Self-supervised learning — labels from the data itself
Training in which the labels come from the data itself, such as predicting the next token of a text. No one has to annotate anything.
A language model is trained on next-token prediction: hide the next token, predict it from the ones before. Every position of every text is a labelled example for free, which is why the training signal grows with the supply of text (section 4.1). Masked-token prediction (BERT) and autoencoders are other self-supervised tasks.