Logits — raw scores
The raw scores a model produces before softmax turns them into probabilities. There is one per vocabulary entry. They can be any real number, negative or positive.
The last layer of a language model outputs a vector with one entry per token in the vocabulary. A larger logit means that token is more likely to come next, but logits are not probabilities: they do not add to 1 and can be negative. Softmax converts them (section 4.3), and temperature divides them first to sharpen or flatten the result.
Example
Order is kept; the scores become shares that add to 1 (0.705 + 0.259 + 0.035).
Where the course uses it
Related terms
- SoftmaxRead "softmax of z". It turns any list of numbers into probabilities: every output is positive and they add up to 1. Bigger inputs get bigger shares.
- TemperatureRead "z i over T". Each logit is divided by the temperature T before softmax. T below 1 makes the distribution sharper; T above 1 makes it flatter.
- |V| — size of the vocabularyRead "the size of V". Single bars around a set mean "how many items it has", so |V| is the number of tokens in the vocabulary.