Softmax
Read "softmax of z". It turns any list of numbers into probabilities: every output is positive and they add up to 1. Bigger inputs get bigger shares.
. Taking to the power of each input makes every value positive. Dividing by the total makes them add up to 1.
It is called soft because it does not just pick the largest input (that would be a "hard" max). Larger inputs get a larger share, but every entry keeps some probability above zero. Only the differences between inputs matter: adding the same constant to every input leaves the output unchanged.
Example
A difference of 1 between two inputs becomes a ratio of e ≈ 2.7 between their probabilities: 0.665 / 0.245 ≈ 2.7.
Where the course uses it
- 4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Datafirst use
- 5.6 The First Attention · Module 5, Sequences Before Transformers
- 6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
- 8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 12.2 The KV Cache · Module 12, Generation
- 15.2 Stretching RoPE, and ALiBi · Module 15, The LLaMA Block
- 20.4 Distillation · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
Related terms
- The exponential functionRead "e to the z" (also written exp(z)). It raises e ≈ 2.718 to the power z. The result is always positive: e⁰ = 1, and it shrinks toward 0 as z becomes very negative.
- ∑ — sumRead "the sum of". The big sigma is a loop that adds: the counter starts at the value below it and stops at the value above it.
- TemperatureRead "z i over T". Each logit is divided by the temperature T before softmax. T below 1 makes the distribution sharper; T above 1 makes it flatter.