Entropy — average surprise
The average surprise of a distribution, H(p) = −Σ p log p: zero if one outcome is certain, largest when all outcomes are equally likely.
Measured in bits with log base 2 (section 3.5). It is the floor that cross-entropy can never go below.
Where the course uses it
Related terms
- Cross-entropy — the training lossThe average surprise when the truth is p but you predict with q. Against a one-hot target it is minus the log of the probability given to the correct class.
- KL — Kullback–Leibler divergenceHow different two probability distributions are: zero when they agree, larger the more they differ. It is not symmetric, so KL(P‖Q) and KL(Q‖P) differ.