\operatorname{KL}(P \,\|\, Q)

KL — Kullback–Leibler divergence

How different two probability distributions are: zero when they agree, larger the more they differ. It is not symmetric, so KL(P‖Q) and KL(Q‖P) differ.

$\operatorname{KL}(P \,\|\, Q) = \sum_x P(x) \log \frac{P(x)}{Q(x)}$ . In RLHF (section 21.5) it measures how far the trained model has drifted from the model it started from, and the objective subtracts it as a penalty so the model stays fluent while it chases reward.

Example

P = [0.5, 0.5], \; Q = [0.9, 0.1] \;\Longrightarrow\; \operatorname{KL}(P \,\|\, Q) \approx 0.51

0.5 ln(0.5/0.9) + 0.5 ln(0.5/0.1) ≈ −0.29 + 0.80.

Where the course uses it

3.5 Information: Entropy, Cross-Entropy, KL, Perplexity · Module 3, Probability and Informationfirst use
20.4 Distillation · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
21.5 RLHF End to End · Module 21, Preference Learning: RL, RLHF and DPO

Related terms

LogarithmRead "log of x". It undoes the exponential: log(eᶻ) = z. It turns products into sums, which is why loss functions use it.