# Module 3: Probability and Information

> Expectation, variance, conditional probability, maximum likelihood, entropy, cross-entropy, perplexity and KL divergence, as language models use them.

- Part I: Foundations · core track · Foundational · 4 hours
- By the end you can use the rules for expectation and variance, read conditional probability notation, and explain cross-entropy, perplexity and KL divergence.
- Builds on: [Module 2: Calculus and Optimization](https://transformers-and-math.drmhse.com/md/02-calculus-and-optimization.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/03-probability-and-information
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

A language model's output is a probability distribution, its training objective is a statement about probabilities, and one of its key design choices (section 6.3) rests on a calculation with averages and spreads. This module teaches the probability the course uses, in the order it uses it. Nothing here is specific to transformers, and every later module leans on it.

## 3.1 Random Variables, Distributions, and Sampling

A *random variable* is a quantity whose value is uncertain until you observe it: the face of a die, the next token of a text, the initial value of a weight. Its *distribution* says how likely each value is.

For a variable with a list of possible values, the distribution gives each value a probability between 0 and 1, summing to 1. A fair die gives each face $\tfrac{1}{6}$; a language model gives one probability to each of its $|V|$ tokens. $P(X = x)$, or just $p(x)$, is "the probability that $X$ comes out as $x$".

A variable that can take any real value, such as a weight, is described by a *density* $p(x)$ instead: the probability of landing in a small interval around $x$ is about $p(x)$ times the interval's width, and the total area under $p$ is 1. The most important density is the *normal distribution* (the bell curve), written $\mathcal{N}(\mu, \sigma^2)$:

$$
p(x) = \frac{1}{\sqrt{2\pi\sigma^2}} \exp\left(-\frac{(x - \mu)^2}{2\sigma^2}\right)
$$

It is centred on $\mu$ with width set by $\sigma$: about 68% of draws land within one $\sigma$ of $\mu$, and 95% within two. Weights are usually initialized from a normal distribution with $\mu = 0$ and small $\sigma$ (section 10.4). $x \sim \mathcal{N}(0, 1)$ reads "$x$ is drawn from a normal distribution with mean 0 and variance 1".

*Sampling* draws a value at random according to the distribution. Generating text is sampling, one token at a time (section 12.3).

Two random variables are *independent* when knowing one tells you nothing about the other: two dice rolls are, the first and second tokens of a sentence are not.

## 3.2 Expectation and Variance

The *expectation* $\mathbb{E}[X]$ is the average value of $X$, weighting each value by its probability. It is also called the *mean*:

$$
\mathbb{E}[X] = \sum_x x \, p(x)
$$

For a fair die, $\mathbb{E}[X] = \tfrac{1}{6}(1 + 2 + 3 + 4 + 5 + 6) = 3.5$. (For a density the sum becomes an integral, and the rules below still hold.) The average of many independent samples settles toward the expectation.

The *variance* is the average squared distance from the mean. Its square root, the *standard deviation* $\sigma$, is in the same units as $X$:

$$
\operatorname{Var}(X) = \mathbb{E}\left[(X - \mathbb{E}[X])^2\right] = \mathbb{E}[X^2] - \left(\mathbb{E}[X]\right)^2
$$

The second form, "the average of the square minus the square of the average", is usually easier to compute.

### The Rules
These rules are used repeatedly, most importantly in section 6.3. Here $a$ and $b$ are fixed numbers.

1. **Expectation is linear, always.** $\mathbb{E}[aX + bY] = a\,\mathbb{E}[X] + b\,\mathbb{E}[Y]$, whether or not $X$ and $Y$ are independent.
2. **For independent variables, the expectation of a product is the product of the expectations.** $\mathbb{E}[XY] = \mathbb{E}[X]\,\mathbb{E}[Y]$.
3. **Scaling a variable scales its variance by the square.** $\operatorname{Var}(aX) = a^2 \operatorname{Var}(X)$: doubling every value quadruples the squared distances. Adding a constant leaves the variance unchanged.
4. **For independent variables, variances add.** $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$. So a sum of $n$ independent variables with variance $s^2$ each has standard deviation $s\sqrt{n}$: spread grows like the *square root* of the number of terms.

By rules 3 and 4, the average of $n$ independent samples has standard deviation shrinking like $1/\sqrt{n}$. A gradient on a random batch is such an average: by rule 1 its expectation is the full-dataset gradient, and its noise shrinks as the batch grows. A quantity whose expectation equals what it estimates is *unbiased*.

> **Interactive figure** `variance-sampler`, on the web page: https://transformers-and-math.drmhse.com/modules/03-probability-and-information#figure-variance-sampler
>
> **Try:** The figure draws two random vectors of length $d$, every entry from $\mathcal{N}(0, 1)$, and records their dot product. Using rules 2 and 4, predict the standard deviation of the dot product at $d = 64$.
>
> **What happens:** Each product $q_i k_i$ has mean 0 and variance $1 \cdot 1 = 1$, and there are 64 independent terms, so the variance is 64 and the standard deviation is $\sqrt{64} = 8$. The histogram's spread grows like $\sqrt{d}$ as you raise $d$. Section 6.3 shows why attention cannot tolerate that growth, and divides it away.

> **In words.** The figure repeatedly draws two random vectors of a chosen length and adds their dot product to a histogram, over a normal curve with variance equal to the length. The measured standard deviation tracks the predicted square root of the length as the length grows.

## 3.3 Joint and Conditional Probability

The *joint probability* $P(A, B)$ is the probability that both $A$ and $B$ happen. The *conditional probability* $P(A \mid B)$, "the probability of $A$ given $B$", is the share of $B$'s probability in which $A$ also happens:

$$
P(A \mid B) = \frac{P(A, B)}{P(B)}
$$

> **Worked example.** In a corpus, the word `the` makes up 5% of tokens, and `the` followed by `cat` makes up 0.01% of adjacent pairs. Then $P(\texttt{cat} \mid \texttt{the}) = 0.0001 / 0.05 = 0.002$: of all the places a `the` appears, `cat` follows in 0.2% of them.

Rearranged, $P(A, B) = P(B)\,P(A \mid B)$. Applied repeatedly, this gives the *chain rule of probability*: a joint probability is a product of one-at-a-time conditionals.

$$
P(w_1, w_2, w_3) = P(w_1)\, P(w_2 \mid w_1)\, P(w_3 \mid w_1, w_2)
$$

This is exact, and it is the foundation of every text-writing model in this course: a model that answers "how likely is the next token, given everything before it?" can multiply its answers into the probability of a whole text (section 8.4).

Independence means $P(A \mid B) = P(A)$, or equivalently $P(A, B) = P(A)\,P(B)$.

## 3.4 Maximum Likelihood

A model with *parameters* $\theta$ assigns a probability to each possible observation. The *likelihood* of $\theta$ is the probability it gives the data actually observed, and *maximum likelihood* chooses the $\theta$ that makes it largest.

For independent observations the likelihood is a product, which underflows in floating point and is awkward to differentiate. The logarithm turns it into a sum, and since the log only increases, maximizing it picks the same $\theta$. Flipping the sign gives the *negative log-likelihood* (NLL), a loss to minimize:

$$
\mathcal{L}_{\text{NLL}}(\theta) = -\sum_{i=1}^{n} \log p_\theta(x_i)
$$

Every log of a probability is at most 0, so the loss is positive, and zero only when every observation gets probability 1.

> **Worked example.** A coin with unknown heads-probability $p$ is flipped 10 times and lands heads 7 times. The likelihood is $p^7 (1 - p)^3$, so the NLL is $-7 \log p - 3 \log(1 - p)$. Its derivative is $-\frac{7}{p} + \frac{3}{1 - p}$. Setting it to zero gives $7(1 - p) = 3p$, so $p = 0.7$: the maximum-likelihood estimate is the observed frequency.

With many values, value $k$ seen $n_k$ times out of $n$ gets probability $n_k / n$. A language model is fitted by the same principle, but its probabilities come from a neural network, so there is no formula for the best $\theta$ and gradient descent (section 2.4) does the minimizing.

> **Browser lab** `mle-coin`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/03-probability-and-information#lab-mle-coin

## 3.5 Information: Entropy, Cross-Entropy, KL, Perplexity

The *surprise* of an outcome with probability $p$ is $-\log p$: 0 when $p = 1$, growing without limit as $p$ shrinks. With base-2 logs it is in *bits* (a fair coin flip carries 1 bit); with natural logs, *nats*.

The *entropy* of a distribution is its average surprise: how unpredictable it is.

$$
H(p) = -\sum_x p(x) \log p(x)
$$

A distribution that puts all its probability on one value has entropy 0. A uniform distribution over $K$ values has the largest possible entropy, $\log K$.

If the true distribution is $p$ but you predict with $q$, your average surprise, using $q$'s probabilities over what really happens, is the *cross-entropy*:

$$
H(p, q) = -\sum_x p(x) \log q(x)
$$

It is never smaller than $H(p)$, and equals it only when $q = p$. The excess is the *Kullback–Leibler divergence*, a measure of how far $q$ is from $p$:

$$
\operatorname{KL}(p \,\|\, q) = H(p, q) - H(p) = \sum_x p(x) \log \frac{p(x)}{q(x)}
$$

It is never negative, is 0 only when the distributions agree, and is not symmetric: $\operatorname{KL}(p \| q)$ and $\operatorname{KL}(q \| p)$ usually differ. Section 21.5 uses it to stop a model drifting far from where it started.

### Why This Is the Training Loss
For one training example, the "true distribution" over the next token is one-hot: probability 1 on the token $c$ that actually came next. The cross-entropy collapses to one term:

$$
H(\text{one-hot}_c, q) = -\log q(c)
$$

That is the NLL of section 3.4. **Minimizing cross-entropy, minimizing NLL and maximizing likelihood are the same thing.** This *cross-entropy loss* trains every classifier and language model in this course.

### Perplexity
*Perplexity* is $\exp$ of the average NLL per token. A perplexity of 20 means the model is as unsure as if choosing uniformly among 20 tokens; 1 is perfect. Section 13.1 explains why perplexities across different tokenizers cannot be compared.

> **Interactive figure** `entropy-explorer`, on the web page: https://transformers-and-math.drmhse.com/modules/03-probability-and-information#figure-entropy-explorer
>
> **Try:** Drag the model's distribution $q$ until it matches the true distribution $p$ exactly. What will the cross-entropy and the KL divergence read then?
>
> **What happens:** The KL divergence reads 0 and the cross-entropy equals the entropy of $p$. That entropy is a floor no model can beat: even a perfect model is still surprised by a genuinely random outcome. Section 10.5 meets the same floor as the "irreducible loss" in scaling laws.

> **In words.** Two bar charts over four outcomes: a fixed true distribution and a draggable model distribution. The readout shows entropy, cross-entropy and KL divergence. Cross-entropy never drops below entropy, and the gap, the KL divergence, closes only when the bars match.

## 3.6 The Sigmoid and Preferences Between Pairs

The *sigmoid* squashes any real score into a probability between 0 and 1:

$$
\sigma(z) = \frac{1}{1 + e^{-z}}
$$

$\sigma(0) = 0.5$, and $\sigma(-z) = 1 - \sigma(z)$. Its slope is $\sigma(z)(1 - \sigma(z))$, the "$p(1-p)$" from Module 2's exercises, because the sigmoid *is* the two-class softmax: $\frac{e^a}{e^a + e^b} = \sigma(a - b)$.

That makes it the natural model for comparisons. Give each item a score $r$, and say item $i$ beats item $j$ with probability

$$
P(i \succ j) = \sigma(r_i - r_j)
$$

Only the *difference* matters, so adding a constant to every score changes nothing. This is the *Bradley–Terry* model, long used to rate chess players from match results by maximum likelihood. Section 21.4 fits it to people's judgments of which of two answers is better, and calls the scores *rewards*.

## Exercises

**Exercise (check).** Which of these need the two variables to be independent? (a) $\mathbb{E}[X + Y] = \mathbb{E}[X] + \mathbb{E}[Y]$; (b) $\mathbb{E}[XY] = \mathbb{E}[X]\mathbb{E}[Y]$; (c) $\operatorname{Var}(X + Y) = \operatorname{Var}(X) + \operatorname{Var}(Y)$.

*Solution:* (b) and (c). Linearity of expectation, (a), holds for any two variables.

**Exercise (apply).** Prove that $\operatorname{Var}(aX) = a^2 \operatorname{Var}(X)$ from the definition $\operatorname{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$.

*Solution:* $\operatorname{Var}(aX) = \mathbb{E}[a^2 X^2] - (\mathbb{E}[aX])^2$. By linearity, $\mathbb{E}[a^2X^2] = a^2\mathbb{E}[X^2]$ and $\mathbb{E}[aX] = a\mathbb{E}[X]$. So $\operatorname{Var}(aX) = a^2\mathbb{E}[X^2] - a^2(\mathbb{E}[X])^2 = a^2 \operatorname{Var}(X)$.

**Exercise (apply).** $X$ is a sum of 256 independent terms, each with mean 0 and variance 1. What is its standard deviation?

*Answer:* 16 (within 0)

*Solution:* Variances of independent terms add: $256 \times 1 = 256$. The standard deviation is $\sqrt{256} = 16$.

**Exercise (apply).** Factor $P(a, b, c, d)$ with the chain rule of probability, in left-to-right order.

*Solution:* $P(a)\,P(b \mid a)\,P(c \mid a, b)\,P(d \mid a, b, c)$. Each factor conditions on everything before it, exactly as a language model predicts each token from the tokens before it.

**Exercise (apply).** A model gives probability $\tfrac{1}{20}$ to every token of a test text. What is its perplexity?

*Answer:* 20 (within 0)

*Solution:* The NLL per token is $-\log(1/20) = \log 20$, so the perplexity is $e^{\log 20} = 20$. This is where the reading "as unsure as choosing among 20 tokens" comes from.

**Exercise (apply).** What is the entropy, in bits, of a distribution that gives probability $\tfrac{1}{2}, \tfrac{1}{4}, \tfrac{1}{4}$ to three outcomes?

*Answer:* 1.5 (within 0.001)

*Solution:* $H = -\left(\tfrac12 \log_2 \tfrac12 + \tfrac14 \log_2 \tfrac14 + \tfrac14 \log_2 \tfrac14\right) = \tfrac12 \cdot 1 + \tfrac14 \cdot 2 + \tfrac14 \cdot 2 = 1.5$ bits.

**Exercise (apply).** Show that $\operatorname{KL}(p \| p) = 0$, and compute $\sigma(2) + \sigma(-2)$.

*Answer:* 1 (within 0)

*Solution:* $\operatorname{KL}(p\|p) = \sum_x p(x) \log \frac{p(x)}{p(x)} = \sum_x p(x) \log 1 = 0$. And by symmetry $\sigma(-2) = 1 - \sigma(2)$, so the sum is exactly 1.

**Exercise (extend).** Under the Bradley–Terry model, answer A has reward 1.5 and answer B has reward 0.5. What is the predicted probability that A is preferred, and what would it be if both rewards were raised by 10?

*Answer:* 0.731 (within 0.001)

*Solution:* $\sigma(1.5 - 0.5) = \sigma(1) = 1/(1 + e^{-1}) \approx 0.731$. Raising both by 10 leaves the difference at 1, so the probability is unchanged. A reward model's scores therefore only mean anything relative to each other.

### Core Academic References
- **Cover & Thomas (2006)**: *Elements of Information Theory (2nd ed.)*. Wiley.
- **Shannon (1948)**: *A Mathematical Theory of Communication*. Bell System Technical Journal, 27: 379–423.
- **Bradley & Terry (1952)**: *Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons*. Biometrika, 39(3/4): 324–345.
- **Goodfellow, Bengio & Courville (2016)**: *Deep Learning*, chapter 3 (Probability and Information Theory). MIT Press.