P(w_1 \dots w_T) = \prod_t P(w_t \mid w_{<t})

Chain rule of probability

The probability of a whole sentence equals the probability of each word given the words before it, all multiplied together.

This is exact, not an approximation. It is the chain rule of probability applied from left to right. Its consequence shapes the whole architecture: a model that can predict one next token can score or generate text of any length by repeating that step. A model that works this way is called autoregressive.

Example

P(\text{the cat}) = P(\text{the}) \times P(\text{cat} \mid \text{the})

Two words, two factors. A sentence of T words has T factors.

Where the course uses it

3.3 Joint and Conditional Probability · Module 3, Probability and Informationfirst use
8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder

Related terms

P(a | b) — conditional probabilityThe vertical bar reads "given". P(a | b) is "the probability of a given b": how likely a is when you already know b is true.
∏ — productRead "the product of". The big pi works like ∑, but it multiplies the terms instead of adding them.