# Module 8: Three Families: Encoder, Encoder-Decoder, Decoder

> Encoder, encoder-decoder and decoder-only transformers: cross-attention, BERT, T5, the causal mask, Pre-LN versus Post-LN, and why decoder-only won.

- Part II: The Transformer · core track · Intermediate · 4 hours
- By the end you can choose the right family for a task and explain why decoder-only won for generation.
- Builds on: [Module 7: The Block and the Full Model](https://transformers-and-math.drmhse.com/md/07-the-block-and-the-full-model.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/08-three-families
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

The block of Module 7 can be wired into three kinds of model. An *encoder* lets every token see the whole input, which suits understanding text. A *decoder* lets each token see only what came before, which suits writing it. The 2017 Transformer used both, joined by cross-attention. This module covers the three families, why decoder-only won for language models, and a small wiring choice, where the normalization goes, that decides whether deep models train at all.

## 8.1 The 2017 Encoder-Decoder and Cross-Attention

The 2017 Transformer was built for translation: an *encoder* reads the source sentence and a *decoder* writes the translation one token at a time. Each is a stack of six identical layers:

```mermaid
graph TD
    subgraph Encoder["Encoder (6 layers)"]
        E_In["Source tokens + positional encoding"] --> E_Attn["Multi-head self-attention (sees the whole source)"]
        E_Attn --> E_Add1["Add & LayerNorm (after the add, as in 2017)"]
        E_Add1 --> E_FFN["Feed-forward network (d_ff = 4 · d_model)"]
        E_FFN --> E_Add2["Add & LayerNorm"]
        E_Add2 --> E_Out["Encoder output H_enc"]
    end

    subgraph Decoder["Decoder (6 layers)"]
        D_In["Target tokens so far + positional encoding"] --> D_MaskAttn["Masked self-attention (future tokens hidden)"]
        D_MaskAttn --> D_Add1["Add & LayerNorm"]
        D_Add1 --> D_Cross["Cross-attention: queries from decoder, keys and values from H_enc"]
        E_Out --> D_Cross
        D_Cross --> D_Add2["Add & LayerNorm"]
        D_Add2 --> D_FFN["Feed-forward network"]
        D_FFN --> D_Add3["Add & LayerNorm"]
        D_Add3 --> D_Linear["Linear layer (d_model → vocabulary size)"]
        D_Linear --> D_Smax["Softmax → next-token probabilities"]
    end
```

> **In words.** Six encoder layers represent the source, each token seeing the whole sentence. In each of six decoder layers, a target token first attends to the target tokens before it, then across to the encoder's output. Each "Add & LayerNorm" adds a sub-layer's input to its output and normalizes.

### Cross-Attention: Shapes and Mechanics

Cross-attention is how the decoder reads the source. Queries come from the decoder's vectors $H_{\text{dec}} \in \mathbb{R}^{N \times d_{\text{model}}}$ ($N$ target tokens), keys and values from the encoder's output $H_{\text{enc}} \in \mathbb{R}^{M \times d_{\text{model}}}$ ($M$ source tokens):

$$
Q_{\text{cross}} = H_{\text{dec}} W_{\text{cross}}^Q \in \mathbb{R}^{N \times d_k} \quad (\text{Queries come from the Decoder})
$$

$$
K_{\text{cross}} = H_{\text{enc}} W_{\text{cross}}^K \in \mathbb{R}^{M \times d_k} \quad (\text{Keys come from the Encoder})
$$

$$
V_{\text{cross}} = H_{\text{enc}} W_{\text{cross}}^V \in \mathbb{R}^{M \times d_v} \quad (\text{Values come from the Encoder})
$$

The weights form an $N \times M$ table, one row per target token and one column per source token, each entry between 0 and 1:

$$
A_{\text{cross}} = \operatorname{Softmax}\left(\frac{Q_{\text{cross}} K_{\text{cross}}^\top}{\sqrt{d_k}}\right) \in [0, 1]^{N \times M}
$$

**There is no causal mask here.** The decoder's self-attention must hide future *target* tokens, but the source is complete before decoding starts, so every target token may attend to every source token.

## 8.2 Encoder-Only: BERT and Masked Language Modeling

BERT (Devlin et al., 2019) keeps only the *encoder*, with no causal mask, and uses a different self-supervised task: hide about 15% of the tokens and predict them from everything else. With $\mathcal{S}$ the hidden positions and $W_{\setminus \mathcal{S}}$ the text with them hidden:

$$
\mathcal{L}_{\text{MLM}} = -\sum_{i \in \mathcal{S}} \log P_\theta(w_i \mid W_{\setminus \mathcal{S}})
$$

Of the chosen tokens, 80% become a special `[MASK]` token, 10% a random token and 10% stay unchanged, so the model cannot learn that only `[MASK]` positions matter. Seeing both sides makes encoders strong at classification, search, tagging and embeddings, but they cannot generate text left to right, since nothing trains them to predict from the past alone. Encoders reappear wherever a model only has to read: image encoders in Module 25 and text recognizers in Module 27.

## 8.3 Encoder-Decoder After 2017: T5

T5 (Raffel et al., 2020) treats *every* task as text in, text out: translation, summarization, classification (the label written as a word) and question answering share one model and one loss.

Its pretraining task, *span corruption*, sits between BERT's blanks and next-token prediction: random spans are each replaced by one sentinel token, and the decoder writes the missing spans in order:

- input: `Thank you <X> me to your party <Y> week.`
- target: `<X> for inviting <Y> last <Z>`

Encoder-decoders remain strong when input and output are different kinds of sequence and the whole input is available up front: translation, speech recognition (Whisper is one) and summarization. But they spend parameters on both halves, and decoder-only models proved simpler to scale and just as capable once large, which is why this course's language models are decoder-only.

| family | attention inside the model | typical pretraining task | good at |
|---|---|---|---|
| encoder-only (BERT) | every token sees every token | fill in masked tokens | classification, search, embeddings |
| encoder-decoder (T5) | encoder: all; decoder: causal + cross | fill in masked spans | translation, summarization, speech |
| decoder-only (GPT) | causal | predict the next token | generation, chat, general use |

## 8.4 Decoder-Only: Predicting the Next Token

The GPT models keep only the decoder: a stack of masked self-attention blocks trained to predict the next token. GPT-1 (2018), GPT-2 (2019) and GPT-3 (2020) showed that this design keeps improving with size and data, and most large language models since, including Meta's LLaMA family, use it.

By the chain rule of probability (section 3.3), the probability of a text of $T$ tokens $W = (w_1, \dots, w_T)$ is a product of next-token probabilities:

$$
P(w_1, w_2, \dots, w_T) = \prod_{t=1}^T P(w_t \mid w_1, w_2, \dots, w_{t-1})
$$

$\prod$ means "multiply these together". So a model that answers "what is the next token, given everything so far?" can score any text and generate text of any length. Training maximizes the likelihood of real text (section 3.4) by minimizing the **negative log-likelihood** (NLL):

$$
\mathcal{L}_{\text{NLL}}(\theta) = -\sum_{t=1}^T \log P_\theta(w_t \mid w_{<t})
$$

$w_{<t}$ means all tokens before position $t$. This is the cross-entropy loss of section 3.5 applied at every position, with the gradient $\mathbf{p} - \mathbf{y}$ of section 4.3, and $e^{\mathcal{L}_{\text{NLL}}/T}$ is the perplexity.

```mermaid
graph TD
    Tokens["Input tokens: 'The', 'Transformer', 'changed', 'AI'"] --> Embed["Token embeddings + position embeddings"]
    Embed --> CausalMask["Add the causal mask M: minus infinity above the diagonal"]
    CausalMask --> Attn["Masked self-attention: row i sees only columns j ≤ i"]
    Attn --> Proj["Output layer: one score per vocabulary token"]
    Proj --> Loss["Cross-entropy loss at every position, all in one pass"]
```

> **In words.** The whole sequence goes in at once, the mask hides each position's future, and the loss is computed at every position in one pass. Training is parallel even though generation is sequential.

### The Causal Attention Mask Matrix
In training the model sees all $N$ tokens at once, but position $i$ predicts token $i+1$ and must not see it, or it could copy the answer. The **causal mask** of section 6.4 prevents this (row $i$ is the looking token, column $j$ the token looked at):

$$
M_{ij} = \begin{cases} 0, & \text{if } j \le i \\ -\infty, & \text{if } j > i \end{cases}
$$

Added inside the softmax, as in Module 6:

$$
\operatorname{MaskedAttention}(Q, K, V) = \operatorname{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right) V
$$

every future position gets exactly zero weight:

$$
e^{-\infty} = 0 \implies \operatorname{Softmax}\left(\frac{QK^\top}{\sqrt{d_k}} + M\right)_{ij} = 0 \quad \forall j > i
$$

($\forall$ means "for all".) Each row's remaining weights still sum to 1. So one parallel pass gives every position the view it would have during generation, and a 1,024-token training sequence yields 1,024 predictions and loss terms from a single forward pass.

> **Interactive figure** `mask-explorer`, on the web page: https://transformers-and-math.drmhse.com/modules/08-three-families#figure-mask-explorer
>
> **Try:** Switch to the causal mask and hover the third row. Before you do, say how many keys it can see, and how many next-token predictions one 6-token training sequence gives.
>
> **What happens:** The third row sees three keys, positions 1 to 3, including itself. A 6-token sequence gives six predictions in one pass, one per row, each made from exactly the tokens that came before it. Compare the bidirectional mask, where every row sees all six, and the cross-attention mask, which is a full rectangle because the source is complete before decoding starts.

> **In words.** An attention mask as a grid, queries as rows and keys as columns, in three settings: bidirectional (encoder), causal (decoder) and cross-attention. Hovering a row highlights the keys it can see and, for the causal mask, names the token that position predicts.

## 8.5 Where the Norm Goes: Post-LN and Pre-LN

LayerNorm can go in one of two places around each residual addition. The 2017 Transformer put it *after*, **Post-LN**:

$$
\mathbf{x}_{l+1} = \operatorname{LayerNorm}(\mathbf{x}_l + \operatorname{SubLayer}(\mathbf{x}_l))
$$

$\operatorname{SubLayer}$ is attention or the feed-forward network, and $l$ counts sub-layers, not blocks. GPT-2 (2019) moved LayerNorm to each sub-layer's *input*, **Pre-LN**, as GPT-3 and LLaMA do (LLaMA with RMSNorm, section 15.3):

$$
\mathbf{x}_{l+1} = \mathbf{x}_l + \operatorname{SubLayer}(\operatorname{LayerNorm}(\mathbf{x}_l))
$$

Pre-LN never normalizes the main path, so it adds one final LayerNorm, `ln_f`, before the output layer. Xiong et al. (2020) explained why Pre-LN trains more easily: in Post-LN the gradients near the output are large at the start, so the learning rate must be ramped up slowly (a **warmup**); Pre-LN can often train without one.

> **In the reference implementation.** `src/transformer/transformer_block.py` uses the Post-LN wiring from 2017, so on this point it differs from GPT-2. Lab 3b runs the two wirings against each other.

```mermaid
graph LR
    subgraph PostLN["Post-LN (Vaswani 2017, this repository)"]
        x1["x_l"] --> Sub1["SubLayer"]
        x1 --> Add1((+))
        Sub1 --> Add1
        Add1 --> LN1["LayerNorm"]
        LN1 --> out1["x_{l+1}"]
    end

    subgraph PreLN["Pre-LN (GPT-2, GPT-3, LLaMA)"]
        x2["x_l"] --> LN2["LayerNorm"]
        x2 --> Add2((+))
        LN2 --> Sub2["SubLayer"]
        Sub2 --> Add2
        Add2 --> out2["x_{l+1}"]
    end
```

> **In words.** Post-LN normalizes the sum, so the normalization sits on the main path. Pre-LN normalizes only the copy entering the sub-layer, leaving the main path a plain addition.

### Mathematical Proof of Gradient Highway in Pre-LN
In Pre-LN nothing on the residual stream is normalized, so unrolling across $L$ sub-layers gives the input plus every layer's contribution:

$$
\mathbf{x}_L = \mathbf{x}_0 + \sum_{l=0}^{L-1} \operatorname{SubLayer}(\operatorname{LayerNorm}(\mathbf{x}_l))
$$

Differentiate with respect to $\mathbf{x}_0$; the first term gives the identity $I$:

$$
\frac{\partial \mathbf{x}_L}{\partial \mathbf{x}_0} = I + \sum_{l=0}^{L-1} \frac{\partial \operatorname{SubLayer}(\operatorname{LayerNorm}(\mathbf{x}_l))}{\partial \mathbf{x}_0}
$$

$I$ depends on no layer's weights, so part of the gradient always reaches the bottom unscaled. In the RNN of section 5.2 the gradient is a product of per-step factors; here the layer terms are added to $I$, not multiplied into it. Post-LN has no such clean term, because the gradient passes back through a LayerNorm at every layer.

> **Caveat.** The $I$ term protects only the residual stream, the direct route from $\mathbf{x}_0$ to $\mathbf{x}_L$. The other terms in the sum each pass through a LayerNorm and a sub-layer, and those still rescale the gradient.

> **Measured** (lab 3b, `experiments/ab_layernorm.py`). The same 49,792 parameters, wired two ways, give outputs that differ by $0.83$ at initialization. The identity path shows up as a gradual trend, not a sudden break. As depth grows from 4 to 32 layers, the gradient reaching the embeddings changes by a factor of $0.9749$ under Post-LN and grows by a factor of $1.6227$ under Pre-LN, so the ratio between the two widens from $1.5$ to $2.5$.

> **Measured** (lab 3b). At this size the effect is modest: at 32 layers Post-LN's gradient is 2.5 times smaller than Pre-LN's, not gone. A 600-step training comparison at two layers separates the two wirings by $0.015$ validation loss, which is within the run-to-run noise of a single seed.

## Exercises

**Exercise (check).** Which family would you choose for (a) classifying support tickets into ten categories, (b) translating documents, (c) a chat assistant? Give one reason each.

*Solution:* (a) Encoder-only: the whole ticket is available and only needs to be understood, so two-sided attention helps and nothing must be generated. (b) Encoder-decoder, or a large decoder-only model: the source is complete before the translation starts, which an encoder exploits. (c) Decoder-only: chat is open-ended generation, which is exactly what next-token training produces.

**Exercise (apply).** Write out the causal mask for four tokens and say how many entries are $-\infty$.

*Answer:* 6 (within 0)

*Solution:* Every entry above the diagonal is $-\infty$: rows 1 to 3 have 3, 2 and 1 such entries, so $3 + 2 + 1 = 6$. In general an $N \times N$ causal mask blocks $N(N-1)/2$ entries.

**Exercise (apply).** A decoder-only model is trained on one sequence of 2,048 tokens. How many next-token predictions contribute to the loss in that single forward pass?

*Answer:* 2047 (within 1)

*Solution:* Each position predicts the next token, so positions 1 to 2,047 each have a target; the last position has no next token in the sequence. (Implementations often count 2,048 by shifting the targets by one and dropping the last prediction, which is the same count of real predictions.)

**Exercise (apply).** Unroll the Pre-LN update $\mathbf{x}_{l+1} = \mathbf{x}_l + f_l(\operatorname{LN}(\mathbf{x}_l))$ over three sub-layers and write $\mathbf{x}_3$ in terms of $\mathbf{x}_0$.

*Solution:* $\mathbf{x}_3 = \mathbf{x}_0 + f_0(\operatorname{LN}(\mathbf{x}_0)) + f_1(\operatorname{LN}(\mathbf{x}_1)) + f_2(\operatorname{LN}(\mathbf{x}_2))$. The input appears on its own, untouched by any layer, which is the identity term of the gradient highway.

**Exercise (extend).** BERT replaces only 80% of its chosen tokens with `[MASK]`, 10% with random tokens and leaves 10% unchanged. What would go wrong if it used `[MASK]` for all of them?

*Solution:* The model would learn that only `[MASK]` positions ever need a prediction, and that every other token can be trusted as given. But `[MASK]` never appears when the model is later used on real text, so its representations of ordinary tokens would not have been trained to carry the information needed to predict them. Mixing in random and unchanged tokens forces the model to build a useful representation of every position.

### Core Academic References
- **Vaswani et al. (2017)**: *Attention Is All You Need*. NeurIPS 2017. [arXiv:1706.03762](https://arxiv.org/abs/1706.03762)
- **Devlin et al. (2019)**: *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. NAACL 2019. [arXiv:1810.04805](https://arxiv.org/abs/1810.04805)
- **Raffel et al. (2020)**: *Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer* (T5). JMLR 21(140): 1–67. [arXiv:1910.10683](https://arxiv.org/abs/1910.10683)
- **Radford et al., OpenAI (2018)**: *Improving Language Understanding by Generative Pre-Training* (GPT-1). Technical Report.
- **Radford et al., OpenAI (2019)**: *Language Models are Unsupervised Multitask Learners* (GPT-2). Technical Report.
- **Radford et al., OpenAI (2023)**: *Robust Speech Recognition via Large-Scale Weak Supervision* (Whisper). ICML 2023. [arXiv:2212.04356](https://arxiv.org/abs/2212.04356)
- **Xiong et al. (2020)**: *On Layer Normalization in the Transformer Architecture* (Pre-LN analysis). ICML 2020. [arXiv:2002.04745](https://arxiv.org/abs/2002.04745)