The meaning of a word depends on the words around it, sometimes many words back. Before 2017, sequences were modelled by recurrent networks, which read one token at a time and carry a running summary forward. This module explains how they work, the two walls they hit (no parallelism, and gradients that vanish over long distances), and the fix invented inside them: attention. Module 6 shows what happens when attention is all that is left.

5.1 Word Vectors and What Similarity Means

After training, the embedding vectors of section 1.5 carry meaning in their directions. Nobody labels which words are related: tokens in similar contexts receive similar updates, so their vectors drift together, and "cat" ends up closer to "dog" than to "carburetor". Word2vec (Mikolov et al., 2013) showed that some relations become directions: "king" minus "man" plus "woman" lands near "queen". Inside large language models these regularities are rougher and differ by layer, but direction encoding meaning is what makes the dot product a sensible score.

Figure 5.1

Direction carries meaning

Pick two words to see the angle between them. Then watch the word2vec analogy: the step from man to woman, added to king, lands near queen.

Illustrative 2D positions chosen by hand. Real embeddings have hundreds of dimensions, and the analogy holds only approximately.

Predict, then try

The arrow from man to woman is copied to start at king. Before looking, predict where it ends, and what it would mean if it ended far from queen.

In words. Words as arrows in a plane: cat and dog point nearly the same way, cat and carburetor almost opposite. The arrow from man to woman, moved to start at king, ends close to queen.

5.2 Recurrent NetworksRun this section

Recurrent neural networks (RNNs), including the Elman RNN and its gated variants LSTM and GRU, read one token at a time and keep a hidden state , a fixed-length summary of everything read so far, computed from the previous state and the current token:

\mathbf{h}_t = \tanh(W_{hh} \mathbf{h}_{t-1} + W_{xh} \mathbf{x}_t + \mathbf{b})

$W_{hh}$ , $W_{xh}$ and $\mathbf{b}$ are learned. This section writes vectors as columns, as the RNN papers do, so matrices multiply from the left.

The same weights apply at every step, so an RNN reads any length. But the state has a fixed size: everything about the first 500 words must fit in the same few hundred numbers as the first 5.

Architecture Flowchart

In words. The hidden state passes along a chain, one step per token; the first token reaches the third only through the second. The chain causes both problems below: steps cannot run at the same time, and gradients must travel back along the whole chain.

5.3 No Parallelism Across Time

Step $t$ needs $\mathbf{h}_{t-1}$ , so an $N$ -token sequence takes $N$ matrix multiplications one after another: sequential steps. A GPU that could do thousands at once sits mostly idle.

5.4 Vanishing and Exploding GradientsRun this section

To learn from a mistake at step $T$ , training needs how much an earlier state $\mathbf{h}_t$ contributed. By the chain rule (section 2.3), that is a product of one Jacobian per step in between:

\frac{\partial \mathbf{h}_T}{\partial \mathbf{h}_t} = \prod_{k=t+1}^T \frac{\partial \mathbf{h}_k}{\partial \mathbf{h}_{k-1}} = \prod_{k=t+1}^T \operatorname{diag}(1 - \tanh^2(\cdot)) W_{hh}

introduces 4 notations

Each factor is the diagonal Jacobian of $\tanh$ (slopes $1 - \tanh^2$ , never more than 1; the $(\cdot)$ is whatever was inside the $\tanh$ at step $k$ ) times $W_{hh}$ .

Let be the most one multiplication by $W_{hh}$ can stretch a vector, the largest (formally the largest singular value, section 20.2). The $\tanh$ factor never stretches, so after $T - t$ steps:

\left\|\frac{\partial \mathbf{h}_T}{\partial \mathbf{h}_t}\right\| \le c \cdot \sigma_{\max}^{T-t} \xrightarrow[T-t \to \infty]{} 0

introduces 2 notations

$T$ here is a step number, not a temperature, and $c$ is a constant independent of distance. If $\sigma_{\max} < 1$ the bound shrinks exponentially: . This is the vanishing gradient; factors above 1 give the exploding gradient (section 2.6). LSTMs ease it with a memory path updated by addition rather than repeated multiplication, but still pass information between tokens $N$ apart through $\mathcal{O}(N)$ steps.

Browser lab

How a gradient shrinks travelling back through an RNN

A random 32-unit tanh RNN runs for 60 steps. A gradient of size 1 at the last step is then carried backward, multiplied by each step’s Jacobian. Section 5.4 bounds its size after k steps by σ_max^k. The lab sets σ_max (the largest stretch of W_hh) to four values and measures what actually arrives.

The bound is a ceiling, not a prediction: the tanh slopes are below 1 and the gradient is rarely aligned with the largest stretch, so the measured sizes usually fall faster than σ_max^k. With large σ_max the tanh saturates, its slopes collapse, and even a stretching matrix can end up vanishing.

5.5 Seq2seq and the Fixed-Vector Bottleneck

In translation, input and output are sequences of different lengths with words in different orders. Sutskever, Vinyals and Le (2014) used two RNNs, sequence-to-sequence (seq2seq):

The encoder reads the source sentence; its final hidden state summarizes it.
The decoder starts from that summary and writes the translation one token at a time, feeding each output back in, until an end-of-sentence token.

It worked well on short sentences and badly on long ones, because the only link between the halves is one fixed-size vector: 5 words or 50 must fit in the same few hundred numbers. This fixed-vector bottleneck made quality fall as sentences grew.

5.6 The First AttentionRun this section

Bahdanau, Cho and Bengio (2014) removed the bottleneck: let the decoder look back at every encoder state $\mathbf{h}_1, \dots, \mathbf{h}_M$ and choose which to read at each step. Before writing output word $i$ , with its own state , the decoder:

Scores every encoder state with a small learned network, $e_{ij} = \operatorname{score}(\mathbf{s}_{i-1}, \mathbf{h}_j)$ .
Turns the scores into weights summing to 1 with a softmax: .
Reads their weighted average, the context vector:

\mathbf{c}_i = \sum_{j=1}^{M} \alpha_{ij} \mathbf{h}_j

introduces 1 notation

and uses $\mathbf{c}_i$ , alongside its own state, to choose the next word.

Nobody tells the model which words correspond; the weights are learned from the translation loss alone. Yet plotted, they line up: writing the French zone, the decoder weights the English area, and where word order differs, the weights cross over to follow it.

Figure 5.6

Where the decoder looks while writing each word

Rows are the French words being written; columns are the English source. Click a row to see the weights it uses.

Weights set by hand to reproduce the pattern in Bahdanau et al. (2014), whose trained model produced the same crossing for “European Economic Area” / “zone économique européenne”. Each row sums to 1.

Predict, then try

In a French–English pair, adjectives usually come after the noun in French and before it in English. Before looking, predict what that does to the shape of the alignment pattern.

In words. A grid with the source sentence across the top and the translation down the side; each cell's shade is the weight on that source word while writing that output word. Most bright cells run along the diagonal, with detours where word order differs.

This first attention mechanism solved the bottleneck: the decoder could reach any source word directly. But the RNNs underneath kept both their problems. In 2017, Vaswani et al. asked whether the recurrence was needed at all.

The Transformer Breakthrough: Maximum Path Length $\mathcal{O}(1)$

It was not. The transformer keeps only attention: between decoder and encoder, as Bahdanau had it, and within each sequence, every token attending to the others (self-attention). That removes the chain:

Path length. Any two tokens are one step apart, $\mathcal{O}(1)$ , so no long product of factors builds up.
Parallelism. In training, all tokens are processed at once as a few large matrix multiplications. (Generation is still one token at a time.)

Exercises

Check

Name the two problems of recurrent networks that the transformer removes, and say which design feature causes both.

Apply

A gradient passes back through 40 RNN steps, each scaling it by at most $\sigma_{\max} = 0.85$ . What is the largest it can be, as a fraction of its original size?

Apply

How many sequential steps does an RNN need to process a 1,000-token sequence, and how many sequential attention steps does one self-attention layer need for the same sequence?

Apply

In Bahdanau attention, the decoder's weights over four source words are $[0.1, 0.6, 0.2, 0.1]$ and the encoder states are the scalars $[1, 3, -1, 2]$ . Compute the context value.

Extend

Explain what information a seq2seq decoder loses on a 200-word sentence, and why attention fixes it without making the encoder's states any larger.

Core Academic References

Mikolov et al. (2013): Efficient Estimation of Word Representations in Vector Space (word2vec). arXiv:1301.3781
Elman (1990): Finding Structure in Time. Cognitive Science, 14(2): 179–211.
Hochreiter & Schmidhuber (1997): Long Short-Term Memory. Neural Computation, 9(8): 1735–1780.
Bengio, Simard & Frasconi (1994): Learning long-term dependencies with gradient descent is difficult. IEEE Transactions on Neural Networks, 5(2): 157–166.
Sutskever, Vinyals & Le (2014): Sequence to Sequence Learning with Neural Networks. NeurIPS 2014. arXiv:1409.3215
Bahdanau, Cho & Bengio (2014): Neural Machine Translation by Jointly Learning to Align and Translate. ICLR 2015. arXiv:1409.0473

5.1 Word Vectors and What Similarity Means

5.2 Recurrent NetworksRun this section

5.3 No Parallelism Across Time

5.4 Vanishing and Exploding GradientsRun this section

How a gradient shrinks travelling back through an RNN

5.5 Seq2seq and the Fixed-Vector Bottleneck

5.6 The First AttentionRun this section

The Transformer Breakthrough: Maximum Path Length O(1)\mathcal{O}(1)O(1)

Exercises

Core Academic References

The Transformer Breakthrough: Maximum Path Length $\mathcal{O}(1)$