# Module 1: Shapes and Linear Algebra

> The linear algebra transformers run on: shapes and tensors, matrix multiplication by hand, dot products, embeddings, projections and number storage.

- Part I: Foundations · core track · Foundational · 5 hours
- By the end you can read any shape in the course, multiply matrices by hand, and explain what a projection keeps and throws away.
- Builds on: [Module 0: Start Here: What an LLM Does](https://transformers-and-math.drmhse.com/md/00-start-here.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/01-shapes-and-linear-algebra
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

A transformer is arithmetic on tables of numbers. This module gives you the toolkit to follow it: read a symbol such as $X \in \mathbb{R}^{N \times d_{\text{model}}}$ and know what it holds, multiply matrices by hand, and say what a multiplication keeps and what it throws away.

## 1.1 Scalars, Vectors, Matrices, and Tensors

Everything a transformer reads, stores and produces comes in one of four arrangements, which differ in their number of *axes*: directions you can step along from one number to the next. Know a symbol's arrangement and the length of each axis, and you can follow most formulas before you know what they compute.

### Scalar: One Number
A *scalar* is a single number. It has no axes, because there is nothing to step along. The temperature that sets how random a model's output is (section 4.3), the step size used in training (section 2.4) and the loss that measures a model's error are all scalars. The course writes a scalar as a plain italic letter, and writes "$x$ is one real number" as $x \in \mathbb{R}$.

### Vector: A List
A *vector* is a list of numbers in a fixed order. It has one axis, and its length is how many numbers it holds:

$$
\mathbf{x} = [0.2,\, -1.7,\, 3.0,\, 0.5] \in \mathbb{R}^{4}
$$

Vectors are written in bold. The raised $4$ is the length of the list, not a power. One index picks one entry: $x_3 = 3.0$ is the third number. The entry is written plain rather than bold because it is a scalar. Section 1.5 shows how every token of text becomes a vector like this one, only hundreds or thousands of numbers long.

### Matrix: A Table
A *matrix* is a table of numbers with two axes, rows and columns. Take the three tokens of "the cat sat", give each a vector of four numbers, and stack the vectors as rows:

$$
X = \begin{bmatrix} 0.2 & -1.7 & 3.0 & 0.5 \\ 1.1 & 0.4 & -0.3 & 2.2 \\ -0.6 & 0.9 & 1.5 & 0.0 \end{bmatrix} \in \mathbb{R}^{3 \times 4}
$$

A matrix's size is said rows first, so this is a "3 by 4" matrix holding 12 numbers. Matrices are written as capital letters. Two indices pick one entry, row first: $X_{23} = -0.3$ is row 2, column 3. Row 2 on its own is the vector for "cat".

Some matrices hold data, like $X$, one row per token. Others hold learned numbers, like the $W^Q$ of section 1.6.

### Tensor: A Stack of Tables
Add a third axis and you have a stack of matrices. A model processes several sentences at once, a *batch*, so the data entering a layer is a stack of matrices like $X$, one per sentence:

$$
\mathcal{X} \in \mathbb{R}^{B \times N \times d_{\text{model}}}
$$

Here $B$ is the number of sentences in the batch, $N$ the number of tokens in each, and $d_{\text{model}}$ the length of each token's vector. An array with any number of axes is called a *tensor*. Two sentences of three tokens with four numbers each make a 2 by 3 by 4 tensor, and multiplying the sizes gives the 24 numbers it holds. A color image is a tensor too: height by width by three color channels (section 25.2).

In code, `torch.Tensor` holds all four arrangements (a scalar has zero axes). In prose, "tensor" usually means three axes or more.

> **Interactive figure** `tensor-shapes`, on the web page: https://transformers-and-math.drmhse.com/modules/01-shapes-and-linear-algebra#figure-tensor-shapes
>
> **Try:** A batch of 2 sentences, each 3 tokens long, each token 4 numbers. How many indices do you need to pick out one single number, and how many numbers does the whole tensor hold?
>
> **What happens:** Three indices (sentence, token, feature), and $2 \times 3 \times 4 = 24$ numbers. Click a box in the stack to check: the readout lists the three indices that reach it.

> **In words.** The figure draws one boxed number, a row of boxes, a grid, and a stack of grids like sheets of paper; the depth of the stack is the third axis. Clicking a box shows the indices that reach it (none, one, two or three), and the readout multiplies the axis lengths to count the numbers.

### Reading a Shape
The list of axis lengths is the *shape*. The course writes it as the raised part of $\mathbb{R}$, and PyTorch prints it as a tuple from `x.shape`. Here are the four arrangements with sizes from GPT-2 Small, whose tokens are vectors of 768 numbers:

| arrangement | axes | example | written | PyTorch shape |
|---|---|---|---|---|
| scalar | 0 | the loss | $\mathcal{L} \in \mathbb{R}$ | `()` |
| vector | 1 | one token | $\mathbf{x} \in \mathbb{R}^{768}$ | `(768,)` |
| matrix | 2 | a sentence of $N$ tokens | $X \in \mathbb{R}^{N \times 768}$ | `(N, 768)` |
| tensor | 3 | a batch of $B$ sentences | $\mathcal{X} \in \mathbb{R}^{B \times N \times 768}$ | `(B, N, 768)` |

Sentences in a batch go through the same computation and never mix, so the course drops the batch axis and follows one sequence: attention in Module 6 works on an $N \times d_{\text{model}}$ matrix, and the code puts $B$ in front of every shape.

The course counts indices from 1, as mathematics does. PyTorch counts from 0, so $x_3$ is `x[2]` in code and $X_{23}$ is `X[1, 2]`.

Some texts call the number of axes a tensor's "rank". This course reserves rank for something else (section 1.3) and says "number of axes".

## 1.2 Matrix Multiplication, Done by Hand

Almost every line of a transformer is a matrix multiplication, so do one by hand until the rule is automatic.

Row $i$ of $A$ and column $j$ of $B$, multiplied entry by entry and summed, give entry $(i, j)$ of $AB$. With $n$ the shared length:

$$
(AB)_{ij} = \sum_{k=1}^{n} A_{ik} B_{kj}
$$

The $\sum$ ("sum") lets $k$ run from $1$ to $n$ and adds one term for each value. The row and column must pair up entry for entry, which is why the inner sizes have to agree.

> **Worked example.** Multiply a 2 by 3 matrix by a 3 by 2 matrix:
> $A = \begin{bmatrix} 1 & 2 & 0 \\ -1 & 0 & 3 \end{bmatrix}$, $B = \begin{bmatrix} 2 & 1 \\ 0 & -1 \\ 1 & 4 \end{bmatrix}$.
> Entry $(1,1)$ is row 1 of $A$ with column 1 of $B$: $1 \cdot 2 + 2 \cdot 0 + 0 \cdot 1 = 2$. Entry $(1,2)$ is row 1 with column 2: $1 \cdot 1 + 2 \cdot (-1) + 0 \cdot 4 = -1$. Entry $(2,1)$ is $-1 \cdot 2 + 0 \cdot 0 + 3 \cdot 1 = 1$, and entry $(2,2)$ is $-1 \cdot 1 + 0 \cdot (-1) + 3 \cdot 4 = 11$. So $AB = \begin{bmatrix} 2 & -1 \\ 1 & 11 \end{bmatrix}$, a 2 by 2 matrix.

> **Interactive figure** `matmul-stepper`, on the web page: https://transformers-and-math.drmhse.com/modules/01-shapes-and-linear-algebra#figure-matmul-stepper
>
> **Try:** In the figure, before hovering, decide which row of $A$ and which column of $B$ produce the bottom-right entry of the result. Then check whether changing a number in the *first* row of $A$ can ever change the *bottom* row of the result.
>
> **What happens:** The bottom-right entry comes from the last row of $A$ and the last column of $B$. Changing the first row of $A$ changes only the first row of the result: each row of the output depends on one row of $A$ and all of $B$. That is why a transformer can push every token, one row each, through the same weight matrix without the tokens mixing.

> **In words.** The figure shows two small editable matrices and their product. Pointing at an entry of the product highlights the row and column that produce it and spells out the sum.

Two facts follow. Order matters: $AB$ and $BA$ usually differ, and often only one exists. And each row of $AB$ depends on one row of $A$ only, so multiplying a one-token-per-row matrix by $W$ transforms every token separately, with the same $W$.

### What Operations Do to Shapes
Checking the shape of each output is the quickest way to follow a formula.

- **Entry by entry.** Adding two tensors, or multiplying them entry by entry ($\odot$), needs the same shape and keeps it.
- **By a scalar.** Multiplying by a scalar scales every entry and keeps the shape.
- **Dot product.** Two vectors of the same length give one scalar (section 1.4).
- **Along an axis.** Summing or averaging along an axis removes it: the average of each row of $X$ is a vector of 3 numbers. Softmax (section 4.3) keeps the shape but also works along one axis, turning each row into probabilities separately.
- **Matrix multiplication.** An $m$ by $n$ matrix times an $n$ by $p$ matrix needs the two inner sizes to agree, and gives an $m$ by $p$ matrix:

$$
\underbrace{X}_{N \times d_{\text{model}}} \; \underbrace{W}_{d_{\text{model}} \times d_k} = \underbrace{Y}_{N \times d_k}
$$

The shared $d_{\text{model}}$ disappears: every token's $d_{\text{model}}$ numbers become $d_k$ numbers. That is the shape of every projection in section 1.6.

> **Worked example.** Multiply the 3 by 4 matrix $X$ above by a 4 by 2 matrix $W$. The inner sizes are both 4, so the product exists, and it is 3 by 2: three tokens, two numbers each. $W$ times $X$ is not defined at all, because a row of $W$ holds 2 numbers and a column of $X$ holds 3.

## 1.3 The Matrix Toolkit

These operations recur throughout the course; later sections use them without stopping.

### Transpose
The *transpose* $A^\top$ swaps rows and columns: row $i$ of $A$ becomes column $i$ of $A^\top$, so a 3 by 2 matrix becomes 2 by 3, and $(A^\top)_{ij} = A_{ji}$. A row vector transposed is a column vector. One rule is used repeatedly: the transpose of a product reverses the order, $(AB)^\top = B^\top A^\top$.

### Identity and Diagonal Matrices
The *identity matrix* $I$ is square, with 1 on its diagonal (top-left to bottom-right) and 0 elsewhere. Like the number 1, it changes nothing: $AI = IA = A$.

A *diagonal matrix* $\operatorname{diag}(\mathbf{s})$ has the entries of $\mathbf{s}$ on its diagonal and zeros elsewhere. On the right it scales each column of a matrix by one entry of $\mathbf{s}$; on the left, each row. $\operatorname{diag}(2, 3)$ doubles a vector's first entry and triples its second.

### Inverse
The *inverse* $A^{-1}$ of a square matrix undoes it: $A^{-1}A = AA^{-1} = I$. A matrix that squashes some direction to zero has none, because that information is gone. A diagonal matrix with no zeros is easy to invert: divide instead of multiply, so $\operatorname{diag}(2, 3)^{-1} = \operatorname{diag}(\tfrac{1}{2}, \tfrac{1}{3})$. Sections 24.3 and 24.4 use this.

### Element-Wise Product
The element-wise product $A \odot B$ multiplies entries in matching positions, so both matrices must have the same shape and the result does too. It is not matrix multiplication: $\begin{bmatrix} 1 & 2 \end{bmatrix} \odot \begin{bmatrix} 3 & 4 \end{bmatrix} = \begin{bmatrix} 3 & 8 \end{bmatrix}$. Some papers write it $\otimes$.

### Outer Product
For column vectors $\mathbf{u}$ (length $m$) and $\mathbf{v}$ (length $n$), the *outer product* $\mathbf{u}\mathbf{v}^\top$ is the $m$ by $n$ matrix with entry $(i, j) = u_i v_j$. Every row is a multiple of $\mathbf{v}^\top$, so it is the simplest possible matrix. The reverse order, $\mathbf{u}^\top\mathbf{v}$, is 1 by 1: the dot product. Sections 18.1 and 20.2 build matrices from sums of outer products.

### Independent Directions and Rank
Vectors are *linearly independent* when none can be made by scaling and adding the others: $[1, 0, 0]$ and $[0, 1, 0]$ are, but $[1, 1, 0]$ adds no new direction, being their sum. The *rank* of a matrix is the number of independent directions among its rows (equivalently, its columns), at most the smaller of its two sizes. Section 1.6 uses it to say how much of a token vector a projection keeps.

### The Row-Vector Convention
This course writes a token's vector as a **row** (a 1 by $d_{\text{model}}$ matrix), so a weight matrix multiplies it from the right: $\mathbf{y} = \mathbf{x}W$. PyTorch's `nn.Linear` stores the matrix transposed and computes $\mathbf{x}W^\top$, the same thing. Sections 15.1 and 18.1 switch to **column** vectors, $\mathbf{y} = R\mathbf{x}$, to match their papers, and say so.

## 1.4 Dot Products, Length, and Angle

### Geometric Meaning of the Dot Product
The model compares two vectors with the *dot product* (or *inner product*): multiply entry by entry and add. The same number is the product of the two lengths times the cosine of the angle between them.

$$
\langle \mathbf{u}, \mathbf{v} \rangle = \mathbf{u}^\top \mathbf{v} = \sum_{i=1}^d u_i v_i = \|\mathbf{u}\|_2 \|\mathbf{v}\|_2 \cos(\theta)
$$

$\|\mathbf{u}\|_2$ is the length of $\mathbf{u}$ (the square root of the sum of its squared entries) and $\theta$ the angle between the vectors, which makes sense in any number of dimensions.

```mermaid
graph TD
    A["Vector x_i"] --> Dot["Dot product ⟨x_i, x_j⟩ = ||x_i|| · ||x_j|| · cos(θ)"]
    B["Vector x_j"] --> Dot
    Dot --> Parallel["Same direction (θ = 0°): cos = 1, largest positive score"]
    Dot --> Ortho["Perpendicular (θ = 90°): cos = 0, score is zero"]
    Dot --> Anti["Opposite directions (θ = 180°): cos = −1, most negative score"]
```

> **In words.** Vectors pointing the same way score high, perpendicular vectors score zero, and opposite vectors score negative. The angle sets the sign; the lengths set the size.

Perpendicular vectors are called *orthogonal*: neither has any component along the other.

### Cosine Similarity vs. Unnormalized Dot Product
Divide out both lengths and only direction remains: *cosine similarity*, between $-1$ and $1$.

$$
\operatorname{CosineSim}(\mathbf{u}, \mathbf{v}) = \frac{\mathbf{u}^\top \mathbf{v}}{\|\mathbf{u}\|_2 \|\mathbf{v}\|_2}
$$

Attention uses the raw dot product, so a longer vector produces larger scores. Length becomes a second learned signal alongside direction, for example to make one token's scores sharper than another's. The cost is that raw dot products have no fixed range, which is why Module 2 scales them down.

## 1.5 Embeddings: From a Token to a Vector

A model cannot do arithmetic on text, so each *token* (a word, part of a word, or a punctuation mark) becomes a vector of fixed length:

$$
\mathbf{x} \in \mathbb{R}^{d_{\text{model}}}
$$

The length $d_{\text{model}}$ is the model's *width*: $768$ in GPT-2 Small, $4096$ in LLaMA-3 8B and $12{,}288$ in GPT-3 175B, according to their published descriptions. Section 19.1 compares current models.

### From Text to Vectors: The Embedding Lookup Walk-through
How `"cat"` becomes a vector of length $d_{\text{model}}$:

1. **Tokenization.** A fixed dictionary, the *vocabulary* $V$, gives every token a whole-number ID from $0$ to $|V|-1$, where $|V|$ is the number of entries. Say `"cat"` gets ID $3797$.
2. **One-hot encoding.** On paper, the ID can be written as a *one-hot vector* $\mathbf{e}_i$: a vector of length $|V|$ that is $1$ at position $i$ and $0$ everywhere else.
3. **Row lookup.** The *embedding matrix* $W_E$ has one row per vocabulary entry and $d_{\text{model}}$ columns ($|V| \times d_{\text{model}}$). The one-hot row $\mathbf{e}_i^\top$ times $W_E$ picks out row $i$, because every other row is multiplied by zero.
   $$
   \mathbf{x} = \mathbf{e}_i^\top W_E = W_E[i, :] \in \mathbb{R}^{d_{\text{model}}}
   $$

Real code skips the multiplication and reads row $i$ straight from memory, which costs the same however large the vocabulary: $\mathcal{O}(1)$, read "order one", meaning the cost does not grow with input size. In PyTorch this is `nn.Embedding(vocab_size, d_model)`. The rows start random and are learned in *training*, which nudges the model's numbers, its *weights* (or *parameters*), to make fewer mistakes on example text.

## 1.6 Linear Projections: One Vector Into Three

In attention, each token plays three roles: it looks for other tokens, it is found by them, and it hands information to those that find it. Rather than make one vector do all three jobs, the transformer makes three vectors from each token with three learned matrices:

$$
\mathbf{q}_i = \mathbf{x}_i W^Q \quad (W^Q \in \mathbb{R}^{d_{\text{model}} \times d_k})
$$

$$
\mathbf{k}_i = \mathbf{x}_i W^K \quad (W^K \in \mathbb{R}^{d_{\text{model}} \times d_k})
$$

$$
\mathbf{v}_i = \mathbf{x}_i W^V \quad (W^V \in \mathbb{R}^{d_{\text{model}} \times d_v})
$$

$\mathbf{q}_i$ is the *query*, $\mathbf{k}_i$ the *key* and $\mathbf{v}_i$ the *value* of token $i$. Each matrix reads $d_{\text{model}}$ numbers and writes $d_k$ or $d_v$, usually far fewer. Multiplying by such a matrix is a *linear projection*.

```mermaid
graph LR
    X["Token vector x_i (length d_model)"] --> WQ["× W^Q"] --> Q["Query: what am I looking for?"]
    X --> WK["× W^K"] --> K["Key: what can I be found by?"]
    X --> WV["× W^V"] --> V["Value: what do I pass on?"]
```

> **In words.** One token vector, three learned matrices: the query is what the token looks for, the key what it is matched on, and the value what it passes on.

### Concrete Numerical Toy Example of Q, K, V Projections
Take $d_{\text{model}} = 4$, $d_k = d_v = 2$, two tokens $\mathbf{x}_1 = [1.0, \, 0.0, \, 2.0, \, -1.0]$ and $\mathbf{x}_2 = [0.0, \, 1.0, \, 1.0, \, 1.0]$, and these learned matrices:
$$
W^Q = \begin{bmatrix} 1 & 0 \\ 0 & 1 \\ 1 & -1 \\ 0 & 1 \end{bmatrix}, \quad W^K = \begin{bmatrix} 0 & 1 \\ 1 & 0 \\ -1 & 1 \\ 1 & 0 \end{bmatrix}
$$

Each output entry is the dot product of the input vector with one column of the matrix.

1. **Query for token 1.** The first column of $W^Q$ is $[1, 0, 1, 0]$ and the second is $[0, 1, -1, 1]$:
   $$
   \mathbf{q}_1 = \mathbf{x}_1 W^Q = [1(1) + 0(0) + 2(1) - 1(0), \;\; 1(0) + 0(1) + 2(-1) - 1(1)] = [3.0, \, -3.0]
   $$
2. **Key for token 2.** The first column of $W^K$ is $[0, 1, -1, 1]$ and the second is $[1, 0, 1, 0]$:
   $$
   \mathbf{k}_2 = \mathbf{x}_2 W^K = [0(0) + 1(1) + 1(-1) + 1(1), \;\; 0(1) + 1(0) + 1(1) + 1(0)] = [1.0, \, 1.0]
   $$
3. **Raw score.** Take the dot product of the query and the key:
   $$
   \mathbf{q}_1 \mathbf{k}_2^\top = (3.0 \times 1.0) + (-3.0 \times 1.0) = 3.0 - 3.0 = 0.0
   $$

The raw embeddings have dot product $\mathbf{x}_1 \cdot \mathbf{x}_2 = 0 + 0 + 2 - 1 = 1$, but the query-key score is $0$: the score depends on the learned matrices. Training adjusts $W^Q$ and $W^K$ until tokens that should attend to each other score high.

### Rank and Projection Subspaces
A $d_{\text{model}} \times d_k$ matrix has rank at most $d_k$ when $d_k$ is smaller (section 1.3), so each projection keeps only a $d_k$-dimensional slice of the token vector, and each matrix can learn a different slice:
- the query $\mathbf{q}_i$ holds the features token $i$ is searching for;
- the key $\mathbf{k}_j$ holds the features token $j$ can be found by;
- the score $\mathbf{q}_i \mathbf{k}_j^\top$ is high when the two sets of features line up.

Module 6 turns these scores into attention.

> **Interactive figure** `projection`, on the web page: https://transformers-and-math.drmhse.com/modules/01-shapes-and-linear-algebra#figure-projection
>
> **Try:** Two tokens sit far apart in the plane. Can a projection onto a line ever put them on top of each other? Try to find a line that does it before you turn the controls.
>
> **What happens:** Yes. A projection keeps only the component along its line, so any two points whose difference is perpendicular to the line land on the same spot. Turn the query line until it is perpendicular to the segment joining the two tokens and their projections coincide. Whatever the perpendicular part held is gone.

> **In words.** The figure projects six two-dimensional token vectors onto a query line and a key line. Each projection keeps only the part along its line, so distant tokens can land close together. Turning a line changes which tokens end up near each other, as training does when it adjusts W Q and W K.

## 1.7 How Numbers Are Stored

Each entry of a tensor has a *data type*. The standard is 32-bit floating point (fp32), 4 bytes per number. Large models usually train in 16-bit formats (bf16 or fp16, 2 bytes) and are often served in 8 bits or fewer (Module 24). A tensor's memory is entries times bytes per entry. GPT-2's token embedding matrix has 50,257 rows and 768 columns, which is 38,597,376 numbers: about 154 MB in fp32 and 77 MB in bf16.

Two kinds of tensor fill memory: the learned matrices, fixed while the model runs, and the *activations*, computed fresh from each input.

Every symbol and term is collected in the [glossary](/glossary).

## Exercises

**Exercise (check).** What is the shape of $XW$ when $X \in \mathbb{R}^{5 \times 8}$ and $W \in \mathbb{R}^{8 \times 3}$? And is $WX$ defined?

*Solution:* $XW$ is $5 \times 3$: the inner sizes (8 and 8) agree and disappear. $WX$ would pair the 3 columns of $W$ with the 5 rows of $X$, which do not match, so it is not defined.

**Exercise (apply).** Compute entry $(2, 1)$ of $AB$ for $A = \begin{bmatrix} 1 & 0 & 2 \\ 3 & -1 & 1 \end{bmatrix}$ and $B = \begin{bmatrix} 4 & 0 \\ 2 & 5 \\ -1 & 1 \end{bmatrix}$.

*Hint:* Row 2 of $A$ with column 1 of $B$.

*Answer:* 9 (within 0)

*Solution:* Row 2 of $A$ is $[3, -1, 1]$ and column 1 of $B$ is $[4, 2, -1]$. Their dot product is $3 \cdot 4 + (-1) \cdot 2 + 1 \cdot (-1) = 12 - 2 - 1 = 9$.

**Exercise (apply).** Compute the cosine similarity of $\mathbf{u} = [3, 4]$ and $\mathbf{v} = [4, 3]$.

*Answer:* 0.96 (within 0.001)

*Solution:* The dot product is $3 \cdot 4 + 4 \cdot 3 = 24$. Both lengths are $\sqrt{9 + 16} = 5$. So the cosine similarity is $24 / (5 \cdot 5) = 0.96$: the vectors point in nearly the same direction.

**Exercise (apply).** A projection matrix $W^Q$ has shape $768 \times 64$. What is the largest rank it can have?

*Answer:* 64 (within 0)

*Solution:* Rank is at most the smaller of the two sizes, $\min(768, 64) = 64$. However the 768 numbers of a token vector are set, its query lives in a space of at most 64 independent directions.

**Exercise (apply).** Use $(AB)^\top = B^\top A^\top$ to write $(\mathbf{x}W)^\top$ for a row vector $\mathbf{x}$, and say what shape it has when $\mathbf{x}$ is $1 \times 4$ and $W$ is $4 \times 2$.

*Solution:* $(\mathbf{x}W)^\top = W^\top \mathbf{x}^\top$. $\mathbf{x}W$ is $1 \times 2$, so its transpose is $2 \times 1$: a column vector. The right-hand side agrees: $W^\top$ is $2 \times 4$ and $\mathbf{x}^\top$ is $4 \times 1$, giving $2 \times 1$. This is exactly the switch between the row convention of this course and the column convention of sections 15.1 and 18.1.

**Exercise (apply).** An embedding table has 32,000 rows and 4,096 columns, stored in bf16 (2 bytes per number). How many megabytes (millions of bytes) does it take?

*Answer:* 262.144 (within 0.5)

*Solution:* $32{,}000 \times 4{,}096 = 131{,}072{,}000$ numbers, times 2 bytes, is $262{,}144{,}000$ bytes, about 262 MB.

**Exercise (extend).** Explain, using the row-by-column rule, why multiplying a matrix of token vectors $X$ (one token per row) by a weight matrix $W$ never mixes information between tokens. Which operation in a transformer *must* therefore be responsible for moving information between tokens?

*Solution:* Row $i$ of $XW$ is built only from row $i$ of $X$ (and all of $W$), so token $i$'s output depends on token $i$ alone. No matrix multiplication of the form $XW$ can move information between tokens. Something must multiply *across* the rows of $X$ instead: that is attention (Module 6), whose weights form an $N \times N$ matrix that multiplies the token axis.

### Core Academic References
- **Strang (2016)**: *Introduction to Linear Algebra (5th ed.)*. Wellesley-Cambridge Press.
- **Golub & Van Loan (2013)**: *Matrix Computations (4th ed.)*. Johns Hopkins University Press.
- **Mikolov et al. (2013)**: *Efficient Estimation of Word Representations in Vector Space* (word2vec). [arXiv:1301.3781](https://arxiv.org/abs/1301.3781)