X \in \mathbb{R}^{m \times n}

Matrix — a table of numbers

A rectangular table of numbers, with two axes: rows and columns. An m×n matrix has m rows and n columns. Written as a capital letter, such as W.

A matrix has two axes, so two indices pick one entry: $X_{ij}$ is row $i$ , column $j$ . Its shape is always said rows first: $\mathbb{R}^{3 \times 4}$ is 3 rows by 4 columns, 12 numbers in all.

Matrices play two roles in a transformer:

Data. A sentence of $N$ tokens, each a vector of width $d_{\text{model}}$ , is stacked into an $N \times d_{\text{model}}$ matrix, one token per row.
Weights. A learned $W$ turns vectors of one width into another by matrix multiplication.

In PyTorch a 3×4 matrix has shape (3, 4).

Example

X = \begin{bmatrix} 1 & 0 & 2 & 5 \\ 3 & 1 & 0 & 4 \\ 0 & 2 & 2 & 1 \end{bmatrix} \in \mathbb{R}^{3 \times 4} \;\Longrightarrow\; X_{23} = 0

Row 2, column 3. Each row could be one token's vector.

Where the course uses it

1.1 Scalars, Vectors, Matrices, and Tensors · Module 1, Shapes and Linear Algebrafirst use
1.6 Linear Projections: One Vector Into Three · Module 1, Shapes and Linear Algebra
6.1 The Question Attention Answers · Module 6, Attention
6.6 Multi-Head Attention · Module 6, Attention
7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
7.5 The Output Layer · Module 7, The Block and the Full Model
8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
12.2 The KV Cache · Module 12, Generation
25.2 The Vision Transformer · Module 25, Vision Transformers and Contrastive Learning
26.1 Connecting Vision to Language · Module 26, Vision-Language Models
26.4 Training on Images and Text Together · Module 26, Vision-Language Models

Related terms

Vector — a list of numbersAn ordered list of numbers, with one axis. One index picks out one entry. Each token becomes a vector before the model can compute with it.
Tensor — numbers on any number of axesAn array of numbers with any number of axes. In code every array is called a tensor: a scalar has 0 axes, a vector 1, a matrix 2. In prose the word usually means 3 or more.
A matrix written outSquare brackets around a grid of numbers show a matrix entry by entry. Each line is a row; each position across a line is a column.
Matrix multiplicationTwo matrices written side by side, like AB, are multiplied. Each entry of the result is a row of A dotted with a column of B. An (m×n) times an (n×p) gives an (m×p).
Shape and axesA tensor's shape lists the size of each axis, in order. (2, 3, 4) has three axes of sizes 2, 3 and 4. Checking shapes is the quickest way to follow a formula.