d_model — the model's width
Read "d model". It is how many numbers represent one token inside the network: 768 for GPT-2 small, 4096 for LLaMA-3 8B.
It is the one size that stays the same through every layer. Every vector in the residual stream, and every sub-layer's input and output, has numbers. That is what lets the residual addition work at every layer: you can only add two vectors of the same length.
The weight matrices in each layer are roughly in size, so a layer's parameter count grows with . Doubling the width roughly quadruples it.
Where the course uses it
- 0.3 What You Need, and How the Course Is Organized · Module 0, Start Here: What an LLM Doesfirst use
- 1.1 Scalars, Vectors, Matrices, and Tensors · Module 1, Shapes and Linear Algebra
- 1.2 Matrix Multiplication, Done by Hand · Module 1, Shapes and Linear Algebra
- 1.3 The Matrix Toolkit · Module 1, Shapes and Linear Algebra
- 1.5 Embeddings: From a Token to a Vector · Module 1, Shapes and Linear Algebra
- 1.6 Linear Projections: One Vector Into Three · Module 1, Shapes and Linear Algebra
- 6.6 Multi-Head Attention · Module 6, Attention
- 7.1 Positional Encodings · Module 7, The Block and the Full Model
- 7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
- 7.5 The Output Layer · Module 7, The Block and the Full Model
- 7.6 The Complete Model · Module 7, The Block and the Full Model
- 8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 9.4 Choosing a Vocabulary Size · Module 9, Tokenization
- 15.1 Rotary Position Embedding (RoPE) · Module 15, The LLaMA Block
- 15.3 RMSNorm · Module 15, The LLaMA Block
- 15.4 SwiGLU · Module 15, The LLaMA Block
- 16.4 Sandwich Normalization and Embedding Scaling · Module 16, Efficient and Long Attention
- 19.1 Comparing the Architectures · Module 19, Reading a Model Card
- 19.2 A Config File, Field by Field · Module 19, Reading a Model Card
Related terms
- d_k, d_v — the width of one headRead "d k" and "d v". They are the number of entries in one attention head's key and value vectors, usually d_model divided by the number of heads.
- ℝ^d — all vectors of length dRead "R d" or "R to the d". It is the set of all lists of d real numbers, so "x ∈ ℝ^d" says x is a vector of d numbers.
- d_ff — the feed-forward widthRead "d f f". It is the width of the hidden layer inside the feed-forward block: traditionally 4 × d_model, and about 8/3 × d_model in gated versions like SwiGLU.