Bold letters are vectors
A bold lowercase letter, like x in bold, is a vector (a list of numbers). A plain italic letter is one number. A capital letter is a matrix.
The font tells you what kind of object a letter is:
- : a scalar, which means one number
- : a vector, a list of numbers
- or : a matrix, a 2-D table of numbers
This matters for subscripts. is the whole vector for token . is the -th number inside a vector. Only the boldness tells the two apart.
Example
Bold x is the whole list; plain x with subscript 2 is one entry of it (counting from 1 here).
Where the course uses it
- 1.1 Scalars, Vectors, Matrices, and Tensors · Module 1, Shapes and Linear Algebrafirst use
- 1.3 The Matrix Toolkit · Module 1, Shapes and Linear Algebra
- 1.4 Dot Products, Length, and Angle · Module 1, Shapes and Linear Algebra
- 1.5 Embeddings: From a Token to a Vector · Module 1, Shapes and Linear Algebra
- 1.6 Linear Projections: One Vector Into Three · Module 1, Shapes and Linear Algebra
- 2.3 The Chain Rule and the Jacobian · Module 2, Calculus and Optimization
- 2.5 Backpropagation · Module 2, Calculus and Optimization
- 4.2 Neurons, Layers, and the MLP · Module 4, How a Model Learns From Data
- 4.3 Logits, Softmax, and Cross-Entropy · Module 4, How a Model Learns From Data
- 5.2 Recurrent Networks · Module 5, Sequences Before Transformers
- 5.3 No Parallelism Across Time · Module 5, Sequences Before Transformers
- 5.4 Vanishing and Exploding Gradients · Module 5, Sequences Before Transformers
- 5.6 The First Attention · Module 5, Sequences Before Transformers
- 6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
- 6.3 Why Divide by the Square Root of the Key Width · Module 6, Attention
- 6.5 Attention Cannot See Word Order · Module 6, Attention
- 7.1 Positional Encodings · Module 7, The Block and the Full Model
- 7.2 Residual Connections and the Residual Stream · Module 7, The Block and the Full Model
- 7.3 LayerNorm · Module 7, The Block and the Full Model
- 7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
- 7.5 The Output Layer · Module 7, The Block and the Full Model
- 8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 8.5 Where the Norm Goes: Post-LN and Pre-LN · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 10.2 AdamW · Module 10, Pretraining
- 11.4 Tensor Parallelism · Module 11, Training Across Many GPUs
- 12.2 The KV Cache · Module 12, Generation
- 12.3 Decoding: From Logits to Text · Module 12, Generation
- 14.1 The Residual Stream as Shared Memory · Module 14, What the Model Learns
- 15.1 Rotary Position Embedding (RoPE) · Module 15, The LLaMA Block
- 15.2 Stretching RoPE, and ALiBi · Module 15, The LLaMA Block
- 15.3 RMSNorm · Module 15, The LLaMA Block
- 15.4 SwiGLU · Module 15, The LLaMA Block
- 16.2 FlashAttention and the Online Softmax · Module 16, Efficient and Long Attention
- 16.4 Sandwich Normalization and Embedding Scaling · Module 16, Efficient and Long Attention
- 17.2 Many Small Experts · Module 17, Mixture of Experts
- 18.1 Linear Attention · Module 18, Beyond Attention: Linear Attention and Hybrids
- 18.2 The Gated Delta Rule · Module 18, Beyond Attention: Linear Attention and Hybrids
- 18.3 State-Space Models · Module 18, Beyond Attention: Linear Attention and Hybrids
- 20.2 The Singular Value Decomposition · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
- 20.4 Distillation · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
- 24.3 Outlier Channels and SmoothQuant · Module 24, Quantization and Number Formats
- 24.4 Post-Training Quantization: AWQ and GPTQ · Module 24, Quantization and Number Formats
- 25.2 The Vision Transformer · Module 25, Vision Transformers and Contrastive Learning
- 26.1 Connecting Vision to Language · Module 26, Vision-Language Models
- 26.3 2D RoPE for Image Patches · Module 26, Vision-Language Models
- 26.4 Training on Images and Text Together · Module 26, Vision-Language Models
- 27.2 The Alignment Problem and the Blank · Module 27, Recognition: OCR and Speech With CTC
- 27.3 The CTC Objective and Its Recursion · Module 27, Recognition: OCR and Speech With CTC
- 28.2 Autoencoders and Latent Spaces · Module 28, Generating Audio: Codecs and TTS
- 28.3 Vector Quantization and Residual VQ · Module 28, Generating Audio: Codecs and TTS