Transpose
Read "A transpose". The raised T flips a matrix so its rows become its columns. On a vector it turns a column into a row.
Entry by entry, : the number in row , column of is the number in row , column of . A matrix becomes .
In this course its main job is making shapes fit so a multiplication is allowed. works because is and is , giving the matrix of scores: one row per query, one column per key. In self-attention , so it is square. without the transpose would not be a valid product.
is the dot product written as a matrix product: a row times a column gives one number.
Example
The first row (1, 2, 3) becomes the first column. A 2×3 matrix becomes 3×2.
Where the course uses it
- 1.3 The Matrix Toolkit · Module 1, Shapes and Linear Algebrafirst use
- 1.4 Dot Products, Length, and Angle · Module 1, Shapes and Linear Algebra
- 1.5 Embeddings: From a Token to a Vector · Module 1, Shapes and Linear Algebra
- 1.6 Linear Projections: One Vector Into Three · Module 1, Shapes and Linear Algebra
- 2.3 The Chain Rule and the Jacobian · Module 2, Calculus and Optimization
- 2.5 Backpropagation · Module 2, Calculus and Optimization
- 5.4 Vanishing and Exploding Gradients · Module 5, Sequences Before Transformers
- 6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
- 6.3 Why Divide by the Square Root of the Key Width · Module 6, Attention
- 6.5 Attention Cannot See Word Order · Module 6, Attention
- 6.6 Multi-Head Attention · Module 6, Attention
- 7.5 The Output Layer · Module 7, The Block and the Full Model
- 8.1 The 2017 Encoder-Decoder and Cross-Attention · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 8.4 Decoder-Only: Predicting the Next Token · Module 8, Three Families: Encoder, Encoder-Decoder, Decoder
- 12.2 The KV Cache · Module 12, Generation
- 15.1 Rotary Position Embedding (RoPE) · Module 15, The LLaMA Block
- 15.2 Stretching RoPE, and ALiBi · Module 15, The LLaMA Block
- 16.4 Sandwich Normalization and Embedding Scaling · Module 16, Efficient and Long Attention
- 16.5 Logit Soft-Capping and QK-Norm · Module 16, Efficient and Long Attention
- 18.1 Linear Attention · Module 18, Beyond Attention: Linear Attention and Hybrids
- 18.2 The Gated Delta Rule · Module 18, Beyond Attention: Linear Attention and Hybrids
- 20.2 The Singular Value Decomposition · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
- 24.4 Post-Training Quantization: AWQ and GPTQ · Module 24, Quantization and Number Formats
- 26.4 Training on Images and Text Together · Module 26, Vision-Language Models
- 27.3 The CTC Objective and Its Recursion · Module 27, Recognition: OCR and Speech With CTC
Related terms
- Dot product (inner product)Read "the dot product of u and v". Multiply the two vectors entry by entry and add the results. You get one number, which is large when the vectors point the same way.
- Matrix multiplicationTwo matrices written side by side, like AB, are multiplied. Each entry of the result is a row of A dotted with a column of B. An (m×n) times an (n×p) gives an (m×p).