Norm — the length of a vector
Read "the norm of x". Double bars mean the length of a vector: square each entry, add them up, and take the square root.
. This is Pythagoras' theorem with sides instead of two. The small subscript says which kind of length:
- : ordinary (Euclidean) length, the default
- : the Frobenius norm, the same formula applied to every entry of a matrix
- : the sum of the absolute values of the entries
Dividing a vector by its norm gives a vector of length 1 (a unit vector). Cosine similarity does this to both vectors before taking their dot product.
Example
The 3–4–5 right triangle: the vector's length is the hypotenuse.
Where the course uses it
- 1.4 Dot Products, Length, and Angle · Module 1, Shapes and Linear Algebrafirst use
- 3.5 Information: Entropy, Cross-Entropy, KL, Perplexity · Module 3, Probability and Information
- 5.4 Vanishing and Exploding Gradients · Module 5, Sequences Before Transformers
- 20.2 The Singular Value Decomposition · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
- 20.4 Distillation · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation
- 21.5 RLHF End to End · Module 21, Preference Learning: RL, RLHF and DPO
- 28.2 Autoencoders and Latent Spaces · Module 28, Generating Audio: Codecs and TTS
Related terms
- Dot product (inner product)Read "the dot product of u and v". Multiply the two vectors entry by entry and add the results. You get one number, which is large when the vectors point the same way.
- |V| — size of the vocabularyRead "the size of V". Single bars around a set mean "how many items it has", so |V| is the number of tokens in the vocabulary.
- √ — square rootRead "the square root of". It is the non-negative number that, multiplied by itself, gives the number inside: √9 = 3.