Query, key, value
Three learned versions of each token's vector. The query is what the token is looking for, the key is what it offers to others, and the value is what it passes on. W^Q is read "W Q", the matrix that makes queries.
One vector would have to serve as a search term, a search label and the content itself, and those roles would get in each other's way. So the model learns three separate weight matrices and makes three versions of each token:
- : what am I looking for?
- : what do I offer?
- : what do I contribute if chosen?
Attention compares every query with every key to get scores, then mixes the values using those scores. The raised , , are labels, not powers. A fourth matrix, , maps the mixed result back to the model's width.
Where the course uses it
- 1.1 Scalars, Vectors, Matrices, and Tensors · Module 1, Shapes and Linear Algebrafirst use
- 1.6 Linear Projections: One Vector Into Three · Module 1, Shapes and Linear Algebra
- 6.2 Scaled Dot-Product Attention, by Hand · Module 6, Attention
- 7.4 The Feed-Forward Network · Module 7, The Block and the Full Model
- 7.6 The Complete Model · Module 7, The Block and the Full Model
- 12.2 The KV Cache · Module 12, Generation
- 19.3 Estimating Parameters, Cache, and Compute · Module 19, Reading a Model Card
Related terms
- Matrix multiplicationTwo matrices written side by side, like AB, are multiplied. Each entry of the result is a row of A dotted with a column of B. An (m×n) times an (n×p) gives an (m×p).
- Scaled dot-product attentionEach token scores every other token, turns the scores into weights that add up to 1, and takes the weighted average of their value vectors.
- d_k, d_v — the width of one headRead "d k" and "d v". They are the number of entries in one attention head's key and value vectors, usually d_model divided by the number of heads.