\operatorname{rank}

Rank of a matrix

Read "rank of W". It is the number of truly independent directions a matrix can produce, which equals its number of nonzero singular values.

A $4096 \times 4096$ matrix stores about 16.8 million numbers, but its rank can be as low as 8: every output it produces is then a mix of just 8 directions. Low rank means the matrix can be stored much more compactly, as two thin matrices.

LoRA assumes that the change a fine-tune makes to the weights has low rank, even though the weights themselves do not. So the change can be learned as two thin matrices.

Example

\operatorname{rank}\begin{bmatrix} 1 & 2 \\ 2 & 4 \end{bmatrix} = 1

The second row is twice the first, so there is only one independent direction.

Where the course uses it

1.3 The Matrix Toolkit · Module 1, Shapes and Linear Algebrafirst use
20.2 The Singular Value Decomposition · Module 20, Adaptation: Fine-Tuning, LoRA, Distillation

Related terms

SVD — singular value decompositionRead "W equals U Sigma V transpose". Any matrix can be split into three steps: a rotation, a stretch along each axis, and another rotation.
LoRA — low-rank adaptationRead "W zero plus alpha over r, B A". Keep the pretrained weights W₀ frozen and learn a small low-rank correction BA to add to them.