A pretrained model knows a great deal and does one thing: continue text. Adaptation changes what it does by training it a little further: to follow instructions, to specialize in a domain, or to pass its abilities to a smaller model. This module covers instruction tuning, the low-rank trick that makes fine-tuning affordable (and the singular value decomposition behind it), and distillation.

20.1 Supervised Instruction Tuning

Asked a question, a pretrained model may answer, or continue with three more questions, which is also plausible text. Post-training turns it into an assistant in two stages: supervised instruction tuning, here, and preference learning (Module 21).

Instruction tuning fine-tunes on prompts paired with good responses, written or checked by people. The loss is pretraining's next-token loss, scored only on response tokens: with mask $m_t$ equal to 1 for response tokens and 0 for prompt tokens, . The prompt is read as context but not learned (section 26.4 applies the same mask to image-and-text data). A fixed chat template of special tokens marks where the system instructions, the user's turn and the assistant's turn begin and end. A few tens of thousands of high-quality examples suffice; they teach the format of a helpful answer more than new knowledge.

20.2 The Singular Value DecompositionRun this section

The (SVD) breaks any matrix into a sum of simple pieces, most important first. For $W$ with $m$ rows, $n$ columns and rank $r$ :

W = U \Sigma V^\top = \sum_{k=1}^r \sigma_k \mathbf{u}_k \mathbf{v}_k^\top

introduces 3 notations

$U$ is $m \times m$ and orthogonal: its columns $\mathbf{u}_k$ , the left singular vectors, have length 1 and are mutually perpendicular.
$\Sigma$ is $m \times n$ , zero except for the singular values $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0$ on its diagonal.
$V$ is $n \times n$ and orthogonal; its columns $\mathbf{v}_k$ are the right singular vectors.

Each piece $\sigma_k \mathbf{u}_k \mathbf{v}_k^\top$ is an outer product (section 1.3), a rank-one $m \times n$ matrix, and $\sigma_k$ says how much it contributes.

Architecture Flowchart

In words. SVD writes a matrix as rank-one pieces sorted by size; keeping the top pieces gives the closest low-rank matrix. LoRA (section 20.3) works from the other direction, learning a weight update that is low-rank from the start, as the product of two thin matrices.

Figure 20.2

SVD: rotate, stretch, rotate

Every matrix acts on the plane in three moves. Step through them, then keep only the largest piece and watch the ellipse flatten.

A 2×2 matrix, so every step can be drawn. A weight matrix with thousands of rows does the same three moves in more dimensions.

Predict, then try

Before dropping the second piece, predict the error of the rank-1 approximation in terms of the singular values shown.

In words. A two-by-two matrix applied to a circle in three moves: V transposed rotates the right singular vectors onto the axes, Sigma stretches them by sigma one and sigma two into an ellipse, and U rotates the ellipse into place. Keeping only the rank-one piece flattens the ellipse onto a line, leaving an error of exactly sigma two.

The Eckart-Young-Mirsky Theorem

Keep the first $k < r$ pieces. The Eckart–Young–Mirsky theorem says no other rank- $k$ matrix is closer to $W$ in the Frobenius norm , the square root of the sum of squared entries of the difference:

W_k = \sum_{i=1}^k \sigma_i \mathbf{u}_i \mathbf{v}_i^\top, \quad \min_{\operatorname{rank}(B)=k} \|W - B\|_F = \sqrt{\sum_{i=k+1}^r \sigma_i^2}

introduces 3 notations

The error depends only on the dropped singular values; if they are small, rank $k$ is almost as good as full. LoRA rests on a related bet: that the change fine-tuning (further training of a pretrained model) makes to a weight matrix is close to low-rank. The theorem does not prove that bet, but it holds in practice, where ranks of $4$ , $8$ or $16$ often work.

20.3 LoRA and QLoRARun this section

Fine-tuning normally updates every entry of a pretrained $d_{\text{out}} \times d_{\text{in}}$ matrix $W_0$ . LoRA freezes $W_0$ and learns a correction from two thin matrices:

W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A

introduces 3 notations

$B$ is $d_{\text{out}} \times r$ and $A$ is $r \times d_{\text{in}}$ , with $r$ much smaller than both widths, so $BA$ has $W_0$ 's shape but rank at most $r$ .
$\alpha$ is a fixed constant (often $2r$ or $16$ ); dividing by $r$ keeps the update's size steady across choices of $r$ .
Initialization. $A$ starts small and random, $B$ at exactly zero, so
$\Delta W = B A = 0 \cdot A = 0 \quad (\text{at step } t=0)$
and the adapted model starts identical to the pretrained one.

Figure 20.3

How much of an update a rank-r product can capture

The bars are the singular values of a 256 × 256 weight update, largest first. A rank-r LoRA update can match at most the first r of them; the error is everything to the right.

A synthetic spectrum: exponentially decaying with a small flat tail. By Eckart–Young–Mirsky the best rank-r error is the root sum of squares of the dropped values, shown relative to the whole.

Predict, then try

The target update is built to have most of its size in a few directions. Before raising the rank, predict how quickly the relative error falls, and what a rank-8 update costs in parameters compared with the full matrix.

In words. A weight update with a chosen spectrum of singular values, approximated by its best rank-r truncation. The plot highlights the kept singular values; the readout gives the relative error and the trainable parameters against the full matrix.

Browser lab

Eckart–Young–Mirsky, checked numerically

Compute the SVD of a random 24 × 16 matrix (by Jacobi rotations, in your browser), keep the top k pieces, and measure the Frobenius error of what is left. Section 20.2 says it must equal the square root of the sum of the squares of the dropped singular values.

This checks the theorem, which is exact. It does not test LoRA's own bet, that real fine-tuning updates are close to low-rank; that is an empirical finding of Hu et al. (2022).

QLoRA

LoRA shrinks what is , but the frozen base model still sits in memory. QLoRA (Dettmers et al., 2023) stores it in 4 bits, in NF4 (a grid with levels spaced to match a bell curve), dequantizing to bf16 as each layer runs; only the bf16 adapters train, with gradients flowing through the frozen weights. At about half a byte per parameter instead of two, a 65-billion-parameter model fine-tunes on one 48 GB GPU, with quality matching 16-bit fine-tuning on the paper's benchmarks. Section 24.2 explains quantization.

20.4 DistillationRun this section

trains a small student to imitate a large teacher. A one-hot label says "the next token is cat"; the teacher's distribution says "cat 0.6, dog 0.2, kitten 0.1, …", which also says which wrong answers are nearly right: the teacher's dark knowledge (Hinton et al., 2015). The classic loss compares the two with KL divergence, both softened by a temperature $T > 1$ so the small probabilities count:

\mathcal{L}_{\text{KD}} = T^2 \operatorname{KL}\left(\operatorname{softmax}(\mathbf{z}_{\text{teacher}}/T) \,\Big\|\, \operatorname{softmax}(\mathbf{z}_{\text{student}}/T)\right)

introduces 3 notations

$T^2$ keeps the gradient's size comparable across temperatures, and the loss is often mixed with ordinary cross-entropy on the true labels. For language models:

Logit distillation matches the teacher's full distribution at every position. It needs the teacher's logits, so it is used when one organization owns both models; Gemma 2's smaller sizes were trained this way.
Sequence-level distillation fine-tunes the student on text the teacher generates, needing no logits.

"Full-parameter distillation" (section 18.4) trains all of the student's weights, rather than only a LoRA adapter. Section 22.5 uses distillation to pass reasoning ability to small models.

Exercises

Apply

How many trainable parameters does LoRA add to one $4096 \times 4096$ weight matrix at rank $r = 8$ , and what fraction of the full matrix is that?

Apply

A matrix has singular values $[5, 3, 1, 0.5]$ . What is the Frobenius-norm error of the best rank-2 approximation?

Check

Why does LoRA start $B$ at zero rather than at small random values?

Apply

In QLoRA, roughly how many gigabytes do the frozen 4-bit weights of a 65-billion-parameter model take, at half a byte per parameter?

Extend

Why does distillation use a temperature above 1, and what would the KL loss reduce to at $T \to 0$ ?

Core Academic References

Ouyang et al. (2022): Training language models to follow instructions with human feedback (InstructGPT). NeurIPS 2022. arXiv:2203.02155
Wei et al. (2022): Finetuned Language Models Are Zero-Shot Learners (FLAN). ICLR 2022. arXiv:2109.01652
Eckart & Young (1936): The approximation of one matrix by another of lower rank. Psychometrika, 1(3): 211–218.
Hu et al. (2022): LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685
Dettmers et al. (2023): QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. arXiv:2305.14314
Hinton, Vinyals & Dean (2015): Distilling the Knowledge in a Neural Network. arXiv:1503.02531
Kim & Rush (2016): Sequence-Level Knowledge Distillation. EMNLP 2016. arXiv:1606.07947