# Module 20: Adaptation: Fine-Tuning, LoRA, Distillation

> Adapting a pretrained LLM cheaply: instruction tuning, the singular value decomposition, LoRA and QLoRA low-rank fine-tuning, and distillation.

- Part V: From Model to Assistant · elective · Advanced · 4 hours
- By the end you can adapt a pretrained model cheaply and explain why low-rank updates work.
- Builds on: [Module 10: Pretraining](https://transformers-and-math.drmhse.com/md/10-pretraining.md), [Module 12: Generation](https://transformers-and-math.drmhse.com/md/12-generation.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/20-adaptation
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

A pretrained model knows a great deal and does one thing: continue text. *Adaptation* changes what it does by training it a little further: to follow instructions, to specialize in a domain, or to pass its abilities to a smaller model. This module covers instruction tuning, the low-rank trick that makes fine-tuning affordable (and the singular value decomposition behind it), and distillation.

## 20.1 Supervised Instruction Tuning

Asked a question, a pretrained model may answer, or continue with three more questions, which is also plausible text. *Post-training* turns it into an assistant in two stages: supervised instruction tuning, here, and preference learning (Module 21).

Instruction tuning fine-tunes on prompts paired with good responses, written or checked by people. The loss is pretraining's next-token loss, scored only on response tokens: with mask $m_t$ equal to 1 for response tokens and 0 for prompt tokens, $-\sum_t m_t \log P_\theta(w_t \mid w_{<t})$. The prompt is read as context but not learned (section 26.4 applies the same mask to image-and-text data). A fixed *chat template* of special tokens marks where the system instructions, the user's turn and the assistant's turn begin and end. A few tens of thousands of high-quality examples suffice; they teach the format of a helpful answer more than new knowledge.

## 20.2 The Singular Value Decomposition

The *singular value decomposition* (SVD) breaks any matrix into a sum of simple pieces, most important first. For $W$ with $m$ rows, $n$ columns and rank $r$:

$$
W = U \Sigma V^\top = \sum_{k=1}^r \sigma_k \mathbf{u}_k \mathbf{v}_k^\top
$$

- $U$ is $m \times m$ and *orthogonal*: its columns $\mathbf{u}_k$, the *left singular vectors*, have length 1 and are mutually perpendicular.
- $\Sigma$ is $m \times n$, zero except for the *singular values* $\sigma_1 \ge \sigma_2 \ge \dots \ge \sigma_r > 0$ on its diagonal.
- $V$ is $n \times n$ and orthogonal; its columns $\mathbf{v}_k$ are the *right singular vectors*.

Each piece $\sigma_k \mathbf{u}_k \mathbf{v}_k^\top$ is an outer product (section 1.3), a rank-one $m \times n$ matrix, and $\sigma_k$ says how much it contributes.

```mermaid
graph LR
    W["Weight matrix W (m × n)"] --> SVD["SVD: sum of rank-one pieces, largest first"]
    SVD --> Trunc["Keep the top k pieces: best rank-k approximation"]
    Trunc --> Idea["Idea: a low-rank matrix can stand in for a full one"]
    Idea --> LoRA["LoRA: learn the update as ΔW = B · A <br/> (B is m × r, A is r × n)"]
```

> **In words.** SVD writes a matrix as rank-one pieces sorted by size; keeping the top pieces gives the closest low-rank matrix. LoRA (section 20.3) works from the other direction, learning a weight update that is low-rank from the start, as the product of two thin matrices.

> **Interactive figure** `svd`, on the web page: https://transformers-and-math.drmhse.com/modules/20-adaptation#figure-svd
>
> **Try:** Before dropping the second piece, predict the error of the rank-1 approximation in terms of the singular values shown.
>
> **What happens:** Exactly $\sigma_2$: the leftover is the dropped piece $\sigma_2 \mathbf{u}_2\mathbf{v}_2^\top$, whose size is $\sigma_2$ because $\mathbf{u}_2$ and $\mathbf{v}_2$ have length 1. That is the Eckart–Young–Mirsky theorem below, for a 2 × 2 matrix.

> **In words.** A two-by-two matrix applied to a circle in three moves: V transposed rotates the right singular vectors onto the axes, Sigma stretches them by sigma one and sigma two into an ellipse, and U rotates the ellipse into place. Keeping only the rank-one piece flattens the ellipse onto a line, leaving an error of exactly sigma two.

### The Eckart-Young-Mirsky Theorem
Keep the first $k < r$ pieces. The Eckart–Young–Mirsky theorem says no other rank-$k$ matrix is closer to $W$ in the *Frobenius norm* $\|W - B\|_F$, the square root of the sum of squared entries of the difference:

$$
W_k = \sum_{i=1}^k \sigma_i \mathbf{u}_i \mathbf{v}_i^\top, \quad \min_{\operatorname{rank}(B)=k} \|W - B\|_F = \sqrt{\sum_{i=k+1}^r \sigma_i^2}
$$

The error depends only on the dropped singular values; if they are small, rank $k$ is almost as good as full. LoRA rests on a related bet: that the *change* *fine-tuning* (further training of a pretrained model) makes to a weight matrix is close to low-rank. The theorem does not prove that bet, but it holds in practice, where ranks of $4$, $8$ or $16$ often work.

## 20.3 LoRA and QLoRA

Fine-tuning normally updates every entry of a pretrained $d_{\text{out}} \times d_{\text{in}}$ matrix $W_0$. LoRA *freezes* $W_0$ and learns a correction from two thin matrices:
$$
W = W_0 + \Delta W = W_0 + \frac{\alpha}{r} B A
$$
- $B$ is $d_{\text{out}} \times r$ and $A$ is $r \times d_{\text{in}}$, with $r$ much smaller than both widths, so $BA$ has $W_0$'s shape but rank at most $r$.
- $\alpha$ is a fixed constant (often $2r$ or $16$); dividing by $r$ keeps the update's size steady across choices of $r$.
- **Initialization.** $A$ starts small and random, $B$ at exactly zero, so
  $$
  \Delta W = B A = 0 \cdot A = 0 \quad (\text{at step } t=0)
  $$
  and the adapted model starts identical to the pretrained one.

> **Interactive figure** `lora-rank`, on the web page: https://transformers-and-math.drmhse.com/modules/20-adaptation#figure-lora-rank
>
> **Try:** The target update is built to have most of its size in a few directions. Before raising the rank, predict how quickly the relative error falls, and what a rank-8 update costs in parameters compared with the full matrix.
>
> **What happens:** The error falls steeply over the first few ranks and then flattens, tracking the singular values that are dropped. A rank-8 update of a 256 × 256 matrix trains $2 \times 256 \times 8 = 4{,}096$ numbers instead of $65{,}536$, one-sixteenth.

> **In words.** A weight update with a chosen spectrum of singular values, approximated by its best rank-r truncation. The plot highlights the kept singular values; the readout gives the relative error and the trainable parameters against the full matrix.

> **Browser lab** `svd-truncation`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/20-adaptation#lab-svd-truncation

### QLoRA
LoRA shrinks what is *trained*, but the frozen base model still sits in memory. QLoRA (Dettmers et al., 2023) stores it in 4 bits, in NF4 (a grid with levels spaced to match a bell curve), dequantizing to bf16 as each layer runs; only the bf16 adapters train, with gradients flowing through the frozen weights. At about half a byte per parameter instead of two, a 65-billion-parameter model fine-tunes on one 48 GB GPU, with quality matching 16-bit fine-tuning on the paper's benchmarks. Section 24.2 explains quantization.

## 20.4 Distillation

*Distillation* trains a small *student* to imitate a large *teacher*. A one-hot label says "the next token is `cat`"; the teacher's distribution says "`cat` 0.6, `dog` 0.2, `kitten` 0.1, …", which also says which wrong answers are nearly right: the teacher's *dark knowledge* (Hinton et al., 2015). The classic loss compares the two with KL divergence, both softened by a temperature $T > 1$ so the small probabilities count:

$$
\mathcal{L}_{\text{KD}} = T^2 \operatorname{KL}\left(\operatorname{softmax}(\mathbf{z}_{\text{teacher}}/T) \,\Big\|\, \operatorname{softmax}(\mathbf{z}_{\text{student}}/T)\right)
$$

$T^2$ keeps the gradient's size comparable across temperatures, and the loss is often mixed with ordinary cross-entropy on the true labels. For language models:

- **Logit distillation** matches the teacher's full distribution at every position. It needs the teacher's logits, so it is used when one organization owns both models; Gemma 2's smaller sizes were trained this way.
- **Sequence-level distillation** fine-tunes the student on text the teacher *generates*, needing no logits.

"Full-parameter distillation" (section 18.4) trains all of the student's weights, rather than only a LoRA adapter. Section 22.5 uses distillation to pass reasoning ability to small models.

## Exercises

**Exercise (apply).** How many trainable parameters does LoRA add to one $4096 \times 4096$ weight matrix at rank $r = 8$, and what fraction of the full matrix is that?

*Answer:* 65536 (within 0)

*Solution:* $B$ is $4096 \times 8$ and $A$ is $8 \times 4096$: $2 \times 4096 \times 8 = 65{,}536$ parameters, against $16{,}777{,}216$ for the full matrix, about 0.4%.

**Exercise (apply).** A matrix has singular values $[5, 3, 1, 0.5]$. What is the Frobenius-norm error of the best rank-2 approximation?

*Answer:* 1.118 (within 0.001)

*Solution:* By Eckart–Young–Mirsky it is the square root of the sum of the squares of the dropped values: $\sqrt{1^2 + 0.5^2} = \sqrt{1.25} \approx 1.118$.

**Exercise (check).** Why does LoRA start $B$ at zero rather than at small random values?

*Solution:* So that $\Delta W = BA = 0$ at the first step: the adapted model starts exactly equal to the pretrained one, and training moves it away gradually. If both were random, the model would start from a randomly perturbed version of the pretrained weights and lose some of what it knew before training even began.

**Exercise (apply).** In QLoRA, roughly how many gigabytes do the frozen 4-bit weights of a 65-billion-parameter model take, at half a byte per parameter?

*Answer:* 32.5 (within 0.5)

*Solution:* $65 \times 10^9 \times 0.5 = 32.5 \times 10^9$ bytes, about 32.5 GB, which with the small adapters, their optimizer state and activations fits on one 48 GB GPU. In bf16 the same weights would take 130 GB.

**Exercise (extend).** Why does distillation use a temperature above 1, and what would the KL loss reduce to at $T \to 0$?

*Solution:* At $T = 1$ a confident teacher puts nearly all its probability on one token, so its distribution is close to one-hot and carries little information about the other tokens. A higher temperature flattens it, exposing the relative probabilities of the near-misses for the student to learn. As $T \to 0$ the teacher's softened distribution becomes exactly one-hot on its top token, and the loss reduces to cross-entropy against the teacher's top choice: training on the teacher's hard labels.

### Core Academic References
- **Ouyang et al. (2022)**: *Training language models to follow instructions with human feedback* (InstructGPT). NeurIPS 2022. [arXiv:2203.02155](https://arxiv.org/abs/2203.02155)
- **Wei et al. (2022)**: *Finetuned Language Models Are Zero-Shot Learners* (FLAN). ICLR 2022. [arXiv:2109.01652](https://arxiv.org/abs/2109.01652)
- **Eckart & Young (1936)**: *The approximation of one matrix by another of lower rank*. Psychometrika, 1(3): 211–218.
- **Hu et al. (2022)**: *LoRA: Low-Rank Adaptation of Large Language Models*. ICLR 2022. [arXiv:2106.09685](https://arxiv.org/abs/2106.09685)
- **Dettmers et al. (2023)**: *QLoRA: Efficient Finetuning of Quantized LLMs*. NeurIPS 2023. [arXiv:2305.14314](https://arxiv.org/abs/2305.14314)
- **Hinton, Vinyals & Dean (2015)**: *Distilling the Knowledge in a Neural Network*. [arXiv:1503.02531](https://arxiv.org/abs/1503.02531)
- **Kim & Rush (2016)**: *Sequence-Level Knowledge Distillation*. EMNLP 2016. [arXiv:1606.07947](https://arxiv.org/abs/1606.07947)