# Module 10: Pretraining

> The decisions behind LLM pretraining: data and deduplication, AdamW, learning-rate warmup and decay, keeping training stable, and Chinchilla scaling laws.

- Part III: Building an LLM · core track · Intermediate · 5 hours
- By the end you can plan a pretraining run (data, optimizer, schedule, model size) and justify each choice.
- Builds on: [Module 9: Tokenization](https://transformers-and-math.drmhse.com/md/09-tokenization.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/10-pretraining
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

*Pretraining* is the long first phase in which a model learns from raw text by next-token prediction (section 8.4), and where almost all of its knowledge and training compute go. This module covers the decisions that shape a run: the data, the optimizer and its learning-rate schedule, keeping the run stable, and splitting a compute budget between model size and data.

## 10.1 Pretraining Data

A model can only learn what is in its training text, so the data is as much a part of the design as the architecture. Building a pretraining set of trillions of tokens is a pipeline of its own:

1. **Collect.** Mostly web crawls (Common Crawl is the usual start), plus books, encyclopedias, papers, code, and conversational and mathematical text.
2. **Extract and filter.** Web pages are mostly menus, boilerplate and spam. Text is extracted from the HTML, filtered by rules (minimum length, share of lines ending in punctuation, repeated lines) and often by a small classifier for high-quality text such as reference or educational material, plus filters for language, toxicity and personal information.
3. **Deduplicate.** Duplicates waste compute, make models reproduce text verbatim, and leak test sets into training (section 13.3). Exact copies are removed by hashing; near-duplicates need the fuzzier test below.
4. **Mix.** Sources are combined in chosen proportions, the *data mixture*, tuned by training small models on candidate mixes. More code or mathematics improves those skills, often with benefits elsewhere, and many runs finish on the highest-quality data.
5. **Synthesize.** Increasingly, other models write part of the data: rewritten web text, textbook-style explanations, worked problems with verified answers. It is cheap and targets gaps, but overuse narrows the data's variety.

### Finding Near-Duplicates
Near-duplicates share most of their short word sequences. Cut each document into overlapping *shingles* of $k$ consecutive words and compare the sets with the *Jaccard similarity*, shared shingles over all shingles:

$$
J(A, B) = \frac{|A \cap B|}{|A \cup B|}
$$

$J$ runs from 0 (nothing shared) to 1 (identical). Comparing every pair is impossible at web scale, so pipelines use *MinHash*: hash every shingle with many hash functions and keep each function's smallest value. Two documents agree on one such minimum with probability exactly $J$, so a signature of, say, 128 minimums estimates $J$ without comparing full sets, and matching signatures are quick to find.

> **Interactive figure** `dedup-demo`, on the web page: https://transformers-and-math.drmhse.com/modules/10-pretraining#figure-dedup-demo
>
> **Try:** The second document is the first with one phrase reworded. Before looking, estimate the Jaccard similarity of their 3-word shingles: closer to 0.2, 0.5 or 0.9?
>
> **What happens:** About 0.74 with 3-word shingles. Rewording "the coming year" as "next year" breaks every shingle that overlaps the change, about seven of them, while the rest of the document still matches. With longer shingles each edit breaks more of them, and the similarity drops (about 0.59 at 5 words). The unrelated text scores 0, and the MinHash estimate tracks the exact value, which is what lets a threshold catch the copy.

> **In words.** An original, a lightly edited copy and an unrelated text, with each pair's shared shingles, exact Jaccard similarity and MinHash estimate. The estimate settles near the exact value as the signature grows, and the edited copy scores far above the unrelated text.

> **Browser lab** `minhash-accuracy`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/10-pretraining#lab-minhash-accuracy

## 10.2 AdamW

Plain gradient descent (section 2.4) uses one learning rate for every weight. Adam (Kingma & Ba, 2015) gives each weight its own effective step size, from two running averages of its gradient $\mathbf{g}_t$:

$$
\mathbf{m}_t = \beta_1 \mathbf{m}_{t-1} + (1 - \beta_1)\,\mathbf{g}_t, \quad \mathbf{v}_t = \beta_2 \mathbf{v}_{t-1} + (1 - \beta_2)\,\mathbf{g}_t^2
$$

$\mathbf{m}_t$ tracks the gradient's recent average (direction) and $\mathbf{v}_t$ its average square, entry by entry (size). Language models typically use $\beta_1 = 0.9$, $\beta_2 = 0.95$. The update divides one by the square root of the other:

$$
\theta \longleftarrow \theta - \eta \left(\frac{\hat{\mathbf{m}}_t}{\sqrt{\hat{\mathbf{v}}_t} + \epsilon} + \lambda \theta\right)
$$

The hats correct for the averages starting at zero: $\hat{\mathbf{m}}_t = \mathbf{m}_t / (1 - \beta_1^t)$, likewise for $\mathbf{v}$. Dividing by $\sqrt{\hat{\mathbf{v}}_t}$ makes the step size roughly independent of the gradient's scale, so rarely updated weights, such as rare-token embeddings, still move. $\lambda \theta$ is *weight decay* (section 4.4), typically $\lambda = 0.1$; the "W" in AdamW (Loshchilov & Hutter, 2019) means it is applied directly, as here, rather than mixed into the gradient, where Adam's division would distort it.

The price is memory: two extra numbers per weight. With bf16 weights and gradients plus fp32 master weights and averages, the rule of thumb is 16 bytes per parameter, before activations.

## 10.3 Learning-Rate Warmup and Decay

Training starts with a *warmup*: the rate rises linearly from near zero over the first few thousand steps, because early on the weights are random and Adam's averages unreliable, and full-size steps can throw the model somewhere it never recovers from (Post-LN models need this most, section 8.5). Then the rate decays, usually along a cosine to about a tenth of its peak, so late training settles with small steps.

> **Interactive figure** `lr-schedule`, on the web page: https://transformers-and-math.drmhse.com/modules/10-pretraining#figure-lr-schedule
>
> **Try:** Set the warmup to zero. What is the learning rate at the very first step, and why is that the riskiest moment of the whole run?
>
> **What happens:** The full peak rate. At step one the weights are random and Adam's running averages hold a single gradient, so the update size is at its least reliable exactly when the step is largest. Warmup keeps early steps small until both have settled.

> **In words.** The learning rate climbs linearly during warmup, peaks, then falls along half a cosine to a floor of a tenth of the peak. Longer warmup delays the peak; a higher floor keeps late steps larger.

## 10.4 Keeping Training Stable

A large run lasts weeks, so one divergence (the loss shooting up and never recovering) is expensive. Three standard defences, alongside warmup:

- **Gradient clipping.** A gradient longer than a threshold, usually $1.0$, is scaled down to it, so one bad batch cannot throw the weights far (Pascanu et al., 2013).
- **Initialization.** Weights start small, typically normal with standard deviation around $0.02$: too large and activations grow layer by layer, too small and the signal fades (the variance argument of section 6.3). GPT-2 also scales each output projection's initial weights by $1/\sqrt{N}$ for $N$ residual layers, since $N$ random additions to the residual stream would otherwise grow its variance $N$-fold.
- **Mixed precision.** Forward and backward passes run in bf16 (fp32's range, section 4.3), while master weights and optimizer averages stay fp32 so tiny updates are not rounded away (Micikevicius et al., 2018).

## 10.5 Scaling Laws: How Big, and on How Much Text

Kaplan et al. (2020) found that loss falls as a smooth power law in parameters $N$, training tokens $D$ and compute $C$, over many orders of magnitude. (In this section only, $N$ counts parameters.) A run's compute is about:

$$
C \approx 6 N D
$$

Each token costs about $2N$ operations forward (a multiply and an add per weight) and $4N$ backward (twice the forward, section 2.5). Hoffmann et al. (2022), the "Chinchilla" paper, fitted the loss in $N$ and $D$ separately:

$$
\mathcal{L}(N, D) = E + \frac{A}{N^{a}} + \frac{B}{D^{b}}
$$

$E$ is the loss no model can beat, the randomness of text itself; the other terms penalize too small a model and too little data. The paper's fitted values are $E = 1.69$, $A = 406.4$, $B = 410.7$, $a = 0.34$, $b = 0.28$ (it names the exponents $\alpha$ and $\beta$). Its direct fits of the best model at each budget give a simple rule: grow $N$ and $D$ together, at about 20 tokens per parameter. Minimizing the formula with those constants disagrees, favouring a smaller model on more data; a replication (Besiroglu et al., 2024) traced this to the fit, and refitted constants ($E = 1.82$, $A = 482.01$, $B = 2085.43$, $a = 0.3478$, $b = 0.3658$) agree with the 20-to-1 rule. Earlier models had been far too large for their data: Chinchilla, 70B parameters on 1.4 trillion tokens, beat the 280B Gopher on 300 billion tokens with about the same compute.

> **Interactive figure** `scaling-laws`, on the web page: https://transformers-and-math.drmhse.com/modules/10-pretraining#figure-scaling-laws
>
> **Try:** Slide along the dashed line of equal compute from the smallest model to the largest. Predict the shape of the loss along the way before you look.
>
> **What happens:** It falls and then rises, with a single lowest point. At the small-model end the model cannot hold what the data teaches; at the large-model end it sees too few tokens to fill its capacity. Either way, the same compute buys a worse model than the balanced point.

> **In words.** Lines of equal predicted loss over model size and training tokens, with a dashed line of equal-compute runs. Along it the loss falls then rises; the lowest point is the compute-optimal model. With the refitted constants, at Chinchilla's budget it lands near 70 billion parameters and 1.3 trillion tokens, while Gopher sits on the same budget at a higher loss.

The rule minimizes *training* cost and ignores the cost of *using* the model, paid on every generated token and growing with $N$. So current models train far past the Chinchilla point: LLaMA 3 8B saw 15 trillion tokens, nearly 2,000 per parameter. Smaller and longer-trained costs more to train but less to run.

## Exercises

**Exercise (check).** Name the five stages of building a pretraining dataset described in section 10.1, in order.

*Solution:* Collect, extract and filter, deduplicate, mix, and (increasingly) synthesize.

**Exercise (apply).** Estimate the training compute of an 8-billion-parameter model trained on 15 trillion tokens, in FLOPs. Give the mantissa in units of $10^{23}$.

*Answer:* 7.2 (within 0.05)

*Solution:* $C \approx 6ND = 6 \times 8 \times 10^{9} \times 15 \times 10^{12} = 7.2 \times 10^{23}$ FLOPs.

**Exercise (apply).** How many gigabytes of training state does AdamW need for a 7-billion-parameter model at 16 bytes per parameter, before activations?

*Answer:* 112 (within 1)

*Solution:* $7 \times 10^9 \times 16 = 112 \times 10^9$ bytes, 112 GB: more than one 80 GB GPU holds. Module 11 shows how the state is split across GPUs.

**Exercise (apply).** Under the 20-tokens-per-parameter rule, $C = 6ND$ with $D = 20N$. What is the compute-optimal model size, in billions of parameters, for $C = 5.76 \times 10^{23}$ FLOPs?

*Answer:* 69.3 (within 1)

*Solution:* $C = 6N \cdot 20N = 120N^2$, so $N = \sqrt{C / 120} = \sqrt{4.8 \times 10^{21}} \approx 6.93 \times 10^{10}$: about 69 billion parameters, trained on about 1.4 trillion tokens. That is Chinchilla.

**Exercise (apply).** Two documents have shingle sets of sizes 40 and 50 and share 30 shingles. What is their Jaccard similarity?

*Answer:* 0.5 (within 0.001)

*Solution:* The union has $40 + 50 - 30 = 60$ shingles, so $J = 30 / 60 = 0.5$.

**Exercise (extend).** Give two separate reasons why training on many copies of the same document is harmful, beyond wasted compute.

*Solution:* First, the model sees that text many times and tends to memorize it, so it can reproduce it verbatim, which raises privacy and copyright problems and spends capacity on rote recall. Second, copies of benchmark questions or other evaluation text in the training data inflate evaluation scores, making the model look better than it generalizes (section 13.3). Duplicates also skew the effective data mixture toward whatever happens to be copied most.

### Core Academic References
- **Kingma & Ba (2015)**: *Adam: A Method for Stochastic Optimization*. ICLR 2015. [arXiv:1412.6980](https://arxiv.org/abs/1412.6980)
- **Loshchilov & Hutter (2019)**: *Decoupled Weight Decay Regularization* (AdamW). ICLR 2019. [arXiv:1711.05101](https://arxiv.org/abs/1711.05101)
- **Pascanu, Mikolov & Bengio (2013)**: *On the difficulty of training recurrent neural networks* (gradient clipping). ICML 2013. [arXiv:1211.5063](https://arxiv.org/abs/1211.5063)
- **Micikevicius et al. (2018)**: *Mixed Precision Training*. ICLR 2018. [arXiv:1710.03740](https://arxiv.org/abs/1710.03740)
- **Kaplan et al. (2020)**: *Scaling Laws for Neural Language Models*. [arXiv:2001.08361](https://arxiv.org/abs/2001.08361)
- **Hoffmann et al. (2022)**: *Training Compute-Optimal Large Language Models* (Chinchilla). NeurIPS 2022. [arXiv:2203.15556](https://arxiv.org/abs/2203.15556)
- **Besiroglu et al. (2024)**: *Chinchilla Scaling: A Replication Attempt*. [arXiv:2404.10102](https://arxiv.org/abs/2404.10102)
- **Broder (1997)**: *On the Resemblance and Containment of Documents* (MinHash). Compression and Complexity of Sequences 1997.
- **Lee et al. (2022)**: *Deduplicating Training Data Makes Language Models Better*. ACL 2022. [arXiv:2107.06499](https://arxiv.org/abs/2107.06499)
- **Penedo et al. (2024)**: *The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scale*. [arXiv:2406.17557](https://arxiv.org/abs/2406.17557)