# Module 13: Evaluation

> How LLMs are evaluated: perplexity and its limits, benchmark scoring, contamination, long-context tests, LLM-as-judge, and confidence intervals.

- Part III: Building an LLM · core track · Intermediate · 3 hours
- By the end you can design an evaluation for a model and spot misleading numbers.
- Builds on: [Module 8: Three Families: Encoder, Encoder-Decoder, Decoder](https://transformers-and-math.drmhse.com/md/08-three-families.md), [Module 3: Probability and Information](https://transformers-and-math.drmhse.com/md/03-probability-and-information.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/13-evaluation
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

The loss says how well a model predicts held-out text, not how useful it is: whether it answers correctly, writes working programs, follows instructions, or finds a fact on page 300. This module covers how models are actually measured and how far to trust the numbers. The habits of section 4.6 carry over: a benchmark score is a measurement, with noise, a scale, and ways to be gamed.

## 13.1 Perplexity and Its Limits

Perplexity (section 3.5) is the cheapest evaluation: no labels needed, and within one model family it tracks capability well.

Its main limit is easy to miss. **Perplexity is per token, so it depends on the tokenizer.** Long tokens mean fewer, harder predictions for the same text, so per-token perplexity is higher even when the text is predicted equally well. The total negative log-likelihood of a text, its total surprise, does not depend on how it was cut up, so dividing it by characters or bytes gives *bits per character* or *bits per byte*, comparable across tokenizers:

$$
\text{bits per byte} = \frac{1}{n_{\text{bytes}}} \sum_{t=1}^{T} -\log_2 P_\theta(w_t \mid w_{<t})
$$

Perplexity on web text also says little about skills rare in that text, such as multi-step arithmetic. And post-training (Modules 20 to 22) deliberately makes models *less* like average web text, so their perplexity can rise as they become more useful.

> **Browser lab** `perplexity-vocab`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/13-evaluation#lab-perplexity-vocab

## 13.2 Benchmarks

A *benchmark* is a fixed set of test questions with a scoring rule:

- **Multiple choice** (MMLU, 57 academic subjects, is the best known). Scored either by asking for the letter, or by comparing the likelihood of each option. The two can differ noticeably, so a number means little without its method.
- **Exact match.** A short answer must equal the reference after normalization (lower-casing, stripping punctuation); mathematics benchmarks such as GSM8K compare the final number.
- **Execution.** For code, hidden tests run the model's program (HumanEval, MBPP). The standard measure is *pass@k*, the probability that at least one of $k$ samples passes, estimated without bias from $n \ge k$ samples of which $c$ pass:

$$
\text{pass@}k = 1 - \frac{\binom{n - c}{k}}{\binom{n}{k}}
$$

The exact *prompt* (wording, and examples shown first, *few-shot* prompting) and the *answer extraction* each move scores by several points, so comparisons are fair only with both fixed. Benchmarks also *saturate*: once the best models score near 100%, the field moves to harder ones.

## 13.3 Contamination

*Contamination* means the test questions or answers appeared in the pretraining data. Benchmarks are published, discussed and copied across the web, so a crawl contains many unless they are removed, and a contaminated model scores by recall instead of skill. Imperfect defences:

- **Decontaminate the training data** by removing long overlaps (say 13 words) with benchmark items, using the shingle matching of section 10.1.
- **Test for memorization**: compare published questions with freshly written ones, or check whether the model completes a question verbatim from its first half.
- **Use dynamic or private benchmarks**, refreshed regularly, dated after the training cutoff, or never published.

A softer form: when a field tunes its methods against one benchmark for years, the methods overfit to it even if no model saw a test question.

## 13.4 Long-Context Evaluation

Accepting hundreds of thousands of tokens is not the same as using them. The simplest test, *needle in a haystack*, hides one fact at a chosen depth in long irrelevant text and asks for it. Plotting accuracy by length and depth exposes blind spots; models often miss facts in the middle more than at the start or end.

Passing is necessary but far from sufficient: retrieving one distinctive sentence is much easier than combining scattered facts, following chains of references, or tracking a quantity through a document. Harder benchmarks such as RULER test these and show many models' effective context well short of the advertised length. Sections 15.2 and 16.3 cover the architecture behind long contexts.

## 13.5 Judging Open-Ended Answers

Many tasks, such as writing an email or summarizing a report, have no single correct answer.

- **Human preference.** People pick the better of two unlabelled answers, and the Bradley–Terry model of section 3.6 turns many such votes into ratings; Chatbot Arena ranks public models this way from millions of crowd votes.
- **LLM-as-judge.** A strong model scores answers against a rubric. It is cheap and agrees with humans reasonably often, but it favors the answer shown *first*, *longer* answers, and its own style. Mitigations: evaluate both orderings, control for length, and check against a sample of human judgments.

Both measure preference, not correctness: a fluent, confident, wrong answer can win.

## 13.6 How Sure Is a Score?

Less sure than a single number suggests. A benchmark of $n$ questions samples the skill it measures. If each question is answered correctly with probability $p$, each is a 0-or-1 variable with variance $p(1-p)$, and by section 3.2 the accuracy has *standard error*

$$
\text{SE} = \sqrt{\frac{p(1 - p)}{n}}
$$

An approximate 95% *confidence interval* is the accuracy plus or minus two standard errors. At 70% on 200 questions, $\text{SE} \approx 0.032$, so the interval is roughly 64% to 76%, and models scoring 70% and 73% cannot be told apart.

> **Interactive figure** `benchmark-ci`, on the web page: https://transformers-and-math.drmhse.com/modules/13-evaluation#figure-benchmark-ci
>
> **Try:** A model's true accuracy is 70%. On a 100-question benchmark, guess how far the measured score typically wanders from run to run of fresh questions, then try 1,000 questions.
>
> **What happens:** On 100 questions the score typically lands anywhere from about 61% to 79%. On 1,000 it stays within about 67% to 73%. The interval narrows like $1/\sqrt{n}$, so ten times the questions buys only about three times the precision.

> **In words.** Fresh benchmarks of a chosen size are drawn for a model of chosen true accuracy and their scores histogrammed, against a shaded band for the predicted 95% interval.

Sampling randomness in generation (run several samples or use greedy decoding, and say which) and prompt sensitivity (section 13.2) add more noise. A careful report gives the interval, the number of samples and the prompt.

## Exercises

**Exercise (check).** Why can't the perplexity of a model with a 32,000-token vocabulary be compared with that of a model with a 256,000-token vocabulary on the same text?

*Solution:* Perplexity is an average per token, and the two tokenizers cut the same text into different numbers of tokens of different difficulty. The model with longer tokens makes fewer, harder predictions, so its per-token perplexity is higher even at equal quality. Normalize by characters or bytes instead.

**Exercise (apply).** What is the approximate 95% confidence interval half-width (two standard errors) for an accuracy of 70% measured on 200 questions?

*Answer:* 0.0648 (within 0.002)

*Solution:* $\text{SE} = \sqrt{0.7 \times 0.3 / 200} = \sqrt{0.00105} \approx 0.0324$, so two standard errors are about $0.065$: the interval is roughly 63.5% to 76.5%.

**Exercise (apply).** Ten programs are sampled and 3 pass. Estimate pass@1 and pass@5.

*Answer:* 0.9167 (within 0.001)

*Solution:* pass@1 $= 1 - \binom{7}{1}/\binom{10}{1} = 1 - 7/10 = 0.3$. pass@5 $= 1 - \binom{7}{5}/\binom{10}{5} = 1 - 21/252 \approx 0.917$. (The checked answer is pass@5.)

**Exercise (apply).** A model's total negative log-likelihood on a 1,000-byte text is 1,100 bits. What is its bits per byte?

*Answer:* 1.1 (within 0.001)

*Solution:* $1{,}100 / 1{,}000 = 1.1$ bits per byte, a figure that can be compared with any other model's on the same text, whatever its tokenizer.

**Exercise (extend).** An LLM judge prefers answer A over answer B 65% of the time when A is shown first, and prefers B over A 55% of the time when B is shown first. What do you conclude, and how would you report the comparison?

*Solution:* The judge has a position bias: it favours whichever answer comes first. Shown first, A wins 65%; shown second, A wins 45% (since B wins 55%). Averaging the two orders gives A winning $(65 + 45)/2 = 55\%$. Report the order-averaged figure, note the bias, and ideally check the judge against a sample of human judgments.

### Core Academic References
- **Hendrycks et al. (2021)**: *Measuring Massive Multitask Language Understanding* (MMLU). ICLR 2021. [arXiv:2009.03300](https://arxiv.org/abs/2009.03300)
- **Chen et al. (2021)**: *Evaluating Large Language Models Trained on Code* (HumanEval, pass@k). [arXiv:2107.03374](https://arxiv.org/abs/2107.03374)
- **Cobbe et al. (2021)**: *Training Verifiers to Solve Math Word Problems* (GSM8K). [arXiv:2110.14168](https://arxiv.org/abs/2110.14168)
- **Liu et al. (2024)**: *Lost in the Middle: How Language Models Use Long Contexts*. TACL 12. [arXiv:2307.03172](https://arxiv.org/abs/2307.03172)
- **Hsieh et al. (2024)**: *RULER: What's the Real Context Size of Your Long-Context Language Models?*. [arXiv:2404.06654](https://arxiv.org/abs/2404.06654)
- **Zheng et al. (2023)**: *Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena*. NeurIPS 2023. [arXiv:2306.05685](https://arxiv.org/abs/2306.05685)
- **Sainz et al. (2023)**: *NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark*. Findings of EMNLP 2023. [arXiv:2310.18018](https://arxiv.org/abs/2310.18018)
- **Miller (2024)**: *Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations*. [arXiv:2411.00640](https://arxiv.org/abs/2411.00640)