# Module 22: Reasoning Models

> How reasoning models are trained: chain-of-thought, test-time compute, RL with verifiable rewards and GRPO, process rewards, and their failure modes.

- Part V: From Model to Assistant · core track · Advanced · 4 hours
- By the end you can explain how models are trained to reason, and how compute at answer time trades against model size.
- Builds on: [Module 21: Preference Learning: RL, RLHF and DPO](https://transformers-and-math.drmhse.com/md/21-preference-learning.md), [Module 13: Evaluation](https://transformers-and-math.drmhse.com/md/13-evaluation.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/22-reasoning-models
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

Since 2024 the strongest models write a long working-out before answering hard questions, sometimes thousands of tokens of trying, checking and backtracking. These *reasoning models* are trained with reinforcement learning on problems whose answers can be checked automatically. This module explains why writing out steps helps a transformer, how answer-time compute can substitute for model size, how the training works, and what goes wrong. The mechanisms are well established; numbers from particular models are left out because they date fast.

## 22.1 Chain-of-Thought

Demand an immediate answer to a multi-step arithmetic question and a model often fails; ask it to "think step by step" and it often succeeds. Writing intermediate steps first, *chain-of-thought* (Wei et al., 2022), helps for a structural reason.

A transformer does a fixed amount of computation per token, one pass through its $L$ blocks, so a question needing more sequential steps than one pass cannot be answered in one token, however large the model. But each written token is fed back as input (section 12.1) for later tokens to attend to. **Generating more tokens buys more computation**, with the written text as working memory between passes.

The steps also break a hard prediction into easy ones that look like familiar text: worked solutions, derivations, code.

Chain-of-thought can be *prompted*, by asking or by showing worked examples (section 14.3). Reasoning models are *trained* to produce long chains on their own (section 22.3).

## 22.2 Spending Compute at Answer Time

So a model can trade compute at answer time (*test-time compute*) for accuracy:

- **Longer reasoning.** Reasoning models often improve steadily with their thinking budget.
- **Self-consistency.** Sample $n$ chains and take a *majority vote* over their answers (Wang et al., 2023). If the right answer is likelier than any single wrong one, the majority beats a single sample.
- **Best-of-$n$ with a verifier.** A *verifier* picks among $n$ answers: unit tests for code, an exact checker for maths, or a learned scoring model. With a perfect verifier, success needs only one right sample:

$$
P(\text{success with } n) = 1 - (1 - p)^n
$$

With an imperfect learned verifier, gains flatten and can reverse at large $n$, as more candidates mean more chances to fool it: reward hacking again (section 21.5).

So model size and answer-time compute are partly interchangeable: on some problems a smaller model with a larger budget matches a larger model answering directly (Snell et al., 2024).

> **Interactive figure** `best-of-n`, on the web page: https://transformers-and-math.drmhse.com/modules/22-reasoning-models#figure-best-of-n
>
> **Try:** Each sample is right with probability 0.3. Before moving the slider, estimate how many samples are needed for best-of-$n$ with a perfect verifier to succeed 90% of the time, and what majority voting does at the same $p$.
>
> **What happens:** About 7: $1 - 0.7^7 \approx 0.92$. Majority voting climbs too, but slowly: with wrong answers spread over four values (about 0.175 each), the right answer is still the single most common one, so the vote converges on it as $n$ grows. If the wrong answers piled onto one value instead (probability 0.7), voting would converge on that wrong answer. A verifier helps when it is reliable; voting helps when the right answer is the most common one.

> **In words.** Accuracy against samples, from 1 to 64, for best-of-n with a perfect verifier, best-of-n with an imperfect one, and majority voting. Sliders set per-sample accuracy and verifier reliability.

> **Browser lab** `self-consistency`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/22-reasoning-models#lab-self-consistency

## 22.3 RL With Verifiable Rewards and GRPO

Reasoning models are made by *RL with verifiable rewards* (RLVR) on problems a program can check: mathematics with known answers, code with unit tests, verifiable puzzles. The reward is often 1 for a correct final answer and 0 otherwise, sometimes plus a little for the required format. With no reward model, only the checker itself can be hacked.

The model is never shown how to reason. The policy gradient (section 21.2) simply makes whatever reasoning led to correct answers more likely, and long chains, self-checking ("wait, let me verify that") and backtracking emerge because they raise the reward (DeepSeek-AI, 2025).

### GRPO
*Group Relative Policy Optimization* (GRPO, Shao et al., 2024) replaces PPO's value model (section 21.3) with something cheaper: sample a *group* of $G$ responses per prompt and use the group's own statistics as the baseline:

$$
A_i = \frac{r_i - \operatorname{mean}(r_1, \dots, r_G)}{\operatorname{std}(r_1, \dots, r_G)}
$$

Every token of response $i$ gets advantage $A_i$, positive if it beat its siblings, and the update is PPO's clipped objective plus a KL penalty. Dropping the value model and its training matters at the scale of long reasoning chains.

A prompt that every sample gets right (or wrong) has all advantages zero and teaches nothing, so learning concentrates at the edge of the model's ability, and reasoning training sets are often filtered to that difficulty.

> **Interactive figure** `grpo-advantages`, on the web page: https://transformers-and-math.drmhse.com/modules/22-reasoning-models#figure-grpo-advantages
>
> **Try:** A group of eight answers to one problem has rewards [1, 0, 0, 1, 0, 0, 0, 0]. Before computing, predict the sign and rough size of the advantage for a correct answer, and what happens if all eight are correct.
>
> **What happens:** The mean is 0.25 and the standard deviation about 0.43, so each correct answer gets $A \approx +1.73$ and each wrong one $A \approx -0.58$: rare successes are pushed up strongly. If all eight are correct, every reward equals the mean and every advantage is 0: nothing is learned from a problem the model already always solves.

> **In words.** A group of answers as a row of tiles marked correct or incorrect, toggled by clicking, each with its group-relative advantage drawn as a bar above or below zero.

## 22.4 Outcome and Process Rewards

An *outcome reward*, on the final answer alone, is easy to compute and hard to game but sparse: a 5,000-token chain earns one bit, a chain right until its last line scores the same 0 as one wrong from the start, and a right answer from flawed reasoning is rewarded.

A *process reward* scores each step. Lightman et al. (2023) trained a *process reward model* on human step labels and found it a better verifier than an outcome-only one on hard mathematics. But step labels are expensive, and a learned step-scorer can itself be exploited, so large-scale reasoning training mostly uses outcome rewards, with process supervision used for verification and data selection.

## 22.5 Distilling Reasoning

RLVR is expensive and works best on large models. Small models inherit much of it by sequence-level distillation (section 20.4): fine-tune on a large reasoning model's chains that reach correct answers. DeepSeek-AI (2025) found this beat training the same small models with RL directly, at a fraction of the cost. Needing only the teacher's text is part of why reasoning spread quickly to open models.

## 22.6 Failure Modes

- **Unfaithful reasoning.** Altering a chain or adding a misleading hint shows models sometimes answer for reasons their stated reasoning omits (Turpin et al., 2023). A chain of thought is evidence about the reasoning, not a transcript of it.
- **Overthinking.** Thousands of tokens on easy questions, sometimes talking itself out of a correct answer. Answer-time compute is paid on every query.
- **Reward hacking, again.** A weak checker can be satisfied without solving the problem, by code that special-cases the tests, say. The checker defines what the model learns, flaws included.
- **Narrow transfer.** Skills trained on mathematics and code transfer only partly to domains with no checker, such as open-ended writing.

## Exercises

**Exercise (check).** Why can writing intermediate tokens let a transformer solve problems it cannot answer in one token?

*Solution:* Each generated token costs one fixed-depth pass through the model, and each written token is fed back as input. Writing steps gives the model more passes of computation, with the written text acting as working memory that later tokens can attend to.

**Exercise (apply).** Each sample is correct with probability 0.4. With a perfect verifier, what is the success rate of best-of-5?

*Answer:* 0.92224 (within 0.001)

*Solution:* $1 - (1 - 0.4)^5 = 1 - 0.6^5 = 1 - 0.07776 = 0.922$.

**Exercise (apply).** A GRPO group of four responses has rewards $[1, 1, 0, 0]$. What is the advantage of a correct response (use the population standard deviation)?

*Answer:* 1 (within 0.001)

*Solution:* The mean is 0.5 and the standard deviation is $\sqrt{\tfrac14(0.25 \times 4)} = 0.5$. A correct response gets $(1 - 0.5)/0.5 = 1$, an incorrect one $-1$.

**Exercise (apply).** Why does GRPO need no value model, and what does a prompt whose group is all correct contribute to the update?

*Solution:* The baseline comes from the other responses to the same prompt: the group mean replaces the value model's prediction of the expected reward. If every response in the group is correct, every reward equals the mean, every advantage is zero, and the prompt contributes nothing to the gradient.

**Exercise (extend).** A team's verifier for a coding task runs three unit tests. After RL training, the pass rate rises sharply, but users report that programs often fail on their own inputs. What probably happened, and how would you fix it?

*Solution:* The policy learned to satisfy the three tests rather than to solve the task, for example by special-casing the tested inputs: reward hacking against a weak verifier. Fixes include much larger and hidden test suites, randomized property-based tests, held-out evaluation on fresh tests, and inspecting high-reward samples for test-specific code.

### Core Academic References
- **Wei et al. (2022)**: *Chain-of-Thought Prompting Elicits Reasoning in Large Language Models*. NeurIPS 2022. [arXiv:2201.11903](https://arxiv.org/abs/2201.11903)
- **Wang et al. (2023)**: *Self-Consistency Improves Chain of Thought Reasoning in Language Models*. ICLR 2023. [arXiv:2203.11171](https://arxiv.org/abs/2203.11171)
- **Cobbe et al. (2021)**: *Training Verifiers to Solve Math Word Problems*. [arXiv:2110.14168](https://arxiv.org/abs/2110.14168)
- **Lightman et al. (2023)**: *Let's Verify Step by Step* (process reward models). ICLR 2024. [arXiv:2305.20050](https://arxiv.org/abs/2305.20050)
- **Snell et al. (2024)**: *Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters*. [arXiv:2408.03314](https://arxiv.org/abs/2408.03314)
- **Shao et al. (2024)**: *DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models* (GRPO). [arXiv:2402.03300](https://arxiv.org/abs/2402.03300)
- **DeepSeek-AI (2025)**: *DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning*. [arXiv:2501.12948](https://arxiv.org/abs/2501.12948)
- **Turpin et al. (2023)**: *Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting*. NeurIPS 2023. [arXiv:2305.04388](https://arxiv.org/abs/2305.04388)
- **Merrill & Sabharwal (2024)**: *The Expressive Power of Transformers with Chain of Thought*. ICLR 2024. [arXiv:2310.07923](https://arxiv.org/abs/2310.07923)