# Module 21: Preference Learning: RL, RLHF and DPO

> Reinforcement learning from human feedback from first principles: policy gradients, PPO, reward models, RLHF, and direct preference optimization (DPO).

- Part V: From Model to Assistant · core track · Advanced · 5 hours
- By the end you can explain RLHF and DPO from first principles, including the RL vocabulary.
- Builds on: [Module 12: Generation](https://transformers-and-math.drmhse.com/md/12-generation.md), [Module 3: Probability and Information](https://transformers-and-math.drmhse.com/md/03-probability-and-information.md)
- Web page (interactive figures, labs and equation explainers): https://transformers-and-math.drmhse.com/modules/21-preference-learning
- From *Transformers & Math* by Mike Chumba: https://transformers-and-math.drmhse.com

Instruction tuning (section 20.1) teaches by imitation, so it cannot make a model *better* than its examples, which are expensive to write. It is far easier for people to say which of two answers is better. This module turns such judgments into training, using a small piece of *reinforcement learning* (RL), which learns from scores rather than correct answers, then builds the reward model, the RLHF loop, and DPO, the shortcut that removes most of that machinery. Module 22 uses the same tools to train reasoning.

## 21.1 A Reinforcement Learning Primer

- An **agent** takes **actions** in an **environment**, each changing the **state** it sees next.
- A **policy** $\pi(a \mid s)$ is its rule for acting: a distribution over actions given the state.
- A **reward** scores an outcome. An **episode** runs from start to finish, and its **return** $R$ is its total reward.
- The goal is to adjust the policy to maximize the expected return.

Each has a language-model counterpart:

| RL | language model |
|---|---|
| state | the prompt plus the tokens generated so far |
| action | the next token |
| policy $\pi_\theta(a \mid s)$ | the model's next-token distribution |
| episode | generating one complete response $y$ to a prompt $x$ |
| reward | a score for the finished response, usually given once at the end |

So a language model already *is* a policy: $\pi_\theta(y \mid x)$ is the product of its next-token probabilities (section 3.3). The objective is the expected reward of its responses:

$$
J(\theta) = \mathbb{E}_{x \sim \mathcal{D},\; y \sim \pi_\theta(\cdot \mid x)}\left[R(x, y)\right]
$$

$y \sim \pi_\theta(\cdot \mid x)$ reads "the response is sampled from the model". Unlike supervised learning, there is no correct answer to imitate, only a score for whatever the model produced, so it must *explore* and learn from how its responses scored.

## 21.2 Policy Gradients

$J$ averages over responses *sampled from the model itself*, and sampling has no derivative. The *log-derivative trick*, $\nabla \pi = \pi \, \nabla \log \pi$, gets around it:

$$
\nabla_\theta J = \mathbb{E}_{y \sim \pi_\theta}\left[R(x, y)\, \nabla_\theta \log \pi_\theta(y \mid x)\right]
$$

This *policy gradient* is estimated from samples by REINFORCE (Williams, 1992): sample a response, score it, and step toward making it more likely, scaled by its reward. $\nabla_\theta \log \pi_\theta(y \mid x)$ is the gradient of the response's log-probability, which backpropagation computes as in training.

The estimate is right on average but very noisy: if every reward is positive, *every* sample is pushed up, and only the relative sizes carry information. The fix subtracts a **baseline** $b$, an estimate of the typical reward:

$$
\nabla_\theta J = \mathbb{E}_{y \sim \pi_\theta}\left[\big(R(x, y) - b\big)\, \nabla_\theta \log \pi_\theta(y \mid x)\right]
$$

The average gradient is unchanged, since $\mathbb{E}[\nabla \log \pi] = \nabla \sum \pi = \nabla 1 = 0$, but the noise falls: better-than-usual responses are pushed up and worse ones *down*. $A = R - b$ is the **advantage**.

> **Interactive figure** `policy-gradient-bandit`, on the web page: https://transformers-and-math.drmhse.com/modules/21-preference-learning#figure-policy-gradient-bandit
>
> **Try:** The simplest RL problem: three actions with fixed average rewards 5.0, 5.5 and 6.0, all positive. Run REINFORCE without a baseline, then with one. Before you do, predict which version converges to the best action faster and more reliably.
>
> **What happens:** With the baseline. Without it, every sampled action has a large positive reward and is pushed up, so the policy drifts toward whatever it happens to sample often: the runs differ wildly, some barely finding the best action. With the running-average baseline, only better-than-average actions are pushed up and worse ones pushed down, so the runs head to the best action together. Overlay six runs of each to see the spread.

> **In words.** A three-armed bandit (one state, three actions with noisy rewards around fixed means) trained with REINFORCE, with and without a baseline. The chart traces each action's probability over training, and several runs can be overlaid to compare their spread.

## 21.3 PPO

At language-model scale, one large step can change the policy so much that its samples no longer describe it, and training collapses. *Proximal Policy Optimization* (PPO, Schulman et al., 2017) limits each update. It takes several gradient steps on each sampled batch, comparing each token's new probability with the sampling policy's, $\rho = \pi_\theta(a \mid s) / \pi_{\text{old}}(a \mid s)$, in a *clipped* objective:

$$
\mathcal{L}_{\text{PPO}} = \mathbb{E}\left[\min\left(\rho\, A, \; \operatorname{clip}(\rho, 1 - \epsilon, 1 + \epsilon)\, A\right)\right]
$$

With $\epsilon$ typically 0.2, inside $[0.8, 1.2]$ this is the advantage-weighted policy gradient; once a good action is 20% more likely (or a bad one 20% less), the clip removes the incentive to go further on this batch. The policy moves in many small, safe steps.

For language models, PPO adds a **value model**, a copy of the language model with a scalar output that predicts the expected reward from each partial response and supplies a per-token baseline, and a **KL penalty** toward the starting model (section 21.5). That is four models in memory at once (policy, reference, reward, value), the main reason simpler methods were sought.

## 21.4 Reward Models From Preferences

A chat response's reward has no formula, so RLHF, as used for InstructGPT (Ouyang et al., 2022), learns it from people. They compare pairs of responses to a prompt $x$, a winner $y_w$ and a loser $y_l$, and a *reward model* $r_\phi$, usually a copy of the language model with a scalar output, is trained to score the winner higher:

$$
\mathcal{L}_{\text{RM}} = -\log \sigma\left(r_\phi(x, y_w) - r_\phi(x, y_l)\right)
$$

This is the Bradley–Terry model of section 3.6 fitted by maximum likelihood: $\sigma(r_w - r_l)$ is the predicted probability of preferring $y_w$, and only differences of reward are meaningful.

## 21.5 RLHF End to End

Then PPO trains $\pi_\theta$ toward responses the reward model scores highly, without drifting far from the instruction-tuned model $\pi_{\text{ref}}$ it started from:

$$
\max_{\theta} \; \mathbb{E}_{x, \, y \sim \pi_\theta}\left[r_\phi(x, y)\right] - \beta \, \operatorname{KL}\left(\pi_\theta(\cdot \mid x) \,\|\, \pi_{\text{ref}}(\cdot \mid x)\right)
$$

$\beta$ sets how strongly drift is penalized; in practice $-\beta \log \frac{\pi_\theta}{\pi_{\text{ref}}}$ is added to the reward token by token. The penalty matters: the reward model only approximates human judgment, and a policy optimized against it without restraint finds outputs that score well and read badly, *reward hacking*. The result was striking: people preferred the 1.3B-parameter InstructGPT's answers to the 175B-parameter GPT-3's.

## 21.6 Direct Preference Optimization

RLHF's reward model, sampling and RL loop are all hard to tune. Rafailov et al. (2023) showed that the RLHF optimum has a closed form, which eliminates the reward model, in three steps.

1. **The best policy has a closed form.** For a fixed reward $r$, section 21.5's objective is maximized by reweighting the reference model by the exponentiated reward:
$$
\pi^*(y \mid x) = \frac{1}{Z(x)}\, \pi_{\text{ref}}(y \mid x)\, \exp\left(\frac{r(x, y)}{\beta}\right)
$$
where $Z(x)$ makes the probabilities sum to 1. Responses the reward likes become more likely, by a factor $\beta$ controls; responses the reference never produces stay unlikely.
2. **So the reward can be read off the policy:** $r(x, y) = \beta \log \frac{\pi^*(y \mid x)}{\pi_{\text{ref}}(y \mid x)} + \beta \log Z(x)$. Any policy implicitly defines the reward for which it is optimal.
3. **Substitute into Bradley–Terry.** Preferences depend only on the *difference* of two rewards for the same prompt, so $\beta \log Z(x)$ cancels, leaving only the policy and the reference model.

Maximum likelihood on the preferences, with $\pi_\theta$ standing in for $\pi^*$, gives a loss on the pairs themselves:

$$
\mathcal{L}_{\text{DPO}} = -\log \sigma\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\text{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\text{ref}}(y_l \mid x)}\right)
$$

Each log-ratio measures how much more likely the model has made a response than the reference did; the loss pushes the winner's up and the loser's down. ($\beta$ plays its RLHF role, unrelated to Adam's $\beta_1$ and $\beta_2$.) It is ordinary supervised training on a fixed dataset, which is why DPO and its variants are widely used alongside or instead of RLHF.

> **Browser lab** `dpo-toy`: an experiment that runs in the reader's browser on the web page: https://transformers-and-math.drmhse.com/modules/21-preference-learning#lab-dpo-toy

## Exercises

**Exercise (check).** Map these RL terms onto a language model: state, action, policy, episode.

*Solution:* State: the prompt plus the tokens generated so far. Action: the next token. Policy: the model's next-token distribution. Episode: generating one complete response.

**Exercise (apply).** A reward model scores the preferred answer 2.0 and the other 0.5. What is $\mathcal{L}_{\text{RM}}$ for this pair?

*Answer:* 0.2014 (within 0.001)

*Solution:* $\sigma(2.0 - 0.5) = \sigma(1.5) = 1/(1 + e^{-1.5}) \approx 0.8176$, so $\mathcal{L}_{\text{RM}} = -\log 0.8176 \approx 0.201$.

**Exercise (apply).** In PPO with $\epsilon = 0.2$ and a positive advantage $A = 1$, what is the clipped objective's value for a token whose probability ratio is $\rho = 1.5$?

*Answer:* 1.2 (within 0)

*Solution:* $\min(\rho A, \operatorname{clip}(\rho, 0.8, 1.2) A) = \min(1.5, 1.2) = 1.2$. Beyond $\rho = 1.2$ the objective stops rising, so there is no incentive to push this token's probability further on this batch.

**Exercise (apply).** Derive the optimal policy $\pi^* \propto \pi_{\text{ref}} \exp(r/\beta)$ for a single prompt with two possible responses, by maximizing $p\,r_1 + (1-p)\,r_2 - \beta \operatorname{KL}$, where $p$ is the probability of response 1 and the reference gives it $q$.

*Solution:* The KL term is $p \log\frac{p}{q} + (1-p)\log\frac{1-p}{1-q}$. Setting the derivative with respect to $p$ to zero: $r_1 - r_2 - \beta\left(\log\frac{p}{q} - \log\frac{1-p}{1-q}\right) = 0$, so $\frac{p}{1-p} = \frac{q}{1-q}\, e^{(r_1 - r_2)/\beta}$. That is exactly $p \propto q\, e^{r_1/\beta}$ and $1 - p \propto (1-q)\,e^{r_2/\beta}$, normalized: $\pi^* \propto \pi_{\text{ref}} \exp(r/\beta)$.

**Exercise (extend).** Explain reward hacking with an example, and say which part of the RLHF objective guards against it.

*Solution:* The reward model is only an approximation of human judgment, so a policy optimized hard against it finds outputs it scores highly but people would not prefer: for example, if raters tended to prefer longer answers, the policy learns to pad answers with unneeded detail, or to sound confident and agreeable rather than correct. The KL penalty toward the reference model limits how far the policy can move into such territory, which the reward model never saw in training and is least reliable about.

### Core Academic References
- **Sutton & Barto (2018)**: *Reinforcement Learning: An Introduction (2nd ed.)*. MIT Press.
- **Williams (1992)**: *Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning* (REINFORCE). Machine Learning, 8: 229–256.
- **Schulman et al. (2017)**: *Proximal Policy Optimization Algorithms*. [arXiv:1707.06347](https://arxiv.org/abs/1707.06347)
- **Christiano et al. (2017)**: *Deep Reinforcement Learning from Human Preferences*. NeurIPS 2017. [arXiv:1706.03741](https://arxiv.org/abs/1706.03741)
- **Ouyang et al. (2022)**: *Training language models to follow instructions with human feedback* (InstructGPT). NeurIPS 2022. [arXiv:2203.02155](https://arxiv.org/abs/2203.02155)
- **Rafailov et al. (2023)**: *Direct Preference Optimization: Your Language Model is Secretly a Reward Model*. NeurIPS 2023. [arXiv:2305.18290](https://arxiv.org/abs/2305.18290)
- **Gao, Schulman & Hilton (2023)**: *Scaling Laws for Reward Model Overoptimization*. ICML 2023. [arXiv:2210.10760](https://arxiv.org/abs/2210.10760)