Policy — a rule for choosing actions
In reinforcement learning, a probability distribution over actions given the current state. A language model is a policy over next tokens.
The goal of RL is to change the policy to maximize expected reward (section 21.1).
Related terms
- Reward — a score for an outcomeA number saying how good an outcome was. For a chat response it comes from a learned reward model; for maths or code it can come from a checker.
- Advantage — reward relative to a baselineHow much better a response scored than a typical response, R − b. Policy gradients weighted by the advantage are far less noisy than those weighted by the raw reward.