Introduction

Next-token prediction in a language model does not directly teach it which answer a person prefers. For the same question, both a factual answer and a plausible but incorrect answer may be natural text.

Preference learning uses data that compares two responses.

$$(x,y_w,y_l)$$

• $x$: prompt
• $y_w$: the preferred winner response
• $y_l$: the less-preferred loser response

RLHF turns human preferences into rewards and optimizes a policy with reinforcement learning. RLAIF replaces some of the people producing preferences with AI feedback. DPO learns a policy directly from preference pairs, without a separate reward model or on-policy RL loop.

Rather than memorizing the names, let us follow where the data comes from and which objective changes the policy.

Viewing Language Generation as Reinforcement Learning

The generation of one answer can be mapped to an MDP.

• State $S_t$: the prompt and tokens generated so far
• Action $A_t$: the next token to generate
• Transition: append the selected token to the sentence
• Episode: the process until one response ends
• Reward: a preference score for the complete response

$$S_t = (x,y_{<t}), \qquad A_t=y_t$$

The policy is the language model.

$$\pi_\theta(y_t\mid x,y_{<t})$$

The probability of a complete response is the product of token probabilities.

$$\pi_\theta(y\mid x) = \prod_{t=1}^{T} \pi_\theta(y_t\mid x,y_{<t})$$

In practice, we use the numerically stable sum of log probabilities.

$$\log\pi_\theta(y\mid x) = \sum_{t=1}^{T} \log\pi_\theta(y_t\mid x,y_{<t})$$

The First Stage of RLHF: SFT

We begin with supervised fine-tuning on demonstrations written or selected by people.

$$L_{\text{SFT}}(\theta) = - \mathbb E_{(x,y)\sim\mathcal D_{\text{demo}}} \left[ \log\pi_\theta(y\mid x) \right]$$

SFT teaches the format for answering prompts and establishes basic behavior. It turns a base model that generates arbitrary text into an initial policy that follows instructions.

One demonstration, however, cannot express the fine-grained relative preference among many possible answers. We therefore compare several responses to the same prompt.

The Second Stage: Preference Data

Generate two responses from the current model and ask a reviewer to choose the better one.

$$y_w\succ y_l$$

Pairwise comparison may be more consistent than assigning an absolute score such as $7.3$. The label can still depend on the preference criteria, the reviewer's knowledge, and the prompt distribution.

The Third Stage: A Reward Model

A reward model $r_\phi(x,y)$ receives a prompt and response and outputs a scalar score.

We can use a Bradley–Terry probability so that the winner receives the higher score.

$$P(y_w\succ y_l\mid x) = \sigma \left( r_\phi(x,y_w)-r_\phi(x,y_l) \right)$$

Here,

$$\sigma(z) = \frac{1}{1+e^{-z}}.$$

The reward-model loss can be written as

$$L_{\text{RM}}(\phi) = - \mathbb E \left[ \log\sigma \left( r_\phi(x,y_w)-r_\phi(x,y_l) \right) \right].$$

Training increases the difference between the winner and loser scores.

The Fourth Stage: Optimizing the Policy with PPO

PPO updates the policy to increase the reward-model score. Maximizing only the reward, however, can produce text that exploits weaknesses in the reward model. A KL penalty keeps the policy from moving too far from the SFT policy or another reference policy.

$$\max_\theta \mathbb E_{y\sim\pi_\theta(\cdot\mid x)} \left[ r_\phi(x,y) - \beta D_{\mathrm{KL}} \left( \pi_\theta(\cdot\mid x) \parallel \pi_{\text{ref}}(\cdot\mid x) \right) \right]$$

A larger $\beta$ keeps the policy closer to the reference. A smaller value lets it move more freely toward the reward.

RLAIF: Changing the Source of Feedback

In Reinforcement Learning from AI Feedback, or RLAIF, an AI system compares responses instead of, or alongside, human reviewers. We can give the AI reviewer principles or a constitution, then ask it to produce critiques and preferences indicating which answer better follows those principles.

Importantly, RLAIF does not necessarily imply an entirely different training pipeline. The preference label can retain the same form.

$$(x,y_w,y_l)$$

We may train a reward model on those labels and run RL, or use a direct preference objective such as the one introduced below.

AI feedback can produce many pairs quickly, but it does not make preferences objective.

• Situations omitted by the principles remain blind spots.
• The AI reviewer's biases and errors enter the labels.
• The evaluator and policy may share similar weaknesses.
• Human audits remain necessary in important domains.

DPO: No Separate Reward Model

Direct Preference Optimization, or DPO, uses the same preference pairs but skips training a separate reward model and running PPO rollouts.

The core idea is to train the policy directly so that the winner becomes relatively more likely than under the reference, while the loser becomes relatively less likely.

First consider the difference in log probability between the policy and reference.

$$\log \frac{ \pi_\theta(y\mid x) }{ \pi_{\text{ref}}(y\mid x) } = \log\pi_\theta(y\mid x) - \log\pi_{\text{ref}}(y\mid x)$$

The DPO loss is

$$\begin{aligned} L_{\text{DPO}}(\theta) = -\mathbb E \Bigg[ \log\sigma \Bigg( \beta \Bigg[ &\log\frac{\pi_\theta(y_w\mid x)}{\pi_{\text{ref}}(y_w\mid x)} \\ - &\log\frac{\pi_\theta(y_l\mid x)}{\pi_{\text{ref}}(y_l\mid x)} \Bigg] \Bigg) \Bigg]. \end{aligned}$$

Read the expression inside the brackets as two questions.

1. How much did the winner's probability increase relative to the reference?
2. How much did the loser's probability increase relative to the reference?

The preference is reflected correctly when the first quantity becomes greater than the second.

How Should We Read β in DPO?

$\beta$ controls the relationship to the reference policy and the scale of the preference signal. Explanations emphasize different aspects depending on the implementation and derivation, but in practice it is an important hyperparameter that determines how strongly the policy reacts to the preference pairs.

If training increases only the winner probability too aggressively, response diversity may shrink or the model may overfit the training pairs. If the response is too weak, the preference may not be reflected sufficiently.

Is RLHF or DPO Always Better?

Neither method is universally superior.

RLHF + PPO

• The reward model can also support other analyses or online sampling.
• Training generates new responses from the current policy and explores their rewards.
• The pipeline and hyperparameters are complex, and the computational cost is high.
• Reward hacking and training instability must be managed.

DPO

• Its implementation resembles supervised fine-tuning and is simpler.
• It requires neither a separate reward model nor on-policy RL infrastructure.
• It depends heavily on the coverage of a fixed preference dataset.
• It does not explore behavior absent from the pairs through online sampling.

The choice should depend on the available data, compute, and evaluation method.

Preference Is Not the True Objective

Neither approach can exceed the quality of its preference data.

• Short, confident answers may be preferred more often than they should be.
• A reviewer unable to check facts may choose a plausible error.
• Different people's values may be mixed into one label.
• Training prompts may differ from prompts in actual use.

An increase in preference score does not prove that factuality, safety, and usefulness all improved. Each must be checked with a separate evaluation.

What to Remember

RLHF, RLAIF, and DPO all use comparative preferences, but they differ in the source of feedback and the route by which the policy is optimized.

1. RLHF follows the sequence SFT → human preferences → reward model → PPO.
2. The KL penalty prevents the learned policy from moving too far from the reference.
3. RLAIF replaces part of the preference judgment with principle-guided AI feedback.
4. DPO directly optimizes winner–loser probabilities without a reward model or on-policy RL loop.
5. No method automatically solves bias and coverage problems in preference data.

The final article turns from algorithms to reward design and evaluation, which should be examined first.

References

• Ouyang et al., Training language models to follow instructions with human feedback
• Bai et al., Constitutional AI: Harmlessness from AI Feedback
• Rafailov et al., Direct Preference Optimization