Preliminary

Learning human preferences

The most basic way to train an LLM is simpler than it might seem: give it the beginning of a sentence and ask it to predict the next token.

For example, consider the following Korean sentence fragment:

For lunch today, I ate kimchi stew

Training simply increases the probability that an appropriate continuation follows. By learning from countless sentences on the internet this way, the model acquires grammar, knowledge, and even some reasoning ability.

There is one problem. Predicting the next token well and giving the answer a person wants are not exactly the same task.

The internet contains kind and unkind answers, accurate and incorrect writing, logically organized explanations, and a remarkable amount of confident nonsense.

To a pretrained model, they are all equivalent training data. It does not judge whether an answer is good or bad; it only learns which token is likely to appear next. It resembles a student who has read an enormous number of books but has no idea how to write an exam response that earns a high score.

SFT, or Supervised Fine-Tuning, was introduced to address this problem.

People collect well-written questions and answers into a dataset:

$$\mathcal{D}_{\mathrm{SFT}} = \left\{ (x_i,y_i) \right\}_{i=1}^{N}$$

Here $x_i$ is a question and $y_i$ is an ideal answer written by a person. Training increases the probability that the model generates that answer.

$$\mathcal{L}_{\mathrm{SFT}}(\theta) = - \mathbb{E}_{(x,y)\sim\mathcal{D}_{\mathrm{SFT}}} \left[ \log \pi_\theta(y|x) \right].$$

This method is intuitive and works well. Instruction tuning produces a model that follows user directions far better than a base model.

It is nevertheless difficult for people to write a perfect answer to every question.

Compare the following two requests:

1. “Write the perfect explanation of PPO for someone learning reinforcement learning for the first time.”
2. “Choose which of these two answers is better.”

For most people, the second is easier. Writing a perfect answer from scratch requires very different effort from selecting the better of two existing answers.

You do not need to know how to run a restaurant to judge whether its food tastes good. Online reviews sometimes make even that proposition doubtful, but relative comparisons are often easier than writing demonstrations directly.

Training an LLM from such comparisons is called preference learning, and RLHF—Reinforcement Learning from Human Feedback—is a representative method.

RLHF

The name RLHF may evoke a person sitting beside the model, praising or scolding it after every generated response.

In practice, people are not placed inside every iteration of the reinforcement-learning loop. They would become exhausted and, more importantly, they are expensive.

Conventional RLHF has people compare answers in advance to construct a preference dataset, then trains a Reward Model to stand in for human judgment. The Reward Model subsequently evaluates responses generated by the policy.

A standard RLHF pipeline such as InstructGPT consists of three broad stages:

1. SFT on a dataset of good answers
2. Reward Model training on human preferences
3. Policy optimization to maximize Reward Model scores

First, provide the same question $x$ to the SFT model several times and generate different answers:

$$y_1,y_2 \sim \pi_{\mathrm{SFT}}(y|x).$$

Show two responses to a person and ask which is better. Denote the selected response by $y_w$ and the unselected one by $y_l$:

$$y_w \succ y_l.$$

$w$ and $l$ stand roughly for winner and loser. Papers also call them the preferred and dispreferred responses.

The resulting preference dataset has the form

$$\mathcal{D} = \left\{ (x^{(i)},y_w^{(i)},y_l^{(i)}) \right\}_{i=1}^{N}.$$

Importantly, the person did not assign exact scores to the answers.

They did not say, “The first answer is worth 8.37 points and the second 6.24.” They said only that the first was better than the second.

Reinforcement learning, however, needs a numerical evaluation of the model’s response. A Reward Model is therefore trained to turn this relative ranking into a scalar reward.

Reward Model

Let the Reward Model be $r_\phi(x,y)$. Given question $x$ and response $y$, it emits one scalar:

$$r_\phi(x,y) \in \mathbb{R}.$$

A high reward is assumed to indicate a response that people are likely to prefer, and a low reward one they are unlikely to prefer.

The Bradley-Terry Model is commonly used for these preferences.

Given two responses $y_1$ and $y_2$, it defines the probability that a person prefers $y_1$ as

$$P(y_1 \succ y_2|x) = \frac{ \exp(r_\phi(x,y_1)) }{ \exp(r_\phi(x,y_1)) + \exp(r_\phi(x,y_2)) }.$$

Using a sigmoid gives the simpler form

$$P(y_1 \succ y_2|x) = \sigma \left( r_\phi(x,y_1) - r_\phi(x,y_2) \right).$$

What matters is not each reward’s absolute value, but the difference between the rewards.

For example, consider

$$r_\phi(x,y_w)=10, \qquad r_\phi(x,y_l)=8$$

and

$$r_\phi(x,y_w)=2, \qquad r_\phi(x,y_l)=0.$$

Both reward differences are 2, so the Bradley-Terry Model assigns them the same preference probability.

The Reward Model should not blindly make the selected answer’s score large. It should make the selected answer’s reward higher than that of the unselected answer.

Its loss is therefore

$$\mathcal{L}_{\mathrm{RM}}(\phi) = - \mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \left[ \log \sigma \left( r_\phi(x,y_w) - r_\phi(x,y_l) \right) \right].$$

The loss decreases when the Reward Model gives the selected response a higher score and increases when it scores the unselected response higher.

So far, this does not look very different from binary classification.

The problem is that training a good Reward Model is not the end. We do not ultimately want a judge that scores responses well; we want an LLM that generates good responses.

The Reward Model only imitates human preferences. The actual policy must now be trained to increase that reward.

A Policy That Maximizes Reward

Let the language model we want to train be $\pi_\theta$.

Given a question $x$, the Policy generates a response $y$.

$$y \sim \pi_\theta(y|x).$$

At its simplest, we could maximize the following objective so that the model generates responses to which the Reward Model assigns high scores.

$$\max_{\pi_\theta} \mathbb{E}_{x\sim\mathcal{D},\, y\sim\pi_\theta(y|x)} \left[ r_\phi(x,y) \right].$$

With only this objective, however, the Policy may find weaknesses in the Reward Model and generate strange responses.

The Reward Model is not human preference itself. It is an approximation trained on a limited preference dataset. If we optimize the Policy too aggressively, it may generate responses that please only the Reward Model rather than responses people actually like.

This is like discovering loopholes in an exam rubric and writing an answer that earns a high score despite saying very little. That skill may have had its uses in school, but it is not what we want from an LLM.

We therefore add a KL divergence penalty so that the Policy being trained does not move too far from the existing Reference model.

$$\max_{\pi_\theta} \mathbb{E}_{x\sim\mathcal{D},\, y\sim\pi_\theta(y|x)} \left[ r_\phi(x,y) \right] - \beta D_{\mathrm{KL}} \left( \pi_\theta(y|x) \| \pi_{\mathrm{ref}}(y|x) \right).$$

Here $\pi_{\mathrm{ref}}$ is generally a frozen copy of the SFT model used as the Reference model.

The first term raises the Reward.

$$\mathbb{E} \left[ r_\phi(x,y) \right].$$

The second term prevents the Policy from moving too far from the Reference model.

$$\beta D_{\mathrm{KL}} \left( \pi_\theta \| \pi_{\mathrm{ref}} \right).$$

A large $\beta$ binds the Policy more strongly to the Reference model, while a small $\beta$ allows the Policy to change more in pursuit of higher Reward.

The objective balances two goals:

1. Generate high-Reward responses that people are likely to prefer.
2. Do not stray too far from the language ability and response distribution of the existing SFT model.

PPO has commonly been used to optimize this objective in practice.

One point deserves emphasis here: RLHF and PPO are not the same thing.

RLHF is the overall framework for training a model with Human Feedback. PPO is the reinforcement-learning algorithm used within that framework to update the Policy with a Reward Model.

Other reinforcement-learning methods can be used instead of PPO. The two concepts have come to be mentioned almost as a pair because PPO has been used so frequently in LLM-based RLHF.

The Complexity of PPO-Based RLHF

Running PPO-based RLHF requires more models than one might expect.

• The Policy Model that is actually trained
• The Reference Model that anchors the Policy
• The Reward Model that evaluates generated responses
• The Value Model that predicts the future Reward of each State

It amounts to bringing in three more models behind the scenes just to teach one model well. Education has always been expensive.

The Policy Model generates a response. The Reward Model evaluates the complete response. At each Token-generation step, the Value Model predicts the future Reward and uses it to calculate the Advantage.

The PPO training process must also Sample new responses from the current Policy.

$$y \sim \pi_{\theta_{\mathrm{old}}}(y|x).$$

It evaluates the generated response with the Reward Model, calculates a KL penalty against the Reference Model, estimates the Advantage with the Value Model, and then updates the Policy using PPO clipping.

Repeating this process creates the following cycle:

$$\text{Response Sampling} \rightarrow \text{Reward Evaluation} \rightarrow \text{Advantage Estimation} \rightarrow \text{PPO Update}.$$

LLM responses are long and the models themselves are large. Generating the rollout is therefore expensive, and training also requires Forward passes through several models and a Backward pass through the Policy.

There are also many factors to manage, including the scales of the Reward and Value, the KL coefficient, the PPO clipping range, and Advantage normalization.

This is exactly why I wrote such a long article about PPO earlier. PPO works well, but it is by no means a simple training method.

This raises a rather fundamental question:

People have already told us which response is better. Must we really train a separate Reward Model and then train the Policy again with PPO?

The current RLHF structure is as follows:

$$\text{Human Preference} \rightarrow \text{Reward Model} \rightarrow \text{Policy}.$$

The Reward Model first learns human preferences, and the Policy then learns the Reward Model's preferences.

It is literally doing the work twice.

Could we instead use the preference dataset to train the Policy itself directly?

The method that begins with this question is Direct Preference Optimization, or DPO.

Direct Preference Optimization

Training Without a Reward Model

The full title of the DPO paper is:

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

It means that your language model is, in fact, a Reward Model. It is a fairly provocative paper title—almost like a YouTube thumbnail.

DPO's main idea is not a proposal for a new Reward.

Rather, if we rearrange the KL-constrained Reward-maximization objective used in conventional RLHF, the Preference probability can be expressed directly in terms of Policy probabilities without explicitly training a Reward Model.

Recall the conventional RLHF objective:

$$\max_{\pi} \mathbb{E}_{y\sim\pi(y|x)} \left[ r(x,y) \right] - \beta D_{\mathrm{KL}} \left( \pi(y|x) \| \pi_{\mathrm{ref}}(y|x) \right).$$

Suppose the Reward function $r(x,y)$ in this objective is given.

The Optimal policy that maximizes the objective can be expressed as

$$\pi_r(y|x) = \frac{1}{Z(x)} \pi_{\mathrm{ref}}(y|x) \exp \left( \frac{1}{\beta}r(x,y) \right).$$

Here $Z(x)$ is the Partition function that makes the probabilities of all responses sum to 1.

$$Z(x) = \sum_y \pi_{\mathrm{ref}}(y|x) \exp \left( \frac{1}{\beta}r(x,y) \right).$$

Intuitively, a response with a higher Reward receives a higher probability than it had under the Reference model.

$$r(x,y) \uparrow \quad\Rightarrow\quad \pi_r(y|x) \uparrow.$$

However, a response that the Reference model would almost never generate cannot gain an arbitrarily high probability merely because its Reward is slightly higher. The term $\pi_{\mathrm{ref}}(y|x)$ is the anchor.

Now let us rearrange the Optimal policy equation in terms of the Reward.

$$\pi_r(y|x) = \frac{1}{Z(x)} \pi_{\mathrm{ref}}(y|x) \exp \left( \frac{1}{\beta}r(x,y) \right)$$

Taking the Log of both sides gives

$$\log \pi_r(y|x) = \log \pi_{\mathrm{ref}}(y|x) + \frac{1}{\beta}r(x,y) - \log Z(x).$$

The Reward can therefore be written as

$$r(x,y) = \beta \log \frac{ \pi_r(y|x) }{ \pi_{\mathrm{ref}}(y|x) } + \beta\log Z(x).$$

This equation is DPO's most important starting point.

Instead of representing the Reward function with a separate Neural Network, we can represent it as the Log probability ratio between the Policy and the Reference policy.

$$r(x,y) = \beta \log \frac{ \pi_r(y|x) }{ \pi_{\mathrm{ref}}(y|x) } + \beta\log Z(x).$$

But $Z(x)$ still remains.

Computing $Z(x)$ exactly for an LLM is practically impossible because it requires summing probabilities over every possible response $y$. That should be clear from considering how many possible sentences there are.

Fortunately, the Bradley-Terry Model used for the Reward Model depends not on the absolute values of the Rewards but only on their difference.

$$P(y_w \succ y_l|x) = \sigma \left( r(x,y_w) - r(x,y_l) \right).$$

Substitute the Reward expression derived above.

$$\begin{aligned} r(x,y_w)-r(x,y_l) &= \beta \log \frac{ \pi_r(y_w|x) }{ \pi_{\mathrm{ref}}(y_w|x) } + \beta\log Z(x) \newline &\quad - \beta \log \frac{ \pi_r(y_l|x) }{ \pi_{\mathrm{ref}}(y_l|x) } - \beta\log Z(x). \end{aligned}$$

Because both responses answer the same question $x$, their $\beta\log Z(x)$ terms cancel.

$$\begin{aligned} r(x,y_w)-r(x,y_l) &= \beta \log \frac{ \pi_r(y_w|x) }{ \pi_{\mathrm{ref}}(y_w|x) } \newline &\quad - \beta \log \frac{ \pi_r(y_l|x) }{ \pi_{\mathrm{ref}}(y_l|x) }. \end{aligned}$$

This is precisely where the separate Reward Model disappears.

The difference in Rewards can be expressed using only the Policy and Reference probabilities.

DPO Loss

Substituting the actual Policy $\pi_\theta$ to be trained for the Optimal policy $\pi_r$ gives the following Preference probability:

$$\begin{aligned} P_\theta(y_w \succ y_l|x) = \sigma \Bigg( & \beta \log \frac{ \pi_\theta(y_w|x) }{ \pi_{\mathrm{ref}}(y_w|x) } \newline & - \beta \log \frac{ \pi_\theta(y_l|x) }{ \pi_{\mathrm{ref}}(y_l|x) } \Bigg). \end{aligned}$$

Because the preference dataset tells us that a person actually selected $y_w$, we can use a Binary cross-entropy-style Loss that increases this probability.

$$\begin{aligned} \mathcal{L}_{\mathrm{DPO}}(\theta) = - \mathbb{E}_{(x,y_w,y_l)\sim\mathcal{D}} \Bigg[ \log\sigma \Bigg( & \beta \log \frac{ \pi_\theta(y_w|x) }{ \pi_{\mathrm{ref}}(y_w|x) } \newline & - \beta \log \frac{ \pi_\theta(y_l|x) }{ \pi_{\mathrm{ref}}(y_l|x) } \Bigg) \Bigg]. \end{aligned}$$

That is almost the entirety of the DPO Loss.

The rollout, Reward Model, Value Model, GAE, and PPO clipping used in PPO have all disappeared from the equation.

We can simply load preference pairs in Batches and perform Gradient descent much like ordinary Language Model Fine-tuning.

This is why it is called Direct Preference Optimization. It optimizes the Policy directly from the preference dataset without the Reward Model as an intermediary.

What DPO Actually Compares

At first glance, the DPO Loss may look complicated because it contains so many Logs and fractions, but the value it actually compares is simple.

Define the following quantity as the Implicit reward that the Policy assigns to response $y$:

$$\hat{r}_\theta(x,y) = \beta \log \frac{ \pi_\theta(y|x) }{ \pi_{\mathrm{ref}}(y|x) }.$$

The DPO Loss can then be written as

$$\mathcal{L}_{\mathrm{DPO}}(\theta) = - \mathbb{E} \left[ \log \sigma \left( \hat{r}_\theta(x,y_w) - \hat{r}_\theta(x,y_l) \right) \right].$$

It looks almost identical to the Reward Model Loss.

In a conventional Reward Model, a separate Network $r_\phi$ predicts the Reward difference.

$$r_\phi(x,y_w) - r_\phi(x,y_l).$$

In DPO, the Log probability ratio between the Policy and Reference policy plays the role of the Reward.

$$\hat{r}_\theta(x,y_w) - \hat{r}_\theta(x,y_l).$$

In other words, the Reward has not disappeared.

It is represented by changes in probability within the Policy rather than emitted by a separate model.

This is what the paper's title, “Your Language Model is Secretly a Reward Model,” means.

We thought we had fired the Reward Model, only to discover that the Policy was now doing the Reward Model's job as well. It is a structure commonly seen in companies.

The Relative Probability Matters

It is also worth considering how DPO differs from simple SFT.

SFT only increases the probability of the Preferred response $y_w$.

$$\mathcal{L}_{\mathrm{SFT}} = - \log \pi_\theta(y_w|x).$$

It does not use information about the Rejected response $y_l$.

DPO, by contrast, learns the relative Margin between the Preferred and Rejected responses.

$$\begin{aligned} m_\theta(x,y_w,y_l) = & \log \frac{ \pi_\theta(y_w|x) }{ \pi_{\mathrm{ref}}(y_w|x) } \newline & - \log \frac{ \pi_\theta(y_l|x) }{ \pi_{\mathrm{ref}}(y_l|x) }. \end{aligned}$$

DPO trains this Margin to grow.

$$\mathcal{L}_{\mathrm{DPO}} = - \log \sigma \left( \beta m_\theta \right).$$

Put simply, it does two things together:

1. It makes the Policy prefer the Preferred response more than the Reference model does.
2. It makes the Policy prefer the Rejected response less than the Reference model does.

The absolute size of $\pi_\theta(y_w|x)$ is insufficient on its own.

DPO considers how much the relative probability of the Preferred response has increased compared with the Reference model.

For example, if the Reference model already generated a certain response with high probability, it is not enough for the Policy merely to assign that response a high probability too.

$$\log \frac{ \pi_\theta(y_w|x) }{ \pi_{\mathrm{ref}}(y_w|x) }$$

This value indicates how much more the Policy has come to prefer that response relative to the Reference model.

DPO learns not the simple Likelihood of the Preferred response, but its relative change in preference from the Reference model.

Why Does the Reference Model Remain?

The Reward Model and Value Model disappear in DPO, but the Reference Model remains.

Without the Reference Model, we could simply learn the difference in Log probabilities between the Chosen and Rejected responses:

$$\log \pi_\theta(y_w|x) - \log \pi_\theta(y_l|x).$$

Training this way, however, would remove the KL constraint from the original RLHF Objective.

The Reference probability in DPO incorporates into the Loss the KL regularization that kept the Policy from moving too far from the baseline model in conventional RLHF.

$$\log \frac{ \pi_\theta(y|x) }{ \pi_{\mathrm{ref}}(y|x) }.$$

It is therefore more accurate to view DPO not as a method that eliminates the Reference Model, but as one that redefines Reward through its relationship with the Reference Model.

The Reference Model is generally a frozen SFT model. The Policy being trained also starts from the same SFT checkpoint.

Because the two models are identical at the beginning of training, the following value is nearly 0:

$$\log \frac{ \pi_\theta(y|x) }{ \pi_{\mathrm{ref}}(y|x) } \approx 0.$$

As training progresses, the Policy changes so that this value becomes relatively larger for Preferred responses and smaller for Rejected responses.

The Role of $\beta$

$\beta$ also appears in DPO.

$$\mathcal{L}_{\mathrm{DPO}} = - \log\sigma \left( \beta \left[ \log \frac{ \pi_\theta(y_w|x) }{ \pi_{\mathrm{ref}}(y_w|x) } - \log \frac{ \pi_\theta(y_l|x) }{ \pi_{\mathrm{ref}}(y_l|x) } \right] \right).$$

This $\beta$ comes from the coefficient in the original KL-constrained RLHF Objective that controlled how far the Policy could move from the Reference Model.

$$\mathbb{E}[r] - \beta D_{\mathrm{KL}} \left( \pi_\theta \| \pi_{\mathrm{ref}} \right).$$

In the theoretical derivation, a larger $\beta$ makes it more costly to change the Policy substantially for a small increase in Reward. The Optimal policy is therefore bound more strongly to the Reference policy.

In the actual DPO Loss, however, $\beta$ also directly scales the Logit. It is therefore better to understand it as a Hyperparameter that jointly determines the scale of the Preference margin and the Regularization against the Reference policy, rather than merely saying that “a larger $\beta$ always weakens training.”

If $\beta$ is too small, the Policy may overfit the preference pairs. If it is too large, the desired difference in preferences may not be learned sufficiently.

Like any other Hyperparameter, it ultimately has to be tuned for the Dataset and Model. Eliminating reinforcement learning did not bring the happy ending of eliminating Hyperparameter tuning as well.

Intuition Behind the DPO Gradient

Examining the Gradient of the DPO Loss also shows which preference pairs receive stronger training.

First, recall the definition of Implicit reward:

$$\hat{r}_\theta(x,y) = \beta \log \frac{ \pi_\theta(y|x) }{ \pi_{\mathrm{ref}}(y|x) }.$$

The DPO Gradient has the following form:

$$\begin{aligned} \nabla_\theta \mathcal{L}_{\mathrm{DPO}} = - \beta \mathbb{E} \Big[ & \sigma \left( \hat{r}_\theta(x,y_l) - \hat{r}_\theta(x,y_w) \right) \newline & \times \left( \nabla_\theta \log\pi_\theta(y_w|x) - \nabla_\theta \log\pi_\theta(y_l|x) \right) \Big]. \end{aligned}$$

The latter term is intuitive.

$$\nabla_\theta \log\pi_\theta(y_w|x) - \nabla_\theta \log\pi_\theta(y_l|x).$$

It raises the probability of the Preferred response and lowers the probability of the Rejected response.

The preceding Sigmoid term determines the Weight according to how badly the current Policy misjudges the ordering of the two responses.

$$\sigma \left( \hat{r}_\theta(x,y_l) - \hat{r}_\theta(x,y_w) \right).$$

If the Policy already gives the Preferred response a much higher Implicit reward, this Weight becomes small.

Conversely, if it rates the Rejected response more highly, the Weight becomes large.

Thus, preference pairs that the current Policy ranks incorrectly receive stronger training than pairs it already distinguishes well.

This is slightly different from indiscriminately increasing every Chosen probability and decreasing every Rejected probability.

Differences Between RLHF and DPO

What Has Disappeared?

Conventional PPO-based RLHF required the following process:

$$\begin{aligned} & \text{Preference Dataset} \newline &\rightarrow \text{Reward Model Training} \newline &\rightarrow \text{Policy Rollout} \newline &\rightarrow \text{Reward Evaluation} \newline &\rightarrow \text{Value Estimation} \newline &\rightarrow \text{PPO Update}. \end{aligned}$$

DPO simplifies it to the following:

$$\text{Preference Dataset} \rightarrow \text{DPO Loss} \rightarrow \text{Policy Update}.$$

No separate Reward Model is required.

A Value Model is unnecessary as well, so GAE is not calculated.

There is no PPO clipping or Advantage normalization.

Nor is there an Online rollout process that continuously Samples responses from the current Policy and evaluates them with a Reward Model inside the training Loop.

Ordinary Offline training can be performed using preference pairs collected in advance.

From an implementation perspective, only the following information is needed:

• Prompt $x$
• Preferred response $y_w$
• Rejected response $y_l$
• The Policy's Log probability
• The Reference Model's Log probability

Because the Reference Model is not trained, it does not need Gradients. Its Log probabilities for the Dataset can even be computed in advance.

These differences make DPO simpler to implement than PPO-based RLHF and allow it to train more like a conventional Supervised Fine-tuning Pipeline.

What Has Not Disappeared?

The name DPO may suggest that even Human Feedback is no longer necessary, but that is not the case.

DPO still requires preference pairs of the following form:

$$(x,y_w,y_l).$$

Someone must decide which response is better.

That someone may be a person, a strong LLM Judge, or a Rule-based verifier. When AI supplies the Preference labels, the setup is closer to RLAIF.

In other words, DPO does not eliminate the Feedback itself.

It eliminates the intermediate process of training a separate Reward Model from that Feedback and then optimizing the Reward Model again through reinforcement learning.

A simple comparison looks like this.

Conventional RLHF:

$$\text{Preference} \rightarrow r_\phi(x,y) \rightarrow \pi_\theta(y|x)$$

DPO:

$$\text{Preference} \rightarrow \pi_\theta(y|x)$$

Human preferences have not suddenly become free; the path that conveys them to the Policy has merely become shorter.

Is DPO Reinforcement Learning?

DPO is derived from a reinforcement-learning Objective, but its actual training process does not use a Reinforcement Learning algorithm in the conventional sense.

It does not interact with an environment to collect new Trajectories, nor does it calculate a Policy Gradient from a Scalar reward emitted by a Reward Model.

Instead, it minimizes a Binary classification-style Loss on a fixed preference dataset.

The DPO paper therefore describes DPO as an RL-free algorithm.

That does not mean DPO suddenly invented a completely unrelated objective.

It analyzes the Optimal policy of the following Objective used in conventional RLHF,

$$\max_\pi \mathbb{E}_{y\sim\pi} [r(x,y)] - \beta D_{\mathrm{KL}} (\pi\|\pi_{\mathrm{ref}})$$

expresses the relationship between the Reward and the Policy in Closed form, and then substitutes the Preference Loss directly into the Policy.

The following two statements are therefore both true:

1. DPO training itself does not use a reinforcement-learning Loop like PPO.
2. The DPO Loss is derived from the KL-regularized RLHF Objective and the Bradley-Terry Preference Model.

I personally find it helpful to think of DPO as “a way to solve a reinforcement-learning Objective without doing reinforcement learning.” The phrasing is a little odd, but that is how its structure actually works.

Advantages and Limitations of DPO

Training Has Become Simpler

DPO's greatest advantage is, of course, its simplicity.

PPO-based RLHF requires managing the interactions among the Policy, Reference, Reward, and Value Models. Response Sampling and Reward calculation also take place inside the training Loop.

DPO, by contrast, only needs to calculate the Log probabilities of preference pairs and minimize a single Loss.

Anyone with experience in ordinary Language Model Fine-tuning can understand a DPO implementation relatively easily.

There is also no need to train and store separate Reward and Value Models, reducing the management burden of the overall Pipeline.

At this point, DPO may look like a strictly superior method that completely replaces RLHF, but that is not always the case.

Bound to an Offline Preference Dataset

Basic DPO uses a fixed preference dataset collected in advance.

Even as the Policy changes during training, it does not automatically receive new Feedback on responses generated by the new Policy.

Suppose, for example, that we construct a preference dataset from responses generated by the initial SFT model.

As DPO training proceeds, the Policy's response distribution changes. Yet the Dataset still contains only responses generated by the initial Model.

There is no direct Preference signal for strange responses that the Policy begins to generate during training, or for good responses that were not in the Dataset.

Online RLHF, by contrast, can continually evaluate responses from the current Policy with the Reward Model.

This requires assuming that the Reward Model generalizes well, but it can at least evaluate new Samples from the current Policy's Output distribution.

Put simply:

• DPO repeatedly studies answer sheets that have already been graded.
• Online RLHF has the Reward Model grade each new problem as it is answered.

If the existing answer sheets are good enough, DPO is much more efficient. But an Online method may be advantageous when the Policy moves far beyond the Dataset or needs to explore new behaviors.

Sensitive to the Quality of Preference Labels

DPO directly incorporates the difference between Preferred and Rejected responses into the Policy.

Consequently, it learns the stated relationship even when a Preference label is wrong, or when the two responses are nearly equal in quality but are forcibly assigned a Winner and Loser.

Human preferences are not always consistent.

Some people like short, direct answers, while others prefer long, friendly ones. Even the same person's judgment may change with their condition that day.

The Bradley-Terry Model assumes that each response has a latent Scalar reward and that the difference between two Rewards determines the Preference probability.

$$P(y_w \succ y_l|x) = \sigma \left( r(x,y_w)-r(x,y_l) \right).$$

In reality, however, it is difficult to believe that human preferences can always be neatly ordered by a single Scalar value.

A may be better than B and B better than C, yet in a particular context C may once again be better than A. Human feelings are not that consistent to begin with.

DPO inherits both this assumption about the Preference model and the quality of the Dataset.

Eliminating the Reward Model does not eliminate the difficulty of Preference modeling.

The Reward Model Cannot Be Reused

In conventional RLHF, the Reward Model is a separate evaluation model.

Once trained, it can be used to compare multiple Policies, perform Rejection sampling, filter a Dataset, or evaluate new Responses.

In DPO, by contrast, the Reward is implicitly contained in the Log probability ratio between the Policy and Reference Model.

$$\hat{r}_\theta(x,y) = \beta \log \frac{ \pi_\theta(y|x) }{ \pi_{\mathrm{ref}}(y|x) }.$$

This Implicit reward depends on the relationship between the current Policy and Reference.

It is difficult to use it to score the Outputs of other Policies independently, as one could with a general, separately trained Reward Model.

DPO is therefore convenient when the final goal is to Fine-tune a single Policy to a preference dataset. When the Reward Model itself must be used across several processes, the conventional RLHF structure may be more appropriate.

Directly Optimizing Preferences Is Not Always Safe

DPO increases the relative Margin between the Preferred and Rejected responses.

But an increase in that relative Margin does not necessarily mean that the absolute Likelihood of the Preferred response always rises.

For example, even if the Probability of the Preferred response decreases slightly, the relative Margin between the two responses can grow if the Probability of the Rejected response decreases much more.

$$\begin{aligned} & \left[ \log\pi_\theta(y_w|x) - \log\pi_\theta(y_l|x) \right] \uparrow \newline & \not\Rightarrow \pi_\theta(y_w|x) \text{ always increases}. \end{aligned}$$

DPO's actual objective is not to memorize the Chosen response unconditionally, but to shift the preference difference between Chosen and Rejected in the correct direction relative to the Reference model.

This property is also an advantage of DPO, but it means that an unsuitable Dataset or Hyperparameter can produce unexpected Probability changes.

Simplifying the Loss leaves the work of Dataset analysis and training-result validation in place.

Summary

The basic idea of RLHF is to turn human preferences into Reward and use reinforcement learning to train a Policy that generates responses with high Reward.

Conventional PPO-based RLHF generally proceeds in the following order:

$$\begin{aligned} & \text{SFT} \newline &\rightarrow \text{Preference Dataset} \newline &\rightarrow \text{Reward Model} \newline &\rightarrow \text{PPO} \newline &\rightarrow \text{Aligned Policy}. \end{aligned}$$

DPO uses the relationship between the Reward Model and the Optimal policy to reduce this process to

$$\text{Preference Dataset} \rightarrow \text{DPO} \rightarrow \text{Aligned Policy}.$$

In DPO, Reward does not disappear; it is represented implicitly inside the Policy by the following Log probability ratio:

$$\hat{r}_\theta(x,y) = \beta \log \frac{ \pi_\theta(y|x) }{ \pi_{\mathrm{ref}}(y|x) }.$$

Training then makes the selected response's Implicit reward higher than that of the unselected response.

$$\hat{r}_\theta(x,y_w) > \hat{r}_\theta(x,y_l).$$

What DPO changes, in the end, is not whether Human Feedback exists.

It replaces the indirect process of first teaching human preferences to a Reward Model and then conveying them to the Policy through PPO with a direct connection from preference pairs to the Policy.

My simplest summary of DPO would be this:

Instead of translating human tastes into scores and then teaching them back to the model, show the model directly which response people preferred.

Showing the model directly does not mean that it perfectly understands the human mind, of course. Humans do not understand the human mind very well either.

At the very least, however, it substantially reduces the complicated process of bringing in additional Reward and Value Models to train a single model, continually generating responses, and running PPO.

DPO's greatest significance is not that it discovered a new Preference signal.

It lies in reconsidering the problem that conventional RLHF was solving and demonstrating mathematically that the Reward Model and Policy need not be trained as separate stages.