Preliminary

Reflections on Reinforcement Learning

We want a robot to perform some task.

Since we are discussing DeepSeek—an LLM—the task in this case is roughly “making the model answer well.” A Korean proverb says to give one more rice cake to the person you dislike, but the world of AI is not so forgiving.

Instead, we will give an extra rice cake to the one that behaves well. This is the concept of a reward. We run an adequately trained model in many question-answering environments and repeatedly praise it when it answers well, with the goal of creating a model that answers even better.

The Evolution of Policy-Based Methods

A value-based method, one of the traditional approaches to reinforcement learning, trains a model to evaluate the value of each action an AI agent can take. The agent estimates the value of taking a particular action in the current state and chooses its next action accordingly.

The rule that determines which action to take in each situation is the policy. A representative method that arose from the idea of learning an action-value function with deep learning is the Deep Q-Network (DQN).

A Q-network estimates the expected cumulative reward of the actions available in each state. Rather than selecting an action from its immediate reward alone, it learns an action-value function $Q$ that includes the future value of the current action, thereby mitigating some instability in rewards.

On reflection, however, the ultimate purpose of training a value estimator is not “to assess the value of every action accurately” for its own sake, but “to choose the action with the highest value.”

This raises a question: rather than evaluating every possible action one by one, would it not be better to learn the policy itself so that it directly predicts a desirable next action in the current state?

In that case, the action space need not even be discrete. We can sample the next action from the probability distribution output by the policy model and update that model with a reward when the resulting sequence of actions produces a good outcome. This learning method is the policy gradient.

Reinforcement learning assumes simulations in which an agent interacts with an environment. The process of repeatedly acting from an initial state until reaching a terminal state is called an episode. Within an episode, states, actions, rewards, and other values occur sequentially over time.

The goal of reinforcement learning is ultimately to collect diverse decision-making processes from many episodes and maximize the expected cumulative reward obtained when the model makes decisions in the most desirable way.

A policy can be constructed from predetermined rules, but in deep reinforcement learning it is generally parameterized, for example by a neural network. We represent it as a function $\pi_\theta(\cdot)$ with parameters $\theta$, which outputs a probability distribution over the actions $a$ available in each state $s$.

We therefore want to train a policy model so that its actions maximize expected reward. The direction in which the parameters should change to improve this objective is the policy gradient.

Computing a policy's gradient directly is difficult. Expected reward depends on the actions selected by $\pi_\theta$ and on the state-visitation distribution induced by that policy.

When the policy changes, the action distribution changes; when the action distribution changes, so does the distribution of states visited afterward. The objective therefore includes both the policy's direct effect and its indirect effect through the state distribution.

The Policy Gradient Theorem shows that the policy gradient can be obtained without explicitly differentiating this complicated state distribution. In one sentence:

$$\text{The gradient of the reward objective can be computed from the gradient of the policy.}$$

Let us briefly examine the formal derivation.

$$\nabla_{\theta} J(\theta) = \nabla_{\theta} \sum_{s \in \mathcal{S}} d^{\pi}(s) \sum_{a \in \mathcal{A}} Q^{\pi}(s,a) \pi_{\theta}(a|s)$$

According to the Policy Gradient Theorem, the gradient above can be written in the following form.

$$\nabla_{\theta}J(\theta) \propto \sum_{s \in \mathcal{S}} d^{\pi}(s) \sum_{a \in \mathcal{A}} Q^{\pi}(s,a) \nabla_{\theta}\pi_{\theta}(a|s).$$

The reward objective is expressed in terms of the probability of visiting each state and the expected value of actions taken in that state. To see why, expand the gradient of the state-value function as follows.

$$\begin{aligned} \nabla_{\theta}V^{\pi}(s) &= \nabla_{\theta} \left( \sum_{a \in \mathcal{A}} \pi_{\theta}(a|s)Q^{\pi}(s,a) \right) \newline &= \sum_{a \in \mathcal{A}} \left( \nabla_{\theta}\pi_{\theta}(a|s)Q^{\pi}(s,a) + \pi_{\theta}(a|s)\nabla_{\theta}Q^{\pi}(s,a) \right) \newline &= \sum_{a \in \mathcal{A}} \left( \nabla_{\theta}\pi_{\theta}(a|s)Q^{\pi}(s,a) + \pi_{\theta}(a|s) \nabla_{\theta} \sum_{s',r} P(s',r|s,a) \left(r+\gamma V^{\pi}(s')\right) \right) \newline &= \sum_{a \in \mathcal{A}} \left( \nabla_{\theta}\pi_{\theta}(a|s)Q^{\pi}(s,a) + \pi_{\theta}(a|s) \sum_{s'} P(s'|s,a) \gamma\nabla_{\theta}V^{\pi}(s') \right). \end{aligned}$$

The gradient of the full expression is distributed according to the product rule. Here, $Q^\pi$ can be expressed as the sum of the current reward and a future state value.

Here, $P(s'|s,a)$ is the environment's state-transition probability. Assuming that this transition probability does not directly depend on the policy parameters $\theta$, the gradient applies only to the value function of the next state.

The key point is that the gradient of the next state's value function appears recursively inside the gradient of the current state's value function.

Expanding the embedded $\nabla_\theta V^\pi(s')$ introduces $\nabla_\theta V^\pi(s'')$. This process can be unrolled continually into future states.

In the expression below, $\rho^\pi(s \rightarrow s', k)$ denotes the probability of reaching state $s'$ after $k$ steps under policy $\pi$.

$$\phi(s) = \sum_{a \in \mathcal{A}} \nabla_\theta \pi_\theta(a|s) Q^\pi(s,a)$$

Defining this quantity as $\phi(s)$ lets us expand the expression as follows.

$$\begin{aligned} \nabla_{\theta}V^{\pi}(s) &= \phi(s) + \sum_{s'} \rho^{\pi}(s\rightarrow s',1) \nabla_{\theta}V^{\pi}(s') \newline &= \phi(s) + \sum_{s'} \rho^{\pi}(s\rightarrow s',1) \phi(s') + \sum_{s''} \rho^{\pi}(s\rightarrow s'',2) \nabla_{\theta}V^{\pi}(s'') \newline &= \phi(s) + \sum_{s'} \rho^{\pi}(s\rightarrow s',1) \phi(s') + \sum_{s''} \rho^{\pi}(s\rightarrow s'',2) \phi(s'') +\cdots \newline &= \sum_{x \in \mathcal{S}} \sum_{k=0}^{\infty} \rho^{\pi}(s\rightarrow x,k) \phi(x). \end{aligned}$$

This unrolls the paths from the current state to all future states. Substituting the initial state $s_0$ gives the following.

$$\begin{aligned} \nabla_{\theta}J(\theta) &= \nabla_{\theta}V^{\pi}(s_0) \newline &= \sum_s \sum_{k=0}^{\infty} \rho^{\pi}(s_0\rightarrow s,k) \phi(s) \newline &= \sum_s \eta(s)\phi(s) \newline &= \left(\sum_s\eta(s)\right) \sum_s \frac{\eta(s)} {\sum_s\eta(s)} \phi(s) \newline &\propto \sum_s d^{\pi}(s) \sum_a \nabla_{\theta}\pi_{\theta}(a|s) Q^{\pi}(s,a). \end{aligned}$$

This shows that the gradient of the policy function can be used to compute a direction that increases expected cumulative reward.

In practice, the log-derivative trick writes this in a more familiar form.

$$\nabla_\theta\pi_\theta(a|s) = \pi_\theta(a|s) \nabla_\theta\log\pi_\theta(a|s).$$

The policy gradient is therefore

$$\nabla_\theta J(\theta) = \mathbb{E}_{s,a\sim\pi_\theta} \left[ \nabla_\theta \log\pi_\theta(a|s) Q^\pi(s,a) \right].$$

In other words, the policy is trained to increase the log probability of actions that achieved high value and decrease the probability of actions that achieved low value.

TRPO to PPO

The discussion so far covered the basics of policy gradients. Before they could be applied effectively to LLM reinforcement learning, however, many approaches were proposed to improve training stability.

TRPO, PPO, GAE, and GRPO, which we examine next, belong to this line of development.

Of TRPO and PPO, PPO is now more widely used. To understand the problem PPO was designed to solve, however, we first need to examine its foundation, TRPO.

TRPO includes some mathematically challenging material. At its simplest, it applies the following principle when training a policy model.

Improve the policy, but update it only within a trustworthy region that does not move too far from the old policy.

Why is such a trust region important?

A policy-gradient method collects trajectories from the current policy and updates the policy according to the reward produced by each action.

This resembles the fact that we do not always begin walking or running from exactly the same posture. At every moment, the agent chooses its next action according to its current state, then reaches a new state as a result of that action.

When a robot acts, we quantify the value and reward produced by that action and update the policy to increase the probability of actions that led to good outcomes. In theory, if this process repeats stably, cumulative reward under the policy should continue to rise.

A policy gradient, however, tells us only the local direction that increases reward at the current point. It does not directly tell us how far to move in that direction.

Even if the gradient points in the right direction, an excessive update magnitude—a step size that is too large—can drastically change the policy's action-probability distribution.

This creates a large mismatch between the advantages computed from trajectories collected under the old policy and the state distribution actually visited by the updated policy.

This leaves two problems.

1. How far can we update the policy while learning stably and preserving useful behavior from the old policy?
2. How accurately can data collected under the old policy predict the performance of the new policy?

Let the old policy be $\pi$ and the new policy be $\tilde{\pi}$. The new policy's actual performance can be written as follows.

$$\eta(\tilde{\pi}) = \eta(\pi) + \sum_s d^{\tilde{\pi}}(s) \sum_a \tilde{\pi}(a|s) A^\pi(s,a).$$

Here, $d^{\tilde{\pi}}(s)$ is the probability that the new policy visits state $s$, and $A^\pi(s,a)$ measures how much better action $a$ is than an average action under the old policy.

The problem is that the new policy's state-visitation distribution $d^{\tilde{\pi}}(s)$ is difficult to compute exactly after every update.

We therefore use a surrogate objective that replaces it with the old policy's distribution $d^\pi(s)$.

$$L_\pi(\tilde{\pi}) = \eta(\pi) + \sum_s d^\pi(s) \sum_a \tilde{\pi}(a|s) A^\pi(s,a).$$

At the point where the old and new policies coincide, the true objective and surrogate objective have the same value and gradient. If the new policy changes only slightly from the old one, we can therefore expect increasing $L_\pi(\tilde{\pi})$ to increase actual performance $\eta(\tilde{\pi})$ as well.

If the policy is updated too aggressively, however, the state-visitation distribution also changes substantially and this approximation becomes inaccurate.

TRPO addresses this problem with a KL-divergence constraint that prevents the new policy from moving more than a specified distance from the old one.

$$\begin{aligned} \max_\theta \quad & \mathbb{E}_{s,a\sim\pi_{\theta_{\text{old}}}} \left[ \frac{ \pi_\theta(a|s) }{ \pi_{\theta_{\text{old}}}(a|s) } A^{\pi_{\theta_{\text{old}}}}(s,a) \right] \newline \text{subject to}\quad & \mathbb{E}_{s\sim\pi_{\theta_{\text{old}}}} \left[ D_{\mathrm{KL}} \left( \pi_{\theta_{\text{old}}}(\cdot|s) \| \pi_\theta(\cdot|s) \right) \right] \leq \delta. \end{aligned}$$

The objective trains the new policy to increase the probability of high-advantage actions. At the same time, constraining KL divergence to remain below $\delta$ prevents the policy from changing too much in a single update.

Strictly speaking, KL divergence is not a mathematical distance metric because it is not symmetric. It can nevertheless measure the discrepancy between two probability distributions and is therefore used to define a trust region.

TRPO's greatest difficulty is that this constrained optimization cannot be solved easily with ordinary gradient descent alone.

TRPO approximates the objective with a first-order Taylor expansion and the KL-divergence constraint with a second-order Taylor expansion.

$$L(\theta+\Delta\theta) \approx L(\theta) + g^\top\Delta\theta$$ $$D_{\mathrm{KL}} (\theta,\theta+\Delta\theta) \approx \frac{1}{2} \Delta\theta^\top F \Delta\theta.$$

Here, $g$ is the gradient of the surrogate objective, and $F$ is the Fisher Information Matrix corresponding to the Hessian of the KL divergence.

The optimization problem then takes approximately the following form.

$$\begin{aligned} \max_{\Delta\theta}\quad &g^\top\Delta\theta \newline \text{subject to}\quad & \frac{1}{2} \Delta\theta^\top F\Delta\theta \leq\delta. \end{aligned}$$

Its solution has the following form, associated with the natural-gradient direction.

$$\Delta\theta \propto F^{-1}g.$$

For a large neural network, however, computing the full matrix $F$ and inverting it is impractical.

TRPO approximates $F^{-1}g$ using Fisher-vector products and conjugate gradient. It also applies a line search to verify that the final update satisfies the actual KL constraint.

Thus, although TRPO does not directly compute the full Hessian, it requires a complex procedure involving

• a second-order approximation of KL divergence,
• the Fisher Information Matrix,
• Fisher-vector products,
• conjugate gradient, and
• backtracking line search.

This method substantially improved the stability of policy-gradient training, but its implementation and computation were complicated.

PPO proposes a way to reduce TRPO's complexity while still preventing the policy from changing excessively in one update.

Its key idea is to use the ratio between the action probabilities of the old and new policies.

$$r_t(\theta) = \frac{ \pi_\theta(a_t|s_t) }{ \pi_{\theta_{\text{old}}}(a_t|s_t) }.$$

This ratio measures how much the new policy has changed the probability of a particular action relative to the old policy.

• $r_t(\theta)>1$: the new policy increased the action's probability.
• $r_t(\theta)<1$: the new policy decreased the action's probability.
• $r_t(\theta)=1$: the old and new policies assign the same probability.

The standard importance-sampling surrogate objective is

$$L^{\mathrm{CPI}}(\theta) = \mathbb{E}_t \left[ r_t(\theta)A_t \right].$$

Actions with positive advantage become more probable, while those with negative advantage become less probable.

If $r_t(\theta)$ becomes too large or too small, however, the policy can change dramatically in a single step. PPO prevents this by clipping the ratio between $1-\epsilon$ and $1+\epsilon$.

$$L^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta)A_t, \operatorname{clip} \left( r_t(\theta), 1-\epsilon, 1+\epsilon \right) A_t \right) \right].$$

For a positive advantage, increasing the action's probability is desirable. Once the ratio exceeds $1+\epsilon$, however, PPO gives no further benefit for increasing it.

Conversely, a negative advantage calls for lowering the action's probability, but PPO provides no additional benefit once the ratio falls below $1-\epsilon$.

In short, it suppresses excessive updates by limiting objective improvement once the policy change leaves a specified range.

PPO's clipping does not mathematically guarantee the same KL trust region as TRPO. But it can suppress abrupt policy changes using ordinary first-order gradient optimization, without second derivatives, a Fisher Information Matrix, conjugate gradient, or line search.

PPO can therefore be understood as a simpler approximation to TRPO's objective:

Increase the probability of high-advantage actions while preventing the new policy from moving too far from the old policy.

Advantage Function

The TRPO and PPO equations above used the advantage function $A^\pi(s,a)$ instead of $Q^\pi(s,a)$.

The action-value function is the expected cumulative reward obtained by selecting a particular action in a particular state.

$$Q^\pi(s,a) = \mathbb{E}_\pi \left[ \sum_{k=0}^{\infty} \gamma^k r_{t+k} \mid s_t=s, a_t=a \right].$$

The state-value function is the average expected cumulative reward from following policy $\pi$ in the current state.

$$V^\pi(s) = \mathbb{E}_{a\sim\pi(\cdot|s)} \left[ Q^\pi(s,a) \right].$$

The advantage function measures how much the value of a particular action exceeds the average value in that state.

$$A^\pi(s,a) = Q^\pi(s,a)-V^\pi(s).$$

For example, even if $Q^\pi(s,a)$ is large, the action is not especially good if most actions in that state also have high value.

Conversely, even when its absolute reward is modest, an action has positive advantage if it is better than the alternatives.

Another reason to use advantage is that it can reduce the variance of the policy gradient.

The policy gradient has the following form.

$$\nabla_\theta J(\theta) = \mathbb{E} \left[ \nabla_\theta \log\pi_\theta(a|s) Q^\pi(s,a) \right].$$

Subtracting a baseline $b(s)$ that depends only on the state and not on the action does not change the gradient's expectation.

$$\mathbb{E}_{a\sim\pi_\theta} \left[ \nabla_\theta \log\pi_\theta(a|s) b(s) \right] = 0.$$

Thus, using $b(s)=V^\pi(s)$ gives

$$\nabla_\theta J(\theta) = \mathbb{E} \left[ \nabla_\theta \log\pi_\theta(a|s) A^\pi(s,a) \right].$$

Using the value function as a baseline preserves the expected policy gradient while reducing its variance.

In a real environment, however, $Q^\pi$, $V^\pi$, and $A^\pi$ are unknown and must be estimated from data.

Generalized Advantage Estimation (GAE) is a method for performing this estimation.

Generalized Advantage Estimation

PPO is usually implemented with an actor-critic architecture that uses both a policy model and a value model.

The policy model acts as the actor that selects actions, while the value model acts as the critic that estimates expected cumulative reward from the current state.

The value function estimated by the critic can be used to compute advantage. The simplest one-step temporal-difference residual is

$$\delta_t^V = r_t + \gamma V(s_{t+1}) - V(s_t).$$

This measures how much the sum of the current reward and the estimated value of the next state exceeds the estimated value of the current state.

The corresponding one-step advantage estimator is

$$\hat{A}_t^{(1)} = r_t + \gamma V(s_{t+1}) - V(s_t) = \delta_t^V.$$

Using rewards from farther into the future gives a $k$-step advantage estimator.

$$\begin{aligned} \hat{A}_t^{(1)} &= r_t + \gamma V(s_{t+1}) - V(s_t) \newline \hat{A}_t^{(2)} &= r_t + \gamma r_{t+1} + \gamma^2 V(s_{t+2}) - V(s_t) \newline \hat{A}_t^{(3)} &= r_t + \gamma r_{t+1} + \gamma^2r_{t+2} + \gamma^3V(s_{t+3}) - V(s_t) \newline &\quad\vdots \newline \hat{A}_t^{(\infty)} &= r_t + \gamma r_{t+1} + \gamma^2r_{t+2} +\cdots - V(s_t). \end{aligned}$$

This involves a bias-variance trade-off.

An estimator with small $k$ relies more heavily on the value model's prediction of future reward. An inaccurate value model can introduce bias, but variance is relatively low because fewer observed rewards are summed.

An estimator with large $k$ uses more observed rewards and therefore depends less on bias in the value model. Its variance is higher, however, because it includes many future rewards and sampled actions.

The difference between the two cases is simple:

• small $k$: high bias, low variance
• large $k$: low bias, high variance

Rather than selecting a single $k$-step estimator, Generalized Advantage Estimation takes a weighted average of advantage estimators with several horizons.

$$\begin{aligned} \hat{A}_t^{\mathrm{GAE}(\gamma,\lambda)} &= (1-\lambda) \left( \hat{A}_t^{(1)} + \lambda\hat{A}_t^{(2)} + \lambda^2\hat{A}_t^{(3)} +\cdots \right). \end{aligned}$$

Here, $\lambda$ controls the balance between short and long horizons.

Each $k$-step advantage can be expressed in terms of TD residuals as follows.

$$\hat{A}_t^{(k)} = \sum_{l=0}^{k-1} \gamma^l \delta_{t+l}^V.$$

Substituting this into the GAE equation gives

$$\begin{aligned} \hat{A}_t^{\mathrm{GAE}(\gamma,\lambda)} &= (1-\lambda) \left( \hat{A}_t^{(1)} + \lambda\hat{A}_t^{(2)} + \lambda^2\hat{A}_t^{(3)} +\cdots \right) \newline &= (1-\lambda) \Big( \delta_t^V + \lambda ( \delta_t^V + \gamma\delta_{t+1}^V ) \newline &\qquad + \lambda^2 ( \delta_t^V + \gamma\delta_{t+1}^V + \gamma^2\delta_{t+2}^V ) +\cdots \Big) \newline &= (1-\lambda) \Big( \delta_t^V (1+\lambda+\lambda^2+\cdots) \newline &\qquad + \gamma\delta_{t+1}^V (\lambda+\lambda^2+\cdots) +\cdots \Big) \newline &= \delta_t^V + \gamma\lambda\delta_{t+1}^V + (\gamma\lambda)^2\delta_{t+2}^V +\cdots \newline &= \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}^V. \end{aligned}$$

GAE therefore sums future TD residuals with weights that decay exponentially according to $\gamma\lambda$.

Its behavior depends on $\lambda$:

• $\lambda=0$: equivalent to the one-step TD estimator; variance is low, but it is more sensitive to value-model error.
• $\lambda\rightarrow1$: approaches the Monte Carlo return; bias decreases, but variance increases.
• $0<\lambda<1$: balances bias and variance.

PPO usually computes advantage with GAE and then uses it in the clipped objective.

$$L^{\mathrm{CLIP}}(\theta) = \mathbb{E}_t \left[ \min \left( r_t(\theta) \hat{A}_t^{\mathrm{GAE}}, \operatorname{clip} \left( r_t(\theta), 1-\epsilon, 1+\epsilon \right) \hat{A}_t^{\mathrm{GAE}} \right) \right].$$

Where PPO controls the magnitude of policy updates through clipping, GAE controls the bias and variance of the advantage estimates used in those updates.

The two methods solve different problems.

• PPO clipping: stability of policy updates
• GAE: stability of advantage estimation

Applying PPO to LLMs

When PPO is applied to an LLM, the policy is a language model that outputs a probability distribution over the next token.

Given a question $q$, the policy model generates an output $o$ consisting of multiple tokens.

$$o=(o_1,o_2,\ldots,o_{|o|}).$$

At each time $t$, the state consists of the question and all tokens generated so far.

$$s_t=(q,o_{<t}).$$

The action is the next token $o_t$ to generate.

$$a_t=o_t.$$

The PPO objective for an LLM can be written as follows.

$$\begin{aligned} \mathcal{J}_{\mathrm{PPO}}(\theta) = \mathbb{E} \Bigg[ & q\sim P(Q), \quad o\sim\pi_{\theta_{\mathrm{old}}}(O|q) \newline & \frac{1}{|o|} \sum_{t=1}^{|o|} \min \left( r_t(\theta)A_t, \operatorname{clip} \left( r_t(\theta), 1-\epsilon, 1+\epsilon \right) A_t \right) \Bigg]. \end{aligned}$$

The token-level policy ratio is

$$r_t(\theta) = \frac{ \pi_\theta(o_t|q,o_{<t}) }{ \pi_{\theta_{\mathrm{old}}}(o_t|q,o_{<t}) }.$$

The parameters before the update are $\theta_{\mathrm{old}}$, and the parameters being updated are $\theta$. The question $q$ is sampled from the dataset, and $o$ is the answer generated by the old policy.

Training an LLM with RLHF generally requires the following models.

1. Policy model
2. Value model
3. Reward model
4. Reference model

The policy model is the language model actually being trained. The reward model takes the question and complete answer as input and scores the answer's preference or quality.

The value model estimates expected future reward at each generation step and is used to compute GAE.

The reference model is generally a frozen copy of the initial SFT model. A KL penalty between the policy and reference model prevents the policy from exploiting the reward model or drifting too far from its existing language abilities.

Conceptually, token-level reward in LLM RLHF can be constructed as follows.

$$r_t = r_{\varphi,t} - \beta \log \frac{ \pi_\theta(o_t|q,o_{<t}) }{ \pi_{\mathrm{ref}}(o_t|q,o_{<t}) }.$$

Typically, the final scalar reward output by the reward model is assigned to the answer's last token, while a KL penalty is applied to intermediate tokens.

The KL penalty discourages the policy from moving too far from the reference model's token distribution.

PPO-based LLM training requires substantial memory because forward passes run through the policy model as well as the value, reference, and reward models.

In particular, maintaining a separate value model comparable in size to the policy is a major burden in large-scale LLM training.

This is where the GRPO idea proposed by DeepSeekMath enters.

GRPO

In an earlier post explaining how ChatGPT was trained, I briefly discussed RLHF and PPO.

DeepSeek used its own method, GRPO, instead of PPO to train some of its reasoning models. GRPO is a reinforcement-learning method proposed in “DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models”.

GRPO stands for Group Relative Policy Optimization. As the name suggests, it generates several answers to one question and trains the policy by comparing their rewards relative to the group.

Like PPO, GRPO uses the probability ratio between the old and new policies and applies PPO-style clipping.

Its largest difference from PPO, however, is that it does not use a separate value model.

In PPO, the value model estimates expected cumulative reward for each token state, and GAE computes a per-token advantage.

GRPO instead generates multiple answers to the same question and constructs advantage according to how much better each answer is than the group average.

Suppose we sample $G$ answers from the old policy for one question $q$.

$$\{o_1,o_2,\ldots,o_G\} \sim \pi_{\theta_{\mathrm{old}}}(O|q).$$

Evaluating each answer with a reward model or rule-based reward produces

$$\{r_1,r_2,\ldots,r_G\}.$$

Normalizing these rewards within the group gives the following relative advantage.

$$A_i = \frac{ r_i- \operatorname{mean} (\{r_1,r_2,\ldots,r_G\}) }{ \operatorname{std} (\{r_1,r_2,\ldots,r_G\}) }.$$

If a reward is above the group mean, then $A_i>0$; if it is below the mean, then $A_i<0$.

The policy is thus trained according to how good an answer is relative to other answers generated for the same question, rather than according to absolute reward magnitude alone.

GRPO's key changes are as follows.

• Unlike PPO, it does not use a separate value model.
• Without a value model, it does not use GAE.
• It generates several answers $o_{1\ldots G}$ to the same question.
• It normalizes each answer's reward using the group mean and standard deviation.
• It uses relative reward within the group as advantage.
• It applies policy-ratio clipping similar to PPO.
• It retains a KL penalty against a reference model.

GRPO's group normalization is different from GAE and does not directly address GAE's bias-variance trade-off.

GAE combines multiple $k$-step returns along the time axis to estimate advantage within a token trajectory.

GRPO compares several complete answers generated for the same question and constructs a relative advantage without a separate critic.

The two methods therefore operate along different axes.

• GAE: estimates advantage along the time axis within one trajectory.
• GRPO: compares relative rewards across several responses to the same prompt.

The GRPO objective can be expressed as follows.

$$\begin{aligned} \mathcal{J}_{\mathrm{GRPO}}(\theta) = \mathbb{E} \Bigg[ & q\sim P(Q), \quad \{o_i\}_{i=1}^{G} \sim \pi_{\theta_{\mathrm{old}}}(O|q) \newline & \frac{1}{G} \sum_{i=1}^{G} \frac{1}{|o_i|} \sum_{t=1}^{|o_i|} \Bigg( \min \left( r_{i,t}(\theta)A_i, \operatorname{clip} \left( r_{i,t}(\theta), 1-\epsilon, 1+\epsilon \right) A_i \right) \newline & \qquad\qquad - \beta D_{\mathrm{KL}} \left( \pi_\theta \| \pi_{\mathrm{ref}} \right) \Bigg) \Bigg]. \end{aligned}$$

Here, the token-level policy ratio is

$$r_{i,t}(\theta) = \frac{ \pi_\theta (o_{i,t}|q,o_{i,<t}) }{ \pi_{\theta_{\mathrm{old}}} (o_{i,t}|q,o_{i,<t}) }.$$

One group-relative advantage $A_i$ computed for answer $o_i$ is generally applied to every token in that answer.

The sample estimator for the KL term used by GRPO can be written in the following form.

$$D_{\mathrm{KL}} \left( \pi_\theta \| \pi_{\mathrm{ref}} \right) = \frac{ \pi_{\mathrm{ref}}(o_{i,t}|q,o_{i,<t}) }{ \pi_\theta(o_{i,t}|q,o_{i,<t}) } - \log \frac{ \pi_{\mathrm{ref}}(o_{i,t}|q,o_{i,<t}) }{ \pi_\theta(o_{i,t}|q,o_{i,<t}) } -1.$$

$x-\log x-1$ is always nonnegative for $x>0$ and equals zero at $x=1$. The penalty therefore grows as the policy and reference distributions diverge.

GRPO's main advantage is reduced memory use from eliminating the critic model.

PPO must train a value model similar in size to the policy and compute both value loss and GAE. GRPO instead derives advantage from the relative rewards of several responses, so it needs no separate value model.

The trade-off is that several responses must be sampled for each prompt. Training efficiency therefore depends on group size, response length, sampling cost, and related factors.

In one sentence, GRPO works as follows.

A PPO-family reinforcement-learning method that makes multiple answers to the same question compete, increasing the probability of answers above the group average and decreasing that of answers below it.

Methods / Approaches

How DeepSeek-R1-Zero Was Trained

DeepSeek-R1-Zero uses DeepSeek-V3-Base as its base model and applies GRPO as its primary reinforcement-learning method.

R1-Zero's central claim is that strong mathematical and coding reasoning can be elicited by applying reinforcement learning directly to a base LLM, without a large supervised fine-tuning stage.

Rather than first teaching a reasoning style through long human-written chain-of-thought data, the model explores diverse reasoning paths on problems whose answers can be verified, and generations that produce good outcomes receive high rewards.

Distilled models in the DeepSeek-R1 family are relatively easy to run in environments such as Ollama. Their output shows a separate reasoning process before the final answer.

I use an M3 MacBook Pro, and in terms of speed I could run distilled models up to roughly deepseek-r1:14b fairly comfortably.

These distilled models are not R1-Zero itself, however. They are Qwen- or Llama-family models fine-tuned on reasoning data generated by R1.

In output from DeepSeek-family models, the reasoning process appears in a separate region. Public examples and some deployment templates place it between <think> and </think> tags.

The tag itself matters less than the behavior seen during reinforcement learning: intermediate reasoning, review, retrying, and self-correction while reaching an answer.

Rather than relying mainly on outcome-based or process-based neural reward models, R1-Zero used rule-based rewards that could verify whether an answer was correct.

Its principal rewards were of two kinds.

• Accuracy reward evaluates whether the response's final result is correct. Answers to mathematical problems can be verified when presented in a specified format, while coding problems can be checked with a compiler and test cases.
• Format reward evaluates whether the model followed the required output format, for example whether it separated the reasoning process and final answer as specified.

Unlike a learned reward model, a rule-based reward has explicit, verifiable evaluation criteria.

For tasks such as mathematics and coding, where answers can be checked automatically, this can reduce reward hacking and incorrect judgments by a reward model.

An Interesting Observation

The paper describes an interesting phenomenon observed during training as an “aha moment.” I initially thought I had misread it, but that is indeed the expression used in the paper.

An intermediate policy model, while solving a problem, recognized an error in its current approach and then recalculated or verified the result using another method.

Training produced more than longer answers. The following patterns also appeared without being written out by people.

• reconsidering the current solution method
• recognizing the possibility of an error
• trying a different approach
• independently verifying a computed result
• voluntarily extending the reasoning process

An answer-generating LLM is a policy model that makes decisions token by token.

When this policy model was trained with verifiable rewards and allowed to explore diverse reasoning paths, patterns of self-review and correction appeared without humans having to write every reasoning process explicitly.

This need not be interpreted as awareness or understanding in the human sense. It is more appropriate to view it as reinforcement of token-generation patterns associated with high reward.

How DeepSeek-R1 Was Trained

R1-Zero as described so far mainly targets tasks with automatically verifiable answers, such as mathematics and coding.

Real LLMs must support a much wider variety of tasks, however, and R1-Zero's output had the following limitations.

• difficult-to-read reasoning processes
• mixed languages
• weak general conversation and instruction-following ability
• excessively long or poorly organized responses

DeepSeek-R1-Zero demonstrated that a template and reinforcement learning alone could elicit substantial reasoning ability.

This raises two questions.

1. Can training first on a small amount of high-quality reasoning data reduce instability in early reinforcement learning and improve convergence speed and output quality?
2. Can the chain of thought become more consistent and readable while performance also improves on general tasks beyond mathematics and coding?

DeepSeek-R1 used a cold-start approach to address these questions.

Instead of immediately applying large-scale RL to DeepSeek-V3-Base, it first performs supervised fine-tuning on several thousand high-quality, long chain-of-thought samples.

The cold-start data was constructed from

• long CoT generated through few-shot prompting,
• reflection and verification of model-generated answers,
• human-written answers, and
• rewrites of R1-Zero's difficult-to-read answers into an organized format.

Rather than a massive SFT dataset intended to inject all reasoning capability directly, this data provides an output format and basic reasoning patterns so the subsequent reinforcement learning can begin stably.

After cold-start SFT, the model undergoes reasoning-oriented RL similar to R1-Zero, focused on verifiable tasks in mathematics, coding, science, logic, and related domains.

Language mixing emerged during reasoning RL. For example, a reasoning process for one question might mix English, Chinese, and other languages.

To mitigate this, DeepSeek added a language-consistency reward. Language consistency can be evaluated by metrics such as the proportion of words in the response that belong to the target language.

Adding this reward can slightly reduce performance on pure reasoning benchmarks, but the developers judged that answers readable in one consistent language matter more to actual users.

After reasoning RL, they assembled roughly 800,000 examples for an additional round of SFT.

The data contains two broad categories.

• roughly 600,000 reasoning-related samples
• roughly 200,000 general non-reasoning samples

The reasoning data includes mathematics, coding, science, and logic problems. For some problems, multiple model-generated answers were verified or their quality was evaluated with a generative reward model.

The non-reasoning data includes general LLM tasks such as writing, factual QA, self-cognition, and translation.

In the final stage, the model undergoes additional reinforcement learning across all scenarios.

Reasoning tasks use accuracy-focused rewards, while general conversational tasks use reward models that evaluate helpfulness and harmlessness.

The full DeepSeek-R1 training process can therefore be simplified as follows.

1. DeepSeek-V3-Base
2. Cold-start SFT on a small amount of high-quality CoT data
3. Reasoning RL focused on mathematics and coding
4. SFT on roughly 800,000 examples covering reasoning and general tasks
5. Final RL covering both reasoning and general preferences

If R1-Zero is best viewed as an experiment showing that reasoning can emerge through reinforcement learning alone, R1 is the model that refines that result into a practical, general-purpose LLM.

Distilled Model

Through distillation, DeepSeek also released models that are relatively smaller than the original R1.

Rather than simply compressing R1's parameters, the distilled models are Qwen- and Llama-family base models supervised fine-tuned on reasoning data generated by R1.

DeepSeek used approximately 800,000 samples collected during R1 training to fine-tune Qwen- and Llama-family models.

The distilled models did not undergo another separate round of large-scale RL. Instead, they learned from high-quality reasoning trajectories generated by R1 through SFT.

A smaller student model was taught to imitate reasoning patterns learned at great expense by a larger teacher model.

In the experiments, distillation from reasoning data generated by the large R1 model often outperformed applying reinforcement learning directly to a smaller model.

Perhaps this is why being born into a wealthy family helps.

This does not mean that directly applying RL to a small model is meaningless. With limited model capacity, however, learning reasoning traces generated by an already powerful teacher proved more efficient than discovering reasoning ability from scratch through exploration.

Distillation is practical because it can transfer reasoning ability to smaller models. But because it depends on teacher-generated data, the teacher's errors, stylistic tendencies, and biases can be transferred as well.

Quantitative benchmark scores also do not capture the entire real-world experience. Accuracy on reasoning problems matters, but so do the following factors.

• consistency of the reasoning process
• frequency of unnecessarily long answers
• ability to correct a false premise independently
• instruction-following ability
• mixed languages or formatting errors
• stability on the distribution of real user questions

What I found most interesting about DeepSeek-R1 was its training method, rather than any single benchmark score.

At scale, it showed that reinforcement learning with verifiable rewards can strengthen long-horizon reasoning, reconsideration, and self-correction in a base model without requiring people to write every reasoning process themselves.