Introduction

Policy Gradient increases the probability of good actions. Even when the direction is correct, taking too large a step can cause problems.

If the policy changes substantially while we perform several updates on trajectories collected by the current policy, that data no longer comes from the policy we are updating. The probability of one action may suddenly dominate and erase other good actions.

Proximal Policy Optimization, or PPO, implements a simple idea with a relatively simple objective: move in a good direction, but do not move too far from the old policy.

This article reads the probability ratio and clipping mechanism of PPO-Clip with concrete numbers.

The Probability Ratio between the Old and New Policies

Let $\pi_{\theta_{\text{old}}}$ be the policy that collected the trajectories and $\pi_\theta$ the policy currently being updated.

For a sample $(S_t,A_t)$, calculate

$$r_t(\theta) = \frac{ \pi_\theta(A_t\mid S_t) }{ \pi_{\theta_{\text{old}}}(A_t\mid S_t) }.$$

This ratio shows how much the probability of the selected action has changed.

• $r_t=1$: the probability is unchanged.
• $r_t=1.2$: it is $20\%$ larger than under the old policy.
• $r_t=0.7$: it has fallen to $70\%$ of the old probability.

For example, if an action's probability was $0.4$ under the old policy and $0.5$ under the new policy,

$$r_t=\frac{0.5}{0.4}=1.25.$$

The Clipped Objective

The central PPO-Clip objective is

$$L^{\text{CLIP}}(\theta) = \mathbb E_t \left[ \min \left( r_t(\theta)\hat A_t, \operatorname{clip} \left( r_t(\theta),1-\epsilon,1+\epsilon \right) \hat A_t \right) \right].$$

It looks long, but it simply chooses the more conservative of two candidates.

1. $r_t\hat A_t$: the unconstrained Policy Gradient objective
2. $\operatorname{clip}(r_t,1-\epsilon,1+\epsilon)\hat A_t$: the objective after restricting the ratio to an interval
3. $\min$: select the less optimistic value

With the common choice $\epsilon=0.2$, the reference interval is

$$[1-\epsilon,1+\epsilon] = [0.8,1.2].$$

When the Advantage Is Positive

$\hat A_t>0$ means the chosen action was better than expected, so we want to increase its probability.

Suppose

$$\hat A_t=2, \qquad r_t=1.35, \qquad \epsilon=0.2.$$

The unconstrained value is

$$r_t\hat A_t = 1.35\times2 =2.7.$$

The clipped value is

$$\operatorname{clip}(1.35,0.8,1.2)\times2 = 1.2\times2 =2.4.$$

The objective therefore uses

$$\min(2.7,2.4)=2.4.$$

Once the probability ratio exceeds $1.2$, increasing it further no longer improves the objective for this sample.

When the Advantage Is Negative

$\hat A_t<0$ means the selected action was worse than expected, so we want to reduce its probability. But we should also prevent an excessive reduction.

If

$$\hat A_t=-2, \qquad r_t=0.7, \qquad \epsilon=0.2,$$

then

$$r_t\hat A_t = 0.7\times(-2) =-1.4,$$

while the clipped value is

$$0.8\times(-2) =-1.6.$$

Using

$$\min(-1.4,-1.6) =-1.6$$

removes any additional benefit from pushing $r_t$ below $0.8$.

The $\min$ can be confusing when the values are negative. Just remember that $-1.6$ is smaller than $-1.4$ on the number line.

Clipping Does Not Hard-Constrain the Probability

PPO does not force every probability ratio to stay inside $[0.8,1.2]$. Its objective merely removes the incentive for improvement beyond that interval.

Because gradients from other samples share network parameters, some ratios can still cross the boundary. Implementations therefore often monitor an approximate KL divergence and stop an epoch early if it becomes too large.

$$\widehat{D}_{\mathrm{KL}} \left( \pi_{\text{old}} \parallel \pi_\theta \right)$$

It is more accurate to view clipping as a mechanism that removes the reward for one sample pushing the policy too far, rather than as a hard constraint.

How PPO Trains One Batch

PPO commonly follows this sequence.

1. Collect several trajectories with the old policy $\pi_{\theta_{\text{old}}}$.
2. Calculate returns and advantages $\hat A_t$.
3. Save the old action probability for every sample.
4. Shuffle the batch into mini-batches.
5. Optimize the PPO objective for several epochs over the same data.
6. Treat the updated policy as the next old policy and collect fresh data.

What Else Is in the Practical Loss?

An Actor-Critic implementation of PPO does not use only the policy loss.

$$L = L_{\text{policy}} + c_vL_{\text{value}} - c_eH(\pi_\theta)$$

• $L_{\text{policy}}$: trains the Actor with the clipped objective.
• $L_{\text{value}}$: reduces the Critic's value-prediction error.
• $H(\pi_\theta)$: encourages entropy so the policy does not collapse onto one action too early.
• $c_v,c_e$: control the relative scale of the terms.

Papers may write an objective to maximize, while code may prepend a minus sign and minimize a loss. When signs differ across implementations, check what each one is maximizing or minimizing.

Why PPO Is Commonly Paired with GAE

PPO is strongly affected by the quality of its advantage estimate. GAE is a common choice.

$$\hat A_t = \delta_t + (\gamma\lambda)\delta_{t+1} + (\gamma\lambda)^2\delta_{t+2} +\cdots$$

where

$$\delta_t = R_{t+1} + \gamma V(S_{t+1}) - V(S_t).$$

$\lambda$ tunes the bias-variance trade-off between short TD estimates and long returns.

Implementations also often normalize advantages within a mini-batch to approximately zero mean and unit standard deviation. This stabilizes gradient scale, but it means that learning uses the relative ordering of advantages in the batch.

What PPO Does Not Solve

PPO is a practical way to stabilize policy updates, not an answer to every RL problem.

• It still requires continually collecting fresh on-policy data.
• A poorly designed reward can make it stably optimize the wrong goal.
• It is sensitive to hyperparameters and implementation details.
• Clipping alone does not guarantee safety or generalization.

PPO appears again in RLHF. There, too, the central problem is to increase a preference reward without moving too far from a reference policy.

What to Remember

PPO compares the action probabilities of the old and new policies and clips the incentive for an advantage estimate to push the policy too far in one update.

1. $r_t=\pi_\theta(A_t\mid S_t)/\pi_{\text{old}}(A_t\mid S_t)$.
2. If $\hat A_t>0$, increase the selected action's probability; if it is negative, decrease it.
3. PPO-Clip uses the more conservative value before and after clipping.
4. Clipping is not a hard constraint that forces the ratio into an interval.
5. Practical PPO is used with a value loss, entropy, and GAE.

References

• Schulman et al., Proximal Policy Optimization Algorithms
• David Silver, Lecture 7: Policy Gradient Methods
• Sutton & Barto, Reinforcement Learning: An Introduction, 2nd Edition