Introduction

DQN predicts a Q-value for every action and selects the largest one.

$$a = \arg\max_{a'}Q(s,a')$$

The policy is therefore hidden behind the value function: learn the Q-values well, and the policy follows.

Policy Gradient reverses that order. It explicitly represents a policy parameterized by $\boldsymbol\theta$ and moves $\boldsymbol\theta$ in a direction that increases the probability of actions that produced good returns.

$$\pi_\theta(a\mid s)$$

The first question is usually, “If the environment is not differentiable, how can we differentiate action probabilities?” This article introduces the log-derivative trick that answers that question and REINFORCE, the most basic Policy Gradient algorithm.

Why Represent the Policy Directly?

For discrete actions, we can use a Softmax policy.

$$\pi_\theta(a\mid s) = \frac{ \exp h_\theta(s,a) }{ \sum_b\exp h_\theta(s,b) }$$

$h_\theta(s,a)$ is a score representing the preference for an action.

For continuous actions, we can use a Gaussian policy that outputs a mean and standard deviation.

$$A_t \sim \mathcal N \left( \mu_\theta(S_t), \sigma_\theta(S_t)^2 \right)$$

The policy can sample a continuous action directly, without listing the Q-values of every possible action or calculating an $\arg\max$.

Learning the policy directly has several advantages.

• It handles continuous actions naturally.
• It can represent problems that require stochastic policies.
• It changes action probabilities smoothly.

Its disadvantages are that the gradient can have higher variance than in value-based methods, and on-policy experience is difficult to reuse for long.

What Are We Maximizing?

Let the objective be the expected return of the policy.

$$J(\theta) = \mathbb E_{\tau\sim\pi_\theta} \left[ G_0 \right]$$

$\tau$ is a trajectory generated by the policy.

$$\tau = (S_0,A_0,R_1,S_1,A_1,\ldots)$$

We want the direction in which $J(\theta)$ increases:

$$\nabla_\theta J(\theta).$$

Make Actions That Produced Good Results More Likely

REINFORCE uses the following update.

$$\theta \leftarrow \theta + \alpha G_t \nabla_\theta \log\pi_\theta(A_t\mid S_t)$$

We can read the equation as a sentence.

• If $G_t>0$, increase the log-probability of the selected action $A_t$.
• If $G_t<0$, decrease that action's log-probability.
• The larger $|G_t|$ is, the larger the update becomes.

Suppose the only actions are LEFT and RIGHT, and currently

$$\pi_\theta(\text{RIGHT}\mid s)=0.6.$$

If we sample RIGHT and obtain a good return, we move in a direction that increases $\log\pi_\theta(\text{RIGHT}\mid s)$. Because the probabilities must sum to $1$, increasing RIGHT's probability decreases LEFT's.

The important point is that we do not train only the action with the largest probability. We evaluate the action that was actually sampled according to its outcome.

Why Use the Logarithm?

Consider an expectation under a probability $p_\theta(x)$.

$$J(\theta) = \sum_x p_\theta(x)R(x)$$

Differentiating gives

$$\nabla_\theta J(\theta) = \sum_x \nabla_\theta p_\theta(x)R(x).$$

We use the identity

$$\nabla_\theta p_\theta(x) = p_\theta(x) \nabla_\theta\log p_\theta(x),$$

which follows from

$$\nabla_\theta\log p_\theta(x) = \frac{ \nabla_\theta p_\theta(x) }{ p_\theta(x) }.$$

Substituting it produces

$$\nabla_\theta J(\theta) = \sum_x p_\theta(x) \nabla_\theta\log p_\theta(x) R(x),$$

or, as an expectation,

$$\nabla_\theta J(\theta) = \mathbb E_{x\sim p_\theta} \left[ R(x)\nabla_\theta\log p_\theta(x) \right].$$

We no longer need to sum over every possible outcome. A sample $x$ from the current policy and its observed $R(x)$ are enough to estimate the gradient.

Why the Environment Does Not Need to Be Differentiable

The environment's response after an action may be impossible to differentiate. Collisions, game rules, and human reactions are often discontinuous.

Policy Gradient does not backpropagate through the environment. It uses the return from the environment as a weight and differentiates only the log-probability of the differentiable policy.

$$\underbrace{G_t}_{\text{environment feedback}} \times \underbrace{ \nabla_\theta\log\pi_\theta(A_t\mid S_t) }_{\text{policy gradient}}$$

The environment reports the outcome; the policy changes the probability of the choice that produced it.

The Policy Gradient Theorem

In an MDP that revisits states, the Policy Gradient Theorem gives

$$\nabla_\theta J(\theta) \propto \mathbb E_{\pi_\theta} \left[ Q^{\pi_\theta}(S_t,A_t) \nabla_\theta \log\pi_\theta(A_t\mid S_t) \right].$$

The key result is that we can calculate the gradient from trajectories sampled under the current policy without directly differentiating the state-visitation probabilities.

REINFORCE substitutes the observed return $G_t$ for the unknown $Q^\pi(S_t,A_t)$.

$$Q^\pi(S_t,A_t) \approx G_t$$

Why Use $G_t$ Instead of the Entire Episode Return?

An action at time $t$ cannot affect rewards that were received before it. We should therefore evaluate $A_t$ using only the return from that point onward.

$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2R_{t+3} +\cdots$$

Including earlier rewards adds noise unrelated to the action. Respecting this causal relationship reduces gradient variance.

The Complete REINFORCE Procedure

1. Generate one episode using the current policy $\pi_\theta$.
2. Calculate return $G_t$ at every time $t$.
3. Accumulate $G_t\nabla_\theta\log\pi_\theta(A_t\mid S_t)$.
4. Update the parameters after the episode ends.

$$\theta \leftarrow \theta + \alpha \sum_{t=0}^{T-1} G_t \nabla_\theta\log\pi_\theta(A_t\mid S_t)$$

The implementation is simple, but when episode returns fluctuate widely, so do the gradients.

The High Variance of REINFORCE

Even after choosing the same action in the same state, $G_t$ can change because of environment randomness and subsequent actions. One rollout may produce $+10$ and the next $-3$.

REINFORCE multiplies by this return directly, so its updates have high variance. We may need to average many trajectories before a stable direction emerges.

The next article subtracts the value normally expected in the state as a baseline and introduces a Critic that learns that baseline. This is the starting point of Actor-Critic.

What to Remember

Policy Gradient directly updates policy parameters in a direction that increases the log-probability of actions that produced good returns.

1. A policy $\pi_\theta(a\mid s)$ directly represents a probability distribution over actions.
2. The log-derivative trick lets us estimate a gradient from samples without differentiating the environment.
3. REINFORCE uses $G_t\nabla\log\pi_\theta(A_t\mid S_t)$.
4. The sign of $G_t$ determines whether to increase or decrease a probability; its magnitude determines the update strength.
5. The method is simple, but the Monte Carlo return gives it high variance.

References

• David Silver, Lecture 7: Policy Gradient Methods
• Seoul National University DSBA Lab, Introduction of Reinforcement Learning
• Sutton & Barto, Reinforcement Learning: An Introduction, 2nd Edition