Introduction

REINFORCE was the simplest method we had seen for learning a policy directly.

$$\theta \leftarrow \theta + \alpha G_t \nabla_\theta\log\pi_\theta(A_t\mid S_t)$$

The problem is that we must wait until the end of the episode to know $G_t$. Even the return following the same action can vary widely because of what happens by chance later in the episode, making the update noisy.

Actor-Critic divides the job into two parts.

• Actor: learns the policy that determines which action to choose.
• Critic: evaluates how good the current state or action is.

The Actor improves its policy using the Critic's evaluation, while the Critic improves that evaluation using actual rewards. Both functions learn together from the same experience.

A Raw Return Has No Point of Comparison

Suppose the agent chooses action $a$ in state $s$ and receives a return of $6$. Was that a good result?

• If this state normally produces $2$, it was excellent.
• If it normally produces $10$, it was actually poor.

The absolute return is difficult to judge on its own. Subtracting what we expected on average in the current state tells us how much better or worse the action was than usual.

$$A^\pi(s,a) = Q^\pi(s,a)-V^\pi(s)$$

$A^\pi$ is the advantage function.

• $A^\pi(s,a)>0$: an action better than the average for this state
• $A^\pi(s,a)<0$: an action worse than the average
• $A^\pi(s,a)=0$: an action about as good as expected

For example, if

$$V(s)=4, \qquad Q(s,A)=6, \qquad Q(s,B)=3,$$

then

$$A(s,A)=6-4=2$$

and

$$A(s,B)=3-4=-1.$$

The Actor should increase the probability of A and decrease the probability of B.

A Baseline Does Not Change the Expected Direction

Subtracting a state-only function $b(s)$ from a policy gradient does not change the expected gradient.

$$\mathbb E_{A\sim\pi} \left[ b(S)\nabla_\theta\log\pi_\theta(A\mid S) \right] =0$$

This follows from the fact that action probabilities sum to $1$.

$$\begin{aligned} \sum_a \pi_\theta(a\mid s) \nabla_\theta\log\pi_\theta(a\mid s) &= \sum_a\nabla_\theta\pi_\theta(a\mid s) \\ &= \nabla_\theta\sum_a\pi_\theta(a\mid s) \\ &= \nabla_\theta 1 \\ &=0 \end{aligned}$$

A baseline reduces unnecessary variation without changing the average update direction. The most natural baseline is $V^\pi(s)$.

What Does the Critic Learn?

Let the Critic be a state-value function parameterized by $\mathbf w$.

$$V_{\mathbf w}(s) \approx V^\pi(s)$$

The Critic can be trained with a TD target.

$$R_{t+1} + \gamma V_{\mathbf w}(S_{t+1})$$

The TD error is

$$\delta_t = R_{t+1} + \gamma V_{\mathbf w}(S_{t+1}) - V_{\mathbf w}(S_t).$$

It tells us whether the outcome was better than expected after actually taking one step. This makes $\delta_t$ a useful one-step estimate of the advantage.

$$\delta_t \approx A^\pi(S_t,A_t)$$

Reading a TD Error with Numbers

Suppose we observe

$$R_{t+1}=1, \quad \gamma=0.9, \quad V(S_t)=4, \quad V(S_{t+1})=5.$$

The TD error is

$$\begin{aligned} \delta_t &= 1+0.9\times5-4 \\ &=1.5. \end{aligned}$$

We expected about $4$ from the current state, but after receiving reward $1$, the next state still held $5$ worth of future value. The one-step outcome was therefore $1.5$ better than expected.

Conversely, if $\delta_t=-0.7$, we regard the chosen action as worse than expected.

Updating the Actor

The Actor uses the TD error in place of the return.

$$\theta \leftarrow \theta + \alpha_\theta \delta_t \nabla_\theta \log\pi_\theta(A_t\mid S_t)$$

• If $\delta_t>0$, increase the probability of the selected action.
• If $\delta_t<0$, decrease its probability.
• If $\delta_t$ is close to $0$, barely move the policy.

REINFORCE waits for an episode-wide return. Actor-Critic can update after every step.

Updating the Critic

The Critic updates its value parameters to reduce the TD error.

$$\mathbf w \leftarrow \mathbf w + \alpha_w \delta_t \nabla_{\mathbf w}V_{\mathbf w}(S_t)$$

For a linear function,

$$\nabla_{\mathbf w}V_{\mathbf w}(S_t) = \mathbf x(S_t),$$

while a neural network obtains the gradient through backpropagation.

The Actor and Critic usually have separate learning rates. If the Critic learns too slowly, the Actor follows inaccurate evaluations. If it changes too aggressively, it sends the Actor an unstable signal.

The Complete Flow in One Step

1. The Actor samples $A_t\sim\pi_\theta(\cdot\mid S_t)$.
2. The environment returns $R_{t+1}$ and $S_{t+1}$.
3. The Critic computes $\delta_t$.
4. The Critic uses $\delta_t$ to update its value estimate.
5. The Actor uses the same $\delta_t$ to update the probability of the selected action.
6. Repeat from $S_{t+1}$.

The Actor chooses a direction; the Critic quickly reports whether that direction worked better than expected.

Trading Bias for Variance

The Monte Carlo return $G_t$ uses the completed episode, so it has no bootstrapping bias but high variance.

A one-step TD error uses the Critic's estimate to reduce variance, but it becomes biased when the Critic is wrong.

Generalized Advantage Estimation, or GAE, follows the same idea by mixing TD errors over several horizons.

$$\hat A_t^{\text{GAE}(\gamma,\lambda)} = \sum_{l=0}^{\infty} (\gamma\lambda)^l \delta_{t+l}$$

A small $\lambda$ places more trust in short TD errors. A value near $1$ incorporates more of the distant future.

Actor-Critic Can Still Be Unstable

When the Actor changes, the policy that the Critic must evaluate changes as well. When the Critic changes, so does the advantage signal received by the Actor. Each learner is moving the other's target.

A large policy update can quickly separate the current policy from the one that collected the available experience and destabilize training. PPO, the subject of the next article, examines the probability ratio between an old and a new policy and prevents the policy from moving too far in a single update.

What to Remember

The Actor learns an action distribution, the Critic learns expected value, and TD error connects their updates.

1. Advantage, $Q(s,a)-V(s)$, measures how much better an action is than the state's average.
2. Subtracting a state-dependent baseline preserves the expected gradient while reducing variance.
3. $\delta_t=R+\gamma V(S')-V(S)$ provides a one-step estimate of advantage.
4. The Actor improves action probabilities; the Critic improves value predictions.
5. n-step returns and GAE let us tune the trade-off between bias and variance.

References

• David Silver, Lecture 7: Policy Gradient Methods
• Sutton & Barto, Reinforcement Learning: An Introduction, 2nd Edition
• Seoul National University DSBA Lab, Introduction of Reinforcement Learning