Introduction

Reinforcement learning maximizes cumulative reward.

$$J(\pi) = \mathbb E_\pi \left[ \sum_{t=0}^{T} \gamma^tR_{t+1} \right]$$

Does a rising training reward mean that the problem has been solved?

No. An agent optimizes the number we actually implemented, not the goal we intended in our minds. If the reward is an incomplete proxy for that goal, the agent can earn a high score while behaving differently from what we wanted.

The same code can also produce entirely different learning curves under different random seeds. A policy may earn high rewards in the training environment and fail in a new one.

This final article organizes reward design, reward hacking, multi-seed evaluation, generalization, and safety metrics into one experimental checklist.

A Reward Is a Number, Not an Instruction Manual

An environment does not directly tell an agent, “finish the task safely and efficiently.” We usually combine several scalars into a reward.

$$R_t = w_1R_t^{\text{progress}} + w_2R_t^{\text{success}} - w_3C_t^{\text{collision}} - w_4C_t^{\text{energy}}$$

Changing one weight can change the optimal behavior.

If the progress reward is too large, the agent may repeat an action that counts as progress without ever reaching the goal. If the collision penalty is too severe, doing nothing may become the safest optimal solution.

When designing a reward, ask which behaviors can maximize the exact equation. Sensible-looking terms are not enough.

Sparse and Dense Rewards

A sparse reward that gives $+1$ only when the goal is reached has a clear intention, but makes exploration difficult.

$$R_t = \begin{cases} 1,&\text{goal reached} \\ 0,&\text{otherwise} \end{cases}$$

Dense shaping that gives a reward whenever the agent moves closer to the goal provides a richer learning signal.

$$R_t^{\text{shape}} = d(S_t,\text{goal}) - d(S_{t+1},\text{goal})$$

Reducing the distance is not always equivalent to completing the task, however. The agent may need to move temporarily away from the goal to go around an obstacle.

Good shaping should help exploration without arbitrarily changing the optimal policy of the original problem. Potential-based shaping is a representative method for addressing this property.

Reward Hacking and Specification Gaming

When an agent exploits a gap in the reward function to earn a high score, we call the behavior reward hacking or specification gaming.

Suppose completing a course gives $+10$, while collecting an intermediate bonus gives $+1$. If the bonus respawns and the episode does not end, the agent can keep circling the bonus area instead of crossing the finish line.

The agent has not disobeyed. If anything, it followed the equation we wrote too faithfully.

Before deploying a reward, ask the following questions.

1. Can any behavior be repeated to collect reward indefinitely?
2. Can part of the success signal be produced without actual success?
3. Does doing nothing become a solution that avoids every penalty?
4. Can the agent exploit episode boundaries or sensor errors?
5. Can a reward model assign a high score to strange outputs outside what it has seen?

Distinguishing Terminated from Truncated

The reason an episode ends also affects the target.

• Terminated: the MDP reached a genuine terminal state such as a goal or failure.
• Truncated: the episode was cut by a time limit or another external condition.

At a genuine terminal state, future value is zero.

$$Y_t=R_{t+1}$$

If a time limit cut the episode but the state could have continued, bootstrapping may need to remain.

$$Y_t = R_{t+1} + \gamma V(S_{t+1})$$

Combining both cases into one done flag can underestimate the value of a good state that merely reached a time limit. Evaluation should also count “ended by failure” separately from “cut when time expired.”

One Good Run Is Not a Result

Neural-network initialization, environment randomness, replay sampling, and action sampling can make the same algorithm produce different outcomes.

A high score under one seed is not the algorithm's general performance.

At minimum, report the following together.

• The number of random seeds used
• The same training budget for every seed
• Mean or median
• Uncertainty such as standard deviation, standard error, or a confidence interval
• The distribution across all runs

With few seeds, a mean can be dominated by one or two outliers. We must also fix which point of the learning curve is compared and whether the score averages the final several episodes.

Separate Training and Evaluation Curves

Training may include exploration noise. Returns collected with a stochastic PPO policy or $\epsilon$-greedy DQN actions can differ from the performance of the policy that will actually be deployed.

At fixed intervals, freeze the policy and run separate evaluation episodes.

• Do not perform training updates.
• Fix the number of evaluation episodes.
• Use the same evaluation protocol for every method.
• When necessary, report deterministic and stochastic actions separately.
• Check the evaluation environment for training normalization or data leakage.

What Should We Measure Besides Reward?

One scalar reward compresses several behaviors. It is therefore useful to expand the actual concerns into separate metrics.

Success Rate

A high episode return does not necessarily mean the task was completed. Calculate the success condition separately.

Safety and Constraint Violations

Count safety-related events such as collisions, entry into restricted areas, the proportion of dangerous actions, or maximum force.

Generalization

Evaluate on initial states, layouts, dynamics, and noise not seen during training.

Sample Efficiency

Compare the final score and the number of environment steps needed to reach the same performance.

Compute and Wall-Clock Time

Even with the same number of environment steps, actual cost differs according to model size and update count. Report GPU time, wall-clock time, and memory when relevant.

Robustness and Failure Cases

Directly classify failure types hidden beneath an average and identify the states in which failures concentrate.

Separate Training and Test Environments

Like supervised learning, reinforcement learning needs a generalization evaluation.

$$\mathcal M_{\text{train}} \neq \mathcal M_{\text{eval}}$$

This does not require an entirely different task. Within the same objective, we can vary

• starting positions,
• obstacle layouts,
• dynamics parameters,
• observation noise, and
• background elements not directly represented in the reward.

If we repeatedly view an evaluation setting while choosing hyperparameters, it effectively becomes a validation set. It is better to preserve a separate final test set.

A Checklist for Fair Comparisons

Suppose we compare a new algorithm A with baseline B. The following conditions should be aligned.

1. Do they receive the same environment-step budget?
2. Are network size and observation preprocessing comparable?
3. Are their hyperparameter-tuning budgets similar?
4. Are there enough evaluation episodes and seeds?
5. Are variance and individual runs reported alongside the mean?
6. Were failed seeds retained rather than removed arbitrarily?
7. Are training reward and held-out evaluation distinguished?
8. Are sample efficiency and compute efficiency kept separate?

A good result should explain the conditions under which a high number repeats and how reliably it does so.

Closing the Series

The first article began with the agent, environment, state, action, and reward. We then moved through MDPs, the Bellman equation, dynamic programming, Monte Carlo, TD, control, and exploration.

When the table became too large, we encountered function approximation and DQN, then continued to REINFORCE, actor–critic, and PPO, which learn policies directly. Model-based RL, offline RL, and Decision Transformer examined how experience is produced and the range covered by the data. Finally, we reached preference learning and reward design.

There are many algorithm names, but the recurring questions are similar.

• From which experience does the method learn?
• Is the target an actual return or a bootstrap estimate?
• Does it learn a policy directly, or derive one through a value?
• Can it produce new data?
• Does the optimized reward represent the actual intention?

Writing down these five questions when reading a new paper makes it easier to locate an unfamiliar equation within the broader structure.

What to Remember

Success in reinforcement learning should not be judged by one training reward, but by whether the intended behavior is produced safely and repeatedly across several conditions and seeds.

1. An agent optimizes the implemented reward, not a person's intention.
2. Dense shaping can help learning but also create new loopholes.
3. Terminated and truncated episodes must be distinguished in both bootstrapping and evaluation.
4. Report means and uncertainty across several seeds.
5. Measure success, safety, generalization, sample efficiency, and compute in addition to reward.
6. Separate the training environment from a held-out evaluation environment.

References

• DeepMind, Specification gaming: the flip side of AI ingenuity
• Henderson et al., Deep Reinforcement Learning That Matters
• Agarwal et al., Deep Reinforcement Learning at the Edge of the Statistical Precipice
• Sutton & Barto, Reinforcement Learning: An Introduction, 2nd Edition