Introduction

Until now, our agents could keep interacting with the environment to generate new experience. They could retry after a failure and explore actions they did not understand.

That assumption is often unrealistic.

• We cannot try a dangerous medical treatment merely for training.
• A robot will break if we make it fall millions of times.
• We may have only recorded user logs and no way to run a new experiment.

This creates the problem of learning a policy from a fixed dataset

$$\mathcal D = \left\{ (s_i,a_i,r_i,s'_i) \right\}_{i=1}^{N}$$

alone.

This article compares imitation learning, which copies an expert's actions, with offline RL, which also uses rewards in a fixed dataset to search for a better policy.

Behavior Cloning: Treat the Action as a Label

Suppose a demonstration dataset contains pairs of states and expert actions.

$$\mathcal D_{\text{demo}} = \left\{ (s_i,a_i^{\text{expert}}) \right\}$$

Behavior Cloning treats this as supervised classification or regression. For discrete actions, it can minimize the negative log-likelihood

$$L_{\text{BC}}(\theta) = - \mathbb E_{(s,a)\sim\mathcal D} \left[ \log\pi_\theta(a\mid s) \right].$$

The equation increases the probability of the action selected by the expert in the dataset. Its advantage is that neither rewards nor next states are required.

A Small Mistake Creates a New State

A policy can fail during execution even when its supervised validation accuracy is high.

The expert dataset is concentrated on states visited by the expert. A small action error moves the learned policy slightly away from the expert path. The new state barely appears in the dataset, so the next action error becomes larger, pushing the trajectory farther away.

This is covariate shift. During training,

$$s\sim d_{\text{expert}},$$

but during execution,

$$s\sim d_{\pi_\theta}.$$

The distribution of input states has changed.

DAgger Also Asks About States Visited by the Policy

DAgger reduces this distribution gap.

1. Run the current policy in the environment.
2. Collect the states it actually visits.
3. Ask the expert for the correct action in those states.
4. Add the new labels to the dataset and train again.

$$\mathcal D \leftarrow \mathcal D \cup \left\{ (s,\pi_{\text{expert}}(s)) \right\}$$

The dataset now contains expert examples for states reached after a mistake. DAgger, however, requires access to the environment during training and the ability to query the expert again. It cannot be used in a completely fixed offline setting.

What Makes Offline RL Different?

An offline RL dataset includes rewards and next states.

$$(s,a,r,s')$$

Behavior Cloning asks only which action appears in the data. Offline RL uses the reward and future following that action and attempts to improve upon the behavior policy that generated the dataset.

For example, action A may appear frequently near a state but produce a low reward, while action B appears rarely but produces a high reward. Offline RL can increase the weight of B.

That possibility is attractive, but it also creates the central difficulty: we cannot observe the outcome of actions absent from the dataset.

The Q-Value of an Unseen Action

In online Q-Learning, an agent can execute an action with a high Q-value and test the prediction. If it is wrong, a new transition corrects the Q-value.

Offline RL cannot execute an action outside the dataset. If function approximation accidentally gives that action a high Q-value, no contradictory data arrives.

$$Y = r + \gamma\max_{a'}Q(s',a')$$

If the $\max$ selects an $a'$ that appears rarely in the dataset, an unsupported Q-value enters the bootstrap target. Training Q on that target can amplify the error. This is called extrapolation error, or the out-of-distribution action problem.

Staying Close to the Dataset

Offline RL methods generally prevent the policy or value function from moving too far outside the dataset's support.

Policy Constraints

Restrict the learned policy from becoming too different from the behavior policy.

$$D \left( \pi_\theta(\cdot\mid s), \pi_\beta(\cdot\mid s) \right) \leq \epsilon$$

$\pi_\beta$ is the behavior policy that generated the dataset.

Conservative Values

Penalize Q-values for actions absent from the dataset: if there is no evidence, do not assume the action is good.

Advantage-Weighted Imitation

Rather than copy every data action equally, imitate actions with high advantages more strongly.

$$L(\theta) = - \mathbb E_{(s,a)\sim\mathcal D} \left[ w(s,a) \log\pi_\theta(a\mid s) \right]$$

$w(s,a)$ is larger for better actions.

Dataset Quality Determines the Algorithm's Limits

Offline RL cannot magically evaluate behavior entirely absent from the dataset.

When examining a dataset, ask:

• Coverage: does it contain the important states and actions?
• Quality: does it include both successes and failures, or is it skewed toward one side?
• Reward: does the recorded reward represent the real objective?
• Behavior policy: who generated the data, and under what rule?
• Confounding: did an unrecorded variable affect both the action and its outcome?

When only good trajectories are available, Behavior Cloning can be a strong baseline. When the data contains diverse successes and failures and the rewards are reliable, offline RL can make better use of the relative value of actions.

Comparing the Four Settings

The question “Can we generate new data during training?” often reveals the nature of the problem more clearly than the terminology does.

What to Remember

Imitation learning learns to follow actions in a dataset. Offline RL uses rewards and transitions in a fixed dataset to search for better behavior, while guarding against optimistic estimates outside that dataset.

1. Behavior Cloning is supervised learning that minimizes $-\log\pi(a\mid s)$.
2. Small execution errors can lead to states outside the training distribution and accumulate.
3. DAgger adds expert labels for states visited by the current policy.
4. Offline RL uses rewards, but cannot collect new experience to verify Q-values for OOD actions.
5. Dataset coverage and quality strongly limit the policies that can be learned.

The next article turns offline trajectories into token sequences of returns, states, and actions, allowing a Transformer to predict actions as a Decision Transformer.

References

• Levine et al., Offline Reinforcement Learning: Tutorial, Review, and Perspectives
• Ross et al., A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
• Sutton & Barto, Reinforcement Learning: An Introduction, 2nd Edition