Introduction

When I wrote about RLHF some time ago, I spent a fair amount of effort explaining terms such as PPO and reward model. To be honest, though, I was closer to following the training process by piecing together explanations scattered across several sources than to properly understanding reinforcement learning. I could read the equations, but the reason they had to take those particular forms remained slightly vague. If someone asked about on-policy and off-policy, I could recite the definitions, but no concrete scene immediately came to mind.

I wondered whether continuing to read the latest papers would eventually make everything clear. Predictably, there was no such shortcut. Reinforcement learning does not reinforce its reader. I had to begin again from the foundations.

This series uses David Silver's UCL Reinforcement Learning course as its main structure, together with the Seoul National University DSBA Lab seminar introducing reinforcement learning and Sutton and Barto's Reinforcement Learning: An Introduction. Rather than translating the lectures as-is, my aim is to unpack, one step at a time, the points where I personally became stuck while reading the equations.

The series therefore moves slowly. We first understand a situation through an example, identify the symbols in a figure, and only then read the equation. This first article introduces only the most basic participants in reinforcement learning.

• Who are the agent and environment?
• What are states, actions, and rewards?
• Why do we add future rewards instead of looking only at the one directly in front of us?
• Why are exploration and exploitation both necessary?

The Bellman equation and Q-Learning do not appear yet. Memorizing a pile of names in a hurry makes the equations look even more alike later.

How Is Reinforcement Learning Different?

Reinforcement learning is ultimately a form of machine learning. The way it presents a problem, however, differs considerably from familiar supervised learning.

Consider a classification task that distinguishes cats from dogs. In supervised learning, each image comes with its correct label. If the model calls a cat a dog, we can immediately compare its prediction with the answer and calculate how wrong it was. It is like a workbook with an answer key in the back.

In reinforcement learning, no one supplies the correct action at every moment. An agent placed in the middle of a maze is not told “move right now, then move up.” Instead, it receives points for reaching the exit and loses points for hitting a wall. From those scores, it must infer whether its recent choices were good.

David Silver's first lecture summarizes four distinguishing features of reinforcement learning.

1. There is no supervisor, only a reward signal.
2. Feedback can be delayed rather than immediate.
3. The temporal sequence matters.
4. The agent's actions change the data it will observe later.

The final point is especially important. In ordinary image classification, misclassifying the first image as a cat does not change the content of the second image. In a game, moving left once changes the next screen itself. The data in reinforcement learning is not a static dataset. It is experience continually created by the agent's behavior.

Agent and Environment

The terminology may appear to multiply suddenly, but the problem becomes simple if we divide reinforcement learning into “the part that acts” and “the rest of the world that responds.”

The agent is the decision-maker. In a game, it is the algorithm controlling the character. For a robot, it is the controller that moves the motors. The environment is everything outside the agent: the game's map and rules, the floor beneath the robot, gravity, obstacles, and so on.

At every time $t$, the two interact in the following order.

$$S_t \;\xrightarrow{\;A_t\;}\; \text{Environment} \;\longrightarrow\; (R_{t+1}, S_{t+1})$$

Rather than skipping over the equation, read it once as a sentence.

The agent observes the current state $S_t$ and selects action $A_t$. After receiving that action, the environment returns reward $R_{t+1}$ and next state $S_{t+1}$.

The symbols can be summarized as follows.

The reward and next state use the subscript $t+1$ because the action occurs first and its results arrive afterward. Textbooks sometimes index these quantities differently, which can be surprisingly confusing at first. The important part is not the index itself, but the sequence: act in the current state, then receive the next state and reward.

Can Choosing Lunch Be a Reinforcement-Learning Problem?

Let us force the choice of a lab lunch into a reinforcement-learning formulation. Suppose it is raining, 25 minutes remain for lunch, and we ate gukbap yesterday.

• State: weather, remaining time, current location, and recently eaten meals
• Action: choose the gukbap restaurant, a newly opened restaurant, or a convenience store
• Reward: satisfaction combining taste, price, and waiting time
• Next state: time, location, and fullness after lunch

There is no correct label saying “go to the gukbap restaurant.” A nearby restaurant may be better in the rain, or an unexpectedly long line may lead to a low reward. The outcome of the same action depends on the state.

Of course, quantifying lunch this carefully may take until the end of the workday. The point is that reinforcement learning is not a problem of guessing a labeled correct action. It is a problem of learning what makes an action good through interaction.

Reward Is Not an Answer Key

A reward is scalar feedback telling the agent how well it has just done. “Scalar” sounds grand, but here it simply means one number.

For example, a maze might define rewards as follows.

• $+10$ for reaching the exit
• $-3$ for hitting a wall
• $-1$ for every movement

This causes the agent to prefer reaching the exit by a short path. Even if both paths eventually receive the same $+10$, the shorter one has the greater final total because each step costs $-1$.

A reward, however, does not directly say “this action was correct.” Receiving $-1$ immediately after moving left does not necessarily make left a bad action. That step may be the only route to the exit.

This is where reinforcement learning becomes difficult. A small immediate reward may lead to a larger reward later, while an action that looks good now may reach a dead end after several steps. The agent's goal is therefore not to maximize the reward at every moment, but to maximize the sum of future rewards.

David Silver's lecture expresses this through the reward hypothesis: a goal can be described as maximizing expected cumulative reward. This is a strong claim, and whether every human objective can truly be represented by one number is a separate philosophical question. To treat reinforcement learning mathematically, however, we begin from this assumption.

Return: Viewing Future Rewards Together

The sum of rewards to be received after time $t$ is called the return and is usually written as $G_t$.

In its simplest form,

$$G_t = R_{t+1} + R_{t+2} + R_{t+3} + \cdots$$

In a very long or non-terminating environment, adding rewards directly can make the sum grow without bound. A reward in the distant future is also uncertain because the agent may never reach it. We therefore usually use a discount factor $\gamma$.

$$G_t = R_{t+1} + \gamma R_{t+2} + \gamma^2 R_{t+3} + \cdots$$

Read the symbols one at a time.

• $G_t$: the total discounted reward from now into the future
• $R_{t+1}$: the reward at the very next step
• $\gamma$: the proportion of a future reward retained from the present perspective
• $\gamma^2 R_{t+3}$: the third reward after two discounts

$\gamma$ is usually between $0$ and $1$.

$$0 \leq \gamma \leq 1$$

When $\gamma=0$, only the next reward matters, producing a rather impatient agent focused on immediate gain. As $\gamma$ approaches $1$, distant rewards receive nearly the same weight as immediate ones.

Let us substitute numbers. If the next rewards are $+2$, $0$, and $+5$, in that order, with $\gamma=0.9$, then

$$\begin{aligned} G_t &= 2 + 0.9 \times 0 + 0.9^2 \times 5 \\ &= 2 + 0 + 4.05 \\ &= 6.05. \end{aligned}$$

The third reward of $+5$ has not disappeared. It is two steps from the present, so two factors of $0.9$ reduce its contribution to $4.05$.

The subscripts and powers can make the equation look unnecessarily complex at first. In practice, it simply adds a chronological receipt: the reward one step away is unchanged, the reward two steps away receives one $\gamma$, and the reward three steps away receives two.

Why Not Keep Making Only the Good Choice?

Suppose the agent has eaten lunch several times and learned that its usual gukbap restaurant gives a high reward on average. It may seem best to visit only that restaurant from then on.

The problem is that a newly opened restaurant may be much better. Having never visited it, the agent does not know its reward. Repeating a choice already known to be good is exploitation. Trying a new choice to acquire information is exploration.

With exploitation alone, the agent can keep using the best behavior found so far but may never discover something better. With exploration alone, it gathers plenty of information but may gain little reward because every choice is an experiment.

The example resembles a lecture because David Silver's first lecture also explains the same problem through restaurant selection. Visiting a favorite restaurant is exploitation; trying a new one is exploration. In reinforcement learning, this small lunchtime dilemma becomes fundamental. The agent must try unknown actions to learn, while avoiding excessive loss of reward during that learning.

We will return to concrete ways of balancing the two in the articles on multi-armed bandits and $\epsilon$-greedy. For now, it is enough to remember that “always select the action that currently looks best” is not necessarily optimal.

A Policy Is the Agent's Habit of Behavior

The rule by which an agent selects an action from a state is called a policy and is written as $\pi$.

A deterministic policy directly chooses one action for each state.

$$A_t = \pi(S_t)$$

A stochastic policy gives the probability of selecting each action in the current state.

$$\pi(a|s) = P(A_t=a \mid S_t=s)$$

Read the second equation as

The probability of choosing action $a$ when the state is $s$.

If 25 minutes remain for lunch and the probabilities of selecting the gukbap restaurant, new restaurant, and convenience store are $0.7$, $0.2$, and $0.1$, respectively, those numbers are part of the current policy.

The goal of reinforcement learning is ultimately to find a good policy. More precisely, we want a policy whose expected return is as large as possible.

$$\pi^* = \arg\max_{\pi} \mathbb{E}_{\pi}[G_t]$$

It is fine if $\arg\max$ and expectation are not yet familiar. In words, the equation says, “find the policy $\pi^*$ whose average return is greatest among all possible policies.” Dynamic programming, Q-Learning, and policy gradient look different, but they are different routes toward this same objective.

Distinguishing Reward, Return, and Value

In reinforcement-learning writing, reward, return, and value can appear to blur together. The first article need not define all three perfectly, but it is important to see that they are not synonyms.

A reward is the one-step number actually returned by the environment. A return is the result of combining several rewards over time. A value goes one step further and predicts a future return that has not yet occurred.

Reinforcement-learning equations become suddenly more complicated in later articles largely because of this “quantity that predicts the future.” We cannot directly know the return of a path we have not taken, so we must estimate its value from experience and probability.

What to Remember

The central idea of this article is

Reinforcement learning is the problem of learning a policy that maximizes return, the sum of future rewards, while an agent interacts sequentially with an environment.

More specifically:

1. The agent observes a state and selects an action.
2. The environment returns the next state and a reward.
3. An action changes even the data the agent will observe next.
4. The objective is not the immediate reward, but the return formed by future rewards.
5. Finding a good policy requires exploration as well as exploitation.

So far, we have only examined the grammar of reinforcement learning. The next article begins with the question, “is knowing the current state really enough?” It distinguishes history, observation, and state, and introduces the Markov property. Nearly every reinforcement-learning equation begins from $S_t$, so we will first ask which parts of the past that state must contain.

References

• David Silver, UCL Reinforcement Learning Course
• David Silver, Lecture 1: Introduction to Reinforcement Learning slides
• Google DeepMind, RL Course by David Silver - Lecture 1
• Seoul National University DSBA Lab, Introduction of Reinforcement Learning
• Sutton & Barto, Reinforcement Learning: An Introduction, 2nd Edition