This article translates and adapts Hugging Face's “Illustrating Reinforcement Learning from Human Feedback (RLHF)” so that readers without much deep-learning background can follow it. The original post is available here.

Lambert, et al., "Illustrating Reinforcement Learning from Human Feedback (RLHF)", Hugging Face Blog, 2022.

Is This an AI Boom?

In Korea, public interest in AI surged after AlphaGo and then gradually faded. Meanwhile, expressions such as “the Fourth Industrial Revolution” and “big data” began appearing on every kind of platform. (To be candid, I dislike it when people who have never properly studied deep learning or machine learning insert “artificial intelligence” into business pitches.)

Then ChatGPT reignited interest, this time in natural-language processing, and began drawing public attention.

As computer science entered the spotlight in Korea, there was a tendency to dress business platforms in impressive-sounding terms such as AI and the metaverse without understanding the underlying principles. There were certainly legitimate startups and companies, but many failed to offer a clear technical solution.

Even so, the keyword “artificial intelligence” was enough to excite people. At the same time, services with no real connection to deep learning—even ones feasible as undergraduate toy projects—began appearing on the web. Some services did genuinely apply machine learning, of course, but it was hard to call them broadly useful across social and cultural groups.

ChatGPT, the subject of this post, is a representative example that pushes marketability and practicality far beyond earlier AI businesses. As some readers will know, Stable Diffusion, released by Stability AI, was another large-scale model that reached the public as a form of content before ChatGPT. Its utility differs, however, from ChatGPT's ability to search, tell stories, and answer questions. We will have to see whether LLMs such as ChatGPT can completely overturn the previous trend of relying on search engines such as Google.

Introduction

As deep-learning researchers will know, language models for natural-language processing (NLP) have advanced dramatically in recent years. The Transformer paper, “Attention Is All You Need,” proposed a new model for machine translation, and models built on it—such as GPT and BERT—began to diversify and improve.

Broadly speaking, GPT focuses on “generating” sentences, whereas BERT focuses on “understanding” the context of a given sentence. The GPT model underlying ChatGPT is therefore research centered on the process of generating plausible sentences.

What Is a Plausible Sentence?

Computer algorithms are built from mathematics. Generating a sentence ultimately means computing over a complex collection of numbers, yet the way humans define “good text” is subjective and can vary with the context in which the sentence is generated.

GPT may need to write a creative novel, an essay, a paper abstract, or even part of a code algorithm for a developer. A deep neural network—the entity being trained—consists of many parameters, or collections of numbers. To set those parameters so that they generate plausible sentences, we need an objective function that represents that purpose. In machine learning, this is the problem of choosing a loss function or objective function.

Most language models use an objective with a trivial solution, such as cross-entropy, a categorical function that predicts the next word from preceding text. Here, a trivial solution means a prediction such as, “In the training data, this word often appeared in this sentence structure.” We cannot be sure that this properly fits situations requiring contextual understanding. To mitigate such problems, optimization began using metrics such as BLEU and ROUGE, which reflect human preferences—subjective human judgments—about generated sentences to some extent.

$$\begin{cases} \text{ROUGE}, &(w_{ref} \in S_{gen} \vert w_{ref} \in S_{ref}) / \vert S_{ref} \vert \newline \text{BLEU}, &(w_{gen} \in S_{ref} \vert w_{gen} \in S_{gen}) / \vert S_{gen} \vert \end{cases}$$

In short, ROUGE measures the proportion of words $w_{ref}$ in the reference sentence $S_{ref}$ that appear in the generated sentence $S_{gen}$. BLEU measures the proportion of words $w_{gen}$ in the generated sentence that appear in the reference sentence. Even when used as subjective metrics, they therefore measure only how many words overlap; they cannot determine whether each generated word appears in the proper context or position. This exposes a limitation in the optimization process.

RLHF Enters the Scene

That suggests another idea. Although it would require more work, if people could directly provide feedback on generated sentences and we could measure performance numerically from that feedback, perhaps we could adjust and optimize a language generator's parameters more delicately. This is the idea behind Reinforcement Learning from Human Feedback (RLHF), the method used to train ChatGPT.

RLHF treats human feedback as a kind of penalty or reward for reinforcement learning and optimizes accordingly. It allows an LLM (Large Language Model) to learn to generate sentences closer to those a person would produce. But can ChatGPT properly explain the RLHF with which it was trained?

ChatGPT apparently decided that adding exclamation marks was the key to explaining something to a child. As it says, the basic idea of RLHF is to use reinforcement learning to train in a faster and more accurate direction. The discussion so far has built up an intuitive explanation; now let us examine the specific training process.

Understanding RLHF Step by Step

I do not fully understand reinforcement learning myself, so I feel a little guilty writing this. Still, I wanted to translate the Hugging Face post into a friendlier form. RLHF training consists of several pipeline stages:

• Pre-train a Language Model (LM)
• Collect data and train a reward model
• Fine-tune the LM with reinforcement learning

Let us examine each stage in detail.

Pretraining language models

RLHF requires a language model that can receive feedback. A language model is trained with a conventional pretraining objective, which can include cross-entropy loss, BLEU, or ROUGE as discussed above. OpenAI, for example, used a smaller model from the GPT-3 family to create InstructGPT, its first model trained with RLHF. Anthropic, which launched Claude, applied RLHF to pretrained Transformers ranging from $10 \times 10^6$ to $52 \times 10^9$ parameters, while DeepMind used its Gopher as the pretrained LM.

This initial model may itself be fine-tuned. OpenAI, for example, fine-tuned on “preferable” prompts written directly by people. Anthropic guided the initial LM for RLHF with context expressing its “helpful, honest, and harmless” criteria. Both approaches effectively use a curated dataset. Because constructing a high-quality dataset is generally expensive, however, this fine-tuning stage is not mandatory for RLHF.

There is no definitive answer as to which model is the best starting point for reinforcement learning. Many companies train large language models, but no study has compared the performance of every network architecture in the RLHF design space. In other words, methods for “reinforcing a language model” with RLHF are still discussed without a clear account of architecture-dependent performance differences.

Reward model training

To evaluate a language model, we need data produced by the model. To build a language model that generates text people actually prefer, we need a reward model (RM) that turns human feedback into a reward based on how well the sentence was written.

More specifically, we want a reward system or model architecture that takes a sequence of text emitted by the LM and returns a scalar reward—a score—for human preference toward that generated text. For example, the reward model ranks a piece of text and emits a corresponding scalar reward.

The RM's scalar output is an essential requirement of the RL algorithm. Put simply, defining a good criterion for turning human preference into a score is what allows RLHF—using human preference to reinforce a network—to fit naturally into an RL algorithm.

Because reward modeling ultimately quantifies rankings over text, it can be understood as training another language model on preference data—either from scratch or by fine-tuning a pretrained LM. Anthropic, for example, applied a special fine-tuning method called preference model pretraining (PMP) to a pretrained LM and reported that it was sample-efficient, meaning it could achieve the desired behavior with relatively few samples. But just as research has not identified the most effective LM architecture for RLHF, there is still no definitive answer for training an RM effectively.

Training the RM requires a dataset of text that a trained network generated from specific prompts—prompt-generation pairs—sampled from an existing dataset. Anthropic generated text from prompts collected through a chat tool on Amazon Mechanical Turk, while OpenAI reportedly used prompt submissions from users of its GPT API. (Nothing is really free, after all.)

These LM outputs are given to human annotators, who apply their own criteria and rank the generated text. One might ask, “Why not skip ranking, assign each text a scalar score directly, and dispense with the reward model?” In practice, that is difficult. As a human annotator viewing several responses to a prompt, it is hard to measure exactly how much better one sentence is than another. Clear quality gaps may be easy to score, but consistently distinguishing values within a fixed range—for example, $0 \sim 10$—creates calibration and noise problems. Instead of assigning absolute scores, annotators compare texts from several models to construct relative rankings, thereby normalizing the dataset.

There are many ways to rank text. The most effective has been to compare the outputs of two different LMs on the same prompt. This head-to-head choice of a winner and loser resembles a chess match. Comparing outputs against other LMs lets us express their relative positions with the Elo system, a rating method used not only in chess but in many sports. In short, ratings rise or fall according to predicted win probability: if an underdog defeats a favorite, it receives a larger bonus. Other ranking methods exist, of course. The scores used for training are normalized from the resulting labeled rankings.

The reward model has an intrinsic limitation: in an RLHF system, its size must grow to some extent with the language model being trained. For example, OpenAI used a $6 \times 10^9$-parameter RM for a $175 \times 10^9$-parameter LM, while Anthropic and DeepMind tended to scale LM and RM size proportionally. A reward model must ultimately have capacity comparable to the language model in order to provide effective feedback on its performance.

This means the RM must be retrained whenever the target LM changes. A pretrained RM cannot simply be reused across multiple LMs, and this also prevents rigorous comparisons among model architectures. Intuitively, it makes sense that understanding the representations of a language model with more parameters requires an RM with correspondingly greater representational capacity.

This can be compared to the ability of tutors teaching top-ranked students versus lower-performing students. A lower-performing student—a network with fewer parameters—may not require an exceptional tutor to evaluate and teach them. A top-ranked student—a network with many parameters—would require an excellent tutor. If academic and teaching ability stand in for representation, it becomes intuitive that as a model grows larger, its reward system or model must grow with it.

We have now covered pretraining the LM and constructing the RM. We have both a model that generates text from a prompt (the LM) and a model that maps human preference for that text to a scalar value (the RM).

Fine-tuning with Reinforcement Learning

Training a large language model with reinforcement learning was once close to algorithmically infeasible. Unlike backpropagation, which enables direct optimization through differentiation, reinforcement learning quantifies how “good” an agent's decisions are in a given situation. Training the deep neural networks used in deep learning means working with complex functions, so the task was difficult both algorithmically and from the engineering perspective of designing the training pipeline.

Several organizations addressed this by using Proximal Policy Optimization (PPO), an RL policy-gradient algorithm (reference), to fine-tune some or all parameters of a copy of the initial LM. Because LMs are enormous, ranging from roughly $10B$ to more than $100B$ parameters, fine-tuning every parameter is inefficient. Inspired by research suggesting that high-dimensional parameters actually represent a manifold of low intrinsic dimension, the Low-Rank Adaptation (LoRA) paper (reference) demonstrated a computationally efficient fine-tuning method.

DeepMind's Sparrow model (reference), for example, built a pipeline that gradually improves the RL policy by bootstrapping data from preferred prompt responses and negative responses.

I will not explore each of these topics in greater detail, but interested readers may find the linked material worthwhile.

PPO is widely used in RL and mature enough to have many guides and tutorials. Because it is relatively well understood, it could be applied to RLHF with a scalable approach. RL in RLHF ultimately answers how a large language model can be optimized using a familiar algorithm.

Using the figure above, we can formulate the fine-tuning task as an RL problem. First, the policy is the language model, which produces a sequence of text in response to a prompt. During LM fine-tuning, the parameterized network is the policy's underlying agent, while the network's output—its text sequence or probability distribution over text—can be regarded as the policy. Its action space consists of every token in the LM's vocabulary. The observation space is the environment in which the agent makes decisions. Because no conventional environment can be defined for an LM undergoing RL training, it can simply be understood as the set of possible input token sequences. More precisely, the observation space is the distribution over input token sequences. It is enormous compared with ordinary RL; considering all possible word combinations, its rough size is

$$\text{vocabulary size}^\text{token length of the input sequence}$$

More varied observations mean greater diversity in the text the model can generate. The reward function consists of the preference model—an evaluation metric for the output—and a constraint on policy shift, or how the policy is updated.

The reward function integrates all the models and training pipelines discussed so far into the RLHF process. Suppose a prompt $x$ from the dataset produces text $y_1$ under the initial language model and text $y_2$ under the policy updated at the current iteration. Passing $y_2$ through the preference model yields a scalar value $r_{\theta}$ that reflects “preference,” as described earlier. This text is compared with $y_1$ from the initial model, and their difference is used to compute a penalty. Most papers from OpenAI, Anthropic, and DeepMind use a scaled Kullback–Leibler (KL) divergence, a standard metric for distributional distance based on the degree of overlap between distributions. The KL-divergence term $r_{\text{KL}}$ is computed between distributions over successive tokens.

The KL-divergence metric appears not only in RLHF but throughout tasks and network architectures involving distributions. In most distribution-optimization tasks, training minimizes KL divergence to align an encoder or decoder output or feature map with an ideal distribution. In RLHF, by contrast, it serves as regularization: the text generated by the initial model and the policy-optimized model should differ, but remain an appropriate distance apart. Without this regularization term, the policy could earn a high reward simply by generating bizarre text that fools the preference model (see the figure above). The reward used by the RL update rule therefore follows

$$r = r_{\theta} - \gamma r_{\text{KL}}$$

The objective maximizes reward, but reduces it if the policy strays too far from the distribution of text sequences produced by the initial LM.

Some RLHF frameworks add another term to this reward function. OpenAI's RLHF-trained InstructGPT, for example, mixes in pretraining gradients as follows.

$$L(\phi) = E\_{(x, y)\sim D\_{\pi_\phi^{\text{RL}}}} (r_\theta (x, y) - \beta \log (\pi_\phi^\text{RL}(y \vert x) / \pi^\text{SFT} (y \vert x))) + \gamma E_{x \sim D_\text{pretrain}} (\log (\pi_\phi^\text{RL} (x)))$$

Here, “pretraining” refers to the gradient over a human-annotated set. The RLHF reward function is thus an area that still requires substantial research.

Finally, parameters are updated according to an update rule that maximizes the reward metric for the current data batch. PPO is called on-policy because parameters are updated using only the current batch. PPO follows an algorithm called trust-region optimization. In simple terms, unconstrained and erratic gradient updates add noise to learning, so deliberate constraints help stabilize training. DeepMind used a similar reward setup for Gopher, but differed in optimizing gradients with synchronous advantage actor-critic (A2C) (reference).

RLHF can also update the reward model and policy together. After the RL policy is updated, users can rank its output by comparing it with the initial model's output. Because this demands continual user involvement, it is laborious, and few papers have studied the procedure.

Anthropic calls this optional training method Iterated Online RLHF (reference); the policy at each iteration becomes a competitor in an Elo ranking system.

Open-Source Tools for RLHF

The preceding sections cover the core of RLHF. This section introduces tools for RLHF training. OpenAI released one of the first codebases applying RLHF to an LM in TensorFlow in 2019 (lm-human-preferences on GitHub).

Repositories for long-suffering PyTorch users also began to appear. Representative examples are Transformers Reinforcement Learning (TRL, TRLX) and Reinforcement Learning for Language Models (RL4LMs).

TRL was created to fine-tune pretrained LMs from the Hugging Face ecosystem with PPO-based reinforcement learning. TRLX, a fork of TRL, is an API for applying PPO- or ILQL-based (Implicit Q-Learning) RLHF to LLMs. TRLX cannot yet handle LLMs with very large parameter counts, but is expected to add support for scaled-up pretrained models.

RL4LMs supports a broader range of RL algorithms (PPO, NLPO, A2C, and TRPO), reward functions, and metrics for fine-tuning LLMs. It is also easy to customize: users need only define a reward function to apply it to any Transformer-based encoder or decoder. Both TRLX and RL4LMs are actively used in experiments and are expected to gain additional features.

Closing Thoughts

RLHF has demonstrated strong performance and begun to influence AI research substantially, but it still has clear limitations. Although model performance improves, models continue to generate harmful and factually inaccurate text. For example, they can produce the following nonsense extremely quickly.

Addressing these shortcomings will be central to improving future LMs. RLHF is not perfect, and this points to the direction in which services such as ChatGPT must evolve.

In an RLHF system, collecting human preference data happens outside the training loop and is therefore labor-intensive. The upper bound of RLHF performance is also inevitably constrained by the quality of human annotation: text written by people and preferences judged by people.

Producing well-written human text for a particular prompt is difficult and expensive. Fortunately, the data scale needed to train a reward model is not prohibitively costly, perhaps because annotation requires less specialized expertise than image-dataset annotation. Even so, the cost is far from negligible for an ordinary research lab, making RLHF research difficult outside large companies. To my knowledge, Anthropic provides the only publicly available large-scale dataset. Even after paying to acquire a dataset, another problem remains: human annotators' subjective opinions conflict. Preferences without a single correct answer inevitably produce noisy labels with variance in rankings.

RLHF therefore has clear limitations: it is difficult to run ablations across diverse network architectures and loss functions, and the field is inaccessible to broad exploration across research-lab settings. One major issue is its reliance on PPO, an older RL optimizer. Demonstrating that another algorithm is more efficient than PPO requires an alternative that can accelerate today's RLHF research, leaving several problems tightly entangled. In the RL framework, the environment corresponds to the reward model's output, so its computational cost is unavoidable as well. To reduce the cost of online RL, researchers have recently explored offline RL methods such as ILQL as optimization tools. Other mechanisms, such as RL's exploration–exploitation balance—the trade-off between diversity and efficiency—have not yet been fully applied. It remains to be seen how successfully adapting these reinforcement-learning algorithms to RLHF will affect LLM performance.