How Are LLMs Trained? — From Pre-training to Distillation and RL

How LLMs Are Trained — 1. Datasets · 2. Modeling · 3. Sovereign AI

LLM training is often summarized in one sentence: “predict the next token.” The principle is correct, but it does not quite explain how a model reaches a production service. Pre-training builds a linguistic foundation from a large corpus; SFT teaches instruction following; Preference Optimization incorporates human preferences; and reinforcement learning teaches the model to search for answers. Each has a different objective and uses different data. More recently, Distillation, which transfers the capabilities of a large Teacher to a smaller Student, has also entered this process. Rather than one long training run, it is closer to a chain of training stages with different goals, each inheriting the previous checkpoint.

Pre-training: Building a Foundation by Predicting the Next Token

The objective function for pre-training is surprisingly simple. Given the preceding tokens, the model minimizes Cross Entropy so that the probability of the next token increases.

$$\mathcal{L}_{PT} = -\sum_t \log p_\theta(x_t \mid x_{<t})$$

A simple objective does not make for a simple project. You have to choose the tokenizer vocabulary, update billions of parameters stably from random initialization, and operate Data Parallelism alongside Tensor, Pipeline, and Expert Parallelism. The learning rate, warmup, number of tokens per batch, sequence length, weight decay, and gradient clipping all interact. Even a small instability becomes enormously expensive over trillions of training tokens. Llama 3 disclosed a dataset of roughly 15T tokens along with its training and post-training recipe. OLMo and OLMo 2 released data, code, checkpoints, and evaluations together, illustrating the difference between publishing model weights and opening the training process itself.

The Base Model produced by pre-training can continue text naturally, but it is not yet an Assistant that follows user requests reliably. It has encountered question-answer formats as part of web documents, but it has no clear priorities around usefulness or safety. Continued Pre-training is used here to increase the weight of a particular domain or language without relearning the entire data distribution. Mid-training for genuinely long contexts and stages that increase the share of recent documents or code fall into much the same category. Training too long on a small specialist corpus, however, can cause catastrophic forgetting of general capabilities, so teams may mix in general data or lower the learning rate.

SFT: Knowing the Answer Is Different from Answering as Requested

Supervised Fine-Tuning connects a question or instruction with a reference response. A common implementation applies loss only to the Assistant tokens that the model must generate; System and User messages provide context but are not reproduced as targets. At this stage, the model learns behaviors such as conversation format, output structure, tool-call syntax, and refusal style.

SFT is straightforward to implement, stable, and capable of changing behavior substantially with relatively little data. Conversely, it imitates the data's mistakes as they are. If every example is a verbose answer from a Teacher, the Student also becomes needlessly long. If the dataset contains only correct answers, the model has little opportunity to learn how to recover from failure or express uncertainty. Recent SFT datasets therefore sometimes include reasoning traces, critiques, tool calls, and self-corrections as well as final responses. Longer written reasoning is not automatically better, however; the verifiability and cost of the final answer still matter.

Distillation: Transferring a Large Model's Distribution to a Smaller One

Knowledge Distillation transfers knowledge from a large Teacher to a Student. The original work on Knowledge Distillation trained the Student on hard labels and the softer probability distribution produced by the Teacher. In an LLM, storing logits over the entire vocabulary is expensive, so Sequence-Level Distillation, which trains on sequences, explanations, reasoning, or critiques generated by the Teacher, is also widely used. Research such as Distilling Step-by-Step has further shown that distilling an answer together with its rationale can improve a smaller model with fewer examples.

Off-policy Distillation

Off-policy Distillation means that the output sequences used for training were not generated by the current Student policy. The model learns Teacher token probabilities or generated outputs along trajectories drawn from outside the Student's distribution: human-written ground truth, Teacher-generated sequences, or responses saved by an earlier checkpoint. In practice, Teacher outputs are often generated in advance and used as a fixed dataset, so Off-policy can look synonymous with Offline, but the terms describe different axes. Offline emphasizes when the data was prepared; Off-policy identifies which policy produced its distribution. Even if a Teacher is called live during training, the procedure remains Off-policy when it uses the Teacher's sequence or a fixed target rather than a trajectory produced by the current Student.

Off-policy methods are easy to implement and highly reproducible because a Teacher API can be called once and its results reused. They are especially practical for expanding prompts or generating solutions at multiple difficulty levels in domains short on human data. This is also why most teams begin by creating synthetic data with a large Teacher and fine-tuning a smaller model on it.

The problem is that the Student's actual error distribution differs from the distribution of data generated in advance by the Teacher. Easy problems the Student already handles may be repeated, while the moments when the Student goes badly off course may not exist in the fixed dataset at all. The Teacher's hallucinations and style become frozen into the dataset, and without records of the Teacher version and prompt used for generation, tracing the origin of an error later is difficult.

On-policy Distillation

On-policy Distillation updates the Student to follow the Teacher's token-level distribution over sequences generated by the current Student itself. It does not mean that the Teacher's weights are trained with the Student, nor does merely calling a Teacher API in real time make a method On-policy. What matters is which policy generated the trajectory. Generalized Knowledge Distillation has the Teacher provide feedback on error sequences actually generated by the Student, reducing the distribution mismatch between fixed Teacher or ground-truth sequences and the Student's sequences at inference time.

This focuses training on the areas where the Student is actually making mistakes. The tradeoff is that every training iteration now includes Student generation and Teacher inference, increasing cost and latency and requiring a system that connects Student generation, Teacher evaluation, and updates reliably. If the Teacher is inconsistent or sensitive to its prompt, the target itself moves. On-policy is therefore not always superior to Off-policy. It makes sense when closing a distribution gap left by fixed synthetic data is worth the added cost. Mixed-policy designs that combine both distributions in a chosen ratio are also possible.

Preference Optimization: Choosing the Direction of a Good Answer

SFT teaches the model to imitate a reference answer, but many real questions do not have one correct response. Whether a friendly but verbose answer is better than a brief, direct one depends on the service. Preference training compares several responses to the same question and adjusts the model's behavioral criteria.

The traditional RLHF pipeline popularized by InstructGPT first creates an SFT model, trains a Reward Model on human-selected chosen/rejected pairs, and then optimizes the policy with PPO. The model explores answers that earn higher rewards while a KL penalty keeps it from moving too far from a reference model. Reusing the Reward Model avoids asking people to score every new answer, but the Policy, Reference, Reward, and Value models must all be operated together, increasing memory use and implementation complexity. Reward hacking can also appear: an answer that exploits a weakness in the Reward Model receives a higher score than one that is genuinely better.

Direct Preference Optimization optimizes the same preference objective directly, increasing the probability of chosen responses and decreasing that of rejected ones without a separate Reward Model or online rollouts. Its simple, stable implementation has made it common in open-model post-training, but it has limited ability to explore strategies outside the range of already collected preference pairs. Strictly speaking, DPO is better understood as preference optimization derived from reinforcement learning; it is not an Online RL loop that gathers new samples through interaction with an environment.

Reinforcement Learning and Exploration

For mathematics, code, and structured output, where results can be checked automatically, correctness, a compiler, unit tests, or a theorem verifier can replace human preference judgments as the reward. This is called RL with Verifiable Rewards, or RLVR. The model generates several solutions to the same problem and raises the probability of successful paths, exploring strategies absent from SFT data. The reward is clearer than one produced by a Reward Model approximating human taste, but the approach applies directly only to verifiable problems. It can also learn shortcuts whose final answer happens to be right even though the process is wrong.

GRPO, introduced in DeepSeekMath, generates several responses for each prompt and uses relative rewards within the group as a baseline, reducing the burden of a separate critic model. Its memory architecture is simpler than PPO's, which has made it popular for LLM reasoning training, but rollout costs do not disappear. If every response in a group is wrong or nearly identical, the learning signal is weak. A poorly designed reward can optimize the model toward merely matching a format or producing unnecessarily long reasoning.

RL is not a universal stage for injecting facts into a model, either. A model missing current knowledge is better served by improved data or retrieval than by rewards alone, while SFT is cheaper and more predictable for stabilizing tone and format. Reinforcement learning is particularly effective where the answer is verifiable but writing down every good solution path as training data is difficult.

Combinations Commonly Used in 2026

Not every team pre-trains a Foundation Model from the beginning. Large model-development organizations typically follow pre-training with Continued Pre-training on high-quality data and context extension, then use SFT to establish the Assistant's basic behavior. Preference Optimization and safety training follow, with Online RL or RLVR added for areas where rewards are verifiable, such as mathematics, code, and tool use. If a smaller deployment model is needed, the strong model produced by this process becomes the Teacher for Distillation.

For organizations with limited GPUs and domain data, the most practical combination is often to start with an open-weight Base Model, perform Continued Pre-training or LoRA/SFT, and use synthetic data from a large Teacher for Off-policy Distillation. An Offline implementation with Teacher outputs generated in advance is common. DPO can be added when human comparison data is available, while Online RL is reserved for core functions with clear rewards and enough value to justify the cost. A few common recipes look like this:

1. General-purpose Assistant: Base Model → SFT → DPO → Safety/Evaluation
2. Specialist domain model: Base Model → Continued Pre-training → SFT → Domain Evaluation
3. Small on-device model: Synthetic or logit data from a large Teacher → Distillation → Quantization
4. Mathematics and code Reasoning model: SFT → Verifiable rollout → GRPO/PPO-family RL → Contamination-Free Evaluation
5. Agent model: Tool-use SFT → Execution-based Preference/RL → Replay of failures from the real environment

Before asking which training method is best, I find it more useful to ask what the current model lacks. Does it lack knowledge, fail to follow instructions, choose the worse of several answers, or fail to explore for a solution? Each failure calls for a different stage. Pre-training establishes the range of possibilities; SFT sets default behavior; preference training adjusts direction; and RL teaches exploration toward a verifiable objective. Distillation transfers the capabilities created along the way into a smaller model or a different architecture. Stacking similarly named techniques matters less than stating which failure each stage should solve and checking whether that failure actually decreases in evaluation.

Previous: What Do LLMs Feed On? — Building and Acquiring Datasets
Next: Whose Model Can We Call Our AI? — Sovereign AI in South Korea