Autoregression: Why LLMs Answer One Token at a Time

A typical LLM is autoregressive. It computes a probability distribution for the next token conditioned on everything generated so far, selects one, and repeats.

Suppose an answer reads:

Speculative Decoding makes LLM output faster.

The model does not generate the whole sentence at once.

1. It generates Speculative.
2. Given Speculative Decoding, it generates makes.
3. Given the text so far, it generates LLM.
4. It examines the full context again and generates the next token.

The probability of a length-$T$ answer factors as follows.

$$p(x_{1:T}) = \prod_{t=1}^{T} p(x_t \mid x_{<t}).$$

The problem is that $x_t$ must be known before $x_{t+1}$ can be computed. A 100-token answer requires at least 100 sequential target-model calls.

A KV cache avoids recomputing keys and values for previous tokens, but every new token must still pass through every layer of a model with billions of parameters.

At small batch sizes, LLM decoding is often limited by GPU memory bandwidth, not arithmetic. It is slow because generating every token repeatedly reads enormous weights from GPU memory.

This raises a question:

If the weights must be read anyway, why compute only one next token? Can we check several tokens at once?

This is the idea behind Speculative Decoding.

Speculative Decoding

“Speculative” means based on a guess. The method predicts several future tokens in advance, and the target checks those guesses together.

Two models participate:

• The target model is the large, expensive LLM that normally generates the answer.
• The draft model is a small model that quickly proposes likely future tokens.

Let draft model $q$ and target model $p$ receive prefix $x_{1:t}$. The process is as follows.

1. Draft

The small draft first generates the next $k$ tokens autoregressively.

$$\hat{x}_{t+1},\hat{x}_{t+2},\cdots,\hat{x}_{t+k} \sim q.$$

It still generates sequentially, but it is much smaller than the target, so this step is relatively cheap.

2. Verification

Feed the original prefix and draft tokens to the target together.

$$[x_{1:t},\hat{x}_{t+1:t+k}]$$

During training, a Transformer uses a causal mask to compute next-token distributions at multiple positions in parallel. The same principle lets the target evaluate all drafted positions in one forward pass.

3. Accept or Reject

Inspect proposals from left to right.

• Accept draft tokens that agree with the target distribution.
• At the first rejection, discard every later draft token.
• Resample the rejected position using a correction based on the target distribution.
• If all draft tokens are accepted, also take one additional token already computed by the target.

In simple terms, the draft quickly writes a sentence, and the target reviews it all at once with a red pen.

If the draft is accurate, one target call can commit several tokens.

$$\text{Vanilla AR} : 1\text{ Target Forward} \rightarrow 1\text{ Token}$$ $$\text{Speculative Decoding} : 1\text{ Drafting} + 1\text{ Target Verification} \rightarrow \text{Multiple Tokens}$$

Why Is Checking Multiple Tokens at Once Faster?

Checking $k$ draft tokens is not free; it requires more computation than checking one.

Even so, combining several positions into one larger matrix multiplication can be far more GPU-efficient than performing $k$ target forwards sequentially.

In token-by-token decoding, one matrix dimension is tiny. The GPU cannot fully use its compute units, yet pays the cost of reading model weights every time.

Verification computes several token positions together.

• It performs more computation per weight read.
• It combines many small matrix multiplications into one larger operation.
• It reduces sequential target calls.
• It uses otherwise idle GPU compute capacity.

Speculative decoding does not simply eliminate all target computation.

It turns memory-bound decoding into more compute-bound verification and reduces sequential calls to the expensive target model.

Actual speed depends on

$$\text{Actual Gain} \approx \frac{ \text{Tokens Committed per Cycle} }{ \text{Draft Cost} + \text{Verification Cost} }.$$

An expensive draft takes too long to propose candidates. An inaccurate draft is mostly rejected, yielding only one or two tokens despite a target call.

Good speculative decoding therefore requires:

1. A sufficiently small, fast draft model.
2. High probability that the draft proposes tokens the target will choose.
3. Efficient GPU verification of multiple candidates.

The EAGLE family focuses on the second problem: how can a small draft predict a large target's next token more accurately?

Does Using a Small Model's Answer Reduce Quality?

This is the first question many people ask.

The draft is necessarily weaker than the target, so will using its tokens degrade the original LLM's answer?

No: the draft model does not make the final decision.

It only proposes candidates; acceptance and correction follow the target distribution.

For greedy decoding:

• Accept if the proposal equals the target's argmax token.
• Otherwise use the target token and discard the remaining draft.

No matter how strange the proposal, the result matches target-only greedy decoding. Draft quality affects speed, not correctness.

Sampling uses a more careful rejection-sampling procedure.

For proposed token $x$, with draft probability $q(x)$ and target probability $p(x)$, acceptance probability is

$$A(x) = \min \left( 1, \frac{p(x)}{q(x)} \right).$$

The total probability mass that proposes and accepts $x$ is

$$q(x)A(x) = q(x) \min \left( 1, \frac{p(x)}{q(x)} \right) = \min(p(x),q(x)).$$

Only the overlap between draft and target distributions comes from the proposal.

On rejection, resample from target probability mass underrepresented by the draft.

$$p_{\mathrm{residual}}(x) = \operatorname{norm} \left( \max(0,p(x)-q(x)) \right).$$

Adding the accepted draft mass and corrected residual mass recovers exactly target distribution $p$.

$$\min(p(x),q(x)) + \max(0,p(x)-q(x)) = p(x).$$

Strictly, residual normalization and total rejection probability must also be included, but this is the core argument.

When draft and target are similar, overlap and acceptance are high. When they differ, rejection increases, but correction still preserves the target sampling distribution.

This is what lossless means in speculative decoding:

It does not replace the original model's answer with a smaller model's answer; it samples the same target output distribution with fewer sequential calls.

It does not change target weights either. EAGLE-3 does not produce a better answer; it produces the target's answer faster.

Limits of Vanilla Speculative Decoding

The simplest method uses a smaller LLM from the target model's family as the draft.

For a 70B target, a 1B or 3B model might write the draft.

But the small model operates independently. Even with the same tokenizer and similar training data, its next-token distribution can differ from the target's.

It is especially difficult for a small draft to match the target on ambiguous continuations, complex reasoning, or specialized knowledge.

A larger draft may improve acceptance but costs more; a smaller draft is faster but incurs more rejection.

$$\text{Larger Draft} \Rightarrow \begin{cases} \text{Higher Acceptance} \\ \text{Higher Draft Cost} \end{cases}$$

Vanilla speculative decoding is bound by this trade-off.

EAGLE begins with the following question:

If the draft can see internal features already computed by the target, could even a tiny draft follow the target's next choice more accurately?

EAGLE: Predict Features, Not Tokens

EAGLE reuses hidden features from the target's previous forward pass instead of adding an entirely independent small LLM.

The final part of a typical LLM can be simplified as

$$h_t \xrightarrow{\text{LM Head}} z_t \xrightarrow{\text{Softmax}} p(x_{t+1} \mid x_{\le t}).$$

$h_t$ is the target's top-layer feature; the LM head turns it into vocabulary-sized logits $z_t$.

A vanilla draft sees only the token sequence. EAGLE receives target features and predicts the feature at the next time step.

$$\hat{h}_{t+1} = D_\theta(h_{\le t}, e_{\le t+1}).$$

Here, $D_\theta$ is the small draft and $e$ is the embedding of the actually sampled token. EAGLE also inputs the token sequence one step ahead to reduce uncertainty in feature autoregression.

Passing the predicted feature through the target LM head yields the draft token distribution.

$$q(x_{t+2}) = \operatorname{softmax} (W_{\mathrm{LM}}\hat{h}_{t+1}).$$

The advantages are clear:

• The draft reuses rich internal target information.
• One decoder layer, far smaller than the full target, can draft tokens.
• It follows the target distribution more easily than an independent small LLM.
• It can reuse the target's LM head and token embeddings.

EAGLE combines feature- and token-prediction losses.

$$\mathcal{L}_{\mathrm{EAGLE}} = \lambda_{\mathrm{fea}} \mathcal{L}_{\mathrm{fea}} + \mathcal{L}_{\mathrm{token}}.$$

The feature loss moves draft output toward the true target feature; the token loss makes the LM-head distribution predict target tokens well.

This permits some multi-step drafting with self-generated inputs despite training one-step feature prediction.

EAGLE-2: Does Every Context Need the Same Draft Tree?

Speculative decoding can draft one chain or a tree with multiple candidates.

If both I and Today are likely next tokens, it can create two branches and extend candidates after each.

Tree attention lets the target verify branches together. Its mask allows each node to see only the shared prefix and its own ancestors.

EAGLE-1 used a fixed, predefined tree, but contexts differ in difficulty.

• Code and formulaic sentences with obvious continuations support deep drafts.
• Open-ended dialogue may benefit from broader early candidates rather than one deep branch.

EAGLE-2 exploits the fact that draft confidence approximates actual acceptance fairly well.

It expands high-confidence branches, prunes unlikely ones, and builds a context-specific dynamic draft tree.

Rather than changing what to predict, EAGLE-2 improves how to allocate a limited draft budget across the tree.

EAGLE-3 retains this dynamic draft tree.

Must the Feature Be Predicted Exactly?

EAGLE-3 begins by questioning feature prediction itself, seemingly EAGLE's strength.

The ultimate goal is not a vector identical to the target feature.

It is a token candidate the target will accept.

Suppose hidden vectors $a$ and $b$ are far apart.

$$\Vert a-b \Vert_2 \gg 0$$

They can still assign high probability to the same token after the LM head.

$$\arg\max W_{\mathrm{LM}}a = \arg\max W_{\mathrm{LM}}b.$$

That is sufficient for speculative decoding: only the target's chosen token must match.

Feature regression, however, forces draft output to resemble the target feature's coordinates.

$$\mathcal{L}_{\mathrm{fea}} = \Vert \hat{h}_{t+1}-h_{t+1} \Vert.$$

Many representations may predict the same token, but feature loss restricts the draft to one target representation.

The paper calls this the feature prediction constraint.

Indeed, EAGLE gains saturated quickly as training data grew. Feature regression constrained the draft even when more token patterns remained learnable.

Why not remove feature loss and predict tokens directly?

Removing it substantially increases first-token acceptance.

But it creates another problem.

Removing the Feature Constraint Creates Train–Inference Mismatch

EAGLE's feature loss was restrictive, but it served an important role.

By making $\hat{h}_{t+1}$ resemble true feature $h_{t+1}$, it kept self-generated inputs at later steps similar to features seen during training.

Without feature loss, draft output $a_{t+1}$ need not resemble $h_{t+1}$.

The first step receives exact target features.

$$g_1,g_2,\cdots,g_t \rightarrow a_{t+1}$$

From the second step onward, exact features at unverified positions are unavailable. Calling the target to obtain them would defeat speculative decoding.

The next step must therefore use previous draft output $a_{t+1}$.

$$g_1,\cdots,g_t,a_{t+1} \rightarrow a_{t+2}.$$

Under ordinary one-step teacher forcing, however, training sees mostly exact target features.

Inference feeds back the model's imperfect outputs.

$$\begin{aligned} \text{Training Input} &: g_1,g_2,\cdots,g_t \\ \text{Inference Input} &: g_1,\cdots,g_t,a_{t+1},a_{t+2},\cdots \end{aligned}$$

A small first-step error changes the second input distribution; the second error then enters the third input. Later steps move progressively farther from the training distribution, and acceptance drops sharply.

This resembles exposure bias in sequence models.

EAGLE-3 addresses it with Training-time Test.

Training-time Test

The name can be confused with test-time training, which updates weights during testing.

EAGLE-3 does not update gradients during inference.

It unrolls the real test-time multi-step drafting process during training.

Training does not stop after one draft call.

1. Input target features to produce first draft output $a_{t+1}$.
2. Feed $a_{t+1}$ back to produce $a_{t+2}$.
3. Feed the output back again to produce $a_{t+3}$.
4. Compute token-prediction loss at every step.

Conceptually, the objective is

$$\mathcal{L}_{\mathrm{TTT}} = \sum_{s=1}^{S} \mathcal{L}_{\mathrm{token}}^{(s)}.$$

The model sees contexts containing its own outputs during training.

$$[g_{\le t}] \rightarrow [g_{\le t},a_{t+1}] \rightarrow [g_{\le t},a_{t+1},a_{t+2}]$$

Deeper drafting therefore does not suddenly expose it to an unseen input distribution.

In the paper's words, the test process is brought into training.

The process does more than imitate one chain: predictions from multiple positions become inputs to the next round. Because they form a tree with different parents, its attention mask differs from a standard lower-triangular mask.

• The original training sequence uses ordinary causal attention.
• Draft nodes can see the shared prefix.
• Each node can see only its parent and ancestors.
• It cannot see future information from other branches.

This reproduces test-time draft-tree dependencies during training.

That is why EAGLE-3 can remove feature loss while retaining stable multi-step drafting.

Fuse Multiple Layers Instead of Using Only the Top Layer

Without the feature constraint, draft inputs need not remain in the target top-layer feature space.

EAGLE-3 uses this freedom to combine low-, middle-, and high-level target features.

Let hidden states from these layers be $l_t,m_t,h_t \in \mathbb{R}^{d}$.

First concatenate the three features.

$$[l_t;m_t;h_t] \in \mathbb{R}^{3d}.$$

A fully connected layer fuses them back to hidden size $d$.

$$g_t = W_g[l_t;m_t;h_t].$$

Combine this with current sampled-token embedding $e_t$ as input to the small draft decoder.

$$u_t = W_u[g_{t-1};e_t].$$

The indices are simplified, but the essential structure is as follows.

$$\text{Multi-layer Target Features} + \text{Current Token Embedding} \rightarrow \text{1-layer Draft Decoder} \rightarrow \text{LM Head} \rightarrow \text{Draft Token}.$$

The top-layer feature sits directly before the LM head and is strongly aligned to next-token logits.

The paper argues that if LM-head weights are full-rank, the top-layer feature corresponding to given logits is unique, so it is effectively aligned to information for the immediate next token.

But a draft must look beyond one token to several future tokens.

Different Transformer depths can contain information ranging from local patterns to abstract context. Fusing low, middle, and high features gives EAGLE-3 broader drafting information than a top-layer-only input.

It also does not recompute fusion feature $g$ with the target at every future step.

• Verified prefixes use target fusion feature $g$.
• Unverified draft regions use previous draft output $a$ instead.
• Training-time Test learns inputs mixing $g$ and $a$ in advance.

The Complete EAGLE-3 Inference Process

Now combine the pieces.

Given a prefix, EAGLE-3 repeats these steps.

1. Reuse Target Features

Store low, middle, and high hidden states from prefill or the previous verification and construct fusion feature $g$.

2. Generate a Draft Tree

The small one-layer decoder uses fusion features, token embeddings, and previous draft outputs to generate several future tokens.

EAGLE-2's dynamic tree assigns more candidate budget to high-confidence branches.

3. Verify in Parallel with Tree Attention

Feed the generated tree to the target once.

An attention mask lets each candidate see only the shared prefix and its ancestors, while target probabilities for branches are computed in parallel.

4. Strict Acceptance

For greedy decoding, accept the longest prefix matching target choices.

For sampling, use $\min(1,p/q)$ acceptance and residual correction to preserve the target distribution.

5. Update the KV Cache

Keep only the KV cache for the accepted path, discard rejected branches, and resume drafting from the committed prefix.

The procedure is:

$$\text{Target Feature} \rightarrow \text{Cheap Dynamic Draft Tree} \rightarrow \text{One Target Verification} \rightarrow \text{Accepted Token Prefix} \rightarrow \text{Repeat}.$$

What Each EAGLE-3 Component Solves

EAGLE-3 is not accelerated by one trick; each component addresses a different bottleneck.

Removing the feature constraint alone improves the first draft token but can collapse acceptance at later positions.

Conversely, Training-time Test alone still limits expressiveness and input-feature choice if exact feature regression remains mandatory.

EAGLE-3 applies both changes together.

Discard the constraint that forces feature resemblance to gain freedom in token prediction, then directly train on the resulting self-output distribution shift with Training-time Test.

Why More Data Finally Helps

Another key result is that data scaling benefits the draft model.

In EAGLE, more training data produced limited gains in acceptance and speedup.

The bottleneck was not data scarcity but the feature-regression constraint on the functions the model could learn.

EAGLE-3 changes this in three ways:

1. It focuses on token prediction, directly tied to the final objective.
2. Multi-layer features enrich the input.
3. Training-time Test learns the real multi-step input distribution.

More data can therefore teach target token distributions and recovery after self-prediction across more contexts.

The paper adds roughly 464K UltraChat-200K examples to about 68K ShareGPT examples and trains on target-generated responses. With about eight times EAGLE's data, EAGLE-3's scaling curve continues to rise.

This “scaling law” is an empirical observation that acceptance and speedup continue improving over the paper's tested range, not a theoretical guarantee of the same acceleration in every environment.

How Much Faster Is It?

The paper evaluates chat and reasoning models on MT-Bench, HumanEval, GSM8K, Alpaca, and CNN/DailyMail.

Selected results at temperature 0 are shown below.

Vicuna 13B reaches up to 6.47× speedup and mean accepted length 7.54 on HumanEval.

Code generation contains repeated syntax and fixed patterns, making candidates easier to predict and HumanEval speedups especially high.

Ablations also clarify the design.

Both removing the feature constraint and adding multi-layer fusion improve accepted length and speedup.

Is It Always Faster at Larger Batch Sizes?

Speculative decoding usually helps most at small batches.

At small batches, target decoding underutilizes GPU compute and is often memory-bandwidth-bound. Draft verification uses the idle compute capacity.

At large batches, vanilla decoding already performs large matrix multiplications. With high GPU utilization, drafting and tree management can become pure overhead.

EAGLE-3 is notable because it improves throughput at relatively large batches in production serving frameworks.

Results with SGLang 0.4.4, H100, LLaMA 3.1 8B, and MT-Bench are:

EAGLE becomes slower than baseline at batch 24, while EAGLE-3 still reaches 1.38× throughput at batch 64.

These numbers do not transfer directly to every GPU and serving environment.

In the same paper's vLLM experiment, EAGLE-3 falls to 1.01× at batch 56. The break-even point depends on draft length, tree width, kernels, GPU, quantization, context length, and request scheduling.

EAGLE-3 is lossless, but does not guarantee speedup.

• Lossless means that correct verification preserves the target output distribution.
• Speedup requires the saved target cost to exceed draft cost on the actual hardware and workload.

These concepts must be distinguished.

Interpreting “No Quality Loss” Precisely

The paper does not separately compare target generation quality.

EAGLE-3 leaves target weights unchanged and preserves its distribution through strict speculative sampling.

This does not mean every run produces an identical string under all conditions.

Greedy Decoding

Under the same numerical environment and deterministic kernels, it follows the target's original argmax token path.

Stochastic Sampling

It guarantees the same probability distribution as vanilla target sampling, not the same string every time.

Different random seeds or sampling order can change individual strings, while the sampled distribution remains the target distribution.

When the Draft Is Wrong

Rejections increase and speed falls; answer quality does not.

If the draft is consistently wrong, all benefit can disappear and only overhead remains. As long as strict verification is retained, however, the draft's incorrect distribution never replaces the final target distribution.

Draft performance is therefore a speed knob, not a quality knob.

Practical Limitations

A Draft Must Be Trained for Each Target

EAGLE-3 depends heavily on target hidden states, token embeddings, and LM head.

One EAGLE-3 draft cannot simply attach to an unrelated target; each target needs a compatible checkpoint.

Fine-tuned weights or changes in vocabulary, chat template, or architecture can affect acceptance.

Training Cost Remains

Vanilla speculative decoding can reuse an existing small model. EAGLE-3 must collect target responses and hidden features and train a dedicated draft head.

Services with large inference savings and repeated requests can recover this initial cost more easily.

Implementation Is More Complex Than a Simple Two-Model Setup

The system must fuse hidden states from multiple layers, build dynamic trees and tree-attention masks, and prune the KV cache to accepted branches.

Without efficient kernel and scheduler support in the serving framework, reproducing reported speedups is difficult.

Acceptance Varies by Domain

Repeated patterns in code make future tokens easy to predict.

Creative writing, open-ended generation, and domains far from training data may have lower acceptance.

Reasoning models are not necessarily difficult to draft. A DeepSeek-R1 draft additionally trained on mathematical data achieves high GSM8K speedup. What matters is how well draft training data covers the target's common output distribution.

Lossless Is Not Numerical Identity

The target distribution is preserved mathematically, but floating-point precision, kernels, quantization, and random-number consumption order can create numerical differences.

If an implementation substitutes an approximate threshold for strict acceptance to gain performance, it can no longer claim losslessness in the same sense.

What Is the Core of EAGLE-3?

At first, EAGLE-3 may look like merely “a small model that predicts several future tokens.”

Its core contribution goes beyond attaching another draft model.

First, speculative decoding separates proposal from decision.

• The draft proposes quickly.
• The target determines the final distribution.

This lets the draft remain lightweight without reducing answer quality to the draft's level.

Second, EAGLE reuses target features so a small draft can follow the large target's intent.

Third, EAGLE-3 recognizes that exact feature replication is not the final objective.

• Remove feature regression to free token prediction.
• Learn the resulting self-output distribution shift with Training-time Test.
• Fuse multiple layers rather than relying only on the top layer.
• Use EAGLE-2's dynamic tree to focus computation on likely candidates.

Finally, strict verification keeps final output in the original target distribution.

The entire mechanism can be summarized in one sentence:

A lightweight draft proposes several target futures in advance; the expensive target verifies them in parallel and corrects wrong proposals back to the target distribution.

The fast model writes the draft; the strong model approves it. Better drafts pass more tokens at once and increase speed. Even when the draft is wrong, the decision maker remains unchanged, preserving the answer distribution.

The most interesting part, in my view, is not better feature prediction but discarding the premise that features must match at all.

Removing the constraint initially breaks multi-step prediction. EAGLE-3 directly trains on the real test process, resolving that failure and enabling broader features and more data.

EAGLE-3 is therefore more than an LLM serving trick; it demonstrates a modeling principle:

If the final goal is token acceptance, do not constrain the exact shape of intermediate representations. Include the model's own outputs from the real deployment setting in training.

This is how EAGLE-3 makes drafts more accurate, accepts longer candidates, and reduces the sequential bottleneck of autoregressive decoding without degrading the original model.