How Does a Conventional LLM Work?

The generative Transformers commonly known as LLMs, including GPT and Claude, are trained and used for inference autoregressively. The model writes a sentence from left to right, one token at a time, conditioning every step on the tokens that came before it. In that sense, it behaves much like a person writing sequentially. But must generation always proceed one token at a time?

Autoregressive Generation

In autoregressive generation, the next predicted token depends on every preceding token. It resembles choosing the next word while being allowed to see only what has already been written. Every later prediction assumes the earlier predictions.

• Token 3 depends on tokens 1 and 2; token 4 depends on tokens 1, 2, and 3; and so on.
• The process is completely sequential, making it difficult to parallelize.
• As the model grows, the FLOPs per token increase and raise the total response latency.

Training follows the same principle. The model predicts the next token while attention is masked so that it cannot see future tokens. The following figures illustrate the process.

Tokenization

An LLM receives many kinds of input, such as a user's request, and produces a complete piece of text.

Before inference on a request like the one above, it must be tokenized. Tokenization converts sentences across languages—English, Korean, German, Spanish, Japanese, Chinese, and more—into numbers representing the smallest units from which text can be composed. Those numbers become the model's input.

How should this “smallest unit” be defined?

The Ideal Form of Tokenization

Splitting by words, as suggested by the figure, would be simple. Mapping every word in the world to a distinct number, however, is far too expensive.

Even if spaces define word boundaries, a word can have many surface forms. Without a suitable basic unit, representing words as numbers requires an enormous vocabulary.

Splitting into individual characters such as letters creates the opposite problem: sequences become very long. One of the Transformer's largest disadvantages, discussed in an earlier article, is that computation becomes excessive as the context length grows.

Most LLMs therefore use BPE, which encodes frequently occurring byte pairs in UTF-8 as new units. I will skip the algorithmic details. It is enough to understand BPE as

a way to obtain units that can express many words without making the context excessively long.

Training and Word Prediction

During training, masking enables parallel processing. Autoregressive generation requires causality: the preceding words must be known before the next word can be predicted, so inference is difficult to parallelize. During training, however, we can simply mask the later words as though they did not exist.

Consider the following sentence.

Causality is an influence by which one event, process, state, or object (a cause) contributes to the production of another event, process, state, or object (an effect) where the cause is at least partly responsible for the effect, and the effect is at least partly dependent on the cause. The cause of something may also be described as the reason for the event or process.

To train a particular token, prepare a mask that hides that token and everything after it. It resembles covering the Korean translation with a hand while memorizing English vocabulary as a child. To learn process, the word after one event in the first line, we present

Causality is an influence by which one event, [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK] [MASK]

The model cannot refer to any later words and must predict the one immediately following the visible prefix.

The mask behaves like an array of zeros and ones. It can arrange for several token predictions at the same time, whose losses are then combined, usually averaged, for training. For sequence length $T$, token vocabulary $\mathcal{V}$, predicted probability $\hat{p}_{t,v}$, and target $y_{t,v}$,

$$\mathcal{L}_{\text{CE}} = -\sum_{t=1}^{T}\sum_{v=1}^{|\mathcal{V}|} y_{t,v}\,\log\!\bigl(\hat{p}_{t,v}\bigr) = -\sum_{t=1}^{T} \log\!\bigl(\hat{p}_{t,y_t}\bigr).$$

Cross-entropy matches the predicted distribution to the target. After seeing an enormous amount of text, the LLM learns to place the most plausible word after a given sequence.

How Does Text Diffusion Work?

When Google introduced its latest Gemini family at I/O, it also presented Gemini Diffusion, a new model based on a diffusion architecture. According to the presentation, its performance on reasoning, mathematics, coding, and several other metrics is comparable to existing SOTA models.

Diffusion models for text generation had long been treated as interesting research, while their practical performance lagged behind other architectures. Unlike an autoregressive LLM, a diffusion approach does not predict one next token at a time. It begins with an unfinished sentence in which every position is masked and samples several tokens together. It was unclear whether predicting many tokens at once could produce a coherent answer, or whether such a sampling structure was well suited to language.

Text Diffusion

Image diffusion begins with random noise and reconstructs an image by denoising it over several steps. Text diffusion denoises tokens rather than pixels.

1. Fill a sentence with [MASK] tokens—the text equivalent of noise.
2. Gradually replace masks with actual tokens over several steps.

Google did not disclose the exact training procedure and architecture, but recent work such as LLaDA provides a rough picture.

The underlying claim is that the essence of a language model is not autoregression itself, but the ability to model a word distribution accurately. A model should therefore be able to predict a word at an arbitrary position in a paragraph and fill the remaining blanks as well.

Diffusion Pretraining

Because the model will not generate only one word at a time, it receives an entire sequence as input and emits an entire sequence as output. In other words, it trains a sequence-to-sequence Transformer.

• The output layer predicts a token at every position, but the loss is calculated only at masked positions.
• This is quite similar to BERT's masked modeling.
• During later supervised fine-tuning, the prompt remains visible and only the response is masked for training.

Diffusion Inference

So far, this looks like an ordinary sequence-to-sequence model. Many studies, however, find that predicting an entire answer in one pass performs poorly. Diffusion provides an iterative solution.

1. Begin with every position masked.
2. Predict every position with the model.
3. Commit only some tokens according to a random or confidence-based rule, schedule that proportion by the time step, and mask the rest again.
4. Run inference again and repeat steps 2–3 for $T$ steps.

Early in the process, roughly 90% of positions may be masked again. Near the end, only about 10% may be remasked. If each step commits two or more tokens and the total number of steps $T$ is smaller than the autoregressive sequence length, inference can become considerably faster.

In short: mask the entire prediction region → predict → remask → repeat.

Conclusion and Personal View

Will It Truly Become Mainstream?

Google expects diffusion to become a mainstream architecture for future frontier models.

• It is fast. As LLMs are increasingly used in loop-based workflows such as agents and code generation, speed becomes crucial.
• It can reduce GPU resources. Faster inference reduces time on chip, and therefore the number of chips required.

A Major Disadvantage: Existing Algorithms May No Longer Apply

I still lack a clear intuition for whether diffusion can model language as powerfully as autoregression. Existing LLM techniques such as chain of thought, which follow preceding reasoning sequentially, may not carry over directly. Context engineering and prompt engineering became central partly because of the autoregressive process. If diffusion becomes dominant, implementing reasoning models in the current DeepSeek-like form may be difficult.

Strictly speaking, this training method also differs substantially from the diffusion process defined for images. It may be more accurate to view it not as image-style diffusion, but as a way of running autoregressive-like prediction in parallel and improving an answer over several steps to overcome the limits of a single parallel pass.