Introduction

I recently reviewed several papers on diffusion models, including DDIM, along with a post that works through the DDPM equations. I also covered classifier-guided and classifier-free diffusion, which add conditions to diffusion models, as well as ControlNet, one of the most prominent recent conditioning papers. GLIDE, the paper introduced here, is likewise about conditional diffusion. Whereas the classifier-based methods discussed earlier apply only to discrete categories with labels, GLIDE asks how a text description can be used effectively as a condition during diffusion sampling.

After DALL-E 2 was released, I hurriedly began studying diffusion and only then noticed GLIDE among the related work. At the time, I knew almost nothing about diffusion, and the reviews I found through Google were not especially helpful, which made the paper difficult to approach. It has now been nearly a year since I began studying diffusion. GLIDE is admittedly no longer a new paper, but I think it is an excellent work for seeing an intermediate stage in the progression through which diffusion models came to surpass earlier generative methods such as GANs.

The Rise of Diffusion Models

SMLD and DDPM, the foundations of diffusion models, are score-based generative models. They spawned a variety of related studies and made the broader deep-learning community aware of diffusion modeling's potential. Among them, GLIDE anticipated that diffusion would eventually be applied to many tasks, just as GAN research had been, and focused on one of the most representative multimodal generation problems: text-to-image synthesis (T2I).

CLIP and T2I

T2I research had already progressed actively using state-of-the-art generative models. Two major accelerators were the Transformer architecture and CLIP, a Transformer-based model that aligns images and text. My earlier CLIP post provides a fuller explanation, but the basic idea is as follows.

Given a large dataset of paired images and captions (text prompts) that describe them, CLIP learns to increase the embedding similarity between each image and its positive text prompt while decreasing its similarity to the remaining negative prompts. By using contrastive learning, it moves beyond the discrete labels used in conventional classification tasks—one-hot encoded classes—and introduces a training structure that can associate images with a wide variety of textual expressions.

Turning Embedding Similarity into a Classifier

The expansion in usable text enabled by this architectural change had enormous implications. Having more usable text makes it possible to expand the dataset domain $p(x)$ while scaling networks up without the same constraints. Because improvements in semantic understanding tend to produce a snowball effect across deep-learning research, it also suggested that multimodal research and performance would grow at an unprecedented rate. GLIDE began by combining ideas from two lines of work: conditioning diffusion models with classifiers, and using CLIP to understand image-text similarity.

Guided Diffusion Models

Two representative studies in diffusion conditioning, Diffusion Models Beat GANs on Image Synthesis and Classifier-Free Diffusion Guidance, brought the diversity-quality trade-off known from GANs and autoregressive modeling—low-temperature sampling—to diffusion models. They also improved the controllability of diffusion generation and made latent manipulation easier.

$$\log p_\phi(y \mid x_t) \approx \left.\log p_\phi(y \mid x_t)\right|_{x_t = \mu} + (x_t - \mu)\left.\nabla_{x_t} \log p_\phi(y \mid x_t)\right|_{x_t = \mu} = (x_t - \mu)g+C_1$$

Classifier guidance assumes that the classifier's score has relatively low curvature near the mean and applies gradient guidance through a first-order Taylor approximation. It was proposed as an effective way to improve sample quality in diffusion models.

$$\tilde{\epsilon}(z_\lambda, c) = (1+w)\epsilon_\theta(z_\lambda, c) - w\epsilon_\theta(z_\lambda)$$

Classifier-free guidance arose in response to this approach. Relying on a classifier requires training it at every noise scale, which complicates the pipeline. More importantly, because classifier-gradient training acts directly on metrics such as FID or IS, critics argued that the method itself was not the principal reason for improved real-world sampling performance.

CLIP Guidance and Classifier-Free Guidance

The authors built a framework in which both approaches could be tested—using a classifier or avoiding one—and observed how each affected text-to-image sample quality.

A conventional classifier for a single task cannot handle arbitrary text descriptions, so the authors devised a method based on the CLIP score, or similarity. Classifier-free guidance needs no classifier, so it instead uses the ADM architecture from Diffusion Models Beat GANs and the conditioning technique introduced in the classifier-free paper. We will examine these details later alongside the official code.

A Brief Introduction to Diffusion Models

DDPM as a Score Estimator

Readers who want to understand GLIDE but have not studied diffusion may find the sudden appearance of epsilon notation intimidating. My posts introducing DDPM and score-based generative modeling provide more detail, but the basic picture is this.

The foundation of variational inference for solving the intractable distributions involved in sampling is to choose an easily sampled prior $p_\theta(z)$. GANs address this with direct sampling, while VAEs train an autoencoder and apply KL-divergence regularization. As its name suggests, a diffusion model solves the problem through a diffusion process.

$$q(x_t \mid x_{t-1}) := \mathcal{N}(x_t; \sqrt{\alpha_t}x_{t-1}, (1-\alpha_t)I)$$

The gradual dispersal of perfume through air can be described as Brownian motion. A stochastic differential equation describes that motion, and the diffusion process above is a solution to such an SDE. As small amounts of Gaussian noise are added step by step, $x_T$ eventually becomes Gaussian noise after sufficient time $T$.

Now suppose we want to generate data from Gaussian noise through a reverse process. Unlike the forward distribution, the reverse distribution $q(x_{t-1} \mid x_t)$ is intractable. The goal is therefore to use deep learning to learn a parametric approximation $p_\theta(x_{t-1} \mid x_t)$.

This leads to the simplified loss, the training objective of the diffusion process (DDPM):

$$L_\text{simple} := \mathbb{E}_{x_0 \sim q(x_0), \epsilon \sim \mathcal{N}(0, I)} \left( \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert_2^2 \right)$$

Through the score-matching formulation, we can show that the epsilon predicted by the DDPM network is a normalized version of the score function; the proof is omitted here.

$$\begin{aligned} &x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left(x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta(x_t, t)\right)+\sigma_tz \\ \rightarrow~~&x_{t-1} = \frac{1}{\sqrt{\alpha_t}} \left(x_i + \beta_i s_{\theta^\ast}(x_i, i)\right) + \sqrt{\beta_i}z_i \end{aligned}$$

The Improved DDPM paper extended this baseline to learn the variance as well. Various conditions can be added to the base equation. One conditioning method that enabled better diffusion performance on high-resolution images was the super-resolution diffusion model, which concatenates a downsampled input $x$ along the channel dimension. We will return to it in the code review.

$$p_\theta(y_{t-1} \mid y_t, x)$$

Guided Diffusion

This method uses a classifier $p_\phi(y \mid x_t)$ for the sample at every noisy stage of the process to steer the generation gradient. Here, $s$ controls the guidance strength.

$$\hat{\mu}_\theta(x_t \mid y) := \mu_\theta(x_t \mid y) + s \cdot \Sigma_\theta(x_t \mid y)\nabla_{x_t} \log p_\phi(y \mid x_t)$$

Classifier-Free Guidance

The preceding approach requires the inconvenience of training a classifier on noisy samples. Suppose instead that we assume an implicit classifier $p^i(y \mid x_t)$. By Bayes' rule, it satisfies the proportionality:

$$p^i(y \mid x_t) \propto \frac{p(x_t \mid y)}{p(x_t)}$$

The gradient, or score, of its log-likelihood can be written as:

$$\nabla_{x_t} \log p^i(y \mid x_t) \propto \nabla_{x_t} \log p(x_t \mid y) - \nabla_{x_t} \log p(x_t) \propto \epsilon^\ast(x_t \mid y) - \epsilon^\ast(x_t)$$

Thus, by training the network on both classifier-conditioned and unconditional samples, we obtain the following extrapolated direction:

$$\hat{\epsilon}_\theta(x_t \mid y) = \epsilon_\theta(x_t \mid \emptyset) + s \cdot \left(\epsilon_\theta(x_t \mid y) - \epsilon_\theta(x_t \mid \emptyset)\right)$$

CLIP Guidance

We can now describe the CLIP guidance used in this paper. CLIP computes the similarity between an image embedding extracted by an image encoder $f(x)$ and a text embedding extracted by a text encoder $g(c)$ using their inner product—more precisely, their cosine similarity.

$$f(x) \cdot g(c)$$

If an image and text are highly similar, their inner product is also large. Because CLIP can measure caption-image similarity this way, we can interpret that value as a kind of classification probability:

$$\hat{\mu}_\theta(x_t \mid y) := \mu_\theta(x_t \mid y) + s \cdot \Sigma_\theta(x_t \mid y)\nabla_{x_t} \left(f(x_t) \cdot g(c)\right)$$

As in the original classifier-guidance paper, GLIDE trains CLIP on noisy images $x_t$ so that it can obtain more accurate gradients during the reverse process. The paper calls this model Noised CLIP.

Earlier community experiments with CLIP-based text conditioning for diffusion had claimed that fine-tuning was possible without a dedicated Noised CLIP. However, those approaches required additional techniques or metrics, such as data augmentation or perceptual loss. The use of corruption datasets such as CIFAR-10C for domain-shift problems illustrates why: a noisy input lies outside the classifier's training distribution and can therefore interfere with correct classification.

Related Work

Text-Conditional Image Generation

Many methods for generating images conditioned on text already existed. Numerous approaches used then-state-of-the-art GANs as their architectural foundation and trained on captioning datasets. DALL-E, one of the works closest to GLIDE in time, instead used vector-quantized learning as its baseline. Although earlier work did not use a diffusion baseline, attempts to combine text conditioning with diffusion grew as diffusion models improved. Vector-quantized diffusion, which draws heavily on DALL-E's ideas, belongs to the earliest wave of diffusion-based T2I research.

Diffusion-Based Generation Tasks

Diffusion research also began expanding beyond text-to-image generation. Papers such as SDEdit showed that stochastic differential equations—the diffusion process—could solve conditioned tasks such as rough-sketch or stroke-to-image generation, rather than only tasks such as inpainting.

The Palette paper further showed that diffusion models trained separately for individual image-to-image translation tasks could perform all of those tasks successfully.

CLIP-Guided Image Generation with GANs

CLIP-guided image generation based on GANs also became an active research area. Examples include StyleCLIP (StyleGAN + CLIP), BigSleep (BigGAN + CLIP), and StyleGAN-NADA, including papers reviewed previously. The linked posts explain in detail how GANs and CLIP guidance were combined. The LAFITE paper trained a GAN conditioned on CLIP image embeddings perturbed with predefined noise to approximate CLIP text embeddings. This enabled text-to-image conditioning even without a high-quality paired image-text dataset—that is, in a language-free setting.

CLIP-Guided Image Generation with Diffusion

GLIDE was not the first attempt to apply CLIP guidance to diffusion. Earlier examples include Colab code released by Crowson and DiffusionCLIP, which fine-tuned a diffusion model during DDIM reconstruction to follow a CLIP loss.

Text-Based Image Editing

Papers such as Paint by Word and Diffusion-Based Image Editing introduced methods that use CLIP to train GANs or diffusion models to transform images according to a desired prompt.

Training

Because GLIDE studies T2I generation with diffusion, understanding its training procedure is essential. In the main experiment, the authors trained a large 3.5-billion-parameter diffusion model with text conditioning on ordinary $64 \times 64$ images, together with a large 1.5-billion-parameter upsampling diffusion model that maps $64 \times 64$ images to $256 \times 256$. For CLIP guidance, they also trained a ViT-L CLIP model on $64 \times 64$ images with noisy inputs, as described above.

Training Text-Conditional Diffusion Models

The paper uses the ADM model from Diffusion Models Beat GANs. My diffusion-paper overview traces how this architecture was derived; in summary:

• Each resolution has two BigGAN residual blocks, and the width is adjusted by resolution.
• Every attention head has 64 channels, with attention layers at resolutions 32, 16, and 8.
• BigGAN residual blocks perform upsampling and downsampling, while AdaGN injects timestep and class embeddings.

In the original ADM, class conditioning was performed by feeding class and time embeddings into attention during training. GLIDE replaces the class embedding with the text embedding produced by a Transformer. Because the paper itself does not explain this part in much detail, let us examine the official GitHub code.

Extracting Text Embeddings with a Transformer

xf_in = self.token_embedding(tokens.long())
xf_in = xf_in + self.positional_embedding[None]
if self.xf_padding:
    assert mask is not None
    xf_in = th.where(mask[..., None], xf_in, self.padding_embedding[None])
xf_out = self.transformer(xf_in.to(self.dtype))
if self.final_ln is not None:
    xf_out = self.final_ln(xf_out)
xf_proj = self.transformer_proj(xf_out[:, -1])
xf_out = xf_out.permute(0, 2, 1)  # NLC -> NCL

outputs = dict(xf_proj=xf_proj, xf_out=xf_out)

This is the get_text_emb method of the Text2ImUNet class. xf_in turns the input prompt into tokens and embeddings, adds positional encoding, and feeds the result to the Transformer. After all Transformer operations are complete, the embedding of the final output token (xf_out[:, -1]) is projected to the required dimensionality and used for conditioning. The meaning of this final token can be seen here:

tokens = [self.start_token] + tokens[: text_ctx - 2] + [self.end_token]

The Transformer tokenizer uses BPE during encoding. It makes sequences the same length and appends an <EOS> token to mark the end. This is analogous to adding a class token at the front of a ViT sequence, passing it through the encoder, and using the resulting token feature as class information. After the Transformer, the attention information accumulated in the <EOS> token summarizes the entire text, so it can serve as the conditioning signal in place of a class embedding.

Conditioning the U-Net Modules

emb = self.time_embed(timestep_embedding(timesteps, self.model_channels))
if self.xf_width:
    text_outputs = self.get_text_emb(tokens, mask)
    xf_proj, xf_out = text_outputs["xf_proj"], text_outputs["xf_out"]
    emb = emb + xf_proj.to(emb)

During the actual U-Net forward pass, the model uses both the xf_proj extracted above and xf_out, whose feature and token dimensions have been permuted.

xf_proj is added to the timestep embedding, while xf_out is passed directly into the modules:

for module in self.input_blocks:
    h = module(h, emb, xf_out)
    hs.append(h)
h = self.middle_block(h, emb, xf_out)
for module in self.output_blocks:
    h = th.cat([h, hs.pop()], dim=1)
    h = module(h, emb, xf_out)

As the inherited UNet superclass shows, each module resolves to a TimestepBlock or an AttentionBlock:

class TimestepEmbedSequential(nn.Sequential, TimestepBlock):
    def forward(self, x, emb, encoder_out=None):
        for layer in self:
            if isinstance(layer, TimestepBlock):
                x = layer(x, emb)
            elif isinstance(layer, AttentionBlock):
                x = layer(x, encoder_out)
            else:
                x = layer(x)
        return

Different text conditions are used depending on the layer type. Layers such as ResBlock that inherit from TimestepBlock receive the combined time-and-text embedding (xf_proj) and use it for group normalization, just as in the original ADM:

if self.use_scale_shift_norm:
    out_norm, out_rest = self.out_layers[0], self.out_layers[1:]
    scale, shift = th.chunk(emb_out, 2, dim=1)
    h = out_norm(h) * (1 + scale) + shift
    h = out_rest(h)
else:
    h = h + emb_out
    h = self.out_layers(h)

Layers implemented as AttentionBlock, meanwhile, use all text-token information as keys and values in cross-attention. The noisy image x, which supplies the queries, can therefore interact directly with the text information:

def forward(self, x, encoder_out=None):
    b, c, *spatial = x.shape
    qkv = self.qkv(self.norm(x).view(b, c, -1))
    if encoder_out is not None:
        encoder_out = self.encoder_kv(encoder_out)
        h = self.attention(qkv, encoder_out)
    else:
        h = self.attention(qkv)
    h = self.proj_out(h)
    return x + h.reshape(b, c, *spatial)

Because the architecture has attention layers at resolutions $32$, $16$, and $8$, the image features are projected to match each attention map (self.qkv(self.norm(x).view(b, c, -1))) before acting as queries against the text keys and values.

Training Dataset and Architecture

The model was trained on the same dataset used for DALL-E. Diffusion Models Beat GANs avoided substantially increasing network width because of the computational cost relative to sampling efficiency. GLIDE, however, boldly increases the width to 512 channels so that the model can carry richer textual information. Its Transformer contains 24 residual blocks with 2,048 channels. Together, the U-Net and Transformer contain 3.5 billion parameters.

The upsampling diffusion model uses the same type of conditioning but a smaller Transformer than the image-generation U-Net, reducing its width from 2,048 to 1,024. The paper contains the remaining training details, so I will not repeat them here.

Classifier-Free Guidance

The architecture above merely conditions on the Transformer's text output; it does not yet include CLIP guidance or classifier-free guidance. After pretraining is complete, 20% of text-token sequences are replaced with an empty sequence (NULL $= \emptyset$) so that the model is fine-tuned to learn an unconditional representation alongside its text-conditioned behavior. As in the original classifier-free guidance paper, the network thereby represents both an unconditional diffusion distribution $p_\theta(z)$ and a conditional distribution $p_\theta(z,c)$. The only conceptual change is that $c$ is now a text embedding rather than a class embedding.

Image Inpainting

As discussed in the related-work section, Palette trained directly for each image-to-image translation task. Other studies did not necessarily train their diffusion models specifically for the target editing task. If an input perturbed for a different purpose is simply diffused into noise and reconstructed, discontinuities and artifacts can appear around the perturbed edges.

GLIDE therefore trains for inpainting in a manner similar to Palette. The inpainting U-Net performs the following forward pass:

def forward(self, x, timesteps, inpaint_image=None, inpaint_mask=None, **kwargs):
    if inpaint_image is None:
        inpaint_image = th.zeros_like(x)
    if inpaint_mask is None:
        inpaint_mask = th.zeros_like(x[:, :1])
    return super().forward(
        th.cat([x, inpaint_image * inpaint_mask, inpaint_mask], dim=1),
        timesteps,
        **kwargs,
    )

It concatenates x, the original unmasked RGB image, with an RGB copy of x to which the mask has been applied. It then adds the inpainting mask itself, producing a seven-channel input: RGB original + RGB masked image + mask.

Because the input channel count changes, the receiving module's channels must change as well. Since this stage is fine-tuning, the first three channels reuse pretrained weights, while the remaining channels are initialized to zero. During fine-tuning, the corresponding upsampling model receives the entire low-resolution image—the original RGB—but only the unmasked region of the high-resolution image, as shown below.

def forward(self, x, timesteps, inpaint_image=None,
                  inpaint_mask=None, low_res=None, **kwargs):
    if inpaint_image is None:
        inpaint_image = th.zeros_like(x)
    if inpaint_mask is None:
        inpaint_mask = th.zeros_like(x[:, :1])
    _, _, new_height, new_width = x.shape
    upsampled = F.interpolate(
        low_res, (new_height, new_width), mode="bilinear", align_corners=False
    )
    return super().forward(
        th.cat([x, inpaint_image * inpaint_mask, inpaint_mask, upsampled], dim=1),
        timesteps,
        **kwargs,
    )

The ordinary upsampling module concatenates a noisy $256 \times 256$ sample $x_t$ with the bicubic-upsampled $64 \times 64$ ground truth. For inpainting, the masked high-resolution ground truth and its mask are inserted conditionally between those two inputs.

Noised CLIP Models

Just as Diffusion Models Beat GANs trained its classifier on inputs at every noise level to obtain more accurate classifier guidance, GLIDE trains CLIP on noisy images $x_t$ for CLIP guidance.

Results

CLIP Guidance vs. Classifier-Free Guidance

The qualitative results are shown above. Classifier-free guidance preserves detail better and produces fewer artifacts or unnatural regions.

Classifier-free guidance also performs better overall on quantitative metrics. In the precision-recall plot, the CLIP-guidance curve bends backward in the precision region. As the earlier paper review explained, the normal trade-off is for precision to rise as recall is sacrificed.

If precision fails to increase properly, $P_g$ is not sufficiently contained within $P_r$; in practical terms, the sampling process cannot cover diverse text prompts. Classifier-free guidance also spans a wider range in IS and FID. Most strikingly, when FID is plotted against CLIP score (similarity), classifier-free guidance is more effective than directly guiding with CLIP.

Classifier-free guidance also earns substantially higher human-rated Elo scores. GLIDE outperforms DALL-E in both photorealism and caption similarity. This comparison applies reranking to DALL-E—selecting preferred samples according to their CLIP scores—but not to GLIDE; the percentages denote win rates. The last row applies DALL-E's d-VAE to GLIDE's output.