Introduction

This review covers a series of three papers. All build on diffusion models—DDPM and DDIM—and seek to improve diffusion sampling quality. Improved DDPM adds several modifications to DDPM’s basic experiments and improves sample log likelihood. Diffusion Beats GANs presents broader architecture ablations and classifier guidance, showing that diffusion models can exceed GAN sampling quality. Finally, Classifier-Free Diffusion Guidance discusses the limitations of classifier guidance and proposes a method that retains the advantages of conditional generation without explicitly training a classifier. Since little additional proof is required, it seems useful to cover all three together.

Improved DDPM

After DDPM attracted attention as a new way to train and sample a generative model, researchers began asking whether it could handle datasets with diverse samples, such as ImageNet, beyond CIFAR-10 and LSUN. GANs offered fast, high-quality sampling but struggled with training instability and low sample diversity. Likelihood methods such as VAEs provided stability and diversity, but sampled more slowly and with lower quality than GANs. DDPM arrived in this context, but it too was no universal solution. It was only a new methodology, with network design and training still open to future work.

The paper improves sampling quality in two main ways. First, it uses a hybrid objective. DDPM optimized a simplified modification of the variational lower bound (VLB); this work adds VLB loss back into the objective.

Second, it learns rather than fixes the variance. DDPM needed hundreds of forward-process steps for good samples, but the learned variance achieves similar quality with as few as $50$ forward passes. Unlike DDIM, which introduces a non-Markovian sampling process, this work preserves DDPM’s Markovian process and modifies training itself.

Conventional DDPM

The earlier post covers the full DDPM derivation. Omitting those proofs, let data follow $x_0 \sim q(x_0)$. The forward noising process adds tiny Gaussian perturbations:

$$q(x_t \vert x_{t-1}) = \mathcal{N}(x_t; \sqrt{1-\beta_t}x_{t-1}, \beta_t I)$$

Given a sufficiently long time $T$ and a well-scheduled $\beta_t$, the parameterized prior can sample $x_T \sim \mathcal{N}(0, I)$: the forward process gradually approaches Gaussian noise. If the reverse conditional $q(x_{t-1} \vert x_t)$ were known, we could sample $x_0$ from arbitrary Gaussian noise, but this distribution is intractable.

A parameterized neural network therefore predicts $p_\theta(x_{t-1} \vert x_t)$ from each noised sample $x_t$, gradually removes noise, and samples $x_0$. The reverse of adding tiny Gaussian noise can be approximated as subtracting Gaussian noise.

$$p_\theta(x_{t-1} \vert x_t) := \mathcal{N}(x_{t-1}; \mu_\theta (x_t, t), \Sigma_\theta (x_t, t))$$

The variational lower bound optimized by DDPM is

$$\mathcal{L} \le D_{KL}(q(x_T \vert x_0) \vert\vert p_\theta(x_T)) -\sum_{t > 1} D_{KL} (q(x_{t-1} \vert x_t, x_0) \vert\vert p_\theta(x_{t-1} \vert x_t)) -\mathbb{E}_q\left(\log p_{\theta}(x_0 \vert x_1) \right)$$

The first term is naturally near zero after perturbing arbitrary data $x_0$ for sufficiently long $T$, so it need not be optimized. The middle term is a KL divergence that trains the reverse-predicting network to follow the forward-process posterior. The last projects the probability of the 256-valued RGB image when generating $x_0$ from $x_1$. See the DDPM post for the detailed proof and an explanation of every term. Simplifying the bound gives

$$\begin{aligned} &\mathbb{E}_{x_0,\epsilon} \left( \frac{\beta_t^2}{2\sigma_t^2 \alpha_t(1-\bar{\alpha}_t)} \parallel \epsilon - \epsilon_\theta(\sqrt{\bar{\alpha}_t}x_0 + \sqrt{1-\bar{\alpha}_t}\epsilon, t) \parallel^2\right) \\ \approx& \mathbb{E}_{t, x_0, \epsilon} \left( \parallel \epsilon - \epsilon_\theta(x_t, t) \parallel^2\right) \end{aligned}$$

Increasing DDPM log likelihood

DDPM achieved strong FID and IS (Inception Score), common sample-quality metrics, but weak log likelihood. Log likelihood measures how well a generative model reflects the modes of the data distribution. Optimizing it encourages the model to capture the distribution’s overall form. A network that produces attractive samples from only one portion of the true distribution cannot be said to model that data well. This paper analyzes why DDPM’s log likelihood is weak, then shows that improving it can substantially improve actual sample quality.

Learnable standard deviation (variance)

DDPM fixes a predefined variance $\sigma_t^2I$. Curiously, sampling quality differs little whether $\sigma^2_t$ is set to $\beta_t$ or to the forward-posterior value $\tilde{\beta}_t = \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t} \beta_t$. The former makes the $t$th kernel variance isotropic Gaussian with respect to $q(x_0)$, while the latter makes it a delta function. If $\beta_t$ and $\tilde{\beta}_t$ are the two endpoints available to the variance, why do they affect sampling so little? I failed to address this adequately in the earlier DDPM post, where I speculated:

The paper says that using $\beta_t$, the variance at step $t$, as $\alpha_t^2$ makes little difference in practice. I thought this was because the cumulative variance product $\bar{\alpha}_t$ varies little with $t$ when the number of steps is large—though I might be wrong.

In retrospect, perhaps ten percent of that was right. Let us examine the Improved DDPM analysis.

As the diffusion step grows, $\beta_t$ and $\tilde{\beta}_t$ become nearly identical. Setting $\sigma_t$ therefore has little effect on sample quality at large diffusion steps. Their meaningful difference lies near $t = 0$, where the image is nearly complete and contains little noise, so sample quality again changes little.

As the number of diffusion steps grows, changing $\Sigma_\theta(x_t, t)$ has little influence on determining the image distribution. But this does not imply that fixing $\sigma_t$ is optimal. Consider the next graph.

Optimizing log likelihood means optimizing the diffusion VLB. The closer a step is to the sample at $T=0$, rather than noise at $T=4000$, the more important its loss becomes. One reason DDPM failed to improve log likelihood effectively is that it did not model $\Sigma_\theta(\cdot)$ where the variance endpoints differ most. $\Sigma_\theta(x_t, t)$, which DDPM omitted from the loss, must also be predicted. The authors reason that if the prediction range of $\Sigma_\theta(x_t, t)$ is too small—as $t$ grows, even its log infimum and supremum barely differ—it is difficult for a neural network to predict. They instead parameterize variance by interpolating between $\beta_t$ and $\tilde{\beta}_t$. The network predicts a vector $v$, one element per dimension, and interpolates the variance as

$$\Sigma_\theta(x_t, t) = \exp(v\log \beta_t + (1-v)\log \tilde{\beta}_t)$$

Interpolating in log space is numerically more stable than interpolating $\beta_t$ directly. Although $v$ is allowed outside $0\sim1$ rather than being restricted to a convex combination, the trained network never predicted outside that interval. The simplified loss discards every variance-normalization term from the VLB, so it cannot learn the parameterized variance. The paper therefore combines $L_\text{simple}$ with a weighted VLB term:

$$L_\text{hybrid} = L_\text{simple} + \lambda L_\text{vlb}$$

Because $L_\text{simple}$ contributes to DDPM’s easy optimization, the paper uses the small value $\lambda = 0.001$. It also stops gradients through $\mu_\theta(x_t, t)$ while optimizing the VLB term. Thus $\Sigma_\theta$ learns stable variance under guidance from the simplified loss through $L_\text{hybrid}$, while $\mu_\theta$ is optimized only by the simplified loss.

Better noise scheduling

The paper also proposes a replacement for DDPM’s noise schedule. Linear scheduling works for high-dimensional, high-resolution images but is less effective for low-resolution ($32 \times 32$) images, whose forward process approaches Gaussian noise too rapidly.

With the linear schedule in the upper row, almost all image information disappears after only a few processes. To learn an effective sampling process, cumulative variance $\bar{\beta}_t$ should make samples noisy more gradually.

Models trained with a linear schedule can skip $20\% \sim 30\%$ of the diffusion process with little FID loss. Those extra steps therefore contribute nothing to sample quality.

The authors use a cosine schedule, producing variance that decreases more progressively—approximately linearly:

$$\bar{\alpha}_t = \frac{f(t)}{f(0)},~f(t) = \cos \left( \frac{t/T + s}{1+s} \cdot \frac{\pi}{2}\right)^2$$

This definition makes $\beta$ approach $1$ too closely near $t = T$, so it is clipped at $0.999$ to avoid the resulting singularity. A small offset $s$ also prevents values near $t = 0$ from becoming too small. Accounting for pixel bins gives $1/127.5$, which determines $s = 0.008$.

Reducing gradient noise

Why use the hybrid loss? If the direct objective is better log likelihood, optimizing the VLB alone seems natural. The results are less cooperative.

Although VLB is the loss directly related to log likelihood, the graph shows much noisier training and poorer convergence than the hybrid objective. Hybrid loss has a lower loss curve overall. The authors hypothesize that pure VLB performs worse because its gradients contain more noise than hybrid-loss gradients.

Unlike simplified loss, $L_\text{VLB}$ has different magnitudes at different steps. Uniformly sampling $t$ therefore does not help the VLB objective. The paper introduces importance sampling:

$$\begin{aligned} &L_\text{VLB} = \mathbb{E}_{t \sim p_t} \left( \frac{L_t}{p_t} \right) \\ &\text{Where }p_t \propto \sqrt{\mathbb{E}(L_t^2)} \text{ and }\sum p_t = 1 \end{aligned}$$

This resembles focal loss. Sampling follows probabilities continually updated from the previous ten loss records at each time step. Initially, $t$ is sampled uniformly until ten observations have accumulated for every value.

Improved DDPM appears to focus on improving DDPM log likelihood relative to likelihood-based networks. Parameterizing the variance also seems to improve FID considerably.

Diffusion Beats GANs

The next paper proposes a way to surpass GAN sampling quality. Baseline DDPM was relatively weak at sampling complex datasets such as ImageNet. With a provocative title from OpenAI, an organization that clearly loves diffusion, “Diffusion Beats GANs” became one of the papers that popularized diffusion. It beats the then-state-of-the-art BigGAN on ImageNet and covers both unconditional and conditional generation.

Why was diffusion not good enough?

Despite strong sample diversity and stable training, why had diffusion sampling quality remained insufficient? The paper’s core is to form hypotheses about the problem and address them. The authors identify two reasons:

1. GANs had received much longer study than diffusion, producing extensive research into optimal architectures, training methods, and hyperparameters.
2. GANs trade diversity for fidelity, making them difficult to beat on sample quality alone.

DDPM was simply young—this paper appeared about one year later—while GANs were explicitly designed to sacrifice diversity for high-quality samples. The paper begins by importing both GAN advantages—optimized network structure and sampling quality—into diffusion, with the ambition of surpassing GANs outright.

Background

The framework begins with DDPM, supplemented by Improved DDPM’s trainable variance and DDIM for fast sampling. Improved DDPM obtains high quality with fewer time steps by changing training; DDIM instead changes sampling through a non-Markovian process with the same marginals. The approaches are therefore distinct. I recommend reading my DDIM post first.

In summary, training uses Improved DDPM’s hybrid loss, while generation with fewer than 50 steps uses DDIM. This relates to an Improved DDPM experiment not discussed above.

The graph shows that DDIM sampling quality becomes better beyond roughly $50$ steps.

Sample-quality metrics

GAN research commonly measures sampling quality with IS and FID, but both are imperfect. This is one reason generative-model research can be persuasive through qualitative evaluation yet struggle to persuade through quantitative evaluation. Beyond a certain point, deciding which of two generated images is “better” has little objective meaning. Adversarially trained GANs can also raise classifier-based metrics naturally, because fake samples attack the discriminator through gradients.

$$IS(G) = \exp\left(\mathbb{E}_{x \sim G}\left[D_{\mathrm{KL}}(p(y \mid x)\Vert p(y))\right]\right)$$

Inception Score is defined above. $p(y)$ measures whether generated samples cover all classes evenly, while $p(y \vert x)$ measures each sample’s quality. IS cannot measure diversity within a class. A CIFAR-10 generator could output one excellent sample for each of the ten classes and still receive a satisfying IS; collapse would go undetected. FID addresses this by using features from an Inception network and modeling multivariate Gaussian distributions from their means and covariances.

$$FID(x, g) = \lVert \mu_x - \mu_g \rVert_2^2 + \operatorname{Tr}\left(\Sigma_x + \Sigma_g - 2(\Sigma_x \Sigma_g)^{1/2}\right)$$

Another paper proposes precision and recall to separate sample fidelity from diversity. Let the model’s implicit distribution be $P_g$ and the real distribution $P_r$. The fraction of generated samples lying inside the real distribution measures fidelity, while the fraction of real samples covered by the generated distribution measures diversity.

$$(Precision) = \frac{TP}{TP+FP}$$

The proportion of true positives—samples in both $P_r$ and $P_g$—among true positives plus false positives—samples in $P_g$ but not $P_r$—corresponds to sample quality.

$$(Recall) = \frac{TP}{TP+FN}$$

The proportion of true positives among true positives plus false negatives—samples in $P_r$ but not $P_g$—corresponds to sampling diversity. This paper uses precision and IS for fidelity and recall for diversity.

Architecture improvements

One limitation of early DDPM-based diffusion research was insufficient exploration of network structure. The authors search for architectures that improve diffusion sampling quality:

• Increase width—channel count—relative to depth while keeping total model size approximately constant.
• Increase the number of attention heads; the baseline U-Net includes attention inside residual blocks.
• Apply attention not only at the $16 \times 16$ feature-map level, but also at $32 \times 32$ and $8 \times 8$.
• Use BigGAN residual blocks for activation upsampling and downsampling.

• Scale residual connections by $\frac{1}{\sqrt{2}}$.

Experiments use ImageNet images at $128 \times 128$, batch size $256$, 250 sampling steps, and FID.

In the left table, every architectural proposal except rescaling improves FID. Increasing depth also tends to improve performance, as the lower graph shows, but the authors stop because training time grows excessively.

Attention experiments show that more heads with fewer channels per head produce the best FID. Sixty-four channels offer the best performance-efficiency trade-off in training speed, so the paper adopts that choice. Interestingly, the architectural trend matches the Transformer.

Adaptive group normalization

AdaGN stylizes each residual block with time-step and class embeddings. Given hidden activation $h$ and the linear projection $y = [y_s,~y_b]$ of time-step and class embeddings,

$$\text{AdaGN}(h,~y) = y_s \cdot\text{GroupNorm}(h) + y_b$$

This is identical to AdaIN apart from GroupNorm, as readers of StyleGAN may notice.

AdaGN performs well and is used throughout training; the figure shows its ablation. The final architecture is

• two BigGAN residual blocks at each resolution, with width adjusted to resolution;
• 64 channels per attention head, with attention layers at resolutions 32, 16, and 8;
• BigGAN residual blocks for upsampling and downsampling, with AdaGN injecting time-step and class embeddings.

Classifier guidance

Conditional image synthesis has proven useful for high quality on datasets with limited labels. Viewing a GAN as a probability distribution, a discriminator supplying explicit $p(y \vert x)$ information can guide the generator to produce images of each label more effectively than merely distinguishing real from fake.

AdaGN already provides class embeddings as style information alongside time steps, but this differs from explicitly supplying discriminator information. The authors therefore develop classifier guidance. Suppose a pretrained classifier $p_\phi(y \vert x_t,~t)$ has learned classification over noised images $x_t$ at every time step. Its log-likelihood gradient $\nabla_{x_t} \log p_\phi(y \vert x_t,~t)$ can guide diffusion sampling. The paper separately derives conditional guidance for the Markovian DDPM sampler and non-Markovian DDIM sampler.

Conditional reverse noising process

For each noised image, pretrained classifier $p_\phi(y \vert x_t,~t)$ supplies information entirely external to the diffusion process. Normalizing factor $Z$ can therefore be treated as constant; see the appendix for details.

$$p_{\theta,\phi}(x_t \vert x_{t+1} , y) = Zp_\theta (x_t \vert x_{t+1}) p_\phi(y \vert x_t)$$

Recall the unconditional diffusion reverse process. For predicted $\mu, \Sigma$ at each time,

$$\log p_\theta(x_t \vert x_{t+1}) = -\frac{1}{2}(x_t-\mu)^\top \Sigma^{-1} (x_t - \mu)+C$$

The curvature of $\log p_\phi(y \vert x_t)$ is expected to be small relative to $\Sigma^{-1}$. The reverse-process log likelihood is a quadratic whose curvature is related to $1/2\parallel \Sigma \parallel$. Since $\Sigma$ is near zero for most diffusion steps, this coefficient is very large. The classifier function can therefore reasonably be assumed to have much smaller curvature.

At $x_t = \mu$, the vertex of the reverse process’s quadratic, classifier guidance can be represented by a first-order Taylor approximation. Sampling is governed mainly by $\mu$; the classifier curvature is negligible relative to $p_\theta$ there.

$$\log p_\phi(y \vert x_t) \approx \log p_\phi(y \vert x_t) \vert_{x_t = \mu}+(x_t - \mu)\nabla_{x_t} \log p_\phi (y \vert x_t) \vert_{x_t = \mu} = (x_t - \mu)g+C_1$$

Here $g$ is the classifier log-likelihood gradient at $x_t = \mu$. Substitution gives

$$\begin{aligned} \log(p_\theta(x_t \vert x_{t+1}) p_\phi(y \vert x_t)) \approx& -\frac{1}{2}(x_t-\mu)^\top \Sigma^{-1} (x_t - \mu)+(x_t - \mu)g + C_2 \\ =& -\frac{1}{2}(x_t-\mu-\Sigma g)^\top \Sigma^{-1} (x_t - \mu -\Sigma g)+ \frac{1}{2}g^\top \Sigma g + C_2 \\ =& -\frac{1}{2}(x_t-\mu-\Sigma g)^\top \Sigma^{-1} (x_t - \mu -\Sigma g)+ C_3 \\ =& \log p(z) + C_4,~z \sim \mathcal{N}(\mu + \Sigma g, \Sigma) \end{aligned}$$

Classifier guidance therefore turns the sampling direction by adjusting the drift.

Conditional sampling for DDIM

The method above adjusts drift and assumes a Markov process, so it cannot be used directly for deterministic DDIM sampling.

$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\underset{\text{predicted }x_0}{\left( \frac{x_t - \sqrt{1-\bar{\alpha}_t}\epsilon_\theta^{(t)}(x_t)}{\sqrt{\bar{\alpha}_t}} \right)} + \underset{\text{direction pointing to }x_t}{\sqrt{1-\bar{\alpha}_{t-1} - \sigma_t^2} \cdot \epsilon_\theta^{(t)}(x_t)} + \underset{\text{random noise}}{\sigma_t z},~z \sim \mathcal{N}(0, I)$$

Deterministic DDIM predicts $x_t$ from $x_0$, so a classifier gradient over $x_t$ cannot be applied in the same way. Here the paper connecting SDEs with diffusion models becomes useful. Its VP-SDE expresses DDPM sampling as

$$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}}_t} \epsilon_\theta(x_t, t)\right)+\sigma_tz$$

and rewrites it as an SDE over score estimate $s_{\theta^\ast}(\cdot)$:

$$x_{t-1} = \frac{1}{\sqrt{\alpha_t}} (x_i + \beta_i s_{\theta^\ast}(x_i, i)) + \sqrt{\beta_i}z_i$$

The score can thus be defined at time $t$ independently of ancestral sampling:

$$\nabla_{x_t} \log p_\theta (x_t) = -\frac{1}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta (x_t)$$

Applying this to the score of $p(x_t)p(y \vert x_t)$ gives

$$\begin{aligned} \nabla_{x_t}\log \left( p_\theta(x_t)p_\phi(y \vert x_t)\right) =& \nabla_{x_t} \log p_\theta(x_t) + \nabla_{x_t} \log p_\phi(y \vert x_t) \\ =& -\frac{1}{\sqrt{1-\bar{\alpha}_t}}\epsilon_\theta(x_t) + \nabla_{x_t} \log p_\phi(y \vert x_t) \end{aligned}$$

We can redefine epsilon, changing the gradient just as for DDPM:

$$\hat{\epsilon}_\theta(x_t) := \epsilon_\theta(x_t) - \sqrt{1-\bar{\alpha}_t}\nabla_{x_t} \log p_\phi (y \vert x_t)$$

Scaling the classifier gradient

Classifier score guidance requires training $p_\phi$. Its architecture takes downsampled features from the diffusion U-Net and applies $8 \times 8$ attention pooling for the final output. Because it must classify every noise step, it trains on noised inputs from every time step. Sampling then uses its gradient as described above.

Initial experiments with an unconditional ImageNet model found that unless classifier guidance scale $s$ exceeded $1$, the probability of generating the desired class fell by half. Even when such samples were produced, they often did not visually belong to the class.

For “Pembroke Welsh corgi,” scale $1.0$ on the left fails to produce convincing corgis, while scale $10.0$ improves them substantially.

The table notably shows that sufficiently strong classifier guidance on an unconditional model—guidance $10.0$—achieves FID and IS comparable to a conditional model.

The paper also surpasses BigGAN with a two-stage diffusion process conditioned on a low-resolution image. Sampling speed remains a problem, however, and separate classifier training restricts the method to labeled samples.

Classifier-Free Diffusion Guidance

Low-temperature sampling

By adjusting gradients with a classifier, classifier guidance gains fidelity while sacrificing some diversity. Its focus is sample quality rather than diversity.

This trade-off was already studied in GANs and other generative models under “low-temperature sampling,” terminology derived from the Boltzmann machine, an energy-based model.

View the prior as a set of energy-based states $S(\tau)$. High-energy states are unstable and occupy a larger set; increasing temperature ($\tau \uparrow$) corresponds to greater sampling diversity. Low-energy states ($\tau \downarrow$) are stable and occupy a smaller region. Diversity falls, but denser sampling within this restricted state space produces more plausible samples—higher fidelity.

Thus low-temperature sampling trades diversity for fidelity. Examples include the truncation trick, which samples high-feasibility regions, and rejection of poor samples in autoregressive models such as Glow.

“Diffusion Beats GANs” proposes two analogous methods—reducing Gaussian noise at every process or reducing the predicted score—but neither works well.

Lower temperature should improve fidelity or prediction, but the results show no such trend. The method therefore relies on class-guidance scale $s$ to control the trade-off.

Guidance without a classifier?

The situation resembles this:

The paper’s motivation comes from the following pipeline. Classifier guidance complicates diffusion training: alongside the U-Net diffusion model, one must separately train a classifier on downsampled features of noised samples at every time step. A conventional pretrained classifier cannot be used. Even if the number of time steps is minimized, this extra model complicates training.

Classifier-guided sampling can also be interpreted as a gradient-based adversarial attack that fools an image classifier. FID and IS are themselves classifier-based metrics. Generating samples that look meaningful to a classifier therefore becomes a direct objective for improving those metrics. The method may score well not because classifier guidance improves true sampling quality, but because its formulation is particularly suited to improving the metrics. A clever observation. Incidentally, “Diffusion Beats GANs” came from OpenAI, while this paper came from Google Brain—almost a clash for the ages.

Background

The training method is surprisingly simple, though its base setting differs slightly from DDPM. It trains a continuous-time diffusion model. For sample $x$ from dataset $p(x)$ and latent $z_\lambda$ with hyperparameter $\lambda \in [\lambda_{\min},~\lambda_{\max}]$, forward process $q(z \vert x)$ is a variance-preserving Markov process:

$$q(z_\lambda \vert x) = \mathcal{N}(\alpha_\lambda x, \sigma_\lambda^2I), \text{ where }\alpha_\lambda^2 = 1/(1+e^{-\lambda}),~\sigma_\lambda^2 = 1-\alpha_\lambda^2$$

Defining this marginal for arbitrary continuous $z_\lambda$ gives the adjacent-latent conditional

$$q(z_\lambda \vert z_{\lambda^\prime}) = \mathcal{N}((\alpha_\lambda/\alpha_{\lambda^\prime})z_{\lambda^\prime}, \sigma_{\lambda \vert \lambda^\prime}^2) , \text{ where }\lambda < \lambda^\prime,~\sigma^2_{\lambda \vert \lambda^\prime} = (1-e^{\lambda-\lambda^\prime})\sigma_\lambda^2$$

$\lambda$ can be interpreted like SNR in decibels for $\alpha_\lambda$ and $\sigma_\lambda$: it reduces the preceding input signal while increasing added noise. Conditioning on $x$, applying Bayes’ rule to obtain a posterior, and deriving the loss against parameterized reverse process $p_\theta$ are identical to DDPM and are omitted. The network objective is

$$\mathbb{E}_{\epsilon, \lambda}(\parallel \epsilon_\theta(z_\lambda)- \epsilon \parallel_2^2)$$

where $\epsilon \sim \mathcal{N}(0, I)$ and $z_\lambda = \alpha_\lambda x + \sigma_\lambda \epsilon$. This is score matching over a continuous function. Uniform $p(\lambda)$ yields the familiar variational lower bound, but the authors borrow the cosine schedule from classifier-guidance work. Its more gradual noise decay lets the network train evenly across noise distributions.

$$\begin{aligned} &\lambda = -2\log \tan(au+b),~u \sim \mathcal{U}(0, 1) \\ &a = \arctan (e^{-\lambda_{\min}/2})-b,~b = \arctan(e^{-\lambda_{\max}/2}) \end{aligned}$$

Classifier guidance

GANs and flow-based models can trade FID against IS through low-temperature sampling, but bringing the idea to diffusion is difficult because the diffusion process fixes construction of the prior. Classifier guidance approximates this effect by adding a noisy-image classifier gradient to the diffusion score:

$$\hat{\epsilon}_{\theta, \phi}(z_\lambda, c) := \epsilon_\theta(z_\lambda, c) - w\sigma_\lambda\nabla_{z_\lambda} \log p_\phi (c \vert z_\lambda)$$

Classifier influence $w$ is a probability scale factor. It encourages generated data to fall inside the desired label’s category under log likelihood, sacrificing diversity to increase fidelity.

$$\tilde{p}_{\theta, \phi}(z_\lambda \vert c) \propto p_\theta(z_\lambda \vert c)p_\phi(c \vert z_\lambda)^w$$

The toy experiment makes this clear. Classifier guidance pushes Gaussian distributions farther apart, improving separability while shrinking the volume occupied by each.

Classifier-free guidance

Classifier guidance produces the expected IS/FID trade-off, but remains imperfect low-temperature sampling and depends on an image classifier. Classifier-free guidance seeks the same effect without changing $\epsilon_\theta(z_\lambda, c)$ into auxiliary $\hat{\epsilon}_{\theta, \phi}(z_\lambda, c)$. Above all, it removes dependence on classifier parameters $\phi$.

Instead of a classifier, the paper jointly trains unconditional diffusion model $p_\theta(z)$ and conditional model $p_\theta(z, c)$. Rather than build and train two separate networks, one network parameterizes both probabilities:

1. For the unconditional model, insert null token $\emptyset$ instead of class identifier $c$: $\epsilon_\theta(z_\lambda) = \epsilon_\theta(z_\lambda, \emptyset)$.
2. With hyperparameter probability $p_\text{uncond}$, generate null-class samples for unconditional training.
3. Combine conditional and unconditional predictions as $\tilde{\epsilon}(z_\lambda, c) = (1+w)\epsilon_\theta(z_\lambda, c) - w\epsilon_\theta(z_\lambda)$.

The equation contains no classifier gradient $\phi$, avoiding approximation issues such as the earlier first-order Taylor expansion. Since it does not directly manipulate sampling gradients, it is not an adversarial attack.

Experimental results

I expected an equal split between unconditional and conditional sampling, but $0.5$ does not appear uniformly best. The paper experiments with three probabilities.

Conclusion

The three papers address problems that form a progression. Improved DDPM first analyzes several reasons conventional DDPM samples poorly and tests modifications to address them.

The first classifier paper, from OpenAI, focuses on using classifier guidance to obtain high-quality sampling like GANs and flow-based models while also optimizing the diffusion architecture.

Finally, classifier-free guidance shows that a single diffusion network can learn both unconditional $p_\theta(z)$ and conditional $p_\theta(z, c)$ through joint optimization with and without class conditions. It then constructs classifier-guidance-like score estimation independently of classifier parameter $\phi$, without separately training a classifier.