Introduction

Unlike GANs, whose adversarial training does not allow a latent prior to be specified directly, DDPM emerged as a promising new kind of generative model because it avoids mode collapse and can be trained stably.

During training, however, DDPM adds tiny Gaussian noise values to individual pixels over many iterations ($T = 1000$), and sampling is based on the noised input ($x_t$) at each step. It therefore cannot escape a Markov process.

A Markov process is a useful way to construct a stable prior $p(z)$, but every advantage comes with a cost: DDPM sampling is remarkably slow. In a Markov process, each next state depends on the preceding one in a cascade, which cannot be parallelized on a hardware accelerator such as a GPU. In practice, generating images at the same resolution requires a thousand times more computation than an implicit model such as a GAN. For example, a GAN can generate $50k$ images of size $32 \times 32$ in under a minute, whereas DDPM takes 20 hours. For more detail on the equations and diffusion itself, see the introductory DDPM post and the post deriving the DDPM equations.

This paper focuses on the excessively slow DDPM sampling process and proposes a new method that retains the same objective as DDPM while enabling a non-Markovian process during sampling. In contrast to DDPM’s fully stochastic sampling, this process is deterministic—conditioned on the original image $x_0$—and creates a one-to-one mapping trajectory like a neural-network decoder. The authors call it a DDIM (Denoising Diffusion Implicit Model).

Overview

The idea behind DDPM was to train the reverse process to follow the posterior $q(x_{t-1} \vert x_t, x_0)$, which makes the Markovian forward process tractable. DDIM assumes a non-Markovian process and thus enables deterministic sampling. By explicitly stating that the noised input at time $t$ originates from $x_0$, as shown below, it escapes the approach that depended only on the preceding state.

In other words, we can ensure that the distribution followed by the original $p_\theta(x_{t-1} \vert x_{t})$ is non-Markovian rather than Markovian. The diffusion process can thereby remain implicitly aware, during sampling, of the image $x_0$ that it must recover.

Most importantly, this change does not alter the objective function. There is no need to retrain with a different objective: a pretrained DDPM can be used as-is by applying only the sampling procedure derived in the figure above. A further advantage is that, because deterministic sampling is aware of $x_0$, the reverse process need not traverse all $T = 1000$ steps used by the forward process to create a noised sample. It can instead follow a subsequence $T_\text{sub} = [\tau_n,~\tau_{n-1}, \cdots,~\tau_0]$ and sample without a large loss of quality.

Finally, because every noised image $x_{t(t < T)}$ in the sampling process is conditioned on $x_0$, interpolation in the latent space can produce meaningful semantic interpolation.

Image editing with GANs was comparatively straightforward—for example, through GAN inversion—because the mapping from latent $z$ through a style mapper $F$ and image synthesizer $G$ to the implicitly generated $x = F(G(z))$ nonlinearly warped the information in the spaces of $z$ and $x$. To some degree, interpolating the two latents $z_1$ and $z_2$ that produce $x_1$ and $x_2$ as $\alpha \cdot z_1 + (1-\alpha) \cdot z_2$ could therefore have an effect in image space analogous to $\beta \cdot x_1 + (1-\beta)x_2$. Because the mapping is nonlinear, this affine expression is not exact, but it enables the intuitive semantic operation of producing, for instance, a brunette between a blonde woman and a woman with black hair.

Stochasticity prevented such intuitive sampling in a diffusion process. DDIM offers one way to overcome that limitation.

Deriving the equations

DDIM uses notation different from DDPM, which can be confusing. I prefer the DDPM notation, so I will rewrite the entire DDIM derivation using DDPM notation.

$$q(x_{1:T} \vert x_0) := \prod_{t=1}^T q(x_t \vert x_{t-1}) , \text{ where }q(x_t \vert x_{t-1}):= \mathcal{N}(\sqrt{\alpha_t}x_{t-1}, ~(1-\alpha_t)I)$$

Here $\alpha_t$ is one minus $\beta_t$, the variance scheduled for each time step $t$. As discussed in the equation-derivation post, adding noise through this forward process produces a Gaussian distribution while keeping the variance at $1$. Accumulating noise in this way gives the following expression for $q(x_t \vert x_0)$, where $\bar{\alpha}_t = \prod_{\tau=1}^t \alpha_\tau$.

$$q(x_t \vert x_0) := \int q(x_{1:t} \vert x_0)dx_{1:(t-1)} = \mathcal{N}(x_t;~\sqrt{\bar{\alpha}}_t x_0, (1-\bar{\alpha}_t)I)$$

As in DDPM, the actual implementation uses a reparameterization trick rather than sampling directly from the stochastic process above.

$$x_t = \sqrt{\bar{\alpha}_t} x_0 + \sqrt{1-\bar{\alpha}_t} \epsilon, \text{ where }\epsilon \sim \mathcal{N}(0, I)$$

Remarkably, the DDIM equation can be derived directly from this expression. The proof is based on Equation 2.115 in Bishop’s Pattern Recognition—the formula for marginalizing a particular variable from a Gaussian conditional distribution. In brief, suppose we have

$$\begin{aligned} p(x) =& \mathcal{N}(x \vert \mu,~\Lambda^{-1}) \\ p(y \vert x) =& \mathcal{N}(y \vert Ax+b,~\mathrm{L}^{-1}) \end{aligned}$$

Then the distribution of $y$ after projecting out $x$ is

$$p(y) = \mathcal{N}(y \vert A\mu+b,~\mathrm{L}^{-1}+ A \Lambda^{-1}A^\top)$$

The conditional mean was parameterized by $x$; after marginalization it uses the prior mean, while the variance acquires the additional term at the end. Returning to DDIM, we want to inductively learn the “posterior” of a non-Markovian process satisfying the following constraint:

$$q_\sigma(x_{t-1} \vert x_t,~x_0)$$

If this looks familiar, in the DDPM derivation it appeared when the original forward distribution $q(x_{t-1} \vert x_t)$ was conditioned on $x_0$ for every time step $t>1$ and expanded with Bayes’ rule while retaining the Markov process:

$$q(x_{t-1} \vert x_t,x_0) = q(x_t \vert x_{t-1},x_0) \frac{q(x_{t-1} \vert x_0)}{q(x_t \vert x_0)}$$

This time, however, we seek a new non-Markovian distribution $q_{\sigma}(x_{t} \vert x_{t-1}, x_0)$ with the same marginal $q(x_{t} \vert x_0)$ at every time step $t$. The inductive proof proceeds as follows.

1. For $t = 1$, it is clear that $q(x_{1} \vert x_0) = q_{\sigma}(x_1 \vert x_0)$.
2. At $t = \tau$, suppose $q(x_{\tau-1} \vert x_0) = \int q_{\sigma}(x_{\tau-1} \vert x_{\tau}, x_0) dx_\tau$ holds ($\tau \ge 1$).
3. If we can also make $q(x_{\tau} \vert x_0) = \int q_{\sigma}(x_{\tau} \vert x_{\tau+1}, x_0) dx_{\tau+1}$ hold for $t = \tau + 1$, every marginal of the DDIM forward process can be matched to DDPM for all $t \ge 1$.

Keeping Bishop’s equation firmly in mind, consider the marginal that must be matched:

$$q(x_{t-1} \vert x_0) = \mathcal{N}(x_{t-1};~\sqrt{\bar{\alpha}}_{t-1} x_0, (1-\bar{\alpha}_{t-1})I)$$

Conditioning this expression on $x_{t}$ gives the following. Shown together with the distribution $q(x_t \vert x_0)$, it is

$$\begin{aligned} q(x_t \vert x_0) =& \mathcal{N}(x_{t};~\sqrt{\bar{\alpha}_t}x_0,(1-\bar{\alpha}_{t})I) \\ q_\sigma(x_{t-1} \vert x_t, x_0) =& \mathcal{N}(x_{t-1};~\sqrt{\bar{\alpha}_{t-1}}x_0 + C\cdot(x_t - \sqrt{\bar{\alpha}_t}x_0),\sigma_t^2 I) \end{aligned}$$

Here $\sigma_t$ can be regarded as an arbitrary standard deviation of any Gaussian distribution satisfying the same marginal. The constant $C$ is introduced into the mean for the following reason. According to Equation 2.115 in Bishop,

$$\mathbb{E}\left(q_\sigma(x_{t-1} \vert x_0)\right) = \mathbb{E}\left(q_\sigma(x_{t-1} \vert x_t, x_0)\right) \rvert_{x_t = \mathbb{E}(q(x_t \vert x_0))}$$

Therefore, the expression $\sqrt{\bar{\alpha}_{t-1}}x_0 + C\cdot(x_t - \sqrt{\bar{\alpha}_t}x_0)$ was intentionally constructed so that substituting $\sqrt{\bar{\alpha}_t}x_0$ for $x_t$ produces $\sqrt{\bar{\alpha}_{t-1}}x_0$. Since we introduced the arbitrary constant $C$, we solve for it by applying the variance identity in the same way:

$$\mathrm{Var}(q_\sigma(x_{t-1} \vert x_0)) = \mathrm{Var}(q_\sigma(x_{t-1} \vert x_t, x_0))+C^2 \cdot \mathrm{Var}(q(x_t \vert x_0))$$

Substituting the known variances of the marginal distributions gives

$$(1-\bar{\alpha}_{t-1})I = \sigma_t^2I + C^2 \cdot (1-\bar{\alpha}_t)I$$

Removing the identity matrix $I$ and solving for the coefficient $C$ gives

$$C = \frac{\sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}}{\sqrt{(1-\bar{\alpha}_t)}}$$

The non-Markovian posterior for an arbitrary variance $\sigma_t^2$ is therefore

$$q_\sigma(x_{t-1} \vert x_t, x_0) = \mathcal{N}(x_{t-1};~\sqrt{\bar{\alpha}_{t-1}}x_0 + \frac{\sqrt{1-\bar{\alpha}_{t-1}-\sigma_t^2}}{\sqrt{(1-\bar{\alpha}_t)}} \cdot(x_t - \sqrt{\bar{\alpha}_t}x_0),\sigma_t^2 I)$$

To emphasize once more, this equation has a completely different meaning from the posterior derived for DDPM. Converting the non-Markovian posterior above into a likelihood gives

$$q_\sigma(x_{t} \vert x_{t-1}, x_0) = q_\sigma(x_{t-1} \vert x_t,x_0) \times \frac{q_\sigma(x_t \vert x_0)}{ q_\sigma(x_{t-1} \vert x_0)}$$

We have thus found a non-Markovian forward process $q_\sigma$ with the same marginals as the Markovian forward process $q$, which is exactly what the paper set out to do.

From a non-Markovian posterior to the DDPM loss

Now let us verify why optimizing the $q$ defined above yields the same loss as DDPM. Only the $q$ term changes relative to the DDPM loss, so we can express the result as follows by replacing the forward process in DDPM’s lower bound with the non-Markovian one.

$$\mathbb{E}(-\log p_\theta (x_0)) \leq \mathbb{E}_q \left( -\log p_\theta(x_T) - \sum_{t \ge 1}\log \frac{p_\theta(x_{t-1} \vert x_t)}{q_\sigma(x_t \vert x_{t-1}, x_0)} \right)$$

The conversion to a posterior and the rest of the algebra were already covered for DDPM, so we can skip directly to the result. The variational-inference loss is

$$D_{KL}(q(x_T \vert x_0) \vert\vert p_\theta(x_T)) -\sum_{t > 1} D_{KL} (q_\sigma(x_{t-1} \vert x_t, x_0) \vert\vert p_\theta(x_{t-1} \vert x_t)) -\mathbb{E}_q\left(\log p_{\theta}(x_0 \vert x_1) \right)$$

The only change is that the distribution $q$ representing the forward process now carries $\sigma$, the standard deviation of the non-Markovian posterior. As in DDPM, we discard the first term and express the training of the diffusion model on the remaining terms in terms of the reverse process $p_\theta$:

$$p_\theta^{(t)}(x_{t-1} \vert x_t) = \begin{cases} \mathcal{N}(x_0(x_t, \epsilon_\theta), \sigma_1^2I), &\text{if }t = 1\\ q_\sigma (x_{t-1} \vert x_t,~x_0(x_t,\epsilon_\theta)),& \text{otherwise}\end{cases}$$

Here $x_0(x_t, \epsilon_\theta)$ denotes $x_0 = (x_t - \sqrt{1-\bar{\alpha}_t} \cdot \epsilon_\theta^{(t)}(x_t)) / \sqrt{\bar{\alpha}_t}$, reconstructed from the noise $\epsilon_\theta^{(t)}$ that the network predicts was added to each $x_t$. Looking only at the case $t>1$, the KL divergence between Gaussian distributions simplifies because their exponential terms cancel:

$$\begin{aligned} &D_{KL} (q_\sigma(x_{t-1} \vert x_t, x_0) \vert\vert p_\theta^{(t)}(x_{t-1} \vert x_t)) \\ &=D_{KL} (q_\sigma(x_{t-1} \vert x_t, x_0) \vert\vert q_\sigma(x_{t-1} \vert x_t,~x_0(x_t,~\epsilon_\theta))) \\ &=\mathbb{E}_q\left(\frac{\parallel x_0 - x_0(x_t,\epsilon_\theta) \parallel_2^2}{2\sigma_t^2} \right) \end{aligned}$$

Rewriting the expression through $\epsilon$, the noise actually added during reparameterization, gives

$$=\mathbb{E}_q\left( \frac{\parallel (x_t-\sqrt{1-\bar{\alpha}_t}\epsilon)/\sqrt{\bar{\alpha}_t} - (x_t-\sqrt{1-\bar{\alpha}_t}\epsilon_\theta^{(t)}(x_t))/\sqrt{\bar{\alpha}_t} \parallel_2^2}{2\sigma_t^2} \right)$$

The familiar form is beginning to appear. After collecting the coefficients, we recover the familiar simplified objective:

$$=\mathbb{E}_q \left( (1-\bar{\alpha}_t)\cdot\frac{\parallel \epsilon - \epsilon_\theta^{(t)}(x_t) \parallel^2}{2\sigma_t^2\bar{\alpha}_t} \right) = \mathbb{E}_q\left( \frac{1-\bar{\alpha}_t}{2\sigma_t^2\bar{\alpha}_t} \cdot \parallel \epsilon - \epsilon_\theta^{(t)}(x_t) \parallel^2 \right)$$

This proves that the non-Markovian posterior introduced for DDIM sampling indeed converges to the DDPM loss.

Sampling with DDIM

Using the predicted $\epsilon_\theta$ in the non-Markovian forward process defined above yields the following sampling equation:

$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\underset{\text{predicted }x_0}{\left( \frac{x_t - \sqrt{1-\bar{\alpha}_t}\epsilon_\theta^{(t)}(x_t)}{\sqrt{\bar{\alpha}_t}} \right)} + \underset{\text{direction pointing to }x_t}{\sqrt{1-\bar{\alpha}_{t-1} - \sigma_t^2} \cdot \epsilon_\theta^{(t)}(x_t)} + \underset{\text{random noise}}{\sigma_t z},~z \sim \mathcal{N}(0, I)$$

The equation first predicts $x_0$ and then reconstructs $x_t$. There is no strict need to predict the sample at $t-1$; predicting the sample at $t-2$ is also reasonable. DDIM sampling can therefore use only selected values of $t$ rather than consuming all $T$ time steps. For example, given a subsequence $[x_{\tau_1},~\cdots,~x_{\tau_S}] (S < T)$ of the full length-$T$ time sequence $[1,~2,~\cdots,~T]$, we can redefine the non-Markovian forward process as $q(x_{\tau_i} \vert x_0) = \mathcal{N}(\sqrt{\bar{\alpha}_{\tau_i}}x_0,~(1-\bar{\alpha}_{\tau_i})I)$. Because it no longer depends only on the preceding state, dependence on a contiguous sequence disappears as well.

Experiments

The sampling equation contains a new hyperparameter, $\sigma$:

$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\underset{\text{predicted }x_0}{\left( \frac{x_t - \sqrt{1-\bar{\alpha}_t}\epsilon_\theta^{(t)}(x_t)}{\sqrt{\bar{\alpha}_t}} \right)} + \underset{\text{direction pointing to }x_t}{\sqrt{1-\bar{\alpha}_{t-1} - \sigma_t^2} \cdot \epsilon_\theta^{(t)}(x_t)} + \underset{\text{random noise}}{\sigma_t z},~z \sim \mathcal{N}(0, I)$$

Computing the variance separately gives the sum of the squared coefficients attached to epsilon:

$$\mathrm{Var}(x_{t-1}) = \frac{\bar{\alpha}_{t-1} \cdot (1-\bar{\alpha}_t)}{\bar{\alpha}_t} + (1-\bar{\alpha}_{t-1} - \sigma_t^2) + \sigma_t^2 = \frac{(1+\alpha_t-2\bar{\alpha}_t)}{\alpha_t}$$

Thus, $\sigma$ controls the transition between DDPM’s stochasticity and DDIM’s determinism without changing the actual sampling variance. Recall the DDPM sampling equation:

$$x_{t-1} = \frac{1}{\sqrt{\alpha_t}}\left( x_t - \frac{\beta_t}{\sqrt{1-\bar{\alpha}}_t} \epsilon_\theta(x_t, t)\right)+\sigma_tz$$

In deriving this equation, the standard deviation of the tractable probability distribution $q(x_{t-1} \vert x_t, x_0)$ for the Markovian forward process was

$$\tilde{\sigma}_t^2 = \beta_t \times \left( \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t} \right)$$

Remarkably, when DDIM’s $\sigma_t$ takes this form, its sampling equation becomes exactly the DDPM equation. Defining $\eta$ as a hyperparameter therefore lets us interpolate between DDPM and DDIM sampling:

$$\sigma_{t} = \eta\sqrt{(1-\alpha_t) \times \left( \frac{1-\bar{\alpha}_{t-1}}{1-\bar{\alpha}_t} \right)}$$

$\eta = 1$ gives pure DDPM, while $\eta = 0$ gives pure DDIM.

The figure reports FID after sampling with fewer than $T = 1000$ steps for DDPM ($\eta = 1$), DDIM ($\eta = 0$), and intermediate values. DDPM performance deteriorates sharply as the number of steps decreases, while DDIM preserves its quality.

Unsurprisingly, more steps yield better sampling performance—but the sampling time becomes excessive.

Roughly $20 \sim 100$ steps provide generation quality comparable to $1000$ steps, corresponding by a simple calculation to a 10-to-50-fold speedup.

Best of all, latent interpolation now has a meaningful relationship with the semantics of the generated images. It is hilarious how the Shutterstock watermark slowly appears along the way—maybe that is just me, though.

Neural ODEs with DDIM

Although DDIM and DDPM are diffusion models, their discrete treatment of diffusion makes it difficult to call them differential equations. DDIM sampling, however, has a form that admits an Euler approximation. Recall the sampling equation:

$$x_{t-1} = \sqrt{\bar{\alpha}_{t-1}}\underset{\text{predicted }x_0}{\left( \frac{x_t - \sqrt{1-\bar{\alpha}_t}\epsilon_\theta^{(t)}(x_t)}{\sqrt{\bar{\alpha}_t}} \right)} + \underset{\text{direction pointing to }x_t}{\sqrt{1-\bar{\alpha}_{t-1} - \sigma_t^2} \cdot \epsilon_\theta^{(t)}(x_t)} + \underset{\text{random noise}}{\sigma_t z},~z \sim \mathcal{N}(0, I)$$

There it is. For DDIM, as discussed above, we simply set $\sigma = 0$.

$$\frac{x_{t-1}}{\sqrt{\bar{\alpha}_{t-1}}} = \frac{x_t - \sqrt{1-\bar{\alpha}_t}\epsilon_\theta^{(t)}(x_t)}{\sqrt{\bar{\alpha}_t}} + \frac{\sqrt{1-\bar{\alpha}_{t-1}} \cdot \epsilon_\theta^{(t)}(x_t)}{\sqrt{\bar{\alpha}_{t-1}}}$$

Because this was originally an equation for discrete sampling, replacing every time interval with $\Delta$ converts it into a differential-equation form:

$$\frac{x_{t-\Delta}}{\sqrt{\bar{\alpha}_{t-\Delta}}} = \frac{x_t }{\sqrt{\bar{\alpha}_t}} + \left(\frac{\sqrt{1-\bar{\alpha}_{t-\Delta}} }{\sqrt{\bar{\alpha}_{t-\Delta}}} - \frac{\sqrt{1-\bar{\alpha}_t}}{\sqrt{\bar{\alpha}_t}} \right)\cdot \epsilon_\theta^{(t)}(x_t)$$

Substituting $x_t/\sqrt{\bar{\alpha}_t} = \bar{x}_t$ and $\frac{\sqrt{1-\bar{\alpha}_t}}{\sqrt{\bar{\alpha}_t}} = \sigma$ gives

$$d\bar{x}(t) = \epsilon_\theta^{(t)}\left(\frac{\bar{x}(t)}{\sqrt{\sigma^2+1}}\right)d\sigma(t)$$

This is a differential equation. Compared with DDPM, the continuous diffusion process formulated through an SDE presents $x(T)$ as the solution to a Neural ODE, allowing a deterministic one-to-one mapping.

Because an ODE is available, $x_0$ can be encoded directly into $x_T$ or reconstructed back into $x_0$. The trajectory error depends on $\Delta$, so a larger $S$—a shorter time interval—brings the result closer to the true solution of the differential equation. The figure above shows reconstruction error on CIFAR-10.