LIT-038: Denoising Diffusion Implicit Models

Song et al. (2020) — ARXIV-2010.02502

Key takeaways

• Generalises DDPM to a family of non-Markovian forward processes that share DDPM’s exact training objective — so the sampler can be replaced on an already-trained network with no retraining. Choosing the deterministic member gives DDIM
• 10–50× fewer steps: quality comparable to a 1000-step model within 20–100 steps; on CelebA a 100-step DDPM matches a 20-step DDIM. Sampling time scales linearly with trajectory length, so step count is the whole cost
• Determinism buys what stochastic sampling cannot: with x_T fixed, high-level features survive any trajectory length, so x_T is a latent code — interpolation in it is semantically meaningful, and samples can be encoded and reconstructed with very low error
• This is a discrete-time paper. Its only continuous connection is §4.3, relating the deterministic sampler to neural ODEs

Standing in the anthology

Read — NOTE-067.

The source of SOTA-207 (use the deterministic sampler when the noise input has to mean something), and the corroborating source of SOTA-203 (sample with a higher-order ODE solver), where it is named as the first-order case rather than as a technique of its own.

Filed a year after the reading, and the reading is why. NOTE-067 ends “nothing is sourced to this paper and this reading files no practice” — the policy of a pass that deliberately filed nothing, not a judgement that there was nothing to file. The audit in #121 is what came back for it.

Two of the four takeaways this document carried were about a different paper. “Continuous-time formulation of diffusion models” and “connection to SDE theory” describe Song et al.’s Score-Based Generative Modeling through Stochastic Differential Equations — a separate 2020 paper. DDIM is a discrete-time, non-Markovian reformulation with a deterministic sampler.

That is the #114 failure mode appearing in this population, where the first fifteen readings had found only vagueness. Worth recording: this corpus is not uniformly “true but empty”.

The reading also supplies a cause the record was carrying only the effect of. LIT-075’s reading treats sampler improvements transferring to pretrained networks (ImageNet-64, FID 2.07 → 1.55, no retraining) as EDM’s evidence for its design-space framing. That transfer is possible because of the shared objective established here.

As a baseline for later samplers and distillers. DPM-Solver++ (LIT-676) runs DDIM as the first-order case, on Dhariwal & Nichol’s ImageNet 256×256 model at classifier guidance 8.0. At 10 function evaluations DDIM scores FID 13.04, against 114.62 for DPM-Solver-2 and 164.74 for DPM-Solver-3, because large guidance narrows the higher-order solvers’ convergence radius. The paper’s own multistep second-order solver still beats DDIM: 9.10 at 20 evaluations, against DDIM’s 9.37 at 250. DMD (LIT-643) lists DDIM in its CIFAR-10 table (App. Table 6) at FID 8.23 with 10 network passes, against 3.77 for its unconditional one-step generator. The DDIM figure is copied from Song et al., not rerun.