anthology-of-the-sota

LIT-036: Denoising Diffusion Probabilistic Models

4 min read

Properties8
StatusActive
Version2
Tagsgenerative-modeling
ExtendsLIT-439 — Deep Unsupervised Learning using Nonequilibrium Thermodynamics
Extended byLIT-446 — Variational Diffusion Models, LIT-627 — Video Diffusion Models, LIT-643 — One-step Diffusion with Distribution Matching Distillation, LIT-699 — Diffusion Models Beat GANs on Image Synthesis, LIT-710 — Efficient Diffusion Training via Min-SNR Weighting Strategy
Compared againstLIT-630 — Flow Matching for Generative Modeling, LIT-636 — Flow Straight and Fast: Learning to Generate and Transfer Data with Rectified Flow, LIT-644 — Building Normalizing Flows with Stochastic Interpolants, LIT-699 — Diffusion Models Beat GANs on Image Synthesis, LIT-789 — Multisample Flow Matching: Straightening Flows with Minibatch Couplings, LIT-793 — Flow map matching with stochastic interpolants: A mathematical framework for consistency models
Cited as “paper” byNOTE-068 — Denoising Diffusion Probabilistic Models
Cited as “source” bySOTA-187 — Train the generative model in a learned compressed latent, not at full resolution

Graph View

Ho et al. (2020) — ARXIV-2006.11239

Key takeaways

  • The paper that made diffusion produce good samples, via parameterisation rather than a new model class: predict the noise ε, keep the reverse-process variances fixed, and train on a simplified, unweighted objective instead of the true variational bound
  • Establishes the equivalence that reframed the field — this parameterisation is denoising score matching over multiple noise levels, with sampling as annealed Langevin dynamics
  • Table 2 is the underquoted part. Predicting ε is only about as good as predicting the posterior mean under the variational bound with fixed variances, and much better under the simplified objective. Target and weighting are not separable choices. Learning the reverse variances “leads to unstable training and poorer sample quality”
  • The correct objective is not the best one, and the paper says so: the true variational bound “yields better codelengths… but the latter [simplified objective] yields the best sample quality.” The field’s foundational result is a deliberate mis-weighting of a principled loss
  • FID 3.17 / IS 9.46 on unconditional CIFAR-10; LSUN 256×256 comparable to ProgressiveGAN. §4.3 measures that most of the model’s codelength describes imperceptible detail

Standing in the anthology

Read — NOTE-068. Nothing is sourced here; SOTA-188 is correctly sourced to EDM, which derives what this paper set by hand.

The reading supplies provenance the record was missing. SOTA-187 rests on the claim that most of a pixel-space model’s capacity goes to perceptually irrelevant detail, which LIT-062’s reading records as “the paper’s premise, supported by the compression working at all”. §4.3 here is the measurement, two years earlier. And SOTA-188’s subject — that the loss weighting across noise levels is a decision — is already visible as the gap between the variational bound and the simplified objective.

Of the takeaways replaced, “fixed noise schedule” named the forward schedule; the consequential fixed quantity is the reverse-process variance, which is what was ablated and what destabilises training when learned.

Where the construction comes from. Sohl-Dickstein et al. (LIT-439) supplied the model: a slow forward diffusion that destroys structure, and a learned reverse process that restores it, each step small enough to approximate simply. This paper keeps that construction and adds the parameterisation that made it produce good samples: ε-prediction, which makes it denoising score matching across noise levels; fixed reverse variances; and the unweighted simplified objective.

How later papers measured against it.

  • Flow Matching (LIT-630) trains a DDPM baseline on its own U-Net (Table 1). DDPM scores FID 7.48 on CIFAR-10, 6.99 on ImageNet-32 and 17.36 on ImageNet-64, against 6.35, 5.02 and 14.45 for the straight OT path. On CIFAR-10 it beats flow matching on the diffusion path (8.06). The paper’s budget statement contradicts itself, and DDPM may have been allowed more iterations.
  • Rectified Flow (LIT-636) quotes DDPM at 1,000 steps as FID 3.21 in its literature table (Table 1b), where this paper reports 3.17. Against that it sets 2.58 for its 1-rectified flow, on a different (DDPM++) architecture. Its controlled comparison is against the VP probability-flow ODE, which it calls the continuous-time limit of DDIM, derived from DDPM. On the same architecture that is 2.58 against 3.93, while the 2,000-step VP SDE, which it calls equivalent to DDPM, scores 2.55.
  • Stochastic interpolants (LIT-644) put InterFlow’s CIFAR-10 FID of 10.27 beside this paper’s quoted 3.17, with no baseline retrained. On that quoted comparison, DDPM’s samples are much the better.
  • Dhariwal and Nichol (LIT-699) re-score DDPM’s LSUN samples in their own evaluation and beat them with ADM at 1,000 steps: FID 1.90 against 4.89 on bedrooms and 5.57 against 17.1 on cats.
  • Multisample Flow Matching (LIT-789) reproduces DDPM at its own hyperparameters on ImageNet-32 and 64. DDPM needs more evaluations for a worse FID than the minibatch-OT flow, 5.72 at 330 NFE against BatchOT’s 4.68 at 146 on ImageNet-32, and falls apart under few-step Euler sampling, 362 FID at four steps (that paper’s Table 7).
  • Flow map matching (LIT-793) quotes those ImageNet-32 rows rather than re-running them: DDPM at 63.08 FID with 20 steps and 362.37 with four, against its own teacher-free flow map’s 9.68 and 16.90.

Backlinks