anthology-of-the-sota

LIT-075: Elucidating the Design Space of Diffusion-Based Generative Models

4 min read

Properties8
StatusActive
Version2
Tagsgenerative-modeling
Extended byLIT-714 — Analyzing and Improving the Training Dynamics of Diffusion Models
Compared againstLIT-555 — Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models, LIT-643 — One-step Diffusion with Distribution Matching Distillation, LIT-646 — Improved Distribution Matching Distillation for Fast Image Synthesis, LIT-692 — Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation, LIT-710 — Efficient Diffusion Training via Min-SNR Weighting Strategy, LIT-714 — Analyzing and Improving the Training Dynamics of Diffusion Models, LIT-770 — DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation, LIT-786 — Improved Techniques for Training Consistency Models, LIT-790 — Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models, LIT-791 — Improving the Training of Rectified Flows, LIT-794 — Normalizing Flows are Capable Generative Models
Cited as “introduced by” bySOTA-188 — Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal
Cited as “paper” byNOTE-019 — Elucidating the Design Space of Diffusion-Based Generative Models
Cited as “source” bySOTA-188 — Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal

Graph View

Karras et al. (2022) — ARXIV-2206.00364

Key takeaways

  • Argues the theory and practice of diffusion models are “unnecessarily convoluted”, and presents a design space that separates the concrete design choices rather than a new model
  • Improves three of those choices independently: the sampler, the preconditioning of the score network, and the distribution of training noise levels
  • The preconditioning is derived, not tuned. Writing the denoiser as D(x;σ) = c_skip·x + c_out·F(c_in·x; c_noise), the paper solves for c_in and c_out by requiring unit-variance inputs and targets, for c_skip by minimising the amplification of the network’s own error, and fits c_noise empirically — the only one of the four with no derivation
  • FID 1.79 class-conditional and 1.97 unconditional on CIFAR-10, at 35 network evaluations per image
  • Modularity is the evidence for the framing: a previously trained ImageNet-64 model goes from FID 2.07 to 1.55 by design changes alone, and to 1.36 after retraining. Most of the gain arrives without retraining
  • Stochastic sampling helps the ImageNet-64 model and hurts CIFAR-10 once the network is trained under the paper’s own improvements — the authors read it as compensating for model error

Measured against it

EDM is the baseline or the teacher for most of the diffusion work the record holds since:

  • Kingma and Gao (LIT-692) derive EDM’s implied weighting, N(λ; 2.4, 2.4²)·(e^{−λ} + 0.25), which is not monotone. They re-implement EDM at ImageNet-64 and tie it: 1.45 for their reproduction, 1.43 for their best monotone weighting. Neither reaches this paper’s published 1.36.
  • EDM2 (LIT-714), from the same group, re-runs EDM as its baseline. On ImageNet-512 latents config A scores 8.00 and the final magnitude-preserving config 2.56. In ImageNet-64 pixel space, EDM’s 2.22 deterministic becomes 1.58 at matched complexity. It also finds that EDM’s loss weight equalises the noise levels only at initialization.
  • Min-SNR (LIT-710) multiplies EDM’s loss by min(SNR, 5)/SNR and converges faster over the first 200M training images. It stops there for compute and does not reach EDM’s full schedule.
  • DMD (LIT-643) and DMD2 (LIT-646) distil the ImageNet-64 EDM model into one step. DMD scores 2.62 against EDM’s 2.32 at 512 passes with the ODE sampler. DMD2 scores 1.51, which beats the ODE teacher, and 1.28 with longer training, the only run that also beats EDM’s stochastic 1.36.
  • DMAD (LIT-770) distils the same model into one step with discriminator heads in place of DMD’s fake score. It reports 1.24, and 1.04 with an ImageNet-pretrained projected discriminator, after 570K iterations at batch 384, a longer budget than DMD2’s 1.51.
  • Warm Diffusion (LIT-555) keeps EDM’s network, training and Heun sampler and adds blur. Its blur-noise ratio 0 is pure-noise EDM at 1.97 on CIFAR-10, and the best ratio, 0.5, gives 1.85. That is a 0.12 gain, best of three sampling rounds and with no variance.
  • iCT (LIT-786) re-runs EDM as the many-step reference for consistency training from random weights, and EDM stays ahead of the one- and two-step students: 2.04 on CIFAR-10 against iCT-deep’s 2.51 and 2.24, and 2.44 at 79 steps on ImageNet-64 against 3.25 and 2.77.
  • sCM (LIT-790) starts every model from the EDM weights on CIFAR-10, consistency training included. Its two-step sCT reaches 2.06 against the teacher’s 2.01.
  • Lee et al. (LIT-791) initialize every 2-rectified flow from EDM’s denoiser (their Prop. 1) and train on pairs from EDM’s Heun sampler. One step reaches 3.07 on CIFAR-10 and 4.31 on ImageNet-64, against EDM’s 1.97 and 2.30 at many steps, so the student does not reach its teacher.
  • TarFlow (LIT-794) quotes EDM at 1.55 on class-conditional ImageNet-64 against its own normalizing flow’s 2.66. That 1.55 is unguided and is not EDM’s best on the benchmark, the 1.36 this note records, while TarFlow’s 2.66 uses a searched guidance weight.

Standing in the anthology

The source of SOTA-188, and confirmed by a full reading — NOTE-019, Section 5 and Appendix B.6.

The reading was Skimmed first: ar5iv has no rendering for this identifier, so the abstract was all that was available through the route the rest of #114 used, and SOTA-188 was left visibly unverified rather than quietly assumed. dmarx supplied the full text; the note is now Read and the practice checks out, with one correction — SOTA-188 had implied all four preconditioning scalings follow from the unit-variance requirement, and two of them follow from other criteria.

That round trip is what the Skimmed status is for. The gap was recorded where the next reader would find it, and it was the note’s own open question that got closed.

Backlinks