LIT-093: Consistency Models

Song et al. (2023) — ARXIV-2303.01469

Key takeaways

• Trains a model to map any point on a diffusion trajectory directly to its origin — self-consistency along the trajectory — so generation is one function evaluation. Multistep sampling stays available, so the compute/quality dial survives
• Two training modes, and the second is the contribution. Consistency distillation needs a pretrained diffusion model; consistency training needs none, which makes these an independent model family rather than an acceleration technique
• One-step FID 3.55 on CIFAR-10, 6.20 on ImageNet-64, beating prior distillation methods; trained in isolation they beat existing one-step non-adversarial models on CIFAR-10, ImageNet-64 and LSUN-256
• A bias/variance schedule on the discretisation: the training loss has less variance and more bias when N is small and the reverse when N is large, so N is increased during training on a schedule, with the EMA decay μ scheduled alongside it. Fast biased progress early, accurate noisy refinement late
• Zero-shot inpainting, colorization and super-resolution with no task-specific training. The authors flag that Theorem 3 is vacuous for the ℓ₁ norm and prove a separate statement

Standing in the anthology

Read — NOTE-053.

The source of two practices, filed by the #121 audit: SOTA-206

(keep a multi-step option in a few-step model) and SOTA-204 (anneal a discretisation from coarse to fine), the second Proposed because this is the only instance the record holds of a general-looking mechanism.

The transferable finding is the discretisation schedule, and the record has

nothing like it. Its schedule practices are about learning rates (SOTA-140, SOTA-141, SOTA-156) and batch sizes; this schedules a step count on an explicit bias/variance argument, and the same shape applies wherever a loss approximates its target through a discretisation.

The takeaways this replaced had the emphasis backwards: “distillation of diffusion models” is the mode the paper is at pains to say it does not require.

Distribution matching distillation is the one-step line measured against it. DMD (LIT-643) reports 2.62 FID on ImageNet-64 against this paper’s 6.20, but loses on unconditional CIFAR-10, 3.77 against 3.55; its baseline numbers are copied from this paper rather than rerun. DMD2 (LIT-646) puts this paper’s 6.20 in the same ImageNet-64 table as its own 1.51 in one step, and takes its multi-step sampling, alternating denoising with noise injection, from this paper. DMAD (LIT-770) copies the same row into its own Table 1, beside its one-step 1.24.

iCT (LIT-786), by Song and Dhariwal, corrects this paper’s consistency-training argument. With an EMA teacher the limiting objective carries no information about the data, so the teacher should be the student itself. Its tables set improved consistency training against this paper’s rows: 3.25 at one step on ImageNet-64, against consistency distillation’s 6.20 and the original consistency training’s 13.0.

Three more papers compare against it. Inductive Moment Matching (LIT-775) shows the consistency loss is its own one-particle, first-moment case, which collapses on ImageNet-256, and lists consistency distillation with LPIPS at 2.93 in two steps on CIFAR-10 against its own 1.98. Shortcut Models (LIT-787) re-implements consistency distillation and training on DiT-B in velocity space with an L2 loss and a different schedule, where they score 136.5 and 69.7 at one step on ImageNet-256; that measures the methods as implemented there, not at this paper’s strength. Lee et al. (LIT-791) copy consistency distillation at 3.55 on CIFAR-10 and 6.20 on ImageNet-64, against their one-step reflowed model’s 3.07 and 4.31.