LIT-067: Progressive Distillation for Fast Sampling of Diffusion Models

Salimans et al. (2022) — ARXIV-2202.00512

Key takeaways

• Progressive distillation: distil an N-step DDIM sampler into a student taking N/2 steps, whose target is two teacher steps; promote the student and repeat. 8192 → 4 steps, FID 3.0 on CIFAR-10, for no more total compute than training the original model
• This is where v-prediction comes from, and the reason is diagnosed: as signal-to-noise → 0, “the effect of small changes in the neural network output on the implied prediction in x-space is increasingly amplified”, and distillation drives evaluation toward exactly that end. The stated criterion is that the implied x̂ must stay stable as λ_t = log(α²/σ²) varies
• Three parameterisations satisfy it — predict x; predict x and ε on separate channels and merge; or predict v ≡ α_t·ε − σ_t·x. All three also work for training an original diffusion model, which is how v-prediction escaped into general use
• The loss weighting must change too: DDPM’s implicit exp(λ_t) weighting gives weight zero to zero-SNR data. Replacements are max(α²/σ², 1) (“truncated SNR”) and (1 + α²/σ²), which is exactly ‖v − v̂‖²
• Halving beats quartering at fixed budget — better to take fewer parameter updates per halving than to skip distillation iterations

Standing in the anthology

Read — NOTE-048. Nothing is sourced here, but the reading identifies the antecedent of a practice the record does carry.

SOTA-188 is sourced to LIT-075 (EDM), correctly — EDM derives the preconditioning coefficients from a unit-variance requirement. But the problem SOTA-188 solves is stated here four months earlier, with a concrete failure (low-SNR amplification), a concrete criterion (implied x̂ stable across log-SNR), and three concrete solutions. v-prediction is one of them, and the record mentions it nowhere.

Whether that means a second source: on SOTA-188 or a v-prediction practice of its own is a registry decision, deliberately not taken from a note-repair pass.

Three later papers run it as a baseline. Both distribution-matching distillers list progressive distillation’s one-step ImageNet-64 FID as 15.39, against 2.62 for DMD (LIT-643) and 1.51 for DMD2 (LIT-646). DMD’s CIFAR-10 table has it at 8.34 against DMD’s 3.77 unconditional. Those baseline rows are copied, not re-run, and they are not this paper’s own numbers either: the 15.39 and the 8.34 are iCT’s re-implementation (LIT-786), starred in its Tables 2 and 3 as run by its authors. They compare at one step, where halving has the furthest to go. Hang et al. (LIT-710) run this paper’s truncated-SNR weighting, max(SNR, 1), as one of four weightings under x0, ε and v targets on ImageNet-256 latents. Their Min-SNR-5 converges faster in every cell, and with ε output max(SNR, 1) diverges, since it puts weight 1/SNR on the ε loss.

The few-step papers filed later carry it too. iCT (LIT-786) re-ran it for the rows above and beats it from random weights, with one-step consistency training at 2.83 on CIFAR-10 and 4.02 on ImageNet-64 (iCT-deep 2.51 and 3.25). sCM (LIT-790) lists it on ImageNet-64 at 10.70 and 4.70, from a different re-implementation by Heek et al., against one-step sCT’s 2.04. Lee et al. (LIT-791) copy the 15.39 and two-step 8.95 against their reflowed model’s 4.31 and 3.64. Shortcut Models (LIT-787) is the one controlled rerun. Halving in eight 50K phases from the same flow-matching teacher on DiT-B, progressive distillation is the best one-step model in its Table 1, at 14.8 FID on CelebA-HQ-256 and 35.6 on ImageNet-256, against the shortcut model’s 20.5 and 40.3 and one reflow’s 23.2 and 44.8. It has no many-step mode, scoring 302.9 and 201.9 at 128 steps.