anthology-of-the-sota

LIT-646: Improved Distribution Matching Distillation for Fast Image Synthesis

7 min read

Properties6
StatusActive
Tagsgenerative-modeling, few-step-generation, inference-optimization, vision-and-graphics, training-optimization
ExtendsLIT-643 — One-step Diffusion with Distribution Matching Distillation
Extended byLIT-631 — From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
Compared againstLIT-062 — High-Resolution Image Synthesis with Latent Diffusion Models, LIT-067 — Progressive Distillation for Fast Sampling of Diffusion Models, LIT-075 — Elucidating the Design Space of Diffusion-Based Generative Models, LIT-093 — Consistency Models, LIT-566 — SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, LIT-770 — DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation, LIT-783 — Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency, LIT-790 — Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models
Cited as “source” byTHEORY-121 — Distribution-matching distillation loses coverage because it descends a reverse divergence on the student's own samples, and a consistency term on teacher trajectories restores it

Graph View

Yin, Gharbi, Park, Zhang, Shechtman, Durand and Freeman, MIT and Adobe Research (2024) — ARXIV-2405.14867. Known as “DMD2”. Read at v2.

Key takeaways

  • The regression loss can go (Table 3, one-step ImageNet-64 from EDM). DMD with regression gets 2.62 FID. Removing it gives 3.48, and brightness oscillates without converging (§4.2, App. C, Fig. 8). With the fake score updated 5 times per generator update (TTUR) it gets 2.61, with no paired dataset. The authors blame a fake critic that lags the generator. Fig. 9 compares 1, 5 and 10 updates and a 5× critic learning rate, as curves only. It also shows TTUR converging faster than DMD with regression.
  • A GAN loss adds the most (Table 3). A classifier head on the fake denoiser’s bottleneck (App. F.1) scores noised real images against noised generator outputs (Eq. 4). Adding it takes 2.61 to 1.51. GAN alone gets 2.56, and GAN with TTUR 2.52. The standard setup is 200K iterations at batch 280 on 7 A100s with GAN weight 3e-3 (App. F.2). The DMD row is the original paper’s number, and the paper does not say whether the other rows share one budget.
  • Backward simulation feeds a multi-step student its own noised outputs during training, not noised real images (§4.5, Fig. 4). The only ablation is 4-step SDXL on 10K COCO prompts (Table 4). Without it: FID 20.66, patch FID 24.21, CLIP 0.332. With it: 19.32, 20.86, 0.332.
  • The SDXL ablation (Table 4, same setup): without the GAN term, FID 26.90 and “oversaturated and oversmoothed” images. Without distribution matching, FID 13.77, the best in the table, with CLIP 0.307, the worst. The authors read FID as rewarding a pure GAN for matching the unguided real distribution.
  • Headline results. ImageNet-64: 1.51 in one step, 1.28 with longer training, against EDM’s 2.32 (ODE) and 1.36 (SDE) (Table 1). COCO-30k from SD v1.5: 8.35 against DMD’s 11.49 (App. Table 5). SDXL, 4 steps: 19.32 FID against 19.36 for the teacher at guidance 6 and 20.39 at guidance 8 (Table 2).
  • Diversity is measured, once (App. B, Table 6). The score is mean pairwise LPIPS over 4 images per prompt on 128 PartiPrompts. It is 0.61 for DMD2 against 0.64 for the SDXL teacher at guidance 6 or 8. LCM gets 0.61, SDXL-Turbo 0.58 and SDXL-Lightning 0.63. §6 calls it “a slight degradation”.

Where the hedges are

Per DP-010:

  • “We eliminate the regression loss” (abstract). The one-step SDXL model was first pretrained with a regression loss on 10K pairs, with a timestep shift to 399 (App. F.4). The authors call this possibly “specific to SDXL”.
  • “Surpassing the original teacher” (abstract). On ImageNet, the standard 1.51 beats the ODE teacher but not the SDE teacher (1.36). Only the 550K-iteration two-stage run (1.28, App. F.2) beats both. On COCO, 8.35 beats the 50-step ODE teacher (8.59) but not 200-step SDE sampling at guidance 2 (7.21, Table 5). The student’s real score used guidance 1.75 against the teacher’s 3 (App. F.3), and FID favours lower guidance, as Table 4’s pure-GAN row shows.
  • “Outperforms its teacher in image quality for 24% of samples” (§5.2). Fig. 5 shows 62.0% against 38.0%, so 24 is the margin in points, not a share of samples. Prompt alignment is 50.5% against 49.5%. Each pair was rated by 5 evaluators on 128 prompts (§5.2, App. H).
  • The diversity loss is not given a cause. The paper never uses “reverse KL” or “mode”. It argues SDXL-Lightning’s higher score is “partially due to random outputs”. It says a larger GAN weight would raise diversity by pulling toward “the more diverse unguided distribution”. That is not measured (App. B). The real score runs at guidance 8 (App. F.4), so guidance is as live a suspect as the KL direction. No ImageNet diversity or recall is reported after the regression loss is removed.
  • The update ratio is per dataset. It is 5 on ImageNet and SDXL and 10 on SD v1.5 (App. F). The authors advise the smallest ratio that keeps “pixel brightness” stable (App. C).

Standing in the anthology

CausVid (LIT-631) is a direct descendant. NOTE-343 records it using “a DMD2 update ratio of 5”. Its four fixed timesteps [999, 748, 502, 247] are close to §4.4’s 999, 749, 499, 249. Self Forcing (LIT-629) runs DMD inside its rollout framework. SOTA-394’s condition that “the record does not hold DMD or DMD2” is answered by this filing and the DMD note.

On SOTA-394’s diversity condition, this is the only measurement in the line: a 4-step image student loses 0.03 LPIPS diversity against its teacher, with competing distillers on both sides. It does not separate the reverse-KL objective from real-score guidance. The practice’s “characteristic of reverse KL” is CausVid’s explanation, and neither DMD paper tests it.

Backward simulation is the image-distillation form of SOTA-395’s self-rollout training, published a year before Self Forcing. Its evidence is one SDXL ablation row, measured by patch FID.

The TTUR is borrowed from Heusel et al. (LIT-611), who proposed it for GAN discriminators. SDXL (LIT-566) is the main teacher.

It extends DMD (LIT-643) by keeping its distribution-matching gradient, a frozen real score minus an online fake score, and removing what DMD needed to make that train: the LPIPS regression onto precomputed teacher outputs goes, and its stabilizing role passes to updating the fake score five times per generator step. It adds the GAN loss on the fake denoiser’s bottleneck and multi-step students trained by backward simulation. Table 3 measures the steps from DMD on ImageNet-64: 2.62, then 3.48 without regression, 2.61 with TTUR and 1.51 with the GAN term.

On ImageNet-64 (Table 1) its one-step 1.51 sits against progressive distillation (LIT-067) at 15.39 and Consistency Models (LIT-093) at 6.20, and against its EDM teacher (LIT-075) at 2.32 with the ODE sampler and 1.36 with the SDE sampler; only the 1.28 longer run beats both. Its multi-step sampling, alternating denoising with noise injection, is taken from Consistency Models. On COCO-30k, distilling SD v1.5, the latent diffusion model of LIT-062, it scores 8.35 against the 50-step teacher’s 8.59 and 200-step SDE sampling’s 7.21.

DMAD (LIT-770) uses this paper as its main baseline on the same ImageNet-64 and SDXL protocols. It replaces the fake critic and its five updates per step with discriminator heads updated once per step, and reports 1.24 against this paper’s 1.28 on ImageNet-64 and 14.47 against 19.32 on four-step SDXL, from copied rows and a longer ImageNet budget.

Two later papers set it beside continuous-time consistency. sCM (LIT-790) copies the one-step 1.28 on ImageNet-64, ahead of its own sCD at 2.44 and sCT at 2.04. rCM (LIT-783) re-implements it on 14B-scale image and video teachers, with a discriminator on the fake-score network’s features and no configuration given. On Wan2.1-1.3B at four steps DMD2 stays slightly ahead on VBench, 84.56 against rCM’s 84.43. On Cosmos-Predict2 GenEval DMD2 scores 0.77 and 0.80, where rCM’s four-step students score 0.79 to 0.83. rCM keeps this paper’s fake-score network and its several critic updates per student step, and drops the GAN.

Filed without a NOTE: the takeaways come from one full reading done for this filing, of the main text and Appendices A–I, with Fig. 5 read from the rendered page. Figs. 8–9 have no numeric values, so the TTUR frequency comparison is described, not quantified.

Backlinks