anthology-of-the-sota

LIT-062: High-Resolution Image Synthesis with Latent Diffusion Models

2 min read

Properties8
StatusActive
Version3
Tagsgenerative-modeling, signal-structure
Extended byLIT-566 — SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesis, LIT-621 — Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models, LIT-625 — Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets, LIT-626 — Movie Gen: A Cast of Media Foundation Models, LIT-633 — AnimateDiff: Animate Your Personalized Text-to-Image Diffusion Models without Specific Tuning, LIT-635 — Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning
Compared againstLIT-623 — Language Model Beats Diffusion — Tokenizer is Key to Visual Generation, LIT-643 — One-step Diffusion with Distribution Matching Distillation, LIT-646 — Improved Distribution Matching Distillation for Fast Image Synthesis, LIT-796 — InstaFlow: One Step is Enough for High-Quality Diffusion-Based Text-to-Image Generation
Cited as “introduced by” bySOTA-187 — Train the generative model in a learned compressed latent, not at full resolution
Cited as “paper” byNOTE-009 — Latent Diffusion Models
Cited as “source” bySOTA-187 — Train the generative model in a learned compressed latent, not at full resolution

Graph View

Rombach et al. (2022) — ARXIV-2112.10752

Key takeaways

  • Pixel-space diffusion spends most of its capacity on perceptually irrelevant detail, which is a compression problem wearing a generative model’s clothes
  • Split it: a pretrained autoencoder maps images to a compact, perceptually equivalent latent; the diffusion model is trained there and asked only to model semantics
  • The autoencoder is trained once and reused across diffusion models, so its cost amortises
  • Conditioning is introduced into the latent-space denoiser rather than bolted onto pixel space
  • The load-bearing assumption is perceptual equivalence: whatever the autoencoder discards is unavailable to the diffusion model no matter how good it is

Standing in the anthology

The source of SOTA-187.

Read in full for #114; the reading is NOTE-009. The practice is confirmed.

The reading adds the condition the practice most needs: the latent must be perceptually equivalent, and what the autoencoder discards is gone for good. That is a hard ceiling rather than a quality margin, and it is why the compression rate is a decision.

Later work measures itself against this design. MAGVIT-v2 (LIT-623) lists LDM-4 in its ImageNet 256×256 table at FID 10.56 unguided and 3.60 guided (400M parameters, 250 steps), against its own masked token model’s 3.65 and 1.78 (307M, 64 steps). The rows are copied from each paper, not re-run. The two distribution-matching distillers take Stable Diffusion v1.5, a latent diffusion model they cite to this paper, as their text-to-image teacher on COCO-30k. One-step DMD (LIT-643) reaches FID 11.49 against the 50-step teacher’s 8.78, at 0.09s against 2.59s. DMD2 (LIT-646) reaches 8.35, which beats the 50-step teacher (8.59 in its own measurement) but not 200-step SDE sampling at 7.21.

InstaFlow (LIT-796) takes Stable Diffusion 1.4 and 1.5, latent diffusion models of this design, as the teacher it reflows and distils to one step. Distilling SD 1.4 directly reaches one-step FID-5k 40.9 on COCO-2017; reflowing it first, at the same 100K steps and 3.2M generated pairs, reaches 31.0. The scaled InstaFlow-0.9B scores 13.10 FID-30k in one step at 0.09 s. The reflowed model trails 25-step SD 1.5 slightly at 25 steps, 21.5 against 20.1 FID-5k.

Backlinks