anthology-of-the-sota

LIT-448: Scalable Diffusion Models with Transformers

4 min read

Properties4
StatusActive
Tagsgenerative-modeling, model-architecture
Extended byLIT-447 — SiT: Exploring Flow and Diffusion-based Generative Models with Scalable Interpolant Transformers, LIT-619 — Wan: Open and Advanced Large-Scale Video Generative Models, LIT-622 — CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer, LIT-624 — Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model, LIT-652 — Video generation models as world simulators, LIT-658 — FiT: Flexible Vision Transformer for Diffusion Model
Compared againstLIT-623 — Language Model Beats Diffusion — Tokenizer is Key to Visual Generation, LIT-626 — Movie Gen: A Cast of Media Foundation Models, LIT-710 — Efficient Diffusion Training via Min-SNR Weighting Strategy, LIT-714 — Analyzing and Improving the Training Dynamics of Diffusion Models, LIT-775 — Inductive Moment Matching, LIT-784 — Mean Flows for One-step Generative Modeling, LIT-785 — STARFlow: Scaling Latent Normalizing Flows for High-resolution Image Synthesis, LIT-787 — One Step Diffusion via Shortcut Models

Graph View

Key takeaways

  • The U-Net in diffusion models was inherited, not chosen. DDPM took it from PixelCNN++ and conditional GANs; Dhariwal and Nichol ablated its internals but left the high-level design intact. Every diffusion model since had used it. This replaces it wholesale with a transformer over latent patches.
  • The scaling claim is the contribution, not the FID. FID falls consistently with forward-pass GFLOPs, and it does not matter which lever buys the GFLOPs — more depth, more width, or more input tokens from a smaller patch size all land on the same trend. That is what makes the architecture a backbone rather than a model.
  • Conditioning is injected through adaptive layer norm, the variant the paper finds best among the several it ablates.
  • DiT-XL/2 reaches 2.27 FID on class-conditional ImageNet 256×256 and state of the art at 512×512, beating the U-Net models it replaces at comparable or lower compute.

Standing in the anthology

Filed as context, and filed unread — there is no NOTE for it, which per ADR-025 means exactly that. This entered the record because two papers filed alongside it, LIT-447 and LIT-449, hold this architecture fixed while changing what is transported across it. SiT’s entire method is “the same model structure, the same number of parameters, the same GFLOPs as DiT, with the transport changed”; without this note that comparison names something the record does not have.

It is also the backbone under most of the current image and video generation line, and the record held nothing on it — not because anything complained, but because nothing in the record had needed to name a diffusion backbone before. A gap that produces no complaint is the kind this record finds only when something else arrives pointing at it.

Filing it unread is a deliberate choice rather than a shortcut: what the neighbouring documents need from it is the scaling claim and the identity of the backbone, both of which are in the abstract and the headline figure, and claiming a reading the record has not done would be worse than saying so.

What later papers set beside it. Four papers in the record compare against DiT. Three quote its published numbers rather than re-run it, so those three are positions on a leaderboard, not controlled tests.

  • MAGVIT-v2 (LIT-623) lists DiT-XL/2’s published FIDs: 9.62 unguided and 2.27 guided at ImageNet 256, and 12.03 and 3.04 at 512. Its masked transformer has 307M parameters against DiT-XL/2’s 675M, and it beats both DiT figures at both resolutions: 3.65 and 1.78 at 256, 3.07 and 1.91 at 512.
  • Min-SNR weighting (LIT-710) sets its ViT-XL system result beside DiT-XL/2’s 2.27 at ImageNet 256. That result is 2.06 FID, with x0-prediction, Min-SNR-5, guidance 1.5 and EDM’s Heun sampler. Its “3.3× faster than DiT” compares its 2.08 at 2.1M iterations with DiT’s reported run, across codebases and optimizers.
  • EDM2 (LIT-714) quotes DiT in its comparison tables and re-runs only EDM. Its discussion asks whether magnitude-preserving layers would help a DiT, and does not try it.
  • Movie Gen (LIT-626) is the one controlled comparison. It trains a 5B text-to-video model with a DiT block and one with a Llama 3-style block, holding everything else fixed. In human evaluation the Llama 3 block wins by a net +18.6 on quality and +12.6 on text alignment (its Table 8c). The two blocks differ only in norm, activation and biases, so this compares two versions of DiT’s block design rather than DiT against a different architecture.

Four more arrived with the flow-map batch, and two of them re-run a DiT at their own settings.

  • Shortcut Models (LIT-787) trains a DiT-B diffusion model as one arm of its matched Table 1, at 39.7 FID with 128 steps, beside flow matching and the shortcut model on the same backbone. Its XL shortcut model, 10.6 at one step, is set beside DiT-XL/2’s published rows at a different budget.
  • STARFlow (LIT-785) trains an in-house 2.1B DiT on 200M samples with the stock VAE decoder as a near-matched baseline. It plots its own 1.4B latent flow below it at every checkpoint on 50K-sample FID (Fig. 10a), with a GAN-fine-tuned decoder and an unstated position on its own longer schedule. Against published DiT-XL/2 it is 2.40 to 2.27.
  • Inductive Moment Matching (LIT-775) keeps the DiT-XL/2 architecture with a second time input, and quotes DiT’s 2.27 at 250 × 2 evaluations against its own 1.99 at eight guided steps, on an unmatched budget of 1.2M iterations at batch 4,096.
  • MeanFlow (LIT-784) trains DiT-style backbones from scratch and quotes DiT-XL/2’s 2.27 against its own one-step 3.43 and two-step 2.20.

Backlinks