Podell, English, Lacey, Blattmann, Dockhorn, Müller, Penna, Rombach, Stability AI (2023) — ARXIV-2307.01952
Key takeaways
- Scale and text encoders: the UNet moves most transformer compute to lower resolutions, and the cross-attention context doubles with two concatenated text encoders. 2.6B parameters in the UNet
- Size conditioning: instead of discarding small images (39% of the
pretraining data below 256² would have been dropped) or upsampling them
(which teaches blur), embed
(h_original, w_original)with Fourier features and add it to the timestep embedding. At inference, the size condition sets the apparent resolution - Table 2 (class-conditional ImageNet 512², FID-5k / IS-5k): discard below 512 (70k images) 43.84 / 110.64. All data, no conditioning 39.76 / 211.50. All data, size-conditioned 36.53 / 215.34
- Crop conditioning: feed the random-crop offsets
(c_top, c_left), and set them to zero at inference. This removes SD 1.x/2.x’s cut-off objects. Shown qualitatively - Multi-aspect fine-tuning on buckets of aspect ratios, and a refiner that re-noises and denoises the base latents with a model specialized to high resolution
- Evaluation: user-preference studies against SD 1.5 and 2.1 (Figure 1). The authors note FID does not track the improvement
Standing in the anthology
Filed from #163 (the stable diffusion lineage, “sdxl”). It extends
LDM (LIT-062) and sources SOTA-341. The other SDXL mentions in the
record are as a quantization target (LIT-512).
As a teacher and as a baseline. DMD2 (LIT-646) distils SDXL into a 4-step student and compares the two on 10K COCO prompts (its Table 2): FID 19.32 for the student against 19.36 for SDXL at guidance 6 and 20.39 at guidance 8, with the student slightly less diverse (pairwise LPIPS 0.61 against 0.64). DMAD (LIT-770) distils SDXL into a four-step student on the same protocol and reports FID 14.47 against SDXL’s 19.36, with CLIP 0.328 against the teacher’s 0.332; its SDXL and DMD2 rows are DMD2’s published numbers, not reruns. The DALL-E 3 captioning paper (LIT-648) puts SDXL last of its three systems on 4,096 MSCOCO captions, CLIP score 30.5 against 31.4 for DALL-E 2 and 32.0 for DALL-E 3, and at −189.5 in a human prompt-following Elo over 170 prompts, where DALL-E 3 scores 153.3. That paper says its DALL-E 3 comparisons do not isolate recaptioning, so they measure systems, not training captions.
Two later papers list SDXL’s published GenEval of 0.55 as a reference row. Flow-GRPO (LIT-779) sets it beside SD3.5-M at 0.63 before and 0.95 after RL on GenEval’s own prompt templates. STARFlow (LIT-785) edges it at 0.56, the one text-to-image row that latent normalizing flow beats.
Diffusion-DPO (LIT-774) fine-tunes SDXL base on Pick-a-Pic preference pairs and is compared against it by human vote: the tuned model is preferred 70.0% of the time on PartiPrompts, and 69% against base plus refiner, which has about 1.9 times its parameters (its §5.2, Figs. 3–4). Fine-tuning SDXL on the preferred images alone makes it worse, because SDXL is already better than the models that generated the data.
Read — NOTE-306.