Blattmann, Dockhorn, Kulal et al., Stability AI (2023) — ARXIV-2311.15127
Key takeaways
- Training runs in three stages. Stage I is image pretraining (Stable Diffusion 2.1), stage II video pretraining at low resolution on a large curated set, and stage III fine-tuning on a small high-quality set (§3). Human raters prefer image-initialized over randomly initialized spatial layers (Fig. 3a).
- Curation beats quantity, in a controlled design. Each annotation axis (CLIP score, aesthetics, OCR text area, optical flow) is filtered on its own at 12.5%, 25% and 50%. The model is held fixed: 8 frames at 256², 40k steps, batch 256 (App. E.2.2). The curated LVD-10M-F is a quarter the size of LVD-10M and is preferred on quality and prompt alignment (Fig. 3b). It is also preferred over WebVid-10M and InternVid-10M (Fig. 4a–b).
- The pretraining advantage survives fine-tuning. After the same stage III, the curated-pretrained model leads the image-initialized one by 127 Elo at 10k steps and 103 at 50k, against 89 and 70 for the uncurated one (Fig. 4e). Both leads shrink from 10k to 50k.
- Scale. LVD is 577M clips. After filtering, LVD-F is 152M (Table 1). Processing yields about 4× more clips per video than the raw metadata cuts, 2.65 → 11.09 (Fig. 2).
- Image-to-video by concatenating a conditioning latent. The design is the one later reports, Wan (LIT-619) among them, cite as the standard approach.
Where the hedges are
Per DP-010:
- The motion filter lost its own ablation. “The aggregated preference score of the model trained with this filtering method does not rank as high in human preference as the non-filtered score” (App. E.2.2). The 25% threshold was kept anyway. §3.1 motivates motion filtering, and the appendix shows it did not win.
- The ablations use a different recipe from the released model. They train with a linear schedule, offset noise and v-parameterization. The released model uses EDM preconditioning (App. E.2.1). It is not shown that the curation findings transfer.
- All evidence is human preference over 64 prompts. Every curation claim rests on about three votes per prompt, per task, per pair (App. E.1.1). That is roughly 190 votes per task per model pair: small, but not three. No automatic metric supports them.
- Asserted, not ablated: the noise-schedule shift toward higher noise at high resolution, called “essential” (§2), and the CFG ramp across frames.
- “Competitive with closed-source” (abstract) rests on a comparison with Gen-2 and Pika that rated visual quality only (§4.3).
Standing in the anthology
This is the evidence the video line’s data sections point to, and the most controlled of them. Wan’s curation (LIT-619 §3) has the same design: filter on several axes, grade motion, keep a small top slice for post-training. Wan does not ablate any of it, so this paper is the evidence for that design and Wan is adoption of it.
The lineage is direct. The architecture is Video LDM’s (LIT-621): temporal convolution and attention after every spatial layer of a pretrained image model. SVD holds that design fixed, trains all layers, and adds what Video LDM did not have, a controlled study of the training data. Under both is latent diffusion (LIT-062): the spatial layers start from Stable Diffusion 2.1, so the video model diffuses in that image model’s latent space rather than in pixels.
The one head-to-head with a concurrent system is Emu Video’s (LIT-635), not this paper’s. On image animation over 65 prompts, raters preferred Emu Video to SVD 72.3% on quality and 73.9% on faithfulness. Those rows were added in Emu Video’s v2, and SVD itself does not compare against Emu Video.
The overridden motion filter is the underreported sentence to look for under DP-010. The field now treats motion filtering as standard, and the one controlled test of it in this line did not favour it.
Read in full as NOTE-336.