Source
Ho, Salimans et al. (2022), LIT-627; Blattmann et al. (2023), LIT-621; Blattmann, Dockhorn, Kulal et al. (2023), LIT-625. All three were read in full as NOTE-347, NOTE-341 and NOTE-336.
The claim
Don’t train a video diffusion model on video alone. Either initialize its spatial layers from a text-to-image model, or pretrain it on images first. Then keep still images in the batch once video training starts.
Every video report in the record does some version of this. The three comparisons that come closest to testing it all point the same way, and none of them isolates the effect:
- Adding frames to video batches (LIT-627, Table 4). Same model, data, batch size and 200K steps. FVD falls from 202 with no extra frames to 68 with four and 58 with eight, with temporal attention masked for the extra frames. The extra frames are also extra compute: the arms process 16, 20 and 24 frames per example. The paper calls the change “a memory optimization to fit more independent examples in a batch”. The table cannot separate “images help” from “more independent frames per step help”.
- Image initialization against end-to-end training (LIT-621, Table 1). On driving scenes, FVD is 534 against 1155. The pretrained arm gets 73K image-model steps the other arm never gets (Table 7), and trains only its temporal layers while the other trains everything. Initialization, compute and freezing change together.
- Image-initialized against random spatial layers (LIT-625, Fig. 3a). Human raters prefer the image-initialized model. The figure gives no counts, and the text states neither its resolution nor its step count.
- A separate image corpus, on and off (LIT-661, W.A.L.T Table 5). Two 419M models trained with and without ~970M image-text pairs beside ~89M text-video pairs. Zero-shot UCF-101 FVD is 598.8 without images and 344.5 with them. This is the only comparison that uses a separate image corpus. Steps, batch size and the video share of each batch are not stated, and it is a single run.
Why it is Proposed
The practice is probably right, and the record cannot yet say it believes
it on this evidence. Each comparison gives the image arm more compute, or
bundles initialization with freezing, or states too little of its setup to
check. Only W.A.L.T uses the thing the practice recommends, a separate
image corpus, and it does not say what compute each arm got. VDM’s and
Video LDM’s “images” are frames drawn from their own video datasets. Every large report adopts the practice, and none of them tests it.
The promote_when asks for the one missing comparison.
Conditions
- What “images” means changed. Every report since Imagen Video (LIT-637 §2.6) trains on a separate, much larger image-text corpus. Imagen Video asserts that this “significantly increases the overall quality” and shows no numbers. Imagen Video also states no image initialization, so it is the “alongside” half without the “before” half.
- Freezing is a third way to keep what the image model knows. Emu Video (LIT-635) and AnimateDiff (LIT-633) freeze the image layers and show the video stages no images. Emu Video’s controlled result is that freezing beats full fine-tuning narrowly, 55.0 / 58.1 on quality / faithfulness (its Table 1). Its unfrozen arm was unfrozen only during the 512px stage, and both arms were given the same conditioning images. So the test cannot see whether unfreezing damages the image model, which is the reason to freeze. Nobody compares freezing with joint image-video training.
- SVD tests only the first half. It initializes from an image model and never trains on images alongside video.
- The large reports give a different reason. Wan presents image-first pretraining as a throughput fix: long, high-resolution video starves the batch and causes “training instability caused by spikes in gradient variance”. That is an argument about compute. It may be the real reason the practice works, and it is exactly what the three comparisons do not control for.
- VDM’s own explanation was never measured. VDM says joint training works because it “reduces the variance of minibatch gradients”, and shows no gradient statistics.