Ho, Salimans et al., Google (2022) — ARXIV-2204.03458
Key takeaways
- The architecture moves image diffusion into time with little change. The 2D U-Net becomes 3D by using 1×3×3 convolutions and adding a temporal-attention block with relative position embeddings after every spatial-attention block (§3, Fig. 1). This factorization is adopted, not ablated.
- Joint image-video training is the result that lasted. Independent frames are appended to each video and temporal attention is masked off for them. On 16×64×64 text-to-video with the small model, FVD falls from 202.28 with no extra frames to 68.11 with four and 57.84 with eight (Table 4). It is controlled for model, data, batch size and steps (App. A.4), but not for frames processed per step: the 4- and 8-frame arms denoise 20 and 24 frames per example against 16. The paper reports no seed count and no variance. Every later report in this line trains on images and video together, Wan (LIT-619) included.
- Reconstruction guidance beats replacement for extending a video autoregressively from an unconditional model. At 64 frames, FVD is 136 against 451 at w=2 (Table 6), with the model and guidance weight held fixed.
- Per-frame and video metrics disagree about guidance. As the classifier-free guidance weight rises, IS keeps improving while FID-type metrics improve and then degrade (Table 5).
Where the hedges are
Per DP-010:
- The mechanism is not measured. The abstract says joint training reduces “the variance of minibatch gradients”. §4.3.1 argues this, but no gradient variance or training curve is shown, only final metrics.
- The guidance claim runs against the paper’s own table. The conclusion claims the benefits of classifier-free guidance “on both video and image sample quality metrics”. At frameskip 1, FVD gets worse as the guidance weight rises: 41.65 at w=1, 50.19 at w=2, 163.74 at w=5 (Table 5). Table 5 has no unguided (w=0) row. Under Eq. 6, w=1 is already guided, so the table does not compare guidance against none.
- The images come from the videos. The “image” half of joint training is frames drawn from the same video dataset, and a separate image corpus is left to future work. Later reports use one, and this paper does not test that.
- Not released. “We have decided not to release our models” (§6).
Standing in the anthology
This is the root of the video generation line. It is filed as the earliest controlled evidence that video diffusion models should see still images as well as video. The text-to-video results use an unnamed 10M-video dataset and have no external baseline, so its benchmark results are not what the record keeps it for.
It keeps DDPM’s (LIT-036) model, a Gaussian diffusion trained to predict the noise through a U-Net, almost unchanged (ε-prediction for most models, v-prediction for BAIR and Kinetics). What it changes is the network’s shape and the data: the U-Net is factorized over space and time so that it takes video, and still frames are trained alongside.
Later systems used it as their diffusion baseline, and it loses each time. Make-A-Video (LIT-632) ran a human study on the 28 prompts from this paper’s website, and raters preferred Make-A-Video 84.38% on quality and 78.13% on faithfulness. MAGVIT-v2 (LIT-623) and W.A.L.T (LIT-661) both list this paper’s Kinetics-600 frame-prediction FVD, 16.2 at 1.1B parameters and 256 steps, as the diffusion row in their Table 1, against 4.3 and 3.3 at about 310M. W.A.L.T also repeats the joint-training ablation on a latent transformer and says its result agrees with this paper’s. There the image half is a separate corpus of about 970M text-image pairs, not frames of the training videos.
Read in full as NOTE-347.