Ho et al., Google Research, Brain Team (2022) — ARXIV-2210.02303
Key takeaways
- The cascade. A base model, three spatial and three temporal super-resolution models, 11.6B diffusion parameters, and a frozen T5-XXL text encoder produce 128 frames at 1280×768 and 24 fps (§2.2).
- v-prediction for video super-resolution. The same group’s video result agrees with SOTA-195: on one super-resolution stage, v-prediction converges faster and avoids cross-frame colour shift (Figs. 12–13, curves only). It is not independent support. Ho and Salimans are authors of both this report and Progressive Distillation (LIT-067), the practice’s source.
- Model size matters more than it did for images. Scaling the base model’s width and depth (500M, 1.6B, 5.6B) improves FVD and CLIP score (Fig. 11, curves only). The paper does not state what else was held fixed. It notes this runs “contrary” to Imagen’s image result.
- Distillation. Distilling each stage to 8 steps cuts sampling from 618s to 35s. With constant guidance, CLIP score goes from 25.19 to 25.03 and R-precision from 92.12 to 89.68 (Table 1).
- The joint-training corpus. All seven models train on 14M video-text pairs, 60M image-text pairs and LAION-400M, with images packed as single-frame videos and temporal layers masked (§§2.6, 3).
Where the hedges are
Per DP-010:
- The version of joint training everyone now uses is asserted here. “Consistent with Ho et al. (2022b), we observe that joint training with images significantly increases the overall quality” (§2.6). There are no numbers. VDM (LIT-627) tested joint training on frames drawn from its own videos. This paper adds a separate image corpus, which is the version in use now, and shows nothing for it. Style transfer from images to video is shown qualitatively (Fig. 8).
- Nothing external is compared. No table includes another system.
- “Without any noticeable loss in perceptual quality” (§2.7) cannot be settled by the metrics either way. Under constant guidance, R-precision falls 2.4 points (Table 1). Under oscillating guidance, full distillation raises both metrics (CLIP 25.02 → 25.12, R-precision 89.91 → 90.97). Every generated row also scores above ground truth (24.27, 86.18).
- Neither the model nor the code was released (§4).
Standing in the anthology
This is the bridge between VDM and the current line. It is the first report whose joint training looks like modern practice: a large, separate image corpus. It gives no evidence for it, which is why SOTA-386 cites VDM and not this report. Where the record needs this paper is its two controlled results: v-prediction for high-resolution video, and guided distillation of a cascade.
It has been compared from outside. Emu Video (LIT-635) set its factorized latent model against this paper’s released samples on its 55 prompts: raters preferred Emu Video on quality 81.8% of the time but on faithfulness only 56.4%, the narrowest margin in that paper. Emu Video’s authors note that these prompts ask for rendered text, a known failure mode of latent diffusion models.
Read in full as NOTE-348.