anthology-of-the-sota

LIT-635: Emu Video: Factorizing Text-to-Video Generation by Explicit Image Conditioning

4 min read

Properties8
StatusActive
Version2
Tagsgenerative-modeling, vision-and-graphics
ExtendsLIT-062 — High-Resolution Image Synthesis with Latent Diffusion Models, LIT-689 — Common Diffusion Noise Schedules and Sample Steps are Flawed
Extended byLIT-626 — Movie Gen: A Cast of Media Foundation Models
Compared againstLIT-637 — Imagen Video: High Definition Video Generation with Diffusion Models, LIT-632 — Make-A-Video: Text-to-Video Generation without Text-Video Data, LIT-621 — Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Models, LIT-625 — Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasets
Cited as “paper” byNOTE-351 — Emu Video
Cited as “source” bySOTA-393 — Freeze the image model's spatial layers when adding time if checkpoint compatibility matters; do not expect a quality gain from it, SOTA-422 — Rescale the noise schedule so the last timestep is pure noise, and train with v-prediction so the model can learn there

Graph View

Girdhar et al., GenAI, Meta (2023) — ARXIV-2311.10709

Key takeaways

  • The factorization. Generate an image first, with the temporal layers off, then generate the video conditioned on that image and the text. The image is zero-padded in time, masked and concatenated on channels (§3.2), which is the same conditioning Wan’s image-to-video uses. The 2.7B spatial parameters come from the Emu text-to-image model and stay frozen, and 1.7B temporal parameters are trained. The network is a latent diffusion model in the sense of LIT-062, which the paper cites for the approach: it denoises autoencoder latents, not pixels, and the 256px stage uses that paper’s noise schedule. What it changes is the added temporal layers and the image condition.
  • The ablations are controlled (Table 1: human win rates on quality and faithfulness, 307 prompts, majority of 5 raters, each design against the same model without it):

    • factorized against direct text-to-video: 70.5 / 63.3

    • zero terminal SNR at 512px: 96.8 / 88.3

      The fix is Lin et al.’s (LIT-689): rescale the schedule so that α_N = 0, leaving no signal at the last training timestep. That paper argued it for images; this one argues that the leak is worse for high-resolution video, “due to redundant pixels across both space and time” (§3.2), and gives it a human-preference test of its own. As in Lin et al.’s one numeric result, the fix is not isolated: v-prediction comes in the same step (see the hedges).

    • multi-stage (256px then 512px) against 512px only, at the same budget: 81.8 / 84.1

    • high-quality fine-tune: 65.1 / 79.6

    • frozen against fully fine-tuned spatial layers: 55.0 / 58.1

  • Low resolution first, with a measured sweet spot. About 70K iterations at 256px (one epoch) is best, and fewer or more is worse (App. Fig. 1).
  • A tiny, motion-filtered fine-tune set. 1.6K of 34M clips are kept, on CLIP score, aesthetics and a minimum H.264 motion-vector score. The motion score of generated videos rises from 0.61 to 12.7 after fine-tuning at text guidance 8 and image guidance 2, and from 2.87 to 14.9 at image guidance 1 (§3.3, App. Table 4).
  • Against earlier systems. Human raters prefer it on quality to Imagen Video (81.8%), Make-A-Video (96.8%), Video LDM (92.3%) and, on image-to-video, SVD (72.3%). Each baseline is judged on its own prompt set from its released samples. Against Imagen Video (LIT-637), 55 prompts, the faithfulness margin is the narrowest in the paper at 56.4%, and the authors note those prompts ask for rendered text, a known weakness of latent models. Against Make-A-Video (LIT-632), 307 prompts, the samples were non-cherry-picked ones from its authors, and on zero-shot UCF-101 FVD is 317.1 against 367.2 (Table 2). Against Align Your Latents (LIT-621), 65 prompts, quality is 92.3% and UCF-101 FVD 317.1 against 550.6.

Where the hedges are

Per DP-010:

  • The zero-SNR baseline is under-specified. The 512px stage switches to v-prediction and zero-SNR rescaling together (App. Table 3), and the paper does not say whether the baseline also used v-prediction. The 96.8% may measure both changes.
  • “At no additional training cost unlike [Imagen Video]” (§3.2) contrasts freezing with joint image-video training. The two are never compared.
  • “81% vs Imagen Video” (abstract) is quality. On faithfulness it is 56.4%, which the body calls outperforming “by 56%”.
  • Every comparison is a human win rate with no confidence interval. On unprocessed videos the rates are higher, and the authors attribute that to rater bias (App. Table 10).
  • Weights were not released.

Standing in the anthology

This is the report Movie Gen (LIT-626) builds on. The multi-stage schedule and a high-quality fine-tune on a motion-, aesthetic- and CLIP-filtered set are tested here. Motion filtering is not tested on its own: Table 1d tests the resulting fine-tune as a whole. Its controlled results are the evidence for several things the later reports adopt:

  • low-resolution-then-high training (the curriculum Wan asserts)
  • a small curated fine-tune set (SVD and Movie Gen). SVD’s stage-III sets of 250K and about 1M clips (LIT-625) are small against its 152M pretraining set, not in Emu Video’s sense. They are roughly 150× to 600× its 1.6K.
  • image-first conditioning (Wan’s image-to-video)

On SOTA-386 it adds one adjacent fact. It keeps image knowledge by freezing the spatial layers rather than training on images, and freezing beats fine-tuning by a small margin. It does not test image initialization against random initialization, or joint training against video-only.

Read in full as NOTE-351.

Backlinks