LIT-626: Movie Gen: A Cast of Media Foundation Models

The Movie Gen team, Meta (2024) — ARXIV-2410.13720

Key takeaways

• The ablations are controlled. Each Table 8 comparison is a 5B model at 352×192, trained on 21M videos and judged by humans on 381 prompts, with “every aspect of the model except for the design decision being tested … held constant”.
  • Flow matching beats diffusion (v-prediction, zero terminal SNR) by a net win of +16.5 on overall quality and +7.1 on text alignment (Table 8a).
  • Video captions beat frame captions by +10.8 on alignment (Table 8b), most of it motion alignment (+10.7, and +16.1 on high-motion prompts; §3.6.2, no σ). The frame-caption baseline captions three frames.
  • The Llama 3 block (LIT-179) beats the DiT block by +18.6 on quality and +12.6 on alignment (Table 8c).
• The temporal autoencoder. It compresses 8×8×8 into 16 channels and is inflated from an image autoencoder. On video it is close to a frame-wise autoencoder, PSNR 32.25 against 34.11. On images it is better, 32.16 against 30.83 (Table 11). An outlier-penalty loss removes “spot” artifacts in the latent (Table 13).
• Fine-tuning is where much of the quality comes from. Supervised fine-tuning on a small curated set gives a +34.7 net win over the pretrained model (Table 7).
• Sampling. A 50-step linear-quadratic schedule is asserted to match 250 linear steps (§3.4.2). Fig. 10 shows the motivation, not a measurement.
• External comparisons. Movie Gen is on par with Kling 1.5 (+3.9) and ahead of Sora (+8.2, σ = 5.1), Runway Gen-3 and Luma (Table 6). It loses to Kling on motion completeness.

Where the hedges are

Per DP-010:

• The scaling claim is image-only and narrow. Four sizes (5B–30B) are trained only on the 256px text-to-image stage, over 1e22–1.7e22 FLOPs, under a quarter of a decade. IsoFLOP minima are then plotted against the Llama 3 law, which is overlaid and not refit. The caption says that law “may serve as a reasonable predictor … even for media generation models” (Table 10). No video scaling experiment backs the abstract’s “reap the benefits of scaling”.
• “Llama 3 beats DiT” is narrower than it sounds. The two blocks differ only in norm, activation and bias (Table 9). Both keep adaLN and cross-attention. The ablation compares two versions of one block design, not two architectures.
• “Holds across a range of model sizes” for flow matching comes with no data. The alignment win (+7.1) is under twice the annotation σ the paper reports elsewhere (Table 40).
• The Sora comparison uses a different recipe. It used 500 linear steps and a manual best-of-5 selection (App. C.2), not the 50-step recipe of §3.4.
• No weights were released. Only the benchmark prompts and generations were published.

Standing in the anthology

This is the line’s source for flow matching over diffusion, and the only controlled test of it at video scale. Every later model here (HunyuanVideo, Step-Video, Wan) trains with flow matching. None of them tests it, and Movie Gen is the report that did. It is equally the source for video-native captioning, which the others assert.

The scaling claim is the sentence to watch. It will be quoted as “video models follow LLM scaling laws”, and the evidence is image-only, four points, and an overlaid rather than fitted curve. HunyuanVideo (LIT-620) fits its own video law, on a proxy model.

What it is built from. The objective is the straight “OT” path of Flow Matching (LIT-630), X_t = t·X₁ + (1 − (1 − σ_min)t)·X₀ with σ_min set to 1e-5, a value that paper never gives. The timesteps are not that paper’s uniform draw but SD3’s logit-normal (LIT-449), cited “as in prior work” (§3.1.2). What Movie Gen adds to both is the move to video and the controlled test against diffusion at 5B. The generator runs in the latent space of an autoencoder, the latent diffusion design of LIT-062, extended to time: the TAE is inflated from an image autoencoder and compresses 8× in time as well as 8×8 in space. The training recipe extends Emu Video’s (LIT-635): low resolution before high (256 px, then 768 px) and a final fine-tune on a small curated set. It does not keep Emu Video’s objective: the diffusion arm of Table 8a, v-prediction with zero terminal SNR, is what Emu Video trained with in its 512 px stage.

Where it is compared. The Table 8a baseline is the fix Lin et al. (LIT-689) proposed for leaked terminal signal, v-prediction with a zero-terminal-SNR schedule, and flow matching beats it (+16.5 quality, +7.1 alignment). The Table 8c baseline is a DiT block (LIT-448), and the Llama block beats it by +18.6 on quality and +12.6 on alignment, but per Table 9 the two differ only in norm, activation and bias. The Sora comparison (LIT-652) is a net win of +8.23 on overall quality (σ 5.07) and +17.72 on text alignment. Sora has no API, so it runs on the prompts and videos OpenAI published, against one of five Movie Gen samples picked by hand (Table 6, §3.6.1). Later papers measure against Movie Gen in turn. Step-Video (LIT-624) finds parity on Movie Gen Bench, 485 wins, 315 ties and 489 losses for Step-Video over 1,289 evaluations (its Table 14). CausVid (LIT-631) scores Movie Gen at 91.5 / 61.1 / 28.8 on VBench temporal quality, frame quality and text alignment over 128 MovieGen prompts, below its own 94.7 / 64.4 / 30.1, with each model run at its own length (its Table 1).

Read in full as NOTE-334.