Yin et al., MIT and Adobe (2024) — ARXIV-2412.07772. Known as “CausVid”.
Key takeaways
- Asymmetric distillation. A bidirectional teacher, “similar to CogVideoX”, is distilled into a block-causal student with DMD2-style distribution matching, sampling in 4 steps (§4). Attention is bidirectional within each 5-latent-frame chunk and causal across chunks, with a KV cache.
- A bidirectional teacher beats a causal one at the same ODE initialization, architecture, data and steps (Table 4). Scores are temporal quality / frame quality / text alignment: 94.7 / 64.4 / 30.1 against 91.9 / 61.7 / 28.2. The authors attribute this to the causal teacher’s error accumulation carrying into its student (Fig. 8). The causal teacher is also the weaker model (92.4 / 60.1 / 28.5 against 94.6 / 62.7 / 29.6 at 10 s, Table 4), and the ablation does not separate the two.
- ODE-regression initialization helps. Initializing the student by regressing onto the teacher’s ODE endpoints gives 94.7 / 64.4 / 30.1, against 93.4 / 60.6 / 29.4 without it (Table 4).
- Latency. First-frame latency is 1.3s and throughput 9.4 fps, against 219s and 0.6 fps for the teacher, on one H100 at 640×352 for 120 frames, including the VAE and text encoder (Table 3).
- Against CogVideoX and Movie Gen. CogVideoX-5B (LIT-622), which §5.1 calls “similarly scaled” to the teacher, takes 208.6s to first frame at 0.6 fps in the same latency table. On zero-shot image-to-video it scores 87.0 / 64.9 / 28.9 against CausVid’s 92.0 / 65.0 / 28.9 (Table 6), so the gap is temporal quality alone. On 128 Movie Gen prompts (Table 1), Movie Gen (LIT-626) is the strongest other system at 91.5 / 61.1 / 28.8 against 94.7 / 64.4 / 30.1, with each model run at its own clip length.
Where the hedges are
Per DP-010:
- “Effectively mitigates error accumulation” (abstract) sits against “we still observe quality degradation when generating videos that are extremely long” (§6).
- The student is worse than the teacher on two axes. It “performs worse in temporal flickering and output diversity” (§5.2, last lines). Its flicker score is the lowest in the VBench table (App. Table 7). The diversity loss is “characteristic of reverse KL” (§6).
- It does not claim real time. “Limited to generating videos at around 10 FPS… could potentially enable real-time” (§6). The VAE sets the latency floor, since it needs five latent frames before producing any pixels.
- The 14-minute video is one example. The user study is 29 prompts, 3 ratings each per model pair from different evaluators (87 per pair, §5.1).
Standing in the anthology
This opens the causal branch of the line. Every other video model in the record generates a fixed clip bidirectionally. The transferable result is the asymmetric one: when distilling into a causal few-step generator, the teacher should not be causal. The authors’ reason is that its accumulated errors distill through, which the ablation does not separate from the causal teacher being weaker. Self Forcing (LIT-629) builds directly on it.
It extends DMD (LIT-643) and DMD2 (LIT-646), the image distillation method it carries to causal video. DMD2 also reports a student less diverse than its SDXL teacher (0.61 against 0.64), but gives it no cause; the reverse-KL attribution is this paper’s.
Read in full as NOTE-343.