Source
The Movie Gen team (2024), LIT-626, Table 8b.
The claim
A text-to-video model learns what text means about motion from the captions it trains on. Generate those captions with a captioner that sees the video, not by captioning frames and rewriting the frame captions into a paragraph.
The recommendation is Movie Gen’s (LIT-626), and that report is the only controlled comparison of it in the record: its caption ablation (§3.6.2, Table 8b) is the evidence below. HunyuanVideo, Wan and Open-Sora 2.0 adopted video-native recaptioning without one.
Movie Gen compares the two at 5B, with the generator, data and budget fixed. The baseline captions three frames and rewrites the frame captions into one caption:
- Text alignment. The video-captioned model wins by a net +10.8 (Table 8b).
- Visual quality is roughly unchanged, −0.8 (Table 8b).
- Where the gain comes from. §3.6.2’s prose says “most of the increase coming from motion alignment (+10.7%)”, “particularly on prompts that… ask for a high degree of motion” (+16.1%). The breakdown is not in the table and has no σ. The +16.1 could be motion alignment or total alignment on the high-motion subset. The sentence allows either.
- Direct caption comparison. Raters preferred the video captions themselves 67% of the time and the frame-rewrite captions 15%.
The gain is concentrated on the thing frame captions cannot describe.
Why the mechanism is plausible
Make-A-Video (LIT-632) is the extreme case of the alternative. All of its text knowledge came from images, and its discussion concedes it “can not learn associations between text and phenomenon that can only be inferred in videos”. Frame captions are a milder version of the same limit. They can name what is in the scene but not what happens, and “what happens” is what separates a video model from an image model.
The analogy is loose. Make-A-Video’s text reaches its video decoder only through one CLIP image embedding, so its limit is partly architectural. Captioned video alone would not have fixed it (NOTE-349).
Conditions
-
SVD measured a related choice and found the opposite. In its caption ablation (LIT-625, App. E.2.2), CoCa, an image captioner run on each clip’s middle frame, “surprisingly” beat the video captioner VideoBLIP on SVD’s human-preference Elo. The two findings don’t straightforwardly conflict. SVD’s video captioner was a weak 2023 model, and Movie Gen’s baseline was a multi-frame rewrite, not a single image caption. They do show that “video-native” is not enough on its own: the captioner has to be good at the video part. A second comparison should report which captioner it used.
-
One controlled source. The result is a human-rated win rate on 381 prompts at 5B and 352×192. It has not been reproduced elsewhere, and the field adopted the practice without testing it.
-
The captioner matters and is not separated out. Movie Gen’s captioner is a Llama 3-based video model fine-tuned for the job. The comparison is that captioner against frame captions, so it cannot say how much a weaker video captioner would give.
-
Camera motion is its own problem. Wan (LIT-619 §3.3) reports that general multimodal models, GPT-4o and Gemini included, predict camera angle and motion poorly, and trains a dedicated annotator for them. If the captioner cannot see camera motion, captions will not teach it, whichever kind of model writes them.
-
The image-domain antecedent is weaker than its reputation. The observation that descriptive synthetic captions improve prompt following is usually credited to DALL-E 3 (LIT-648), which Wan cites. Its controlled evidence is CLIP score only, not human-rated alignment, and it credits Parti with training on synthetic captions first.