Singer et al., Meta AI (2022) — ARXIV-2209.14792
Key takeaways
- The design. It builds on an unCLIP-style text-to-image model (LIT-070): a prior, a 64px decoder, and super-resolution to 256px and 768px. 1D temporal convolutions and 1D temporal attention are added after every spatial layer and initialized as the identity, so training starts from the image model (§3.2).
- Text alignment comes from images, and motion from unlabeled video. The prior is trained on 2.3B English LAION pairs and never sees video. The decoder’s temporal layers and the interpolation network train on WebVid-10M only, with its text thrown away. HD-VILA-10M is used only for the spatiotemporal super-resolution network SR_l^t (§4.1).
- Results. Zero-shot MSR-VTT FID is 13.17 and CLIPSIM 0.3049 (Table 1). Zero-shot UCF-101 FVD is 367.23 (Table 2). Human raters prefer it to VDM (LIT-627) 84% on quality over 28 prompts (Table 3).
- Where later systems place it. Emu Video (LIT-635) generated on this paper’s 307-prompt set, against non-cherry-picked Make-A-Video samples supplied by its authors, and raters preferred Emu Video on quality 96.8% of the time. On zero-shot UCF-101 the same comparison lists this paper’s FVD 367.2 against Emu Video’s 317.1 (Emu Video Table 2). W.A.L.T (LIT-661) quotes the same 367.2 and IS 33.0 in its system-level UCF-101 table, against 344.5 / 31.7 for its 419M joint image-video model and 258.1 / 35.1 at 3B (W.A.L.T Table 5).
- Frame rate as a curriculum. It conditions on fps and trains from high fps (less motion) to low fps (more motion). This is asserted.
Where the hedges are
Per DP-010:
- “Without text-video data” describes the training signal, not the data. WebVid-10M is a captioned stock-footage set whose captions were discarded, and HD-VILA pairs its clips with speech transcripts.
- The consequence is stated, in the discussion. The model “can not learn associations between text and phenomenon that can only be inferred in videos”, for example “a person waving their hand left-to-right or right-to-left” (§5). This is the design’s real limit, and it is not in the abstract.
- There are no ablations. Starting from the image model “significantly accelerates the T2V training process” (§1) is the first advantage listed in the abstract, and no run without it exists. Every table is a system-level comparison.
- §3.4 contradicts itself. It says the prior was “trained on images alone (no aligned text)”, but the prior is trained on text-image pairs.
- The resolution is overstated. Footnote 1 says the 768px output is downsampled to 512px “for a cleaner aesthetic”.
Standing in the anthology
This is the far end of SOTA-386’s claim: an image model supplies all the text knowledge, and video supplies only motion. The paper shows what that costs. Anything about text and motion together cannot be learned. That is why later reports train on captioned video, and why Movie Gen found video captions beat frame captions on alignment, most of it motion alignment (LIT-626, Table 8b and §3.6.2).
It supplies no controlled evidence for SOTA-386. It starts from an image model and never trains without one.
Read in full as NOTE-349.