Yu et al., Google and Carnegie Mellon (2023) — ARXIV-2310.05737
Key takeaways
- Lookup-free quantization lets the vocabulary scale. Each latent dimension is sign-quantized and an entropy penalty is added. With standard VQ, generation FID gets worse past a moderate vocabulary. With LFQ, reconstruction and generation both keep improving as the vocabulary grows (Fig. 1, a controlled sweep on ImageNet 128).
- A causal 3D CNN lets images and video share a tokenizer. The first frame is encoded on its own, so a single image is a valid one-frame video. No experiment uses the video tokenizer for images: ImageNet results use separate ImageNet-trained image tokenizers (App. A.1). It beats C-ViViT variants on FVD, 96 against 437 and 317 (Table 5a). Table 5a does not say whether that FVD is reconstruction or generation, its MAGVIT row (107.15) does not match MAGVIT’s reconstruction FVD in Table 5c (24.55), and the parameter counts differ (58M, 90M, 67M). It also beats a non-causal LFQ baseline on Kinetics generation, 5.2 against 11.6 (Table 1).
- The same generator improves with the tokenizer swapped. Kinetics-600 FVD goes from 9.9 to 5.2 at 12 decoding steps each; Table 1 also gives MAGVIT-v2 4.3 at 24 steps, the figure later papers such as W.A.L.T quote. UCF-101 FVD goes from 76 (MAGVIT, 12 steps) to 58 (MAGVIT-v2, 24 steps), and Table 1 reports no 12-step UCF number for MAGVIT-v2. The backbone also changes slightly: the 2^18 vocabulary is factorized into two 2^9 heads with weight tying (§3.2, §4.2). This is the evidence for the “tokenizer is key” half of the title.
- Most of the other design choices are shown on reconstruction only. The up- and downsamplers, late temporal downsampling, adaptive GroupNorm and depth are ablated cumulatively (Tables 5b and 5c). None of them is shown to help generation.
Where the hedges are
Per DP-010:
- “Language model” does not mean an autoregressive LLM. It is a bidirectional masked model decoded in 12–64 steps. The one autoregressive result (Table 8, UCF FVD 109) is worse than the MLM’s 58 and is not compared against diffusion.
- “Beats diffusion” needs guidance. Without guidance MAGVIT-v2 loses: 3.07 against VDM++ 2.99 at 512, and 3.65 against 2.40 at 256. With guidance at 256 it is a tie, 1.78 against MDT’s 1.79 (Tables 2 and 7). The latent-diffusion baselines in the same tables lose either way. At ImageNet 256, LDM-4 (LIT-062) scores 10.56 unguided and 3.60 guided, and DiT-XL/2 (LIT-448) 9.62 and 2.27, both at 250 steps, against MAGVIT-v2’s 3.65 and 1.78 at 307M parameters and 64 steps (Table 7). At 512, DiT-XL/2 scores 12.03 and 3.04 against 3.07 and 1.91 (Table 2).
- The claim of matched data and budget is never quantified.
- The video diffusion baselines are old (VDM, RIN), and UCF-101 has no diffusion baseline at all (Table 1). VDM (LIT-627) is a 1.1B pixel-space model at 256 steps with Kinetics-600 FVD 16.2. MAGVIT-v2 reaches 5.2 at 307M and 12 steps, and 4.3 at 24 steps.
- There are no text-conditional results, which the paper admits in App. B.
Standing in the anthology
The title makes a claim about generator families. The record keeps the paper for its tokenizer design, which later work adopted. Its causal, first-frame- separate 3D encoder is the design the video VAEs in this line use. Wan’s VAE cites it by name for treating the first frame separately. CogVideoX’s and HunyuanVideo’s causal 3D VAEs have the same shape, though they use a continuous latent and not LFQ. W.A.L.T (LIT-661), two months after this paper, did the same: it takes MAGVIT-v2’s causal 3D CNN encoder-decoder, drops the quantization for a real-valued latent, and runs diffusion on it. Its Table 1 then puts W.A.L.T-L ahead of MAGVIT-v2 on the benchmarks this paper reports, 3.3 against 4.3 on Kinetics-600 and 46 against 58 on UCF-101, taking MAGVIT-v2’s 24-step figures.
The title’s claim about generator families is not what the record relies on. Anything citing this paper for “language models beat diffusion” should cite the guided numbers and name the masked decoder.
Read in full as NOTE-342.