anthology-of-the-sota

LIT-622: CogVideoX: Text-to-Video Diffusion Models with An Expert Transformer

4 min read

Properties8
StatusActive
Version2
Tagsgenerative-modeling, vision-and-graphics, model-architecture, representation-and-encoding
ExtendsLIT-448 — Scalable Diffusion Models with Transformers, LIT-623 — Language Model Beats Diffusion — Tokenizer is Key to Visual Generation
Compared againstLIT-449 — Scaling Rectified Flow Transformers for High-Resolution Image Synthesis, LIT-618 — LTX-Video: Realtime Video Latent Diffusion, LIT-619 — Wan: Open and Advanced Large-Scale Video Generative Models, LIT-631 — From Slow Bidirectional to Fast Autoregressive Video Diffusion Models
Cited as “introduced by” bySOTA-390 — In a video diffusion transformer, attend over space and time jointly rather than factorizing, and budget for the cost
Cited as “paper” byNOTE-346 — CogVideoX
Cited as “source” bySOTA-390 — In a video diffusion transformer, attend over space and time jointly rather than factorizing, and budget for the cost

Graph View

Yang, Teng et al., Tsinghua University and Zhipu AI (2024) — ARXIV-2408.06072

Key takeaways

  • The causal 3D VAE’s compression was chosen by ablation. At 8×8×4 with 16 channels, flicker is 86.3 and PSNR 28.7, against 93.2 and 28.4 for SDXL’s 2D VAE (Table 1). 32 channels reach 30.5 PSNR. At 16×16×8, “convergence becomes extremely difficult”. HunyuanVideo and Wan use the same 8×8×4, 16-channel point. Movie Gen, Step-Video and LTX-Video compress further. The VAE is built on MAGVIT-v2’s (LIT-623) and takes its temporally causal convolution, with all padding at the start of the time axis (§2.1). It replaces MAGVIT-v2’s lookup-free quantization with a continuous latent trained on L1, LPIPS and KL losses.
  • Text and video share one sequence and differ only in their adaLN. The “expert” adaLN is compared against SD3’s MM-DiT (LIT-449) at equal parameters and at equal depth (Fig. 8). This design choice is what distinguishes CogVideoX. HunyuanVideo later moved to a FLUX-style dual-then-single-stream design, and Wan and Step-Video to cross-attention. The modulation itself is DiT’s (LIT-448): following DiT, the diffusion timestep drives an adaLN, and CogVideoX’s change is to give vision and text tokens separate adaLN parameters within one transformer stack.
  • Full 3D attention beats factorized 2D+1D attention and costs more. It has lower FVD early in training (Fig. 8b), and the factorized variant is reported unstable. No values are given. One forward pass at 768×1360×5s takes 9.60s, against 4.17s for factorized attention, a ratio that falls to 1.67× at 480×720×6s and 1.08× at 256×384×6s (Table 8). This is where the line gives up VDM’s factorization.
  • Explicit uniform sampling. Timestep ranges are split across data-parallel ranks, which lowers validation loss at every timestep at 40k steps, e.g. 0.116 against 0.119 at t=500 (Table 9).

Where the hedges are

Per DP-010:

  • The ablations arrived late. The version-1 preprint’s ablations were loss curves only, and version 3 dropped two of them. The architecture ablations (Figs. 8 and 10) are absent from version 1 and present in version 3. Version 2 was not checked. Their model size and step count are not stated in the text.
  • The released 2B model uses the rejected option. CogVideoX-2B uses sinusoidal position encoding, not the RoPE the ablation favours (App. Table 5).
  • The claimed output grew between versions. Version 1’s introduction gives the open model as 720×480, 6s at 8 fps. Version 3 claims 768×1360, 10s at 16 fps. The paper does not say whether a different checkpoint produced the new figures, and the Kling human-evaluation table is unchanged between them.
  • The reported model may have changed between versions. Version 1 says the reported model’s captions came from the frame-caption summarization pipeline, and version 3 removed the sentence. The 2B VBench row also changed between versions, while the 5B row and the Kling table did not.
  • The captioning claim is not ablated. The abstract says dense recaptioning “significantly improves generation quality”.
  • The scaling claim is two model sizes.
  • Quality fine-tuning costs semantic ability. Fine-tuning on the top 20% of data caused “a slight degradation in the model’s semantic ability” (App. A). This appears only in the appendix.

Standing in the anthology

This is the node the open line branches from. Its full 3D attention became the default, and HunyuanVideo and Wan kept its VAE ratio. Its expert adaLN did not survive, and the successors split two ways: HunyuanVideo went to FLUX-style joint streams, and Wan, Step-Video and LTX-Video went to cross-attention (LTX-Video calls it “better than MM-DiT” without an ablation). The successors disagree on how text should enter a video DiT, and none of them tests the alternatives against each other at scale.

The successors also use it as their baseline. Wan (LIT-619) puts CogVideoX1.5-5B at 82.17 on VBench against Wan 14B’s 86.22 and Wan 1.3B’s 83.96 (Table 4), and includes CogVideoX’s VAE in a reconstruction plot whose values are not printed (Fig. 7). LTX-Video (LIT-618) runs CogVideoX-2B at 768×512 and 40 steps rather than its own 720×480, and its raters prefer LTX-Video in 85% of text-to-video pairs (Table 2). CausVid (LIT-631) distils a teacher “similar to CogVideoX”: a 10s video takes CogVideoX-5B 208.6 s at 0.6 fps, against CausVid’s 1.3 s to the first frame and 9.4 fps (Table 3), and on zero-shot image-to-video CausVid scores 92.0 / 65.0 / 28.9 on VBench temporal quality, frame quality and text alignment against CogVideoX-5B’s 87.0 / 64.9 / 28.9 (Table 6).

Read in full as NOTE-346.

Backlinks