Source
Yang, Teng et al. (2024), LIT-622 — CogVideoX.
The claim
The early video models factorized attention to save compute: spatial attention within each frame, then temporal attention across frames at each position. VDM (LIT-627), Video LDM (LIT-621) and Make-A-Video (LIT-632) all do this. In a transformer over a compressed latent, attend over all space-time tokens at once.
The recommendation comes from CogVideoX (LIT-622), which adopts full 3D attention on the argument that under factorized attention a moving object in one frame cannot attend directly to itself in the previous frame, only through other patches (Fig. 5). It compares the two in its video DiT (Fig. 8). The text reports the whole result in two sentences. With factorized 2D+1D attention, “the FVD will become much higher than 3D attention in early steps”, and factorized attention “is unstable and prone to collapse”. The only setup it gives is the evaluation: FVD on 500 WebVid test videos. Model size, steps, resolution, patch size, parameter matching and FVD values are not stated. The loss curve (Fig. 10b) is cited without comment.
The cost is measured. Table 8 gives inference time for one DiT forward step on an H800. Full attention is 1.08× the factorized time at 256×384, 1.67× at 480×720 and 2.30× at 768×1360. The ratio grows with resolution because attention is quadratic in tokens.
Budget for that cost, because it grows. Attention is quadratic in token count, so it dominates at video lengths. Wan (LIT-619 §4.3) measures attention at up to 95% of step time at 1M tokens. That is why the line turns to context parallelism (2D ring and Ulysses) and to more aggressive latent compression (Step-Video LIT-624, LTX-Video LIT-618). Full attention decides the systems design as well as the quality.
Conditions
- One comparison, reported in words. The evidence is a direction (“much higher… in early steps”) and an instability observation. It is not clear whether the FVD gap lasts past early training. The ablation is not in the paper’s first arXiv version. Step-Video (LIT-624) ran a 4B comparison and reports “better, particularly high motion” with no numbers. So the record has two unquantified agreements and no measured margin.
- The cost grows with resolution. At 256×384 full attention is nearly free (1.08×). At 768×1360 it is 2.30×, and it keeps rising with token count. No report says where factorizing becomes worth it again.
- The motivation is motion. Factorized attention can relate two positions in different frames only through an intermediate step, and large motion is where related content changes position. That is an argument, not a measurement.
- Full attention gives up plug-in compatibility. AnimateDiff (LIT-633) keeps the image model’s spatial layers seeing one frame at a time so that personalized image checkpoints can be dropped in. Joint 3D attention removes that option. This is a reason some systems factorize on purpose.
- A controlled comparison points the other way, at small scale. W.A.L.T (LIT-661, Table 3b) alternates per-frame spatial blocks with local 3D-window blocks. On UCF-101 at 35K steps it matches or beats full 3D attention in every block: FVD 55.3–59.6 against 59.9, at 1.7× the training speed. It is a different group and a numbered result. It is not the 2D+1D factorization CogVideoX compared, and it is 17 frames at 128px, where full attention is cheap. So it does not contest the practice, but it is the only numbered comparison against full attention the record holds.
- The inverse question is untested. Nothing in the record tests whether a factorized model given the extra compute back, as depth or width, would close the gap.