HaCohen et al., Lightricks (2024) — ARXIV-2501.00103
Key takeaways
- The line’s most aggressive compression. The VAE compresses 32×32 in space and 8× in time into 128 channels, a 1:8192 pixels-to-tokens ratio with no patchifier. CogVideoX and HunyuanVideo are 1:1024 (Table 1). The transformer therefore sees 8× fewer tokens than CogVideoX and HunyuanVideo, and 4× fewer than Movie Gen and PyramidFlow (1:2048). The paper calls it “four times the typical ratio” (§2.1, Table 1). This is where the speed comes from.
- The decoder denoises. It is trained on t ∈ [0, 0.2]. The §4.4 user study decodes at t = 0.05, and the paper gives no default inference t. The high-frequency detail a compressed latent cannot hold is generated during decoding rather than reconstructed. Wan’s related work names this as LTX-Video’s distinguishing idea.
- The speed claim is 121 frames of 768×512 in about 2s on an H100 with 20 steps (§1). The quality survey used 40 steps.
- Human preference. 85% text-to-video and 91% image-to-video win rates across the survey, against 38% and 47% for CogVideoX-2B (Table 2). The survey used 1,000 prompts and 20 raters.
- RoPE frequency spacing. Exponential spacing gives lower loss than inverse-exponential over about 140k steps, with everything else fixed (Fig. 17). This is the paper’s one clean ablation.
Where the hedges are
Per DP-010:
- The central idea has no numbers. The denoising decoder was “strongly preferred” over t = 0 in an internal study, with no count, rater number or win rate (§4.4).
- “Greatly enhances GAN stability” (§2.1.2) is supported by one frame (Fig. 16).
- “Real-time” and the quality numbers use different settings. Speed is measured at 20 steps and quality at 40. No timing table is given for any model, and VAE and text-encoder costs are not broken out.
- The baselines were run away from their native resolution. The paper says every model ran at 768×512 “with their default configurations”. CogVideoX-2B’s native resolution is 720×480 (LIT-622), which this paper does not mention.
- The VAE has no reconstruction metric. No PSNR, SSIM, LPIPS or rFVD is reported for it, so the claim that quality is maintained at 1:8192 cannot be checked.
- Architecture claims are asserted. Cross-attention “work[s] better than MM-DiT” and RMSNorm QK-norm are both stated as “we found”.
- The training details are missing. §4.1 is truncated mid-sentence (”…ADAM-W optimizerAfter pre-training…”), with no steps, batch size or data size.
Standing in the anthology
LTX-Video is the speed end of the line. It asks how much the latent can be compressed before the generator suffers, and answers it with a decoder that restores what compression discards. CogVideoX found 16×16×8 “extremely difficult” to converge, and LTX-Video goes further, which makes the decoder the load-bearing part. That part is the least evidenced.
Its diffusion side comes from SD3 (LIT-449): rectified flow with velocity prediction, and SD3’s timestep density, which this paper calls “log-normal”. The density is shifted towards high noise as the token count grows, with μ = 1 at about 256 tokens and μ = 2 at about 4096, and the paper cites SD3 and simple diffusion for the reason (§2.5.2). What it changes is how text enters: cross-attention instead of SD3’s MM-DiT, which it says works better without giving numbers.
Self Forcing (LIT-629) uses LTX-Video as its efficient diffusion baseline. On one H100, LTX-Video at 768×512 runs at 8.98 fps with 13.5 s first-frame latency and scores 80.00 on VBench. Self Forcing runs at 17.0 fps with 0.69 s latency and scores 84.31, at 832×480 (its Table 1).
inference-optimization is justified because the paper exists to make
generation fast, and every design choice is argued from that goal.
Read in full as NOTE-350.