Kong et al., Tencent Hunyuan Foundation Model Team (2024) — ARXIV-2412.03603
Key takeaways
- The line’s only video scaling law. Chinchilla-style envelopes (LIT-068) are fit as N_opt = a·C^b and D_opt = a·C^b, first for text-to-image and then for video. The image fit favours parameters (0.563 against 0.433, §4.4.1). The video fit, initialized from the image-optimal checkpoints, favours data (0.362 against 0.629, §4.4.2). The image proxy family is “DiT-T2X” at seven sizes from 92M to 6.6B parameters. The video fit uses five sizes, B2 to G2 (Fig. 10d). The token amounts are “only related to the first stage of training for images and videos respectively” (§4.4.2). The 13B size was chosen from the fits “taking into account the training consumption and inference cost”, not read off them. No other report in the line measures it.
- Architecture. 20 dual-stream blocks feed 40 single-stream blocks with full attention and 3D RoPE. Text comes from a decoder-only multimodal LLM with a bidirectional token refiner, and pooled CLIP-L is added to the timestep embedding. Training follows SD3 (LIT-449): flow matching, with t drawn from SD3’s logit-normal density (§4.5.1). The dual-to-single stream blocks are credited to FLUX, not to SD3’s MM-DiT directly.
- The VAE. 4×8×8 into 16 channels, causal, trained from scratch on 4:1 video to image data. It reaches PSNR 35.39 on MCL-JCV against CogVideoX-1.5’s 33.22 (Table 1). Fine-tuning with tiling randomly on or off removes tiling seams. Its causal 3D convolution is MAGVIT-v2’s (LIT-623), adopted by name “to handle both videos and images”: T+1 frames become T/4+1 latent frames, so an image is a one-frame video. Where MAGVIT-v2 quantizes with LFQ, HunyuanVideo keeps a continuous latent trained with a KL term.
- Few-step sampling. A timestep shift s·t/(1+(s−1)t) is used, with s raised from 7 at 50 steps to 17 below 20 steps. It is compared against Movie Gen’s (LIT-626) linear-quadratic schedule at 10 steps, qualitatively only (Fig. 11b): two sample videos, from which the authors read the larger shift as the better choice. CFG distillation gives about 1.9× speedup.
- Evaluation. 1,533 prompts, each generated once, “avoiding any cherry-picking” (§6.1), were rated by 60 professional raters. How many ratings each video received is not stated. Overall is 41.3% against 37.7% for the best anonymized competitor, and motion quality is the best at 66.5% (Table 3).
Where the hedges are
Per DP-010:
- “Reduce computational resource requirements by up to 5×” (intro) is supported by no experiment in the paper.
- The scaling law is fit on a different model. The proxy trains with DDPM v-prediction, T5-XXL and cross-attention. The shipped model uses flow matching, a multimodal LLM and dual/single streams (§4.4.1). That the fit transfers is assumed.
- No design choice is ablated. Full attention, the dual-to-single stream, the multimodal-LLM encoder and the token refiner are all adopted or justified by citation. Data filters were checked by “simple experiments using a smaller HunyuanVideo model” whose results are not shown.
- The headline metric is undefined. Table 3’s “Overall” is not defined. On visual quality the model ranks fourth of six (95.7% against 97.7%).
Standing in the anthology
HunyuanVideo was the open baseline Wan measured itself against, in both its VAE comparison and its human evaluation. On Wan’s own benchmark (LIT-619, Table 2) the open HunyuanVideo scores 0.673 against Wan 14B’s 0.724 and Wan 1.3B’s 0.689, and on VBench 83.24 against 86.22 (Table 4). Wan’s VAE reconstructs 2.5× faster than this one on the same hardware, with PSNR shown only as a plot (Fig. 7). The two reports disagree most clearly on the text encoder. HunyuanVideo replaces T5 with a multimodal LLM on the strength of a claim it doesn’t test. Wan tests the same question, finds the LLM ahead by 0.1 FID, and keeps umT5 anyway. The encoder question is open, and neither report settles it.
It is the main open baseline for two other reports as well. Step-Video (LIT-624) reports its 16×16×8 VAE level with this 8×8×4 one on its own benchmark (PSNR 39.37 against 39.56, rFVD 3.61 against 4.17), and wins 615, ties 313 and loses 361 against HunyuanVideo on Movie Gen Bench (Tables 14 and 15), with frame count and resolution not matched. Open-Sora 2.0 (LIT-634) trains its $200k model on this VAE and measures it at LPIPS 0.046 and PSNR 30.240 (Table 1). That is well ahead of Step-Video’s VAE (0.082 and 28.719), and ahead of Open-Sora’s own 4×32×32 DC-AE on LPIPS though slightly behind it on PSNR. Its 100-prompt human evaluation calls Open-Sora 2.0 “comparable to” HunyuanVideo.
The scaling fit is the most quantitative claim about video compute allocation in the record. It should be cited with its proxy caveat.
Read in full as NOTE-344.