LIT-773: Improving Video Generation with Human Feedback

Jie Liu, Gongye Liu, Jiajun Liang, Ziyang Yuan, Xiaokun Liu, Mingwu Zheng, Xiele Wu, Qiulin Wang, Menghan Xia, Xintao Wang, Xiaohong Liu, Fei Yang, Pengfei Wan, Di Zhang, Kun Gai, Yujiu Yang and Wanli Ouyang, CUHK MMLab, Tsinghua, Kuaishou (Kling), Shanghai Jiao Tong University and Shanghai AI Laboratory (2025) — ARXIV-2501.13918. NeurIPS 2025. Read at v2 (27 Oct 2025), appendices included. v1 (23 Jan 2025) was checked for what changed. v1 lists an eighteenth author, Wenyu Qin, whom v2 drops. v1 calls the dataset “200k” in its conclusion, has a human evaluation on TA-Hard prompts (its Fig. 10) that v2 drops, and says plainly that VideoReward “is vulnerable to reward hacking”. The method, the hyperparameters and the main tables are the same. The project’s name for the whole is VideoAlign.

Key takeaways

• The preference data is for current video models (§3.1, Table 1, App. G). 16,000 prompts across eight categories, expanded with GPT-4o and rewritten in-house, rendered by 12 systems from Gen2 and SVD to Kling 1.5 and Gen3, 108k videos, 182k pairs. Each pair has a 1–5 score per video and a win, tie or loss on each of visual quality (VQ), motion quality (MQ) and text alignment (TA). Three trained annotators per sample, with a reviewer settling disagreements. 13,000 pairs whose prompts never appear in training are held out.
• Pairs beat scores, and ties need modelling (§3.2, Figs. 2–3, Table 6). Trained on the same annotators’ labels, a Bradley–Terry model beats pointwise regression at every data fraction. A plain BT model gives tied pairs large reward gaps. The tie-aware version (BTT, Eq. 1, θ = 5) keeps ties near zero. It matches BT without ties and gains with them: overall 61.50 against 61.22 ties-included, VQ 60.52 against 52.33.
• Separate reward tokens per dimension (§3.2). [VQ] and [MQ] tokens sit after the video and before the prompt, so they cannot see the text. [TA] sits after the prompt. The aim is that a video’s visual score does not change with the prompt it is paired with.
• VideoReward leads on the authors’ modern-video benchmark (Table 2). On VideoGen-RewardBench, 26.5k pairs the authors had annotated over third-party videos from 12 recent systems, overall accuracy is 61.26 with ties and 73.59 without. The next best, VisionReward, has 56.77 and 67.59. On GenAI-Bench, short clips from older models, it is 49.41 and 72.89 against VisionReward’s 51.56 and 72.41.
• Flow-DPO, and why β must not follow the derivation (§4.2, Eqs. 4–6, Lemma B.1, Fig. 6). Under the straight path, the noise-prediction error is (1 − t)² times the velocity error. Substituting into Diffusion-DPO’s loss gives Flow-DPO with β_t = β(1 − t)², full strength at the data end and zero at the noise end (t = 1 is noise here). Trained that way, the model “overfit[s] the objective at higher noise levels” and reward-hacks. Dropping the factor, so β is constant, “leads to more stable training and improved alignment across all reward dimensions”. The authors cite the analogy of DDPM’s unweighted loss, and uneven training across timesteps (Min-SNR). Flow-RWR drops the same factor (Eq. 9).
• The multi-dimensional result (Table 3), aligning to the mean of the three VideoReward scores, win rates against the base model:
  • Constant-β Flow-DPO on VideoGen-Eval prompts: VQ 90.95, MQ 81.01, TA 68.26. On TA-Hard: 77.46, 71.43, 73.24.
  • The β(1 − t)² variant: VQ 88.44 and MQ 91.23, but TA 28.14 on VideoGen-Eval and 38.03 on TA-Hard, below the base model’s 50.
  • SFT on the preferred videos and Flow-RWR land between 46 and 79.
  • VBench total is 83.19 for the base model and 83.41 for Flow-DPO. VBench quality falls under every method, from 84.37 to 84.19 for Flow-DPO.
  • Humans, two annotators and a third to break disagreement, on VideoGen-Eval: Flow-DPO preferred 44.0%, base 27.0%, ties 29.0% (Fig. 5). On VQ alone 31.8 / 21.0 / 47.2, MQ 35.5 / 32.2 / 32.2, TA 29.5 / 22.2 / 48.2.
• Aligning to text alignment alone (Table 4). Constant-β Flow-DPO wins 69.09 (VBench prompts), 65.49 (VideoGen-Eval) and 84.51 (TA-Hard). The β(1 − t)² variant gets 63.67, 55.95 and 71.83. SFT gets 52.88, 53.81 and 64.79.
• Reward guidance at inference needs a reward trained on noisy latents (§4.4, Eq. 11, Table 5, App. E). The KL-regularized optimum is the base distribution tilted by exp(w·r). For rectified flow that is a shift of the velocity by −w·t/(1 − t)·∇r. The reward is a small model built from the generator’s own early layers and trained on pairs noised identically. With MQ-only guidance on TA-Hard, the noise-trained reward wins 74.6 and a clean-trained one 38.6, worse than no guidance. Predicting x₀ and scoring that gave “unusable videos”. Weights on the three dimensions trade them off at inference: all on TA gives TA 70.42 and MQ 46.48, all on VQ and MQ gives 86.43 and 93.23 with TA 26.65.
• The settings (App. I, App. D). The base is an internal 1B-parameter DiT with a 3D VAE and T5, trained by flow matching. Alignment uses LoRA, rank 64 and alpha 128 on the attention projections, Adam at 5·10⁻⁶, one epoch, batch 64, on 16 A800s, with Flow-DPO β = 500. Full fine-tuning “can degrade the model’s performance or potentially lead to model collapse”.

Where the hedges are

Per DP-010:

• The judge is the teacher. Training pairs are relabelled by VideoReward, and every automatic win rate is VideoReward’s, “ensuring that models optimised on these synthetic labels can be evaluated fairly by the same reward function” (§5.2). That makes the win rates a measure of how well each method optimizes VideoReward, not of video quality. The independent automatic number, VBench, moves by 0.2 points.
• “Reward hacking” is one reward dimension falling while others rise. For the β(1 − t)² variant, the evidence is VideoReward’s own TA score dropping below the base while its VQ and MQ climb. No human evaluation of that variant, of SFT or of Flow-RWR is reported. The human study covers only constant-β Flow-DPO against the base. The labels themselves were filtered by the scalarized reward, so a gain in VQ and MQ at TA’s expense can satisfy the training pairs.
• The β comparison sweeps different ranges. Fig. 6 runs constant β at 100, 500 and 1000 and the β(1 − t)² variant at 2000 to 8000. Since β(1 − t)² ≤ β, a larger nominal β for the decaying variant is a reasonable match. But no β is run both ways, and the constant-β curve falls at 1000.
• The quality ceiling is admitted. “Excessive training with Flow-DPO led to a significant deterioration in model quality”, which LoRA was used to prevent (§7). The run length was therefore chosen. No curve shows the rise and fall. VideoReward still assigns high scores where “human assessments indicate a marked decrease in video quality”.
• One internal model. All alignment results are on a 1B research model that is not released. One run per setting, no intervals. Human evaluation is on VideoGen-Eval’s prompts, about 400 of them by v1’s caption.
• VideoReward’s lead is on its home benchmark. VideoGen-RewardBench was annotated by the authors’ process over the same era of models as the training set (Fig. 10). On GenAI-Bench, VideoReward is scored by its TA output alone (App. H.2), and VisionReward is ahead with ties counted. Separate tokens, the third design choice, gain on GenAI-Bench and lose slightly on VideoGen-RewardBench overall with ties (61.26 against 61.50, Table 6), against the text’s “further improves performance”.
• Reward guidance is scored by a different reward. Guidance uses the latent reward model and Table 5’s win rates are presumably VideoReward’s. The paper does not say which model scores Table 5.

Which comparisons are like for like

• SFT, Flow-RWR and both Flow-DPO variants (Tables 3–4) share the base model, the relabelled data, LoRA and the optimizer settings. Only the loss differs. The β ranges for the two Flow-DPO variants differ (Fig. 6).
• Human evaluation (Fig. 5) compares Flow-DPO only with the base, same seeds.
• Reward models (Table 2) are run by the authors with per-model averaging rules. LiFT’s discrete scores tie often, and ties were broken at random for the ties-excluded accuracy. The baselines’ training data are older and differ in era.
• Flow-NRG against an unguided baseline (App. E) is like for like within the ablation. It is not compared with the training-time methods.

Standing in the anthology

It extends Diffusion-DPO (LIT-774). Flow-DPO is that loss with the noise error rewritten as a velocity error. One timestep and one noise are shared across the pair, and the reference is frozen; the pseudocode in App. C is Diffusion-DPO’s with velocities in place of noise. What it adds is the finding that the timestep weighting the substitution produces, (1 − t)², is the wrong one for a rectified-flow video model, and that a constant weight works. It also turns the KL-regularized optimum both DPO papers start from into an inference-time guidance rule, which Diffusion-DPO does not have. Its offline setting inherits Diffusion-DPO’s assumption that preference pairs come from the reference model, and breaks it the same way: the pairs are videos from 12 other systems.

The record’s flow-RL papers use it. Flow-GRPO (LIT-779), from an overlapping group, cites this paper with Diffusion-DPO for the Flow-DPO and Flow-RWR baselines it trains online and offline on SD3.5-M. Those baselines improve with training and stay below Flow-GRPO on GenEval. Flow-GRPO does not say whether its Flow-DPO keeps the (1 − t)² factor or drops it as recommended here. Step-Video-T2V (LIT-624) applies Diffusion-DPO to a 30B video flow model and reports that β = 5000 with learning rate 10⁻⁸ converges too slowly. This paper’s β = 500 at 5·10⁻⁶ with LoRA is the same direction, as NOTE-338 advises. The two β values are not on one scale: Diffusion-DPO folds the number of timesteps into its β, and this loss weights a velocity error.

SOTA-442 is the online, policy-gradient route for the same kind of model. This paper is the offline one, with the KL anchored through β rather than per step. Its limits point the same way that practice’s conditions do: run length decides quality, and the reward model scores its own success.

Filed without a NOTE. The takeaways come from one full reading of v2, appendices included, done for this filing. v1 was compared for authors, the conclusion and limitations, the β ablation and its figures. Tables 3–4 and Figs. 5–6 were checked against the rendered pages.