LIT-779: Flow-GRPO: Training Flow Matching Models via Online RL

Liu, Liu, Liang, Li, Liu, Wang, Wan, Zhang and Ouyang, CUHK MMLab, Tsinghua, Kuaishou (Kling), Nanjing University and Shanghai AI Laboratory (2025) — ARXIV-2505.05470. NeurIPS 2025. Read at v5 (27 Oct 2025); v1 (8 May 2025) was checked for what changed.

Key takeaways

• ODE to SDE without retraining (§4, Eqs. 7–9, App. A). The marginals of dx = v dt are matched by the reverse SDE dx = [v − (σ_t²/2)∇log p_t]dt
  • σ_t dw for any σ_t. For the linear path the score is a function of the velocity, ∇log p_t(x) = −x/t − ((1 − t)/t)v(x) (Eq. 27). So the sampler needs only the trained v_θ, and Euler–Maruyama gives the update (Eq. 9). The schedule is σ_t = a·√(t/(1 − t)) with a = 0.7.
• The point is a tractable policy. Each step is an isotropic Gaussian, so the PPO-style ratio is a density ratio and the KL to the reference model is closed form, a weighted squared difference of velocities (§4). The ODE sampler would need divergence estimates for the same ratio, and explores nothing beyond its initial noise.
• Denoising reduction (§4, §5.3, Fig. 7a, App. D). Rollouts use 10 sampling steps, evaluation the default 40. This is over 4× faster in wall-clock to the same final reward. Five steps is no faster. The 10-step SDE samples show visible artifacts — colour drift, blurred detail (Fig. 19) — and the relative advantage still carries signal.
• The headline results (Tables 1–2), SD3.5-M at 512 px with LoRA rank 32, group size 24, KL β 0.04 (0.01 for PickScore), on 24 A800s (App. B):
  • GenEval 0.63 → 0.95; position 0.24 → 0.99, counting 0.50 → 0.95.
  • Visual text accuracy 0.59 → 0.92.
  • PickScore 21.72 → 23.31.
  • On FLUX.1-Dev with PickScore, 22.62 → 23.97 on DrawBench (App. C.4, Table 7).
• The KL term earns its place (Table 2, Fig. 6, App. C.5). Without it, GenEval training takes DrawBench Aesthetic from 5.39 to 4.93, DeQA from 4.07 to 2.77 and ImageReward from 0.87 to 0.44. With it they are 5.25, 4.01 and 1.03, at the same 0.95. On PickScore, removing the KL leaves the quality metrics intact but collapses diversity: “different seeds producing nearly identical results”. The KL run reaches the KL-free reward, only later. The paper’s point is that this is not early stopping. It does not cite Gao et al. (LIT-795), whose finding it answers, and the two measure differently: Flow-GRPO compares at equal reward with other reward models as judges, Gao et al. compare a gold reward at equal KL.
• Group size and noise level matter (Figs. 5, 7b). G = 12 and G = 6 collapsed on PickScore and G = 24 did not. On OCR, a = 0.1 learns slowly, 0.7 and 1.0 equally fast, and “too much noise” degrades images to zero reward.
• It beats the alternatives it ran (Fig. 4, App. C.1, Fig. 8, Tables 5–6). With the same group size per step it beats SFT on the best sample, reward-weighted regression and DPO, each offline and online, on GenEval. On PickScore it beats DDPO (run through the same SDE), which “eventually collapses”, and ReFL. Against ORW its test reward is 29.89 at step 960, where ORW has fallen from 29.15 at step 480 to 23.05.

Where the hedges are

Per DP-010:

• “Outperforming GPT-4o” on GenEval (§1, Table 1). The training prompts are generated by GenEval’s own scripts and templates, weighted towards the base model’s weak categories. The reward is GenEval’s detector pipeline. The test set is deduplicated against training, but it comes from the same distribution. GPT-4o’s 0.84 is zero-shot. The out-of-template evidence is T2I-CompBench++ (Table 3), where 2D-spatial rises 0.2850 → 0.5447 and texture falls 0.7338 → 0.7236. Held-out object classes go 0.64 → 0.90 (Table 4).
• “Very little reward hacking” (abstract) is measured by other reward models — Aesthetic, DeQA, ImageReward, UnifiedReward and PickScore on DrawBench. There is no human evaluation, and no diversity metric in the main table. Diversity appears only against ORW (Table 6: 0.96 → 1.02 on 50 prompts × 64 images). On the PickScore task the reward and an evaluation metric are the same model, which the paper says may be why quality looks intact without KL (§5.3). v1’s abstract said “little to no reward hacking”; v5 says “very little”.
• v1 tried a multi-reward ensemble and dropped it (v1 §5.3). Adding quality and preference rewards to the task reward “led to local blurriness and reduced diversity”. v5 no longer mentions it.
• “The first method to integrate online policy gradient RL into flow matching models” (abstract). The paper cites concurrent GRPO on flow-matching speech (F5R-TTS), earlier online reward-weighted fine-tuning of flow matching (ORW) and adjoint-matching fine-tuning. The claim is true of policy gradient on image flow models, narrowly.
• The noise schedule is singular at the noise end. σ_t = a√(t/(1 − t)) is infinite at t = 1, where sampling starts, and the drift’s σ_t²/(2t) term goes with it. The paper does not say how the first step is taken.
• Evaluation samples with the ODE. Training optimizes the SDE policy and the reported images use a 40-step ODE (App. C.7). That is sound only because the two share marginals for the fine-tuned model too. Discretized at 10 steps they visibly do not (Fig. 19).
• One run per setting, no seeds or intervals. Most curves are figures, and GenEval training to 0.95 takes several thousand GPU hours (Fig. 7a).

Which comparisons are like for like

• SFT, RWR and DPO (Fig. 4, App. C.1) share the model, the reward and the group of samples per step; only the update rule differs. Fig. 8’s caption says its x-axis counts training prompts because DPO’s tuned batch size differs. Only online DPO’s β is reported as searched.
• DDPO and ReFL (LIT-788) (Fig. 8) are run by the authors on SD3.5-M with PickScore. DDPO goes through Flow-GRPO’s own SDE, so this compares the update rule rather than DDPO as published. Their hyperparameters are not given.
• ORW (Tables 5–6) follows Flow-GRPO’s PickScore setup, with ORW’s step frequency chosen from four values.
• Table 1 is Flow-GRPO trained on GenEval against other models’ published zero-shot GenEval. Not like for like.

Standing in the anthology

Its recipe, a same-marginal SDE rollout anchored by a per-step KL, is SOTA-442.

It extends GRPO (LIT-127). The group-normalized advantage (Eq. 4), the clipped ratio and the KL to a reference (Eq. 5) are DeepSeekMath’s, unchanged. What Flow-GRPO adds is a way to make a deterministic generator into a policy GRPO can use. It is the record’s first evidence for SOTA-145 outside language: the baseline from a group of samples for the same prompt, with no critic, carries over to images. The group-size result says the group has to be large enough — 24 here — for the estimate to hold. The objective is the published one, with std normalization. SOTA-146’s corrections are not applied; Dr. GRPO’s length bias cannot arise here because every trajectory has the same T, but the std term can.

On SOTA-265. The conversion rests on that practice’s identity and puts it to a second use. Eq. 7 is the family of SDEs the stochastic-interpolant paper (LIT-645) derives, which Flow-GRPO cites alongside Score SDE (LIT-722). One trained velocity, and therefore one score, gives an SDE for every σ_t with the same marginals. Flow-GRPO depends on the fact that σ_t is not fixed by training: it chooses one after training, on a frozen model, with no retraining. It extends the practice in what σ_t is chosen for. SOTA-265 tunes the coefficient for sample quality. Here it is an exploration knob for RL, set by how fast reward rises (a = 0.7, Fig. 7b), and the sampler it tunes is used only during training. What it does not do is measure the quality side: no FID or quality score is reported as a function of a, only the qualitative statement that 40-step SDE and ODE samples are “visually indistinguishable” (App. D) and that too much noise breaks generation. It is a second group using the freedom, not a second measurement of what tuning it buys.

The base model is SD3.5-Medium from the SD3 line (LIT-449), whose rectified flow transformer and published GenEval rows anchor Table 1, beside SDXL (LIT-566) at 0.55. Its offline and online Flow-DPO baselines are Diffusion-DPO’s loss (LIT-774): DPO (LIT-169) evaluated through denoising errors against a frozen reference, carried to flow matching by Flow-DPO (LIT-773), which drops the (1 − t)² weight the exact substitution produces. Its Flow-DPO and Flow-RWR baselines are those of LIT-773, from an overlapping group; Flow-GRPO does not say whether its DPO keeps the (1 − t)² weight that paper drops. Each step’s pair is the best and worst image of a group. They improve with training and stay below Flow-GRPO on GenEval (Fig. 4, App. C.1), and online beats offline, the direction Diffusion-DPO’s own conclusion anticipated.

DDPO (LIT-776) is where Flow-GRPO’s central move comes from: treat each denoising step as a Gaussian policy with an exact likelihood, and clip the ratio as PPO does. DDPO had a stochastic DDPM sampler. Flow-GRPO manufactures one from a deterministic flow, and adds a group baseline and a KL anchor, neither of which DDPO has. Flow-GRPO reruns DDPO through its own SDE on SD3.5-M with PickScore (Fig. 8). There DDPO’s reward rises more slowly and “eventually collapses”, and Flow-GRPO keeps improving. With DDPO’s hyperparameters unreported, that compares update rules, not DDPO as published.

The record now holds the two roots of this line, DDPO (LIT-776) and Diffusion-DPO (LIT-774), discussed above. DPOK (LIT-778) and ImageReward with ReFL (LIT-788) are now filed. ReFL is a baseline here and ImageReward one of the quality metrics. The other filed relatives are Step-Video-T2V (LIT-624), which applies Diffusion-DPO to a video flow model and finds modest gains, and HyperNoise (LIT-491), which is what SOTA-302 rests on. That practice says to steer a distilled generator through its input noise, because the KL anchor that stops reward hacking is intractable in weight space for a one-step model. Flow-GRPO is the opposite case, and the comparison sharpens the practice’s condition rather than contesting it. For a multi-step flow sampled stochastically, each step’s KL is closed form, and weight-space LoRA tuning with that anchor held quality on the reward models it was checked against. Nothing here tests a distilled model.

Filed without a NOTE: the takeaways come from one full reading of v5, appendices included, done for this filing. v1 was read for its abstract and §5.3 only. Figure values are read only where the text gives them.