Source
Black, Janner, Du, Kostrikov and Levine (2023), LIT-776, §4.2–4.3. Liu et al. (2025), LIT-779, §4, Eqs. 6–9, and App. A. Albergo, Boffi and Vanden-Eijnden (2023), LIT-645, Cors. 10 and 18. Fan et al. (2023), LIT-778, Lemmas 4.1–4.2, App. A.2–A.3.
The account
A likelihood-ratio gradient needs the action’s log-density. REINFORCE and PPO estimate the reward gradient as reward times ∇ log π(action). Both need that density in closed form, and PPO’s clipped ratio needs it under two sets of weights.
As one action, a diffusion sample has none. DDPO (LIT-776, §4.2) casts reward-weighted regression as a one-step MDP whose action is the final image. The diffusion loss “does not involve an exact log-likelihood”, only a variational bound, so that procedure “is not theoretically justified”. Cast instead as a T-step MDP whose state is (prompt, t, x_t) and whose action is x_{t−1}, each step of a standard sampler is an isotropic Gaussian. That “allows for the evaluation of exact log-likelihoods and their gradients” (§4.3).
A flow’s ODE step is not a policy step. A flow-matching sampler integrates dx = v dt, so each step is a deterministic function of the last. Flow-GRPO (LIT-779, §4) names two failures. The ratio needs p(x_{t−1} | x_t), which “becomes computationally expensive under deterministic dynamics due to divergence estimation”. And there is “no randomness beyond the initial seed” to explore with. Strictly, a deterministic step’s transition is a point mass, so the per-step ratio is not defined at all. Only the whole chain’s density exists, through the divergence integral.
A same-marginal SDE restores the policy without retraining. The stochastic-interpolant paper (LIT-645, Cors. 10 and 18) shows that one learned velocity and score define an ODE and, for every noise level ε(t) ≥ 0, an SDE with the same marginals, with ε chosen after training. Flow-GRPO uses that family (Eq. 7, proved in App. A). For the linear path the score is a function of the velocity, so the SDE needs only the trained v_θ (Eq. 8). Its Euler–Maruyama step (Eq. 9) is an isotropic Gaussian with variance σ_t²Δt. Two consequences are closed-form algebra: the ratio between new and old weights is a density ratio of two Gaussians, and the KL between the tuned and reference policies at a step is the squared difference of their means over 2σ_t²Δt (§4).
Summed, the step KL bounds the KL on the sample. DPOK (LIT-778, Lemma 4.2) shows that the KL between the tuned and reference models’ final images is at most the KL between their chains, by the data processing inequality, and the Markov property splits that into a sum of per-step KLs between Gaussians. So the closed-form step KL is computable and, summed, also bounds the divergence on what is actually sampled. DPOK’s gradient drops the part that flows through earlier steps (App. A.3), so the bound is minimized approximately. Its Lemma 4.1 is the same per-step REINFORCE as DDPO’s, derived concurrently. THEORY-054 uses the same inequality in noise space.
What it explains
SOTA-442, all three of its clauses. The practice says to roll the model out as a same-marginal SDE, built from its own velocity, and to anchor each step with the closed-form KL. On this account the SDE is what makes the model a policy at all, building it from the velocity is what lets a model trained only for the ODE be used, and the closed-form KL comes free with the Gaussian step. The practice’s noise level a is an exploration knob, and the account says why it has to be nonzero. Flow-GRPO’s own sweep has a = 0.1 learning slowly and a = 0.7 and 1.0 equally fast (Fig. 7b).
It also says why the practice can sample its evaluations with the ODE. The SDE and the ODE share marginals for any velocity, including the tuned one, so a policy improved through the SDE is the same generative model sampled the other way. SOTA-265 records the same freedom used for sample quality.
Where it stops
- Continuous time only. The marginal equivalence holds for the SDE, not for its ten-step discretization. Flow-GRPO’s 10-step rollouts show colour drift and blurred detail (Fig. 19), which the 40-step ODE evaluation does not share. Its σ_t = a√(t/(1 − t)) is infinite at the noise end, where sampling starts, and the paper does not say how the first step is taken.
- Likelihood-ratio methods only. Methods that differentiate a reward through the sampler, or that fit preferences through the diffusion loss as Diffusion-DPO does (LIT-774) or Flow-DPO in velocity form (LIT-773), need no per-step policy density. The account says nothing against them. ReFL (LIT-788) is the reward-gradient case. It backpropagates the reward through one late step and needs no step density. Its stated reason for not using RLHF, that a diffusion model “cannot yield likelihoods”, is true of the whole chain and not of a step.
- It does not say a KL anchor is enough. It says the anchor is computable per step. Flow-GRPO’s evidence that the anchor prevents reward hacking is measured with other reward models, and DDPO ran with no anchor at all and degraded under over-optimization. DPOK’s ablation (LIT-778, §5.3) is the same kind of evidence, one prompt, judged by an aesthetic predictor. For a language model under PPO, Gao et al. (LIT-795) found a KL penalty acted like early stopping rather than improving true reward at a given distance.
- It does not reach a one-step generator. A distilled one-step model has one deterministic step and no SDE family to swap in, so this construction gives it neither a policy density nor a tractable anchor. That is the case SOTA-302 covers, by moving the regularizer into noise space, and THEORY-054 is the record’s account of why that works.