Source
Black, Janner, Du, Kostrikov and Levine (2023), LIT-776 — DDPO, App. E.1, Fig. 10 and App. D.5.
What to do
A text-to-image diffusion model is sampled with classifier-free guidance: the step uses ε̃ = w·ε(x_t, t, c) + (1 − w)·ε(x_t, t), a mix of the conditional and unconditional predictions. When fine-tuning such a model on a reward, by policy gradient or reward-weighted regression, with several rounds of sampling from the current model:
- Choose the guidance weight w you will sample with, and keep it fixed.
- Compute the likelihoods or losses through the guided prediction ε̃, the quantity that actually produces the samples, not through the conditional prediction ε(x_t, t, c) alone.
- Do not train the unconditional objective on its own. The reward depends on the prompt, so there is nothing for the unconditional branch to be rewarded for.
DDPO calls this “CFG training”.
Evidence
The ablation (LIT-776, App. E.1, Fig. 10). On Stable Diffusion v1.4 with JPEG compressibility as the reward, the paper runs its sparse reward-weighted regression baseline (RWR_sparse) both ways. Training only the conditional ε-prediction, the reward stops improving after the first round and then falls back over the next ones, read from the plot. Training the guided prediction, it keeps rising across all of them. The paper: CFG training “has no effect after a single round of finetuning, but becomes essential for subsequent rounds”.
The reason offered is a hypothesis. Each update changes the conditional prediction but not the unconditional one, so the same w mixes two predictions that have drifted apart. The samples degrade, the next round trains on worse samples, “and so on”. Nothing in the paper measures the miscalibration directly.
DDPO’s own runs use guidance weight 5 for every method (App. D.5), and the appendix lists CFG training among the paper’s design decisions. The main text does not mention it, and DDPO itself is never run without it.
Conditions
- One algorithm, one reward, one curve. The ablation is RWR_sparse on compressibility, not DDPO’s policy gradient, and has one run per arm. The paper’s claim that it is “essential for methods that do more than one round of interleaved sampling and training” generalizes from that.
- Single-round training does not need it. By the paper’s own reading, the difference appears only once the model trains on its own updated samples. An offline method trained once on fixed pairs, such as Diffusion-DPO (LIT-774), is outside the claim; SOTA-445 is the record’s practice for that case.
- The fixed weight is baked in. The tuned model is calibrated for the w it was trained at. Sampling it at another guidance weight was not tested.
- The flow-matching successors are silent. Flow-GRPO (LIT-779) carries DDPO’s policy to flow models and does not say whether guidance is applied during training. Whether this matters for a model whose guidance has been distilled into the weights is untested.
- It is the RL counterpart of an Active practice. SOTA-424 trains one network for both scores by dropping the condition, then picks the weight at sampling. During reward tuning this practice keeps that network and fixes the weight before training instead.
Known implementations
- DDPO, the source’s code, with guidance weight 5.