LIT-774: Diffusion Model Alignment Using Direct Preference Optimization

Wallace, Dang, Rafailov, Zhou, Lou, Purushwalkam, Ermon, Xiong, Joty and Naik, Salesforce AI and Stanford (2023) — ARXIV-2311.12908. Published at CVPR 2024, per Flow-GRPO’s reference list. Read at v1 (21 Nov 2023), the only arXiv version, supplementary material included.

Key takeaways

• The loss (§4, Eq. 14; pseudocode in Supp. S9). Draw one timestep and one noise and use them for both images of a pair. For each image, take the model’s denoising error minus the frozen reference’s. The loss is −log σ(−β[(winner’s difference) − (loser’s difference)]). It falls when the model gets better than the reference at denoising the winner, or worse at the loser (Fig. 2). It costs four forward passes per pair and needs no sampling.
• How it gets there, and the approximations it makes (§4, Supp. S2). DPO needs log p_θ(x_0 | c), which a diffusion model does not give. The paper instead puts the reward and the KL on the whole chain x_0:T. The joint KL is an upper bound on the marginal one (Eq. 10). Jensen’s inequality moves the expectation over the chain outside the log-sigmoid (Eq. 12). Then the chain posterior p_θ(x_1:T | x_0), which cannot be sampled, is replaced by the forward process q(x_1:T | x_0) (Eq. 13). The constant T is folded into β, which is why β is in the thousands. Supp. S3 rederives the loss as off-policy RL in the denoising MDP of DDPO and DPOK, and S4 from a noise-aware preference model.
• The headline (§5.2, Figs. 3–4). SDXL-1.0 base, trained on Pick-a-Pic v2 (851,293 pairs over 58,960 prompts after dropping about 12% ties), evaluated by five Mechanical Turk labelers per comparison, majority vote:
  • Against SDXL-base on PartiPrompts (1,632 prompts), it wins 70.0% on general preference, 64.3% on visual appeal and 64.9% on prompt alignment. On HPSv2 prompts (3,200) the figures are 64.7%, 61.9% and 64.9%.
  • Against SDXL base+refiner (6.6B parameters against its 3.5B) it wins 69% and 64% on general preference. On PartiPrompts’ People category that is 67.2%, against 73.4% over the base alone.
  • Its HPSv2-benchmark images average 28.16 under the HPSv2 reward model, “topping the leaderboard”.
  • In SDEdit image editing on TEdBench, it is preferred 65% to 24%, with 11% draws (§5.3).
• Settings (§5.1). The effective batch is 2,048 pairs on 16 A100s, with AdamW for SD1.5 and Adafactor for SDXL. The learning rate is (2000/β)·2.048·10⁻⁸ with 25% linear warmup, scaled inversely with β because the DPO gradient’s norm grows with β. That is about 8.2·10⁻⁹ at SDXL’s β = 5000. β in [2000, 5000] “offer[s] good performance”. SD1.5 uses 2000. The β sweep (Supp. S5, Fig. S2) is four bars of median PickScore on SD1.5.
• Fine-tuning on the winners hurts a model better than its data (§5.5, Supp. S6). SFT on the preferred images helps SD1.5 (55.5% win rate over base). For SDXL “any amount of SFT deteriorates the performance”, even at lower learning rates, because Pick-a-Pic was generated by SDXL-beta and Dreamlike and SDXL-1.0 is better than both. DPO still improves SDXL on those pairs (Fig. 7). The contrastive signal works where imitating the winner does not.
• AI feedback can replace the labels (§5.4, Fig. 6). On SD1.5 (β = 5000), relabelling the pairs with HPSv2, CLIP, an aesthetic predictor or PickScore moves that model’s own metric, sometimes at another’s expense. Aesthetics-ranked training lowers CLIP. Relabelling Pick-a-Pic with PickScore beats the human labels it was trained on. It raises DPO-SD1.5’s PartiPrompts general-preference win rate from 59.8% (human labels) to 63.3%. The paper calls this a form of data cleaning.
• The model is a reward model too (§5.5, Table 2). The implicit reward ranks Pick-a-Pic v2 validation pairs with 72.0% accuracy for DPO-SDXL and 60.8% for DPO-SD1.5. HPSv2 gets 59.3, CLIP 57.1 and aesthetics 51.4. PickScore’s 64.2 is marked as having seen the evaluated data.

Where the hedges are

Per DP-010:

• The derivation’s last step is an assumption, not a bound. Eqs. 10 and 12 are upper bounds on the loss. Replacing p_θ(x_1:T | x_0) with q(x_1:T | x_0) is an approximation with no stated error. Supp. S2 offers an alternative approximation (L2), argues that its error is lower, and does not test it. The S3 argument that this is “off-policy” RL justifies sampling from q. It does not measure the gap.
• The pairs are not from the reference model. §4 assumes pairs “generated from a reference model p_ref”. Pick-a-Pic was generated by SDXL-beta and Dreamlike (Pick-a-Pic’s own paper, LIT-782, lists SD 2.1 among the generators of its first release), and SDXL-1.0 is neither. The method works anyway (Fig. 7), but the KL-regularized optimum it derives is not obviously the one it trains toward. Dreamlike trained on its own 15% of the data improves only a little, which the authors put down to dataset size.
• Checkpoint selection uses a model trained on the same data. Every run is validated by median PickScore on 500 Pick-a-Pic validation prompts (§5.1). PickScore (LIT-782) was trained on Pick-a-Pic v1. Final testing is human and on other prompt sets, so the headline is not circular, but the β sweep and the selection are.
• “Visual appeal” is not what the aesthetic model sees. Against SDXL on Pick-a-Pic validation prompts (Supp. S7, Fig. S3), DPO-SDXL wins clearly under PickScore, HPS and CLIP. Under the LAION aesthetic predictor the bar sits at the 0.5 line. The text does not discuss it.
• One run per setting, no intervals. The human evaluations are single comparisons with five votes each. The paper does not say how ties between two images were handled outside TEdBench. The SDXL training length is not given. No human evaluation of the main SD1.5 model (β = 2000) is reported. The SD1.5 win rates above come from the AI-feedback and SFT experiments.
• “For the first time” in an open vocabulary (§6) rests partly on dismissing DDPO and DPOK, which it reports as unstable on Pick-a-Pic without numbers (§5.1, Supp. S1). Supp. S1’s explanation of why DDPO fails describes θ_old as the original model, where in DDPO it is the previous iteration’s. It says high-reward prompts dominate the gradient, and DDPO’s per-prompt reward normalization exists to prevent exactly that.
• The model was not released at v1. The ethics paragraph withholds it “until we add additional safety filtering”.

Which comparisons are like for like

• SDXL-base (Fig. 3) is the same architecture and parameter count, the paper’s own “parameter-equal comparison”. It is the fair one.
• SDXL base+refiner (Fig. 4) is a stronger pipeline with 1.9× the parameters, which favours the baseline.
• SFT vs DPO (§5.5) uses the same data and schedule, but SFT’s learning rate (10⁻⁹) differs, and only a win rate over base is given.
• AI-feedback rows (Fig. 6) are win rates under reward models. On the diagonal, the model that ranked the training pairs also scores the result. Only the PickScore relabelling has a human evaluation, the rise from 59.8% to 63.3%.
• Rejection sampling (Supp. S8, Fig. S4) scores the reference model’s best of n draws by PickScore against one DPO sample, also by PickScore. The mean number of draws needed is 13.7 for SDXL and 25.6 for SD1.5, with medians 3 and 7. For 7% and 16% of prompts, 100 draws were not enough.
• DDPO was tried and not reported numerically.

Standing in the anthology

It extends DPO (LIT-169). The Bradley–Terry likelihood, the closed-form optimum of the KL-regularized objective and the β-weighted log-ratio loss (Eqs. 3–8) are DPO’s. What it adds is a way to evaluate that log-ratio for a model with no tractable likelihood. Its SFT result runs against the language-model recipe that does SFT before DPO, SOTA-129’s sequence. The paper’s own reading (Supp. S6) is that the gap is the domain: SDXL is already better than its preference data. That result is filed as SOTA-445: when the base is better than the models that made the pairs, train on the pairs with this loss rather than on the winners. Its SDXL arm has no number, and nobody has repeated it. SOTA-126’s one-epoch DPO practice is about tiny language models and is not tested here.

Its main comparison is SDXL (LIT-566), its base model, and the refiner pipeline from the same paper. The SD1.5 experiments start from Stable Diffusion in the latent diffusion line (LIT-062).

DDPO (LIT-776) is the online, reward-model route it positions against. The paper reports trying public DDPO implementations on Pick-a-Pic and seeing no stable PickScore gain over a hyperparameter sweep. It gives no numbers, and its account of why does not hold up, as above. The two papers use the same denoising MDP (Supp. S3, Eq. 19), on- and off-policy.

Flow-GRPO (LIT-779) compares against it in flow-matching form. Its offline and online Flow-DPO baselines are this loss carried to flow matching by Flow-DPO (LIT-773), which drops the (1 − t)² weight the exact substitution produces, on pairs made from the best and worst image of each group. They improve with training and stay below Flow-GRPO’s online policy gradient on GenEval (its Fig. 4 and App. C.1). Flow-GRPO also finds online DPO ahead of offline, which this paper anticipates as future work (§6).

Flow-DPO (LIT-773) rewrites this loss for rectified flow. Carried over exactly it weights the KL by β(1 − t)², which vanishes at the noise end, and that variant raised its video reward’s quality scores while text alignment fell below the base model; a constant β fixed it.

Step-Video-T2V (LIT-624) builds on it: its Video-DPO objective is “based on the DiffusionDPO method”, extended to flow matching (its §4.4), and on a 30B video model it gets a modest gain. NOTE-338 records its advice not to inherit β = 5000 and the learning rate near 10⁻⁸ for a large video model. The pseudocode here (Supp. S9) already shares the noise and timestep across each pair, which that note also recommends.

Filed without a NOTE. The takeaways come from one full reading of v1, supplementary material included, done for this filing. Figs. 3, 6–7 and S2–S4 were checked against the rendered pages. Figure values are quoted only where the text or a bar label gives them.