Source
Wallace, Dang, Rafailov, Zhou, Lou, Purushwalkam, Ermon, Xiong, Joty and Naik (2023), LIT-774 — Diffusion-DPO, §5.2, §5.5, Fig. 3, Fig. 7 and Supp. S6.
What to do
You have a preference dataset of image pairs, preferred and rejected, for the same prompt, and a base model to improve. If the images were generated by models weaker than your base:
- Do not fine-tune on the preferred images alone. Imitating them pulls the model toward images worse than it already makes.
- Train on the pairs with the Diffusion-DPO loss, against a frozen copy of the base as the reference. The signal is which of the two images the model should get better at denoising relative to the reference, and that can improve a model even when both images are below its level.
When the data are better than the base, the source found SFT helps too, so the choice is not forced there.
Evidence
SFT on the winners hurts the stronger model (LIT-774, §5.5). Fine-tuned on the preferred images of Pick-a-Pic, for the same schedule as DPO at learning rate 10⁻⁹, SD1.5 improves, with a 55.5% win rate over its base. For SDXL, “any amount of SFT deteriorates the performance … even at lower learning rates”. The authors attribute the contrast to the data: Pick-a-Pic was generated by SDXL-beta and Dreamlike, which are better than SD1.5 and worse than SDXL-1.0. No number is given for the SDXL arm.
DPO on the same pairs helps it (§5.2, Fig. 3). DPO-SDXL, trained on Pick-a-Pic v2’s 851,293 pairs, is preferred to the untuned SDXL base on PartiPrompts 70.0% of the time on general preference, by majority vote of five crowdworkers per comparison. Fig. 7 shows median PickScore rising for SDXL trained on pairs it did not generate.
The paper’s reading (Supp. S6) is that this matches the summarization setting of DPO (LIT-169), where a model already tuned for the task does not gain from fine-tuning on preferred outputs.
Conditions
- The SDXL SFT arm is a sentence, not a measurement. The comparison that motivates the practice has one number on one side. The SD1.5 SFT win rate is reported without saying how it was evaluated.
- The learning rates differ. SFT ran at 10⁻⁹, DPO at its β-scaled rate (about 8·10⁻⁹ at β = 5000). The text mentions lower rates for SFT, not higher ones.
- One dataset, one strong base. Every SDXL result is on Pick-a-Pic. The claim’s condition, base better than data, is inferred from which models generated the data and is not measured.
- DPO’s own derivation assumes the pairs come from the reference. They do not here, and the method works anyway; the paper does not say what objective it then optimizes. The note’s hedges set this out.
- It is offline. Online methods that sample from the current model, where the data are the model’s own, face a different question; SOTA-449 is one of their practices. Flow-GRPO (LIT-779) finds online DPO ahead of offline in its flow-matching comparison, which this practice does not address.
- The language-model recipe runs the other way. There SFT usually comes before preference optimization. The source’s reading is that the difference is the domain gap: a pretrained text-to-image model is already doing the task, so there is nothing for an SFT stage to teach.
Known implementations
- DPO-SDXL, the source’s model. DPO-SD1.5 is trained the same way, but its base is weaker than the data, so it is not a case of this practice.