Rafailov et al. (2023) — ARXIV-2305.18290
Key takeaways
- The problem is the machinery, not the objective. RLHF fits a reward model to human preference labels and then optimises the policy against that estimate with RL while a KL term keeps it near the original — a procedure the paper describes as complex and often unstable, with two models, a sampling loop and a sensitive set of hyperparameters.
- The move: a new parameterisation of the reward model under which the corresponding optimal policy can be written in closed form. The reward never has to be materialised, because the policy is the reward model, up to a reparameterisation — which is what the title means.
- What is left is a simple classification loss over preference pairs. No sampling from the language model during fine-tuning, no separate reward network, and little hyperparameter tuning.
- Empirically it matches or beats PPO-based RLHF: better at controlling sentiment, matching or improving response quality on summarisation and single-turn dialogue, at a fraction of the implementation cost.
Standing in the anthology
The source SOTA-126 has been missing. That practice — run DPO on tiny models for one epoch only — recommends an algorithm the record described but never cited, which is exactly the shape DP-001 exists to prevent, and it survived because the practice’s own finding (one epoch, not more) is about DPO’s behaviour at 90M rather than about DPO itself.
Read the two together and the division is clear: this paper is why the stage exists and why it is cheap; LIT-119 and LIT-129 are the evidence that it still converges at 90M–135M, and LIT-119’s observation that a second epoch degrades the model while the DPO reward keeps rising is a caution this paper’s framing predicts — optimising the implicit reward is not the same as improving the policy.
It is also the middle of the post-training sequence the record now holds: SFT → DPO → RLVR (LIT-172), which is the recipe SOTA-129 records and SOTA-123 collapses at tiny scale.
Outside language, Diffusion-DPO (LIT-774) carries this loss to diffusion models, evaluating the likelihood ratio through denoising errors, and Flow-GRPO (LIT-779) runs it in that form, offline and online, as a baseline for RL on a flow-matching image model. With the same SD3.5-M model, reward and group of samples per step, both DPO variants improve GenEval with training and stay below Flow-GRPO’s online policy gradient (its Fig. 4). Only online DPO’s β is reported as searched.