Ouyang et al. (2022) — ARXIV-2203.02155
Key takeaways
- The three steps, which became the template. (1) Collect labeler demonstrations on a prompt distribution and fine-tune supervised. (2) Collect rankings of model outputs and train a reward model to predict them. (3) Fine-tune the supervised model against that reward with PPO.
- Scale is not the axis. Outputs from the 1.3B InstructGPT are preferred to those from 175B GPT-3 — a hundredfold parameter difference losing to post-training. At matched size the 175B model wins 85 ± 3% of comparisons against 175B GPT-3, and 71 ± 4% against few-shot 175B GPT-3.
- It beats instruction-tuning-on-public-tasks on real prompts, which is the comparison that decided the field’s direction: 73.4 ± 2% win rate against the SFT baseline, versus 26.8 ± 2% for T0 and 29.8 ± 2% for FLAN.
- The alignment tax is named here. RLHF fine-tuning regresses performance on SQuAD, DROP, HellaSwag and WMT’15 French-English. The paper’s own fix is PPO-ptx: mix the PPO updates with updates that raise log likelihood on the pretraining distribution, which “greatly reduce[s]” the regressions “without compromising labeler preference scores”.
- A per-token KL penalty from the SFT model is what keeps the policy from over-optimising the reward model. The KL coefficient and the pretraining-mix coefficient are the two knobs the objective exposes. Measured later in this paper’s own setting, the penalty behaved like early stopping rather than protection; see Standing.
- One 6B reward model drives every policy size. Larger 175B reward models could reach lower validation loss and were not used, which makes the size comparisons across policies clean and leaves the reward-model scaling question open.
- 40 screened contractors produced the data, and the paper is explicit about what that means: the procedure aligns the model “to the stated preferences of a specific group of people (mostly our labelers and researchers), rather than any broader notion of ‘human values’”.
Standing in the anthology
The origin of the post-training shape this record documents downstream and had never filed. LIT-169 (DPO) is defined against this pipeline — its title is a claim about the reward model this paper introduced. LIT-082 (Constitutional AI) replaces this paper’s human labels with model-written ones, and SOTA-183 is that practice. SOTA-126 tunes DPO’s schedule. Every one of them assumes the thing described here.
The reading is NOTE-161. Filed under #87, and it is the third document in two sessions to be filed because its descendants were already here — after SOTA-227 for speculative decoding. DP-007’s “refinement filed before the thing it refines”, three times over, is a pattern about how this record acquires documents rather than about any of the three subjects.
It is not the origin of RLHF, and the record should not read it as one.
The paper says its methodology follows Christiano et al. (2017), who
introduced learning from human preferences, and Stiennon et al. (2020), who
applied it to summarization. Both are now filed — LIT-434 and
LIT-433 — and the extends: field carries the line, so the reading
list walks preference learning → summarization → general instruction
following rather than starting here. LIT-432 is FLAN, the arm this
paper measured itself against and beat.
Three text-to-image papers carry it to diffusion models, each taking a different part. Lee et al. (LIT-797) take the reward model and the pre-training-loss term and replace PPO with one round of reward-weighted likelihood. ImageReward (LIT-788) takes the ranked-comparison reward model and the labelling guide, whose second trade-off rule still says “truthfulness and harmlessness”. DPOK (LIT-778) takes the per-token KL penalty and applies it per denoising step.
Gao et al. (LIT-795) use this paper’s environment and its 6B reward model as ground truth to measure over-optimization: true reward peaks and falls with KL from the SFT policy, and in their PPO runs the KL penalty did not raise true reward at a given KL; it acted like early stopping, a result they flag as possibly hyperparameter-sensitive. They also answer part of the reward-model scaling question this paper left open: a larger reward model, or more comparison data, moves the peak later.
Version 1 named the three as open, and they closed three days later — not because anything tracked them, but because a pass went looking for exactly this shape of sentence. #86’s trunk detector reads Standing sections for claims that several papers agree on something the record cannot cite, and this paragraph is what it found. A request written in prose carries no code and resolves to nothing, so nothing was ever going to raise it on its own — which is DP-004 about the record’s own prose rather than about a search.