Stiennon et al. (2020) — ARXIV-2009.01325
Key takeaways
- The loop, on text. Collect human comparisons between candidate summaries, fit a reward model to predict the preferred one, fine-tune the policy against it with reinforcement learning. The method is Ziegler et al.’s (LIT-734), adapted to batch collection
- The models beat the human reference summaries, on a version of the
TL;DR Reddit dataset — and beat much larger models fine-tuned with
supervised learning alone, which is the size-versus-post-training comparison
LIT-377later ran on general instructions - It transfers without task-specific tuning. The TL;DR-trained models produce CNN/DM news summaries nearly as good as the human reference with no news-specific fine-tuning, so what the reward model learned is not the dataset
- Optimizing the reward model beats optimizing ROUGE, judged by humans. This is the methodological half and the paper leads with it: the training loss and the metric are both proxies, and the paper’s closing ask is that researchers “pay closer attention to how their training loss affects the model behavior they actually want”
- The reward model generalises to datasets it was not fit on, stated as a measured result rather than assumed — which is the property PPO is spending when it optimizes against a fixed reward model
Standing in the anthology
The bridge, and the reason the lineage is a lineage rather than a
resemblance. LIT-377 names this paper as the second half of its
methodological inheritance — Christiano introduced learning from human
preferences, and this “applied it to summarization”. The author lists make
the continuity literal rather than reconstructed: Ouyang, Christiano, Lowe
and Amodei appear on both this paper and InstructGPT.
The lineage has three steps, not two. The loop is LIT-434’s — human comparisons, a reward model fit to them, reinforcement learning against the fit — but this paper does not take it from there. It cites LIT-434 only among work using human feedback to train agents in simulated environments, and says its approach “is similar to the one outlined in” Ziegler et al. (LIT-734), who had already fine-tuned GPT-2 with PPO against a reward model fit to human preferences, including on TL;DR and CNN/DM summarization. What this paper changes is what Ziegler et al. reported going wrong: batch rather than online collection of comparisons, labelers held to high agreement with the researchers, separate policy and value networks, and much larger models. Ziegler’s labelers already preferred its summaries to the references, but those summaries were near-extractive copies, and the labelers’ judgement had drifted from the researchers’; this paper’s contribution is making that preference trustworthy.
The evaluation claim is the half the record was weakest on. Its second tag is not decoration: the argument that a widely-used automatic metric disagrees with the thing it stands in for is a claim about measurement, made by running the comparison, and it predates by years the record’s own benchmark-scepticism documents.
Unread — no NOTE.