Christiano et al. (2017) — ARXIV-1706.03741
Key takeaways
- Ask which of two trajectory segments is better, not what the reward is. A non-expert compares pairs; a reward model is fit to those comparisons; the policy is optimized against the fitted reward. The three-part loop is the contribution, and it is what every later preference method rearranges
- Complex RL tasks solved with no access to the reward function at all — Atari games and simulated robot locomotion, which is the evidence that the substitution is not merely expressible but sufficient
- Under 1% of the agent’s interactions need feedback. That ratio is the paper’s real argument: human oversight stops being the thing that bounds how large a system you can supervise
- Novel behaviours from about an hour of human time, which is a claim about specifying goals rather than about sample efficiency — the behaviours are ones no reward function had been written for
Standing in the anthology
The origin of the reward-model loop, filed because five documents here
already depend on it. LIT-377 says so directly: InstructGPT’s methodology
“follows Christiano et al. (2017), who introduced learning from human
preferences”. LIT-169 (DPO) is a claim about what that reward model
secretly is, LIT-082 replaces its human labels with model-written ones,
SOTA-183 is that replacement as a practice, and SOTA-126 tunes the
schedule of the method that removed the explicit reward model. None of them
could name where the construction came from.
DP-007 names the reason and LIT-377 supplied the trace: the note that
filed InstructGPT wrote down that this paper was missing, and the record kept
working without it. Agreement has no author.
It is not a language paper, and the record should not read it as one.
Atari and MuJoCo, an RL method with no language model in it. What crossed
into language was the loop, not the setting — LIT-433 is the step that
carried it, and reading this one as “early RLHF for LLMs” would collapse a
transfer into a continuation.
Unread — no NOTE.