Wei et al. (2022) — ARXIV-2201.11903
Key takeaways
- The method is eight exemplars. Show the model a handful of input–rationale–output triples instead of input–output pairs. No fine-tuning, no training data, no task-specific checkpoint: the paper says plainly that no language model was fine-tuned in the writing of it.
- The ablations are why this is the trunk and not one result among many. Three rival explanations were built and tested. Equation only — emit just the equation — does not help on GSM8K. Variable compute only — emit a row of dots as long as the equation would be — performs at baseline, so it is not merely buying the model more forward passes. Chain of thought after the answer also performs at baseline, so the model is depending on the chain causally rather than being reminded of pretraining knowledge.
- It is absent below roughly 100B parameters, and can be worse than not doing it. Smaller models produce “fluent but illogical” chains and score below standard prompting. The gain is also largest on the hardest datasets and negative or negligible on single-step problems.
- Robust in the ways prompting usually is not. Three annotators wrote independent chains; all beat standard prompting by a large margin. Exemplars sampled from the GSM8K training set did as well as the hand-written ones. Order and count of exemplars do not overturn it.
- The framing line. “Standard prompting only provides a lower bound on the capabilities of large language models.” That sentence is the whole of SOTA-278, written in 2022 by people who had just demonstrated it.
Standing in the anthology
The trunk of a topic the record had somehow never held. Until this was filed, the only chain-of-thought document in the corpus was SOTA-127, which says to filter such traces out of a tiny model’s training data — a recommendation that reads very differently once the paper explaining why small models produce bad chains is present.
It also stands underneath a great deal the record already carries.
SOTA-130 skips the reasoning SFT stage and runs RL with verifiable
rewards; LIT-466 evaluates models whose defining feature is
an extended thinking trace. Both assume the thing this paper established.
Declared as extends: LIT-035, the GPT-3 paper, because few-shot prompting
is what it modifies.
The baseline in every experiment is standard few-shot prompting, “popularized by Brown et al. (2020)”: LIT-035’s in-context exemplars of input–output pairs. The method changes only the exemplars, by adding a chain of thought to each. GPT-3 is also one of the three model families evaluated, through the text-ada-001 to text-davinci-002 endpoints (350M to 175B, which the paper takes to be InstructGPT models), alongside LaMDA and PaLM.