Source
Wei et al. (2022), LIT-467 — ARXIV-2201.11903, read as NOTE-216.
What to do
Write the few-shot exemplars as ⟨input, worked steps, output⟩ rather than
⟨input, output⟩. Eight is enough. Nothing else changes — no fine-tuning, no
training data, no task-specific checkpoint.
This is chain-of-thought prompting as LIT-467 introduced it, and the table below is its evidence: with eight such exemplars PaLM 540B reached the state of the art on GSM8K, beating a fine-tuned GPT-3 with a verifier, and the largest GPT and PaLM models more than doubled their GSM8K score — across LaMDA, GPT-3 and PaLM, and on commonsense and symbolic tasks as well as arithmetic.
When it pays, and when it does not
| condition | effect |
|---|---|
| multi-step problem, model ≥ ~100B | large; GSM8K more than doubled |
| single-step problem | negative or negligible |
| model below ~100B | absent, and often worse than plain prompting |
The second and third rows are the part that gets dropped when this is repeated. Small models produce chains the paper describes as “fluent but illogical” and score below standard prompting, and on SingleOp — the one-step subset of MAWPS — the gain is negative or very small. This is a recommendation with a regime, not a universal improvement.
Why the ablations matter more than the headline
Three rival explanations were built and knocked down, which is why this is the trunk rather than one result among many:
- Not the equation. Prompting for the equation alone does not help on GSM8K.
- Not the extra tokens. Emitting a row of dots as long as the needed equation performs at baseline. Buying the model more positions to compute in, with nothing in them, buys nothing.
- Not knowledge activation. Putting the chain after the answer performs at baseline, so the answer depends on the chain rather than being primed by it.
Conditions
- The evidence is 2022 models — LaMDA, GPT-3, PaLM. The ~100B threshold is a fact about that generation, and small-model reasoning has moved a long way since. Treat the threshold as a caution to measure rather than a number to apply.
- Chains are tokens. The paper reports negative gains on the easiest problems and does not price the ones it wins on. There is a regime where this loses on accuracy and cost at once.
- Search over a large in-context fact base is one such regime, measured on 2024 models. LIT-667 gives frontier models 28.2K facts (or 5.4K retrieved, enough to deduce the answer) and asks a question whose proof needs two bridge entities found among them. Against 33.3% for chance, Gemini-1.5-Pro scores 28.7% answering directly and 11.3% when asked to verbalize; with retrieval, 37.3% against 12.0%. GPT-4-Turbo is at chance either way. The mechanism is visible rather than mysterious: 70.7% of the verbalized responses conclude the answer cannot be decided (58.7% with retrieval) after searching, when it can. What the paper does not report is how the verbalization was elicited — “prompted to verbalize the reasoning”, with no prompt given — so this bounds the regime rather than singling out exemplars, and it lands on SOTA-281 and SOTA-280 equally.
- A correct chain is not guaranteed and a correct answer does not imply one. Their own error analysis found two of fifty correct answers reached through incorrect reasoning. Reading a chain as an explanation of the answer is a separate claim this does not support. LIT-667 makes that much worse on a task where the proofs are mechanically checkable: most of the rationales that reached the right answer were themselves wrong — hallucinated facts or logical errors. That figure is from inspection and carries no count, so take it as a reason to score the rationale rather than as a rate.
- Whether the model is “reasoning” is explicitly left open by the authors, and the practice takes no position either.
- The emergence claim rests on a brittle metric. Exact-match over multi-step answers is the shape SOTA-200 says to distrust. The gains at 540B survive three ablations and are not in doubt; the discontinuity has not been re-measured against a continuous score.
Why Active and universal at once
Active because the effect is large, ablated against its three obvious
alternatives, and robust across annotators, exemplar sets, orders and three
model families. universal because the field stopped arguing about this
years ago — models are now post-trained to produce traces unprompted, and the
live disputes are about how to evaluate traces
(LIT-466, LIT-463) rather than whether to
elicit them. DP-005 is why those are two separate fields: adoption
this complete is not itself evidence, and the evidence column here still
holds one paper about a generation of models nobody serves any more.
What this practice does not cover
The record holds nothing yet on the immediate descendants, and each is a separate recommendation:
- Zero-shot chain of thought — appending “Let’s think step by step” with
no exemplars at all (
ARXIV-2205.11916). - Self-consistency — sampling several chains and taking the majority
answer (
ARXIV-2203.11171). - Whether to elicit chains at all on an easy query, which LIT-466 bears on directly: at low complexity its non-thinking models beat their own thinking counterparts and used fewer tokens, which is this paper’s SingleOp finding arriving again three years later by a different route.
Relation to the neighbours
Extends SOTA-038, in-context few-shot adaptability, which is the mechanism it modifies. SOTA-127 — filter chain-of-thought traces out of a tiny specialist’s training data — is the other side of the scale threshold, and now has the paper explaining why small models’ chains are bad sitting beside it. SOTA-130 and the reasoning-model literature assume this and had no antecedent in the record for what they were extending.