Kojima et al. (2022) — ARXIV-2205.11916
Key takeaways
- One prompt, twelve datasets, no exemplars. The same trigger sentence is used across arithmetic, commonsense, symbolic and logical reasoning. The method is two-stage: extract a reasoning trace, then extract an answer from it. Nothing is task-specific, which is the whole claim.
- The numbers are large and the baseline is what makes them large. MultiArith 17.7 → 78.7, GSM8K 10.4 → 40.7 on text-davinci-002. Those are against zero-shot prompting, which is the right comparison for the claim and not the same as beating the state of the art.
- It underperforms few-shot chain of thought and beats eight-shot standard prompting. The paper states this plainly. That ordering is the practical content: it is the cheap baseline, not the better method, and the hand-written chains of LIT-467 are still worth their labour when the task justifies it.
- Same scale threshold as its parent. Flat or negative below roughly 100B; the gain arrives with size, across Original and Instruct GPT-3 and PaLM. 17 models from 0.3B to 540B.
- The honest prior-art note. Reynolds and McDonell had proposed a similar trigger, and the paper says so — the difference claimed is that theirs was task-specific and never evaluated quantitatively against baselines.
Standing in the anthology
The second of the three documents SOTA-279 named as uncovered when the
chain-of-thought trunk was filed, and the one that most changes what a
practitioner does first. Declared as extends: LIT-467, which it modifies by
removing the exemplars.