anthology-of-the-sota

SOTA-280: Sample several reasoning paths and take the majority answer rather than decoding one greedily

3 min read

Properties6
StatusActive
Consensusemerging
Tagsin-context-learning, inference-optimization
SourceLIT-468 — Self-Consistency Improves Chain of Thought Reasoning in Language Models
Introduced byLIT-468 — Self-Consistency Improves Chain of Thought Reasoning in Language Models
ExtendsSOTA-279 — Put worked reasoning steps in the few-shot exemplars when the task needs more than one step, and only once the model is large enough

Graph View

Source

Wang et al. (2022), LIT-468 — ARXIV-2203.11171, read as NOTE-217.

What to do

Keep the chain-of-thought prompt exactly as it is. Change the decoding: sample several reasoning paths instead of taking the greedy one, extract each path’s final answer, and return the most frequent.

That procedure is self-consistency, which LIT-468 introduced as a drop-in replacement for greedy decoding under a chain-of-thought prompt. Across four model families (UL2, GPT-3/Codex, LaMDA-137B, PaLM-540B) it raised GSM8K by 17.9 points, SVAMP by 11.0, AQuA by 12.2, StrategyQA by 6.4 and ARC-challenge by 3.9 over greedy chain-of-thought decoding.

Start at five or ten paths. That is the authors’ own guidance and it is the practical content — performance saturates quickly, so the marginal path stops paying well before the budget does.

Why it works, and what that implies

A hard problem admits several routes to one correct answer, while wrong answers tend to be reached idiosyncratically. Agreement between independent derivations is therefore evidence, and a single greedy path offers no such check — it is exposed both to local optimality and to the luck of one sample. It is ensembling without an ensemble.

The implication worth carrying: it counts answers, not reasoning. The by-products the authors report — calibration and an uncertainty estimate — come from the answer distribution, not from the traces being any better.

Conditions

  • The task needs a determinate, extractable final answer to compare across samples. Arithmetic and multiple choice qualify; open-ended generation does not, and the paper does not claim it.
  • Cost is linear in paths and is the one limitation the authors name. Five or ten is a rule of thumb without a scaling story — whether the saturation point moves with task difficulty is not measured.
  • Sampling must be diverse enough to produce genuinely different paths, or this degenerates to greedy decoding at N times the price.
  • It does not make the traces trustworthy. The authors note models sometimes generate nonsensical reasoning paths, and LIT-467’s error analysis found correct answers reached through incorrect chains. An evaluation that reads a trace as an explanation is not rescued by this — see SOTA-278.
  • Two of the four models are not public, so part of the evidence is not directly reproducible; the authors supply prompts and two public Codex engines.

Relation to the neighbours

Extends SOTA-279 by changing how its output is read rather than what it is asked. Being a sampling algorithm, it also files under inference-optimization, whose blurb names them.

It composes with SOTA-281 in principle — nothing requires the sampled chains to come from exemplars rather than a trigger sentence — but that pairing is not what either paper measured, and the record should not imply it was.

Backlinks