anthology-of-the-sota

LIT-077: Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models

2 min read

Properties7
StatusActive
Version2
Tagsanalysis-and-evaluation
Extended byLIT-471 — Are Emergent Abilities of Large Language Models a Mirage?
Cited as “introduced by” bySOTA-200 — Check whether an emergent capability is a metric artefact before believing it
Cited as “paper” byNOTE-051 — Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Cited as “source” bySOTA-197 — Account for test-set proximity to the training data when evaluating, and state your contamination exposure, SOTA-200 — Check whether an emergent capability is a metric artefact before believing it

Graph View

Srivastava et al. (2022) — ARXIV-2206.04615

Key takeaways

  • 204 tasks, 450 authors, 132 institutions, spanning linguistics, childhood development, mathematics, common-sense reasoning, biology, physics, social bias and software development — built because existing benchmarks were saturating. Evaluated on dense and Switch-style sparse transformers from millions to hundreds of billions of parameters, with expert human raters performing every task as the baseline
  • The finding worth carrying is about measurement: tasks improving gradually “commonly involve a large knowledge or memorization component, whereas tasks that exhibit ‘breakthrough’ behavior at a critical scale often involve multiple steps or components, or brittle metrics”. A multi-step task scored all-or-nothing improves invisibly until every step works — a property of the metric, not the model. Formalised as a linearity vs breakthroughness axis
  • Performance is remarkably similar across model classes, with benefits from sparsity; accuracy and calibration both improve with scale and are poor in absolute terms and against the human baseline
  • Social bias typically increases with scale under ambiguous context — and few-shot prompting for neutrality significantly reduces it on bbq_lite
  • Contamination is addressed rather than assumed away: training data predates the repository for every model except PaLM, indirect leakage is acknowledged, and a training_on_test_set task ships for future models to probe

Standing in the anthology

Read — NOTE-051. Retagged from model-architecture — a benchmark paper is not an architecture paper — to analysis-and-evaluation. Nothing is sourced here.

The breakthroughness finding is the one the record could use. The corpus carries several claims of the form “capability X appears at scale Y”, and this categorisation says a large share of such observations are metric artefacts. Read with LIT-085’s data-fraction result and LIT-073’s DrawBench, this pass has now produced three independent arguments that the measurement, not the model, produced the surprise — a stronger case for a practice than any one of them alone.

Its takeaways described the artefact and none of its findings.

Backlinks