Chollet (2019) — ARXIV-1911.01547
Key takeaways
- The complaint: benchmarks measure skill at a task, and skill is purchasable with data and priors. A system that has seen enough of a task distribution scores well without having acquired anything, so skill-based measurement cannot distinguish breadth of ability from breadth of exposure
- The proposal: measure skill-acquisition efficiency — how much a system learns from how little — against explicitly stated priors and experience
- The instrument: ARC (later widely called ARC-AGI), a set of visual grid puzzles where each task is intended to be structurally novel, solvable from a handful of demonstrations, and resistant to being learned in advance
- What that buys is the property the record cares about here: a benchmark built to defeat memorization behaves differently under methods that search or adapt at test time than one built to sample a task distribution
Standing in the anthology
Carries no practice, deliberately — ADR-032. Filed for what it
establishes about measurement, and for a disambiguation the record needs.
This is the “ARC” that SOTA-235 reports on, and it is not the only ARC in
this record. SOTA-235 (adapt at test time on the evaluation instance
itself) carries a headline of 53.0%, up to 6× fine-tuned baselines, and
61.9% ensembled — figures on this benchmark, where the whole point is that
the tasks are novel, which is why a test-time-adaptation result there is
close to definitional rather than incidental.
Elsewhere in the record, “ARC” means LIT-407 — the AI2 Reasoning
Challenge, a multiple-choice science-question benchmark with no relation to
this one beyond the acronym. THEORY-007 and LIT-184 use it in that sense.
The two are now separable; SOTA-235 and LIT-379 say ARC-AGI rather
than ARC as of this contribution.
Unread — no NOTE.