Clark et al. (2018) — ARXIV-1803.05457
Key takeaways
- Multiple-choice grade-school science questions, partitioned into ARC-Easy and ARC-Challenge, where the Challenge split is defined by what fails on it: questions both a retrieval baseline and a word-co-occurrence baseline answer incorrectly
- Defining a split by baseline failure is the design decision worth noting — it makes the Challenge set adversarial to two specific strategies, which is narrower and more honest than “hard questions”
- A standard member of the zero-shot evaluation suite, which is how it appears here: alongside HellaSwag, PIQA, WinoGrande and LAMBADA rather than on its own
Standing in the anthology
Carries no practice, deliberately — ADR-032.
Filed to separate two benchmarks this record was calling by one name. Ten
documents here write ARC-Easy or ARC-Challenge and mean this paper —
THEORY-007 measures across ARC-C, LIT-184 names it beside MMLU as a
target FineWeb-Edu moves. Twelve others write a bare ARC, and at least two
of those — SOTA-235 and LIT-379 — mean LIT-405, Chollet’s puzzle
benchmark, which shares nothing with this but four letters.
Nothing in the record distinguished them until now, which matters because a result is only as legible as what it was measured on: “6× a fine-tuned baseline on ARC” and “ARC-C rises with model scale” are claims about different objects.
Unread — no NOTE.