Gandhi et al. (2024) — ARXIV-2404.03683
Key takeaways
- The method is to represent a search process as a flattened string — a stream of search — and pretrain on traces generated by heuristic solvers, so a model sees fruitful mistakes rather than only correct next tokens. Reported as +25% search accuracy over models trained to predict only the solution
- The task is Countdown: combine a set of input numbers with arithmetic operations to reach a target. The paper presents it as an existing game rather than a benchmark it introduces — the numbers round of a British television programme — and it is the demonstration vehicle, not the contribution
- What makes Countdown useful to the literature that borrowed it is mechanical: a solution is verifiable by evaluating it, so a reward exists without a judge, a reference answer or a rubric. That is why it turns up wherever a method needs a task it can score a million times cheaply
- And what that costs: the search space is small and arithmetic, so the task rewards search over a shallow discrete space and says little about anything else. A gain on Countdown is a gain at bounded symbolic search
Standing in the anthology
Carries no practice, deliberately — ADR-032, and filed for the half of
the paper nobody here cites.
Twenty-three documents report on Countdown. Zero cite the method. The
evolution-strategies cluster is measured on it throughout — SOTA-154
averages across Qwen2.5 and LLaMA3 on it, SOTA-212 runs OLMo3-7B on it
across 200 GH200s, SOTA-213 reads LIT-237’s 500-iteration run on it — and
none of them is about learning to search in language. The record borrowed the
benchmark and left the paper behind, which is why this note is about the task.
The record’s own evidence says Countdown behaves unlike its neighbours, and
that is the thing a citing practice implicitly assumes and cannot currently
look up. NOTE-082 records that ES reaches peak accuracy quickly on every
task except Countdown, and that HellaSwag declines steadily as Countdown
training continues past the point Countdown itself has converged. A benchmark
whose training dynamics differ from the rest of the suite is not
interchangeable with them, and four practices report on it as though it were
one task among several.
Unread — no NOTE. ADR-025 makes that mean what it says; the method is
what a reading would be about, and the method is not what this record uses.