Shojaee et al. (2025) — ARXIV-2506.06941
Key takeaways
- The instrument is the point. Established maths and coding benchmarks are contaminated and score only final answers. Controllable puzzle environments — Tower of Hanoi, River Crossing, Blocks World — let compositional complexity be dialled while the logical structure stays fixed, and let the reasoning trace be checked as well as the answer.
- Three regimes under matched inference compute. At low complexity plain models beat reasoning models and use fewer tokens; at medium complexity the extra thinking pays; at high complexity both collapse.
- The counterintuitive scaling limit. As problems get harder, reasoning effort rises and then falls — models spend fewer thinking tokens on the hardest problems while still holding budget in reserve.
- The negative claims about exact computation. Models fail to follow an explicit algorithm when given one, and reason inconsistently across puzzle families.
Standing in the anthology
Held for the instrument and the three-regime result, with the accuracy-collapse interpretation contested in the record rather than endorsed.
LIT-463 is a public comment arguing that the collapse is an artifact of the evaluation: output-token limits, a scorer that cannot tell refusal from failure, and River Crossing instances that are mathematically unsolvable yet scored as failures. The record takes the River Crossing point as established and the rest as argued — NOTE-213 sets out which is which.
This paper is NeurIPS 2025, camera-ready, revised in November 2025 — five
months after the comment. It went through peer review with the objection
already public. That does not settle the dispute and it does rule out reading
this as a debunked preprint, which is how a reader who met only the comment
would be likely to read it. Active, corrected_by declared, and neither
document treated as having won.