nucleation

LIT-1063: Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters

2 min read

Properties4
StatusActive — read in full 2026-10-15 from the arXiv preprint (arxiv.org/pdf/2408.03314, v1 of 2024-08-06, the only arXiv version) and from the ICLR 2025 conference version, open access in the ICLR proceedings (proceedings.iclr.cc, retitled "... than Scaling Parameters for Reasoning"), which is a revision not posted to arXiv: it charges verifier calls to inference FLOPs, which halves the stated R values (0.16/0.79/22 become 0.08/0.40/11) at unchanged budgets, and adds a verifier-quality check. Worth reading as the standard empirical case that the best use of inference compute depends on how hard a question is for the base model, and that choosing the method per difficulty bin matches best-of-N with up to about 4× less test-time compute on MATH with PaLM 2-S*, before counting the 2048-sample difficulty estimate. For QUESTION-095 it shows a FLOPs-matched amortization model: a ~14× larger model's extra pretraining is a fixed cost spread over all inference, so the per-query test-time budget it buys scales as M + 3(M − 1)/R, with R the ratio of inference to pretraining tokens; test-time compute wins at low R on easy and medium questions and loses at high R and on the hardest questions at every R. R is aggregate inference volume, not a count of recurring queries; the larger model's per-token cost is a retrieval-cost term; query predictability appears as difficulty but its cost is not counted; staleness and erosion are absent.
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-869 — Snell, Lee, Xu & Kumar — Scaling LLM test-time compute optimally
Cited as “source” byTHEORY-331 — Counting the inference a model will serve as well as its training, the least-compute way to reach a fixed quality shifts toward up-front compute as expected inference volume grows: a smaller model trained on more data in Sardana et al., a larger pretrained model over test-time search in Snell et al.; test-time compute won only at low volume and on questions the small model already sometimes solved

Graph View

Snell, C., Lee, J., Xu, K. and Kumar, A. (2024). Scaling LLM test-time compute optimally can be more effective than scaling model parameters. ARXIV-2408.03314

Key takeaways

Read in full (NOTE-869). On MATH with PaLM 2-S*, which way of spending test-time compute works best (beam search against a process reward model versus best-of-N; sequential self-revision versus parallel sampling) depends on how hard the question is for the base model, and choosing the method per difficulty bin matches best-of-N with up to about 4× less test-time compute (the discussion says 2–4×). In a FLOPs-matched comparison with a ~14× larger model, the extra test-time compute a small model may spend is set by R, the ratio of total inference tokens to pretraining tokens: when R is small the larger model’s pretraining cost buys many samples per query and test-time compute wins on easy and medium questions; when R is large the budget per query falls toward the parameter ratio and the larger model wins, always on the hardest questions. Difficulty estimation (2048 samples per question) and verifier and revision-model finetuning are not counted in the compute, R is an aggregate token ratio with no term for which queries recur, and the evidence is one model family on one benchmark of 500 test questions.

Why this is here

The owner, 2026-10-15: “there might also be something useful in the scaling law literature, regarding valuation of pre-training investment (amortized compute) vs inference time search/compute.” Read for QUESTION-095: compute spent once in training is amortized over every later query, against compute spent at inference per query. Machine learning, filed here for that question with the anthology flag (ADR-005, ADR-013); the anthology may hold its own reading.

The per-query side of the trade: when to spend at inference rather than up front.

The reading (NOTE-869) adds the amortization term: in the FLOPs-matched comparison the larger model’s extra pretraining is spread over the ratio R of inference to pretraining tokens, and test-time compute wins only at low R on easier questions. R is an aggregate volume, not repeated queries, and the larger model also pays more per inference token, a retrieval cost the amortized-inference readings lacked.

Backlinks