nucleation

LIT-1070: Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws

2 min read

Properties4
StatusActive — read in full 2026-10-15 from the authors' arXiv preprint v3 (arxiv.org/pdf/2401.00448v3, 14 Apr 2025), which is the ICML 2024 paper (PMLR 235, also fetched from proceedings.mlr.press/v235/sardana24a.html and diffed: same equations, tables and numbers, small wording edits); v1 (2023-12-31) is an earlier two-author NeurIPS 2023 workshop version without the experiments. Worth reading as the worked case of choosing how much to precompute by expected use: fixing quality, it adds lifetime inference FLOPs (2N per token) to training FLOPs (6N per token), finds the optimum numerically (no closed form), and shows the optimal model shrinking and training longer as demand grows (30B-Chinchilla quality with 10^13 inference tokens: 13.6B on 2.84× the data, 28% fewer FLOPs; in dollars, with inference at 1% utilisation, savings of 17% to 58%), with 47 training runs up to 10,000 tokens per parameter showing quality still improving but more slowly than the fitted law predicts. For QUESTION-095 it supplies expected reuse and a non-zero per-use cost as explicit terms, and nothing for staleness, uncertainty about demand (assumed known before training) or erosion; for CLAIM-303 it instantiates the break-even n > F/(c_r − c_a − u) with u = 0, and shows the choice as a continuous dial tied by a production function rather than a file-or-not decision.
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-881 — Sardana, Portes, Doubov & Frankle — Beyond Chinchilla-optimal: accounting for inference in scaling laws
Cited as “source” byTHEORY-331 — Counting the inference a model will serve as well as its training, the least-compute way to reach a fixed quality shifts toward up-front compute as expected inference volume grows: a smaller model trained on more data in Sardana et al., a larger pretrained model over test-time search in Snell et al.; test-time compute won only at low volume and on questions the small model already sometimes solved

Graph View

Sardana, N., Portes, J., Doubov, S. and Frankle, J. (2023). Beyond Chinchilla-optimal: Accounting for inference in language model scaling laws. ARXIV-2401.00448

Key takeaways

Read in full (NOTE-881). Fixing a target pre-training loss and adding inference FLOPs (2N per token over a known lifetime demand D_inf) to training FLOPs (6N per token), the compute-minimising model is smaller and trained on more data than Chinchilla-optimal, increasingly so as D_inf approaches and exceeds the training-set size; the optimum has no closed form and is found numerically. The savings are modest in FLOPs (for a 30B-Chinchilla-quality model with 10^13 inference tokens, 28% by training 13.6B on 2.84× the data) and larger in dollars once inference’s low hardware utilisation is priced in (17% to 58% in the paper’s examples). 47 training runs (150M–6B parameters, 10–10,000 tokens per parameter) find loss still falling at extreme ratios, but more slowly than Chinchilla-fitted laws predict, so the formula overstates the gain from long training. Demand is assumed known before training and fixed given quality; there is no term for uncertainty, discounting, obsolescence or a model being replaced.

Why this is here

The owner, 2026-10-15: “there might also be something useful in the scaling law literature, regarding valuation of pre-training investment (amortized compute) vs inference time search/compute.” Read for QUESTION-095: compute spent once in training is amortized over every later query, against compute spent at inference per query. Machine learning, filed here for that question with the anthology flag (ADR-005, ADR-013); the anthology may hold its own reading.

Expected reuse made an explicit term in the choice of how much to precompute.

The reading (NOTE-881) confirms it: lifetime inference demand is an explicit term, so with enough demand a smaller model trained longer is cheaper. Demand is assumed known in advance and independent of model size; there is no staleness, uncertainty or erosion term, and the precomputed thing is a cheaper computation, not a stored answer.

Backlinks