LIT-441: Pre-training under infinite compute

Kim et al. (2025) — ARXIV-2509.14786

Key takeaways

• The premise is a crossing of two curves. Web text grows about 1.03x a year; pretraining compute about 4x. The paper takes the limit that implies — fixed data, unbounded compute — and asks what recipe wins there.
• The standard recipe fails, and fails in a specific way. Holding 200M tokens fixed and adding epochs or parameters, loss eventually turns up. Tuning epoch count per parameter count does not save it. The failure is overfitting, not saturation.
• The first fix is regularization, and the magnitude is the finding. The weight decay everyone inherits from GPT-3 is 0.1. Jointly tuning weight decay, learning rate and epoch count by coordinate descent at each parameter count, the optimum for over-parameterized models is roughly 30x that. With it, loss becomes monotone in parameter count and follows a clean power law with exponent ~0.23 — where Chinchilla’s parameter exponent is ~0.34, so the returns to size are larger once the data is used properly.
• The second fix beats the first. Ensembling K independently trained models — same everything but the seed — and averaging logits gives a lower asymptote than scaling one model to infinite width. Two 300M models beat one 600M. Ensembling and parameter scaling compose.
• Ensemble members want different hyperparameters than a lone model: more epochs and less weight decay per member, which the authors read as each member being usefully more overfit. Tuning for the many-member limit rather than for one model moved the asymptote from 3.34 to 3.17.
• Distillation removes the inference cost. An 8-member ensemble distils into a 300M student retaining ~83% of the gain, and self-distillation at equal size and architecture also helps — so the data efficiency does not require a large final model, or even a large model during training.
• The aggregate number: the joint recipe is ~5.2x more data efficient than the standard one at 200M seed tokens, and the fitted data-scaling laws say the multiple holds at higher token counts.
• Downstream transfer is checked rather than assumed: +9% on PIQA/SciQ/ARC-E, and on continued pretraining over math data the ensembling recipe on 4B tokens beats default CPT on the full 73B.

The methodological proposal, which is separable

The paper asks to evaluate a monotone scaling recipe by the asymptote of its fitted scaling law rather than by its loss at some chosen compute budget. The argument is that when compute is not the constraint, loss at a budget answers a question nobody asked; the asymptote answers “how good can this recipe get”.

That is a claim about how to measure, and it is what licenses every comparison in the paper — the ensembling recipe wins on asymptote while losing at small member counts. It also carries the paper’s largest methodological risk: an asymptote is an extrapolation, and the authors’ own sensitivity analysis puts run-to-run variation at 0.01 loss across three seeds, which is small but is measured on the fits rather than on the double limits that the headline numbers take.

Standing in the anthology

Sources two practices and one measurement claim, and it arrives at the dispute the record holds over data repetition from a fourth direction.

Its baseline is LIT-166’s. The “standard recipe” here is Muennighoff et al.’s data-constrained one, repeating the corpus for more epochs while adding parameters, and the first result is a disagreement with that paper’s scaling law: the decay-based law has loss falling monotonically in epoch count, while at 200M seed tokens the loss here turns up once the epochs are high enough. By this paper’s account, LIT-166 acknowledged the discrepancy and dropped most overfitting runs from its fit (its App. D). What is added on that setting is the regularization that removes the overfitting, the ensembling and distillation that beat parameter scaling, and the asymptote as the measure.

SOTA-124, SOTA-171 and SOTA-173 all ask how many times may a corpus be repeated. This paper’s answer is that the question is underspecified, because the number depends on a hyperparameter everyone was copying rather than tuning. Under-regularized, the answer is “a few”; at 30x the weight decay it is “as many as you can afford”, and the curve stops turning up. That is the same shape of argument as SOTA-173’s — the overfitting is a property of the recipe, not of repetition — reached by a different intervention, and it is the cheaper of the two to try.

The ensembling result also sits awkwardly beside everything the record says about scaling a single model, and deliberately so: it is a claim that at sufficient parameter count the field’s default move is the wrong one. Nothing in this record trains an ensemble, and the largest ensemble member here is 1.4B, so the awkwardness is not yet a contradiction.