Source
Kim et al. (2025), LIT-441 — ARXIV-2509.14786.
What goes wrong without it
Fix a corpus — 200M tokens, in the study — and spend compute by adding epochs and parameters. Loss falls, then rises. Tuning the epoch count separately at each parameter count does not prevent it. The recipe is not saturating; it is overfitting, and past a point every additional parameter makes the model worse.
That failure is what bounds the data-constrained recipes the record already holds. It is also, on this evidence, an artifact of a hyperparameter nobody tunes.
The finding is the magnitude
The weight decay in general use is 0.1, inherited from GPT-3 and carried forward through recipe after recipe. Searched jointly with learning rate and epoch count by coordinate descent at each parameter count, the optimum for the most over-parameterized models is roughly 30x that.
Both the failure and the fix come from LIT-441, which first made this recommendation: on a fixed 200M-token corpus it showed the standard more-epochs-more-parameters recipe overfitting, then ran coordinate descent over weight decay, learning rate and epoch count at each parameter count until no single change helped. The regularized recipe it arrived at is reported as about 2.29x more data efficient than the standard one at that corpus size.
With it, two things change:
- Loss becomes monotone in parameter count — the turn upward disappears, so adding parameters is once again always worth doing if you can afford it.
- The resulting power law has parameter exponent ≈0.23 where Chinchilla’s is ≈0.34 — meaning the returns to model size are larger, not smaller, once the data is being used properly.
Why this is the cheapest thing to try first
The record’s three positions on repetition — SOTA-171’s four epochs, SOTA-173’s objective augmentation, and the diffusion route — all answer “how far may a corpus be repeated” with a number or an intervention. This answers it with a hyperparameter, and the intervention costs one sweep rather than a change of objective or of architecture.
It also says something uncomfortable about the dispute itself: SOTA-124, SOTA-171 and SOTA-173 between them report one setting of weight decay, and on this evidence the answer moves with it. None of those measurements is wrong; they may all be measurements of the under-regularized case.
Not a claim about AdamW
SOTA-001 and the decoupled-decay line are about the mechanism — how weight decay should be applied so that it does not interact with the adaptive step. This is about the value, in a regime the customary value was never chosen for. Nothing here disputes decoupling; it says the constant that decoupling made meaningful has been copied rather than picked.
Conditions, and why this is Proposed
One group, one corpus of 200M tokens, models up to 1.4B. Frontier pretraining is several orders of magnitude away on both axes.
“30x” is a measurement, not a rule. The paper gives no way to predict the optimum from the parameter-to-token ratio, so the practice is really “sweep it, and sweep upward” — the direction is the reusable part. A practitioner who sets 3.0 because it is 30 times 0.1 has taken a number out of one experiment.
The regime matters. This is about parameter-to-token ratios above Chinchilla with repeated data. At or below the compute-optimal ratio the inherited value is not shown to be wrong, and the failure this fixes does not arise.
A law exists now, in the wrong regime
The conditions above say the paper “gives no way to predict the optimum from
the parameter-to-token ratio, so the practice is really sweep it, and sweep
upward”. SOTA-261 is a law in exactly that ratio — the normalized AdamW
timescale, 1/(η λ S), following a power law in tokens-per-parameter with
exponent about −0.52, fitted over three orders of magnitude of compute.
It does not close this gap, and treating it as though it did would be the easy mistake. That law is fitted from 20 to 1280 tokens-per-parameter, and its normalization is introduced on the explicit ground that “LLM pre-training only uses one epoch of data”. This practice’s setting is the opposite end: parameters above Chinchilla-optimal, so TPP below 20 and often below 1, with the corpus repeated many times.
Nobody has checked whether either law extrapolates into the other’s regime. The record now holds two well-measured answers to how weight decay should move under data pressure, from opposite ends of one axis, and an unexamined gap between them. That is the honest state and it is a better question than either practice alone poses.
Known implementations
- None in the record. Every pretraining report the record holds that states a weight decay states something near 0.1.