Source
Bergsma et al. (2025), LIT-147 — Straight to Zero.
The finding
The customary schedule decays the learning rate to 10% of its peak. In a large empirical study across model sizes, batch sizes, datasets and vocabularies, with the peak tuned for each schedule, linear decay to zero consistently beat that and the other shapes tried at compute-optimal token budgets, and by more the further training went past compute-optimal. The reason offered: AdamW acts as an exponential moving average of updates, early training must move away from the initialisation while late training must average over enough updates to cancel gradient noise, and a floor at 10% leaves noise in the final weights that a decay to zero averages out.
The study and the recommendation are both LIT-147’s: it is the paper that argues against the 10× decay and proposes decaying linearly to zero instead, and it reports the advantage growing with the token horizon, so the case is strongest for runs trained well past compute-optimal.
Why this is Proposed
This is one group’s study, and the frontier recipes filed in the record still decay to a floor (Falcon-H1-Tiny’s ×64 exponential decay, LIT-119). The claim is compatible with WSD (SOTA-140) — it is about the end of the decay, not its start — and LIT-145’s cooldown to zero points the same way.
Two reports that look decisive and are not
The sharpest version of “the field has not moved” is DeepSeek-V4 (LIT-139), read for #18. It decays to exactly 10% of peak — 2.7e-4 to 2.7e-5 for Flash, 2.0e-4 to 2.0e-5 for Pro — which is the customary floor this practice’s source names in its first sentence and argues against, at 1.6T parameters over 33T tokens. Not a report that considered decaying to zero and declined; a report that used the default.
Kimi K3 (LIT-131) looks at first like the second half of the promotion condition — a schedule comparison with the hyperparameters retuned per schedule, which is exactly what that clause asks for. It is not, and the reason is in the same sentence of the report: the comparison is run “under a fixed minimum learning rate”. Holding the floor constant is holding this practice’s variable constant, so K3 settles which schedule to use and says nothing about how far to decay it. A study can satisfy the methodology this practice demands and still not be about it.