Bergsma et al., Cerebras (2025) — ARXIV-2502.15938
Key takeaways
- Cosine decay to 10% of the peak — the 10× decay — is the default the paper argues against. With the peak learning rate tuned, a linear decay to zero consistently outperforms the alternatives at compute-optimal dataset sizes, across model sizes, batch sizes, datasets and vocabularies.
- The advantage grows with the token horizon: the further past compute-optimal a model is trained, the more decaying to zero helps.
- The explanation reinterprets AdamW as an exponential moving average of weight updates. Early training needs to move away from the initialisation; late training needs to average over more updates to cancel gradient noise. Linear decay to zero balances the two demands, and a floor at 10% leaves noise unaveraged.
Standing in the anthology
The source of SOTA-141, and the sharpest statement in the record of how a decay stage should end. It sits beside WSD (LIT-144) rather than against it: WSD says when to decay, this says how far.
Its schedule has since been used as the baseline to beat. ScheduleFree+ (LIT-682) ran linear decay to zero, with the peak tuned by grid search, against WSD and its own learning-rate-free optimizer from 120M to 2B. Linear decay beat WSD whenever both were tuned to the same horizon, and a cosine floor of 0.1 came out suboptimal at 500M, which matches this paper’s case against the 10× decay. ScheduleFree+ in turn beat linear decay at 100 tokens per parameter everywhere except 2B, and lost to it at the larger sizes at 20. Those runs use one seed, and the linear-decay arm is AdamW against an AdamC package.