Hu et al., Tsinghua and ModelBest (2024) — ARXIV-2404.06395
Key takeaways
- Warmup-Stable-Decay (WSD): warm up, hold the learning rate at its peak for the bulk of training, then decay it sharply — the paper uses about 10% of the total tokens for the decay. The loss drops steeply during the decay and the final quality matches or beats a cosine schedule of the same length.
- The stable stage has no predefined end, so the total token count need not be fixed when training starts; any checkpoint from the stable stage can be branched into a decay to produce a finished model, which makes scaling-law studies and continued pretraining far cheaper than with cosine.
- The decay stage is where the authors mix in high-quality and SFT-style data, using it for domain adaptation without a separate fine-tuning run.
- The rest of the paper is a model wind tunnel: µP-style scaling experiments (LIT-148) on small proxies to set width, depth, batch size and learning rate before the 1.2B and 2.4B runs.
Standing in the anthology
The source of SOTA-140, the schedule that every 2025–2026 report in the record uses and that retires the single-cycle cosine practice drawn from
GPT-3 (SOTA-039). LIT-145 is the controlled study that says why it works; LIT-146 sets the peak for it.