Source
Hu et al. (2024), LIT-144 — MiniCPM; Hägele et al. (2024), LIT-145, for the comparison against cosine.
Warm up, hold the learning rate at its peak for most of training, then decay it fast over the last stretch — about 10% of the tokens in MiniCPM, up to about 20% in Hägele et al., who found the cooldown’s length and shape to matter and recommend a 1-sqrt shape to zero. Most of the loss improvement arrives in the decay. The schedule scales as predictably as cosine across model sizes.
On quality, be precise about what LIT-145 found, because it is the half that is now contested. Sweeping the maximum learning rate for both schedules, it reports “an almost perfect match between the performance of the best cosine and cooldown schedule even for different training durations”, with the cooldown slightly ahead when it runs for 10–20% of steps and the cooldown length needed to match cosine shrinking as training gets longer. A match, in other words, plus the practical advantages below — not a quality win that the advantages come on top of.
What it buys beyond quality: the total token count need not be fixed when training starts, since the decay can be launched from any stable-stage checkpoint; scaling laws can be fitted from one run’s checkpoints instead of a run per duration; continued pretraining is a matter of resuming the stable stage; and high-quality or SFT-style data can be concentrated in the decay (LIT-144), which is where LIT-119’s recipes put theirs.
Conditions: the peak learning rate must be tuned for the constant stage — a value tuned for cosine is too high to hold. LIT-146 (SOTA-142) is one way to set it. How far the decay goes is its own question: LIT-147
(SOTA-141) argues for all the way to zero; production recipes in the record decay to a floor (Falcon-H1-Tiny: ×64 exponential over 100 GT of 800 GT).
Contested
Kimi K3 (LIT-131) ran the comparison again at 2.8T and came out the other way. Its argument is the same one LIT-145 acted on: the two schedules’ optimal peak learning rates and batch sizes differ substantially even at matched model size and token budget, so a shared hyperparameter setting silently favours whichever schedule it happens to suit. K3 therefore runs an independent scaling-law search per schedule — over batch size, learning rate, tokens-per-parameter and model shape — and reports that under each schedule’s own optimum, cosine decay consistently reaches a lower final loss. K3 adopts cosine.
So both sides of this disagreement accept the same methodology and disagree on the result. Three things stop that from settling the question:
- Scale. LIT-145 ran at 124M–360M; K3 fitted at the scale of a 2.8T model. LIT-145 itself reports that the cooldown length needed to match cosine falls as training lengthens, so the two are not obviously measuring the same regime.
- Decay depth is held fixed, and it is a live variable here. K3 compares “under a fixed minimum learning rate”, while the WSD LIT-145 recommends decays to zero with a 1-sqrt shape — the question SOTA-141 is Proposed about. A WSD arm stopped at a floor is not the WSD this practice’s own evidence endorses, and K3 does not say which shape or depth its arm used.
- The practical case is not contested — it is not addressed. The argument above is about an open token budget, branchable stable-stage checkpoints, and scaling laws fitted from one run. K3 compares final loss and nothing else, and gives those up silently: a whole-run cosine fixes the duration in advance, which is the one thing LIT-145 set out to avoid. (It keeps a late “cooldown phase” — that is where its context window goes from 256K to 1M — so what it drops is the open budget, not the idea of a final stretch.)
The rest of the frontier has not moved with K3. DeepSeek-V4 (LIT-139), whose report precedes it by a month, warms up over 2000 steps, holds the peak for most of training, and decays near the end — warmup-stable-decay, with cosine as the shape of the cooldown rather than as the schedule. That distinction is easy to lose: a report saying “cosine” may mean K3’s whole-run cosine or V4’s cooldown shape, and only the first contests this practice.
Sequence
Warm restarts (LIT-042, retired) → a single cosine cycle (LIT-035,
SOTA-039, retired by this) → WSD, with its decay shape and depth still
being settled (SOTA-141) and its peak set by scaling law (SOTA-142).
There is a third answer, filed as SOTA-156 and Proposed.
Schedule-Free AdamW says the branch point above was the wrong one: the
question is not which shape to decay through but whether to name a stopping
time at all. It reaches this practice’s headline property — the token budget
need not be fixed when training starts — with no decay phase to launch and no
peak re-tuned for a constant stage, since there is no constant stage.
Someone has now run the two against each other, though not in the form
that practice’s promote_when: asks for. LIT-682 (2026) runs
ScheduleFree+ against a WSD arm from 120M to 2B parameters, and WSD loses at
every size and at every horizon it tests. Three things keep that from
displacing this practice. The winning arm is a package (inner momentum,
Polyak steps and AdamC weight decay) set against AdamW. How WSD was tuned is
not stated. And the work is the Schedule-Free author’s own, with one seed per
arm. The same report finds tuned linear decay beating WSD “when tuned to the
same training horizon”. That concerns the known-horizon case, where this
practice’s open-budget property is not needed.
Known implementations
- MiniCPM; Falcon-H1 and Falcon-H1-Tiny; DeepSeek-V4 (constant peak, cosine-shaped cooldown near the end)
- Against: Kimi K3, which compared the two with hyperparameters retuned per schedule and adopted a whole-run cosine