Defazio et al. (2024) — ARXIV-2405.15682
Key takeaways
The problem it removes is the one every schedule in this record works around. A learning-rate schedule that knows the stopping step T outperforms one that does not — that is the starting observation, and it is why cosine fixes the token budget in advance and why WSD was proposed to escape doing so. Schedule-Free removes the dependency instead of routing around it: no T, no shape, and no additional hyperparameter over standard momentum.
Scheduling and iterate averaging are the same thing. The method is a consequence of a theory unifying the two, which is the part worth reading rather than the benchmark table. A schedule’s decay and an average over the trajectory are two spellings of one operation, and once that is seen, the averaged form is the one that does not need to know when it will stop.
The evidence is unusually independent for a single paper. Schedule-Free AdamW was the core of the winning entry to the MLCommons 2024 AlgoPerf Algorithmic Efficiency Challenge, Self-Tuning track — a third-party competitive benchmark with fixed rules, which is not the same kind of claim as a self-reported sweep. Results span convex problems to large-scale deep learning.
Standing in the anthology
The schedule chain has three answers now, and this is the one the record does not hold as a practice.
Warm restarts (LIT-042, retired) → a single cosine cycle (LIT-035, SOTA-039, retired) → warmup-stable-decay (SOTA-140) → and this, which says the branch point was wrong: the choice is not which shape to decay through but whether to specify a stopping time at all.
Read against SOTA-140 specifically, the overlap is exact and the mechanism is not. WSD’s central practical claim is that the total token count need not be fixed when training starts, because the decay can be launched from any stable-stage checkpoint. Schedule-Free delivers the same property with no decay phase to launch — and, unlike WSD, does not need its peak learning rate re-tuned for a constant stage, because there is no constant stage.
The practice is SOTA-156, filed Proposed, and the reason is a gap
rather than a doubt. No frontier report in this record trains under it. The
record’s schedule material is contested between WSD and cosine at the scale
where the disagreement matters (LIT-131 against LIT-145), and a third answer
with no adoption at that scale does not settle it — it widens it. The
condition that would promote it is in that practice’s promote_when:, which
is where a standing query belongs: a frontier report training under
Schedule-Free, or a schedule comparison including it as an arm with each
method’s hyperparameters swept.
The AlgoPerf result deserves its own sentence, because it is the strongest outside evidence in the schedule material and it does not transfer cleanly. AlgoPerf’s workloads top out far below the scale at which LIT-131 and LIT-145 disagree, so a win there is evidence about the method’s robustness under fixed rules rather than about its behaviour at 2.8T.