Loshchilov and Hutter (2016) — ARXIV-1608.03983
Key takeaways
- The schedule is a cosine annealing from the peak learning rate down to a small floor, and then a warm restart: the rate jumps back to near the peak and the anneal begins again, typically with each cycle longer than the one before.
- The motivation is escaping and exploring: a restart knocks the iterate out of the basin the previous cycle settled into, and the ensemble of end-of-cycle checkpoints is useful in its own right.
- The lasting contribution turned out to be the cosine shape, not the restarts. Every schedule in the record’s modern half anneals on a cosine or a variant of one; almost none of them restart.
Standing in the anthology
Superseded by LIT-035 — one long cosine cycle, as GPT-3 and its successors train — and kept because the chain is the content: warm restarts here, a single cycle in LIT-035 and
SOTA-039, and warmup-stable-decay after that. A reader meeting WSD needs to know that the field tried restarts first and what happened to them.
It was retired at migration for the right reason against the wrong paper: the entry named a work that does not exist, so the judgement “restarts lost to single-cycle cosine” was recorded against a fabrication. The judgement stands; it now applies to the paper that actually made the claim.