Loshchilov et al. (2017) — ARXIV-1711.05101
Key takeaways
- The inequivalence: L2 regularization and weight decay coincide for plain SGD (rescaled by the learning rate) and do not for adaptive methods, because the adaptive denominator rescales the L2 gradient along with everything else
- Common Adam implementations use L2 “often calling it weight decay in what may be misleading due to the inequivalence we expose”
- AdamW applies the decay to the parameters, outside the adaptive rescaling
- (i) The optimal weight decay decouples from the learning rate for both SGD and Adam — turning a two-dimensional sweep into two one-dimensional ones
- (ii) Adam’s generalization improves substantially, enough to compete with SGD with momentum on image classification, where it had been losing
Standing in the anthology
The source of SOTA-120.
Read in full for #114; the reading is NOTE-015. The practice is confirmed — an analytic argument, two measured consequences, and universal adoption since.
SOTA-120 was Deferred when this note was read, and the reading did not
settle it — the deferral was never about the source. It waited on evidence
that anyone acts on the result, and LIT-156 and LIT-122 supplied it: the
practice is Active.
A live tension worth carrying: LIT-153 argues that constant decoupled decay pins a layer’s equilibrium norm by hyperparameters rather than by data — a consequence of exactly the mechanism this paper introduced. Both hold. Decoupling is the right way to apply decay; applying it at a constant rate has an effect nobody noticed for years.