Source
Loshchilov et al. (2017), LIT-012 — ARXIV-1711.05101.
What the source showed
The recommendation is LIT-012’s: Loshchilov and Hutter showed that L2 regularization and weight decay coincide for plain SGD (up to a learning-rate rescaling) but not for Adam, whose adaptive denominator rescales the L2 gradient so that parameters with large gradient histories get less decay, and they introduced AdamW to apply the decay to the weights outside that rescaling. They measured two consequences: the optimal weight decay decouples from the learning rate, turning a joint sweep into two one-dimensional ones, and Adam’s generalization improves enough to compete with SGD with momentum on image classification, where it had been losing. That evidence is from image classification of 2017–2019; none of it is language-model training.
How this stopped being deferred
This practice was Deferred for sixteen days on a doubt carried over
verbatim from the old sota_maybe field:
I feel like everyone still uses vanilla Adam though…
llama2 used - β₁=0.9, β₂=0.95
The result was never in question — only whether anybody acts on it. The
promote_when: condition asked for a survey or a frontier training report
that states which of the two it uses, rather than leaving the choice
implicit in a config. The record has since acquired both, and neither had to
be gone looking for:
- LIT-156 is the survey exactly: ten optimizers across four scales, each tuned rather than handed the baseline’s hyperparameters, measured at the end of training. The baseline everything is judged against is a well-tuned AdamW — not Adam.
- LIT-122 is the stronger evidence, because there decoupling is a stated design decision rather than a default. Muon had results at small scale and none above it; the two changes that carried it to a 3B/16B model on 5.7T tokens were applying weight decay and matching AdamW’s update RMS. Someone had to decide to decouple, and wrote it down.
- LIT-132, LIT-139 and LIT-159 all name AdamW as the incumbent whose hyperparameters a challenger must transfer — LIT-139 keeps AdamW for the embeddings while moving the matrices elsewhere, which is a choice only statable if the optimizer is named.
So the doubt is answered against itself: the frontier reports say AdamW, and the survey benchmarks against AdamW. This is the case the pending-decisions report describes as “a decision the codebase has already made and hasn’t written down”.
The consensus: converged axis was already recording this. The gap between
that and a Deferred status was the point of having two axes — and it is now
closed from the status side, which is where it should have been all along.