Source
Kingma et al. (2014), LIT-001 — ARXIV-1412.6980.
The claim
Adam is what to reach for absent a reason to do otherwise, and it is LIT-001 that the record leans on for it.
That is defensible and was more so when it was written. Adam combines AdaGrad’s per-parameter scaling with RMSProp’s decaying average and corrects the bias in both moments, which makes it insensitive to gradient scale and so usable without the per-problem tuning SGD with momentum needs. The cost is memory — two extra states per parameter, the 8 bytes that SOTA-015 keeps in FP32 and that ZeRO exists to partition (SOTA-028).
The title, which was the reason this body was written
Until version 2 this practice was called “Default choice for neural network training”, which never says of what. A title that does not name its subject cannot be cited, searched for, or contested — and this practice is contested: SOTA-121 argues Muon beats the default at scale, and the weight-decay half is SOTA-120. Neither disagreement could attach to the old title.
It read as a fragment of the note it was promoted from, and it is one of
several on the #107 worklist in that shape — SOTA-036 (“gpt training
recipe”) is the other clear case, still open.
Where the default has actually moved
To AdamW rather than Adam — decoupled weight decay is now the default in every framework and every large run in this record — and, at frontier scale, to Muon and its relatives. This practice as written cannot express either, which is the practical cost of the title.