anthology-of-the-sota

LIT-001: Adam: A Method for Stochastic Optimization

1 min read

Properties5
StatusActive
Tagstraining-optimization
Compared againstLIT-686 — On the distance between two neural networks and the stability of learning
Cited as “introduced by” bySOTA-001 — Use Adam as the default optimizer absent a reason to choose otherwise, SOTA-002 — Common hyperparameters: β₁=0.9, β₂=0.999, ε=1e-8, SOTA-003 — Learning rate typically 1e-4 to 1e-3 for most tasks
Cited as “source” bySOTA-001 — Use Adam as the default optimizer absent a reason to choose otherwise, SOTA-002 — Common hyperparameters: β₁=0.9, β₂=0.999, ε=1e-8, SOTA-003 — Learning rate typically 1e-4 to 1e-3 for most tasks

Graph View

Kingma et al. (2014) — ARXIV-1412.6980

Key takeaways

  • Introduces adaptive moment estimation for gradient-based optimization
  • Combines advantages of AdaGrad and RMSProp
  • Corrects bias in moment estimates
  • Provides theoretical convergence guarantees

Historical impact

  • Foundational optimizer still widely used

Measured against later optimizers

Bernstein et al.’s Fromage paper (LIT-686) ran Adam as a baseline beside SGD and Fromage, three seeds each. On the test-set numbers Adam is behind Fromage on all four benchmarks and behind SGD on three: CIFAR-10 loss 0.76 against Fromage’s 0.31, ImageNet loss 1.184 against 1.126, GAN FID 23.9 against 16, and Wikitext-2 transformer perplexity 172.7 against 171.1. On training perplexity, which that paper calls its primary metric, Adam wins the transformer clearly, 36.8 against Fromage’s 66.1. Fromage’s CIFAR-10 and transformer rows also carry a per-layer norm cap the baselines are not said to have had.

Backlinks