anthology-of-the-sota

LIT-010: Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates

1 min read

Properties4
StatusActive
Tagstraining-optimization
Cited as “introduced by” bySOTA-009 — warmup to a large early lr, anneal throughout training to small final lr
Cited as “source” bySOTA-009 — warmup to a large early lr, anneal throughout training to small final lr

Graph View

Smith et al. (2017) — ARXIV-1708.07120

Key takeaways

  • One-cycle learning rate policy (1cp)
  • Enables much faster training
  • Uses very large learning rates
  • Demonstrates regularization effect

Backlinks