anthology-of-the-sota

LIT-007: Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour

1 min read

Properties4
StatusActive
Tagsdistributed-optimization, training-optimization
Cited as “introduced by” bySOTA-008 — linear warmup of LR stabilizes early training with large batch size.
Cited as “source” bySOTA-008 — linear warmup of LR stabilizes early training with large batch size.

Graph View

Goyal et al. (2017) — ARXIV-1706.02677

Key takeaways

  • Linear scaling rule for learning rates with batch size
  • Gradual warmup period crucial for stability
  • Batch normalization with corrected variance
  • Demonstrated successful training with 8192 batch size

Backlinks