anthology-of-the-sota

LIT-362: Don't Use Large Mini-Batches, Use Local SGD

2 min read

Properties5
StatusActive
Tagsdistributed-optimization
Cited as “introduced by” bySOTA-216 — Switch from minibatch SGD to local SGD at the first learning-rate decay, rather than choosing between them
Cited as “paper” byNOTE-117 — Don't Use Large Mini-Batches, Use Local SGD
Cited as “source” bySOTA-216 — Switch from minibatch SGD to local SGD at the first learning-rate decay, rather than choosing between them, THEORY-014 — The outer optimizer is what buys the inner step count

Graph View

Lin et al. (2018) — ARXIV-1808.07217

Key takeaways

Introduces post-local SGD—a two-phase scheme that first trains with large- batch SGD then switches to local SGD—and empirically demonstrates it closes the generalization gap of large-batch training while being more communication-efficient than both large-batch and standard local SGD.

  • Post-local SGD closes the generalization gap (Table 3). Post-local SGD with K=16 workers and H=16 or H=32 local steps matches the single-machine (K=1) baseline accuracy of ~93% top-1 on CIFAR-10, while large-batch mini- batch SGD at K=16 achieves only ~91-92%. The gain is robust across ResNet-20, WideResNet, and DenseNet architectures. Holds when: Validated on CIFAR-10/100 and ImageNet (ResNet-50). Communication rounds are reduced by factor H relative to large-batch SGD at same per-step gradient budget.
  • Local SGD beats mini-batch SGD at fixed effective batch size (Figure 2). At the same effective batch size (K * H * B_loc = K * B), local SGD with H=4 achieves strictly higher test accuracy than mini-batch SGD, across all tested architectures on CIFAR-10. Holds when: Holds for K in {4, 8, 16} and H in {4, 16, 32}; effect is more pronounced at larger K.
  • Sharpness reduction (Figure 4). Local SGD converges to minima with lower sharpness (smaller largest Hessian eigenvalue) than large-batch mini-batch SGD at the same effective batch size. Post-local SGD achieves sharpness similar to single-machine SGD. Holds when: Measured on CIFAR-10 ResNet-20; sharpness correlates with generalization gap across methods.

What the evidence does not cover

  • Experiments limited to vision benchmarks (CIFAR, ImageNet); generalization to NLP or other modalities not validated.
  • Post-local SGD’s transition point t’ requires tuning and is heuristically set at the first LR decay.
  • Local SGD with very large H and K can encounter early-training optimization difficulties that affect later generalization.
  • Theory from Stich (2018) covers only convex objectives; non-convex guarantees for post-local SGD are lacking.

Standing in the anthology

Read — the reading is NOTE-117. Arrived in the imported batch, which brought in the local-update branch — train apart for H steps, then average.

Backlinks