Contribution
Proposes SlowMo, a meta-framework that wraps any communication-efficient distributed base optimizer with a periodic slow momentum update, and proves this achieves linear speedup on smooth non-convex objectives—also providing the first convergence guarantee for BMUF and the Lookahead optimizer.
Key insight
Adding a slow outer momentum step after every tau inner steps of local SGD or a decentralized optimizer consistently improves both optimization and generalization at negligible extra communication cost, because slow momentum corrects accumulated drift between workers at the outer level.
Assumptions
- Objective functions are smooth (L-Lipschitz gradients) and non-convex; bounded gradient variance.
- Base optimizer (local SGD, SGP, OSGP) already provides linear speedup in the number of workers m before SlowMo is applied.
- Total training steps T satisfies T >= m^3 * tau^3 for the linear speedup term to dominate in the convergence bound.
- Workers are homogeneous (same speed); the periodic AllReduce synchronizes all workers at each outer step.
- Slow momentum hyperparameters alpha and beta are fixed throughout training (not adaptive).
Key results
- Theorem 1 (SlowMo convergence). SlowMo with m workers and tau inner steps converges to a stationary point of smooth non-convex functions at rate O(1/sqrt(mtauT)), matching AR-SGD and achieving linear speedup. Holds when: Smooth non-convex objectives; T >= m^3 * tau^3; fixed slow LR alpha and momentum beta.
- BMUF first convergence guarantee (Corollary). BMUF (local SGD + slow momentum with beta=0) achieves linear speedup in m workers, providing its first theoretical convergence guarantee. Holds when: Same conditions as Theorem 1 with beta=0.
- Lookahead convergence (Corollary). Lookahead (single worker SlowMo with beta=0) converges to a stationary point, providing its first convergence guarantee. Holds when: m=1 worker; smooth non-convex objective.
Claims
| id | claim | strength | support |
|---|---|---|---|
| C1 | SlowMo consistently improves validation accuracy over the base optimizer (Local SGD, SGP, OSGP) across CIFAR-10, ImageNet, and WMT’16 tasks. | strong | Table 1: e.g., Local SGD + SlowMo improves ImageNet top-1 from 69.94% to 73.24%; SGP + SlowMo from 75.15% to 75.73%. |
| C2 | SlowMo achieves these improvements with negligible additional communication overhead when tau is large (e.g., tau=48). | strong | Table 2: per-iteration times are nearly identical with and without SlowMo for SGP and OSGP base optimizers. |
| C3 | SlowMo converges to a stationary point of smooth non-convex functions at rate O(1/sqrt(mtauT)), achieving linear speedup in workers m. | strong | Theorem 1 with formal proof; rate matches that of AR-SGD, showing no asymptotic cost from slow momentum. |
| C4 | The slow momentum update (not momentum buffer synchronization) is responsible for the accuracy gains. | moderate | SGP-SlowMo-noaverage (removing the AllReduce before the slow update) achieves 75.78% vs. 75.73% on ImageNet, suggesting the buffer sync is not the key factor. |
Method
Slow Momentum (SlowMo).
SlowMo runs as a two-level loop on top of any base optimizer. In each outer iteration, all m workers independently run tau steps of the base optimizer (e.g., local SGD, SGP). Workers then AllReduce their parameters to obtain a synchronized model. A slow momentum update is applied: the accumulated pseudo-gradient (difference between old and new parameters) is added to a momentum buffer, and the shared model is updated with a slow learning rate alpha and slow momentum beta. Workers are re-initialized to this updated shared model. The base optimizer can itself be distributed (SGP, OSGP) or communication-free (local SGD).
- Inner loop: tau steps of any base optimizer (local SGD, SGP, OSGP) per worker
- Periodic AllReduce to synchronize parameters across workers
- Slow momentum buffer updated with pseudo-gradient (parameter change over tau steps)
- Slow outer update with hyperparameters alpha (slow LR) and beta (slow momentum)
- Worker re-initialization to shared outer model after each outer step
Concepts
- SlowMo (Slow Momentum) — A meta-framework wrapping distributed base optimizers with a periodic slow momentum outer update applied after every tau inner steps.
- BMUF (Block-wise Model Update Filtering) — An existing distributed training method equivalent to SlowMo with local SGD as the base optimizer; SlowMo provides its first convergence guarantee.
- Lookahead optimizer — A single-node optimizer equivalent to SlowMo with one worker and beta=0; SlowMo’s theory covers it as a special case.
- Stochastic Gradient Push (SGP) — A decentralized distributed training method that uses gossip-style peer-to-peer communication at every step, usable as a base optimizer in SlowMo.
- Pseudo-gradient — The difference between the parameter vector before and after tau inner optimizer steps; used as the gradient signal for the slow momentum update.
Connections
Builds on.
- Local SGD Converges Fast and Communicates Little (LIT-361) — Local SGD is one of the primary base optimizers for SlowMo; SlowMo adds a slow momentum outer layer on top.
- Don’t Use Large Mini-Batches, Use Local SGD (LIT-362) — Introduces post-local SGD; SlowMo generalizes the idea of a corrective outer step applied to local SGD workers.
- Scalable Training of Deep Learning Machines by Incremental Block Training (BMUF, Chen & Huo 2016) — BMUF is a special case of SlowMo; this paper provides BMUF’s first convergence proof.
Related.
- DiLoCo: Distributed Low-Communication Training of Language Models (LIT-212) — DiLoCo directly cites SlowMo as inspiration for using Nesterov outer momentum on top of inner local optimization steps.
Recommendations
- R1 — Apply SlowMo on top of local SGD or decentralized optimizers with slow momentum beta in [0.4, 0.8] and slow learning rate alpha=1 as a default starting point. Topic: hyperparameter defaults · Strength: strong · When: Any distributed training setup using local SGD or gossip-based decentralized methods.
- R2 — Set tau=48 for decentralized base optimizers (SGP/OSGP) and tau=12 for local SGD to balance accuracy and communication overhead. Topic: inner step count · Strength: moderate · When: ImageNet-scale training; optimal tau varies by task and may require tuning.
- R3 — Prefer SGP as the base optimizer over local SGD within SlowMo for fixed communication budget, as SGP-SlowMo achieves better accuracy than LocalSGD-SlowMo. Topic: base optimizer selection · Strength: moderate · When: When decentralized peer-to-peer communication is feasible alongside the periodic AllReduce.
Bearing on the record
SlowMo wraps any local-update or decentralized optimizer in an outer momentum step, which is structurally the same move as the record’s DiLoCo practice and predates it by four years. Reading the two together is how the record would learn whether the outer optimizer or the inner step count is doing the work.
Limitations
- Convergence proof covers smooth non-convex functions but requires T >= m^3
- tau^3 for linear speedup to dominate, which may be impractical for large tau.
- SlowMo introduces three additional hyperparameters (alpha, beta, tau) that require tuning per task.
- When combined with SGP/OSGP base optimizers, the periodic AllReduce for synchronization doubles communication at each outer step.
- Experiments are limited to vision (CIFAR-10, ImageNet) and one NLP task; broader applicability to LLM-scale training is unvalidated.
Open questions
- Can the periodic AllReduce in SlowMo be eliminated entirely (as hinted by SGP-SlowMo-noaverage results) without loss of convergence guarantees?
- What is the theoretical reason slow momentum improves generalization, beyond its optimization benefits?
- How does SlowMo interact with gradient compression techniques to further reduce communication?