Li et al. (2020) — ARXIV-2006.15704
Key takeaways
- The paper of record for
DistributedDataParallel, and it names its three techniques in the abstract: bucketing gradients, overlapping computation with communication, and skipping gradient synchronization. Near-linear scalability on 256 GPUs when configured appropriately. - Overlap is the one the anthology was missing a source for. Gradients become ready in reverse layer order during the backward pass, so a bucket can be reduced while earlier layers are still computing — which is what keeps the network busy through a phase that would otherwise be pure compute.
- Bucketing is the same problem Horovod’s Tensor Fusion (LIT-220) solves, arrived at independently in a different framework, and the two together are why the practice is standard rather than novel.
- “Skipping gradient synchronization” is gradient accumulation stated as a communication decision rather than a batch-size one — reducing every k steps instead of every step, which trades staleness for traffic.
Standing in the anthology
The source for SOTA-047 (overlap communication with the backward pass), which had been credited to LIT-051 — a paper that assumes overlap rather than introducing it.
Filed alongside LIT-220 for the same reason: the practices were right and their citations were not.