Source
Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.
The practice is right, and this is not where it comes from
Gradients become available in reverse layer order as the backward pass runs, so the gradient for the last layer is ready long before the first layer’s is computed. Waiting for the whole backward pass and then reducing everything leaves the network idle throughout the backward and the GPU idle throughout the reduction. Issuing each layer’s reduction as its gradient lands overlaps the two, and on a well-tuned job most of the communication disappears behind compute.
That is correct, and it is the foundation the rest of this cluster builds on. It is also standard practice since well before LIT-051 — it is what Horovod and PyTorch’s DistributedDataParallel do by default, and the technique predates both.
What LIT-051 actually contributes
This practice used to cite LIT-051, whose contribution is a unified parameter-server/all-reduce framework and a CPU-side Summation Service. That paper assumes overlap; it does not introduce it, and citing it here credited the wrong work.
The source is now LIT-219, the PyTorch DistributedDataParallel paper,
which names the technique in its abstract — “bucketing gradients, overlapping
computation with communication, and skipping gradient synchronization” — and
reports near-linear scalability on 256 GPUs with them.
The cost, since the title does not
The overlap is what makes the bucket size a tuning parameter: reduce too eagerly and each collective is too small to reach peak bandwidth; too lazily and there is nothing left to hide the last one behind. That trade is the real