Source
Lin et al. (2017), LIT-056 — ARXIV-1712.01887.
The measurement the practice rests on
LIT-056 opens with the number: 99.9% of the gradient exchange in distributed SGD is redundant. Send only the largest-magnitude components, accumulate the rest locally, and send them when they grow large enough to matter — with momentum correction, local gradient clipping and warmup on the sparsity ratio to keep the accumulated staleness from changing the optimisation.
That last list is the part that gets dropped when this practice is quoted, and it is what makes the technique work rather than merely compress. Naive top-k sparsification without momentum correction loses accuracy; the paper’s contribution is the corrections, not the observation that gradients are sparse.
When the condition in the title is met, and when it is not
“Slow networks” is doing real work. Compression trades computation and accumulated staleness for bytes, so it pays only when the interconnect is the bottleneck — commodity Ethernet, multi-datacentre, anything where the collectives dominate step time (which SOTA-045’s kind of measurement is what establishes).
Inside a modern training cluster it usually is not met. NVLink and InfiniBand move gradients fast enough that the overlap in SOTA-047 hides most of the cost, and the compression’s own compute and its interaction with the optimizer are then pure loss.
The record’s more current answer for the genuinely-slow case is SOTA-155 — many local steps with an outer optimizer over the deltas — which attacks frequency rather than volume and composes better with everything else.