Lin et al. (2017) — ARXIV-1712.01887
Key takeaways
- The measurement first: 99.9% of the gradient exchange in distributed SGD is redundant. Communication bandwidth is the scaling limit, and almost all of what is being sent need not be.
- Sparsifying naively costs accuracy, so the contribution is the four corrections that stop it doing so — momentum correction, local gradient clipping, momentum factor masking, and warm-up training. The compression is the easy half; making it lossless in the sense that matters is the paper.
- Compression ratios of 270× to 600× without accuracy loss, across image classification, speech recognition and language modelling: ResNet-50’s gradient from 97MB to 0.35MB, DeepSpeech’s from 488MB to 0.74MB.
- The consequence is economic rather than algorithmic — large-scale distributed training on inexpensive 1Gbps Ethernet instead of specialized high-bandwidth interconnect, and distributed training on mobile.
Standing in the anthology
The source of SOTA-075, use gradient compression when the network is the constraint. The practice’s condition matters and comes from this paper: the technique buys nothing on a fast interconnect, and the corrections above are not optional if accuracy is to hold.
Filed next to LIT-051, which takes the other route to the same bottleneck — use the cluster’s spare CPU and bandwidth better rather than send less — and carries the record’s topology and buffer-tuning practices.