What to do
Keep the part of the gradient the compressor threw away. After compressing,
store residual = g - decompress(compress(g)) locally, and add it to the
next step’s gradient before compressing again. When the optimizer has a
momentum buffer, apply the correction to the buffer rather than to the raw
gradient — sparsification delays the coordinates it drops, and delayed
coordinates arrive with stale momentum already applied to them.
Set the rank or ratio second. LIT-337 reports rank 2 sufficient for convolutional networks and rank 4 for LSTMs; LIT-056 reaches 99.9% sparsity. Neither number means anything without the residual.
Why
The requirement is reported three times, by three compressors, in three forms. Low-rank projection: rank-4 PowerSGD “fails to converge to acceptable accuracy even on simple tasks” without error feedback, and matches uncompressed SGD with it. Sparsification: Deep Gradient Compression reports that at 99.9% sparsity, momentum SGD without momentum correction degrades significantly — because a coordinate held back for hundreds of steps is applied against a momentum buffer that has moved on. Sign-based compression with Adam: LIT-278 finds that applying error-compensated compression to Adam directly “corrupts error cancellation and severely harms convergence”, because Adam’s variance division is non-linear and the residual no longer means what the correction assumes.
That third case is the one that shows what the practice is really about. Error feedback is not a trick that makes compression work; it is the statement that the compressor must sit somewhere the residual is still meaningful. Where it
does not, the fix is to move the compressor, which is SOTA-215.
It is cheap and it is local. The residual is one buffer the size of the gradient, never communicated. There is no coordination cost and no hyperparameter.
What this does not settle
The ranks and ratios are per-architecture and mostly pre-transformer. LIT-337’s own limitations say transformer language models may need rank 32 or more, and its measurements are ResNet18 and an LSTM on 16 GPUs. The requirement generalizes; the numbers should not be assumed to.
Whether compression pays at all is a separate question. LIT-314 gives a lower-bound construction showing unbiased compression cannot improve the uplink and downlink terms simultaneously, which bounds what any of this buys. This practice says what to do if you compress.