What to do
Do not apply 1-bit or error-compensated compression to an Adam update as it stands. Run Adam uncompressed for a warmup period, then stop updating the second-moment estimate and treat it as a fixed preconditioner for the rest of training; compress what remains, which is a momentum SGD update under a constant diagonal scaling and does admit error compensation.
Two settings, both from LIT-278: the warmup must be at least as long as
the learning-rate warmup, and the variance-norm ratio
||v_t||_1 / ||v_{t-delta}||_1 >= 0.96 is usable as an automatic stopping
criterion for it.
Why
The failure is about where the compressor sits, not how hard it squeezes.
Error feedback (SOTA-214) carries a residual forward on the assumption
that adding it back next step undoes the loss. Adam divides by a running
sqrt(v) that changes between those two steps, so the residual is added back
under a different scaling than the one it was subtracted under, and the
cancellation the method depends on does not happen. This is why the fix is not
a smaller compression ratio.
The observation that makes it work is empirical and cheap to check. Adam’s variance term changes rapidly early and then stabilises; after that, freezing it costs little. The paper reports allreduce consuming 80–94% of step time for BERT on Ethernet clusters above 8 nodes, which is the regime where this is worth doing at all.
What this does not settle
One lab, one model family, one network fabric. BERT-scale models on Ethernet. Whether the variance stabilises the same way in a long transformer pretraining run, where the data distribution shifts, is untested here.
Freezing the variance changes the optimizer. The practice is presented as a communication optimization, and it is also a change to what is being optimized. LIT-278 reports matched convergence; nothing in this record checks what it does to a run long enough for the preconditioner to go stale.
The record has a practice this contradicts in spirit. SOTA-218 says
not to assume a metaparameter transfers. Freezing v is exactly such an
assumption, made deliberately, and it is measured rather than assumed — but
the measurement is one paper’s.