What to do
Keep an anchor model on each node alongside the local one. After every
tau local updates, pull the local model toward the anchor — no communication,
since each node holds its own copy — and let a separate thread average the
local models into a new anchor in the background. tau = 1 or 2.
The anchor is the method, not an option. Without it there is nothing to pull toward while the average is still in flight, and the scheme degenerates to ordinary local SGD, which has to wait.
Why
One step of slack is enough when the exchange has somewhere to go. The
condition LIT-323 states is simply that the parallel communication time be
smaller than tau steps of computation; past that, the latency is completely
hidden and more slack buys nothing while costing consensus. That is why the
recommendation is a small number rather than the largest tolerable one, and it
is the opposite of the reasoning behind SOTA-216 and SOTA-155, where a large
interval is the point and something has to recover what it costs.
The reported number is a ratio. At tau = 2, communication-to-computation
falls from 34.6% to 1.5%, with loss-versus-iteration convergence roughly
matching fully synchronous SGD — about 0.1 s of additional latency per epoch
against 1.5 s. Accuracy holding is the premise that makes the ratio
interesting, and the paper reports tau = 1 and 2 slightly exceeding
synchronous accuracy.
It also removes the straggler wait. Because the exchange is non-blocking, a node that finishes its local updates first does not idle — which is a second benefit from the same construction and is not priced into the 1.5%.
Conditions
It is a claim about a ratio, so it is a claim about your hardware. The 34.6% baseline is what communication cost on the paper’s cluster for that model. On a fabric where the exchange is already cheap there is nothing to hide. LIT-323 expects its advantage to be “further magnified” on a slower interconnect — it names 10 Gbps as an example of slower — or on a larger network such as a transformer, and tests neither.
CIFAR-10, 2020. One dataset. Earlier drafts of this document said CIFAR-10 and ImageNet; the paper’s experiments are CIFAR-10. CIFAR-10 is LIT-424 — 32×32 images, cheap enough to run a method on first and small enough that a result there is weak evidence about scale, which is the whole distance between what this paper measured and what the claim above is pitched at.
alpha = 0.6 and beta = 0.7 are not filed here. The pullback strength
and the anchor’s momentum are algorithm-local constants tuned on this task —
the paper reports alpha = 0.6 best for tau >= 2 and alpha = 0.5 at
tau = 1 — and ADR-041 is why they stay in the reading. They are the
standard settings for the method, not, as an earlier draft here said, a
variant for skewed data.
The construction is EASGD’s, and the paper says so. Pulling local models toward an anchor is elastic averaging; what LIT-323 adds is the asymmetric mixing and the observation that EASGD “did not observe and utilize” the overlap its own structure permits.
Known implementations
None recorded.