What to do
Run ordinary synchronous minibatch SGD from initialization until the first
learning-rate decay. Then switch to local SGD: each worker takes H steps on
its own and the workers average every H steps, with H of 16 to 32 as the
reported range.
When choosing H from first principles rather than from that range,
LIT-361 gives H = O(sqrt(T/(Kb))) for T total steps, K workers and
local batch b — the largest interval that still preserves linear speedup.
Why
The two methods are not competing; they are suited to different phases. The argument in LIT-362 is that large-batch SGD loses generalization because it loses gradient noise, and that this matters in the early phase, before the first decay, when the trajectory is still choosing a basin. After that decay the noise is no longer buying anything and per-step communication is paying for nothing.
Local SGD is the better of the two at equal compute per round. The same paper’s second finding is that local SGD beats minibatch SGD at matched effective batch size on both generalization and communication — so the switch is not a concession, and the only reason to run minibatch SGD at all is the early phase.
What this does not settle
Image classification, 2018. ResNets on CIFAR and ImageNet, K = 16
workers, switching at epoch 150. Nothing here tests it on a transformer.
The H = 500 in SOTA-155 is not a contradiction of the 16 to 32 here.
That was this document’s reading and it was wrong. The two numbers are chosen
against different objectives — this one is a swept accuracy peak, LIT-212’s is
a point on a curve flat enough that H = 1000 costs 2.9% perplexity while
communicating 20× less — and, more to the point, the two schemes do different
things at the synchronisation. This one averages parameters. LIT-212 runs
Nesterov momentum over the accumulated deltas, and its own ablation reports
that plain averaging at H = 500 performs poorly. THEORY-014 is the
account: the outer optimizer is what buys the interval, so a practice with no
outer optimizer should be expected to have a lower ceiling.
What that does not touch is this practice’s actual claim. The switch at the first learning-rate decay is about when to start communicating rarely, not about how rarely, and LIT-212 neither tests it nor contradicts it — it starts from a model pretrained for 24,000 steps, which is a phase boundary of a different kind and is not compared against not having one.
The generalization premise is not settled inside this record. The claim that large batches cost generalization, rather than costing tuning effort, is exactly what SOTA-218 says to be careful about: a large-batch run whose metaparameters were transferred rather than retuned will under-perform for reasons that have nothing to do with noise.
“The first learning-rate decay” presumes a schedule with decays. Under a cosine or schedule-free regime there is no such moment, and the practice gives no rule for finding one.