anthology-of-the-sota

LIT-058: Measuring the Effects of Data Parallelism on Neural Network Training

3 min read

Properties6
StatusActive
Version3
Tagstraining-optimization
Cited as “introduced by” bySOTA-218 — Retune the learning rate, momentum and schedule at every batch size you compare, never by a scaling heuristic
Cited as “paper” byNOTE-144 — Measuring the Effects of Data Parallelism on Neural Network Training
Cited as “source” bySOTA-061 — Use largest batch that maintains >80% sample efficiency, SOTA-218 — Retune the learning rate, momentum and schedule at every batch size you compare, never by a scaling heuristic, SOTA-221 — To train past the batch size where your optimizer stalls, change the optimizer's conditioning rather than the scaling rule, THEORY-012 — How far a batch-size scaling heuristic transfers is a property of the optimizer, not of the heuristic, THEORY-013 — The SGD noise scale g = eps*N/B sets generalization, so the optimal batch size grows with the training set

Graph View

Shallue et al. (2018) — ARXIV-1811.03600

Key takeaways

  • 168,160 trained models, yielding 71,638,836 loss measurements, over 454 (workload, batch size) pairs, all aimed at one relationship: how the number of steps to a target accuracy varies with batch size. The 71 million figure is measurements, not runs — it is the published database of training curves, and it is the number most often misquoted as the run count.
  • A workload here is a (model, data set, training algorithm) triple, and there are 35 of them: six model families (fully connected, a simple CNN, ResNet, VGG, Transformer, LSTM), seven data sets (MNIST, Fashion MNIST, CIFAR-10, ImageNet, Open Images, LM1B, Common Crawl) and three optimizers (SGD, momentum, Nesterov momentum). All three optimizers carry a single global step size — no layerwise or curvature-normalized method is in the sweep, which is what SOTA-221 turns on.
  • The shape is consistent even when its position is not. There is an initial regime of near-perfect scaling — double the batch, halve the steps — then a region of diminishing returns, then a plateau past which more parallelism buys nothing. Perfect scaling ends anywhere from a batch of 2^4 to 2^13; the plateau begins between roughly 2^9 and 2^16. Two ranges, two transitions, routinely merged into one.
  • Where the transitions fall depends on the model, dataset and optimizer, and the paper finds no simple rule that predicts it. That negative result is the practical content: the maximum useful batch size has to be measured for a workload rather than inherited.
  • Careful metaparameter tuning at every batch size is what makes the comparison mean anything; several apparent batch-size effects in prior work are artefacts of tuning the schedule for one batch size and reusing it.

Standing in the anthology

The source of SOTA-218 — retune at every batch size you compare — and a corroborating source for SOTA-221, which is the other half of the same subject. THEORY-012 reads this paper together with LIT-265 and argues that its negative result about heuristics is really a result about optimizers, on the strength of the one comparative finding in it: momentum extends perfect scaling past plain SGD.

The paper also asks, in its future work, for a prospective predictor of the scaling behaviour that would spare the sweep. SOTA-198 is that, from a paper published two months later that does not cite this one.

The practice a future contributor is most likely to write here — “pick the largest batch that still scales” — is not filed, and this is the evidence it would need.

Backlinks