Source
Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311.
McCandlish et al. (2018), LIT-017 — ARXIV-1812.06162.
Known implementations
- PaLM
The other end of SOTA-092’s curve
As the model improves, the gradient’s useful component shrinks relative to its noise, so averaging more samples buys a more accurate step where before it bought a redundant one. Past that point the larger batch is strictly better per step and the question becomes whether the extra samples are worth their wall-clock.
In LIT-069 this is the second clause of the same sentence as SOTA-092: PaLM 540B’s batch doubles at step 50k and again at step 115k, from 1M to 4M tokens, because larger batches are “beneficial later in training due to better gradient estimates”. The paper cites Smith et al. (2018) and McCandlish et al. (2018) for that and does not test it.
The record names McCandlish et al., LIT-017, as the origin, and as the evidence. “Better gradient estimates” is their gradient noise scale: the ratio of the per-example gradient variance to the squared true gradient, which sets the batch size past which more samples stop speeding up training. They measured it rising through training, tracking a critical batch size that grows by an order of magnitude or more over a run, and analysed growing the batch to follow it. Smith et al. (arXiv 1711.00489), PaLM’s other citation, also grow the batch late in training, but their reason is that doing so anneals SGD’s noise the way a learning-rate decay does, not that the gradient estimate gets more valuable; it is a different recommendation and is not filed here.
The two halves together are the empirical content of critical batch size: a threshold below which more samples help and above which they mostly do not, which moves upward through training.
Condition, and the thing to hold fixed
Growing the batch changes the effective learning rate per sample, so a ramp that does not adjust the schedule alongside it is running a different optimisation problem after the change than before. That interaction is the usual reason a ramp underperforms in practice, and neither this practice nor SOTA-092 mentions it.
Under gradient accumulation the ramp is nearly free — more micro-batches per step rather than more memory (SOTA-031) — which is worth knowing, because it makes the recommendation actionable on fixed hardware rather than only on a bigger cluster.