Source
Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361.
McCandlish et al. (2018), LIT-017 — ARXIV-1812.06162.
The sub-linearity is LIT-028’s. Its compute-optimal allocation (Eq. 1.7)
grows the model as C^0.73 and the batch as C^0.24, and its Figure 3
draws what that means over a billion-fold increase in compute: more than
1,000,000× in model size, about 100× in batch, under 10× in serial steps.
The same paper measured critical batch size against loss at 3M and 85M
parameters (Figure 10) and found it depends on the loss, not directly on
model size — and Figure 3’s own caption says the batch increase is drawn
from the increase in data. The model-size reading is a projection along that
frontier, which is the reading SOTA-258 later took apart.
LIT-017 is the mechanism. The gradient noise scale that sets the useful batch rises as the loss falls, and on LSTM language models of several sizes it was roughly independent of model size at a fixed loss; larger models reach larger noise scales only because they reach lower loss.
Origin and evidence are the same paper here: LIT-028 does not write batch as a function of model size, but its allocation is where the pairing first appears, and GPT-3 credits it (with McCandlish) for “larger models can typically use a larger batch size”. This practice previously credited Artetxe et al. (LIT-061), which only sets batch size by model size “following Brown et al. (2020)”. That setting is GPT-3’s Table 2.1 (LIT-035): from 125M to 175B parameters — about 1,400× — the batch went from 0.5M to 3.2M tokens, about 6×, chosen with the gradient noise scale measured during training and justified by citing Kaplan and McCandlish.
Sub-linearly, and why the exponent matters more than the direction
Larger models tolerate — and want — larger batches, because the gradient noise scale that sets the useful batch size rises as the model reaches a lower loss. That the growth is sub-linear is the content: doubling parameters does not double the batch, so the ratio of batch to model size falls as scale rises, and a rule that scales them together over-shoots at the top end.
The practical failure is quiet. An over-large batch does not diverge; it spends compute on samples that buy less than they cost, and the run simply reaches a given loss later than it should have — attributed, usually, to the data or the schedule.
What the record has that is better
This is a heuristic read off a 2020 allocation fit and the GPT-3 table that followed it, and the quantity it gropes toward — critical batch size — is measurable and has since been characterised directly. The record’s own material is stronger: SOTA-092 and SOTA-093 give the mechanism, and the ramp implied by them is what large runs actually do.
Kept because it is true and because a rule of thumb is useful when nobody is going to measure. But a reader with a measurement should prefer it, and the practice should not be read as licensing a fixed batch-to-parameter ratio, which is the reading its title most invites.
The alternative the record held beside it, SOTA-061, does not use model size at all: it raises the batch while sample efficiency stays above 80% of its small-batch value and stops there. That rule asks for a measurement, where this one is a proxy for it. The proxy has since been shown to track the wrong variable (SOTA-258). No paper compared the two rules directly.