Source
Marek et al. (2025), LIT-444 — ARXIV-2507.07101.
The rule
Express the decay as a half-life in tokens:
t_half = B · ln(2) / ln(1/beta)
Changing the batch size from B to B', hold t_2 fixed:
beta_2' = beta_2^(B'/B)
Leave beta_1 alone — its usual default works across batch sizes. It is
specifically the second moment that has to move.
Why the default breaks
beta_2 is a rate per optimizer step, and an optimizer step is not a fixed
quantity of data. Shrinking the batch by 512× while holding beta_2 = 0.95
shortens the window over which the second moment — the variance estimate the
update is divided by — is computed, by the same factor. The estimate is then
made from far too little data and the updates become erratic.
This is the whole of “small batches are unstable”. The paper reproduces a
previously published result showing small batches underperforming with fixed
beta_2, applies the rule, and the deficit disappears with no other tuning.
The rule originates in LIT-444. Marek et al. swept learning rate, beta_1,
beta_2 and t_2 independently across batch sizes from 1 to 1024 on a 30M
model and compared holding beta_2 fixed with holding t_2 fixed: only the
second kept training stable down to batch size one, and beta_1’s usual
default worked at every batch size.
Why it is worth filing separately from the batch-size advice
Because it is true independently of whether you take SOTA-260’s advice about which batch size to use. Anyone who changes batch size for any reason — a new cluster, a memory constraint, a fine-tuning run at a different scale — is silently changing the averaging window unless they apply it. That includes every transfer of a recipe from a paper to a different machine.
And it is the safety condition on the other practice: small batches without this rule are the instability, not a demonstration of it.
A constant with no unit attached
The general form of the finding is worth more than the rule. beta_2 = 0.95
is a number the field copies with an implicit denominator — a step — that
varies between every two runs that compare it. Naming the denominator makes
the constant transferable.
SOTA-261 is the same move on a different hyperparameter: weight decay only means something against a learning rate and a step count, and the AdamW timescale is what naming that denominator produces. Two papers, two hyperparameters, one diagnosis — the numbers everyone copies are ratios whose denominators were left out of the recipe.
Conditions, and why this is Proposed
One group. The sweeps isolating the mechanism are at 30M parameters; the scale checks are 124M and one 1.3B run.
The authors say they do not know why it transfers across model scales. That is honest and it is a real limitation: the rule is an empirical regularity with a plausible story, not a derivation, and a regularity of this shape usually has an account waiting.
Untested against a batch-size schedule, which the authors name as an open question — and the record recommends exactly such a ramp at SOTA-093, so that gap is directly load-bearing here.
The square-root learning-rate rule is separately reported to overshoot badly across wide batch ratios (32× prescribed against about 3× measured, from 1 to 1024), which is worth knowing alongside this but is not part of this recommendation.
Known implementations
- None. No framework in the record exposes a token half-life; they all expose
beta_2.