Source
Marek et al. (2025), LIT-444 — ARXIV-2507.07101.
The recommendation, and the one it replaces
Standard practice is to use the largest batch that fits, and to simulate a larger one with gradient accumulation when it does not. This says the target is the smallest batch that still maximizes throughput — in practice at least a few hundred tokens per device, so memory bandwidth does not become the bottleneck, and raised again if a second-order optimizer’s per-step overhead starts to cost more than the step buys.
The recommendation is LIT-444’s. With the half-life rule applied, its batch-1 runs held their own against batch 512: on GPT-2 124M, Adam and Adafactor matched AdamW after per-optimizer learning-rate tuning, and on a 1.3B GPT-3 configuration vanilla SGD with no momentum matched Brown et al.’s AdamW baseline while Adam and Adafactor beat it.
Gradient accumulation is then the wrong tool almost everywhere: it increases memory, because the accumulated gradient must be stored, and it trades away optimizer steps for a batch size the paper argues you did not want. The stated exception is multiple device replicas bottlenecked on interconnect bandwidth, where the larger effective batch is buying communication savings rather than optimization.
This is only safe with the half-life rule
SOTA-259 is not an optional companion. The entire reputation of small
batches for instability comes from holding beta_2 fixed while shrinking the
batch, which shortens the second moment’s averaging window in tokens by the
same factor. A reader who adopts the small batch without rescaling beta_2
gets precisely the instability this practice is explaining away.
What it buys beyond parity
- Robustness. At batch size 1 the loss surface over learning rate and the decay rates is markedly flatter than at 512. Less tuning is needed, which for most practitioners is worth more than the last few thousandths of a nat.
- Optimizer choice stops mattering. SGD, Adafactor, Adam and Muon reach similar loss at batch size 1 on a 30M model; the spread opens up as the batch grows. Vanilla SGD with no momentum and no optimizer state matched the GPT-3 AdamW configuration on a 1.3B model.
- Memory, both from the batch itself and from not storing an accumulated gradient — and, where SGD becomes viable, from carrying no optimizer state at all.
Where it sits against the ceiling
SOTA-258 establishes that critical batch size grows with the token budget. That is a ceiling above which parallelism stops paying; this is an argument about where to sit below it, and the two are compatible. Together they give a range with a reason at each end, which the record did not have: every other batch-size practice it holds — SOTA-097, SOTA-062, SOTA-093, SOTA-198 — assumes bigger is what you want and asks only how much you can afford.
The uncomfortable implication
If optimizer sophistication buys little at small batch, then part of what the record’s optimizer practices recommend is robustness at a batch size this practice says not to use. That is not a refutation of any of them, and it is a question none of them answers.
Conditions, and why this is Proposed
Batch size has to be a free choice, and often it is not. In sharded training the global batch is largely set by the parallelism topology, and nothing here addresses that regime. The recommendation is for the case where the model fits on a device.
The optimizer sweeps are at 30M parameters. The scale checks are 124M and a single 1.3B run trained at about 10 tokens per parameter — under-trained relative to Chinchilla — against an untuned baseline configuration. The case against gradient accumulation follows from the other results plus a memory argument rather than from a matched-conditions measurement.
Weight decay is switched off rather than rescaled in the batch-size-1 configurations, which leaves its interaction with the half-life rule unexamined — and SOTA-261 says weight decay is the hyperparameter that should be moving.
Known implementations
- None. Every pretraining recipe in this record uses a large batch, and most accumulate.