What to do
When a batch size stops reaching the target and you are looking for a bigger
one, spend the effort on the optimizer, not on the scaling heuristic. The
specific move LIT-265 supplies: take an Adam-style per-coordinate update u,
then scale each layer’s step by phi(||x_layer||) / ||u_layer||, with phi
clipping the parameter norm into a fixed range so a degenerate layer cannot
blow the ratio up. Two levels of normalization — per coordinate underneath, per
layer on top.
Both levels are load-bearing, and the paper’s own negative result is how you know. LARS is the same layerwise ratio over momentum SGD with no per-coordinate term; it reaches the ResNet-50 target and fails on BERT at every batch size tried. If your model’s layers differ as much as an attention stack’s do, one level is not enough.
What it buys, exactly
On BERT-Large pre-training, with SQuAD v1.1 F1 as the target and a baseline of 90.395 from the public checkpoint:
- AdamW stops at 16K. LIT-265 tried three tuning strategies for it — default hyperparameters, LAMB’s hyperparameters, and a grid search — and past 16K it reaches 88.1 against a 90.4 target. At the 64K/32K mixed-batch schedule, “even after extensive tuning of the hyperparameters, we fail to get any reasonable result”.
- LAMB reaches 32K with F1 at or above the baseline, and the mixed-batch run finishes in 76 minutes on a TPUv3 Pod against three days for the baseline.
- On ImageNet/ResNet-50 at 16K, comprehensively tuned Adagrad, Adam and AdamW reach 55.4%, 66.0% and 67.3% top-1 against a 76.3% target. LAMB reaches it. The instrument is ILSVRC (LIT-423), the thousand-class classification challenge — not the database it is drawn from, and not the downsampled ImageNet-64 that generative work reports FID on.
The baselines being tuned is what makes these numbers mean anything, and it is worth saying plainly because the opposite is the usual failure: an optimizer paper whose baseline inherited its learning rate is measuring the inheritance. LIT-265’s baselines were grid-searched and its own appendix reports the search spaces.
What this is not
It is not permission to skip the retuning in SOTA-218. LIT-265 reports its large-batch results under the heading untuned and says in three places that tuning does better — at 16K, changing the fine-tuning learning rate alone moves F1 from the reported 91.345 to 91.688. The claim is that a square-root rule plus linear-epoch warmup is good enough to hit a fixed target without a sweep, not that the rule is right. SOTA-218 is about drawing a curve you intend to reason from, and a curve drawn this way is still not one. THEORY-012 is the account under both.
LIT-265 agrees with LIT-058 and cites it saying so — “learning rate scaling heuristics with the batch size do not hold across all problems or across all batch sizes” is LIT-265’s own summary of it, in its related-work section, as motivation. The two papers were never in dispute. This record’s reading that they were is the thing being corrected here.
It is not a claim that LAMB is the current answer. The paper is from 2019, its workloads are BERT-Large and ResNet-50, and it predates everything this record holds about modern pretraining optimizers. LIT-152 reports the same shape from a 2025 model report: the architecture and Muon together move the optimal learning rate and batch size upward and make batch-size warmup unnecessary — a different mechanism reaching the same place, and the reason to carry the instruction at the top forward rather than the specific algorithm.
Conditions
The convergence theory covers less than the algorithm. Theorem 1 is proved
for simplified LAMB with beta1 = 0 and lambda = 0 — no momentum, no weight
decay — while every experiment uses both. The L_avg-not-L_inf result that
makes the mechanism legible is a statement about the simplified version.
Scaling efficiency is not throughput. At 64x hardware it falls to ~76.7% for BERT’s 300M parameters, on gradient communication. The 76-minute figure is a TPUv3 Pod with that interconnect; a GPU cluster on commodity Ethernet is a different machine, and what the record recommends there is SOTA-215 — freeze Adam’s variance and compress the momentum — which is an answer to the communication cost rather than to the batch size, and composes with this one rather than replacing it.
Mixed-batch training is a separate moving part. The 76-minute run changes batch size and sequence length together between phases and re-warms the optimizer at the transition. That re-warmup is tuned, and decreasing the batch between phases is reported as an instability risk.
Known implementations
None recorded. The BERT and ResNet-50 runs above are LIT-265’s own experiments
and belong to the evidence, not to adoption — counting them here would be
DP-005 exactly. That layerwise adaptation is in every framework is
recorded in consensus_note, where counting is the right operation.