Smith et al. (2022) — ARXIV-2201.11990
Key takeaways
- 3D parallelism: 8-way tensor × 35-way pipeline, data-parallel over the rest, on 560 DGX A100 nodes. 530B parameters — 105 layers, hidden 20480, 128 heads, sequence 2048
- Why all three are needed: the batch size a model can train with has an
upper bound, so at ~4000 GPUs pure data parallelism “would only allow for a
batch size of 1 per GPU”. That ceiling is
LIT-017’s critical batch size — the reason model parallelism exists is statistical, not architectural - A complete published recipe, at a detail level frontier reports abandoned
soon after: LR 5.0e-5, 1B tokens linear warmup, cosine to 10% over 340B;
batch ramped 32 → 1920 in steps of 32 over the first 12B tokens; Adam
β₁ = 0.9, β₂ = 0.95, ε = 1e-8; clip 1.0; weight decay 0.1; init
N(0, σ²)with σ ≈ √(1/(3H)) - Three stability findings: β₂ was lowered from the standard 0.99 specifically “to reduce spikes in the training loss”; higher-variance initialisation fails to converge; and the learning rate was projected from a model-size plot rather than swept, because higher rates destabilise
- Per-GPU throughput falls as the cluster grows — 126 / 121 / 113 teraFLOP/s at 280 / 350 / 420 nodes against a 312 peak. And 270B tokens for 530B parameters, about 0.5 tokens per parameter
Standing in the anthology
Read — NOTE-033. Nothing is sourced to this paper and the reading files no practice.
Two things are worth carrying. Lowering β₂ to suppress loss spikes is a
stability lever the record names nowhere, reported here as an operational
finding at 530B; unablated, which is why no practice is filed, but the
stability neighbourhood (SOTA-035, SOTA-052) has a gap where it belongs.
And the LIT-017 connection is the more valuable one: the record carries
tensor and pipeline parallelism as systems techniques with systems
justifications, when the cause of the whole arrangement is a statistical
property of the gradient. The 32→1920 batch ramp is likewise exactly what
LIT-017 predicts from a noise scale that grows as the loss falls. Neither
paper cites the other.
At 0.5 tokens per parameter against Chinchilla’s ~20, this is also the
corpus’s clearest artefact of the era LIT-040 derived — a 530B model trained
on fewer tokens than a 7B model gets today. The recipe outlived the model.