Source
Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740.
Condition
Same argument as SOTA-014, one level down: Adam’s second moment is a running average of squared gradients, so it spans a far wider dynamic range than the weights do, and FP16 cannot hold it. The exponent range runs out before the mantissa does — small squared gradients flush to zero, ε stops doing its job, and the effective step size drifts.
What LIT-011 itself showed is one level up, on the weights, and it is where the FP32 update path this practice extends was first recommended. Its case is that the update, not the forward pass, is where FP16 fails: about 5% of the Mandarin speech model’s weight-gradient values had exponents below −24 and would become zero in the optimizer, and a weight 2048 or more times larger than its update can lose the update in FP16 addition. Updating FP16 weights directly cost 80% relative accuracy on that model, while updating an FP32 master copy matched FP32 training. The paper’s recipe is “single-precision master weights and updates”; it never names Adam’s moments. Carrying the argument to them is this practice’s step, and it is the accounting ZeRO (LIT-027) makes when it counts mixed-precision Adam at 12 bytes of FP32 state per parameter.
Cost, and what has changed since
8 bytes per parameter for Adam’s two moments, on top of the 4 for the master weights. That is the 12 bytes of optimizer state per parameter that dominates the memory of a data-parallel run and that ZeRO (LIT-027) exists to partition — the reason SOTA-028 buys as much as it does is that this practice is being followed.
This is the part of the mixed-precision recipe most actively contested since. Eight-bit optimizer states, and stochastic rounding in place of round-to-nearest, both recover most of the memory while holding accuracy on many workloads. The practice as stated is the conservative default, not a settled bound.
The stochastic-rounding half of that has a large-scale negative result against it. Gopher (LIT-617) trained its 7.1B and 280B models with bfloat16 parameters updated by stochastic rounding, and reports: “We subsequently found that stochastic rounding does not fully recover mixed precision training performance.” Its smaller models used float32 parameters with bfloat16 activations, so the comparison is between the two recipes inside one lineage.
That does not overturn “holds accuracy on many workloads” — it names a workload where it did not, at a scale where the substitution is most tempting, and the finding is one group’s aside rather than an ablation. What it does is put a scale on a sentence that had none. A memory optimization verified at one size is not verified at twenty times that size, and this is the specific case the record can now point at.