Yang and Hu, Microsoft and OpenAI (2022) — ARXIV-2203.03466
Key takeaways
- Under the standard parametrization, the optimal learning rate shifts as a network gets wider, so hyperparameters found on a small model are wrong for a large one. Under the Maximal Update Parametrization (µP) — a specific scaling of initialisation variances and per-layer learning rates with width — many optimal hyperparameters are stable across width.
- µTransfer: parametrize the target model in µP, tune on a narrow proxy, and transfer the values to the full width without tuning it directly.
- Demonstrations: transferring from a 13M-parameter model beats the published BERT-large (350M) at a total tuning cost of one BERT-large pretraining; transferring from 40M beats the published GPT-3 6.7B with tuning that cost about 7% of that model’s pretraining.
- µP itself is not introduced here. It comes from Tensor Programs IV
(LIT-548), which proved that the standard parametrization stays stable
only at an
O(1/width)learning rate and defined µP for MLPs trained with SGD. This paper takes that parametrization, carries it to Adam and Transformers, and adds the empirical claim TP-IV did not make: that the optimal hyperparameters, not just the maximum stable learning rate, hold still across width. - Transfer is across width; depth, batch size and training length were not covered, which is what LIT-149 and LIT-150 take up.
Standing in the anthology
The source of SOTA-143 and the parametrization behind Falcon-H1’s 35 tuned multipliers (LIT-120), MiniCPM’s wind tunnel (LIT-144) and the Power scheduler’s size transfer (LIT-146). Learnable Multipliers (LIT-121) claims to generalise it by learning the multipliers it fixes.