Source
Yang and Hu (2022), LIT-148 — Tensor Programs V.
Scale initialisation variances and per-layer learning rates with width the way the Maximal Update Parametrization prescribes, and the optimal learning rate — along with several other hyperparameters — stops moving as the model gets wider. Then tune on a narrow proxy and transfer the values to the full model without tuning it: the paper beats the published BERT-large from a 13M-parameter sweep, and the published GPT-3 6.7B from a 40M-parameter sweep at about 7% of the large model’s pretraining cost.
Both demonstrations are LIT-148’s, and so is the recommendation: the paper proposes µTransfer — parametrize in µP, tune a narrow proxy, copy the values to the full width untuned — as the way to tune a large model. It takes µP itself from Tensor Programs IV; what it adds, and what this practice rests on, is the empirical claim that the optimal hyperparameters, not only the largest stable learning rate, hold still across width, carried to Adam and Transformers.
The optimum transfers as a window, not a point, and the width of that
window has now been measured. LIT-501 sweeps ten
µP-coordinated learning rates over [5×10⁻⁵, 5×10⁻⁴] across four SiT sizes
with ten training seeds per cell. Under a per-cell tuned FID the valleys are
flat-bottomed near 2–3×10⁻⁴ at every size, with the two rates flanking each
minimum sitting inside its seed envelope — three adjacent learning rates share
the per-size best, a 1.7× window. µP transfers the optimum; what it
transfers is a region.
Two riders from the same sweep. Selecting the rate on unguided single-seed
FID gives a monotone curve whose argmin sits at the right edge, 5×10⁻⁴,
which is also where 3 of 10 small-model seeds diverge — a confident-looking
number pointing at the edge of stability. And the seed variance does not
dip at the optimal rate: it is 1.7–2.3% there, inside the general floor, so
the flat region is not a low-variance region. This is one family at 100k
steps and is a measurement of this practice’s resolution, not a challenge to
its claim; SOTA-307 carries the underlying variance result.
Conditions: transfer is across width. Depth, batch size and training
duration are not covered by the original result — LIT-150 (SOTA-144) extends it to depth, and LIT-146 (SOTA-142) handles tokens and batch size by a separate law. The multipliers µP fixes are what Learnable Multipliers
(LIT-121, SOTA-122) proposes to learn instead. In practice the transferred values are a starting point that production recipes then adjust: Falcon-H1 tuned 35 multipliers and Falcon-H1-Tiny carried them to 90M through the forward multipliers with learning rate and weight decay held fixed (LIT-119).
Sequence and variations
µP (this) → u-µP (LIT-149): add Unit Scaling so the sweep becomes
one-dimensional and FP8 works out of the box → CompleteP (SOTA-144): the depth exponent that makes transfer hold across depth as well.
Known implementations
- Falcon-H1, Falcon-H1-Tiny; MiniCPM; PowerLM; Cerebras-GPT