anthology-of-the-sota

LIT-149: u-µP: The Unit-Scaled Maximal Update Parametrization

1 min read

Properties5
StatusActive
Tagstraining-optimization, numerics-and-precision
ExtendsLIT-148 — Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer
Cited as “introduced by” bySOTA-159 — Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling
Cited as “source” bySOTA-159 — Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling

Graph View

Blake et al., Graphcore and Cerebras (2024; ICLR 2025) — ARXIV-2407.17465

Key takeaways

  • µP (LIT-148) makes activation scale independent of model size; Unit Scaling makes activations, weights and gradients begin training at a scale of one. Combined, the model is size-independent and well-scaled from the first step.
  • u-µP models reach lower loss than comparable µP models and train in FP8 without loss-scaling tricks, because nothing starts far from the representable range.
  • The hyperparameters become close to independent, so a one-dimensional sweep replaces a joint search; a learning-rate sweep alone gets near the optimum.

Standing in the anthology

A variation on SOTA-143 rather than a replacement for it: the transfer is still across width; what changes is how cheaply the proxy sweep is done and that it comes with FP8 for free. Filed for the chain µP → u-µP → CompleteP (LIT-150).

The practice is SOTA-159, Proposed — a sibling of SOTA-144, which extends the same parent across depth on the same evidence profile. Filing one extension and not the other had left the µP material with a fork where only one branch existed as a document.

Backlinks