anthology-of-the-sota

SOTA-159: Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling

2 min read

Properties6
StatusProposed
Consensusunreplicated
Tagstraining-optimization, numerics-and-precision
SourceLIT-149 — u-µP: The Unit-Scaled Maximal Update Parametrization
Introduced byLIT-149 — u-µP: The Unit-Scaled Maximal Update Parametrization
ExtendsSOTA-143 — Parameterize the model with µP and tune hyperparameters on a narrow proxy, transferring them across width

Graph View

Source

Blake et al., Graphcore and Cerebras (2024; ICLR 2025), LIT-149 — ARXIV-2407.17465.

SOTA-143 gets the transfer: under µP the optimal learning rate stops moving with width, so you tune small and transfer. What it does not get is the scale the tensors start at — µP makes activation scale independent of model size without saying what that size-independent scale should be.

Unit Scaling answers that: activations, weights and gradients all begin training at a scale of one. Combined, the model is size-independent and well-scaled from the first step, and two things follow that neither gives alone.

The combination, named u-µP, is LIT-149’s, and so is every result below: Blake et al. propose it as the parametrization to train under and report it reaching lower loss than comparable µP models, training in FP8 without loss scaling, and getting near the optimum from a learning-rate sweep alone.

What the combination buys

  • The sweep collapses. The hyperparameters become close to independent, so a one-dimensional sweep replaces a joint search — a learning-rate sweep alone gets near the optimum. That is a cost reduction on top of the cost reduction SOTA-143 already claims.
  • FP8 without loss-scaling tricks, because nothing starts far from the representable range. Low-precision stability falls out of the parametrization rather than being bolted on.
  • u-µP models reach lower loss than comparable µP models.

Conditions, and why this is Proposed

One paper, one collaboration, and nothing in the record trains under it.

Worth being explicit about the shape, because it is the reason this is filed at all: SOTA-144 is the other one-group extension of SOTA-143 — CompleteP, which extends the transfer across depth — and it carries the same evidence profile, the same status and the same consensus value. Filing one and not the other left the µP material with a fork where only one branch existed as a document. The two are siblings, not rivals: one extends the transfer across depth, this one improves what the transfer costs and what it survives.

Known implementations

  • None in the record.

Backlinks