SOTA-276: For a normalized or preconditioned optimizer, width-depth muP is width muP plus a hidden residual multiplier of order 1/L

Source

Zheng et al. (2026), LIT-462 — ARXIV-2603.00541, read as NOTE-211, Takeaway 2. Accounted for by THEORY-037.

What to do

If you already have a width-scaling muP setup and now want to scale depth as well, and your optimizer normalizes or preconditions its update — Muon, Muon-Kimi, Shampoo, SOAP, AdamW, Sophia, Lion — then:

1. Keep your width-muP initialization and learning-rate rules exactly as they are.
2. Set the hidden residual multiplier α_l = Θ(1/L).
3. Change nothing else.

SGD is the exception and needs a further step. Its update is proportional to the raw gradient, which carries the residual multiplier, so after setting α_l the hidden learning rate needs an additional α_l-dependent rescaling.

Why the depth correction collapses to one number

A hidden layer’s raw gradient inherits the residual multiplier α_l as a factor. An optimizer that normalizes or preconditions its update divides that factor out, so the norm of the update it produces does not depend on α_l at all. The spectral condition constrains that norm — which means the optimizer-specific part of the rule is the same as it was under width-only scaling, and the only thing depth changes is the multiplier itself. SGD does not divide anything out, so for SGD the factor survives into the update and has to be compensated.

The practical consequence is that adopting depth scaling is a one-line change for most of the optimizers the record recommends, rather than a re-derivation.

The simplification is LIT-462’s Takeaway 2, and this recommendation starts there: having derived its width-depth condition per optimizer for nine of them, the paper observes that for every one except SGD the result is width-scaling muP plus α_l = Θ(1/L), and gives SGD its extra learning-rate rescaling. Its depth sweeps to 256 layers on GPT-2-style models (Muon-Kimi-AdamW, Muon-AdamW, Shampoo-AdamW, Sophia) are run with the rule implemented this way and show the optimal learning rate holding across depth.

Conditions

• Proposed. The rule is derived for nine named optimizers and four were run. It is stated as a property of a class — “normalized or preconditioned” — and the evidence for the class is the nine instances that produced it. Whether some other preconditioned optimizer has an update norm that does depend on α_l is a thing to check, not a thing the paper rules out.
• SGD is genuinely different, and an optimizer that is partly raw-gradient should be treated as SGD-like rather than assumed to simplify.
• Hybrid setups need care. The experiments run Muon on hidden matrices and AdamW on embeddings, LM head and biases; both are in the simplifying class, so the hybrid inherits the simplification. A hybrid with an SGD-like component would not.
• 300M tokens per run, GPT-2-style models, widths to 4096 and depths to 256.
• The experiments implement the simplification rather than testing it. No run compares the simplified rule against a non-simplified derivation of the same condition, because for these optimizers the two coincide by the argument. The check is the argument.

Relation to the neighbours

SOTA-143 is the width-muP setup this takes as given. SOTA-275 is what fixes Θ(1/L) rather than Θ(1/√L) as the multiplier to add. The record recommends Muon (SOTA-121) and matrix preconditioning (SOTA-165) and has never said how to muP either; this is the answer, and it is short because of what preconditioning does.

Known implementations

• None reported. The paper’s own runs implement it; no production report in the record trains this way.