anthology-of-the-sota

LIT-462: Spectral Condition for μP under Width-Depth Scaling

2 min read

Properties7
StatusActive
Version2
Tagstraining-optimization
ExtendsLIT-437 — A Spectral Condition for Feature Learning, LIT-150 — Don't be lazy: CompleteP enables compute-efficient deep transformers
Cited as “introduced by” bySOTA-275 — Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule, SOTA-276 — For a normalized or preconditioned optimizer, width-depth muP is width muP plus a hidden residual multiplier of order 1/L
Cited as “paper” byNOTE-211 — Spectral muP under Width-Depth Scaling
Cited as “source” bySOTA-275 — Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule, SOTA-276 — For a normalized or preconditioned optimizer, width-depth muP is width muP plus a hidden residual multiplier of order 1/L, THEORY-037 — Residual-branch depth sets the depth scaling rule, because a branch with two transformations has a second-order update term that a branch with one does not

Graph View

Zheng et al. (2026) — ARXIV-2603.00541

Key takeaways

  • The two depth parameterizations are one family, indexed by residual branch depth. Expanding the one-step feature change of a residual block whose branch holds k transformations gives update terms of order 1 through k. At k = 1 there is only a first-order term and the constraints are looser, giving α = Θ(1/√L) — Depth-muP. At k = 2 the cross term ΔW⁽²⁾ΔW⁽¹⁾, where both branch weights moved in the same step, must also be Θ(1/L), and that extra constraint tightens the multiplier to α = Θ(1/L) — CompleteP. Beyond k = 2 nothing further changes.
  • Every Transformer is the k ≥ 2 case, because attention and FFN blocks hold more than one transformation. The paper’s prediction is therefore that Depth-muP-style scaling should fail to transfer on Transformers, and its depth sweep to L = 256 is where that prediction is tested.
  • The implementation collapses to one extra multiplier. For every modern optimizer they treat except SGD, width-depth muP is width muP plus a hidden residual multiplier α_l = Θ(1/L). Normalization or preconditioning removes the depth factor the raw hidden gradient carries, so the optimizer-specific rule is unchanged. SGD’s update is proportional to the raw gradient and needs a further learning-rate rescaling.
  • Nine optimizers, derived rather than tuned. Muon-Kimi, Muon, Shampoo, SOAP, AdamW, Sophia, Lion, SGD and the Spectral Sphere Optimizer, each mapped from its own update rule to an HP parameterization. Existing width-depth muP formulations come back as special cases.
  • The derivation uses only subadditivity, submultiplicativity and a random matrix norm bound — the paper’s own claim is that it replaces Tensor Programs and dynamical mean-field theory with elementary linear algebra for this purpose, which is a claim about accessibility and is checkable.

Standing in the anthology

The depth half of a condition the record already holds for width. LIT-437 is the spectral condition under width scaling and is declared as the antecedent; LIT-150 is CompleteP, which this recovers as the k ≥ 2 case and corroborates from an independent derivation.

It also bears directly on SOTA-144, whose promote_when asked for an independent group training under CompleteP and reporting depth transfer. What arrived is close to that and not identical to it, and NOTE-211 sets out the difference rather than counting it as met.

Backlinks