Source
Zheng et al. (2026), LIT-462 — read as NOTE-211.
What it explains
| practice | what it says to do | what this says it is |
|---|---|---|
| SOTA-144 | use CompleteP to transfer across depth | the k ≥ 2 member of one family, not a rival parameterization |
| SOTA-275 | pick the rule from the residual branch | the family is indexed by branch depth, and that is the whole index |
| SOTA-276 | add one 1/L multiplier to width muP | preconditioning removes the depth factor, so nothing else moves |
The account
Write the change in a residual block’s output after one step. For a branch
holding a single weight, every term has exactly one ΔW in it. For a branch
holding two, there is also the term where both moved: the product
ΔW⁽²⁾ΔW⁽¹⁾.
In ordinary analysis that cross term is second order and negligible. Under
muP it is not, and the reason is what muP is for. The maximal-update
principle deliberately makes each update as large as stability permits, so
ΔW is not small — it is exactly as large as the constraint allows. That
principle is THEORY-076’s account of width, where µP is the
parametrization under which every layer’s update contributes Θ(1); this
account takes the same bookkeeping and asks what it demands along depth. A
product of two such terms is therefore the same order as the terms
themselves, and it needs its own constraint.
Adding that constraint is the entire difference between the two published
depth parameterizations. Without it the residual multiplier can be
Θ(1/√L), which is Depth-muP. With it the multiplier must be Θ(1/L),
which is CompleteP. Carry the expansion to a branch of depth k and all
orders up to k get the same Θ(1/L) constraint — and the resulting
parameterization stops changing after k = 2, so there is no third regime
to discover.
The consequence for practice is structural rather than empirical. An architecture does not get to choose which depth rule it wants; the number of transformations in its residual branch chooses. Every Transformer has more than one, so every Transformer is in the tighter regime.
What was actually shown
Both the derivation and the test are LIT-462’s: its Condition 3.1 adds the second-order constraint, and its Figure 2 sweeps depth from 4 to 256 under Muon-Kimi-AdamW and Muon-AdamW, with Shampoo-AdamW and Sophia in the appendix.
The derivation is the argument, and the experiment is a prediction test
rather than a demonstration. The theory says the k = 1 rule should fail to
transfer on architectures whose branches hold more than one transformation.
Figure 2 sweeps depth from 4 to 256 on GPT-2-style models under both rules:
the k = 1 optimum drifts with depth, the k ≥ 2 optimum does not. It
could have come out with both transferring, or with neither, and it did not.
What this does not say
It does not say the two rules differ much at the depths people train today. The paper’s own §5.4 is the check that matters here: for Muon-Kimi-AdamW even standard parameterization appears to transfer across depth reasonably well, and the authors attribute it to moderate depths and to LayerNorm and QKNorm masking the pathology. Remove LayerNorm and SP breaks. That is a finding about how much modern normalization is already absorbing, and it means the practical margin at realistic depth is smaller than the asymptotic argument suggests.
It does not establish CompleteP at scale. 300M tokens per run. The
account explains why a parameterization is right; how much it is worth is a
separate question that these runs cannot answer, which is why
SOTA-144 stays Proposed.
It is not a duality claim and does not need one. THEORY-024 holds
that muP is a partial approximation of a duality map. This derives muP —
including for Shampoo and Muon, the other objects in that account — from norm
bookkeeping alone, with no dual space anywhere. The two frames describe the
same objects and neither requires the other, which is worth recording now
that the duality frame is itself Proposed.
And the scale estimates are estimates. The derivation tracks typical magnitudes using subadditivity and submultiplicativity, and their tightness rests on non-cancellation and alignment behaviour that the paper argues for rather than proves.
Why Active
The claim is a decomposition, it is checkable by hand, and the one place it
makes a falsifiable prediction about training — that k = 1 scaling fails on
multi-transformation branches — was tested against the alternative and held
across four optimizers. It also unifies two published results the record
carried separately without saying how they relate.