SOTA-275: Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule

Source

Zheng et al. (2026), LIT-462 — ARXIV-2603.00541, read as NOTE-211. Accounted for by THEORY-037.

What to do

Count the linear transformations inside one residual branch of your architecture. That number, called k, picks the depth rule:

k	residual multiplier	published name
1	α_l = Θ(1/√L)	Depth-muP
≥ 2	α_l = Θ(1/L)	CompleteP

A Transformer’s attention and FFN branches each hold more than one transformation, so a Transformer is always the second row. Nothing changes again past k = 2, so there is no further case to work out for a deeper branch.

This is not a preference between two competing proposals. The two rules are the same condition evaluated at different branch depths, and applying the k = 1 rule to a k ≥ 2 architecture is a mismatch rather than a conservative choice: in a depth sweep from 4 to 256 layers the optimal base learning rate drifts under k = 1 and holds under k ≥ 2, for Muon-Kimi-AdamW, Muon-AdamW, Shampoo-AdamW and Sophia alike.

The rule originates in LIT-462, which derives one muP spectral condition for joint width-depth scaling, gets Depth-muP and CompleteP back as its k = 1 and k ≥ 2 cases, and so makes branch depth the thing that picks the rule. The sweep above is its head-to-head (Figure 2, GPT-2-style models on OpenWebText at 300M tokens per run): the k = 1 condition shifts the optimal learning rate as depth grows and the k ≥ 2 condition does not, which is the evidence for the second row being the Transformer’s.

Why it is a structural question and not an empirical one

Under muP each update is deliberately as large as stability allows, so when a branch holds two weights the term in which both moved in one step is the same order as the terms in which one did. Constraining that cross term is what tightens the multiplier. A branch with one weight has no such term, so its constraint set is genuinely looser. THEORY-037 holds the derivation.

Conditions

• Proposed, and at 300M tokens per run. The depth sweep is deep and short. Nothing here speaks to whether the choice still pays over a realistic token budget.
• The margin at realistic depth is smaller than the theory suggests, and the authors say so. In their §5.4, standard parameterization appears to transfer across depth for Muon-Kimi-AdamW; they attribute it to moderate depths and to LayerNorm and QKNorm masking the scaling pathology, and removing LayerNorm makes SP break. Modern normalization is already absorbing some of what the parameterization is for.
• Only one side of the prediction is tested. The claim is that branch depth is the index. What is shown is k ≥ 2 beating k = 1 on Transformers. Nobody has shown k = 1 beating k ≥ 2 on a one-transformation architecture, which is what would make this a rule rather than a preference for the tighter option.
• Branch depth must be well defined. An architecture with parallel attention and FFN, or with different branch depths at different blocks, is not covered.
• The derivation is a deep linear residual MLP, one step, one example, generalized under assumptions stated in an appendix.

Relation to the neighbours

SOTA-144 says to use CompleteP so one sweep serves deeper models. This says which architectures need it and why, and that the alternative fails on Transformers specifically — which SOTA-144 does not claim, because LIT-150 compared CompleteP against muP rather than against Depth-muP. Declared as extends.