Yang et al. (2023) — ARXIV-2310.17813
Key takeaways
- The condition is on the spectral norm, of the weight matrices and of
their updates, scaled like
sqrt(fan-out/fan-in). That is the whole claim, and the paper states it against the alternatives explicitly: the widely used scalings based on Frobenius norm and entry size are heuristic - Feature learning is the thing being preserved — internal representations evolving nontrivially at all widths, rather than the network degenerating into a kernel regime as width grows
- muP falls out of it. The paper offers an “elementary derivation of maximal update parametrization” from the spectral condition, which reframes a result the record already recommends as a corollary of something simpler
- Written to be understood, by its own statement of intent — the text is structured to give a conceptual account of scaling behaviour rather than only a result
Standing in the anthology
The first step of the line under the record’s optimizer cluster. The record recommends muP in SOTA-143, sourced to LIT-148 (Tensor Programs V), and holds no account of why the parametrization is the right one. This paper derives it from a condition on the spectral norm, which is the same quantity the Muon line acts on.
Its target is LIT-548’s. µP, defined in Tensor Programs IV through abc-parametrizations, is what the spectral condition recovers, and the appendix borrows that paper’s abc framework to prove µP is the unique stable abc-parametrization meeting the condition at large width. What it changes is the variable: it works in the spectral norms of weights and updates rather than in per-layer initialization scales and learning rates, which it says makes the same size calculations simpler.
It corrects LIT-686, which shares an author: that paper’s deep-relative-trust bound assumes every weight and its perturbation is well-conditioned and measures relative change in Frobenius norm. This paper names that conditioning assumption as the flaw — weights have high stable rank and updates low — says it gave the wrong scaling relation with width, and replaces Frobenius with spectral.
Filed from #180, where it sits in the top fourteen of the reading
feed by distinct days revisited. extended_by the modular-norm paper.
Unread — no NOTE.