Keller Jordan, Yuchen Jin, Vlado Boza, You Jiacheng, Franz Cesista, Laker Newhouse and Jeremy Bernstein (2024) — kellerjordan.github.io/posts/muon
Key takeaways
- The rule: for a hidden weight matrix, take the SGD-momentum update and replace it with its nearest orthogonal matrix, approximated by a handful of Newton–Schulz iterations, then step. Everything that is not a hidden matrix — embeddings, the classifier head, gains and biases — stays on AdamW, and the scoping is part of the method rather than an implementation detail.
- The claim it was published on is sample efficiency at small wall-clock cost: substantially fewer tokens to a target, with the orthogonalisation adding only a few percent per step because Newton–Schulz is a short sequence of matrix multiplies.
- It is a speedrunning artefact — developed against NanoGPT and CIFAR wall-clock records — which is why it arrived without scaling evidence and why the papers that followed are all about whether it survives scale.
Standing in the anthology
The origin of the record’s optimizer chain, and until now the one link in it the corpus did not hold. LIT-122, SOTA-121 and SOTA-131 each name it as “a 2024 blog post by Jordan and collaborators, which is not in the corpus”; it is now.
Filed with a url:, the third such source after LIT-119 and LIT-051, and the
clearest case yet for ADR-009 — three laboratories pretrain frontier models
with this and there is no paper to cite. The chain now reads: this →
production form and scaling evidence (LIT-122) → QK-Clip at a trillion
parameters (LIT-132) → per-head orthogonalisation at 2.8T (LIT-131), with
LIT-156 as the outside comparison and LIT-153 as the diagnosis of
where its advantage goes at scale.