Liu et al. (2025) — ARXIV-2502.16982
Key takeaways
- Muon, the orthogonalised-momentum optimizer for matrix parameters, had strong results at small scale and no evidence above it. Two changes carry it: apply weight decay, and scale each parameter’s update so its RMS matches AdamW’s, which keeps the learning rate and weight decay tuned for AdamW transferable without a sweep.
- Scaling-law experiments put Muon at roughly 2× the computational efficiency of AdamW under compute-optimal training. Read that number against LIT-156, which tunes both sides and measures at the end of training rather than mid-run, and gets 1.4× at 0.1B falling to 1.1× at 1.2B. The two are not straightforwardly comparable — different scales, budgets and baselines — but the record should not quote the 2× alone.
- Moonlight, a 3B-active / 16B-total mixture-of-experts model trained on 5.7T tokens, moves the Pareto frontier at that scale with fewer training FLOPs than prior models.
- A distributed implementation that is memory-optimal and communication- efficient is released; the orthogonalisation step is the part that needs care in a sharded setting.
Standing in the anthology
The recipe SOTA-121 adopts, confirmed independently at tiny scale by LIT-119. The optimizer itself was introduced in a 2024 blog post by Jordan and collaborators (LIT-159); this paper is where the production form and the scaling evidence live. LIT-132 adds QK-Clip to the same optimizer at 1T parameters (SOTA-131), and LIT-131 carries it to 2.8T.