Kimi Team, Moonshot AI (2025) — ARXIV-2507.20534
Key takeaways
- Vanilla Muon at this scale drives the maximum attention logit past 1000 within a short run, and logits of that size bring loss spikes and occasional divergence. The token efficiency of Muon (LIT-122) was worth keeping; the instability was not.
- QK-Clip: after each optimizer step, if a head’s maximum attention logit exceeds a threshold, rescale that head’s query and key weights so it does not. It acts on the weights rather than on the activations, so inference is unchanged, and it only fires where the logits demand it.
- MuonClip — Muon with weight decay, RMS-matched updates and QK-Clip — carried a 1T/32B mixture-of-experts model through 15.5T tokens with no loss spike.
- Post-training is multi-stage: a large-scale pipeline synthesising tool-use and agentic trajectories, then a joint reinforcement-learning stage over real and synthetic environments. State of the art among open non-thinking models at release, strongest on agentic tasks.
Standing in the anthology
The source of SOTA-131 and the middle link of the Muon chain: LIT-122 made Muon transfer AdamW’s hyperparameters; this made it survive a trillion parameters; LIT-131 carries the same optimizer to 2.8T. The agentic post-training half is not drawn into a practice.