Source
Kimi Team (2025), LIT-132 — the Kimi K2 report.
Muon with weight decay and RMS-matched updates (SOTA-121) is enough at small scale. At a trillion parameters the K2 team found it drives the maximum attention logit past 1000 early in training, and logits of that size bring loss spikes and occasional divergence. QK-Clip is the addition: after each step, for any head whose maximum logit exceeds a threshold, rescale that head’s query and key projection weights so it does not. The clip acts on weights, so the served model is unchanged, and it fires only where needed. With it, K2 trained on 15.5T tokens without a single loss spike; K3 (LIT-131) keeps the same optimizer at 2.8T.
QK-Clip is first proposed in the K2 report itself, LIT-132, as the fix for the max-logit blow-up it observed with vanilla Muon; the combined optimizer it calls MuonClip (Muon with weight decay, RMS-matched updates and QK-Clip) is what carried the 1T-parameter, 32B-active MoE through 15.5T tokens with no loss spike.
Conditions: the failure this prevents is a large-scale one. LIT-119 trained 90M and 0.6B models with Muon and no clipping and reports stable runs, so the clip is insurance whose premium is a per-head max-logit check per step — cheap, but not free, and unnecessary until the logits say otherwise. The threshold is a hyperparameter; the report’s value is tuned for its model.
Variations
DeepSeek-V4 (LIT-139) pretrains 1.6T and 284B MoE models with Muon and without QK-Clip, and the report says so in as many words: “The attention architecture of DeepSeek-V4 series allows us to directly apply RMSNorm on the attention queries and KV entries, which effectively prevents attention logits from exploding. Consequently, we do not employ the QK-Clip technique in our Muon optimizer.” So the clip is one of two ways to the same invariant, and QK-normalisation is the other — not a reading of secondary coverage but the primary source’s own account, under the heading “Avoiding Exploding Attention Logits”.
The other way now has its own practice and its own source. SOTA-192 files QK-normalisation against LIT-088, which diagnoses the mechanism this practice’s threshold is defending against: logits grow with the weights until the softmax is almost one-hot with near-zero entropy, at which point the gradient vanishes and the run diverges. ViT-22B observed it at ~8B parameters and shows a before/after.
The two are compared_against and nobody has actually compared them. The
open question is whether the clip is only necessary when the normalisation is
absent — DeepSeek-V4’s report is consistent with that and does not establish
it.
Sequence
Muon (LIT-159) → weight decay and RMS matching so AdamW’s hyperparameters transfer (LIT-122, SOTA-121) → QK-Clip so the attention logits stay bounded at scale (this practice) → LIT-131 adds Per-Head Muon on top, orthogonalizing each attention head’s momentum block separately so that heads with larger gradients stop dominating the shared update. Each step keeps the one before.
Mechanism
The failure QK-Clip answers is attention-logit growth, which LIT-155 established as a distinct instability with a normalization remedy, and — the useful part — reproducible in small models at high learning rate rather than only at the scale where it first cost someone a run. That is also why LIT-139 can decline QK-Clip: an RMSNorm on the queries and compressed KV entries bounds the same quantity, so the choice is between normalizing the input to the logit and clipping the weights after the fact.
Known implementations
- Kimi K2, Kimi K3 (QK-Clip); DeepSeek-V4 (Muon with QK-norm, no clip)