LIT-131: Kimi K3: Open Frontier Intelligence

Kimi Team, Moonshot AI (2026) — ARXIV-2607.24653

Key takeaways

• 2.8T total and 104B activated parameters, 16 of 896 routed experts per token under what the report calls Stable LatentMoE, native visual understanding, and a one-million-token context window. Weights released alongside the report.
• The attention stack is a layerwise hybrid: 69 Kimi Delta Attention layers (LIT-133) and 24 gated Multi-Head Latent Attention layers — about the 3:1 ratio Kimi Linear settled on — so most layers carry a finite-state linear-attention memory and a minority keep global attention with a compressed KV cache.
• Attention Residuals (LIT-134) replace the fixed residual sum with learned attention over all preceding layers’ representations.
• Stable LatentMoE is three changes, filed as answers to two failure modes that 56× sparsity brings on. LatentMoE separates full model width from routed-expert width, so 896 experts with 16 active is affordable — but the routed path then composes down-projection, a gated expert FFN and up-projection into nearly four consecutive matmuls, which at 2.8T explodes the routed branch’s activations, and a fixed-step balancing bias stops behaving at ~900 experts.
  • Normalized LatentMoE: an RMSNorm between expert aggregation and the up-projection, so the routed branch’s scale stops varying with which experts were selected. Reported to improve validation loss and downstream benchmarks, not only stability.
  • SiTU-GLU replaces SwiGLU (SOTA-034). Both of SwiGLU’s multiplicative factors are unbounded, so coincident large coordinates give activation outliers and overflow risk in low precision; the plain sigmoid gate is bounded but loses Swish’s near-linear positive regime. SiTU-GLU applies a scaled tanh soft cap to the linear factor of the Swish gate and, independently, to the up branch — approximately linear near the origin, bounded far from it. K3 uses different cap constants for the two branches.
  • Quantile Balancing, below.

• Quantile Balancing keeps auxiliary-loss-free routing (SOTA-148) and replaces its update rule. The original adds a fixed step to each expert’s bias, whose size “trades off slow adaptation against load oscillation”; with 896 experts per layer that trade stops being acceptable. QB instead sets each expert’s bias directly from the router-score quantile that matches its target load, derived from a single forward pass by taking Top-(k+1) so the (k+1)-th entry is the cutoff an expert must beat. The quantile spans a global batch of millions of margins, so it is estimated from a per-expert histogram — one all-reduce over bin counts, exact up to bin width regardless of sharding, a few hundred bins per expert. As in the original, the update takes effect only on the next step, so no batch is routed with a bias derived from itself; the bias is frozen at inference.
• Table 1’s comparison with K2 is the report’s one quantitative account of what changed: 61 layers to 93, 1.04T to 2.78T total and 32.6B to 104.2B active, hidden dimension unchanged at 7,168, routed experts 384 to 896 with 8 to 16 active and 1 to 2 shared, attention heads 64 to 96, vocabulary unchanged at 160K, training context 128K to 1M, MLA to hybrid KDA–MLA, SwiGLU to SiTU-GLU. The vision tower is new: 401M parameters, 27 layers, patch size 14, 12 heads.
• The headline claim is about the rate rather than the size: roughly 2.5× the scaling efficiency of Kimi K2 (LIT-132) at matched compute, attributed to the three architectural changes together with refined training and data recipes.
• Optimization continues the Muon line: Muon on the matrix parameters together with what the report calls the weight-clipping mechanism introduced in Kimi K2 — QK-Clip (SOTA-131) under a different name; the term MuonClip does not appear anywhere in the report — and, for the attention projections, Per-Head Muon, which the report gives its own section (§2.5). Instead of applying Newton-Schulz orthogonalization to the whole Q, K and V projection matrices, their momentum matrices are partitioned along the head dimension and each head’s block orthogonalised separately. The stated intuition: full-matrix orthogonalization treats all heads as one coupled block, so heads with larger gradient or momentum scales dominate the shared update direction while smaller-scale heads receive insufficiently normalized updates. Claimed effects are more balanced learning dynamics across heads and better stability at larger scales, plus a small saving — Newton-Schulz on tall per-head blocks is cheaper than on the full matrix. No ablation is given for it.
• Long context is reached by extending the window progressively during training rather than at once (SOTA-139), and by synthesising tasks that can only be solved by attending across the whole million-token window.
• The scaling-law study prefers cosine decay to warmup-stable-decay, and the methodological half is the interesting one. The team argues that comparing the two under one hyperparameter set is unfair, because their optimal peak learning rates and batch sizes differ substantially even at the same model size and token budget, so a shared setting silently favours whichever schedule it happens to suit. They therefore run an independent scaling-law search per schedule, at a fixed minimum learning rate; under each schedule’s own optimum, cosine decay consistently reaches a lower final loss, and K3 adopts it. This is the opposite conclusion to the one SOTA-140 records, reached by the comparison design that practice’s own evidence (LIT-145) argues for.
• A multi-token-prediction layer shaped like a backbone block is trained with the model and later fine-tuned into an EAGLE-3-style single-layer draft model for speculative decoding — LIT-163 and LIT-185 are the two halves that arrangement is assembled from.
• No positional encoding at all on the MLA layers. Unlike K2 and K2.5, K3 follows Kimi Linear in applying NoPE to every global-attention layer, and lets the intervening KDA layers supply position sensitivity and recency. The consequence the report names is the one worth carrying: extending the context no longer means retuning a RoPE frequency base or applying YaRN, because there is no positional parameter in the global layers to retune.
• Gated MLA adds “an input-dependent, channel-wise full-rank output gate” — SOTA-134’s gate at a finer granularity than the head-specific one its source ablates.
• The RL infrastructure is reported at a scale worth recording: 51,219,741 sandboxes across 1,505,678 images created over training and evaluation.

Standing in the anthology

The current end of two chains the record now holds: the Muon line (Muon → weight decay and RMS matching in LIT-122 → QK-Clip in LIT-132 → here) and the hybrid-attention line (Falcon-H1’s parallel heads in LIT-120; Kimi Linear’s interleaved layers in LIT-133 → here). No practice is sourced to it directly — the practices it instantiates (SOTA-131, SOTA-132, SOTA-133) are sourced to the papers that introduced each change.

What this note is read from, and what the report does not contain. Every claim above is checked against the report’s full text. An earlier version of this section said its ablation tables were still unmined; that was a guess, and it was wrong. The report has five tables — an architectural comparison with K2, two benchmark tables, an in-house web-development comparison, and a table of third-party evaluations — and no component ablations at all. The headline 2.5× figure is read off fitted scaling-law curves on held-out out-of-distribution validation data, and the report attributes it to the architectural, data and training changes collectively. So nothing here apportions the gain between Attention Residuals, the hybrid attention, the LatentMoE and the data recipe, and no number in the report supports doing so. The one controlled comparison it does report is between learning-rate schedules, and it cuts against SOTA-140; what that costs the practice is a separate judgement, filed separately.