Kimi Team, Moonshot AI (2025) — ARXIV-2510.26692
Key takeaways
- Kimi Delta Attention keeps the gated delta-rule recurrence of LIT-137 but replaces the scalar forgetting gate with a vector-valued one, so each channel of the finite-state memory decays at its own rate — a finer use of the fixed memory a linear-attention layer has.
- A chunkwise kernel built on a specialised Diagonal-Plus-Low-Rank transition keeps the layer hardware-efficient while staying close to the classical delta rule.
- The architecture is a layerwise hybrid: KDA layers interleaved with Multi-Head Latent Attention at 3:1, so one layer in four keeps global attention.
- On a 48B-total, 3B-activated model trained for 1.4T tokens, the hybrid outperforms full MLA under matched conditions across short-context, long-context and reinforcement-learning regimes — the paper’s claim to be the first hybrid linear architecture to do so under a fair comparison.
- Efficiency: KV cache reduced by up to 75%; RULER at 128k reached 84.3 with 3.98× acceleration; at 1M tokens, time per output token 1.84 ms against 11.48 ms for MLA, about 6.3×.
- The global layers carry no positional encoding at all. NoPE (LIT-207) is applied to every MLA layer, which hands the whole job of position and recency to the KDA layers — the paper’s phrase is that KDA becomes “the primary position-aware operator”. The ablation is the scale evidence LIT-207’s 107M study lacks: against a Kimi Linear (RoPE) baseline at the same 48B configuration, the NoPE variant wins on long context and the two are level on short, which the paper reads as a difference in how positional bias is distributed across depth. Two practical consequences it names: the MLA layers become convertible to pure MQA at inference, and extending the context needs no base-frequency tuning and no YaRN, because there is no positional parameter to retune.
Standing in the anthology
The source of SOTA-132 and the design LIT-131 adopts at 2.8T. The other hybrid in the record, LIT-120, puts SSM and attention heads side by side in every block rather than alternating layers; the two are the record’s two answers to the same question.
Gated DeltaNet-2 (LIT-802) keeps KDA’s channel-wise decay and splits its scalar β into a channel-wise erase gate on the key and a channel-wise write gate on the value. Tying both gates to one scalar gives KDA back. KDA was retrained there under a single 1.3B / 100B-token recipe and averages 52.28 against 53.11 as a recurrent model and 52.68 against 53.97 as a hybrid with sliding-window attention. Olmo Hybrid (LIT-799) lists Kimi Linear 48B-A3B in its table of open-weight models, trained on different data and budgets: Math 68.5 against Olmo Hybrid’s 55.1, RULER at 64K 90.5 against 85.0. It names this paper among the hybrids that do not use the negative-eigenvalue extension of the delta rule.