Source
Kimi Team (2025), LIT-133 — Kimi Linear.
Most layers carry a linear-attention module with a finite-state memory — here Kimi Delta Attention, a gated delta rule with a channel-wise forgetting gate — and one layer in about four keeps global attention, implemented as Multi-Head Latent Attention so its KV cache is compressed too. On a 48B-total, 3B-active model trained for 1.4T tokens, this hybrid outperformed full MLA under matched conditions on short-context, long-context and RL evaluations, with up to 75% less KV cache and up to 6.3× the decoding throughput at 1M tokens. Kimi K3 (LIT-131) ships the layout at 2.8T parameters with 69 KDA layers and 24 gated-MLA layers.
There is a second benefit the practice was filed without, and it only shows up in the report rather than the abstract. Kimi K3 applies no positional encoding at all to its global-attention layers, letting the interleaved KDA layers supply position sensitivity and recency instead. The consequence the report names: extending the context stops requiring a retuned RoPE frequency base or YaRN, because the global layers hold no positional parameter to retune. The idea is not the report’s — LIT-207 established it in 2023, and Kimi Linear (LIT-133) cites it and runs the ablation: against a RoPE version of the same 48B configuration, the NoPE variant wins on long context and the two are level on short. The design is filed on its own as SOTA-153, because two of its three sources have no linear attention in them at all — this practice and that one travel together in Kimi’s models and are separable everywhere else. So the hybrid is not only cheaper per token — it makes SOTA-139’s staged context extension a smaller operation. Whether that survives without a recurrence that is itself position-sensitive is untested.
Conditions: the layout has two independent adopters in the record with different linear modules — Kimi’s KDA at 2.8T (LIT-131) and Alibaba’s Gated DeltaNet (SOTA-135) from Qwen3-Next’s 80B-A3B (LIT-136) to the dense Qwen3.8-27B (LIT-135) — which is adoption evidence rather than a repeat of the experiment. The ratio is a knob — the paper’s ablations settled on 3:1 — and the global layers are what keep exact retrieval intact, so they should not be removed to chase the throughput number.
This practice used to say the controlled comparison was one group’s. It is not. LIT-195, filed via #40, ran the experiment in 2024, two years before the reports that made the layout visible here: DeltaNet interleaved with sliding-window attention every other layer, and DeltaNet with just two global attention layers (the second and the n/2-th), both beating a strong Transformer++ baseline at 1.3B. Different group, different linear module, different ratios, same conclusion.
It also supplies a mechanism the source report does not. Linear attention is content-addressed and carries little positional information, and it is specifically bad at precise local shift-and-compare; a minority of softmax layers restores exactly that. Which is a better reason to keep the global layers than “retrieval gets worse without them”, and it predicts that the right ratio depends on how much local comparison the task needs rather than on a universal constant.
A fourth laboratory has run it, against a released transformer. Olmo Hybrid (LIT-799) takes Olmo 3 7B, replaces its three-in-four sliding-window layers with Gated DeltaNet, and keeps one full-attention layer in four. Against the released Olmo 3 it reaches the same MMLU in 49% fewer tokens and leads in every domain after mid-training. RULER at 64K is 76.9 against 70.9 with YaRN for both. The data mix, schedule and mid-training batch also changed, so it is close to controlled rather than controlled. Its ablations, on identical data from 60M to 1B, are the cleaner part. All three interleaved ratios beat the transformer at 1B. 3:1 is best at 760M and 1B, and 7:1 is level with it at small scale and ahead at 600M. Spreading the attention layers evenly beat concentrating them in the middle, by 0.003 BPB at 1B. That is the ratio this practice recommends, chosen again by a group that tested the alternatives, with the caveat already above: the ratio is a knob and its best setting moves with scale.
Sequence
The interleave starts earlier than this practice’s sources do. Griffin (LIT-204, 2024) mixes gated linear recurrences with attention layer by layer, and the delta-rule work says so outright — its sliding-window hybrid “follows Griffin and Samba”. So the line runs Griffin → the Gated DeltaNet paper’s own H1 and H2 hybrids → Kimi Linear’s 3:1 global layout → the 2026 production models, and the record held only the last two steps.
The fork at the first step is the one to keep. Griffin and Samba put local attention in the minority layers; the layouts this practice recommends put global attention there. That is not a detail: only global attention preserves exact retrieval at arbitrary distance, which is the property the minority layers are here to protect. A reader taking “interleave at 3:1” from the earlier work would get the ratio and lose the reason.
The Gated DeltaNet paper’s own H1 and H2 hybrids (LIT-137, 2024) → Qwen3-Next ships 3:1 Gated DeltaNet to gated attention in production (LIT-136, September 2025) → Kimi Linear’s fair comparison against full attention (LIT-133, October 2025, this practice’s source) → Qwen3.5, 3.6 and 3.8 (LIT-135) and Kimi K3 (LIT-131) at dense and frontier scale.
Variations
A third laboratory has since adopted the layout independently: LIT-183’s Nemotron 3 Nano is a MoE hybrid Mamba-Transformer at 30B-A3B, using Mamba rather than a linear attention as the cheap layer. That widens what the practice can claim — from “the 3:1 interleave works” toward “a hybrid of a fixed-state mixer with periodic global attention works, and the mixer’s family is a second-order choice”.
The one controlled test of that last clause points the other way. In Olmo Hybrid’s ablations (LIT-799), the same 3:1 layout with Mamba-2 layers trails the plain transformer at 1B (0.698 against 0.682 BPB), while the Gated DeltaNet version leads it (0.669). Mamba-3 and KDA were not in that comparison, so it says that the family can matter, not which one is best. Nemotron 3 Nano is adoption evidence for a Mamba hybrid. It does not report the alternative.
A third layout also exists now. LIT-176’s Native Hybrid Attention
puts the mixture inside one uniform layer — a linear RNN maintains long-term
KV slots, a sliding window supplies short-term tokens, and a single softmax
attends over both — so the ratio becomes a continuous hyperparameter instead
of a layer schedule. The practical argument for it is that an interleave
makes every fourth layer different, which every cache manager and pipeline
schedule then has to know about; the infrastructure sections of LIT-131 and
LIT-152 are largely about managing that. It is now filed as a Proposed
practice of its own; it has no frontier deployment where this has two, so
this practice does not move, and the comparison that would settle it has not
been run.
The other hybrid layout in the record is Falcon-H1’s (LIT-120): SSM heads and attention heads in parallel within every block, with the split a per-block channel allocation rather than a layer schedule. LIT-119’s tiny
ablations (SOTA-125) found SSM capacity worth more than MLP width under that layout; no source in the record compares the two layouts against each other.
Known implementations
- Kimi Linear 48B-A3B, Kimi K3; Qwen3-Next, Qwen3.5, Qwen3.6-27B, Qwen3.8-27B