Source
Yang et al. (2025), LIT-208 — RNoPE-SWA. Puvvada et al. (2025), LIT-209 — SWAN-GPT. Kimi Team (2025), LIT-133 — Kimi Linear.
What to do
In a model whose layers alternate between full attention and something cheaper — sliding-window attention, or a linear-attention recurrence — give the full-attention layers no positional encoding at all, and leave the encoding, or the recurrence, to the cheap layers. All three sources put one global layer per three local ones.
Why it works
Because the two layer types are being asked for different things, and an encoding helps one and hurts the other.
LIT-208 measured it. In a hybrid, the NoPE layers show a sharp spike of attention mass on the tokens being retrieved and comparatively little recency bias; the RoPE layers show strong recency and almost no retrieval. Retrieval and locality are separable jobs, each layer type is bad at the other’s, and the positional encoding is what makes a layer local. So a layer you want to retrieve over the whole sequence is a layer you should not make local.
LIT-207 is why this costs nothing: a decoder-only transformer without a positional encoding can represent absolute and relative position anyway, and in a hybrid it does not even have to, because a windowed layer or a decaying recurrence sitting beneath it already has.
The consequence that matters most
Why a layer can go without an encoding at all
This practice removes positional encoding from some layers, and the reason that is possible is not in the hybrid papers. Haviv et al. (LIT-665) showed in 2022 that a decoder-only LM given no positional information is competitive — 0.05 perplexity from learned embeddings at 1.3B on the Pile — and probed where the position comes from: absent at layer 1, recovered within four layers, as accurate as learned embeddings by mid-network.
THEORY-102 is the mechanism: the causal mask itself carries position, because a token that can count its attendable predecessors knows its index. Two consequences bear directly on this recommendation.
The mechanism is causal, so it is available to a decoder and to nothing else. The same paper’s control is a bidirectional MLM, where NoPos perplexity is 147.18 against 4.00 for position-aware baselines. Nothing about this layout transfers to an encoder.
Position accumulates over depth rather than being present at the input. A hybrid is choosing which layers may go without an encoding, and those layers differ in how much implicit position has built up by the time they run. No document in this record says what that does, including that one — and it is the obvious question this layout raises.
“Lose nothing” was too strong. Haviv et al.’s own limitations section says NoPos is “always slightly worse”, and their scale sweep shows smaller models benefiting from fixed non-parametric encodings with the gap closing as scale grows. What NoPE wins is extrapolation past the training length, and that is LIT-207’s result rather than this one’s. The practice stands on the extrapolation, not on parity.
Context extension stops being an operation. There is no positional parameter in the global layers, so there is nothing to rescale: no retuned base frequency, no interpolation, no YaRN. LIT-131 reports reaching 1M tokens this way and says so in as many words. LIT-209 gets extrapolation well past its training length with no long-context training stage at all.
This is the same problem SOTA-151 solves the other way, which is why the two are filed as alternatives rather than as a sequence.
Conditions, and what the sources disagree about
The local layers have to be genuinely local. LIT-208’s sharpest negative result: widening the RoPE layers by raising the base frequency damages the NoPE layers downstream, because the extra span is noise to a layer trying to compute similarity. Needle attention mass fell from 0.0765 to 0.0369 and the needles score from 8.036 to 6.203 as θ went from 10,000 to 4 million. The fix was to confine the RoPE layers to a 4096-token window. If you take one condition from this practice, take this one — it inverts the reflex that longer context means a bigger base frequency.
Whether inference-time attention scaling is required is unsettled. LIT-209 says a dynamic scaling of attention scores is the crucial element keeping global NoPE layers usable far past the training length, and names LIT-208 as lacking it. LIT-208 reports good long-context results without it, and LIT-133 does not use it either. Two of three say it is not needed; the one that says it is has the strongest extrapolation claim. Nobody has run the comparison.
The ratio is 1:3 in all three and ablated in one. LIT-208 tested 1:1, 1:3 and 1:7 and found 1:3 best; the position of the global layer within the group did not matter. Two independent groups arriving at the same ratio is suggestive, not a law — and the ablation is one lab’s, on one architecture.
It is not only a pretraining decision. This is the correction LIT-209 forces on the record. An 8B RoPE model pretrained on 15T tokens was converted by initialising from its weights, removing the encoding from the global layers, windowing the local ones, and continuing to pretrain; it came back at 71.55 against the original’s 70.95 across GSM8k, MATH500, MBPP, HumanEval, MT-Bench and RULER. Continued pretraining is not free and most readers will not do it, but “an existing model cannot take this option” — which SOTA-063 and SOTA-151 both said until now — is false.
What would move this to converged
A frontier report from outside Cohere, NVIDIA and Kimi shipping the layout; or a controlled comparison against a properly extended RoPE model — YaRN at the same context and matched training cost — rather than against an unextended baseline. Every comparison in the record so far is against a RoPE model that was not given the rescaling treatment SOTA-151 recommends, which is the weakest point in the case.
Half of it has since been run, by a fourth group. Olmo Hybrid (LIT-799), from Ai2, is a 7B hybrid with Gated DeltaNet in three layers of four and full attention in the fourth. It was pretrained with RoPE, and for long context it was extended on the same 100B tokens in two ways: YaRN, and DroPE, which removes RoPE from every attention layer and leaves position to the recurrent layers. RULER at 4K / 16K / 32K / 64K: YaRN 92.8 / 90.0 / 84.7 / 76.9, DroPE 92.2 / 88.4 / 86.2 / 85.0. So, against a properly extended RoPE model at matched cost, the encoding-free global layers win beyond 16K and trail slightly below. Two things keep it from settling the question. It is one model, and the encoding was removed at extension rather than absent from pretraining, which is closer to LIT-209’s conversion than to the designs this practice is sourced to.
Adjacent, and not the same practice
SOTA-132 recommends the 3:1 interleave of linear attention with global attention. This practice is about what the global layers do with position, and it applies to a windowed-attention hybrid just as well — two of the three sources here have no linear attention in them at all. The two travel together in Kimi’s models and are separable everywhere else.