Source
Du et al. (2025), LIT-176 — ARXIV-2510.07019.
The trade-off is the record’s own: full attention is quadratic, linear attention is efficient and compromises recall over long contexts. Every hybrid in the record is an arrangement for having some of both. This one changes where the arrangement lives.
- Long-term context sits in key-value slots updated by a linear RNN.
- Short-term context comes from a sliding window.
- One softmax attention operates over all of them together, so the weighting between long and short term is per-token, per-head and content-dependent — with no extra fusion parameters to learn.
Inter-layer behaviour is then controlled by a single hyperparameter, the sliding-window size, which slides smoothly from purely linear to full attention while every layer stays structurally identical.
The design and the case for it are LIT-176’s, the paper that introduces Native Hybrid Attention: it proposes the single-softmax layer and the window-size dial, and reports it beating Transformers and other hybrid baselines on recall-intensive and commonsense tasks.
The argument that is not about quality
A 3:1 interleave makes every fourth layer different, and that difference propagates: every serving stack, cache manager and pipeline schedule has to know about it. The infrastructure sections of the frontier reports in this record are largely about managing exactly that. A uniform stack with a dial moves the complexity out of the schedule and into a number.
Whether that is worth anything depends on costs the paper does not measure
and the record cannot supply, which is part of why this is Proposed. But it
is a real argument and it is not a quality argument, so a comparison that
reports only benchmark scores will not settle it either way.
Against the interleave, which it does not displace
SOTA-132 records the layer-interleaved hybrid — three linear layers per global one — and contrasts it with the parallel-head layout that puts both head types side by side in one block. This is a third position, and the record now holds all three: mix by layer, mix by head, or mix inside one attention.
It reports beating Transformers and other hybrid baselines on recall-intensive and commonsense tasks, and pretrained models can be structurally hybridised into it after the fact — which is the cheapest path to trying it that any of the three offers.
The relation is deliberately not declared. compared_against records a
comparison somebody ran between two things the record recommends, and this
paper’s baselines are its own, not SOTA-132’s specific 3:1 global layout.
Known implementations
- None in the record.