Dao and Gu (2024) — ARXIV-2405.21060
Key takeaways
- The theoretical claim is the title: state-space models and variants of attention are closely related, connected through decompositions of a well-studied class of structured semiseparable matrices. Two research communities had been building the same object from different ends.
- That framework — state space duality — is not only interpretive. It says which algorithmic tricks from each side transfer to the other, which is what makes the paper productive rather than merely tidy.
- Mamba-2 falls out of it: a refinement of Mamba’s selective SSM whose core layer is 2–8× faster while staying competitive with Transformers on language modelling.
Standing in the anthology
The pivot the record’s whole hybrid half rests on, and the reason a “hybrid” is a coherent thing to build rather than two architectures bolted together. Falcon-H1 (LIT-120) puts Mamba-2 heads and attention heads side by side in one block and tunes the channel split; Kimi Linear (LIT-133) and the Qwen line (LIT-136, LIT-135, LIT-152) interleave layers at 3:1 (SOTA-132). Both layouts presuppose the two mechanisms are commensurable, which is what this establishes.
The record’s linear-attention sequence now reads: Mamba’s selectivity (LIT-161) → the duality and Mamba-2 here → gated delta rule (LIT-137) → channel-wise gating in KDA (LIT-133), with LIT-165 as the 2026 continuation and LIT-173 as the other family’s answer.
No practice of its own, and the reason is what kind of result it is. The
duality is not an instruction; it is the proof that makes one possible. A
hybrid is a coherent thing to build because the two mechanisms are
commensurable, and this is what establishes that. It is a premise of the
practices rather than a rival to them, which is why it appears in their
reasoning and not in their source: lists.
Two 2026 papers retrain Mamba-2 as the decay-only baseline. Gated DeltaNet-2 (LIT-802) puts it in a common fast-weight table with the delta-rule models: it adds correlations to a decayed state rather than writing a residual. At 1.3B / 100B tokens and matched state size it is second-last of six recurrent mixers (51.82 against 53.11). Olmo Hybrid (LIT-799) uses Mamba-2 layers as the rival recurrent mixer in its architecture ablations from 60M to 1B. Pure Mamba-2 trails the transformer at 1B (0.718 against 0.682 BPB), and the 3:1 Mamba-2 hybrid trails it too (0.698), while the Gated DeltaNet hybrid leads (0.669).