Gu and Dao (2023) — ARXIV-2312.00752
Key takeaways
- The diagnosis that unlocked the subquadratic line: linear attention, gated convolutions, recurrences and structured SSMs had all failed to match attention on language, and the shared weakness is inability to do content-based reasoning — the state evolves the same way regardless of what the current token is.
- The fix is one sentence long: let the SSM parameters be functions of the input. The model can then selectively propagate or forget along the sequence depending on the token it is looking at, which is exactly the capability discrete modalities need.
- That breaks the convolutional formulation the efficiency depended on, so the paper supplies a hardware-aware parallel algorithm in recurrent mode — the part that made selective SSMs practical rather than merely correct.
- Packaged as a simplified architecture with no attention and no MLP blocks at all. 5× higher inference throughput than Transformers, linear scaling in sequence length, improving on real data to million-length sequences.
- Mamba-3B beats Transformers its own size and matches Transformers twice its size.
Standing in the anthology
The root of the record’s state-space line, and a note several entries have needed. Falcon-H1 (LIT-120) runs Mamba-2 heads in parallel with attention heads in every block; Gated DeltaNet (LIT-137) is described in the record as “improving Mamba2 with the delta rule” and is the module three layers in four of the Qwen line are built from; SOTA-125 spends a tiny parameter budget on SSM state width.
The selectivity argument is the one to carry forward. Every linear-attention variant the record holds is a different answer to the same question — how does a fixed-size state decide what to keep — and this is where the question was posed sharply enough to be answerable.
No practice of its own: the recommendation it contains has a successor the
record already carries. “Make the state-space parameters input-dependent”
is the claim, and every module the record recommends is a descendant of it —
the gated delta rule and its channel-wise variant both inherit selectivity
and add to it. Filing it separately would give a reader two recommendations
where the later one supersedes the earlier, which is what a Sequence
section is for instead.