anthology-of-the-sota

LIT-165: Mamba-3: Improved Sequence Modeling using State Space Principles

2 min read

Properties7
StatusActive
Version2
Tagsmodel-architecture, attention-techniques
ExtendsLIT-162 — Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality
Compared againstLIT-802 — Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Cited as “introduced by” bySOTA-167 — Build linear-attention layers from the state-space view: expressive discretisation, complex-valued state, and a MIMO readout
Cited as “source” bySOTA-167 — Build linear-attention layers from the state-space view: expressive discretisation, complex-valued state, and a MIMO readout

Graph View

Lahoti et al. (2026) — ARXIV-2603.15569

Key takeaways

  • The framing is inference-first, and the critique is aimed at its own family: many recent linear models trade quality and capability for algorithmic efficiency, failing on tasks such as state tracking, and their theoretically linear inference is often hardware-inefficient in practice. Linear is not automatically fast and not automatically enough.
  • Three improvements, each from the SSM view of linear models: a more expressive recurrence derived from SSM discretisation; a complex-valued state update that enables richer state tracking; and a multi-input multi-output (MIMO) formulation that improves quality without increasing decode latency.
  • At 1.5B, +0.6 downstream accuracy over the next best model — named as Gated DeltaNet — and the MIMO variant +1.2 further, for +1.8 total.
  • Across state-size experiments it matches Mamba-2’s perplexity at half the state size, which is the efficiency claim that matters for a fixed-memory architecture.

Standing in the anthology

The current end of the state-space line the record now holds from the start: Mamba’s selectivity (LIT-161) → the duality and Mamba-2 (LIT-162) → gated delta rule (LIT-137) → channel-wise gating (LIT-133) → here.

It is the only note in that line that measures against Gated DeltaNet as the incumbent, which is a useful correction of perspective: the record’s hybrid practice (SOTA-132) treats GDN and KDA as the modules to build with, and this says the SSM family kept moving in parallel and is ahead at 1.5B. Nothing in the record ships Mamba-3, so it stays a note; what it changes is that “linear attention won” is not the right summary of the last two years.

One rerun under another group’s recipe partly disagrees. Gated DeltaNet-2 (LIT-802) retrains Mamba-3 SISO and MIMO (rank 4) at 1.3B / 100B tokens against Gated DeltaNet at matched state size. MIMO is ahead of Gated DeltaNet there (52.39 against 52.07 average), as this paper says. SISO is behind it (51.42), and last of the six recurrent mixers. In hybrid form both Mamba-3 variants are at 52.7, against 52.25 for Gated DeltaNet and 53.97 for Gated DeltaNet-2. It is one run each, by the group that made Gated DeltaNet, so it qualifies “ahead at 1.5B” for SISO rather than reversing it.

Backlinks