anthology-of-the-sota

LIT-137: Gated Delta Networks: Improving Mamba2 with Delta Rule

3 min read

Properties8
StatusActive
Version2
Tagsattention-techniques, model-architecture
ExtendsLIT-162 — Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality, LIT-195 — Parallelizing Linear Transformers with the Delta Rule over Sequence Length
Extended byLIT-133 — Kimi Linear: An Expressive, Efficient Attention Architecture, LIT-799 — Olmo Hybrid: From Theory to Practice and Back, LIT-802 — Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Compared againstLIT-802 — Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
Cited as “introduced by” bySOTA-135 — Give linear-attention layers a gated delta rule: a decay gate for erasure plus a delta update for targeted writes
Cited as “source” bySOTA-132 — Interleave linear-attention layers with global attention at about 3:1 instead of using full attention throughout, SOTA-135 — Give linear-attention layers a gated delta rule: a decay gate for erasure plus a delta update for targeted writes

Graph View

Yang et al., NVIDIA and MIT (2024; ICLR 2025) — ARXIV-2412.06464

Key takeaways

  • Two mechanisms had been explored separately for linear transformers: a gate for adaptive memory control (Mamba2) and the delta update rule for targeted memory modification (DeltaNet). They are complementary — gating lets the state forget quickly, the delta rule lets it overwrite one association without disturbing the rest — and the gated delta rule combines them in one recurrence.
  • A parallel, chunkwise training algorithm makes the rule efficient on modern accelerators, which the plain delta rule had not been.
  • At 1.3B parameters on 100B tokens, Gated DeltaNet consistently beats Mamba2 and DeltaNet on language modelling, commonsense reasoning, in-context retrieval, length extrapolation and long-context understanding.
  • Two hybrids, GatedDeltaNet-H1 and -H2, interleave the layer with sliding-window attention or Mamba2 layers and improve on the pure form — the seed of the layer-interleaved hybrids that followed.

Standing in the anthology

The source of SOTA-135 and the module three layers in four of the Qwen line are built from (LIT-136, LIT-135). Kimi Delta Attention (LIT-133) extends it with a channel-wise forgetting gate; the sequence of linear-attention recurrences in the record now reads Mamba2 and DeltaNet → this → KDA.

The recurrence is S_t = S_{t−1}·α_t(I − β_t k_t k_tᵀ) + β_t v_t k_tᵀ. From Mamba2 (LIT-162) it takes the data-dependent decay α_t, with Mamba2’s parameterization, which can clear the state quickly but decays every association at the same rate. From DeltaNet (LIT-195) it takes the delta update, which overwrites the value stored under one key, and the WY-based chunkwise parallel algorithm. The paper extends that algorithm to carry the gate. It then beats both parents at 1.3B / 100B tokens. In its single-needle tests Mamba2 degrades past 2K tokens because it forgets too fast, and DeltaNet degrades at longer lengths on the real-essay needle tests because it clears memory poorly.

Two 2026 papers build on it. Gated DeltaNet-2 (LIT-802), from two of its three authors, splits β into a channel-wise erase gate and a channel-wise write gate on top of KDA’s channel-wise decay, and reduces to this recurrence when all three are tied to scalars. It retrains this model as a baseline under one 1.3B / 100B-token recipe. Gated DeltaNet averages 52.07 against 53.11, but it still holds the word-needle task at 4K, 60.6 against 31.8. Olmo Hybrid (LIT-799) uses the recurrence with the negative-eigenvalue extension (β → 2β) for three layers in four of a 7B model trained to 6T tokens. In its ablations from 60M to 1B, Gated DeltaNet beats Mamba-2 at every scale, as a pure model and in a 3:1 hybrid.

Backlinks