Source
Li et al. (2026), LIT-177 — ARXIV-2606.26560.
SOTA-135 pairs a decay gate for erasure with a delta update for targeted writes. This names a limitation in that pairing precisely enough to act on: the delta rule corrects what is stored at the current write address before writing there. The correction is anchored to the write. So stale information sitting at a different address cannot be actively removed — it can only decay passively, which is what the gate is for and is the blunt instrument the gate always was.
EDA decouples the two. A targeted erase step along a learned erase direction runs first, then the ordinary delta-style corrective write along the write direction. The corrective behaviour is preserved and a cleanup path is added beside it.
The evidence, and the part of it that is unusual
Best in both settings tested — dense 2.5B and MoE 25B-A2.8B — and the gain persists after 80B tokens of long-context midtraining, where it also leads long-context evaluations from 4k to 128k. Persistence through midtraining matters more than the headline: a memory-management improvement that washes out once the model has seen long contexts would be measuring something else.
These results are LIT-177’s, the paper that introduces EDA and first makes this recommendation: it names the write-anchored correction as the limitation and tests the decoupled erase against the plain delta rule at both scales, through the 80B-token midtraining.
The analysis is the part worth reading. Update analysis and memory-state probes show EDA allocating its cleanup path most strongly where passive decay is weak — which is what the mechanism predicts, and is the kind of confirmation architecture papers usually assert rather than demonstrate.
The rival answer, which nobody has compared it to
Kimi Delta Attention meets the same pressure with a channel-wise forgetting gate, letting each channel decay at its own rate: finer passive decay. EDA’s answer is an addressed erase: active removal somewhere other than where you are writing.
Both are about giving a fixed-size state better memory management, they are
not variants of each other, and the record now holds both with no comparison
between them. That is the condition in promote_when:, and it is available
to be run — the channel-wise gate ships at 2.8T.
A second group, in a narrower form
Gated DeltaNet-2 (LIT-802), from NVIDIA and published in May 2026, a month before the EDA paper, reached a version of the same idea independently. Its update is S_t = (I − k_t (b_t ⊙ k_t)ᵀ) D_t S_{t−1} + k_t (w_t ⊙ v_t)ᵀ. The write still runs along k_t, but the read that gets erased runs along b_t ⊙ k_t, a channel-wise gate on the key. So the erase address is decoupled from the write address. Unlike EDA’s learned erase direction, it can only reweight the coordinates of the current key. It is built on KDA’s channel-wise decay and measured against it at 1.3B / 100B tokens with matched state size. The variant that keeps only the erase gate channel-wise averages 52.79 against KDA’s 52.28 (Wiki perplexity 16.12 against 16.81). The full model averages 53.11.
That is close to “an independent group running an addressed erase against a
channel-wise decay gate”, and it answers the comparison the section above
asks for in one way: the two remedies stack rather than compete. It is not
treated here as meeting promote_when:, for three reasons. The erase
direction is a restricted case of the one recommended here. It is one run
at one scale. And its ablation does not say whether the scalarised variants
were retrained. A reader who disagrees with that judgement has the numbers
above to disagree with.
Known implementations
- None in the record for the design as filed. GatedDeltaNet-2 (NVlabs) implements the restricted, channel-reweighted form.