Source
Peng et al. (2025), LIT-173 — ARXIV-2503.14456.
A generalised delta rule with vector-valued gating and in-context learning rates, plus a relaxed value-replacement rule. Constant memory and constant inference time per token. Released Apache 2.0 with a 3.1T-token multilingual corpus and four models from 0.19B to 2.9B.
The vector-valued gate is the same move Kimi Delta Attention makes, arrived at independently and from a different tradition — recurrent networks rather than linear attention. Two lines converging on per-channel gating is worth holding as two data points rather than one.
Why this is not just another module
The reported result is efficiency-flavoured: a 2.9B model setting the 3B state of the art on multilingual tasks and matching it in English, while trained on dramatically fewer tokens than what it is compared against.
Both that result and the expressivity one below are LIT-173’s, the RWKV-7 paper that introduces the recurrence this practice recommends. Part of the low token count is that its 1.5B and 2.9B models were converted from RWKV-6 checkpoints and trained on, not trained from scratch.
The expressivity result is the one that cuts against an assumption the rest of the record makes. RWKV-7 can perform state tracking and recognise all regular languages while remaining parallelisable to train — which, under standard complexity conjectures, exceeds what a softmax stack can do, since those are limited to TC⁰.
That inverts the usual justification. SOTA-132 interleaves linear layers with global attention because the linear layers are cheaper and the attention supplies capability they lack. This says the capability gap runs the other way for state tracking: the recurrence can do something attention provably cannot.
The record should hold that claim without leaning on it. A complexity result is about what an architecture can represent, not about what training finds, and whether it matters for language modelling at scale is not settled by a 2.9B model. But a hybrid practice justified by “linear is cheap, attention is capable” is resting on half of a picture, and this is the other half.
Conditions, and why this is Proposed
One group, one line, nothing above 2.9B, and no frontier deployment. The comparison the record would most want — this recurrence against the gated delta rule it recommends, at matched scale — has not been run by anyone.
Known implementations
- RWKV-7 “Goose”, 0.19B to 2.9B. Nothing else in the record.