Peng et al. (2025) — ARXIV-2503.14456
Key takeaways
- Constant memory and constant inference time per token, and a 2.9B model that sets a 3B state of the art on multilingual tasks and matches the 3B state of the art in English — while trained on dramatically fewer tokens than the models it is compared with.
- The mechanism is a generalised delta rule with vector-valued gating and in-context learning rates, plus a relaxed value-replacement rule. The vector-valued gate is the same move Kimi Delta Attention makes (LIT-133), arrived at independently.
- The expressivity result is the part worth carrying: RWKV-7 can perform state tracking and recognise all regular languages while remaining parallelisable to train — which, under standard complexity conjectures, exceeds what Transformers can do, since they are limited to TC⁰.
- Released with a 3.1T-token multilingual corpus and four models from 0.19B to 2.9B, Apache 2.0.
Standing in the anthology
The other expressive linear recurrence, and the reason SOTA-135’s sequence is not a single lineage. The record’s gated-delta line runs through NVIDIA’s Gated DeltaNet into Qwen and Kimi; RWKV arrived at vector-valued gating on its own, from a recurrent-network tradition rather than from linear attention.
It is built on RWKV-6 (LIT-429) directly. The paper replaces RWKV-6’s diagonal transition with its extended delta rule, keeping the matrix-valued state and data-dependent decay that Eagle and Finch introduced. It then turns the scalar in-context learning rate into a vector and decouples the removal and addition keys. It also simplifies the surrounding block: token shift is no longer data-dependent, and the channel-mixing receptance gate is gone. The 1.5B and 2.9B models were not trained from scratch. They were converted from RWKV-6 World checkpoints and trained on, which is part of why the token count behind them is so low.
Its complexity argument is the sharpest claim in the record’s architecture half and cuts against an assumption the rest of the corpus makes implicitly. Every hybrid practice here (SOTA-132) is justified by linear layers being cheaper while global attention supplies capability that linear layers lack. This says the expressivity gap runs the other way for state tracking — the recurrence can do something attention provably cannot. Whether that matters for language modelling at scale is not settled by a 2.9B model, and the record should hold the claim without leaning on it.