Peng et al. (2024) — ARXIV-2404.05892
Key takeaways
- Eagle (RWKV-5) makes the state matrix-valued.
LIT-430carried a vector per channel; a matrix state holds associations rather than a running sum, which is the capacity the recall criticisms of linear attention were about - Finch (RWKV-6) makes the decay data-dependent. The
Wstops being a learned constant per channel and becomes a function of the token — the layer can decide what to forget and when, which a fixed decay cannot - Both released with a multilingual corpus and tokenizer, and the multilingual result is where the line’s headline numbers have come from since
Standing in the anthology
Carries no practice, deliberately — ADR-032.
It is the step that makes LIT-173 legible. RWKV-7’s contribution is
vector-valued gating and in-context learning rates on top of a generalised
delta rule — each of those is an increment on something introduced here, and
SOTA-178 states the practice in exactly those comparative terms without the
record holding the thing being compared to.
The data-dependent decay is also the convergence point. SOTA-178’s own
body notes that “two lines converging on per-channel gating is worth” saying
— and this is where the RWKV side of that convergence happens, a year before
the delta-rule line (LIT-137, LIT-162) reached the same place from
linear attention. LIT-194 records the same observation about timing. Both
claims are about this paper, made while it was absent.
It builds directly on LIT-430: the paper presents Eagle and Finch as improving on RWKV-4, keeps its stacked time-mixing and channel-mixing blocks, its token shift and, nearly unchanged, its channel mixing, and changes the time mixing — matrix-valued multi-head states, LayerNorm over heads and SiLU gating in Eagle, then data-dependent decay and token shift in Finch. The state grows by more than an order of magnitude over RWKV-4’s, and on PG19 at 3B Eagle’s loss along sequence position improves sharply over RWKV-4’s, with Finch lower again.
Unread — no NOTE.