Peng et al. (2023) — ARXIV-2305.13048
Key takeaways
R,W,K,V— receptance, weight (the decay), key, value. TheWisLIT-431’s learned position bias restricted to a decaying function of relative distance, and that restriction is what buys the recurrence: a bias that depends only on how far back a token is can be accumulated incrementally- Train in parallel, run as an RNN. The same layer has a
sequence-parallel form for training and a step-at-a-time form for
generation with state that does not grow — the property the line is
organised around and that
LIT-428first established was possible - Scaled to 14B, which at the time was by a wide margin the largest recurrent language model trained, and the evidence that the construction was not confined to small scale
Standing in the anthology
Carries no practice, deliberately — ADR-032.
This is the missing trunk #161 named. The record held LIT-173 (RWKV-7)
and nothing that led to it, which matters more than a gap usually does
because SOTA-178’s entire evidence base is RWKV-7 — its own conditions
section says “RWKV-7 ‘Goose’, 0.19B to 2.9B. Nothing else in the record.”
A practice resting on one model of one architecture reads differently depending on whether that architecture is a one-off or the current member of a line that has been scaled and re-derived four times. It is the second, and now the record can show it.
SOTA-135 is the other document this changes. It observes that RWKV arrived
at a correction-style update “independently and a year earlier” than the line
it is filed beside — a claim about when this lineage did something, which
was being made against a record that held only the lineage’s latest member.
The construction it starts from is LIT-431’s. The paper writes AFT as keys and values weighted by learned pairwise position biases, says RWKV is “inspired by AFT”, and replaces that pairwise bias matrix with a channel-wise decay vector multiplied by relative position, the change that lets the layer be rewritten as an RNN.
Unread — no NOTE.