Katharopoulos et al. (2020) — ARXIV-2006.16236
Key takeaways
- Replace
softmax(QKᵀ)Vwithφ(Q)(φ(K)ᵀV). Once the similarity is a kernel rather than a softmax, matrix associativity lets theK–Vproduct be formed first, and the sequence-length term drops from quadratic to linear - The autoregressive case is an RNN, exactly. The running
φ(K)ᵀVsum is a fixed-size state carried forward one token at a time — which is the paper’s title and the reason every later recurrent-attention architecture can call itself both things without equivocating - Up to 4000× faster autoregressive inference at long context, because generation stops re-reading the prefix and reads a state instead
- The state is fixed-size, and that is the whole trade. A quadratic layer keeps every past token addressable; this keeps a summary. What later work argues about is what to put in that summary and how to let it forget
Standing in the anthology
Carries no practice, deliberately — ADR-032 for the category, though
this is a method paper rather than a benchmark: the recommendation it would
support is use linear attention, and the record’s position on that is
SOTA-167’s, which is that “linear attention won” is not the right summary.
What it is here for is that the record argues about linear attention in
twelve documents and could not cite its source.
That is DP-007 in its purest form: the identity above is assumed by
LIT-194 (“linear attention has no decay term, so it cannot forget”),
LIT-176 (“efficient and compromises recall over long contexts”),
LIT-195 (“lacks precise associative recall”), SOTA-132, SOTA-153 and
LIT-165 — every one of which is a claim about what this paper’s
construction gives up. Agreement has no author, so the thing they all agree
on went unfiled while six documents discussed its consequences.
It is also the root of two lines the record already holds separately. The
delta-rule and SSM papers (LIT-195, LIT-137, LIT-161, LIT-162) fix
the forgetting problem from one direction; LIT-431 and the RWKV line
fix it from another. Both are answers to the same fixed-size state.
AFT (LIT-431) is also where this model was measured as a baseline: on Enwik8, at the same 12-layer, 512-wide shape, the linear transformer reaches 1.207 test bits per character against 1.180 for AFT-local-32 (AFT’s Table 4).
Unread — no NOTE.