Zhai et al. (2021) — ARXIV-2105.14103
Key takeaways
- No
QKᵀat all. WhereLIT-428keeps a similarity and makes it cheap, this removes the token-token interaction: values are weighted by a learned pairwise position bias plus the key, and the query multiplies the result element-wise - The interaction becomes positional rather than content-based, which is the strong assumption and the source of the trade — a content-dependent lookup is exactly what is given up
- Memory is constant in both context size and model size for the local variants, which is what the architecture was argued for
Standing in the anthology
Carries no practice, deliberately — ADR-032.
It is a bridge document. On its own the record would have little use for it:
AFT is not a live recommendation and nothing here is measured on it. It is
filed because the RWKV line is unreadable without it — the W in RWKV is
this paper’s learned position bias, and RWKV’s own framing is AFT with the
bias made a decaying function of relative position so the whole thing can run
as a recurrence.
Without this note, LIT-430 arrives with four letters and no
explanation of where three of them came from, and a reader tracing the line
from LIT-428 to LIT-173 has to leave the record to follow it.
Its relation to LIT-428 is a measured comparison, not descent. AFT is derived from multi-head attention, and the paper notes that its order of computation, keys and values combined before the query meets them, “is also found in” the “linearized attention” works, as a shared feature rather than a borrowed one. What differs is how they are combined, element-wise rather than by matrix products, which removes the cost quadratic in feature dimension that linear attention pays. On Enwik8 it runs Linear Transformer as a baseline at the same 12-layer, 512-wide shape: 1.207 test bits per character against 1.180 for AFT-local-32 (Table 4).
Unread — no NOTE.