Source
Fu et al. (2023), LIT-115 — ARXIV-2310.12109.
The regime where the substitution is worth considering
Long sequences are where attention’s quadratic term dominates everything else, so they are where a sub-quadratic mixer has the most to give back. That is the whole of the recommendation, and the condition SOTA-111 states is the general form of it.
Two things make “consider” the right verb rather than “use”. Exactness is given up — attention computes a data-dependent all-pairs interaction and a structured mixer approximates the mixing, so the substitution is a modelling decision and not an implementation one. And the ecosystem is not neutral: attention has fused kernels, quantised inference paths and serving systems built around its KV cache (SOTA-105), none of which transfer.
What the record cannot tell you
Where “very long” begins. LIT-115 gives the architecture and its asymptotics; this record holds no measurement of the crossover length for a given model size, and the answer moves every time attention’s constant factor improves — which it has repeatedly (SOTA-085, SOTA-106).
That is a real gap rather than a quibble: a practice whose condition is a threshold nobody has measured cannot be followed except by measuring it yourself, which is what a reader should take from this.