Source
Fu et al. (2023), LIT-115 — ARXIV-2310.12109.
The other axis, which this practice was leaving out
Transformers scale quadratically along two axes — sequence length and model dimension — and almost all efficient-attention work addresses only the first, because it replaces the sequence mixer and leaves the MLP alone.
LIT-115’s contribution is one primitive that is sub-quadratic along both, serving as sequence mixer and dimension mixer alike. That is what separates it from the long-convolution line it builds on, and it is the reason to reach for it: if the model dimension is as much of a constraint as the context, nothing in the efficient-attention literature helps and this does.
The measured form: BERT-base and BERT-large quality on GLUE with up to 27% fewer parameters and up to 9.1× throughput at 4K; ViT-b beaten by 1% at half the parameters.
The condition the whole cluster turns on
Attention’s cost is quadratic in sequence length; Monarch Mixer replaces the mixing with structured Monarch matrices, which are products of block-diagonal factors and give sub-quadratic scaling while staying expressible as dense matrix multiplies — so the hardware still sees GEMMs rather than a sparse kernel it handles badly.
That last part is the reason this is a serious proposal rather than one more sub-quadratic scheme. Most alternatives to attention lose in practice because their theoretical FLOP saving lands on operations the accelerator runs slowly; keeping the work as GEMMs is what makes the asymptotic win reachable.
When the condition is not met
At the sequence lengths most language models train on, attention is not the bottleneck — the MLPs are, and a fused exact-attention kernel (SOTA-085, SOTA-106) has already removed the memory cost that made attention look worse than it is. Substituting a sub-quadratic mixer there gives up exactness for nothing.
So the practice is genuinely conditional, and the title now says on what. SOTA-110 states the other face of the same condition: the regime where it is met by default.
The third sibling, SOTA-112, used to be named here as “what to do when the condition is met only in part” — interleave M2 with attention. It is retired: LIT-115 is an attention-free architecture and argues for replacement, so there is no partial case it describes. If one exists it needs a source.
The record has no measurement of where the crossover sits, which is the honest limit — LIT-115 reports the architecture and its scaling, and nobody here has established the sequence length at which the substitution starts paying for a transformer of a given size.