Fu et al. (2023) — ARXIV-2310.12109
Key takeaways
- Transformers scale quadratically along two axes — sequence length and model dimension — and almost all efficiency work addresses only the first, because it replaces the sequence mixer and leaves the MLP alone
- Monarch matrices are one sub-quadratic primitive for both: expressive enough to capture many linear transforms including the DFT, and decomposable into GEMMs, so they reach high FLOP utilisation where attention — being memory-bound — cannot, even under FlashAttention
- Attention-free, not hybrid. M2 replaces attention; the paper’s own reading of its causal result is that “radically different architectures than Transformers may be performant on causal language modeling”
- BERT-style: matches BERT-base and BERT-large on GLUE with up to 27% fewer parameters and up to 9.1× throughput at sequence length 4K
- ImageNet: M2-ViT beats ViT-b by 1% at half the parameters
- Causality is the hard case: enforcing it by masking reintroduces a quadratic bottleneck, so the paper develops a multivariate-polynomial view of Monarch matrices to parameterise causality directly
- Proof of concept, by the authors’ own framing — BERT-large is the largest model here
Standing in the anthology
The source of SOTA-110 and SOTA-111.
Read in full for #114; the reading is NOTE-014.
Two of three practices confirmed. SOTA-112 — combine M2 with attention rather
than replacing it — is Rejected: this paper is attention-free and argues for
replacement, and hybrid occurs zero times in it. The hybrid argument is real
and belongs to the attention/SSM line elsewhere in the record.
SOTA-111 was also understating its source, naming only the sequence-length axis. The dimension axis is what separates this work from the entire efficient-attention literature it builds on.