Source
Fu et al. (2023), LIT-115 — ARXIV-2310.12109.
Why this is rejected: the source argues the opposite
LIT-115 is an attention-free architecture. The string hybrid occurs
zero times in it. What it does is replace: “attention-free models by
replacing attention layers”, “M2-ViT … replaces the long convolutions with
the M2 operator”, and its own summary of the causal result is that “radically
different architectures than Transformers may be performant on causal language
modeling”.
This practice recommends interleaving M2 with attention. That is not a different emphasis or a transposed number — it is the recommendation the source argues against.
The argument below is real, and it is somebody else’s
What follows was the body before #114, and it is kept because the reasoning is sound: a structured mixer and attention do fail differently, and interleaving them is a live design. But that argument is made by the attention/SSM hybrid literature in this record — SOTA-132’s line — not by Monarch Mixer, and the ratio those papers establish is measured for a different pair of mechanisms.
If the record wants a Monarch/attention hybrid practice it needs a paper that proposes one. Nobody has.
Why hybrids rather than replacement
A structured mixer and attention fail differently. Attention is exact, data-dependent and expensive; a Monarch mixer is cheap, fixed-pattern and approximate. Interleaving them keeps a few layers able to do the exact all-pairs comparison that some tasks need — retrieval from a long context, precise copying — while the majority of layers pay the cheaper cost.
This is the same shape as the hybrid attention/state-space designs elsewhere
in the record (SOTA-132’s line, SOTA-176’s in-layer version): the argument for mixing two mechanisms is not that either is better, it is that their failure modes are complementary and the ratio is a dial.
The dial, and what nobody has set
The unstated parameter is how many attention layers and where. The hybrid literature in this record puts the ratio at a small fraction — a handful of full-attention layers in a stack of tens — but that is measured for attention/SSM hybrids, not for this mixer, and the two are not obviously interchangeable.
So the practice is directionally right and quantitatively empty, which the title should not hide. Its value is as a warning against the wholesale substitution SOTA-110 and SOTA-111 could be read as licensing.