Source
Shazeer (2019), LIT-024 — ARXIV-1911.02150.
What the paper showed
Multi-query attention, and the advice to use it for decoding, come from LIT-024. Its argument starts from a diagnosis: incremental decoding cannot parallelise across positions the way training does, so its speed “is limited by the memory bandwidth necessary to reload the large keys and values tensors”, not by arithmetic. Sharing one key/value head across all query heads shrinks those tensors by the head count, and the paper reports much faster decoding at “only minor quality degradation” from the multi-head baseline, a cost stated qualitatively rather than bounded, which is the room GQA later found.
Superseded
Grouped-query attention (SOTA-109) replaced this. GQA gives each group of query heads its own key/value head rather than sharing one across all of them, which keeps most of multi-query attention’s cache saving without its quality cost, and can be uptrained from a multi-head checkpoint.
This stood as Active alongside SOTA-109’s “prefer GQA to MQA
or MHA”, so the record advised both at once. The body stays; only the status
moved.