Source
Shazeer (2019), LIT-024 — ARXIV-1911.02150.
What the paper showed
This is the architecture LIT-024 introduced as multi-query attention: project queries per head as usual, but project one set of keys and values shared by every head, so the decode-time cache holds a single K and V per layer instead of one per head. The paper’s reason is bandwidth. Incremental decoding is bound by reloading the keys and values, so shrinking them by the head count is what buys the speed. The quality cost is stated qualitatively, as “only minor quality degradation” against the multi-head baseline.
Superseded
Grouped-query attention (SOTA-109) replaced this. GQA gives each group of query heads its own key/value head rather than sharing one across all of them, which keeps most of multi-query attention’s shared key/value projections’s cache saving without its quality cost, and can be uptrained from a multi-head checkpoint.
This stood as Active alongside SOTA-109’s “prefer GQA to MQA
or MHA”, so the record advised both at once. The body stays; only the status
moved.