anthology-of-the-sota

LIT-024: Fast Transformer Decoding: One Write-Head is All You Need

2 min read

Properties6
StatusActive
Version2
Tagsattention-techniques, model-architecture, inference-optimization
Cited as “introduced by” bySOTA-023 — Use multi-query attention for decoder-only models to reduce memory bandwidth, SOTA-024 — Keep key/value projections shared across heads while query projections remain separate
Cited as “paper” byNOTE-016 — Fast Transformer Decoding: multi-query attention
Cited as “source” bySOTA-023 — Use multi-query attention for decoder-only models to reduce memory bandwidth, SOTA-024 — Keep key/value projections shared across heads while query projections remain separate

Graph View

Shazeer (2019) — ARXIV-1911.02150

Key takeaways

  • The diagnosis is the contribution. Incremental Transformer inference “is limited by the memory bandwidth necessary to reload the large keys and values tensors which encode the state of the attention layers” — not by arithmetic. The paper derives this before proposing anything
  • Training parallelises across sequence positions and decoding cannot, which is why the two have different bottlenecks and why an architecture tuned for one may be wrong for the other
  • Multi-query attention: share one set of keys and values across all attention heads, keep the queries per-head. The KV tensors shrink by a factor of the head count, and the bandwidth requirement with them
  • Much faster to decode, at “only minor quality degradation” from the multi-head baseline — reported qualitatively, not bounded, which is the claim GQA later found room in

Standing in the anthology

The source of SOTA-023 and SOTA-024.

Read in full for #114; the reading is NOTE-016. Both practices are confirmed, and both were already Superseded by GQA before the reading — which is unaffected.

A Superseded practice can still be checked, and these were right. The paper is sound, the practices reported it accurately, and something better arrived later. That is the benign case the two-scheme split exists for (ADR-002).

Backlinks