anthology-of-the-sota

LIT-100: GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints

1 min read

Properties6
StatusActive
Version2
Tagsattention-techniques
Cited as “introduced by” bySOTA-109 — Prefer GQA to MQA or MHA
Cited as “paper” byNOTE-021 — Grouped-Query Attention
Cited as “source” bySOTA-109 — Prefer GQA to MQA or MHA

Graph View

Ainslie et al. (2023) — ARXIV-2305.13245

Key takeaways

  • Two contributions, and the record only holds one of them
  • Grouped-query attention: an intermediate number of key-value heads — more than one, fewer than the query heads — generalising MQA, which is the g = 1 endpoint, and multi-head, which is g = h
  • Uptraining: convert an existing multi-head checkpoint to fewer key-value heads for 5% of original pretraining compute, by mean-pooling the key and value projections within each group and continuing briefly
  • Uptrained GQA reaches quality close to multi-head at speed comparable to MQA — the result the record’s models are built on
  • The motivation is explicitly that MQA costs quality and “it may not be desirable to train a separate model just for faster inference”

Standing in the anthology

The source of SOTA-109.

Read in full for #114; the reading is NOTE-021. The practice is confirmed.

Uptraining is not represented in the practice registry, and is arguably the more portable of the two ideas: an inference-side architectural property need not be committed to before pretraining, if a cheap conversion exists. The record holds several pretraining-time commitments where the same question could be asked and has not been.

Backlinks