Pope et al. (2022) — ARXIV-2211.05102
Key takeaways
- The core move is to write down an analytical cost model for the partitioning of a transformer across chips — weights, activations and KV cache each with their own memory and communication terms — and then select the layout that model says is best, instead of reusing the training layout.
- The best layout is not one layout. A latency-optimized target and a throughput-optimized target want different partitionings of the same model, and the paper gives both alongside the regime where each wins.
- Memory bandwidth, not arithmetic, is the binding constraint in decode, and the KV cache is the term that grows with batch and sequence — which is what makes cache layout and access pattern the thing to optimize.
- Multiquery attention is analyzed as a way of shrinking that term, enabling much longer contexts at a given memory budget.
Standing in the anthology
The source of SOTA-228; the reading is NOTE-162.
The previous version of this section said “Nothing in the record cites this note”, and then set out exactly why something should: it is the analytical backing for serving practices sourced elsewhere in the corpus — SOTA-113 on continuous batching (LIT-112), SOTA-115 on prefill/decode overlap (LIT-103), SOTA-105 on paged attention (LIT-112) — all of them answers to bottlenecks this paper located. The diagnosis sat here, correct and unacted on, until the reverse coverage query under #87 went looking for papers supporting no practice. That is DP-007’s shape with the unusual feature that the record had already written it down.