Agrawal et al., Microsoft Research India (2023) — ARXIV-2308.16369
Key takeaways
- The asymmetry that drives everything: prefill processes the whole prompt and saturates GPU compute even at small batch sizes, while decode emits one token per request per step and leaves the machine mostly idle. Serving both in the same system means one phase is always wasting the hardware.
- Chunked prefills split a prefill request into equal-sized chunks; decode-maximal batching then builds each batch from one prefill chunk plus as many decodes as fit. The chunk saturates compute and the decodes piggyback, costing up to an order of magnitude less than they would in a decode-only batch.
- Chunking a single prefill into several batches is what makes the coverage work — more batches to hide decodes in — and it makes batches uniform in cost, which is the second win: under pipeline parallelism the micro-batch imbalance between prefill and decode is what creates bubbles.
- LLaMA-13B on an A6000: up to 10× decode throughput and 1.33× end-to-end. LLaMA-33B on an A100: 1.25× end-to-end, 4.25× decode. GPT-3 with pipeline parallelism: bubbles reduced 6.29×, end-to-end throughput 1.91×.
Standing in the anthology
The source of SOTA-115, overlap prefill and decode compute. It reached the record under a fabricated citation naming CUDA kernels, which is the wrong level: the win here is in how requests are batched, and it needs no custom kernel at all.
The record’s serving practices now sit on their real sources — continuous batching on vLLM (LIT-112), fused attention on FlashAttention (LIT-074), and prefill/decode interleaving here.