Yu et al. (2022) — OSDI ‘22
Key takeaways
- The problem is request-granularity scheduling. Generation is multi-iteration: each forward pass emits one token per request. Existing servers fixed the batch for the whole request, so a request that finished early could not return to the client, and a newly arrived one waited for the entire batch to drain
- Iteration-level scheduling is the fix: the scheduler invokes the engine to run a single iteration on the batch, then re-decides. Finished requests leave, waiting requests join, every step
- Selective batching makes that compatible with a Transformer. Requests at different positions cannot be batched uniformly across every operation, so batching is applied only to the operations where it is valid
- 36.9× throughput improvement over NVIDIA FasterTransformer at the same latency, on GPT-3 175B
- Designed for scale-out to models of hundreds of billions of parameters
Standing in the anthology
The source of SOTA-113.
Filed in #114 because it was missing. SOTA-113 — use continuous batching for
inference — had been sourced to vLLM (LIT-112), which describes
iteration-level scheduling in its background section and cites this paper
for it. vLLM’s contribution is PagedAttention; the batching this practice
recommends is Orca’s.
“Continuous batching” is the name the field settled on and is not this paper’s term, which is why the citation drifted: the practice is filed under a name that appears in neither paper.