Frantar, Castro, Chen, Hoefler, Alistarh, ISTA, Universidade da Coruña and ETH Zürich (2024) — ARXIV-2408.11743
Key takeaways
- The question: weight-only quantization clearly speeds up single-user decoding, which is memory-bound. Does it survive batching, as the arithmetic per weight loaded grows?
- The answer is a roofline. A 4-bit kernel can stay memory-bound, and so keep close to the full 4× speed-up, until the batch makes the matmul compute-bound. On an A10 that is around batch 16–32 for large layers, after which the speed-up decays toward 1.5× at batch 128
- How: asynchronous global loads, careful shared-memory and warp layouts, pipelining, and dequantization straight into the tensor-core register layout. This requires a bespoke weight layout, which is what “Marlin format” names
- End-to-end: integrated in vLLM, the same curve on four GPU classes (Table 2): 2.3–3.2× single-GPU at batch 16 or below, 1.1–1.2× at 128, less when sharded (1.38× at batch 1 for Llama-2-70B on eight A100s). About 2.8× lower time per output token in a serving benchmark, and 3.3× with 2:4 sparsity. The sparse model’s accuracy beats the FP16 baseline only because it was further distilled on synthetic data
Standing in the anthology
Filed from #163 (“marlin format”). It extends GPTQ (LIT-081), whose
quantized weights it serves, from the same first author. It sources
SOTA-340.
What travels and what the practice keeps. “2.8×” is the travelling number, and it is a latency figure from one serving benchmark. The more useful result is the shape of the curve: the speed-up is set by whether the layer is memory-bound, which depends on batch size and GPU. The practice is stated in those terms.
Read — NOTE-305.