Source
Frantar et al. (2024), LIT-567 — MARLIN. Read, as NOTE-305. The quantization itself is SOTA-185 (GPTQ).
The practice
- Know which side of the roofline you serve on. At small batch, decoding is memory-bound and 4-bit weights give close to 4× on the matmuls. As the batch grows, the same weights are reused across more rows, the matmul becomes compute-bound, and the advantage shrinks
- Use a kernel designed to delay the crossover. A generic dequantize then FP16 GEMM loses the speed-up early. MARLIN keeps near 3.9× to batch 16–32 on an A10 by pipelining loads and dequantizing straight into the tensor-core layout, which needs its own weight format
- At high batch, do not expect much from weight-only quantization. About 1.5× at batch 128 in the paper’s layer benchmark, and 1.1–1.2× end-to-end in vLLM (Table 2). Weight-and-activation quantization targets that regime
- Remember what the speed-up costs. The INT4 model the paper serves loses 3.3 points of mean accuracy (4.3 on MMLU) against FP16 (Table 1). A gain that has shrunk to 1.2× at your serving batch may not be worth that
The batch-size framing and the kernel both come from Frantar et al.’s MARLIN paper, LIT-567. Its layer benchmark (Figure 1, a 72k×18k layer on an A10) holds close to the 3.87× ideal to batch 16–32 while ExLlamaV2, AWQ, bitsandbytes and torch kernels degrade quickly after batch 1, and its roofline (Figure 11) puts the crossover below batch 64. Its vLLM runs (Table 2) are the end-to-end numbers above: for Llama-2-7B on an A10, 2.93× at batch 1, 2.74× at 16 and 1.20× at 128.
Conditions
- The crossover depends on GPU and layer shape. The headline curve is one large layer on an A10
- Sharding shrinks it. Split over eight A100s, Llama-2-70B gains 1.38× at batch 1, against 2.55× on two
- Kernel authors’ own benchmarks, on Ampere-class GPUs