Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Inference optimization — serving-time decisions — batching, cache layout, compression, sparsity, distillation, sampling algorithms.
36 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-005 | Use running statistics for inference | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-023 v2 | Use multi-query attention for decoder-only models to reduce memory bandwidth | Shazeer (2019), LIT-024 — ARXIV-1911.02150. | Superseded — by SOTA-109 |
| SOTA-105 | PagedAttention to accelerate batch inference for LLM sampling | Kwon et al. (2023), LIT-112 — ARXIV-2309.06180. | Active |
| SOTA-113 v2 | Use continuous batching for inference | Kwon et al. (2023), LIT-112 — ARXIV-2309.06180. | Active |
| SOTA-115 | Overlap prefill and decode compute | Agrawal et al. (2023), LIT-103 — ARXIV-2308.16369. | Active |
| SOTA-163 v2 | Quantize with block-scaled microscaling formats rather than one scale per tensor | Rouhani et al. (2023), LIT-197 — an MX block is 32 elements sharing one E8M0 scale, so applying the scale is an exponent adjustment rather than a multiply. Shrinking the scope of the scale contains outliers locally instead of letting a few large values cost every small one its precision. 6-bit MX trains large transformers to FP32 accuracy with no recipe change. | Active |
| SOTA-185 | Quantize weights after training by compensating each rounding error into the columns not yet quantized | Frantar et al. (2022), LIT-081 — ARXIV-2210.17323. Round one column at a time and push the resulting error into the remaining columns using approximate second-order information, instead of rounding every weight independently to the nearest level. | Active |
| SOTA-227 v2 | Decode with a draft model and an accept-reject rule, which is exactly lossless | Autoregressive decoding runs one serial model pass per token and each pass is bounded by streaming the weights, so the arithmetic units idle. Have a cheap draft model guess the next gamma tokens, score all gamma + 1 positions in one target pass, and keep a prefix under an accept-reject rule built so the output distribution is the target’s exactly. 2x-3x at 11B and 2-2.5x at 70B, with no retraining, no architecture change and nothing traded away. | Active |
| SOTA-228 | Pick the inference partitioning from where the bottleneck is, and expect it to move between prefill and decode | Pope et al. (2022), LIT-110. Serving is not one workload: prefill parallelises over the prompt, decode is serial over the output, and the two are bounded by different things. Worse, the binding constraint moves — at small batch and short context it is loading the weights, and at large batch and long context it is loading the KV cache, which on a 500B model at batch 512 and 2048 tokens reaches 3 TB, three times the parameters. Choose the layout from a cost model of the regime you are in, and re-choose it when the regime changes. | Active |
| SOTA-229 | Scale the draft model’s training data, once nothing constrains it to predict the target’s features | Draft models had stopped improving with more training data, and LIT-185 identifies why: EAGLE trained its draft to predict the target’s hidden features and read tokens off them, which supplies multi-step training signal and caps expressiveness. Remove the constraint — predict tokens directly, fuse features from several depths, and simulate the multi-step draft during training — and the speedup rises with the data instead of flattening. Up to 6.5x, about 1.4x over the previous version, on roughly 8x the data. | Proposed |
| SOTA-269 | Keep a knowledge base outside the weights; parameters are an expensive and lossy place to memorise facts | Lu et al. (2024), LIT-452 — fact capacity is linear in model size and saturates in epochs, and the extrapolation is damning: all of Wikidata would want about 1000B non-embedding parameters trained for 100 epochs. Derivable facts cost full price unless they happen to share direction and structure. Borgeaud et al., LIT-060, is the record’s evidence that the alternative works at scale. | Active |
| SOTA-280 | Sample several reasoning paths and take the majority answer rather than decoding one greedily | Wang et al. (2022), LIT-468 — replace greedy decoding with sample-and-marginalize: GSM8K +17.9, SVAMP +11.0, AQuA +12.2. Five or ten paths recover most of it, which is the whole cost question. | Active |
| SOTA-289 v2 | When the sampling budget is small, prefer uniform-state discrete diffusion with consistency distillation — masked diffusion cannot revise what it has already emitted | Proposed | |
| SOTA-296 | Add the parallel streams in a short final training stage, not from the start | Chen et al. (2025), LIT-486 — parallel scaling costs P× the training FLOPs, which is what would otherwise make it unaffordable. Train 1T tokens the ordinary way, then switch the streams on for 20B tokens, 2% of the budget. The loss spikes when the random prefixes appear and recovers within 0.0002T tokens, and the logarithmic gains in P are the same ones the from-scratch runs show. | Proposed |
| SOTA-297 | Scale parallel computation with learnable input transforms, not parameters, when inference memory is the binding constraint | Chen et al. (2025), LIT-486 — run the same weights over P learnably-prefixed copies of the input and learn the aggregation. Loss falls as if the parameters had grown by O(log P); a 1.6B model at P = 8 matches a 4.4B one on code. At batch size 1 that costs 22× less added memory and 6× less added latency than the parameter scaling it replaces — and P× the training FLOPs. | Proposed |
| SOTA-302 v2 | Steer a distilled generator by modulating its input noise, not by fine-tuning its weights | Eyring et al. (2025), LIT-491 — train a LoRA hypernetwork to predict an improved initial noise for a frozen step-distilled generator. GenEval on SANA-Sprint goes 0.70 → 0.75 for 0.1 s of added latency, recovering about half of what 30-second test-time optimization buys. Reward fine-tuning the same model instead takes it 0.73 → 0.62: the anchoring KL term is intractable in weight space and tractable in noise space. | Proposed |
| SOTA-314 v2 | Absorb quantization outliers into a high-precision low-rank branch taken from the weights, and fuse its kernels into the low-bit ones | Li et al. (2024), LIT-512 — at 4 bits on both weights and activations, smoothing alone is not enough. Shift outliers from activations into weights, peel the dominant singular values into a 16-bit rank-32 branch, quantize the residual. Then fuse: run naïvely the branch costs 57% latency and cancels the win; fused it gives 3.0× over W4A16 and 3.5× memory on 12B FLUX.1. | Active |
| SOTA-317 | Do not rank singular directions by magnitude when deciding what to discard; check the small end, and check it after fine-tuning | Staats, Thamm and Rosenow (2024), LIT-517 — in a transformer’s non-square matrices, the smallest singular directions carry data directions and their removal is catastrophic: Llama-3 8B on GSM8K falls from 43.2% to 2.0% when the smallest decile of the Down-Projection goes, against 40.0% for the same decile of the square Attention-Output. And the damage only appears if you prune after fine-tuning. | Active |
| SOTA-318 | Choose the rank per matrix when you compress a transformer, because low-rank structure varies by component and by depth | Jaiswal et al. (2024), LIT-516 — “transformer weights are low rank” is true of some matrices and false of others. Query, Key and MLP Gate converge to low rank; MLP Up, MLP Down and Value do not; middle blocks resist while the first and last few give way. Setting the rank per matrix instead of globally is ~6.4× better in perplexity than uniform reduction at 30% on LLaMA-2 7B and ~47× at 40% on 13B. | Active |
| SOTA-327 | Train retrieval embeddings with nested losses, so a prefix of the vector can shortlist and the full vector re-rank | Kusupati, Bhatt, Rege et al. (2022), LIT-547 — put the loss on log(d) nested prefixes of the embedding. Each prefix then matches a separately trained model of that width (ResNet50, ImageNet), and one database supports shortlisting on 16 dimensions and re-ranking on 2048: equal mAP@10, 14× faster. Measured on image retrieval. Widely shipped in text embedding models without an independent comparison in the record. | Proposed |
| SOTA-328 | Train one nested model instead of a family of separately trained sizes, and extract the sizes you need from it | Devvrit, Kudugunta, Kusupati et al. (2023), LIT-546 — nest four FFN widths in one Transformer, train one width per step, and extract any size in between by choosing widths per layer. Up to 850M parameters, at the compute of training the four sizes separately, the largest matches its baseline and the smaller ones beat theirs. It pays only if you were going to train the family. Gemma 3n ships it. | Proposed |
| SOTA-330 | Enforce output structure by masking logits against a grammar, with masks precomputed per automaton state and built on the CPU while the GPU runs the forward pass | Dong et al. (2024), LIT-552 — split the vocabulary by whether a token’s validity depends only on the grammar automaton’s current node (over 99% of tokens for JSON). Precompute those per node, check the rest against the full stack at runtime, and build each mask on the CPU during the GPU forward pass. Mask time is 0.018 ms per token after the ablated optimizations, and end-to-end overhead is 0.1–0.2 ms. It guarantees syntax. It does not measure whether answers stay correct. | Active |
| SOTA-340 | Expect weight-only 4-bit quantization to speed up batched serving only while the batch keeps the matmul memory-bound, and use a kernel built to stay there | Frantar et al. (2024), LIT-567 — weight-only quantization speeds up decoding by loading fewer bytes, so the gain lasts only while the layer is memory-bound. A kernel designed to stay memory-bound (MARLIN) holds about 3.9× over FP16 up to batch 16–32 on an A10, decaying toward 1.5× at 128. End-to-end in vLLM, single-GPU, 2.3–3.2× at batch 16 or below and 1.1–1.2× at 128, on four GPU classes. Pick the format for the batch you actually serve, and pay for it in accuracy only where it buys speed. | Proposed |
| SOTA-355 v2 | Isolate the emergent outlier feature dimensions into 16-bit and quantize the rest to int8 | Dettmers et al. (2022), ARXIV-2208.07339. Quantize the projections vector-wise to int8, but pull the handful of systematically large feature dimensions out into a separate 16-bit multiplication — 99.9% of values in 8-bit, no measured quality cost up to 175B. | Active |
| SOTA-356 | Protect the salient weight channels by scaling them before rounding, choosing them from the activations rather than the weights | Lin et al. (2023), ARXIV-2306.00978. Identify the ~1% of weight channels whose activations are largest, scale them up by an equivalent transformation before rounding, and quantize everything uniformly — rather than keeping the important channels in higher precision. | Active |
| SOTA-392 v3 | To get a one-step sampler from a rectified flow, distil it; treat reflow as an optional extra pass, and keep the pre-reflow model for many steps | Liu, Gong and Liu (2022), LIT-636. On CIFAR-10, one-step FID is 378 from the base model, 6.18 after distillation alone, 12.21 after one reflow, and 4.85 after reflow and distillation (Table 1a). Distillation does most of the work. Reflow adds a further gain at the cost of a second training pass, and worsens many-step quality from 2.58 to 3.36. InstaFlow (LIT-796) contests it at text-to-image scale, starting from a diffusion model rather than a rectified flow. | Proposed |
| SOTA-394 v3 | When distilling a causal few-step video generator, take the teacher bidirectional, not causal | Yin et al. (2024), LIT-631. A block-causal 4-step student distilled from a bidirectional teacher beats one distilled from a causal teacher, at the same initialization, data and step count (Table 4). The causal teacher is also the weaker model before distillation, so the result shows the arrangement works and does not isolate why. | Proposed |
| SOTA-397 v2 | To condition an unconditionally trained diffusion model on known frames, use reconstruction guidance, not replacement | Ho, Salimans et al. (2022), LIT-627. Extending a video autoregressively from a model trained without conditioning, reconstruction guidance (steering each denoising step toward consistency with the known frames) gives FVD 136 against 451 for replacing the known frames’ noisy latents (Table 6). The model and guidance weight are held fixed. | Proposed |
| SOTA-410 v2 | For guided sampling, use a second-order multistep solver on the data-prediction parameterization, not a higher-order solver on the noise prediction | Lu et al. (2022), LIT-676. SOTA-203’s higher-order solver inverts under guidance: at scale 8.0 and 10 evaluations, FID is 13.04 for first-order DDIM against 114.62 at order 2 and 164.74 at order 3. Sample instead with a second-order multistep solver parameterized on the data prediction — which is also what makes SOTA-202’s per-step clamping available at all, since you cannot clamp x̂₀ if the solver’s state is ε̂. 20 evaluations then beat DDIM at 250. | Active |
| SOTA-411 | To let a model spend more compute at inference without more parameters, train a depth-recurrent core on a randomly sampled iteration count, re-injecting the input at every iteration | Geiping et al. (NeurIPS 2025), LIT-683. Put a small core block between a short prelude and coda. At each training step sample how many times to iterate it, from a heavy-tailed distribution with mean 32. Backpropagate through the last few iterations only. Feed the input embedding into every iteration, and start the state from noise. Trained this way, accuracy rises with iterations at test time and saturates per task. A model trained at fixed depth does not gain this. Nothing yet shows it beats a standard transformer of the same FLOPs. | Proposed |
| SOTA-414 | To make an image diffusion model a dense geometric predictor, fine-tune it end to end as a one-step model at t = T with a task loss | Martin Garcia et al. (WACV 2025), LIT-687. Fix the timestep at T, feed zeros as the noise, decode the prediction through the frozen VAE, and train on the task’s own loss: affine-invariant L1 for depth, angular error for normals. From 20K iterations on 74K synthetic images, the one-step model beats Marigold’s 50-step, 10-member ensemble on depth AbsRel on all five zero-shot sets, and on normals by 2.6–3.0° of mean error. Starting from plain Stable Diffusion is nearly as good. | Proposed |
| SOTA-416 v2 | When sampling a diffusion model in few steps, start at t = T: use trailing, not leading, timestep spacing | Martin Garcia et al. (WACV 2025), LIT-687. DDIM’s “leading” spacing never visits the final timestep. So at one step, a pure-noise input is labelled with a timestep that claims an almost clean sample. With the same weights and only the spacing changed, Marigold’s single-step depth goes from noise to NYUv2 AbsRel 5.7. The mismatch vanishes as the step count approaches T, which is why it goes unnoticed at 50 steps. | Active |
| SOTA-420 | When distilling a classifier, match the teacher’s temperature-softened outputs at the same temperature and keep a down-weighted hard-label term | Hinton et al. (2015), LIT-680. Soft-target cross-entropy at temperature T in both teacher and student, plus a hard-label cross-entropy at T = 1 with a considerably lower weight. Multiply the soft term by T², so the balance does not move when T changes. Start low for a small student. The paper’s best realistic setting was T = 2, and a 30-unit MNIST net wanted 2.5–4. | Proposed |
| SOTA-423 | When classifier-free guidance over-exposes a latent diffusion model, rescale the guided prediction toward the conditional prediction’s standard deviation | Lin et al. (WACV 2024), LIT-689. Compute x_cfg as usual, rescale it by std(x_pos)/std(x_cfg), and blend with weight φ back toward x_cfg. The source uses φ ≈ 0.7 at w = 7.5. It is the latent-space counterpart of Imagen’s dynamic thresholding, which needs a known pixel range. The evidence is qualitative, and part of the effect may simply be lower effective guidance. | Proposed |
| SOTA-433 | Guide a diffusion model with a smaller, less-trained copy of itself rather than an unconditional model, and make sure the two models are degraded in the same way | Karras et al. (2024), LIT-721. Replace classifier-free guidance’s unconditional reference with a degraded copy of the same conditional model — smaller and under-trained, same task, same conditioning, same data. The quality gain survives and the diversity loss does not: ImageNet-512 FID 2.56 → 1.34 (EDM2-S), 1.25 (XXL), 1.01 at ImageNet-64, and unconditional EDM2-S 11.67 → 3.86, which CFG cannot do at all. The binding condition is compatibility: degrade the two models differently and the method stops working entirely. | Proposed |
| SOTA-447 | When distilling a large image or video model by continuous-time consistency, add a small DMD distribution-matching term | Zheng et al. (2025), LIT-783. Pure sCM distillation of 2B–14B text-to-image and text-to-video teachers loses small text, texture and temporal coherence. Adding DMD’s loss at weight 0.01, on the student’s own 1–4-step rollouts with a fake-score network updated 5–10 times per step, fixes it in samples. Four-step Wan2.1-1.3B VBench is 84.43 against a re-implemented DMD2’s 84.56. sCM’s own Fig. 7 (LIT-790) is why the consistency term stays primary: distribution matching alone loses recall. | Proposed |