inference-optimization.
28 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-016 | Fast Transformer Decoding: multi-query attention | Incremental decoding is bounded by the memory bandwidth of reloading the keys and values, not by arithmetic. Share one key/value head across all query heads: much faster decoding, “only minor quality degradation”. | Read |
| NOTE-023 | PagedAttention and vLLM | KV cache memory is huge, grows and shrinks dynamically, and was managed as one contiguous block — so fragmentation and duplication capped the batch size. PagedAttention borrows OS paging: near-zero waste, sharing within and across requests, 2–4× throughput at equal latency. | Read |
| NOTE-159 | Accelerating Large Language Model Decoding with Speculative Sampling | The same construction as NOTE-160, reached independently two months later and demonstrated at 70B in a distributed setup. Its contribution to the record is corroboration plus two sharper statements: the guarantee holds “within hardware numerics” rather than absolutely, and the achieved tokens per second exceed the memory-bandwidth ceiling that bounds autoregressive decoding — which is the family’s premise, measured rather than assumed. | Read |
| NOTE-160 | Fast Inference from Transformers via Speculative Decoding | Autoregressive decoding is serial and memory-bandwidth-bound, so the arithmetic units sit idle while weights stream. Speculative decoding spends that idle compute: a cheap draft proposes gamma tokens, the target scores all gamma + 1 positions in one parallel pass, and an accept-reject rule keeps a prefix. The rule is constructed so the output distribution is exactly the target’s, which is what makes this a free speedup rather than a quality trade. | Read |
| NOTE-162 | Efficiently Scaling Transformer Inference | An analytical model of where inference time actually goes, used to pick a partitioning rather than searching for one. Its organising finding is that prefill and decode have different performance characteristics and must be analysed separately, and that the dominant cost moves — weights at small batch and short context, the KV cache at large. On PaLM 540B across 64 TPU v4 chips: 29 ms per token at low batch with int8 weights, 76% MFU at large batch, and 32x longer context from multiquery attention once the partitioning is right. | Read |
| NOTE-163 | EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test | EAGLE predicted the target model’s features and then read tokens off them. That constraint is why feeding it more training data stopped helping. Drop it — predict tokens directly, fuse low, mid and high features instead of reusing the top layer, and simulate the multi-step draft during training so step two sees its own step-one output. The result is a scaling law for inference acceleration: more draft data, proportionally more speedup, up to 6.5x. And it raises throughput 1.38x at batch 64, where speculation is supposed to stop paying. | Read |
| NOTE-217 | Self-Consistency | Sample several chains, take the majority answer. Reading it: it is a decoding change rather than a prompt change, so it composes with everything else in the topic — and it counts answers, not reasoning, so it does not make the traces trustworthy. | Read |
| NOTE-228 v2 | The Diffusion Duality | Reading it: the theorem is elegant and the ablation says it accounts for under half the likelihood gain. What the paper actually establishes is a regime split — masked diffusion wins on quality at many steps, uniform-state wins below about 32, and the reason is architectural rather than tuned. | Read |
| NOTE-235 | ParScale | Read as a candidate third scaling axis. The scaling law is fitted honestly and the inference-cost analysis is better instrumented than most; the claim that carries furthest is the asymmetry — code benefits more than general text, on two independent measurements — and it is the claim the paper labels a conjecture. | Read |
| NOTE-240 | Noise hypernetworks | Read as the amortized counterpart to SOTA-301. The headline — recover half of test-time optimization at 1/100th the latency — is well measured. The finding that carries further is the control: direct reward fine-tuning of the same model makes it worse, and worse the more steps you sample. | Read |
| NOTE-256 | Image-GS: the rate accounting GaussianToken skipped, and an ablation that inverts its own headline | Read from the #180 worklist for its adjacency to the tokenizer trunk. Two things worth keeping: a comparison run the way this record keeps asking for — six baselines at matched model size, bitrate on the axis — and an ablation in which the two unglamorous optimizer choices outweigh the content-adaptive allocation the paper is named for. | Read |
| NOTE-257 | SVDQuant: a low-rank branch on the weights, not on the error | Read from the #180 worklist. The idea that travels is an ordering: decompose the weights and quantize the residual, rather than quantize and then patch the error. Two propositions and Eckart-Young say why, and a published method that does it the other way round is the control. | Read |
| NOTE-263 | An MP bulk plus outliers at both ends, and why magnitude order is not importance order | Read to answer THEORY-059’s open question. It answers it and complicates it: language-transformer weight spectra are steep at the top, as the account needs, but the bottom of the spectrum in non-square matrices holds directions whose removal is catastrophic. Frobenius magnitude and functional importance are ordered differently, and a method that peels the top and quantizes the rest is keeping exactly the part this paper says is fragile. | Read |
| NOTE-264 | Low-rank structure in a transformer is non-uniform, and a global rank is measurably the wrong choice | Read second in the spectral cluster, and it is the one that carries a recommendation. “Transformer weights are low rank” turns out to be true of some matrices and false of others, with the split systematic by component and by depth. The comparison against a uniform rank at matched compression was run and the margins are large. | Read |
| NOTE-273 | The observation outlived the method, and it is the observation the record needed | Read for one figure. The record holds two practices resting on attention being low rank and held no source for the claim; the record also already holds the verdict that this paper’s method was overtaken. Those are different things and the reading keeps them apart. | Read |
| NOTE-289 | MatFormer | One Transformer with four nested FFN widths, trained one width per step, gives four sizes plus Mix’n’Match sizes nobody trained. At the compute of training the four separately, it matches the largest and beats the smaller ones, which see up to 4× the data through shared weights. Read §1–5 and Appendices B, D.1, E.1 and F. | Read |
| NOTE-291 | Matryoshka Representation Learning | Sum the task loss over log(d) nested prefixes of one embedding, and each prefix matches a separately trained model of that width on ResNet50 and ImageNet. Retrieval that shortlists on 16 dimensions and re-ranks on 2048 is 14× faster at equal mAP@10. Read §1–6 and Appendices D.2 and I; the remaining appendix tables were skimmed. | Read |
| NOTE-293 | Gemma 3n | Per-Layer Embeddings: every token id indexes a table with one 256-d vector per layer, gated into the residual at each layer, about 2.35B parameters held off the accelerator. No measurements are published. Read from the model overview, the developer guide, and the Hugging Face transformers implementation, which is the only place the mechanism is fully specified. | Read |
| NOTE-294 | XGrammar | Grammar-constrained decoding made nearly free. It splits the vocabulary into tokens whose validity depends only on the automaton’s stack top, which are precomputed, and the under-1% that need the full stack, which are checked at runtime. The mask is built on the CPU during the forward pass. Each optimization is ablated. Output quality is measured only as syntactic validity. Read §1–6; the appendix was not read. | Read |
| NOTE-305 | MARLIN | An FP16×INT4 GEMM kernel that stays memory-bound, and so near the 3.87× ideal 4-bit speed-up, up to batch 16–32 on a large layer, decaying toward 1.5× at 128. End-to-end in vLLM the same shape holds on four GPU classes (Table 2): 2.3–3.2× single-GPU at batch ≤16, 1.1–1.2× at 128, and much less when the model is sharded over eight A100s. The INT4 model it serves loses 3.3 points of mean accuracy (Table 1). | Read |
| NOTE-335 | Self Forcing | Training a few-step autoregressive video model on its own rollouts, with a distribution-matching loss on the whole clip, beats teacher- and diffusion-forced training in matched few-step comparisons (VBench 84.31 against 82.76 and 82.32 chunk-wise), and does not drop when moved to frame-wise generation, where the baselines do. It runs at 17 fps with 0.69s latency on one H100, and its parity with the slower 1.3B base rests on a 14B teacher: the SiD variant, with a 1.3B teacher, scores 84.07 against the base’s 84.26. | Read |
| NOTE-337 | Rectified Flow | Regressing a velocity onto x1 − x0 along the straight interpolation gives an ODE that preserves the marginals and does not increase any convex transport cost, and retraining on its own couplings (“reflow”) provably straightens it at O(1/K). On CIFAR-10 the single-pass model scores FID 2.58 against 3.93 for the VP probability-flow ODE at the same architecture, with training budget unstated. Most of the one-step result comes from distillation (378 → 6.18 with no reflow), and reflow adds 6.18 → 4.85. | Read |
| NOTE-343 v2 | CausVid | Distilling a bidirectional video DiT into a 4-step block-causal student with distribution matching gives a streaming generator: 1.3s to the first frame and 9.4 fps, against 219.2s and 0.6 fps for the teacher. At equal initialization a bidirectional teacher makes a better causal student than a causal one (VBench 94.7 / 64.4 / 30.1 against 91.9 / 61.7 / 28.2). The causal teacher is also the weaker model, so the paper’s error-accumulation explanation is not isolated. | Read |
| NOTE-348 | Imagen Video | A seven-model pixel-space cascade whose one well-measured result is that guided progressive distillation to 8 steps per stage cuts sampling from 618s to 35s. The CLIP metrics cannot tell the two apart: they fall slightly under constant guidance, rise under oscillating guidance, and rate every sample above the real videos. Joint image-video training, the choice later reports copied, is asserted without a number. | Read |
| NOTE-350 v2 | LTX-Video | A 1.9B video DiT on a 32×32×8, 128-channel VAE with no patchifier (1:8192 pixels to tokens), whose decoder also does the last denoising step. It is fast and preferred over same-size 2024 open models in a 20-rater survey. The VAE has no reconstruction metric at all, and the denoising decoder, the paper’s central idea, rests on one unquantified internal study. | Read |
| NOTE-357 | SEDD | Score entropy makes the ratio view of discrete diffusion trainable and gives it a likelihood bound; with it, absorbing-state diffusion reaches a near-tie with a retrained autoregressive model on LM1B. “Beats GPT-2” is 3 of 5 zero-shot sets against a model trained on different data. The controlled result that lasts is that masking beats uniform corruption on every table. | Read |
| NOTE-360 | MDLM | Masked diffusion’s ELBO is a schedule-weighted average of masked-LM losses, and with a modern recipe it is the best discrete diffusion language model at 110M parameters. It does not beat autoregression in-domain, even with the AR baseline given half the optimizer steps; the paper’s own checklist names that as its limitation. | Read |
| NOTE-361 | DeepSeek-V4 | Two MoE models (1.6T/49B and 284B/13B) reach a native 1M-token context by interleaving two compressed attentions — 4x-compressed KV with top-k sparse selection (CSA) and 128x-compressed dense KV (HCA), each with a 128-token sliding-window branch — at 27% of V3.2’s per-token FLOPs and 10% of its KV cache for Pro at 1M. Multi-token prediction is kept “without modification” from V3 as a depth-1 training objective; the report never mentions speculative decoding or a draft model. | Read |