Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Numerics and precision — how many bits, where, and what that costs — number formats, training precision and the failures it causes, post-training quantization, and the interaction between them.
23 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-013 | Use dynamic loss scaling that doubles every 2000 successful steps | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-014 | Maintain master weights in FP32 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-015 v2 | Store optimizer states in FP32 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-016 | Perform forward/backward passes in FP16 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-043 | Use mixed precision during data loading | Mohan et al. (2020), LIT-050 — ARXIV-2007.06775. | Active |
| SOTA-118 | Employ mixed precision to reduce memory usage | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-134 v2 | Gate each attention head’s output with a sigmoid after the scaled dot-product | Qiu et al. (2025), LIT-138 — the best of 30 gating variants at 15B MoE and 1.7B dense over 3.5T tokens: better quality, more stable training, larger tolerable learning rates, no attention sinks; shipped in every Qwen full-attention layer since Qwen3-Next. | Active |
| SOTA-158 | Bound the activation’s output range when training in low precision | Jiang et al. (2026), LIT-200 — SwiGLU approximates x² for large positive inputs, which is where its expressive capacity comes from and what enlarges the output range and exacerbates outliers. Two groups independently concluded that is intolerable in low precision. What they agree on is the constraint; the functions they chose differ. | Proposed |
| SOTA-159 | Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling | Blake et al. (2024), LIT-149 — µP makes activation scale independent of model size; unit scaling makes activations, weights and gradients start at a scale of one. Together the model is size-independent and well-scaled from the first step, the hyperparameters become close to independent so a one-dimensional sweep replaces a joint search, and FP8 works without loss-scaling tricks. | Proposed |
| SOTA-160 | Treat the pretraining token budget and the post-training quantization plan as one decision, not two | Kumar et al. (2024), LIT-186 — degradation from post-training quantization increases with how much data the model was trained on, to the point where additional pretraining data becomes actively harmful if the model will be quantized afterwards. Fit over 465 runs into one precision-aware scaling law covering both pre- and post-training quantization. | Proposed |
| SOTA-161 v4 | Keep the attention output in FP32 during training, because flash attention’s rounding bias compounds instead of cancelling | Qiu et al. (2025), LIT-198 — putting flash attention in BF16 while the FFN goes to FP8 sometimes explodes, and the cause is two things at once: attention produces low-rank updates repeated across steps and tokens, and low-precision addition rounds with a bias. The biased error rides the repeated update and compounds into a systematic gradient bias instead of averaging out. | Active |
| SOTA-163 v2 | Quantize with block-scaled microscaling formats rather than one scale per tensor | Rouhani et al. (2023), LIT-197 — an MX block is 32 elements sharing one E8M0 scale, so applying the scale is an exponent adjustment rather than a multiply. Shrinking the scope of the scale contains outliers locally instead of letting a few large values cost every small one its precision. 6-bit MX trains large transformers to FP32 accuracy with no recipe change. | Active |
| SOTA-185 | Quantize weights after training by compensating each rounding error into the columns not yet quantized | Frantar et al. (2022), LIT-081 — ARXIV-2210.17323. Round one column at a time and push the resulting error into the remaining columns using approximate second-order information, instead of rounding every weight independently to the nearest level. | Active |
| SOTA-230 | Quantize the frozen base to 4-bit and keep the adapters in 16-bit | Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314. Store the frozen base weights in 4-bit NormalFloat and dequantize to BFloat16 for every matrix multiply, training only 16-bit LoRA adapters. Fine-tuning a 65B model falls from >780GB to <48GB with no measured loss against a 16-bit fully fine-tuned baseline, because the arithmetic never happens in 4 bits. | Active |
| SOTA-234 | Train in the target low-bit format from scratch rather than quantizing a finished model | Ma et al. (2024), LIT-380 — ARXIV-2402.17764. Decide the serving format before training and train in it, rather than training in FP16 and compressing afterwards. Ternary weights trained from scratch match an FP16 model of equal size and token budget from 3B upward — but the compute argument assumes hardware built for the format, and only the memory saving is measured on machines that exist. | Proposed |
| SOTA-248 | Stretch and clip the softmax so an attention head can output exact zeros | Bondarenko et al. (2023), LIT-414 — a head that wants to do nothing has to drive its softmax input to infinity to approximate exact zeros, and that is what creates the activation outliers that break INT8. Stretch the softmax to (γ, ζ) and clip back to (0,1) and zeros become reachable from a finite input. On BERT-base: W8A8 perplexity 1294 → 4.55, max infinity-norm 735 → 20, and the FP16 model gets slightly better rather than worse. Evidence is BERT-base, OPT-125M and ViT-S/16 — small and encoder-heavy. | Proposed |
| SOTA-283 | To fine-tune a model that is already quantized, bank the part of each update that is smaller than the lattice spacing, and rebuild the accumulator from seeds rather than storing it | Proposed | |
| SOTA-314 v2 | Absorb quantization outliers into a high-precision low-rank branch taken from the weights, and fuse its kernels into the low-bit ones | Li et al. (2024), LIT-512 — at 4 bits on both weights and activations, smoothing alone is not enough. Shift outliers from activations into weights, peel the dominant singular values into a 16-bit rank-32 branch, quantize the residual. Then fuse: run naïvely the branch costs 57% latency and cancels the win; fused it gives 3.0× over W4A16 and 3.5× memory on 12B FLUX.1. | Active |
| SOTA-340 | Expect weight-only 4-bit quantization to speed up batched serving only while the batch keeps the matmul memory-bound, and use a kernel built to stay there | Frantar et al. (2024), LIT-567 — weight-only quantization speeds up decoding by loading fewer bytes, so the gain lasts only while the layer is memory-bound. A kernel designed to stay memory-bound (MARLIN) holds about 3.9× over FP16 up to batch 16–32 on an A10, decaying toward 1.5× at 128. End-to-end in vLLM, single-GPU, 2.3–3.2× at batch 16 or below and 1.1–1.2× at 128, on four GPU classes. Pick the format for the batch you actually serve, and pay for it in accuracy only where it buys speed. | Proposed |
| SOTA-355 v2 | Isolate the emergent outlier feature dimensions into 16-bit and quantize the rest to int8 | Dettmers et al. (2022), ARXIV-2208.07339. Quantize the projections vector-wise to int8, but pull the handful of systematically large feature dimensions out into a separate 16-bit multiplication — 99.9% of values in 8-bit, no measured quality cost up to 175B. | Active |
| SOTA-356 | Protect the salient weight channels by scaling them before rounding, choosing them from the activations rather than the weights | Lin et al. (2023), ARXIV-2306.00978. Identify the ~1% of weight channels whose activations are largest, scale them up by an equivalent transformation before rounding, and quantize everything uniformly — rather than keeping the important channels in higher precision. | Active |
| SOTA-399 | Choose pre-training weight decay, dropout, clipping and precision for the quantization you intend to ship | Proposed | |
| SOTA-437 | Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base | Hershcovitch et al. (2024), LIT-750 — ZipNN. In regular trained weights only the float exponent is redundant: about 40 of its 256 values occur and it codes to about a third, while sign and mantissa barely compress. So split the exponent into its own stream and Huffman-code it without an LZ stage. Llama-3.1 BF16 goes to 66.4% against Zstd’s 77.7%, and is faster in both directions. Between checkpoints, XOR deltas against a base up to 10 checkpoints back compress better than standalone files. Sizes and throughputs are measured. Stall time inside a training run is not. | Proposed |