Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Numerics and precision — how many bits, where, and what that costs — number formats, training precision and the failures it causes, post-training quantization, and the interaction between them.
24 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-011 | Mixed Precision Training | Micikevicius et al. (2017), ARXIV-1710.03740. Enables FP16 training with FP32 master weights. | Active |
| LIT-081 v3 | GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers | Frantar et al. (2022), ARXIV-2210.17323. One-shot weight quantization to 3-4 bits: round column by column and push each rounding error into the columns not yet done, using approximate second-order information from a small calibration set. | Active |
| LIT-102 v3 | ZeRO++: Extremely Efficient Collective Communication for Giant Model Training | Rajbhandari et al. (2023), ARXIV-2306.10209. Communication optimization. | Active |
| LIT-139 v2 | DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence | DeepSeek-AI (2026), ARXIV-2606.19348. Two MoE models — V4-Pro at 1.6T/49B and V4-Flash at 284B/13B — with a 1M-token context reached by a hybrid of compressed sparse attention and heavily compressed dense attention, manifold-constrained hyper-connections widening the residual stream, and Muon as the optimizer for most modules over 32–33T tokens; at 1M tokens V4-Pro needs an estimated 27% of V3.2’s per-token FLOPs and 10% of its KV cache. Multi-token prediction is carried over from V3 unchanged, as a training objective; the report does not mention speculative decoding. | Active |
| LIT-149 | u-µP: The Unit-Scaled Maximal Update Parametrization | Blake et al. (2024), ARXIV-2407.17465. Combine µP with Unit Scaling so that activations, weights and gradients start at unit scale regardless of size: lower loss than plain µP, FP8 training out of the box, and hyperparameters that can be swept one at a time, with a learning-rate sweep alone reaching near-optimal loss. | Active |
| LIT-186 | Scaling Laws for Precision | Kumar et al. (2024), ARXIV-2411.04330. Training in low precision reduces a model’s effective parameter count, and post-training quantization degrades more the longer a model was trained — so past some point extra pretraining data is actively harmful. | Active |
| LIT-197 | Microscaling Data Formats for Deep Learning | Rouhani et al. (2023), ARXIV-2310.10537. The MX formats: a block of 32 elements sharing one 8-bit power-of-two scale, with the elements themselves in FP8, FP6, FP4 or INT8. Over two dozen benchmarks — 8-bit MX runs inference on FP32 checkpoints with no calibration, 6-bit matches FP32 for training weights, activations and gradients with no recipe change, and 4-bit weights cost only a minor drop. The first sub-8-bit training result of that kind. | Active |
| LIT-198 v2 | Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention | Qiu et al. (2025), ARXIV-2510.04212. The first mechanistic account of why training with flash attention in low precision explodes. Two things have to coincide: attention representations that are low-rank and similar across steps and tokens, and rounding error in BF16 addition that is biased rather than symmetric. The bias acts as a coefficient on the repeated low-rank update, so the errors accumulate instead of cancelling and the spectral norm of weights runs away. A minimal change to flash attention that de-biases the rounding stops it. | Active |
| LIT-200 v3 | PowLU: An Activation Function for Stable Pre-Training of LLMs | Jiang et al. (2026), ARXIV-2605.25704. For large positive inputs SwiGLU approximates x², and that quadratic amplification is what enlarges the output range and produces outliers, particularly in low-precision training. PowLU replaces it with a rational power function giving adaptive nonlinearity and bounded growth. Scaling-law experiments plus 7.9B and 124B Ling models against SwiGLU, and against SwiGLU-Clip at 7.9B only. | Active |
| LIT-221 | Train With Mixed Precision — NVIDIA deep learning performance guide | NVIDIA, the Train With Mixed Precision user guide. Where the dynamic loss scaling constants everyone runs actually come from: “we successfully trained networks with N = 2000, increasing scaling factor by 2, decreasing scaling factor by 0.5” — reported as one tested configuration, with “many other settings are valid as well.” | Active |
| LIT-378 | QLoRA: Efficient Finetuning of Quantized LLMs | Dettmers et al. (2023), ARXIV-2305.14314. Back-propagate through a frozen 4-bit base model into 16-bit LoRA adapters: 65B fine-tuning drops from >780GB to <48GB with no loss against a 16-bit fully fine-tuned baseline. Carries a second finding the quantization headline tends to bury — LoRA on the query and value projections alone does not reach full fine-tuning at scale, and the adapter COUNT, not the rank, is what closes the gap. | Active |
| LIT-380 | The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | Ma et al. (2024), ARXIV-2402.17764. Every weight is ternary, {-1, 0, +1}, and the model is trained that way from scratch rather than quantized afterwards. From 3B upward it matches an FP16 LLaMA of the same size and token budget on perplexity and end-task accuracy, while the matrix multiply reduces to integer addition — a compute paradigm rather than a compression ratio. | Active |
| LIT-414 | Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothing | Bondarenko et al. (2023), ARXIV-2306.12929, NeurIPS 2023. The diagnosis the attention-sink literature rests on, and two architectural fixes with the experiment attached. A head trying to learn a no-op must drive its softmax input ever larger to approximate exact zeros, which is what creates the outliers that break quantization. Clipped softmax and gated attention each remove them: BERT-base goes from 1294 W8A8 perplexity to 4.55, with max infinity-norm from 735 to 20. Its gated attention is a per-head sigmoid on the attention output — the recommendation SOTA-134 makes, two years earlier, as Qiu et al. themselves say. | Active |
| LIT-473 | Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost | Xu, Miikkulainen and Qiu (2026), ARXIV-2602.03120. Fine-tune a quantized model in its own integer space: accumulate the part of each evolution-strategies update that is smaller than the lattice spacing until it crosses a grid point, and rematerialize that accumulator from stored seeds so it costs kilobytes instead of an FP16 copy of the weights. Countdown, Qwen2.5 at 1.5B and 3B, INT4/INT8/W8A8. | Active |
| LIT-512 | SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models | Li et al. (2024), ARXIV-2411.05007 — 4-bit weights and activations for diffusion transformers. Shift outliers from activations into the weights by smoothing, peel the dominant singular values into a 16-bit rank-32 branch, quantize only the residual. Two propositions bound the output error by the magnitudes rather than only the rounding errors, which is why shrinking the residual works. Naïvely the extra branch costs 57% latency; fused, the system reaches 3.0× over a W4A16 baseline and 3.5× memory reduction on 12B FLUX.1. Read as NOTE-257. | Active |
| LIT-567 | MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models | Frantar, Castro, Chen, Hoefler, Alistarh (2024), ARXIV-2408.11743. An FP16×INT4 matrix-multiply kernel, the “Marlin format” being its quantization-specific weight layout. It keeps weight-only quantization’s memory-bandwidth speed-up through batched serving: about 3.9× over FP16 (near the 3.87× ideal for 4-bit, group 128) up to batch 16–32, falling toward 1.5× at 128 as the problem turns compute-bound. In vLLM, about 2.8× lower time per output token on an A6000, and 2.3–3.2× single-GPU end-to-end at batch 16 or below, falling to 1.1–1.2× at 128. | Active |
| LIT-585 | AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration | Lin et al. (2023), ARXIV-2306.00978. Weight-only low-bit quantization that protects the ~1% of channels whose activations are largest, by scaling them up before rounding rather than keeping them in higher precision. No backpropagation and no reconstruction, so it does not overfit its calibration set. | Active |
| LIT-586 v2 | LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale | Dettmers et al. (2022), ARXIV-2208.07339. Int8 inference for the feed-forward and attention projections, at no measured cost in quality up to 175B, by isolating a small set of systematically emergent outlier feature dimensions into a 16-bit multiplication and quantizing the other 99.9% vector-wise. | Active |
| LIT-656 | Intriguing Properties of Quantization at Scale | Ahmadian et al. (2023), ARXIV-2305.19268. A controlled study from 410M to 52B, same architecture, varying weight decay, dropout, gradient clipping and training dtype: outlier dimensions are not an inherent product of scale but of the optimization conditions during pre-training. Their 52B model gains 0.08% under plain INT8 (77.31 → 77.39 average) where OPT-66B loses about 42%. The record’s largest cluster had the emergence framing in an Active practice’s title. | Active |
| LIT-676 v4 | DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models | Lu, Zhou, Bao, Chen, Li and Zhu (2022), ARXIV-2211.01095. The regime SOTA-203 does not mention. At a classifier guidance scale of 8.0 on ImageNet 256×256, FID at 10 function evaluations is 13.04 for first-order DDIM, 114.62 for DPM-Solver-2 and 164.74 for DPM-Solver-3 — higher order is monotonically catastrophic, and the practice’s own framing of DDIM as “the least accurate member of the family” inverts. Two causes, one of which the record already holds as SOTA-202. The fix is order 2, multistep, on the data-prediction parameterization. | Active |
| LIT-750 | ZipNN: Lossless Compression for AI Models | Hershcovitch et al. (2024), ARXIV-2411.05239. Losslessly compressing a trained model’s file works almost entirely through the float exponent: about 40 of 256 values occur, and the exponent stream codes to ~33% while sign and mantissa stay near 100%. Regular BF16 models therefore shrink to ~66%, FP32 to ~83%, and models rounded after training (“clean”) to as little as 34–48%. Splitting the exponent into its own stream and coding it with Huffman alone, without LZ, beats Zstd by 17% in size and 62% in single-thread speed on Llama-3.1 BF16. | Active |
| LIT-783 | Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency | Zheng et al., Tsinghua and NVIDIA (2025), ARXIV-2510.08431. sCM made to run on 14B text-to-image and text-to-video models, with a FlashAttention-2 JVP kernel under FSDP and context parallelism. Pure sCM loses fine detail and temporal coherence there. Adding a DMD loss at weight 0.01 (rCM) fixes it. Four-step Wan2.1 VBench is 84.43 at 1.3B against a re-implemented DMD2’s 84.56, and 84.92 at 14B; GenEval tracks the teacher at 1–4 steps. The diversity advantage over DMD2 is shown in samples only. One run per row. | Active |
| LIT-790 | Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models | Lu and Song, OpenAI (2024), ARXIV-2410.11081. Continuous-time consistency models had been unstable. A trigonometric flow parameterization, identity time input with positional embeddings, normalized tangents, learned loss weights and a Flash Attention JVP make them train to 1.5B parameters. Continuous time beats every discrete step count tried. Two-step FID is 2.06 on CIFAR-10, 1.48 on ImageNet-64 and 1.88 on ImageNet-512, against teachers at 2.01, 1.33 and 1.73. Every model, “consistency training” included, starts from the pretrained diffusion model. | Active |
| LIT-800 | Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM | Kozyrev and Maiboroda, Minima (2026), ARXIV-2609.04098. Public 4-bit builds of Qwen3.8-27B kept its 48 Gated DeltaNet layers in 8 or 16 bits for fear that errors in a recurrence compound. Quantizing all 496 linear layers to NVFP4 W4A4 by calibration alone stays within seed noise of BF16 on five task suites (average −0.52) at 17.5 GiB. The perplexity cost is the largest of the recipes compared, +0.72 at 4K, but it shrinks along a 32K window. In replayed layers the gate projections everyone protected are the least sensitive. State error holds at a plateau of about 12.6% over 32K tokens, and a state impulse is erased within hundreds to a few thousand steps. One model, one format. | Active |