Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Systems optimization — hardware utilization, kernels, compilation, memory access patterns — how an operation is executed, not how many bits it is executed in.
29 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-044 | Pin memory for CPU-GPU transfers | Afroz et al. (2025), LIT-737 — ARXIV-2511.14124. Host-to-device copies from pinned memory ran at 24.74 GB/s against 10.16 GB/s from pageable memory on PCIe 4.0; the advice itself is NVIDIA’s, not a paper’s. | Active |
| SOTA-081 | Use operator fusion for small operations | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-082 | Optimize memory layout for hardware | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-083 v2 | Implement custom kernels for critical ops | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-084 | Profile-guided optimization for hot paths | Chen et al. (2018), LIT-063 — ARXIV-1802.04799. | Active |
| SOTA-085 v4 | Use flash attention for all attention computations when hardware supports it | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-086 v4 | Tiling size should match hardware SRAM size | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-087 v4 | Recompute attention during backward pass instead of storing it | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-088 | Fuse small operations into larger kernels | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-089 | Align tensor dimensions to hardware boundaries | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-090 | Use hardware-specific memory layouts | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-091 | Profile and optimize memory access patterns | Ivanov et al. (2020), LIT-066 — ARXIV-2007.00072. | Active |
| SOTA-106 v3 | Use flash-attention-2 over original flash-attention when available | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Active |
| SOTA-107 v3 | Keep sequence lengths multiple of 128 for best performance | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Rejected — The 128 is traced and belongs to a different quantity. Every occurrence in the source is a head dimension or a block size; none is a sequence length, and “multiple of” and “divisible” appear zero times |
| SOTA-108 v3 | Pad attention masks to block boundaries for better hardware utilization | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Rejected — Not in the cited paper. “padding”, “padded”, “pad”, “block boundar” and “divisible” appear zero times each; the body’s reasoning about skipping fully-masked blocks is correct and is not the source’s |
| SOTA-114 | Fuse attention operations where possible | Dao et al. (2022), LIT-074 — ARXIV-2205.14135. | Active |
| SOTA-161 v4 | Keep the attention output in FP32 during training, because flash attention’s rounding bias compounds instead of cancelling | Qiu et al. (2025), LIT-198 — putting flash attention in BF16 while the FFN goes to FP8 sometimes explodes, and the cause is two things at once: attention produces low-rank updates repeated across steps and tokens, and low-precision addition rounds with a bias. The biased error rides the repeated update and compounds into a systematic gradient bias instead of averaging out. | Active |
| SOTA-230 | Quantize the frozen base to 4-bit and keep the adapters in 16-bit | Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314. Store the frozen base weights in 4-bit NormalFloat and dequantize to BFloat16 for every matrix multiply, training only 16-bit LoRA adapters. Fine-tuning a 65B model falls from >780GB to <48GB with no measured loss against a 16-bit fully fine-tuned baseline, because the arithmetic never happens in 4 bits. | Active |
| SOTA-234 | Train in the target low-bit format from scratch rather than quantizing a finished model | Ma et al. (2024), LIT-380 — ARXIV-2402.17764. Decide the serving format before training and train in it, rather than training in FP16 and compressing afterwards. Ternary weights trained from scratch match an FP16 model of equal size and token budget from 3B upward — but the compute argument assumes hardware built for the format, and only the memory saving is measured on machines that exist. | Proposed |
| SOTA-249 | Recompute activations from a sqrt(n) subset of checkpoints when activation memory is the binding constraint | Chen et al. (2016), LIT-004 — ARXIV-1604.06174. Store activations at O(sqrt(n)) checkpoints and recompute the rest during the backward pass. The price is one extra forward pass per minibatch; the measured case is a 1000-layer ResNet at 48G to 7G for 30% more wall clock. | Active |
| SOTA-252 | Make the full-resolution path affordable instead of upsampling, when the shortcut is what breaks correctness | Abdal et al. (2023), LIT-113 — ARXIV-2311.17857. Prior 3D GANs render small and upsample in 2D because volume rendering is too slow at training resolution — but a 2D upsampler is multi-view inconsistent by construction. Efficient Gaussian rendering makes native 512x512 affordable, and the inconsistency leaves with the upsampler rather than being mitigated. | Active |
| SOTA-314 v2 | Absorb quantization outliers into a high-precision low-rank branch taken from the weights, and fuse its kernels into the low-bit ones | Li et al. (2024), LIT-512 — at 4 bits on both weights and activations, smoothing alone is not enough. Shift outliers from activations into weights, peel the dominant singular values into a 16-bit rank-32 branch, quantize the residual. Then fuse: run naïvely the branch costs 57% latency and cancels the win; fused it gives 3.0× over W4A16 and 3.5× memory on 12B FLUX.1. | Active |
| SOTA-316 | When you are memory-bound rather than time-bound, train all the parameters a block at a time, partitioned on layer boundaries | Liu et al. (2025), LIT-515 — freeze all but one block of layers, train it to local convergence, rotate. Memory falls to roughly the active block, so a 7B trains on one A800 where full-parameter needs two, and perplexity at 2B matches or beats the full-parameter baseline. The price is about three times the iterations, which the paper’s own table shows and its abstract does not. | Proposed |
| SOTA-326 | Add n-gram embedding memory as one table read early in the network, on top of the expert budget rather than in place of experts | Qiu et al. (2026), LIT-152 — Qwen3.8-Flash-Next’s ablations. One n-gram embedding table at layer 2 takes loss from 1.585 to 1.541 and the benchmark average from 45.44 to 47.94. Splitting the same table across two layers buys nothing, and no depth clearly wins. Trading experts for table parameters at a fixed total lowers loss slightly and moves no downstream benchmark. Grown on top of the budget, the table keeps lowering loss after downstream accuracy has flattened. Layer 2 is a systems choice: the host-memory prefetch overlaps layer 1. | Proposed |
| SOTA-330 | Enforce output structure by masking logits against a grammar, with masks precomputed per automaton state and built on the CPU while the GPU runs the forward pass | Dong et al. (2024), LIT-552 — split the vocabulary by whether a token’s validity depends only on the grammar automaton’s current node (over 99% of tokens for JSON). Precompute those per node, check the rest against the full stack at runtime, and build each mask on the CPU during the GPU forward pass. Mask time is 0.018 ms per token after the ablated optimizations, and end-to-end overhead is 0.1–0.2 ms. It guarantees syntax. It does not measure whether answers stay correct. | Active |
| SOTA-340 | Expect weight-only 4-bit quantization to speed up batched serving only while the batch keeps the matmul memory-bound, and use a kernel built to stay there | Frantar et al. (2024), LIT-567 — weight-only quantization speeds up decoding by loading fewer bytes, so the gain lasts only while the layer is memory-bound. A kernel designed to stay memory-bound (MARLIN) holds about 3.9× over FP16 up to batch 16–32 on an A10, decaying toward 1.5× at 128. End-to-end in vLLM, single-GPU, 2.3–3.2× at batch 16 or below and 1.1–1.2× at 128, on four GPU classes. Pick the format for the batch you actually serve, and pay for it in accuracy only where it buys speed. | Proposed |
| SOTA-372 v2 | Run the encoder on visible tokens only and push the mask tokens into a small decoder you discard | Active | |
| SOTA-429 | Drop half the image patches when training a CLIP-style model, spend the saving on more pairs and a larger batch, and unmask only for a short final tune | Li et al. (2022), LIT-706 — FLIP. Remove 50% of image patches and run the ViT on the rest, with CLIP’s loss and nothing else. The saving buys a 2× larger batch at the same memory and 2× the samples per hour; ViT-L/16 on LAION-400M reaches its unmasked reproduction’s accuracy more than 3× faster. The accuracy comes from what the saving buys: at equal batch, masking is parity. | Proposed |
| SOTA-437 | Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base | Hershcovitch et al. (2024), LIT-750 — ZipNN. In regular trained weights only the float exponent is redundant: about 40 of its 256 values occur and it codes to about a third, while sign and mantissa barely compress. So split the exponent into its own stream and Huffman-code it without an LZ stage. Llama-3.1 BF16 goes to 66.4% against Zstd’s 77.7%, and is faster in both directions. Between checkpoints, XOR deltas against a base up to 10 checkpoints back compress better than standalone files. Sizes and throughputs are measured. Stall time inside a training run is not. | Proposed |