Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Systems optimization — hardware utilization, kernels, compilation, memory access patterns — how an operation is executed, not how many bits it is executed in.
26 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-004 v4 | Training Deep Nets with Sublinear Memory Cost | Chen et al. (2016), ARXIV-1604.06174. Gradient checkpointing: store activations at O(sqrt(n)) layers and recompute the segments between them in the backward pass, for the cost of one extra forward pass per minibatch. A 1,000-layer ResNet drops from 48G to 7G for 30% more running time. | Active |
| LIT-059 v2 | CheckFreq: Frequent, Fine-Grained DNN Checkpointing | Mohan et al. (2021), FAST ‘21. Checkpoint frequency should be computed, not guessed: online profiling picks an iteration-granularity interval, adaptive tuning holds the overhead under a bound, and two-phase checkpointing pipelines the write against compute. | Active |
| LIT-063 v2 | TVM: An Automated End-to-End Optimizing Compiler for Deep Learning | Chen et al. (2018), ARXIV-1802.04799. Graph-level and operator-level optimization in one compiler, with a learned cost model searching the low-level schedule space, so performance is portable across back-ends instead of living in vendor libraries. | Active |
| LIT-066 v2 | Data Movement Is All You Need: A Case Study on Optimizing Transformers | Ivanov et al. (2020), ARXIV-2007.00072. Transformer training is bound by data movement, not compute: existing frameworks use suboptimal layouts, and a systematic movement-first optimization cuts data movement 22.91% for a 1.30x speedup on a BERT encoder layer. | Active |
| LIT-074 v3 | FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness | Dao et al. (2022), ARXIV-2205.14135. IO-aware attention implementation. | Active |
| LIT-116 v2 | TensorRT-LLM: A Compiler and Runtime for LLMs | A synthetic entry the migration produced: no paper of this title exists, and the identifier it carried belongs to an unrelated physics preprint. Retired rather than deleted. The source field points at the software project itself. | Rejected — software with no published paper behind it, and no reading in the record |
| LIT-139 v2 | DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence | DeepSeek-AI (2026), ARXIV-2606.19348. Two MoE models — V4-Pro at 1.6T/49B and V4-Flash at 284B/13B — with a 1M-token context reached by a hybrid of compressed sparse attention and heavily compressed dense attention, manifold-constrained hyper-connections widening the residual stream, and Muon as the optimizer for most modules over 32–33T tokens; at 1M tokens V4-Pro needs an estimated 27% of V3.2’s per-token FLOPs and 10% of its KV cache. Multi-token prediction is carried over from V3 unchanged, as a training objective; the report does not mention speculative decoding. | Active |
| LIT-197 | Microscaling Data Formats for Deep Learning | Rouhani et al. (2023), ARXIV-2310.10537. The MX formats: a block of 32 elements sharing one 8-bit power-of-two scale, with the elements themselves in FP8, FP6, FP4 or INT8. Over two dozen benchmarks — 8-bit MX runs inference on FP32 checkpoints with no calibration, 6-bit matches FP32 for training weights, activations and gradients with no recipe change, and 4-bit weights cost only a minor drop. The first sub-8-bit training result of that kind. | Active |
| LIT-367 | ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning | Rajbhandari et al. (2021), ARXIV-2104.07857. ZeRO-Infinity offloads parameters, gradients and optimizer state to CPU and NVMe with bandwidth- centric partitioning, so model size stops being bounded by aggregate GPU memory. | Active |
| LIT-378 | QLoRA: Efficient Finetuning of Quantized LLMs | Dettmers et al. (2023), ARXIV-2305.14314. Back-propagate through a frozen 4-bit base model into 16-bit LoRA adapters: 65B fine-tuning drops from >780GB to <48GB with no loss against a 16-bit fully fine-tuned baseline. Carries a second finding the quantization headline tends to bury — LoRA on the query and value projections alone does not reach full fine-tuning at scale, and the adapter COUNT, not the rank, is what closes the gap. | Active |
| LIT-380 | The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | Ma et al. (2024), ARXIV-2402.17764. Every weight is ternary, {-1, 0, +1}, and the model is trained that way from scratch rather than quantized afterwards. From 3B upward it matches an FP16 LLaMA of the same size and token budget on perplexity and end-task accuracy, while the matrix multiply reduces to integer addition — a compute paradigm rather than a compression ratio. | Active |
| LIT-418 | HybridFlow: A Flexible and Efficient RLHF Framework | Sheng et al. (2024), ARXIV-2409.19256. The paper behind verl — a hybrid single/multi-controller execution model for the RLHF dataflow, with a 3D resharding engine between the training and generation phases. Named in ADR-032 as one of the four references at the top of the backlog it opened. | Active |
| LIT-512 | SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models | Li et al. (2024), ARXIV-2411.05007 — 4-bit weights and activations for diffusion transformers. Shift outliers from activations into the weights by smoothing, peel the dominant singular values into a 16-bit rank-32 branch, quantize only the residual. Two propositions bound the output error by the magnitudes rather than only the rounding errors, which is why shrinking the residual works. Naïvely the extra branch costs 57% latency; fused, the system reaches 3.0× over a W4A16 baseline and 3.5× memory reduction on 12B FLUX.1. Read as NOTE-257. | Active |
| LIT-515 | Exploiting Block Coordinate Descent for Cost-Effective LLM Model Training | Liu et al. (2025), ARXIV-2506.12037 — full-parameter training one block at a time, with blocks cut on layer boundaries so the optimized kernels still apply. Reaches comparable or better perplexity at 2B on about three times the iterations, using one A800 where full-parameter needs two. Read as NOTE-260. | Active |
| LIT-552 | XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models | Dong et al. (2024), ARXIV-2411.15100. Constrained decoding sets the logits of grammar-violating tokens to −∞. Doing that for a context-free grammar means interpreting a pushdown automaton over the whole vocabulary every step. XGrammar precomputes, per automaton node, the verdict for tokens that do not depend on the rest of the stack (over 99% for JSON on Llama-3.1), and builds the mask on the CPU during the GPU forward pass. Per-token mask time is 36 µs against milliseconds for earlier engines, with end-to-end overhead of about 0.1 ms per token. | Active |
| LIT-567 | MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models | Frantar, Castro, Chen, Hoefler, Alistarh (2024), ARXIV-2408.11743. An FP16×INT4 matrix-multiply kernel, the “Marlin format” being its quantization-specific weight layout. It keeps weight-only quantization’s memory-bandwidth speed-up through batched serving: about 3.9× over FP16 (near the 3.87× ideal for 4-bit, group 128) up to batch 16–32, falling toward 1.5× at 128 as the problem turns compute-bound. In vLLM, about 2.8× lower time per output token on an A6000, and 2.3–3.2× single-GPU end-to-end at batch 16 or below, falling to 1.1–1.2× at 128. | Active |
| LIT-601 | Masked Autoencoders Are Scalable Vision Learners | He et al. (2021), ARXIV-2111.06377. Mask 75% of the patches, run the encoder on the visible quarter only, and reconstruct pixels with a small decoder you then throw away. 3x faster pretraining, 87.8% with ViT-Huge on ImageNet-1K alone. The 75% ratio, five times BERT’s 15%, is found by a sweep, and the paper explains it with an argument about the signal’s redundancy. | Active |
| LIT-655 | Self-attention Does Not Need O(n²) Memory | Rabe and Staats (2021), ARXIV-2112.05682. Six months before FlashAttention: attention in O(1) memory per query, self-attention in O(log n), a practical O(√n) implementation, exact rather than approximate, 59× less memory overhead at sequence length 16,384. And the number the record needed and did not have: on TPU the same algorithm is “within a few percent of the runtime of the standard implementation” — the memory saving is algorithmic and the speedup is not. | Active |
| LIT-706 | Scaling Language-Image Pre-training via Masking | Li et al. (2022), ARXIV-2212.00794 — FLIP. Train CLIP with 50–75% of image patches removed and the image encoder run on the rest: the saved compute buys more pairs per hour and a 2–4× larger batch at the same memory, which outweighs the information lost per image. ViT-L/16 on LAION-400M reaches its CLIP reproduction’s accuracy more than 3× faster. The accuracy gain at a fixed schedule is 1–2 points at L and nothing at B/16, and at equal batch size masking 50% buys nothing — the batch is doing the work. | Active |
| LIT-735 | Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads | Jangda et al. (2021), ARXIV-2105.05720 — CoCoNet (ASPLOS 2022). A DSL and compiler that treat computation and collectives as one program, so they can be fused and overlapped. Overlapping a tensor-parallel MatMul with its AllReduce in fine-grained chunks hides over 80% of the MatMul and runs 1.36x faster than issuing them in sequence; in Megatron-LM, the overlapped schedules speed up model-parallel inference 1.48-1.51x and pipeline-parallel inference further, and the data-parallel gain on BERT training comes from fusing the optimizer into the collective rather than from overlap. | Active |
| LIT-736 | Reducing Activation Recomputation in Large Transformer Models | Korthikanti et al. (2022), ARXIV-2205.05198. Introduces sequence parallelism: the layer norms and dropouts that tensor parallelism leaves replicated are split along the sequence dimension, and the tensor-parallel all-reduces become an all-gather and a reduce-scatter that move the same bytes, so activation memory per layer falls by the tensor-parallel degree at no extra communication. With selective recomputation of the attention core it cuts activation memory about 5x, and over full recomputation lifts end-to-end throughput 29.0-32.1% on models from 22B to 1T parameters. | Active |
| LIT-737 | 10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training | Afroz et al. (2025), ARXIV-2511.14124 (SoCC ‘25). A tensor caching and migration system for offloaded LLM training across GPU, CPU and NVMe, reporting up to 2× faster training than prior offloading methods. Its motivating measurement is the one the record uses: on an L40S over PCIe 4.0 ×16, a 16 MB FP32 host-to-device copy takes 1.65 ms (10.16 GB/s) from pageable memory and 0.68 ms (24.74 GB/s) from pinned memory. | Active |
| LIT-739 | TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems | Abadi et al. (2016), ARXIV-1603.04467 — the TensorFlow white paper (dated November 2015). Describes TensorFlow’s dataflow-graph interface and its execution from a single device to clusters; among its graph features are input operations that read from storage directly and queues that let input be prefetched from disk while the previous batch is still being computed. | Active |
| LIT-746 | Position: Curvature Matrices Should Be Democratized via Linear Operators | Dangel et al. (2025), ARXIV-2501.19183 — a position paper with a library behind it: expose the Hessian, GGN, Fisher flavours and (E)KFAC as linear operators, so applications ask only for matrix-vector products and the batching, scaling and determinism pitfalls live in one tested place. Its measurements are costs in units of a gradient, on ResNet-50 and nanoGPT: computing KFAC’s factors takes 1.5–2.5 gradients and applying them 0.1–0.2, while a Hessian-vector product takes 4.5–5.5 gradients and 3–4.5× the memory in PyTorch. | Active |
| LIT-750 | ZipNN: Lossless Compression for AI Models | Hershcovitch et al. (2024), ARXIV-2411.05239. Losslessly compressing a trained model’s file works almost entirely through the float exponent: about 40 of 256 values occur, and the exponent stream codes to ~33% while sign and mantissa stay near 100%. Regular BF16 models therefore shrink to ~66%, FP32 to ~83%, and models rounded after training (“clean”) to as little as 34–48%. Splitting the exponent into its own stream and coding it with Huffman alone, without LZ, beats Zstd by 17% in size and 62% in single-thread speed on Llama-3.1 BF16. | Active |
| LIT-754 | Efficient AlphaFold2 Training using Parallel Evoformer and Branch Parallelism | Wang et al. (2022), ARXIV-2211.00235. Move the Evoformer’s outer product mean to the end of the block, so the MSA and pair stacks no longer depend on each other within a block; this matches the original block’s accuracy and speed. Then put the two stacks on two GPUs. On a batch of one protein per device, where data parallelism cannot scale further, this cuts end-to-end AlphaFold2 training from 5.8 to 4.2 days on UniFold and from 6.7 to 4.9 days on HelixFold, using twice the GPUs. | Active |