systems-optimization.
11 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-005 | FlashAttention | Attention is bounded by HBM traffic rather than FLOPs. Tiling to on-chip SRAM and recomputing the attention matrix in the backward pass gives exact attention in Θ(N²d²M⁻¹) HBM accesses against standard attention’s Θ(Nd + N²) — and Proposition 3 proves no exact algorithm beats that across all SRAM sizes. | Read |
| NOTE-122 | ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learning | By partitioning model states across all parallel devices and using a bandwidth-centric allgather strategy (rather than per-device broadcast), ZeRO-Infinity aggregates PCIe bandwidth linearly with device count, making NVMe and CPU offloading fast enough to sustain efficient training despite their individually slow bandwidths. | Read |
| NOTE-164 | QLoRA: Efficient Finetuning of Quantized LLMs | Store the frozen base in 4 bits, compute in 16, and train only LoRA adapters: 65B fine-tuning goes from >780GB to <48GB with no measured loss against a 16-bit fully fine-tuned baseline. The quantization is the headline; the more useful result for anyone already using LoRA is that the standard query/value placement does not reach full fine-tuning at scale, and that the adapter count rather than the rank is what closes the gap. | Read |
| NOTE-168 | The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | Ternary weights trained from scratch, matching FP16 from 3B upward at equal size and tokens. The interesting claim is not the compression ratio but that the matrix multiply becomes integer addition — which is a bet on hardware, and the one place the paper’s argument outruns its measurements. | Read |
| NOTE-184 | Training Deep Nets with Sublinear Memory Cost | Two contributions, and the famous one is the second. First, treat memory allocation over the computation graph as a compiler problem — liveness analysis, in-place operations, memory sharing. Then, on top of that, drop most intermediate feature maps and recompute them segment by segment during the backward pass. Dividing an n-layer chain into sqrt(n) segments costs sqrt(n) memory for one extra forward pass; the extreme of the same analysis is log n memory for n log n extra forward computation. | Read |
| NOTE-257 | SVDQuant: a low-rank branch on the weights, not on the error | Read from the #180 worklist. The idea that travels is an ordering: decompose the weights and quantize the residual, rather than quantize and then patch the error. Two propositions and Eckart-Young say why, and a published method that does it the other way round is the control. | Read |
| NOTE-260 | Block coordinate descent: a smaller machine bought with three times the steps | Read from the #180 worklist as the last of the 5-revisit tier. The method is sound and the accounting needs separating: the headline 2.6% mixes the method with the price of a 4090, the same-device figure is 33%, and the iteration counts in the paper’s own Table 2 show BCD paying about 3× the steps for the memory it saves. | Read |
| NOTE-294 | XGrammar | Grammar-constrained decoding made nearly free. It splits the vocabulary into tokens whose validity depends only on the automaton’s stack top, which are precomputed, and the under-1% that need the full stack, which are checked at runtime. The mask is built on the CPU during the forward pass. Each optimization is ablated. Output quality is measured only as syntactic validity. Read §1–6; the appendix was not read. | Read |
| NOTE-305 | MARLIN | An FP16×INT4 GEMM kernel that stays memory-bound, and so near the 3.87× ideal 4-bit speed-up, up to batch 16–32 on a large layer, decaying toward 1.5× at 128. End-to-end in vLLM the same shape holds on four GPU classes (Table 2): 2.3–3.2× single-GPU at batch ≤16, 1.1–1.2× at 128, and much less when the model is sharded over eight A100s. The INT4 model it serves loses 3.3 points of mean accuracy (Table 1). | Read |
| NOTE-361 | DeepSeek-V4 | Two MoE models (1.6T/49B and 284B/13B) reach a native 1M-token context by interleaving two compressed attentions — 4x-compressed KV with top-k sparse selection (CSA) and 128x-compressed dense KV (HCA), each with a 128-token sliding-window branch — at 27% of V3.2’s per-token FLOPs and 10% of its KV cache for Pro at 1M. Multi-token prediction is kept “without modification” from V3 as a depth-1 training objective; the report never mentions speculative decoding or a draft model. | Read |
| NOTE-371 | FLIP | Removing 50% of image patches in CLIP training halves the image encoder’s cost and lets the contrastive batch double at the same memory; ViT-L/16 on LAION-400M then matches its unmasked reproduction more than 3× faster and beats it by about a point at equal epochs. At equal batch the masking gains nothing, and at ViT-B/16 the full-schedule model is 0.2 behind — the improvement is the batch and the extra samples that masking pays for. | Read |