numerics-and-precision.
7 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-029 | ZeRO++: Extremely Efficient Collective Communication for Giant Model Training | Attacks each of ZeRO’s three communication collectives separately — block-quantized weight all-gather, a secondary weight shard held redundantly to trade memory for communication, and an all-to-all quantized gradient reduction replacing reduce-scatter. 4× less communication, up to 2.16× throughput at 384 GPUs. | Read |
| NOTE-164 | QLoRA: Efficient Finetuning of Quantized LLMs | Store the frozen base in 4 bits, compute in 16, and train only LoRA adapters: 65B fine-tuning goes from >780GB to <48GB with no measured loss against a 16-bit fully fine-tuned baseline. The quantization is the headline; the more useful result for anyone already using LoRA is that the standard query/value placement does not reach full fine-tuning at scale, and that the adapter count rather than the rank is what closes the gap. | Read |
| NOTE-168 | The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits | Ternary weights trained from scratch, matching FP16 from 3B upward at equal size and tokens. The interesting claim is not the compression ratio but that the matrix multiply becomes integer addition — which is a bet on hardware, and the one place the paper’s argument outruns its measurements. | Read |
| NOTE-222 | Quantized Evolution Strategies | Reading it: the memory trick is the contribution, not the error feedback. Error feedback is 1-bit SGD’s, correctly credited; what is new is refusing to store the accumulator at all and rebuilding it from seeds, which is the only reason the method fits in the memory budget that motivated it. | Read |
| NOTE-257 | SVDQuant: a low-rank branch on the weights, not on the error | Read from the #180 worklist. The idea that travels is an ordering: decompose the weights and quantize the residual, rather than quantize and then patch the error. Two propositions and Eckart-Young say why, and a published method that does it the other way round is the control. | Read |
| NOTE-305 | MARLIN | An FP16×INT4 GEMM kernel that stays memory-bound, and so near the 3.87× ideal 4-bit speed-up, up to batch 16–32 on a large layer, decaying toward 1.5× at 128. End-to-end in vLLM the same shape holds on four GPU classes (Table 2): 2.3–3.2× single-GPU at batch ≤16, 1.1–1.2× at 128, and much less when the model is sharded over eight A100s. The INT4 model it serves loses 3.3 points of mean accuracy (Table 1). | Read |
| NOTE-361 | DeepSeek-V4 | Two MoE models (1.6T/49B and 284B/13B) reach a native 1M-token context by interleaving two compressed attentions — 4x-compressed KV with top-k sparse selection (CSA) and 128x-compressed dense KV (HCA), each with a 128-token sliding-window branch — at 27% of V3.2’s per-token FLOPs and 10% of its KV cache for Pro at 1M. Multi-token prediction is kept “without modification” from V3 as a depth-1 training objective; the report never mentions speculative decoding or a draft model. | Read |