Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Distributed optimization — parallelism and sharding, communication, memory management, checkpointing.
42 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-017 v2 | Use micro-batch splitting for pipeline parallelism | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-018 v2 | Balance pipeline stages to minimize bubble overhead | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-019 v2 | Choose pipeline chunks based on memory vs. compute trade-off | Huang et al. (2018), LIT-016 — ARXIV-1811.06965. | Active |
| SOTA-028 | Stage optimizer states across data parallel ranks (ZeRO-1) | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-029 | Partition gradients and optimizer states (ZeRO-2) for larger models | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-030 | Use ZeRO-3 only when other strategies insufficient | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-031 v2 | Keep micro-batch size per GPU as large as memory allows | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-046 | Use hierarchical allreduce for tensors > 1MB | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — the ’> 1MB’ crossover has no owner: NCCL measures the algorithm choice per topology rather than publishing a constant, and LIT-051 does not discuss it. What survives is that a modern collectives library already chooses by size and topology, which is not a practice a reader acts on |
| SOTA-047 | Overlap communication with backward pass | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Active |
| SOTA-048 | Group small tensors before communication | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Active |
| SOTA-049 | Set buffer size to network bandwidth-delay product | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — collapses two different quantities into one unfollowable sentence — a TCP socket buffer sized by bandwidth-delay product, and a gradient fusion buffer sized by the overlap trade. The fusion-buffer half is SOTA-048, which now carries Horovod’s attributable 64 MB default; the transport half is a claim about networks with no source here |
| SOTA-054 v2 | Derive the checkpoint interval from online profiling and adapt it at runtime against an overhead bound | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-055 | Save optimizer state every N epochs (N ~ sqrt(total_epochs)) | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Rejected — the practice its own cited source exists to replace. LIT-059 names epoch-granular, hand-tuned checkpointing as the problem it solves, on both counts; SOTA-054 now states what that paper actually recommends |
| SOTA-056 | Use async I/O for checkpoint writing | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-057 | Implement multi-level checkpoint strategy | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Active |
| SOTA-058 v2 | Use sequence parallelism for the layer-norm and dropout regions that tensor parallelism leaves replicated | Korthikanti et al. (2022), LIT-736 — ARXIV-2205.05198. | Active |
| SOTA-059 | Overlap communication with computation when possible | Jangda et al. (2021), LIT-735 — ARXIV-2105.05720. | Active |
| SOTA-073 | Monitor network utilization during training | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — states no signal, threshold or action, so there is nothing to do differently on reading it, and LIT-051 does not argue for it. The specific version worth having is in SOTA-048’s body: the gap between achieved and peak bandwidth during the overlap window tells you the fusion buffer is sized wrong |
| SOTA-074 | Adapt buffer sizes to network conditions | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — names a controller — signal, target, update rule — of which none is stated, and no source here describes one. A measured version would resemble what CheckFreq does for checkpoint intervals (SOTA-054) |
| SOTA-075 | Use gradient compression for slow networks | Lin et al. (2017), LIT-056 — ARXIV-1712.01887. | Active |
| SOTA-076 | Place replicas to minimize cross-rack traffic | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — a cluster-scheduling claim rather than a training one: it is the scheduler’s decision, expressed as a placement-group request, not something in a training run. ADR-026 (superseding ADR-020) scopes this record by the kind of claim, and this is a different kind |
| SOTA-116 | Use FSDP over DDP when model size exceeds single GPU memory | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-117 | Overlap communication with computation using backward prefetch | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-118 | Employ mixed precision to reduce memory usage | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-119 | Choose sharding factor based on model and GPU memory size | Zhao et al. (2022), LIT-083 — ARXIV-2304.11277. | Active |
| SOTA-155 | Train across poorly connected islands: many inner steps per worker, an outer momentum optimizer over the deltas, and a streamed synchronisation | Douillard et al. (2023, 2025), LIT-212 and LIT-214 — federated averaging with the constants pushed hard: many inner AdamW steps per worker, Nesterov momentum as the outer optimizer over the accumulated deltas, synchronised rarely and in streamed subsets. 500× less communication at matched quality, then two more orders off the peak. Filed Proposed: one group, and no frontier report trains this way. | Proposed |
| SOTA-181 | Shard the sequence across devices in a ring and overlap the key-value exchange with the attention it feeds | Liu et al. (2023), LIT-206 — compute attention and the feedforward blockwise, distribute the blocks across devices in a ring, and pass each key/value block to the neighbour while computing on the one you hold. The communication is fully overlapped with the computation, so sequences scale with device count — exactly, with no approximation and no added overhead. | Active |
| SOTA-189 | Set the checkpoint interval from the job’s measured time-to-failure, which shrinks as the job grows | CoreWeave (2025), LIT-222. Expected work lost to a failure is half the inter-checkpoint interval, and mean time to failure falls linearly with GPU count — so the interval belongs to the job’s size and the cluster’s failure rate, not to the write cost alone. | Proposed |
| SOTA-214 | Pair any gradient compressor with error feedback, and correct the momentum it is applied under | Compression is lossy and the loss accumulates. Every scheme in this record that survives high compression ratios keeps the residual locally and adds it back to the next step, and every report of a compressor “not converging” that the record holds is a report of that step being skipped: rank-4 PowerSGD fails on Cifar10 without it, and gradient sparsification at 99.9% degrades badly without the momentum-buffer version of it. | Active |
| SOTA-215 | Freeze Adam’s variance term after a warmup before compressing its update, and never compress the update directly | Error-compensated compression assumes the residual it carries forward still means the same thing next step. Adam’s update divides by a running variance, which is a non-linearity the correction does not survive — compressing it directly corrupts the error cancellation and badly harms convergence. The workaround is structural rather than a tuning fix: Adam’s variance stabilises early, so freeze it after a warmup and the update becomes linear enough to compress. | Proposed |
| SOTA-216 | Switch from minibatch SGD to local SGD at the first learning-rate decay, rather than choosing between them | The two options are usually posed as a choice: minibatch SGD generalises and communicates every step, local SGD communicates rarely and generalises worse. Post-local SGD is the observation that they are wanted at different times — run minibatch SGD until the first learning-rate decay, then switch to local SGD with H around 16 to 32 for the rest. The early phase is where the gradient noise is doing work; the late phase is where communication is pure overhead. | Proposed |
| SOTA-219 | Leave the inner optimizer state unsynchronised in local-update training | When workers train independently for hundreds of steps and synchronise by exchanging deltas, the Adam moments each worker accumulated are local state and should stay local. Synchronising them triples the communication for negligible quality gain — which means most of what the scheme saves is not saved by communicating rarely, but by communicating less each time. | Active |
| SOTA-220 | Aggregate worker gradients coordinate-wise by median when any worker may be faulty | Averaging has a breakdown point of zero: one worker returning an arbitrary vector can move the mean anywhere, and adding workers does not help, because it is the worker’s magnitude and not its share of the population that does the moving. The coordinate-wise median is the robust aggregator that needs no estimate of how many workers are faulty; the trimmed mean has better rates and needs one. | Proposed |
| SOTA-222 v2 | Overlap the synchronisation with one or two local steps instead of communicating every step | Wang et al. (2020), LIT-323. Local SGD is usually argued for as a way to communicate rarely, which costs accuracy at large intervals. The cheaper reading is that it does not take a large interval to win: at tau = 2 the communication-to-computation ratio falls from 34.6% to 1.5% with convergence per iteration roughly unchanged. The interval is not buying infrequency, it is buying somewhere to hide the exchange. | Proposed |
| SOTA-223 | Do not keep the gossip topology static: sample a fresh random neighbourhood every round | de Vos et al. (2023), LIT-315. Decentralized training has spent years choosing good fixed graphs. Epidemic Learning does not choose one: each round, every node sends its update to s peers picked at random. The transient-iteration count falls from the best-known O(n^3) to O(n^3/s^2), and at 96 nodes on CIFAR-10 it converges 1.7x faster and 2.2% more accurately than a static 7-regular graph at equal communication volume. | Proposed |
| SOTA-224 | Ramp the sparsity ratio up over the first few epochs rather than starting at the target | Lin et al. (2017), LIT-056. Gradient sparsification delays the coordinates it drops; error feedback returns them, but it returns them late. Early in training the gradients are large and the trajectory is still choosing where to go, so a delay there costs more than the same delay later. Start at a low sparsity — the paper ramps 75%, 93.75%, 98.4375%, 99.6%, 99.9% over the first few epochs — and reach the target once the run has settled. | Active |
| SOTA-225 | Drop the parameter server for gossip once the network is the bottleneck, and measure where that is rather than inheriting a threshold | Lian et al. (2017), LIT-302. The question is whether a decentralized algorithm can ever beat a centralized one, and the answer is that it can — not by converging better, since the rates match, but by having no node that every other node talks to. The advantage appears where the central node’s link is the constraint, and the paper establishes that by sweeping bandwidth and latency rather than by naming a threshold. | Active |
| SOTA-226 | If the gossip topology is static, make it a directed exponential graph, not a ring | Assran et al. (2018), LIT-254. Having chosen gossip over a parameter server, the next choice is the graph, and it is not a detail: a ring’s consensus rate degrades with the node count badly enough that the speedup regime recedes out of reach. A directed exponential graph — each node sends to peers at distance 1, 2, 4, 8, … in rotation — reaches exact consensus in log2(n) rounds with constant out-degree per round, and spreads the load evenly. | Active |
| SOTA-249 | Recompute activations from a sqrt(n) subset of checkpoints when activation memory is the binding constraint | Chen et al. (2016), LIT-004 — ARXIV-1604.06174. Store activations at O(sqrt(n)) checkpoints and recompute the rest during the backward pass. The price is one extra forward pass per minibatch; the measured case is a 1000-layer ResNet at 48G to 7G for 30% more wall clock. | Active |
| SOTA-364 | Break batch normalization’s cross-sample leak before it solves your contrastive task for you | Active | |
| SOTA-376 | Score each image-text pair independently with a sigmoid, so the loss needs no global normalization | Active | |
| SOTA-437 | Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base | Hershcovitch et al. (2024), LIT-750 — ZipNN. In regular trained weights only the float exponent is redundant: about 40 of its 256 values occur and it codes to about a third, while sign and mantissa barely compress. So split the exponent into its own stream and Huffman-code it without an LZ stage. Llama-3.1 BF16 goes to 66.4% against Zstd’s 77.7%, and is faster in both directions. Between checkpoints, XOR deltas against a base up to 10 checkpoints back compress better than standalone files. Sizes and throughputs are measured. Stall time inside a training run is not. | Proposed |