Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Distributed optimization — parallelism and sharding, communication, memory management, checkpointing.
67 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-004 v4 | Training Deep Nets with Sublinear Memory Cost | Chen et al. (2016), ARXIV-1604.06174. Gradient checkpointing: store activations at O(sqrt(n)) layers and recompute the segments between them in the backward pass, for the cost of one extra forward pass per minibatch. A 1,000-layer ResNet drops from 48G to 7G for 30% more running time. | Active |
| LIT-007 | Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour | Goyal et al. (2017), ARXIV-1706.02677. Linear scaling rule for learning rates with batch size. | Active |
| LIT-016 v2 | GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism | Huang et al. (2018), ARXIV-1811.06965. Introduces pipeline parallelism for training. | Active |
| LIT-022 v2 | Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | Shoeybi et al. (2019), ARXIV-1909.08053. Model parallel transformers. | Active |
| LIT-026 v3 | Advances and Open Problems in Federated Learning | Kairouz et al. (2019), ARXIV-1912.04977. Comprehensive survey of federated learning. | Active |
| LIT-027 | ZeRO: Memory Optimizations Toward Training Trillion Parameter Models | Rajbhandari et al. (2020), ARXIV-1910.02054. Micro-batch size determined by GPU memory. | Active |
| LIT-043 | Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM | Narayanan et al. (2021), ARXIV-2104.04473. 3D parallel training strategy. | Active |
| LIT-051 | A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters | Jiang et al. (2020), OSDI ‘20. BytePS: spare CPU and network capacity in a GPU cluster is a resource, and a communication framework that uses it subsumes both all-reduce and parameter-server as special cases. | Active |
| LIT-056 v2 | Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training | Lin et al. (2017), ARXIV-1712.01887. 99.9% of the gradient exchange in distributed SGD is redundant; sending the rest, with four corrections that keep accuracy, compresses gradients 270–600× and puts multi-node training on commodity 1Gbps Ethernet. | Active |
| LIT-059 v2 | CheckFreq: Frequent, Fine-Grained DNN Checkpointing | Mohan et al. (2021), FAST ‘21. Checkpoint frequency should be computed, not guessed: online profiling picks an iteration-granularity interval, adaptive tuning holds the overhead under a bound, and two-phase checkpointing pipelines the write against compute. | Active |
| LIT-065 v2 | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model | Smith et al. (2022), ARXIV-2201.11990. Large scale training techniques. | Active |
| LIT-083 v2 | PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel | Zhao et al. (2022), ARXIV-2304.11277. Improved memory efficiency over ZeRO. | Active |
| LIT-098 v2 | PipeMare: Asynchronous Pipeline Parallel DNN Training | Yang et al. (2019), ARXIV-1910.05124. Asynchronous pipeline parallelism. | Active |
| LIT-102 v3 | ZeRO++: Extremely Efficient Collective Communication for Giant Model Training | Rajbhandari et al. (2023), ARXIV-2306.10209. Communication optimization. | Active |
| LIT-158 | A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale | Shi et al. (2023), ARXIV-2309.06497. The engineering that made a second-order optimizer usable at scale: block-diagonal Kronecker preconditioners sharded across GPUs, at most 10% per-step wall-clock over diagonal adaptive methods. | Active |
| LIT-187 v2 | GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding | Lepikhin et al. (2020), ARXIV-2006.16668. Puts the mixture-of-experts layer into a transformer and shards it across 2048 TPUs with annotations rather than a rewrite: a 600B multilingual translation model trained in four days, and the top-2 routing that later work names as the thing to fix. | Active |
| LIT-206 | Ring Attention with Blockwise Transformers for Near-Infinite Context | Liu et al. (2023), ARXIV-2310.01889. Compute attention and the feedforward blockwise, distribute the blocks across devices in a ring, and overlap the communication of key/value blocks with the computation of the next block. Sequence length scales with device count — no approximation, no extra communication or computation cost — which is what makes million-token contexts a sharding question rather than a memory ceiling. | Active |
| LIT-212 | DiLoCo: Distributed Low-Communication Training of Language Models | Douillard et al. (2023), ARXIV-2311.08105. Training on islands of badly connected devices instead of one tightly coupled cluster: many inner AdamW steps per worker, Nesterov momentum as the outer optimizer over the resulting deltas, synchronised rarely. On C4 with 8 workers it matches fully synchronous training while communicating 500× less, and tolerates workers appearing and disappearing mid-run. | Active |
| LIT-214 | Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch | Douillard et al. (2025), ARXIV-2501.18512. DiLoCo made synchronisation rare and left its size alone, so peak bandwidth was unchanged. Three changes fix that: synchronise subsets of parameters in sequence rather than all at once, let workers keep training while they synchronise, and quantise what crosses the link. Billion-scale training at matched quality with two orders of magnitude less bandwidth. | Active |
| LIT-219 | PyTorch Distributed: Experiences on Accelerating Data Parallel Training | Li et al. (2020), ARXIV-2006.15704. The design and evaluation of PyTorch’s DistributedDataParallel, whose three acceleration techniques are “bucketing gradients, overlapping computation with communication, and skipping gradient synchronization” — near-linear scalability on 256 GPUs. | Active |
| LIT-220 | Horovod: fast and easy distributed deep learning in TensorFlow | Sergeev and Del Balso (2018), ARXIV-1802.05799. Ring-allreduce for TensorFlow, and the origin of Tensor Fusion — batching small tensors before the collective, because ring-allreduce “utilizes the network in an optimal way if the tensors are large enough, but does not work as efficiently or quickly if they are very small.” | Active |
| LIT-222 | CoreWeave Training Benchmarks Whitepaper | CoreWeave (2025). Fits a right-censored exponential survival model to real job durations and interruptions, giving a per-GPU failure rate of 3,748.25 days/failure and an MTTF of 3.66 days at 1,024 GPUs — the quantity a checkpoint interval should be chosen against, and one the record had no source for. | Active |
| LIT-246 | Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts | Ryabinin and Gusev (2020), ARXIV-2002.04013. Learning@home: train a mixture of experts spread over unreliable volunteer hardware, routed through a DHT, so no worker holds the whole model. | Active |
| LIT-252 | OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training | Jaghouar et al. (2024), ARXIV-2407.07852. OpenDiLoCo: a reproduction and open implementation of DiLoCo, trained across continents at 90-95% compute utilization. | Active |
| LIT-254 | Stochastic Gradient Push for Distributed Deep Learning | Assran et al. (2018), ARXIV-1811.10792. Stochastic gradient push over directed, time-varying graphs, with the communication overlapped against the next local step. | Active |
| LIT-257 | Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates | Yin et al. (2018), ARXIV-1803.01498. Coordinate-wise trimmed mean and coordinate-wise median as Byzantine-robust aggregators, with statistical rates that are order-optimal, and the median needs no knowledge of the corruption fraction. | Active |
| LIT-273 | Distributed Optimization with Gradient Tracking over Heterogeneous Delay-Prone Directed Networks | Makridis et al. (2025), ARXIV-2504.10964. Gradient tracking over directed networks with heterogeneous delays, with an a-priori step-size bound computed from the delay bound. | Active |
| LIT-276 | Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs | Defazio et al. (2025), ARXIV-2512.17131. Replace DiLoCo’s outer optimizer with primal averaging, making the smoothing strength a continuous hyperparameter independent of the inner step count. | Active |
| LIT-278 | 1-bit Adam: Communication Efficient Large-Scale Training with Adam’s Convergence Speed | Tang et al. (2021), ARXIV-2102.02888. Adam’s variance term stabilizes early in training; freeze it after a warmup and the update becomes linear enough for error-compensated 1-bit compression to work. | Active |
| LIT-279 | HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent | Niu et al. (2011), ARXIV-1106.5730. Lock-free parallel SGD on shared memory: let cores overwrite each other without locks, and when the problem is sparse enough the collisions are rare and the convergence rate is nearly the same. | Active |
| LIT-280 | signSGD: Compressed Optimisation for Non-Convex Problems | Bernstein et al. (2018), ARXIV-1802.04434. Sending only the sign of each gradient coordinate is a 32x compression that still converges, and with majority-vote aggregation the reduction also applies to the return path. | Active |
| LIT-285 v2 | Coordination of groups of mobile autonomous agents using nearest neighbor rules | Jadbabaie, Lin and Morse (2003), DOI:10.1109/TAC.2003.812781. Nearest-neighbour heading averaging on undirected graphs converges to a common value for any switching sequence whose graphs are jointly connected across contiguous, bounded time intervals. The proof is via Wolfowitz’s ergodic-product theorem. It is asymptotic only, with no rate, and its priority is contested by Bertsekas and Tsitsiklis against their 1980s agreement results (LIT-325). The system is shown stable although no common quadratic Lyapunov function exists. | Active |
| LIT-288 | Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling | Zhu et al. (2025), ARXIV-2504.11645. Tighter finite-time rates for heterogeneous federated stochastic approximation under Markovian sampling, where naive local averaging converges to the wrong fixed point. | Active |
| LIT-292 | Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms | Wang and Joshi (2018), ARXIV-1808.07576. Cooperative SGD: one framework whose special cases are local SGD, elastic averaging and decentralized SGD, so their error floors can be compared in the same terms. | Active |
| LIT-302 | Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent | Lian et al. (2017), ARXIV-1705.09056. Decentralized parallel SGD, in which workers average only with neighbours, matches centralized SGD in rate while removing the server bottleneck, and wins outright on low-bandwidth or high- latency networks. | Active |
| LIT-303 | SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient | Ryabinin et al. (2023), ARXIV-2301.11913. SWARM parallelism: pipeline stages over unreliable heterogeneous workers connected by slow links, rebalanced as workers come and go. | Active |
| LIT-307 | Communication-Efficient Learning of Deep Networks from Decentralized Data | McMahan et al. (2016), ARXIV-1602.05629. FedAvg: average the weights of clients that each ran several local epochs, rather than averaging one gradient per round, and the round count falls by one to two orders of magnitude. | Active |
| LIT-314 | Proving the Limited Scalability of Centralized Distributed Optimization via a New Lower Bound Construction | Tyurin (2025), ARXIV-2506.23836. A lower-bound construction showing that unbiased compression cannot improve both the uplink and downlink terms at once, which bounds what centralized compression schemes can buy. | Active |
| LIT-315 | Epidemic Learning: Boosting Decentralized Learning with Randomized Communication | Vos et al. (2023), ARXIV-2310.01972. Epidemic Learning: sample a fresh random set of s neighbours each round instead of using a fixed topology, which converges faster than any static graph. | Active |
| LIT-316 | Distributed Deep Learning in Open Collaborations | Diskin et al. (2021), ARXIV-2106.10207. DeDLOC: collaborative training across volunteered hardware with adaptive averaging, keeping the run equivalent to standard large-batch SGD at a fixed global batch size. | Active |
| LIT-323 | Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD | Wang et al. (2020), ARXIV-2002.09539. Overlap Local-SGD hides the averaging latency behind local computation using an anchor model, so small H becomes affordable. | Active |
| LIT-325 | Distributed asynchronous deterministic and stochastic gradient optimization algorithms | Tsitsiklis et al. (1986), DOI:10.1109/TAC.1986.1104412. The founding analysis of distributed asynchronous gradient methods: convergence holds for any finite delay bound B, and the step size must shrink with B. | Active |
| LIT-336 | Async-HFL: Efficient and Robust Asynchronous Federated Learning in Hierarchical IoT Networks | Yu et al. (2023), ARXIV-2301.06646. Async-HFL: hierarchical federated learning over long-tailed wireless delays, selecting devices by gradient diversity rather than by latency alone. | Active |
| LIT-337 | PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization | Vogels et al. (2019), ARXIV-1905.13727. PowerSGD compresses the gradient to a rank-k factorization found by one power-iteration step, which is all- reducible, and error feedback is what makes it converge. | Active |
| LIT-344 | Byzantine-Tolerant Machine Learning | Blanchard et al. (2017), ARXIV-1703.02757. Averaging gradients is not Byzantine-tolerant at any cluster size — one worker can steer the average anywhere. Krum instead selects the gradient closest to its own nearest neighbours. | Active |
| LIT-345 | Generalized EXTRA stochastic gradient Langevin dynamics | Gurbuzbalaban et al. (2024), ARXIV-2412.01993. A generalized EXTRA correction removes the network-induced bias that decentralized SGLD otherwise leaves in the sampled posterior. | Active |
| LIT-350 | Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism | Ramasinghe et al. (2025), ARXIV-2506.01260. Project the model-parallel communication onto a low-rank subspace updated every few hundred steps, reaching about 100x compression for pipeline-parallel decentralized training. | Active |
| LIT-354 | Asynchronous Stochastic Gradient Descent with Delay Compensation | Zheng et al. (2016), ARXIV-1609.08326. Asynchronous SGD applies a gradient computed at stale parameters; a first-order Taylor correction using a cheap Hessian approximation recovers most of the accuracy the delay costs. | Active |
| LIT-361 | Local SGD Converges Fast and Communicates Little | Stich (2018), ARXIV-1805.09767. Local SGD keeps the linear speedup of minibatch SGD while communicating only every H = O(sqrt(T/(Kb))) steps. | Active |
| LIT-362 | Don’t Use Large Mini-Batches, Use Local SGD | Lin et al. (2018), ARXIV-1808.07217. Post-local SGD: run minibatch SGD until the first learning-rate decay and switch to local SGD after it, recovering the generalization that large-batch training loses. | Active |
| LIT-366 | Asynchronous Local Computations in Distributed Bayesian Learning | Bhar et al. (2023), ARXIV-2311.03496. Asynchronous decentralized Langevin sampling with several local unadjusted-Langevin steps per communication round. | Active |
| LIT-370 | Deep learning with Elastic Averaging SGD | Zhang et al. (2014), ARXIV-1412.6651. Let each worker keep its own parameters and pull it toward a centre variable with an elastic force, so workers may explore apart and communication can be infrequent. | Active |
| LIT-372 | Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices | Ryabinin et al. (2021), ARXIV-2103.03239. Moshpit SGD: average over a random grid of small groups rather than by gossip, reaching exponentially fast averaging on unreliable heterogeneous workers. | Active |
| LIT-373 | SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum | Wang et al. (2019), ARXIV-1910.00643. SlowMo: wrap any local-update or decentralized base optimizer in an outer momentum step over the averaged iterates. | Active |
| LIT-374 | LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates | Fokam et al. (2024), ARXIV-2410.05985. LAYUP: decouple the forward and backward passes into separate threads with layer-wise updates, so straggler devices stop setting the step time. | Active |
| LIT-418 | HybridFlow: A Flexible and Efficient RLHF Framework | Sheng et al. (2024), ARXIV-2409.19256. The paper behind verl — a hybrid single/multi-controller execution model for the RLHF dataflow, with a 3D resharding engine between the training and generation phases. Named in ADR-032 as one of the four references at the top of the backlog it opened. | Active |
| LIT-605 v2 | Sigmoid Loss for Language Image Pre-Training | Zhai et al. (2023), ARXIV-2303.15343. Replace CLIP’s softmax with a pairwise sigmoid, so the loss never needs a global view of the similarity matrix — no all-gathers, only a per-device block in memory, and better results below 16k batch. Then trains at batch size one million and finds the benefit saturated at 32k. | Active |
| LIT-619 v2 | Wan: Open and Advanced Large-Scale Video Generative Models | Wan Team, Alibaba (2025), ARXIV-2503.20314. The open Wan2.1 video models, 1.3B and 14B, and the full recipe behind them. The recipe covers data curation, a 127M causal 3D VAE compressing 4×8×8, a cross-attention DiT trained by rectified flow on an image-then-video resolution curriculum, the 2D context parallelism it needs, and a mask-conditioned image-to-video variant. The abstract claims video scaling laws in data and model size, but the report plots no scaling curve. | Active |
| LIT-624 v3 | Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model | Step-Video Team, StepFun (2025), ARXIV-2502.10248. A 30B text-to-video DiT on a 16×16×8 VAE that reconstructs as well as HunyuanVideo’s 8×8×4 on its own benchmark, though not on Open-Sora 2.0’s, so the model covers 204 frames. It also applies DPO to video (a 55/45 preference score, ties split) and runs at 32% MFU. It reaches parity with Movie Gen and loses to the strongest commercial model it names, despite the abstract’s “state-of-the-art”. | Active |
| LIT-735 | Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads | Jangda et al. (2021), ARXIV-2105.05720 — CoCoNet (ASPLOS 2022). A DSL and compiler that treat computation and collectives as one program, so they can be fused and overlapped. Overlapping a tensor-parallel MatMul with its AllReduce in fine-grained chunks hides over 80% of the MatMul and runs 1.36x faster than issuing them in sequence; in Megatron-LM, the overlapped schedules speed up model-parallel inference 1.48-1.51x and pipeline-parallel inference further, and the data-parallel gain on BERT training comes from fusing the optimizer into the collective rather than from overlap. | Active |
| LIT-736 | Reducing Activation Recomputation in Large Transformer Models | Korthikanti et al. (2022), ARXIV-2205.05198. Introduces sequence parallelism: the layer norms and dropouts that tensor parallelism leaves replicated are split along the sequence dimension, and the tensor-parallel all-reduces become an all-gather and a reduce-scatter that move the same bytes, so activation memory per layer falls by the tensor-parallel degree at no extra communication. With selective recomputation of the attention core it cuts activation memory about 5x, and over full recomputation lifts end-to-end throughput 29.0-32.1% on models from 22B to 1T parameters. | Active |
| LIT-737 | 10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training | Afroz et al. (2025), ARXIV-2511.14124 (SoCC ‘25). A tensor caching and migration system for offloaded LLM training across GPU, CPU and NVMe, reporting up to 2× faster training than prior offloading methods. Its motivating measurement is the one the record uses: on an L40S over PCIe 4.0 ×16, a 16 MB FP32 host-to-device copy takes 1.65 ms (10.16 GB/s) from pageable memory and 0.68 ms (24.74 GB/s) from pinned memory. | Active |
| LIT-739 | TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems | Abadi et al. (2016), ARXIV-1603.04467 — the TensorFlow white paper (dated November 2015). Describes TensorFlow’s dataflow-graph interface and its execution from a single device to clusters; among its graph features are input operations that read from storage directly and queues that let input be prefetched from disk while the previous batch is still being computed. | Active |
| LIT-750 | ZipNN: Lossless Compression for AI Models | Hershcovitch et al. (2024), ARXIV-2411.05239. Losslessly compressing a trained model’s file works almost entirely through the float exponent: about 40 of 256 values occur, and the exponent stream codes to ~33% while sign and mantissa stay near 100%. Regular BF16 models therefore shrink to ~66%, FP32 to ~83%, and models rounded after training (“clean”) to as little as 34–48%. Splitting the exponent into its own stream and coding it with Huffman alone, without LZ, beats Zstd by 17% in size and 62% in single-thread speed on Llama-3.1 BF16. | Active |
| LIT-754 | Efficient AlphaFold2 Training using Parallel Evoformer and Branch Parallelism | Wang et al. (2022), ARXIV-2211.00235. Move the Evoformer’s outer product mean to the end of the block, so the MSA and pair stacks no longer depend on each other within a block; this matches the original block’s accuracy and speed. Then put the two stacks on two GPUs. On a batch of one protein per device, where data parallelism cannot scale further, this cuts end-to-end AlphaFold2 training from 5.8 to 4.2 days on UniFold and from 6.7 to 4.9 days on HelixFold, using twice the GPUs. | Active |
| LIT-762 | Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models | Kandpal et al. (2023), ARXIV-2306.04529 — Git-Theta, a Git extension that versions a checkpoint per parameter group rather than as one blob. Unchanged groups (detected with a locality-sensitive hash) are stored as references; changed groups are stored as typed updates (dense, sparse, low-rank, (IA)³) that are reconstructed by walking back through history. In one T0-3B workflow it stored 41.5 GB against Git LFS’s 57.0 GB, the saving coming almost entirely from a LoRA commit (0.27 GB against 11.4); dense fine-tunes were stored nearly in full, and every add was slower than Git LFS. | Active |
| LIT-783 | Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency | Zheng et al., Tsinghua and NVIDIA (2025), ARXIV-2510.08431. sCM made to run on 14B text-to-image and text-to-video models, with a FlashAttention-2 JVP kernel under FSDP and context parallelism. Pure sCM loses fine detail and temporal coherence there. Adding a DMD loss at weight 0.01 (rCM) fixes it. Four-step Wan2.1 VBench is 84.43 at 1.3B against a re-implemented DMD2’s 84.56, and 84.92 at 14B; GenEval tracks the teacher at 1–4 steps. The diversity advantage over DMD2 is shown in samples only. One run per row. | Active |