anthology-of-the-sota

distributed-optimization

Graph View

110 items with this tag.

Oct 03, 2026
LIT-783: Large Scale Diffusion Distillation via Score-Regularized Continuous-Time Consistency
Oct 01, 2026
LIT-735: Breaking the Computation and Communication Abstraction Barrier in Distributed Machine Learning Workloads
- distributed-optimization
- systems-optimization
Oct 01, 2026
LIT-736: Reducing Activation Recomputation in Large Transformer Models
- distributed-optimization
- systems-optimization
Oct 01, 2026
LIT-737: 10Cache: Heterogeneous Resource-Aware Tensor Caching and Migration for LLM Training
- distributed-optimization
- systems-optimization
Oct 01, 2026
LIT-739: TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems
Oct 01, 2026
LIT-750: ZipNN: Lossless Compression for AI Models
Oct 01, 2026
LIT-754: Efficient AlphaFold2 Training using Parallel Evoformer and Branch Parallelism
Oct 01, 2026
LIT-762: Git-Theta: A Git Extension for Collaborative Development of Machine Learning Models
- adaptation-and-tuning
- distributed-optimization
Oct 01, 2026
SOTA-437: Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base
Sep 24, 2026
LIT-619: Wan: Open and Advanced Large-Scale Video Generative Models
Sep 24, 2026
LIT-624: Step-Video-T2V Technical Report: The Practice, Challenges, and Future of Video Foundation Model
Sep 23, 2026
LIT-605: Sigmoid Loss for Language Image Pre-Training
Sep 23, 2026
SOTA-364: Break batch normalization's cross-sample leak before it solves your contrastive task for you
- model-stability
- distributed-optimization
Sep 23, 2026
SOTA-376: Score each image-text pair independently with a sigmoid, so the loss needs no global normalization
Sep 19, 2026
SOTA-249: Recompute activations from a sqrt(n) subset of checkpoints when activation memory is the binding constraint
- distributed-optimization
- systems-optimization
Sep 18, 2026
LIT-418: HybridFlow: A Flexible and Efficient RLHF Framework
- distributed-optimization
- systems-optimization
Sep 16, 2026
SOTA-222: Overlap the synchronisation with one or two local steps instead of communicating every step
- distributed-optimization
Sep 16, 2026
SOTA-223: Do not keep the gossip topology static: sample a fresh random neighbourhood every round
- distributed-optimization
Sep 16, 2026
SOTA-224: Ramp the sparsity ratio up over the first few epochs rather than starting at the target
- distributed-optimization
Sep 16, 2026
SOTA-225: Drop the parameter server for gossip once the network is the bottleneck, and measure where that is rather than inheriting a threshold
- distributed-optimization
Sep 16, 2026
SOTA-226: If the gossip topology is static, make it a directed exponential graph, not a ring
- distributed-optimization
Sep 16, 2026
THEORY-014: The outer optimizer is what buys the inner step count
- distributed-optimization
Sep 15, 2026
LIT-246: Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts
- distributed-optimization
Sep 15, 2026
LIT-252: OpenDiLoCo: An Open-Source Framework for Globally Distributed Low-Communication Training
- distributed-optimization
Sep 15, 2026
LIT-254: Stochastic Gradient Push for Distributed Deep Learning
- distributed-optimization
Sep 15, 2026
LIT-257: Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates
- distributed-optimization
Sep 15, 2026
LIT-273: Distributed Optimization with Gradient Tracking over Heterogeneous Delay-Prone Directed Networks
- distributed-optimization
Sep 15, 2026
LIT-276: Smoothing DiLoCo with Primal Averaging for Faster Training of LLMs
- distributed-optimization
Sep 15, 2026
LIT-278: 1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed
- distributed-optimization
Sep 15, 2026
LIT-279: HOGWILD!: A Lock-Free Approach to Parallelizing Stochastic Gradient Descent
- distributed-optimization
Sep 15, 2026
LIT-280: signSGD: Compressed Optimisation for Non-Convex Problems
- distributed-optimization
Sep 15, 2026
LIT-285: Coordination of groups of mobile autonomous agents using nearest neighbor rules
- distributed-optimization
- analysis-and-evaluation
Sep 15, 2026
LIT-288: Achieving Tighter Finite-Time Rates for Heterogeneous Federated Stochastic Approximation under Markovian Sampling
- distributed-optimization
Sep 15, 2026
LIT-292: Cooperative SGD: A unified Framework for the Design and Analysis of Communication-Efficient SGD Algorithms
- distributed-optimization
Sep 15, 2026
LIT-302: Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent
- distributed-optimization
Sep 15, 2026
LIT-303: SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
- distributed-optimization
Sep 15, 2026
LIT-307: Communication-Efficient Learning of Deep Networks from Decentralized Data
- distributed-optimization
Sep 15, 2026
LIT-314: Proving the Limited Scalability of Centralized Distributed Optimization via a New Lower Bound Construction
- distributed-optimization
Sep 15, 2026
LIT-315: Epidemic Learning: Boosting Decentralized Learning with Randomized Communication
- distributed-optimization
Sep 15, 2026
LIT-316: Distributed Deep Learning in Open Collaborations
- distributed-optimization
Sep 15, 2026
LIT-323: Overlap Local-SGD: An Algorithmic Approach to Hide Communication Delays in Distributed SGD
- distributed-optimization
Sep 15, 2026
LIT-325: Distributed asynchronous deterministic and stochastic gradient optimization algorithms
- distributed-optimization
Sep 15, 2026
LIT-336: Async-HFL: Efficient and Robust Asynchronous Federated Learning in Hierarchical IoT Networks
- distributed-optimization
Sep 15, 2026
LIT-337: PowerSGD: Practical Low-Rank Gradient Compression for Distributed Optimization
- distributed-optimization
Sep 15, 2026
LIT-344: Byzantine-Tolerant Machine Learning
- distributed-optimization
Sep 15, 2026
LIT-345: Generalized EXTRA stochastic gradient Langevin dynamics
- distributed-optimization
Sep 15, 2026
LIT-350: Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelism
- distributed-optimization
Sep 15, 2026
LIT-354: Asynchronous Stochastic Gradient Descent with Delay Compensation
- distributed-optimization
Sep 15, 2026
LIT-361: Local SGD Converges Fast and Communicates Little
- distributed-optimization
Sep 15, 2026
LIT-362: Don't Use Large Mini-Batches, Use Local SGD
- distributed-optimization
Sep 15, 2026
LIT-366: Asynchronous Local Computations in Distributed Bayesian Learning
- distributed-optimization
Sep 15, 2026
LIT-370: Deep learning with Elastic Averaging SGD
- distributed-optimization
Sep 15, 2026
LIT-372: Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices
- distributed-optimization
Sep 15, 2026
LIT-373: SlowMo: Improving Communication-Efficient Distributed SGD with Slow Momentum
- distributed-optimization
Sep 15, 2026
LIT-374: LAYUP: Asynchronous decentralized gradient descent with LAYer-wise UPdates
- distributed-optimization
Sep 15, 2026
SOTA-214: Pair any gradient compressor with error feedback, and correct the momentum it is applied under
- distributed-optimization
Sep 15, 2026
SOTA-215: Freeze Adam's variance term after a warmup before compressing its update, and never compress the update directly
- distributed-optimization
Sep 15, 2026
SOTA-216: Switch from minibatch SGD to local SGD at the first learning-rate decay, rather than choosing between them
- distributed-optimization
Sep 15, 2026
SOTA-219: Leave the inner optimizer state unsynchronised in local-update training
- distributed-optimization
Sep 15, 2026
SOTA-220: Aggregate worker gradients coordinate-wise by median when any worker may be faulty
- distributed-optimization
Sep 09, 2026
LIT-219: PyTorch Distributed: Experiences on Accelerating Data Parallel Training
- distributed-optimization
Sep 09, 2026
LIT-220: Horovod: fast and easy distributed deep learning in TensorFlow
- distributed-optimization
Sep 09, 2026
LIT-222: CoreWeave Training Benchmarks Whitepaper
- distributed-optimization
Sep 09, 2026
SOTA-189: Set the checkpoint interval from the job's measured time-to-failure, which shrinks as the job grows
- distributed-optimization
Sep 08, 2026
SOTA-181: Shard the sequence across devices in a ring and overlap the key-value exchange with the attention it feeds
- distributed-optimization
Sep 07, 2026
LIT-187: GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
- distributed-optimization
- model-architecture
Sep 07, 2026
LIT-206: Ring Attention with Blockwise Transformers for Near-Infinite Context
- distributed-optimization
Sep 07, 2026
LIT-212: DiLoCo: Distributed Low-Communication Training of Language Models
- distributed-optimization
Sep 07, 2026
LIT-214: Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
- distributed-optimization
Sep 07, 2026
SOTA-155: Train across poorly connected islands: many inner steps per worker, an outer momentum optimizer over the deltas, and a streamed synchronisation
- distributed-optimization
Sep 05, 2026
LIT-158: A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale
- distributed-optimization
- training-optimization
Aug 24, 2026
LIT-004: Training Deep Nets with Sublinear Memory Cost
- distributed-optimization
- systems-optimization
Aug 24, 2026
LIT-007: Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour
- distributed-optimization
- training-optimization
Aug 24, 2026
LIT-016: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism
- distributed-optimization
Aug 24, 2026
LIT-022: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism
- distributed-optimization
- model-stability
Aug 24, 2026
LIT-026: Advances and Open Problems in Federated Learning
- distributed-optimization
Aug 24, 2026
LIT-027: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models
- distributed-optimization
Aug 24, 2026
LIT-043: Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM
- distributed-optimization
- model-stability
Aug 24, 2026
LIT-051: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters
- distributed-optimization
Aug 24, 2026
LIT-056: Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training
- distributed-optimization
Aug 24, 2026
LIT-059: CheckFreq: Frequent, Fine-Grained DNN Checkpointing
- systems-optimization
- distributed-optimization
Aug 24, 2026
LIT-065: Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model
- distributed-optimization
- training-optimization
Aug 24, 2026
LIT-083: PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
- distributed-optimization
Aug 24, 2026
LIT-098: PipeMare: Asynchronous Pipeline Parallel DNN Training
- distributed-optimization
Aug 24, 2026
LIT-102: ZeRO++: Extremely Efficient Collective Communication for Giant Model Training
- distributed-optimization
- numerics-and-precision
Aug 24, 2026
SOTA-017: Use micro-batch splitting for pipeline parallelism
- distributed-optimization
Aug 24, 2026
SOTA-018: Balance pipeline stages to minimize bubble overhead
- distributed-optimization
Aug 24, 2026
SOTA-019: Choose pipeline chunks based on memory vs. compute trade-off
- distributed-optimization
Aug 24, 2026
SOTA-028: Stage optimizer states across data parallel ranks (ZeRO-1)
- distributed-optimization
Aug 24, 2026
SOTA-029: Partition gradients and optimizer states (ZeRO-2) for larger models
- distributed-optimization
Aug 24, 2026
SOTA-030: Use ZeRO-3 only when other strategies insufficient
- distributed-optimization
Aug 24, 2026
SOTA-031: Keep micro-batch size per GPU as large as memory allows
- distributed-optimization
- training-optimization
Aug 24, 2026
SOTA-046: Use hierarchical allreduce for tensors > 1MB
- distributed-optimization
Aug 24, 2026
SOTA-047: Overlap communication with backward pass
- distributed-optimization
Aug 24, 2026
SOTA-048: Group small tensors before communication
- distributed-optimization
Aug 24, 2026
SOTA-049: Set buffer size to network bandwidth-delay product
- distributed-optimization
Aug 24, 2026
SOTA-054: Derive the checkpoint interval from online profiling and adapt it at runtime against an overhead bound
- distributed-optimization
Aug 24, 2026
SOTA-055: Save optimizer state every N epochs (N ~ sqrt(total_epochs))
- distributed-optimization
Aug 24, 2026
SOTA-056: Use async I/O for checkpoint writing
- distributed-optimization
Aug 24, 2026
SOTA-057: Implement multi-level checkpoint strategy
- distributed-optimization
Aug 24, 2026
SOTA-058: Use sequence parallelism for the layer-norm and dropout regions that tensor parallelism leaves replicated
- distributed-optimization
Aug 24, 2026
SOTA-059: Overlap communication with computation when possible
- distributed-optimization
Aug 24, 2026
SOTA-073: Monitor network utilization during training
- distributed-optimization
Aug 24, 2026
SOTA-074: Adapt buffer sizes to network conditions
- distributed-optimization
Aug 24, 2026
SOTA-075: Use gradient compression for slow networks
- distributed-optimization
Aug 24, 2026
SOTA-076: Place replicas to minimize cross-rack traffic
- distributed-optimization
Aug 24, 2026
SOTA-116: Use FSDP over DDP when model size exceeds single GPU memory
- distributed-optimization
Aug 24, 2026
SOTA-117: Overlap communication with computation using backward prefetch
- distributed-optimization
Aug 24, 2026
SOTA-118: Employ mixed precision to reduce memory usage
- numerics-and-precision
- distributed-optimization
Aug 24, 2026
SOTA-119: Choose sharding factor based on model and GPU memory size
- distributed-optimization

Created with Quartz v5.0.0 © 2026

Repository
Quartz