Jaghouar et al. (2024) — ARXIV-2407.07852
Key takeaways
Open-source implementation and replication of the DiLoCo method built on the Hivemind library, scaling it to 1.1B parameters and demonstrating real-world globally distributed training across 2 continents and 3 countries at 90–95% compute utilization.
- Replication result (150M model, 8 workers). DiLoCo achieves perplexity 13.73 vs. DDP baseline 13.68 at 500x less inter-island communication after 88k steps on C4. Holds when: 150M Llama; H=500; 8 H100 workers; C4 dataset; Nesterov outer optimizer.
- FP16 pseudo-gradient all-reduce (ablation). FP16 all-reduce of pseudo- gradients causes no measurable perplexity degradation versus FP32, halving communication volume. Holds when: 4 and 8 workers; 150M model; all other settings identical.
- Scaling to 1.1B parameters (4 workers, H=125). DiLoCo at 1.1B parameters with H=125 reaches perplexity within 0.24 of DDP while communicating 125x less after 44k steps. Holds when: 1.1B Llama; 4x H100 nodes; 44k steps; limited scaling experiment.
What the evidence does not cover
- DiLoCo shows slower FLOP efficiency than DDP during early training, requiring long runs to amortize the benefit.
- All-reduce synchronization creates idle time for faster workers; asynchronous outer-step execution is future work.
- Scaling beyond 8 workers does not yet match DDP compute efficiency.
- Experiments limited to up to 1.1B parameters; behavior at larger scales is untested.
Standing in the anthology
Read — the reading is NOTE-130. Arrived in the imported batch, which brought in the local-update branch — train apart for H steps, then average.