Source
Douillard et al. (2023), LIT-212 — DiLoCo; Douillard et al. (2025), LIT-214 — Streaming DiLoCo, for the bandwidth half.
The constraint being attacked is co-location, not bandwidth. Standard distributed training exchanges gradients every step, so every accelerator has to sit behind a low-latency high-bandwidth link — which means one cluster, and a big one. The premise here is that several small clusters are easier to obtain than one large one.
What you run:
-
Many inner steps per worker, with AdamW as the inner optimizer, between synchronisations.
-
Nesterov momentum as the outer optimizer, applied to the accumulated worker deltas, at outer learning rate 0.7 and outer momentum 0.9. This is the part that is not standard federated averaging, and it is what recovers the quality that infrequent synchronisation would otherwise cost — LIT-212’s ablation reports outer SGD, which is plain averaging, and outer Adam both performing poorly at the same interval.
This component is older than DiLoCo and was not its invention. LIT-373 applied slow outer momentum to local SGD and to gossip optimizers in 2019, and LIT-212’s related work names it as having “extended [federated averaging] to more powerful outer optimizers”, work that “inspired our use of Nesterov momentum”. Reading the two together is what THEORY-014 does, and it is why this practice’s order matters: the inner step count is what the outer optimizer buys, not the other way round.
-
Streamed, overlapped, quantised synchronisation: exchange subsets of parameters in sequence rather than the whole model at once, let workers keep training while an exchange is in flight, and quantise what crosses the link.
Why both papers, and why the second is not optional
DiLoCo alone reduces how often the link is used and not how much it must
carry: every synchronisation still exchanges all parameters across all
workers, so peak bandwidth is what fully synchronous training needs. A link
provisioned for the peak is a link that was never cheap, which is why a
recommendation resting on the first paper alone would be recommending
something whose economic case is unmade. The second paper is filed as
corrects: the first for exactly that reason (ADR-017).
LIT-214 is that second paper, and the third bullet above is its design: synchronise fragments of the model in sequence so the peak drops, overlap each exchange with continued inner steps so latency is tolerated, and send the outer gradients in four bits with accumulation kept in FP32. Combined, it reports billion-scale training at quality comparable to data-parallel training with about 400× less bandwidth, which is the measurement the economic case rests on.
Conditions, and what is not established
Reported results: 500× less communication at matched quality on C4 with 8 workers against fully synchronous optimization, robust to workers holding differently distributed data and to resources disappearing mid-run; then two orders of magnitude less bandwidth at billion scale with the streamed form, at comparable quality.
Eight workers on C4 is a long way from the arrangement the motivation describes, and one group published both halves. No report in this record trains a released model this way.
What this is not advice about: the record’s existing distributed material — pipeline parallelism, the ZeRO stages, allreduce tuning, sharding factors — is tuning within a synchronous regime. This is a different regime. Those practices neither apply here nor conflict with this one, and reading either as contradicting the other is a category error.
Known implementations
- None in this record. The published results are the authors’ own.