Diskin et al. (2021) — ARXIV-2106.10207
Key takeaways
Proposes DeDLOC, an algorithmic framework and system for collaborative deep learning on heterogeneous volunteer hardware, featuring an adaptive gradient-averaging algorithm that dynamically selects the optimal communication strategy (AR-SGD, parameter server, BytePS, or hybrid) based on current peer capabilities; demonstrated in a real 40-person collaborative ALBERT pretraining for the Bengali language.
- Mathematical equivalence to large-batch SGD (informal). Accumulating to a fixed global batch size before each optimizer step makes the update sequence identical in distribution to large-batch synchronous SGD, regardless of how many peers contributed each round. Holds when: Holds exactly when all gradients are accumulated before the optimizer step; DPU introduces one step of staleness, which is negligible in practice.
- Adaptive averaging speedup (empirical). LP-selected adaptive strategy runs 1.9x faster than ring All-Reduce on mixed server+workstation hardware with no loss in model quality. Holds when: 4 hardware configurations; 100 averaging rounds; ALBERT- large model.
- Fault tolerance via group averaging (empirical). Group-based averaging with group size m limits failure blast radius to 1/m of the global batch; the remaining groups proceed and the missed gradient contribution is simply absent from that step. Holds when: Groups of size ~8 used in sahajBERT; node dropout rate not formally characterized.
What the evidence does not cover
- Requires recruiting and retaining volunteers, which introduces social and organizational challenges outside the algorithm.
- Authentication and contribution accounting are rudimentary; no defense against gradient poisoning by malicious peers.
- Data-parallel approach limits model size to what fits in the memory of a single participant’s GPU.
- LP-based strategy optimization adds overhead and may not scale to very large collaborations.
- Part-time volunteer participation creates throughput variance that is difficult to plan for.
Standing in the anthology
Read — the reading is NOTE-099. Arrived in the imported batch, which brought in the volunteer- and internet-scale branch, where workers join and leave.