Ryabinin and Gusev (2020) — ARXIV-2002.04013
Key takeaways
Proposes Learning@home, the first training paradigm and infrastructure enabling crowdsourced distributed training of large neural networks on volunteer hardware, using Decentralized Mixture-of-Experts (DMoE) layers and a Kademlia DHT for fault-tolerant, low-bandwidth expert discovery.
- DHT lookup latency scaling. Beam search latency grows from 317ms at 100 nodes to 764ms at 10,000 nodes — sub-linear (empirical O(log N)). Holds when: Kademlia DHT on real cloud instances; d-dimensional grid beam search with fixed beam width.
- Throughput under latency (empirical). DMoE throughput is near-constant for network latency up to 200ms; dense model-parallel throughput degrades by 2-5x over the same range. Holds when: 4x GTX 1080 GPUs; simulated exponential latency distribution; asynchronous Trainer/Runtime architecture.
- Fault tolerance under 10% failure rate (empirical). DMoE with 256 experts converges on MNIST and WikiText-2 under 10% node failure rate with minimal accuracy degradation vs. no failures. Holds when: Crash-stop failures; DHT re-routes expert queries to surviving nodes; no explicit convergence rate formula provided.
What the evidence does not cover
- Only demonstrated at small scale (MNIST, WikiText-2); scalability to truly large models is theoretical.
- Security vulnerabilities: malicious experts can poison gradients propagated through backpropagation.
- DHT lookup latency (hundreds of ms) is non-trivial and grows with network size.
- No mechanism for consistent model checkpointing or recovery of the global model state.
- Asynchronous training introduces stale gradients; theoretical convergence guarantees are limited.
Standing in the anthology
Read — the reading is NOTE-137. Arrived in the imported batch, which brought in the volunteer- and internet-scale branch, where workers join and leave.