anthology-of-the-sota

LIT-051: A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters

1 min read

Properties4
StatusActive
Tagsdistributed-optimization
Cited as “introduced by” bySOTA-046 — Use hierarchical allreduce for tensors > 1MB, SOTA-049 — Set buffer size to network bandwidth-delay product, SOTA-073 — Monitor network utilization during training, SOTA-074 — Adapt buffer sizes to network conditions, SOTA-076 — Place replicas to minimize cross-rack traffic
Cited as “source” bySOTA-046 — Use hierarchical allreduce for tensors > 1MB, SOTA-049 — Set buffer size to network bandwidth-delay product, SOTA-073 — Monitor network utilization during training, SOTA-074 — Adapt buffer sizes to network conditions, SOTA-076 — Place replicas to minimize cross-rack traffic

Graph View

Jiang et al., Tsinghua University and ByteDance (2020) — OSDI ‘20, usenix.org/conference/osdi20/presentation/jiang. Known by its implementation’s name, BytePS.

Key takeaways

  • All-reduce and parameter-server are not two competing architectures but two special cases of one communication framework, which the paper derives and proves optimal for a cluster holding both GPUs and spare CPU and network capacity.
  • Reaching that optimum in practice means splitting the optimizer: a Summation Service that aggregates gradients — common to every optimizer, AVX-friendly, run on CPUs — and the model-dependent optimizer step, left on the GPUs.
  • On jobs up to 256 GPUs, BytePS beats the best open-source all-reduce by up to 84% and the best parameter-server by up to 245%.

Standing in the anthology

Filed against its published title rather than its tool name. The note’s substance is still the four-bullet stub the migration carried; what changed is that the citation now resolves to the work the entry names.

Backlinks