anthology-of-the-sota

SOTA-049: Set buffer size to network bandwidth-delay product

1 min read

Properties4
StatusRejected — collapses two different quantities into one unfollowable sentence — a TCP socket buffer sized by bandwidth-delay product, and a gradient fusion buffer sized by the overlap trade. The fusion-buffer half is SOTA-048, which now carries Horovod's attributable 64 MB default; the transport half is a claim about networks with no source here
Tagsdistributed-optimization
SourceLIT-051 — A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters
Introduced byLIT-051 — A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters

Graph View

Source

Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.

What is wrong with this one

The bandwidth-delay product is the amount of data in flight on a link at full rate, and it is the right sizing rule for a TCP socket buffer or a congestion window. It is not the rule for a gradient fusion buffer, which is what “buffer” means everywhere else in this cluster (SOTA-048): that size is a trade between amortising per-collective overhead and preserving the overlap with the backward pass, and it has nothing to do with round-trip time.

Two different quantities have been collapsed into one sentence. Either reading is defensible on its own; together they are a recommendation that cannot be followed, because it does not say which buffer.

LIT-051 supports neither. Its contribution is the unified communication framework and the Summation Service split (SOTA-047).

What it needs

Splitting or retiring. If it means the fusion buffer, SOTA-048’s body already states the real trade and this adds nothing. If it means transport tuning, that is a claim about the network stack rather than about training, and needs a source that is about networks.

Backlinks