Source
Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang.
Tensor fusion, and whose it is
A collective has a fixed per-call cost — launch, handshake, latency across the fabric — that a small tensor cannot amortise. A transformer has thousands of small parameters (norms, biases) alongside a few large ones, so reducing each as it arrives spends most of the step on overhead. Coalescing them into a buffer and reducing once is the fix.
It is Horovod’s Tensor Fusion, LIT-220, and PyTorch DDP reached the same technique independently as gradient bucketing, LIT-219. Horovod states the problem exactly: ring-allreduce “utilizes the network in an optimal way if the tensors are large enough, but does not work as efficiently or quickly if they are very small.” Fusing first gave up to 65% improvement on models with many layers over an unoptimized TCP network.
It also supplies the number the record was missing: Horovod’s default fusion buffer is 64 MB. That is a real, attributable default — and, like the loss scaling constants in SOTA-013, a tested default rather than a measured optimum. The citation this practice carried before, LIT-051, assumes fusion rather than introducing it.
The trade the title hides
Fusing more means waiting longer for the buffer to fill, which delays the reduction and eats into the overlap SOTA-047 depends on. The bucket size is therefore a dial between two failures — too small and the collectives are latency-bound, too large and the overlap collapses — and there is no setting that is right for every fabric and model.
That is a better statement of what SOTA-049 is reaching for than the bandwidth-delay product it currently names.