anthology-of-the-sota

LIT-102: ZeRO++: Extremely Efficient Collective Communication for Giant Model Training

2 min read

Properties4
StatusActive
Version3
Tagsdistributed-optimization, numerics-and-precision
Cited as “paper” byNOTE-029 — ZeRO++: Extremely Efficient Collective Communication for Giant Model Training

Graph View

Rajbhandari et al. (2023) — ARXIV-2306.10209

Key takeaways

  • Treats each of ZeRO’s three collectives as a separate problem: block quantization for the forward weight all-gather, a secondary redundant weight shard so the backward gather is local, and qgZ, an all-to-all quantized reduction replacing reduce-scatter
  • The reason qgZ exists is structural, not empirical. Reduce-scatter is a sequence of add-then-forward hops, so a quantized value is requantized and resummed at every hop and error compounds with hop count. An all-to-all exchanges data once, so each value is quantized once and reduced locally in higher precision — which is why INT4 gradients survive here and would not survive a compressed reduce-scatter
  • 4× less communication, up to 2.16× throughput at 384 GPUs
  • Deliberately re-introduces the redundancy ZeRO exists to remove: optimization 2 spends memory to buy bandwidth, on the premise that the bottleneck is communication — low-bandwidth clusters, or scale that forces the per-GPU batch small
  • The transferable rule: when compression fights a collective’s structure, change the collective, not the precision

Standing in the anthology

Read — NOTE-029. Nothing is sourced to this paper and the reading files no practice; the record carries neither ZeRO nor this regime.

The finding worth keeping is about the record’s low-precision practices.

SOTA-158, SOTA-159 and SOTA-160 all reason about bit width and range, and none reasons about how many times a value is quantized on its way through a distributed computation. This paper’s central engineering decision turns entirely on that count: one INT4 hop is fine, several are not, and the fix was to restructure the collective so there is only one.

Audited by #121; still unsourced, and one recommendation is already held. R4 — quantize block-wise rather than per-tensor — is SOTA-163, sourced to a paper that specifies the format rather than to this one. R1 is the live candidate: when compression fights a collective’s structure, change the collective rather than the precision. It is the most portable claim in the reading and the record holds no second instance of it, which is why it waits rather than being filed from one.

Backlinks