Yang et al. (2019) — ARXIV-1910.05124
Key takeaways
- Runs pipeline stages asynchronously on stale weights rather than paying GPipe’s bubble or PipeDream’s per-microbatch weight stash — staleness is corrected statistically instead of eliminated
- Two techniques: T1, learning-rate rescheduling (scale the step by the stage’s known delay) and T2, discrepancy correction (approximate the weights the backward pass should have seen). The delay is deterministic — a function of stage index — which is what makes a correction definable
- 4.6× the pipeline utilization of GPipe and up to 2× less weight and optimizer memory than PipeDream, at a quality PipeDream does not reach; largest gap to synchronous training is 0.1% top-1 on ImageNet
- T1 alone gives 7.6× utilization over GPipe at 95.0% accuracy / 34.1 BLEU (synchronous: 95.0% / 34.5). T2 looks optional until the pipeline is deep — on a 150-stage ResNet-152 it is required to prevent divergence
- Activation memory scales quadratically in stage count while throughput scales linearly, so recompute is done across a segment of stages rather than within one
Standing in the anthology
Read — NOTE-050.
Nothing is sourced to this paper and the reading files no practice — the record’s pipeline practices are GPipe-shaped and synchronous, which is what frontier training runs.
What the reading changes is how SOTA-017 should be read. The record presents
the bubble as a cost to be minimised, with M ≥ 4K as the answer. This paper
is the standing demonstration that it can instead be declined, and that the
price of declining it is two corrections — one of which only starts to matter
once the pipeline is deep, which is the ablation’s most instructive cell. No
practice changes; the neighbourhood gains a second option.