LIT-016: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism

Huang et al. (2018) — ARXIV-1811.06965

Key takeaways

• Micro-batch splitting is the mechanism. Split the mini-batch into M micro-batches and pipeline them across K partitions, then accumulate and take one synchronous step — so the optimization is unchanged and only the schedule differs
• Bubble overhead — idle time from partitioning — is O((K−1)/(M + K−1)), amortized over the micro-steps
• Negligible once M ≥ 4×K, the paper’s empirical threshold. Helped by backward re-computation being schedulable early, without waiting on gradients from earlier layers
• Peak activation memory with re-materialization and partitioning is O(N + (L/K) × (N/M)) — N/M the micro-batch size, L/K the layers per partition — against O(N × L) with neither
• Only activation tensors cross partition boundaries, so efficient scaling holds even without high-speed interconnects
• Scope: any model expressible as a sequence of layers. Demonstrated on a 557M AmoebaNet (84.4% top-1, ImageNet-2012) and a 6B multilingual NMT model
• The bubble analysis assumes evenly balanced partitions, stated explicitly

Standing in the anthology

The source of SOTA-017, SOTA-018 and SOTA-019.

Read in full for #114; the reading is NOTE-006. All three practices are confirmed, and all three gain a number where they had an adjective — M ≥ 4K, the bubble expression, and the memory trade in closed form.