Mohan et al., Microsoft Research and UT Austin (2020) — ARXIV-2007.06775
Key takeaways
- The finding that motivates everything else: across nine models, three tasks and four datasets on production cluster hardware, DNN training time is in many cases dominated by data stall time — time the GPU spends waiting for items to be fetched from storage and preprocessed — not by the step itself. The input pipeline had been the unexamined half of training time.
- Knowing that requires measuring it, and measuring it requires a technique: DS-Analyzer establishes the stall differentially, by comparing against runs with the data path removed, rather than by reading a profiler total. It also supports predictive what-if analysis over memory, CPU threads, storage device and GPU generation.
- The mitigations land in a data-loading library, CoorDL, and are three simple things rather than a redesign — the paper’s point being that the bottleneck was unmeasured, not intractable.
- Against DALI, the state-of-the-art loader at the time, PyTorch with CoorDL cuts training time by as much as 5× on a single server.
Standing in the anthology
The source of the record’s data-pipeline practices: SOTA-042 (memory-map large datasets), SOTA-043 and above all SOTA-045, profile data loading separately from training, which is this paper’s central claim rather than a tip. SOTA-044 (pin memory for host-to-device transfers) was sourced here until the correction pass after #395 found that the paper never mentions pinned memory or host-to-device transfer; it now rests on a measurement elsewhere.
Filed alongside LIT-053, which attacks the same bottleneck from the storage format end. The two are the record’s answer to a failure mode that does not show up in a loss curve: a correct model, correctly optimized, waiting.