Aizman, Maltby and Breuel, NVIDIA (2020) — ARXIV-2001.01858
Key takeaways
- The premise is a mismatch, not a performance bug: distributed filesystems were designed for access patterns and usability requirements that deep learning jobs do not have. Petascale training reads enormous numbers of small objects in shuffled order, which is close to the worst case for systems tuned around directory trees and random access to files.
- WebDataset is the format answer — training data as sharded tar archives, a standards-based container needing no special reader, so the job’s access pattern becomes sequential streaming of large files. Shuffling happens within and across shards rather than by seeking.
- AIStore is the storage answer: a scalable, easy-to-deploy backend for the same workload.
- Evaluated on image classification across local SSDs, single-node NFS, and two identical bare-metal clusters running HDFS and AIStore — the point of the comparison being that the format’s benefit is not tied to one backend.
Standing in the anthology
The source of SOTA-077 (tar archives for dataset storage) and of the buffering, prefetch and worker-count practices that follow from streaming shards (SOTA-078, SOTA-079, SOTA-080). Those numbers — a shuffle buffer of two to three batches, four workers per GPU — are operational rules of thumb rather than results this paper reports, and they should be read that way; what the paper establishes is the format and why sequential access is the thing worth buying.
The companion to LIT-050: that paper measures the stall, this one changes the storage layout that causes it.