Source
Aizman et al. (2020), LIT-053 — ARXIV-2001.01858.
The other half of every overlap in this record
Prefetching is SOTA-047’s principle applied to the input pipeline: work that does not depend on the current step should be running during it. The next batch’s read, decode and collation are exactly that, so a loader that starts them while the GPU is busy turns a serial fetch-then-compute loop into two overlapping ones.
It is what makes a streaming format work at all (SOTA-077): sequential reads are only fast if somebody is reading ahead.
The recommendation is older than the source. The TensorFlow white paper, LIT-739, describes queues between graph stages and names this as their first use: input prefetched from disk while the previous batch is still being computed. It gives the mechanism and no measurement, so it is the origin and not evidence; the source below is cited for the pipeline the overlap runs in.
What LIT-053 supplies is the pipeline this overlap runs in, not a measurement of prefetching itself. Its design splits the input path into independently scalable stages — I/O, decoding, augmentation, training — and its benchmark metric is how fast a ResNet-50 loop consumes batches from the PyTorch DataLoader, run with five workers per GPU. On that metric, sharded data on 120 hard drives matched local-SSD training rates, and AIStore and HDFS scaled alike, which the authors attribute to the compute-side client being the bottleneck. It never varies or reports a prefetch depth, so the parameter below is not its result.
The parameter, and the cost that is easy to miss
Depth. One batch ahead hides one batch’s latency and no more, and a pipeline with occasional slow items — a large image, a shard boundary, a cold read — needs enough depth to absorb the worst case rather than the average. Beyond that, depth is memory: every prefetched batch is resident, and in a pinned loader (SOTA-044) it is pinned memory, which competes with the page cache the same pipeline depends on (SOTA-042).
The failure mode is the one SOTA-045 exists for. Prefetch depth cannot fix a pipeline whose steady-state throughput is below the GPU’s demand — it only hides variance. A loader that is genuinely too slow looks, from the outside, exactly like one that needs more depth, and the difference is visible only in the measurement.