Source
Afroz et al. (2025), LIT-737 — ARXIV-2511.14124.
LIT-737 measured the difference this practice rests on. On an NVIDIA L40S over PCIe 4.0 ×16, a 16 MB FP32 copy from host to device took 1.65 ms from pageable memory and 0.68 ms from pinned memory — 10.16 against 24.74 GB/s — and the 8 MB FP16 and device-to-host rows show the same better-than-twofold gap. It was measured to motivate a tensor-offloading system, not an input pipeline, and it reports copy bandwidth rather than overlap with compute; the copy is the same copy, but the overlap argument below is the CUDA programming model’s, not a result anyone in the record measured.
That paper is not where the advice comes from, and nor was the data-stall
study this practice used to cite, which never discusses pinning. The
recommendation is NVIDIA’s own CUDA documentation, and frameworks made it a
loader option long before anyone in this record measured it; no paper can be
named as its origin, so introduced_by is left empty.
What pinning actually buys
Pageable host memory cannot be the source of an asynchronous DMA transfer: the driver has to stage it through an internal pinned buffer first, so the copy is synchronous from the program’s point of view and cannot overlap with compute. Allocating the staging buffer as page-locked from the start removes the extra copy and lets the transfer run on a separate stream while the GPU is still working on the previous batch.
For an input pipeline that is already close to keeping up, that overlap is the difference between a small stall per step and none.
The cost is real and is a system-wide one
Pinned pages cannot be swapped or moved. Pinning aggressively — many worker processes each with a large prefetch depth — takes memory away from the page cache that the same pipeline depends on for repeat reads (SOTA-042), and on a shared machine it takes it away from other jobs. The failure is not an error; it is the loader getting slower for reasons that do not appear in its own metrics.
So this is a default worth having and not a knob worth maximising, and whether it is helping at all is a question for SOTA-045’s measurement rather than for argument.