Ivanov et al., ETH Zurich (2020) — ARXIV-2007.00072
Key takeaways
- The claim that reframes the problem: despite the attention transformer optimization has received, existing implementations do not efficiently use GPUs, and the reason is that training has become memory-bound. Amdahl’s Law plus enormous gains in compute throughput moved the bottleneck to data movement while the mental model stayed on FLOPs.
- Existing frameworks use suboptimal data layouts — a claim about defaults, not about a missing feature, which is why the fix is available to anyone willing to measure rather than requiring new hardware.
- The method is a recipe for globally optimizing data movement rather than a point optimization: classify every operator by whether it is compute- or memory-bound, then fuse and re-lay-out accordingly.
- Data movement reduced by up to 22.91%, for a 1.30× speedup over state-of-the-art frameworks on a BERT encoder layer and 1.19× on the whole of BERT. The authors present the approach as general to deep networks.
Standing in the anthology
The source of SOTA-088 (fuse small operations into larger kernels), SOTA-089 (align tensor dimensions to hardware boundaries), SOTA-090 (hardware-specific memory layouts) and SOTA-091 (profile and optimize memory access patterns).
It is the empirical companion to LIT-063: TVM builds the machinery for fusion and layout search, this measures how much of a real workload’s time that machinery is addressing. It is also the argument FlashAttention (LIT-074) takes to its conclusion two years later — the same observation, memory movement rather than arithmetic, applied to one operator with an IO-aware algorithm instead of a compiler pass.