Mohan, Phanishayee and Chidambaram (2021) — FAST ‘21, usenix.org/conference/fast21/presentation/mohan
Key takeaways
- Checkpointing had been epoch-granular and hand-tuned, which makes the interval a guess: too rare and a failure costs hours of recomputation, too frequent and the write dominates. CheckFreq makes the frequency something the system derives, at the granularity of iterations, from systematic online profiling.
- The interval is then tuned at runtime — adaptive rate tuning — so the checkpointing overhead stays inside a stated bound as conditions change, rather than being fixed once from an offline measurement.
- Correctness detail most implementations get wrong: resuming must preserve the training-data invariant that each item is seen exactly once per epoch, which means checkpointing the data loader’s state, not just the model and optimizer. CheckFreq does it with a lightweight resumable iterator.
- Two-phase checkpointing pipelines the snapshot against ongoing compute, so the write is largely hidden. Together: recovery time falls from hours to seconds with runtime overhead bounded within 3.5%.
Standing in the anthology
The source of SOTA-054 through SOTA-057 — checkpoint frequency as a tuned quantity, asynchronous writes, and a multi-level strategy. Read the
practices against the paper rather than the reverse on one point: SOTA-055 states an epoch-based rule of thumb for optimizer state, while CheckFreq’s argument is precisely that iteration granularity is available and epochs are the wrong unit. That tension is worth leaving visible until someone re-reads the practice.
By the same authors as LIT-050, and the same instinct applied one layer down: measure the thing everyone was estimating.