Source
Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan.
What this is instead of
Checkpointing had been epoch-granular and hand-tuned, which makes the interval a guess in both directions: too rare and a failure costs hours of recomputation, too frequent and the write dominates. LIT-059’s contribution is to stop choosing it. CheckFreq profiles the run online, works at iteration granularity rather than epoch, and tunes the interval at runtime so that checkpointing overhead stays inside a stated bound as conditions change — within 3.5%, while recovery falls from hours to seconds.
The title this practice carried until now — “checkpoint frequency should increase with training time” — was a heuristic standing in for that mechanism, and it was not what the cited paper says. There is a defensible intuition behind it, since the value at risk grows as a run proceeds, but the paper’s answer is to let the profiler and the overhead bound settle the interval
rather than any schedule. Restated to match its source; SOTA-055, which proposed a closed-form epoch interval, is retired for the same reason.
Conditions and cost
The mechanism needs a loader that can measure its own write cost and change the interval, which is a property of the training framework rather than of the model. Without one, the fallback is a fixed interval chosen from a measured write time and an accepted overhead — the same calculation, done once by hand.
It also depends on SOTA-056: the overhead bound is only reachable because the persist is pipelined against compute, so a system checkpointing synchronously cannot hit it at any interval worth having.
What the paper adds that this record still has no practice for: resuming must restore the data loader’s state, or an epoch stops seeing each item exactly once. LIT-059 says most implementations get that wrong.
The other half of the trade
This practice settles the interval from the cost side: how much checkpointing overhead the run will tolerate. SOTA-189 settles it from the risk side: expected work lost to a failure is half the interval, and a job’s mean time to failure falls linearly with its GPU count (3.66 days at 1,024 GPUs on one fleet’s measured rate), so an interval right for a small job is too long once it is resized. Neither source combines the two. Take the write cost from profiling as here, and the failure rate from the fleet as there.