Source
Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311.
Known implementations
- PaLM
The response half of the monitoring cluster
SOTA-069, SOTA-070, SOTA-098 and SOTA-099 all detect. This is what to do, and it is the only practice in either stability cluster that says so: on a loss spike, restore the last checkpoint from before it and skip forward past the batches that were being consumed when it happened.
The reasoning is that a spike is usually caused by specific data — a pathological document, a bad shard — interacting with the current parameters, rather than by a general instability. Rewinding alone would replay the same batches into the same state and reproduce it; skipping alone would leave the damage already done to the weights and the optimizer moments in place. Both together are what makes the recovery reliable, and PaLM reports using it.
The procedure is LIT-069’s, reported as what worked rather than as a studied method. The 540B model spiked roughly 20 times despite gradient clipping, at irregular intervals and sometimes late, and never in the smaller models; the authors restarted from a checkpoint about 100 steps before each spike and skipped roughly 200–500 batches, after which the loss did not spike again at that point. Their ablation qualifies the reading above: the same batches replayed from a different, earlier checkpoint did not spike, so they put it down to particular data meeting a particular parameter state, not to bad data as such.
What it depends on
Three things the record now says elsewhere, and the practice is worth reading as the reason they matter. Checkpoints frequent enough that “the last one” is close (SOTA-054). Detection early enough to know which batches to skip — the NaN case is SOTA-072, and by the time it reaches the loss the moments are already poisoned. And a data loader whose position can be restored and advanced, which LIT-059 says most implementations get wrong and for which this record still has no practice.
Its cost is honest and rarely stated: the skipped data is not seen, and nobody checks afterwards whether what was skipped mattered.