Source
Tay et al. (2021), LIT-052 — ARXIV-2109.10686.
Why this is rejected
The source contains the string “instabilit” zero times. The claim came from LIT-052’s fifth replaced takeaway, “training instability primarily occurs in first few thousand steps”, and the practice then converted “a few thousand” into ~5000 — a number with no owner at any point in the chain.
That is the sharpest thing here and the reason this one is worth reading past
its status. The record has a recurring failure of ownerless constants: a
threshold that arrives as a round number in a summary, gets a tilde and a
practice of its own, and is thereafter quotable. SOTA-013’s 2000 was traced
to a real source. This 5000 was traced to a rounding.
What replaces it
Nothing needs to. The record’s monitoring practices are already sourced and
already say more than this did: SOTA-069 watches exp(loss) and SOTA-099
tracks gradient-norm statistics. Neither is scoped to a step window, which is
the better shape — instabilities cluster early, but a loss spike at step 400k
is not less worth catching, and a practice that names a window invites someone
to stop looking after it.