Source
Tay et al. (2021), LIT-052 — ARXIV-2109.10686.
Why this is rejected
The paper at that identifier is Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers. Its subject is model shape — depth against width — and its finding is the DeepNarrow strategy. Searched in full, it contains the string “warmup” zero times.
The claim came from LIT-052’s first key takeaway, “as model size increases, required warmup period becomes shorter relative to total training time”, which was replaced at that note’s version 2 along with five others. Whatever those bullets were drawn from, it was not this paper.
What is left of the claim
Something in this neighbourhood may well be true, and the record should not pretend otherwise: warmup length is usually set as a fraction of the run or as a fixed step count, and both conventions imply something about how it should move with scale. But the anthology’s rule is that a recommendation names the paper that established it (ADR-001), and no paper in the record establishes this one. Restating it would need a source found first, not a source assumed.
Retired alongside SOTA-065, SOTA-066, SOTA-067 and SOTA-068. All five came from the same six bullets, which is why this is one finding about a note rather than five findings about practices.