Source
Brown et al. (2020), LIT-035 — ARXIV-2005.14165.
What the recipe actually is
Stripped to what a reader can act on, LIT-035’s pretraining setup is four choices, and their durability is the reason this is a practice at all:
- Decoder-only, not encoder-decoder and not a masked objective. One stack, causal attention, no task-specific heads.
- A single next-token objective on everything, with no auxiliary losses and no supervised task mixture during pretraining.
- A broad, filtered web corpus rather than a curated task collection — quality filtering and deduplication over scraped text, weighted toward higher-quality sources.
- A fixed context window, with documents packed and split to fill it.
Everything downstream in this record modifies one of those four rather than
replacing the frame: the objective (SOTA-157’s diffusion alternative), the packing (SOTA-152), the data proportions (SOTA-166), the context (staged extension), the corpus (SOTA-172, SOTA-173).
Why the title was the problem
“gpt training recipe” names an artefact and recommends nothing, so nothing could cite it, contest it, or check it — the same defect as SOTA-001’s old title, and both were promoted one-to-one from a note’s bullet.
Retitled at version 2. What the practice claims is unchanged; it can now be disagreed with, which is the point.
What has moved since
The frame holds and the recipe’s specifics have not. Pretraining is now followed by an alignment stage LIT-035 has no equivalent of, context windows are extended in stages rather than fixed, and the corpus is increasingly synthetic or heavily rewritten rather than filtered-scraped.
Read as “this is the frame everything else varies within”, the practice is Active and well-supported. Read as a recipe to follow in 2026, it is a historical baseline, and the record should not let the second reading pass for the first.