Practice statuses — What a status means for a PRACTICE: whether this record says to do the thing today. It is not the standing of the paper underneath — that is lit-statuses, and the two are allowed to disagree — and it is not what the field does, which is consensus (ADR-002).
Replaced — something newer does this better, and the record says what.
13 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-010 v2 | skip connections promote training stability by smoothing out the loss landscape | Li et al. (2017), LIT-014. Moved to THEORY-011 — the claim is a statement about why deep networks became trainable, not an instruction, and belongs in the scheme ADR-031 created for exactly that. This code stays so links into it resolve. | Superseded — by THEORY-011; Not a recommendation, and never was: it states what is true rather than what to do, which is a THEORY under ADR-031 and was one of the eight that decision named and declined to move. The claim is unchanged and is believed — it now lives at THEORY-011, with the figure’s limits and the five practices it underwrites. Superseded rather than Rejected because Rejected on a practice means do not do this, and nobody should read this code as advice against skip connections. ADR-034 is the rule. |
| SOTA-022 | Place BN after linear/conv layers but before activation functions | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Superseded — by SOTA-004; a duplicate of SOTA-004, which states the same placement rule from batch normalization’s own paper. The two arrived in the same import from different notes and have coexisted since; SOTA-004 is the one with the originating source |
| SOTA-023 v2 | Use multi-query attention for decoder-only models to reduce memory bandwidth | Shazeer (2019), LIT-024 — ARXIV-1911.02150. | Superseded — by SOTA-109 |
| SOTA-024 v2 | Keep key/value projections shared across heads while query projections remain separate | Shazeer (2019), LIT-024 — ARXIV-1911.02150. | Superseded — by SOTA-109 |
| SOTA-039 | single cycle of cosine decay is sufficient lr schedule | Brown et al. (2020), LIT-035 — ARXIV-2005.14165. | Superseded — by SOTA-140; warmup-stable-decay matches it and leaves the token budget open; every report since 2024 in the record uses a stable-then-decay schedule |
| SOTA-062 v3 | Scale batch size with model size but sub-linearly | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. Along the compute-optimal frontier the model grows as C^0.73 and the batch as C^0.24: over a billion-fold increase in compute, more than a million-fold in model size against about a hundred-fold in batch. | Superseded — by SOTA-258; The decoupling experiment this practice’s own body invited has been run: holding the token budget fixed, the dependence of batch size on model size very nearly vanishes. What the heuristic was tracking is the data that scaled alongside the model. Superseded rather than Rejected because following it along the Chinchilla line gives roughly the right answer for the wrong reason — the successor is SOTA-258. |
| SOTA-068 v2 | Use gradient clipping during early training phase | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Superseded by SOTA-035, which cites Pascanu et al. and drops the “early training” qualifier. | Superseded — by SOTA-035; SOTA-035 is the same recommendation with a real source and without the false qualifier — clipping is applied throughout training, not only early |
| SOTA-097 v4 | Optimal batch size scales approximately with compute budget - B ∝ C^(1/4) | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. Equation 1.7 gives B ∝ C^(α_C/α_B) = C^0.24, from α_C ≈ 0.050 and α_B ≈ 0.21. | Superseded — by SOTA-258; The exponent is a fit to a projection. Along the Chinchilla line N and D scale together, so C = 6ND moves with D and a power law in either describes the data; two independent groups measuring off that line find optimal and critical batch size scale in the token budget, not in compute. Superseded rather than Rejected because the fit is not an error and the quantity is real — the successor is SOTA-258, and ADR-017’s corrective succession is the shape. |
| SOTA-102 v2 | Implement dynamic temperature scaling for mixing | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Superseded — by SOTA-103; Names a real component of ODM’s policy imprecisely rather than something the paper does not do — the two were one practice split in the import, and SOTA-103 now describes the whole method |
| SOTA-137 | Widen the residual stream into several streams with freely learned mixing (hyper-connections) | Zhu et al. (2024), LIT-141 — n parallel residual streams with learnable mixing; gains on dense and MoE pretraining, and an identity-mapping property lost that its successor restores. | Superseded — by SOTA-136; constraining the mixing to a doubly-stochastic manifold keeps the width and restores the identity mapping this gave up |
| SOTA-208 | Factorize long-sequence attention into a local window and a second head that escapes it | Child et al. (2019), LIT-225 — ARXIV-1904.10509. Superseded: the field went to exact attention made fast (SOTA-086, SOTA-087) and then to sparsity that is LEARNED rather than fixed (SOTA-138). What survives is the connectivity requirement — a window alone is not the design — which every hybrid the record recommends is an instance of. | Superseded — by SOTA-138 |
| SOTA-382 v3 | Subsample frequent tokens: it is faster and it improves the rare ones | Superseded — by SOTA-374; the same recommendation from the same paper and section; SOTA-374 states the discard rule and carries the independent ablation | |
| SOTA-432 | Maintain two power-function weight averages during training and snapshot them, so the averaging length is a post-hoc sweep instead of a guess you have to make before the run | Karras et al. (2023), LIT-720. Keep two power-function averages of the weights during training (σ_rel 0.05 and 0.10) and store both in each periodic snapshot; afterwards, reconstruct the average for any averaging length by a least-squares fit over the stored profiles, with error falling as O(1/n⁴) in the snapshot count. The averaging length stops being a pre-training commitment. Worth it because that commitment is expensive in a way that does not show up in its own column: with the length pinned at 13%, the learning-rate decay moves FID by up to 72% across t_ref ∈ [30k, 160k]; swept post hoc, the same bracket is within 10% of the optimum. | Superseded — by SOTA-428; A duplicate of SOTA-428, which was filed from LIT-714 hours earlier and is the same recommendation from the same paper. SOTA-428 is also the better document: it carries the architecture, learning-rate, guidance and metric dependences, where this one had only the first two. The Active/unreplicated pairing this note argued for is a judgement SOTA-428 declined, at Proposed, and that is the one that stands. |