Practice statuses — What a status means for a PRACTICE: whether this record says to do the thing today. It is not the standing of the paper underneath — that is lit-statuses, and the two are allowed to disagree — and it is not what the field does, which is consensus (ADR-002).
Retired — no longer believed, with no successor — the attic.
19 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-021 | Monitor loss landscape smoothness during training | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Rejected — states no threshold and no response, and the smoothing it refers to is a mechanism from a paper the record had conflated with this one. Measuring landscape smoothness during training is not something the practice’s source does or recommends; the finding it rests on is Santurkar et al.’s explanation of why batch normalization works, which is a fact about the technique rather than an instruction |
| SOTA-027 v2 | Use a smaller learning rate for LayerNorm parameters | Xu et al. (2019), LIT-025 — ARXIV-1911.07013. | Rejected — The cited paper contains no discussion of learning rates and argues the parameters in question should be removed rather than tuned. The durable claim — treat norm parameters as their own group — is elsewhere in the record, and is about weight decay rather than rate |
| SOTA-041 v2 | lr tuning less important for larger models | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. | Rejected — States the opposite of its source. Kaplan reports that larger models require a smaller learning rate to prevent divergence, and carries an explicit size-dependent rule LR(N) because the right rate is not size-independent |
| SOTA-046 | Use hierarchical allreduce for tensors > 1MB | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — the ’> 1MB’ crossover has no owner: NCCL measures the algorithm choice per topology rather than publishing a constant, and LIT-051 does not discuss it. What survives is that a modern collectives library already chooses by size and topology, which is not a practice a reader acts on |
| SOTA-049 | Set buffer size to network bandwidth-delay product | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — collapses two different quantities into one unfollowable sentence — a TCP socket buffer sized by bandwidth-delay product, and a gradient fusion buffer sized by the overlap trade. The fusion-buffer half is SOTA-048, which now carries Horovod’s attributable 64 MB default; the transport half is a claim about networks with no source here |
| SOTA-055 | Save optimizer state every N epochs (N ~ sqrt(total_epochs)) | Mohan et al. (2021), LIT-059 — https://www.usenix.org/conference/fast21/presentation/mohan. | Rejected — the practice its own cited source exists to replace. LIT-059 names epoch-granular, hand-tuned checkpointing as the problem it solves, on both counts; SOTA-054 now states what that paper actually recommends |
| SOTA-064 v2 | Warmup needed scales sub-linearly with model size | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of warmup. | Rejected — The cited paper does not contain the word “warmup”. Its subject is model shape for downstream fine-tuning; this claim came from a takeaway that belonged to no identifiable source, and no paper in the record supports it |
| SOTA-065 v2 | Initialize layer norm weights closer to 1 for larger models | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of layer normalization, and the claim names no adjustable quantity. | Rejected — The cited paper contains no discussion of layer normalization. The claim is also doubtful on its own terms: LayerNorm’s gain initialises to exactly 1 by convention at every scale, so “closer to 1 for larger models” names no adjustable quantity |
| SOTA-066 v2 | Can use shorter warmup periods for wider models | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of warmup. | Rejected — The same gap as SOTA-064, restated as a width claim. The cited paper contains no discussion of warmup at all |
| SOTA-067 v2 | Monitor loss specifically during first ~5000 steps for instabilities | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of instability, and the 5000 belongs to nobody. | Rejected — The cited paper contains no discussion of training instability. The record’s sourced monitoring practices are SOTA-069 and SOTA-099, which do not depend on a step count |
| SOTA-073 | Monitor network utilization during training | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — states no signal, threshold or action, so there is nothing to do differently on reading it, and LIT-051 does not argue for it. The specific version worth having is in SOTA-048’s body: the gap between achieved and peak bandwidth during the overlap window tells you the fusion buffer is sized wrong |
| SOTA-074 | Adapt buffer sizes to network conditions | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — names a controller — signal, target, update rule — of which none is stated, and no source here describes one. A measured version would resemble what CheckFreq does for checkpoint intervals (SOTA-054) |
| SOTA-076 | Place replicas to minimize cross-rack traffic | Jiang et al. (2020), LIT-051 — https://www.usenix.org/conference/osdi20/presentation/jiang. | Rejected — a cluster-scheduling claim rather than a training one: it is the scheduler’s decision, expressed as a placement-group request, not something in a training run. ADR-026 (superseding ADR-020) scopes this record by the kind of claim, and this is a different kind |
| SOTA-101 v2 | Use perplexity-based filtering for quality assessment | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Rejected — The cited paper performs no filtering — it reweights sampling over domains — and filtering has the opposite failure mode to what it recommends. The string “filter” does not appear in it |
| SOTA-104 v2 | Monitor domain coverage during training | Albalak et al. (2023), LIT-117 — ARXIV-2312.02406. | Rejected — Names no signal and no response, and is in tension with its own source: under ODM the sampling distribution is supposed to move away from uniform coverage. The string “domain coverage” does not appear in the paper |
| SOTA-107 v3 | Keep sequence lengths multiple of 128 for best performance | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Rejected — The 128 is traced and belongs to a different quantity. Every occurrence in the source is a head dimension or a block size; none is a sequence length, and “multiple of” and “divisible” appear zero times |
| SOTA-108 v3 | Pad attention masks to block boundaries for better hardware utilization | Dao et al. (2023), LIT-106 — ARXIV-2307.08691. | Rejected — Not in the cited paper. “padding”, “padded”, “pad”, “block boundar” and “divisible” appear zero times each; the body’s reasoning about skipping fully-masked blocks is correct and is not the source’s |
| SOTA-112 v4 | Combine Monarch Mixer layers with standard attention rather than replacing it wholesale | Fu et al. (2023), LIT-115 — ARXIV-2310.12109. | Rejected — The source is attention-free and argues for replacement, not combination. “hybrid” occurs zero times in it; the hybrid argument belongs to the attention/SSM literature elsewhere in this record |
| SOTA-380 | Use prediction-based word vectors rather than count-based distributional vectors | Baroni, Dinu and Kruszewski (2014), LIT-608 — across 14 benchmarks, word2vec CBOW beat PPMI, SVD and NMF count vectors almost everywhere, and the authors recommend “anybody interested in using DSMs” to “go for the predict models”. Retired: LIT-607 traces the gap to settings the count models were not given, and finds no consistent winner once they are. | Rejected — the advantage it reports disappears when the count-based baselines get the design choices word2vec shipped with and every method is tuned alike (LIT-607): no family wins consistently. Retired with no successor. What replaces it is an evaluation practice, SOTA-379, which corrects it |