Field consensus — How far THE FIELD has converged on a practice, which is orthogonal to status — this record’s own editorial position. The useful cases are the corners: Deferred + converged is a practice everybody follows that we have not satisfied ourselves is deliberate, and Active + contested is this record being ahead of the field and saying so, which an anthology should be able to do (ADR-015).
Assumed — not doing it is what needs justifying.
14 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-001 v2 | Use Adam as the default optimizer absent a reason to choose otherwise | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-032 v5 | Put the layer normalization inside the residual block, before the sublayer | Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. Pre-LN: normalize the input to each sublayer rather than the sum after it, so the gradients near the output are well behaved at initialization. | Active |
| SOTA-164 v2 | Deduplicate the pretraining corpus at both substring and document granularity before training on it | Lee et al. (2021), LIT-202 — a single 61-word sentence appears thousands of times in C4. Deduplicating cuts verbatim emission about tenfold, reaches equal or better accuracy in fewer steps, and removes train-test contamination from standard validation sets. Two granularities, because a repeated boilerplate paragraph is not a duplicated document. | Active |
| SOTA-182 v2 | Compute the normalization statistic without centering (RMSNorm) | Zhang and Sennrich (2019), LIT-023 — ARXIV-1910.07467. Drop the mean subtraction from layer normalization and rescale by the root mean square alone. | Active |
| SOTA-184 v2 | Adapt a pretrained model by training a low-rank update to each weight matrix, not the matrix itself | Hu et al. (2021), LIT-046 — ARXIV-2106.09685. Freeze the pretrained weights and learn a rank-r product BA beside each one, so the trainable parameter count and the optimizer state fall by orders of magnitude and the adapter folds back into the weight at inference. | Active |
| SOTA-187 v5 | Train the generative model in a learned compressed latent, not at full resolution | Active | |
| SOTA-203 v3 | Sample a diffusion model with a higher-order ODE solver on the weights you already trained | Lu et al. (2022), LIT-076 — the diffusion ODE is semi-linear, so solve the linear part exactly and approximate only the neural integral. 10-20 function evaluations, no retraining. Rests on LIT-038, which established that the sampler is not fixed by the training objective. | Active |
| SOTA-249 | Recompute activations from a sqrt(n) subset of checkpoints when activation memory is the binding constraint | Chen et al. (2016), LIT-004 — ARXIV-1604.06174. Store activations at O(sqrt(n)) checkpoints and recompute the rest during the backward pass. The price is one extra forward pass per minibatch; the measured case is a 1000-layer ResNet at 48G to 7G for 30% more wall clock. | Active |
| SOTA-279 v2 | Put worked reasoning steps in the few-shot exemplars when the task needs more than one step, and only once the model is large enough | Wei et al. (2022), LIT-467 — eight exemplars carrying worked steps more than doubled PaLM 540B on GSM8K, beating a fine-tuned model with a verifier. Three ablations rule out the equation, the extra tokens and knowledge activation. Below ~100B it does nothing or hurts. | Active |
| SOTA-281 | Try the single step-by-step instruction before writing exemplars; it recovers most of the gain for none of the labour | Kojima et al. (2022), LIT-469 — one fixed sentence with no exemplars takes MultiArith from 17.7% to 78.7% and GSM8K from 10.4% to 40.7%. It beats eight-shot standard prompting and loses to hand-written few-shot chains, which is the ordering that makes it a first move. | Active |
| SOTA-360 | Score a density ratio against sampled negatives instead of reconstructing the target | Active | |
| SOTA-362 | Train through a projection head and then discard it, taking the layer before as the representation | Active | |
| SOTA-381 v2 | Sample a handful of negatives instead of normalizing over the vocabulary, and let the count fall as the data grows | Active | |
| SOTA-424 v4 | Train one network for both conditional and unconditional scores by dropping the condition on 10% of examples, then pick the guidance weight by which metric you are willing to lose | Ho and Salimans (2022), LIT-693. Replace the conditioning with a null token on a fraction of training examples — 0.1 is the measured best, with 0.2 and 0.5 worse — so one network learns both scores, then extrapolate away from the unconditional estimate at sampling. The weight is not a quality dial: on ImageNet 64×64, w from 0.1 to 4.0 takes FID from 1.55 to 26.22 and IS from 66.11 to 260.2. Choose it by which of those you are prepared to sacrifice, and never compare two models at different weights. | Active |