Field consensus — How far THE FIELD has converged on a practice, which is orthogonal to status — this record’s own editorial position. The useful cases are the corners: Deferred + converged is a practice everybody follows that we have not satisfied ourselves is deliberate, and Active + contested is this record being ahead of the field and saying so, which an anthology should be able to do (ADR-015).
Agreed — the field agrees and dissent is marginal, whether or not each adopter made the choice deliberately.
40 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-120 v2 | Prefer AdamW’s decoupled weight decay to L2 regularization added to the loss | Loshchilov et al. (2017), LIT-012 — ARXIV-1711.05101. | Active |
| SOTA-145 v4 | Estimate the RL baseline from a group of samples for the same prompt instead of training a critic | Shao et al. (2024), LIT-127 — Group Relative Policy Optimization: PPO with the value model dropped and the baseline taken from the scores of several outputs sampled for the same prompt, which removes a model-sized chunk of the RL memory footprint and is what every later reasoning recipe in this record actually runs. | Active |
| SOTA-150 v3 | Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound | Shazeer et al. (2017), LIT-188 — route each token to a few of many feed-forward experts instead of running one dense feed-forward for every token, so total parameters and per-token compute stop being the same number. Measured against dense by LIT-170 seven years later: DeepSeekMoE 16B matches LLaMA2 7B at roughly 40% of the compute, 145B approaches the same team’s dense 67B at 28.5%. | Active |
| SOTA-151 v4 | Extend a trained model’s context by rescaling RoPE, not by fine-tuning at the longer length | Chen et al. (2023) and Peng et al. (2023) — RoPE does not extrapolate, and fine-tuning at the longer length barely helps: more than 10000 batches moved LLaMA’s effective window from 2048 to 2560. Rescaling the position indices so they land back in the trained range reaches 32× that in under 1000 steps. YaRN rescales per wavelength rather than uniformly and gets to 128k with 10× fewer tokens. | Active |
| SOTA-174 | Train autoregressive models with fill-in-the-middle by default: it is a data transformation, and it is free | Bavarian et al. (2022), LIT-124 — cut a document into prefix, middle and suffix, move the middle to the end with sentinel tokens, and the model learns to infill. Transforming a large fraction of the training data does not harm left-to-right perplexity or sampling quality across a wide range of scales, so infilling is an added capability rather than a trade. | Active |
| SOTA-185 | Quantize weights after training by compensating each rounding error into the columns not yet quantized | Frantar et al. (2022), LIT-081 — ARXIV-2210.17323. Round one column at a time and push the resulting error into the remaining columns using approximate second-order information, instead of rounding every weight independently to the nearest level. | Active |
| SOTA-188 v5 | Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal | Active | |
| SOTA-192 v9 | Normalize the queries and keys before the attention dot product | Active | |
| SOTA-193 | Lower Adam’s second-moment decay when the loss spikes, before reaching for the learning rate | Proposed | |
| SOTA-195 v2 | Predict v rather than the noise when the model will be evaluated at low signal-to-noise | Active | |
| SOTA-197 v2 | Account for test-set proximity to the training data when evaluating, and state your contamination exposure | Active | |
| SOTA-199 | Regularize a narrow fine-tune against the pre-fine-tuning model’s own samples | Active | |
| SOTA-202 v2 | Clamp the prediction to the training range at every step when sampling from a model’s own output | Active | |
| SOTA-205 v2 | Replace a large coordinate network with a compact explicit structure and a small decoder | Four independent groups, three structures. Where a field is queried pointwise and the signal is spatially sparse, the capacity belongs in an addressable structure that training optimises directly, not in a network evaluated per point. | Active |
| SOTA-206 v3 | Keep a multi-step sampling option in a few-step generative model | Song et al. (2023), LIT-093. A one-step model that cannot spend more compute has no quality dial: whatever it produces is what you get. Consistency models trade compute for quality at inference without retraining, and that property is worth preserving by design. | Active |
| SOTA-207 | Use the deterministic sampler when the noise input has to mean something | Song et al. (2020), LIT-038. A stochastic sampler injects fresh noise at every step, so nothing about the starting point survives to the output. Setting the family’s stochasticity to zero makes the initial noise a latent code you can interpolate in and invert to. | Active |
| SOTA-227 v2 | Decode with a draft model and an accept-reject rule, which is exactly lossless | Autoregressive decoding runs one serial model pass per token and each pass is bounded by streaming the weights, so the arithmetic units idle. Have a cheap draft model guess the next gamma tokens, score all gamma + 1 positions in one target pass, and keep a prefix under an accept-reject rule built so the output distribution is the target’s exactly. 2x-3x at 11B and 2-2.5x at 70B, with no retraining, no architecture change and nothing traded away. | Active |
| SOTA-228 | Pick the inference partitioning from where the bottleneck is, and expect it to move between prefill and decode | Pope et al. (2022), LIT-110. Serving is not one workload: prefill parallelises over the prompt, decode is serial over the output, and the two are bounded by different things. Worse, the binding constraint moves — at small batch and short context it is loading the weights, and at large batch and long context it is loading the KV cache, which on a 500B model at batch 512 and 2048 tokens reaches 3 TB, three times the parameters. Choose the layout from a cost model of the regime you are in, and re-choose it when the regime changes. | Active |
| SOTA-230 | Quantize the frozen base to 4-bit and keep the adapters in 16-bit | Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314. Store the frozen base weights in 4-bit NormalFloat and dequantize to BFloat16 for every matrix multiply, training only 16-bit LoRA adapters. Fine-tuning a 65B model falls from >780GB to <48GB with no measured loss against a 16-bit fully fine-tuned baseline, because the arithmetic never happens in 4 bits. | Active |
| SOTA-237 | Match densely without a keypoint detector, because the detector fails where matching is hardest | Sun et al. (2021), LIT-387 — ARXIV-2104.00680. Establish dense matches coarse-to-fine instead of detecting keypoints, describing them and matching the descriptors. The detector sets the pipeline’s ceiling, and it fails in low-texture regions — walls, floors, road — where it cannot emit repeatable points, so no downstream matcher can recover a correspondence that was never proposed. | Active |
| SOTA-240 v4 | Apply dropout where the model can memorize what it is shown, and not where it cannot | Srivastava et al. (2014), LIT-395 — dropout has a “sweet spot” in dataset size and reports both of its edges: no gain at all on data small enough to memorize through the noise, and a declining gain once the data is large enough that overfitting is not the problem. It costs 2-3x training time and it is not a drop-in — the paper’s own recipe pairs it with n/p units, 10-100x the learning rate, momentum 0.95-0.99 and a max-norm constraint. Introduced by Hinton et al. (2012), LIT-394. | Active |
| SOTA-306 v2 | Factorize the codebook: look up in a low-dimensional normalized space, embed in a high-dimensional one, and report utilization alongside reconstruction quality | Yu et al. (2021), LIT-500 — project the encoder output down to a low-dimensional lookup space, ℓ₂-normalize it and the codebook, match there, and project the winner back up to a wide embedding. Codebook usage goes 4% → 95% and FID 3.68 → 1.50 at unchanged throughput; removing the normalization alone drops usage to 2%. Confirmed independently by LIT-497 and LIT-494, and in all three the curve turns back up, so there is an optimum to find rather than a direction to follow. | Active |
| SOTA-331 | Pass low-dimensional coordinate inputs through sinusoids of sampled frequencies before an MLP, and tune the frequency scale rather than the distribution | Tancik et al. (2020), LIT-550, with NeRF (LIT-435) — an MLP on raw coordinates never fits high frequencies. Map the input to [cos 2πBv, sin 2πBv] with B ~ N(0, σ²) and tune σ on held-out data: too small blurs, too large aliases, and the distribution’s shape does not matter. Gaussian features beat no mapping and log-spaced positional encoding on all seven image, shape, CT, MRI and view-synthesis tasks tested. | Active |
| SOTA-355 v2 | Isolate the emergent outlier feature dimensions into 16-bit and quantize the rest to int8 | Dettmers et al. (2022), ARXIV-2208.07339. Quantize the projections vector-wise to int8, but pull the handful of systematically large feature dimensions out into a separate 16-bit multiplication — 99.9% of values in 8-bit, no measured quality cost up to 175B. | Active |
| SOTA-356 | Protect the salient weight channels by scaling them before rounding, choosing them from the activations rather than the weights | Lin et al. (2023), ARXIV-2306.00978. Identify the ~1% of weight channels whose activations are largest, scale them up by an equivalent transformation before rounding, and quantize everything uniformly — rather than keeping the important channels in higher precision. | Active |
| SOTA-357 | Build a zero-shot classifier from prompt templates and ensemble them in embedding space | Active | |
| SOTA-358 v3 | Drop the domain inductive bias once pre-training data is large enough, and keep it when it is not | Active | |
| SOTA-359 v2 | Supervise vision from the caption, and match image to caption rather than predicting its words | Active | |
| SOTA-361 | Compose augmentations so the shortcut dies, and tune them for the contrastive objective rather than the supervised one | Active | |
| SOTA-364 | Break batch normalization’s cross-sample leak before it solves your contrastive task for you | Active | |
| SOTA-365 v2 | Learn without negatives by breaking the symmetry: a predictor on one branch and a stop-gradient on the other | Active | |
| SOTA-370 | Add many low-resolution views alongside the two full ones, because view count is what helps and resolution is what costs | Active | |
| SOTA-372 v2 | Run the encoder on visible tokens only and push the mask tokens into a small decoder you discard | Active | |
| SOTA-384 | Calibrate a trained classifier with a single temperature fitted on held-out data, not with a richer map | Guo et al. (2017), LIT-616 — rescale the logits by a single learned 1/T fitted for NLL on a held-out set. It cannot change the argmax, so accuracy is unchanged by construction, and it beats vector scaling, matrix scaling, histogram binning, isotonic regression and BBQ — including the two that strictly contain it. Typical uncalibrated ECE is 4–10%. | Active |
| SOTA-386 v4 | Show a video diffusion model images before and alongside video | Proposed | |
| SOTA-388 v2 | Train a plain ViT on ImageNet-1k with average pooling, fixed 2D sin-cos positions, a 1024 batch and light augmentation | Beyer et al. (2022), LIT-628 — five changes, none novel, take ViT-S/16 on ImageNet-1k from 66.8% to 76.5% at 90 epochs and 80.0% at 300. Global average pooling, fixed 2D sin-cos position embeddings, batch 1024, RandAugment at level 10 and Mixup at p = 0.2. 6h30 on a TPUv3-8 for the 90-epoch run. | Active |
| SOTA-389 v3 | Caption training video with a model that watches the video, not with captions of its frames | Proposed | |
| SOTA-390 v3 | In a video diffusion transformer, attend over space and time jointly rather than factorizing, and budget for the cost | Proposed | |
| SOTA-408 v3 | Average the weights along the tail of training under a cyclical or high constant learning rate, then re-estimate the normalization statistics | Izmailov et al. (2018), LIT-673. Hold the learning rate high — cyclical or constant — over the last stretch of training and keep a running average of the weights, then ship the average. From pretrained torchvision checkpoints, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at one model’s inference cost. With batch normalization, the running statistics must be recomputed with one pass over the data or the averaged model is broken rather than merely worse. | Proposed |
| SOTA-410 v2 | For guided sampling, use a second-order multistep solver on the data-prediction parameterization, not a higher-order solver on the noise prediction | Lu et al. (2022), LIT-676. SOTA-203’s higher-order solver inverts under guidance: at scale 8.0 and 10 evaluations, FID is 13.04 for first-order DDIM against 114.62 at order 2 and 164.74 at order 3. Sample instead with a second-order multistep solver parameterized on the data prediction — which is also what makes SOTA-202’s per-step clamping available at all, since you cannot clamp x̂₀ if the solver’s state is ε̂. 20 evaluations then beat DDIM at 250. | Active |