Field consensus — How far THE FIELD has converged on a practice, which is orthogonal to status — this record’s own editorial position. The useful cases are the corners: Deferred + converged is a practice everybody follows that we have not satisfied ourselves is deliberate, and Active + contested is this record being ahead of the field and saying so, which an anthology should be able to do (ADR-015).
Spreading — several independent groups, moving toward default without being there.
61 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-132 v3 | Interleave linear-attention layers with global attention at about 3:1 instead of using full attention throughout | Kimi Team (2025), LIT-133 — three Kimi Delta Attention layers per gated-MLA layer beat full MLA at 48B/1.4T while cutting KV cache 75%; the layout LIT-131 ships at 2.8T with 69 KDA and 24 MLA layers. | Active |
| SOTA-133 | Replace fixed residual accumulation with learned attention over preceding layers | Kimi Team (2026), LIT-134 — Attention Residuals: a per-layer pseudo-query chooses which earlier layers to read, at O(d) parameters per layer; 1.25× compute advantage on scaling laws at 48B/1.4T, adopted in Kimi K3. | Active |
| SOTA-144 v2 | Extend µP’s transfer to depth with CompleteP so one sweep serves deeper models too | Dey et al. (2025), LIT-150 — the depth exponent α = 1 transfers the optimal learning rate across depth and keeps deep layers learning; 11.8% fewer FLOPs than µP at optimal shapes, 34.4% at 179 layers. One group so far. | Proposed |
| SOTA-146 | Correct the GRPO objective before running it — the published form has three independently identified defects | Liu et al. (2025), LIT-167; Yu et al. (2025), LIT-168; Zheng et al. (2025), LIT-180 — three groups, three defects in GRPO’s objective, three published fixes, none of them the same fix. “We ran GRPO” does not say which of these you ran. | Proposed |
| SOTA-147 v2 | Compress the KV cache into one shared latent vector instead of sharing key and value heads | DeepSeek-AI (2024), LIT-174 — Multi-head Latent Attention projects keys and values into a single low-rank latent and caches that, cutting the KV cache 93.3% and raising maximum generation throughput 5.76× against the same team’s dense 67B. Every DeepSeek model since is built on it. | Active |
| SOTA-148 v3 | Balance mixture-of-experts load with a bias on the routing scores, not an auxiliary loss | Wang et al. (2024), LIT-171 — add a per-expert bias to the routing scores before the top-K decision and update it from that expert’s recent load, so balancing changes which experts are chosen without adding a gradient to the loss. Better balance and better quality than an auxiliary-loss control, and what DeepSeek-V3 runs at 671B. | Proposed |
| SOTA-149 | Build the sparse layers from many small experts plus an always-on shared one, not a few large ones | Dai et al. (2024), LIT-170 — split into mN smaller experts and activate mK of them so the router chooses from a far larger combination space at the same compute, and isolate a few always-on shared experts to hold the common knowledge every routed expert would otherwise learn separately. | Active |
| SOTA-153 v4 | Drop positional encoding from the global-attention layers of a hybrid and let the cheap local layers carry position | Yang et al. (2025), Puvvada et al. (2025) and the Kimi Team (2025) — in a model that already interleaves full attention with a cheap local mixer, the full-attention layers do not need a positional encoding: the local layers carry position and recency, and the global layers do retrieval better without one. Three groups, three different local mechanisms, the same one-global-per-three-local layout. The payoff is that extending the context needs no RoPE rescaling, because there is no positional parameter left to rescale. | Active |
| SOTA-158 | Bound the activation’s output range when training in low precision | Jiang et al. (2026), LIT-200 — SwiGLU approximates x² for large positive inputs, which is where its expressive capacity comes from and what enlarges the output range and exacerbates outliers. Two groups independently concluded that is intolerable in low precision. What they agree on is the constraint; the functions they chose differ. | Proposed |
| SOTA-162 v2 | Train with auxiliary multi-token-prediction heads alongside next-token prediction | Gloeckle et al. (2024), LIT-163 — predict the next n tokens through n independent heads on a shared trunk, as an auxiliary task rather than a replacement. No training-time overhead, the benefit grows with model size and survives multi-epoch training, and the extra heads are a draft model you already trained. | Active |
| SOTA-163 v2 | Quantize with block-scaled microscaling formats rather than one scale per tensor | Rouhani et al. (2023), LIT-197 — an MX block is 32 elements sharing one E8M0 scale, so applying the scale is an exponent adjustment rather than a multiply. Shrinking the scope of the scale contains outliers locally instead of letting a few large values cost every small one its precision. 6-bit MX trains large transformers to FP32 accuracy with no recipe change. | Active |
| SOTA-165 | Precondition the gradient with matrices rather than entrywise scaling | Wen et al. (2025), LIT-156 — under per-optimizer tuning across ten optimizers and four scales, every fastest one multiplies gradients by matrices rather than scaling entrywise. A structural finding that survives a fair comparison, and the class the record’s Muon practice is one member of. The advantage is 1.4× at 0.1B and 1.1× at 1.2B. | Active |
| SOTA-169 | Widen the residual stream into several streams and constrain the mixing between them | Zhu et al. (2024) and three successors, LIT-141 / LIT-140 / LIT-151 / LIT-181 — replace the single residual stream with n parallel streams and learn the mixing. All four agree the width helps and that the mixing needs a constraint; they disagree about which one, and that disagreement is the two practices extending this rather than this itself. | Proposed |
| SOTA-180 | Keep the routed experts’ weights in a down-projected latent space and spend the saved bandwidth on more of them | Elango et al. (2026), LIT-196 — down-project the token before the routed experts and keep their weights in that latent space, dispatching and aggregating there too, while routing and the shared experts stay at full width. Communication volume and weight-loading bandwidth both fall by d/d_l, and the saved budget buys more experts and more active per token. | Active |
| SOTA-181 | Shard the sequence across devices in a ring and overlap the key-value exchange with the attention it feeds | Liu et al. (2023), LIT-206 — compute attention and the feedforward blockwise, distribute the blocks across devices in a ring, and pass each key/value block to the neighbour while computing on the one you hold. The communication is fully overlapped with the computation, so sequences scale with device count — exactly, with no approximation and no added overhead. | Active |
| SOTA-183 | Generate the harmlessness preference labels with the model itself, against a written set of principles | Bai et al. (2022), LIT-082 — ARXIV-2212.08073. Replace the human harmlessness comparisons with model-generated ones: the model critiques and revises its own responses against an explicit list of principles, and the preference model trains on that. | Active |
| SOTA-186 | Bootstrap a large annotation set with the model you are training, staging the automation as it improves | Active | |
| SOTA-194 | When an objective arrives with its own dataset, state the composition before crediting the objective | Active | |
| SOTA-196 v2 | Report zero-shot and in-distribution performance separately; one hyperparameter can move them in opposite directions | Active | |
| SOTA-198 v2 | Measure the gradient noise scale instead of sweeping batch size, and expect it to grow during the run | Active | |
| SOTA-200 v3 | Check whether an emergent capability is a metric artefact before believing it | Active | |
| SOTA-201 | Give each pretrained component its own learning rate when fine-tuning a composite model | Active | |
| SOTA-210 v3 | Report pass@k as well as pass@1 after post-training: reinforcement learning raises one and lowers the other | Ba et al. and Hayes et al. (2026), LIT-230 and LIT-234. GRPO finishes below its own base model on pass@16 and pass@32 in 15 of 18 comparisons while improving pass@1; across Qwen2.5, Qwen3 and published RL checkpoints up to 32B the base model overtakes the RL checkpoint at large k. A post-training result reported at pass@1 alone cannot distinguish a model that learned something from one that stopped trying anything else. | Active |
| SOTA-211 | Spend one fitness evaluation per evolution-strategies direction, not an antithetic pair | Kaya and Hashemi (2026), LIT-232, and Ba et al. (2026), LIT-230 — the antithetic pair’s variance reduction depends on both evaluations sharing randomness, which autoregressive regeneration breaks. Replace the pair with a leave-one-out baseline computed from the population mean the score centering already needs: same expected update, one evaluation per direction, twice as many directions per budget, and half the estimator MSE in transformer blocks. | Proposed |
| SOTA-213 | Control evolution-strategies drift with a larger population or an anchor penalty, not by stopping training early | Schweighofer et al. and Liang et al. (2026), LIT-238 and LIT-236, on the scaling Hoy et al. derived in LIT-235 — ES drift is a random walk whose size falls with population size, so raising the population from 30 to 128 halves the update norm and monotonically reduces prior-task degradation. Anchored Weight Decay buys the same reduction at population 30 for 1-2% runtime. Do not stop early instead: the prior-task dip is often transient and recovers by the end of training. | Proposed |
| SOTA-214 | Pair any gradient compressor with error feedback, and correct the momentum it is applied under | Compression is lossy and the loss accumulates. Every scheme in this record that survives high compression ratios keeps the residual locally and adds it back to the next step, and every report of a compressor “not converging” that the record holds is a report of that step being skipped: rank-4 PowerSGD fails on Cifar10 without it, and gradient sparsification at 99.9% degrades badly without the momentum-buffer version of it. | Active |
| SOTA-217 v4 | Align the hidden-unit permutation before averaging weights from separately trained networks | A neural network’s hidden units can be permuted without changing the function, so two networks trained from different seeds sit in different corners of the same symmetry orbit. Averaging their weights directly averages across that mismatch and traverses a loss barrier that is mostly an artefact of labelling. Match the units first — by weight matching, which needs no data and takes seconds to minutes — and for wide enough networks the barrier is gone. At standard width, and on ImageNet, it shrinks but does not vanish. | Proposed |
| SOTA-219 | Leave the inner optimizer state unsynchronised in local-update training | When workers train independently for hundreds of steps and synchronise by exchanging deltas, the Adam moments each worker accumulated are local state and should stay local. Synchronising them triples the communication for negligible quality gain — which means most of what the scheme saves is not saved by communicating rarely, but by communicating less each time. | Active |
| SOTA-236 | Predict scene geometry directly instead of solving for cameras first and triangulating | Wang et al. (2023), LIT-385 — ARXIV-2312.14132; Wang et al. (2025), LIT-384 — ARXIV-2503.11651. Given uncalibrated, unposed images, regress the 3D structure and let camera parameters and pixel matches fall out of it, rather than estimating calibration and pose first so that triangulation becomes possible. The quantities the classical pipeline needs as inputs are by-products of this one. | Active |
| SOTA-246 | Find mislabelled training data by self-influence, not by training loss | Koh and Liang (2017), LIT-403, with LIT-400 — rank training points by their influence on their own loss and inspect from the top. Mislabelled examples are strong proponents of themselves, so they sort to the front; high training loss does not separate them nearly as well. | Proposed |
| SOTA-258 | Scale batch size with the token budget, not with compute or model size | Zhang et al. (2024), LIT-445, and Bergsma et al. (2025), LIT-443 — critical batch size scales as D^0.46 and D^0.47 respectively, and is nearly flat in model size once the token budget is held fixed. Optimal batch size likewise goes as D^0.38. The compute exponent the field has used since Kaplan fits a projection: along the Chinchilla line C and D move together and only D is doing the work. | Active |
| SOTA-262 v2 | Give each modality its own weights and let the streams attend jointly | Esser et al. (2024), LIT-449 — MMDiT gives the text and image streams separate projections and MLPs, then runs attention over the concatenated sequence so information flows both ways. It beats UViT and DiT at matched budget. Liang et al. (2024), LIT-483 — the same shape reached independently for autoregressive multi-modal LLMs, matching a dense baseline at 55.8% of the FLOPs, and ablated component by component: the feed-forward carries most of the benefit, the attention projections less, the layer norms none. | Active |
| SOTA-263 v4 | Shift the timestep schedule when the resolution changes, because more pixels need more noise | Hoogeboom et al. (2023), LIT-660, and Chen (2023), LIT-659. A timestep is not a fixed amount of corruption. Destroying the signal in an image with more pixels takes more noise, so a schedule set at one resolution under-corrupts at a higher one. Shifting log-SNR down with resolution takes ImageNet 256 FID from 7.65 to 3.76 in pixel space. The direction is measured. The magnitude is tuned in every source. | Active |
| SOTA-265 v3 | Tune the stochastic sampler’s diffusion coefficient after training; it is not fixed by the forward process | Albergo, Boffi and Vanden-Eijnden (2023), LIT-645, state it and Ma et al. (2024), LIT-447, measure it — score-based diffusion conventionally takes the reverse SDE’s diffusion coefficient from the forward process, and presents the two as intrinsically tied. They are not: the coefficient affects neither the velocity nor the score, only the integration. So it is a sampler hyperparameter, tunable on a frozen model, and tuning it tightens the KL divergence to the target. | Active |
| SOTA-266 v5 | Connect data and noise on a straight line, and sample the training timesteps from a logit-normal rather than uniformly | Esser et al. (2024), LIT-449, and Ma et al. (2024), LIT-447 — the straight-line path between data and noise beats the curved variance-preserving one at fixed architecture and compute, and the advantage is largest at few sampling steps. Movie Gen (LIT-626) confirms it for video at 5B. In SD3’s latent text-to-image sweep, the timestep distribution decides the ranking: uniform does not beat tuned ε-prediction and logit-normal does. | Active |
| SOTA-269 | Keep a knowledge base outside the weights; parameters are an expensive and lossy place to memorise facts | Lu et al. (2024), LIT-452 — fact capacity is linear in model size and saturates in epochs, and the extrapolation is damning: all of Wikidata would want about 1000B non-embedding parameters trained for 100 epochs. Derivable facts cost full price unless they happen to share direction and structure. Borgeaud et al., LIT-060, is the record’s evidence that the alternative works at scale. | Active |
| SOTA-270 | Do not read a smooth loss curve as evidence of smooth training; decompose it when the answer matters | Kangaslahti et al. (2025), LIT-455, with Cohen et al. LIT-453 and Kunin et al. LIT-454 — the curve time-averages oscillation, sums over differently-timed abrupt transitions, and reads flat while the weights still travel. Smoothness is what many breakthroughs look like added up, so it is not evidence about the training. | Active |
| SOTA-278 | Before reporting that a model cannot do something, rule out the evaluation | Hu and Frank (2024), LIT-465 — the same capacity asked two ways scores differently, and the gap widens as the model weakens. Lawsen (2025), LIT-463, is the worked example: an output cap, a scorer that cannot express refusal, and instances with no solution. | Active |
| SOTA-280 | Sample several reasoning paths and take the majority answer rather than decoding one greedily | Wang et al. (2022), LIT-468 — replace greedy decoding with sample-and-marginalize: GSM8K +17.9, SVAMP +11.0, AQuA +12.2. Five or ten paths recover most of it, which is the whole cost question. | Active |
| SOTA-291 v2 | Evaluate a specific intervention on a specific affordance rather than the net effect of the system, and never read an individual-level null as the absence of a collective effect | Active | |
| SOTA-308 | When a retrieval change raises recall, measure relevance and the end-task metric before calling it an improvement | Wu et al. (2026), LIT-502 — across three published GraphRAG pipelines on G-Medical, graph expansion raises retrieval recall (GFM-RAG 84.3% against vanilla RAG’s 71.8%) and drops context relevance (38.5% against 62.9%). The wider context is noisier, and generation accuracy falls with it — a system that retrieves more of the right thing and much more of the wrong thing answers worse. | Active |
| SOTA-312 v2 | State the noise or binning assumption behind any mutual information you report for a deterministic network, and show the conclusion survives changing it | Saxe et al. (2018), LIT-509 — in a deterministic network I(h;X) is infinite, so every finite number on an information plane is a property of a noise model the analyst imposed and the network never had. The same tanh run, binned evenly in net input instead of evenly in activity, loses its compression phase entirely; and LIT-507 makes compression appear in ReLU networks by changing the binning the other way. | Active |
| SOTA-327 | Train retrieval embeddings with nested losses, so a prefix of the vector can shortlist and the full vector re-rank | Kusupati, Bhatt, Rege et al. (2022), LIT-547 — put the loss on log(d) nested prefixes of the embedding. Each prefix then matches a separately trained model of that width (ResNet50, ImageNet), and one database supports shortlisting on 16 dimensions and re-ranking on 2048: equal mAP@10, 14× faster. Measured on image retrieval. Widely shipped in text embedding models without an independent comparison in the record. | Proposed |
| SOTA-330 | Enforce output structure by masking logits against a grammar, with masks precomputed per automaton state and built on the CPU while the GPU runs the forward pass | Dong et al. (2024), LIT-552 — split the vocabulary by whether a token’s validity depends only on the grammar automaton’s current node (over 99% of tokens for JSON). Precompute those per node, check the rest against the full stack at runtime, and build each mask on the CPU during the GPU forward pass. Mask time is 0.018 ms per token after the ablated optimizations, and end-to-end overhead is 0.1–0.2 ms. It guarantees syntax. It does not measure whether answers stay correct. | Active |
| SOTA-363 | Decouple the negative set from the batch with a queue, and keep the encoder that fills it slow | Active | |
| SOTA-367 | Or put the anti-collapse constraint in the loss, as a variance floor and a decorrelation term | Active | |
| SOTA-369 | Curate a pretraining set by retrieving neighbours of curated seeds, rather than scaling the uncurated pool | Active | |
| SOTA-373 v3 | Set the masking ratio by the signal’s redundancy, not by the ratio that worked on text | Active | |
| SOTA-376 | Score each image-text pair independently with a sigmoid, so the loss needs no global normalization | Active | |
| SOTA-382 v3 | Subsample frequent tokens: it is faster and it improves the rare ones | Superseded — by SOTA-374; the same recommendation from the same paper and section; SOTA-374 states the discard rule and carries the independent ablation | |
| SOTA-400 | Append a few register tokens to a ViT so it does not commandeer background patches for scratch space | Active | |
| SOTA-407 v2 | Interpolate the zero-shot and fine-tuned weights at about half way rather than shipping the fine-tuned model | Wortsman et al. (2021), LIT-674. Fine-tuning a zero-shot model spends the robustness that made it worth starting from. Ship (1 − α)·θ_zero-shot + α·θ_fine-tuned at α = 0.5 instead: +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model, with reference accuracy falling by at most 0.3 pp and usually rising, at no cost in training or inference. Requires that the fine-tuned weights were obtained from the zero-shot weights. | Active |
| SOTA-409 | Average the fine-tuned models from your hyperparameter sweep instead of keeping only the best, adding each one in validation order and only if it helps | Wortsman et al. (2022), LIT-675. A hyperparameter sweep ends by discarding every model but one. Average them instead — sorted by held-out accuracy, keeping each only if held-out accuracy improves, which makes the result no worse than the best individual model on that set by construction. O(1) inference against an ensemble’s O(k). +0.7 pp on CLIP and +0.5 pp on ALIGN over the best sweep member; +0.16 on ImageNet and +0.34 under distribution shift for ViT-G. Averaging all of them uniformly instead can come out worse than the best one. | Active |
| SOTA-413 v2 | In a scaling-law study, count the output head in parameters and FLOPs, and fit loss against compute with an irreducible-loss offset | Pearce and Song (TMLR 2024), LIT-688. At small scale the embedding and output head are a large share of the model. Excluding them from N, and from the FLOPs in C = 6ND, biases the fitted allocation exponent: 0.49 becomes 0.74 when the same runs are relabelled. An offset-free loss–compute fit biases that exponent too. Do not compare exponents across studies that count differently. | Active |
| SOTA-415 v3 | During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch | Li et al. (2025), LIT-684, extending Hägele et al. (2024), LIT-145. To see where a WSD run would land if annealed now, evaluate the simple average of its last ~10 stable-phase checkpoints. On the one fork measured, that proxy sits within about 1.5 points of a real 250B-token anneal on each benchmark. It costs no training. Still anneal the model you ship: on that fork the anneal is ahead on MMLU at the end. | Proposed |
| SOTA-416 v2 | When sampling a diffusion model in few steps, start at t = T: use trailing, not leading, timestep spacing | Martin Garcia et al. (WACV 2025), LIT-687. DDIM’s “leading” spacing never visits the final timestep. So at one step, a pure-noise input is labelled with a timestep that claims an almost clean sample. With the same weights and only the spacing changed, Marigold’s single-step depth goes from noise to NYUv2 AbsRel 5.7. The mismatch vanishes as the step count approaches T, which is why it goes unnoticed at 50 steps. | Active |
| SOTA-421 | Before fitting a compute-optimal allocation law, tune learning rate, batch size and AdamW beta2 per model size and keep warmup short of the smallest run | Porian et al. (NeurIPS 2024), LIT-690. In a small-scale sweep that will be fitted for N*(C), a warmup longer than the smallest models’ optimal runs moves the exponent by 0.10. So does one batch size, learning rate and β₂ shared across sizes. Tune them per size, with β₂ above 0.95 at small batch, and cap warmup (the source uses warmup tokens = N, at most 20% of the budget). A constant learning rate is enough to fit the exponent, at about half the cost of a per-budget cosine. | Proposed |
| SOTA-422 | Rescale the noise schedule so the last timestep is pure noise, and train with v-prediction so the model can learn there | Lin et al. (WACV 2024), LIT-689, with Emu Video’s comparison, LIT-635. Common VP schedules leave signal at t = T. Stable Diffusion’s leaves √ᾱ_T = 0.068. The model learns to keep the leaked channel mean, and pure-noise inference then limits brightness. Rescale √ᾱ_t so it reaches 0 at T. Then train with v-prediction, because at zero SNR ε-prediction is trivial and v_T = x₀. Sample in the x₀ form, because converting v to ε at zero SNR divides by zero. | Proposed |
| SOTA-425 v2 | Report precision and recall beside FID whenever the generator has a fidelity-diversity knob, because FID’s best value sits in the interior of that trade | Dhariwal and Nichol (2021), LIT-699. FID mixes fidelity and diversity, so on any knob that trades one for the other its optimum is interior and a single FID cannot say which side of the trade you are on. Their own Table 5 is the demonstration: ADM-G beats BigGAN-deep on FID at ImageNet 128, 256 and 512 while losing precision at all three (0.78/0.86, 0.82/0.87, 0.87/0.88) and winning recall by 0.24, 0.24 and 0.13. Report precision and recall, state the knob’s value, and never read a single FID as “better samples”. | Active |
| SOTA-443 | Down-weight large residuals in consistency and MeanFlow training: use a Pseudo-Huber loss or an adaptive inverse-error weight, not plain squared L2 | Song and Dhariwal (2023), LIT-786, and Geng et al. (2025), LIT-784. When the regression target is the model’s own output under stop-gradient, squared L2 trains worse than a loss that limits the pull of large errors. iCT’s Pseudo-Huber beats squared L2 on CIFAR-10 (Fig. 2, curves). MeanFlow’s adaptive weight 1/(‖Δ‖² + c)^p takes one-step ImageNet-256 FID from 79.75 at p = 0 to 61.06 at p = 1 (Table 1e). Two groups, small settings, one run per arm. | Proposed |
| SOTA-444 | When a flow will be sampled with a coarse ODE solver, pair noise and data within each minibatch by exact optimal transport | Pooladian et al. (2023), LIT-789, and Tong et al. (2023), LIT-798. Re-pair each minibatch’s noise and data by exact discrete OT before the flow-matching regression. The marginals stay exact, the learned paths straighten, and on ImageNet-64 reaching FID 20 with Euler takes 12 steps instead of 29, at equal likelihood and 4% more time per iteration. It lowers the step count of an ODE sampler. It does not make one: four-step FID stays near 39 on ImageNet-32. Unconditional, 64 px and below. | Proposed |