Field consensus — How far THE FIELD has converged on a practice, which is orthogonal to status — this record’s own editorial position. The useful cases are the corners: Deferred + converged is a practice everybody follows that we have not satisfied ourselves is deliberate, and Active + contested is this record being ahead of the field and saying so, which an anthology should be able to do (ADR-015).
In dispute — credible groups publicly disagree, now — the fork in a line of work, whose trunk is agreed and whose branches are not.
18 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-034 v6 | Use SwiGLU activation for transformers | Shazeer et al. (2020), LIT-030 — ARXIV-2002.05202. Contested at frontier scale in low precision: two groups now ship bounded replacements, on the grounds that SwiGLU’s unbounded factors produce activation outliers. | Active |
| SOTA-124 v5 | Repeat high-quality data freely when its epoch size exceeds the model’s memorization window | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Roughly 100–500 GT for a 7B model, scaling linearly; the authors call the understanding early. | Proposed |
| SOTA-130 v4 | Skip the reasoning SFT stage and run RL with verifiable rewards directly on the base model | Olmo Team (2025), LIT-130 — the RL-Zero track: RLVR on the base model with no reasoning SFT in between, released as an open benchmark for studying RL rather than as the recommended recipe; a variation on SOTA-129 that the record keeps so the family of reasoning recipes can be traced. | Proposed |
| SOTA-136 v4 | Widen the residual stream into several streams mixed by doubly-stochastic matrices (manifold-constrained hyper-connections) | Xie et al. (2025), LIT-140 — hyper-connections with the residual-mixing matrix constrained to be doubly stochastic, so identity mapping survives; 6.7% training overhead at n = 4 in large-scale training, shipped in DeepSeek-V4. | Proposed |
| SOTA-140 v5 | Use a warmup-stable-decay schedule: hold the learning rate, then decay it sharply over the final 10–20% of tokens | Hu et al. (2024), LIT-144, with the controlled comparison in LIT-145 — matches cosine at equal tuning, leaves the token budget open, makes every stable-stage checkpoint a usable branch point; contested at frontier scale by LIT-131, which retuned per schedule and chose cosine. | Active |
| SOTA-154 v4 | Fine-tune with evolution strategies instead of policy-gradient reinforcement learning | Qiu et al. (2025), LIT-211 — evolution strategies over the full parameter space of a billion-scale LLM, which the field had assumed impossible. +36.4% over base on average against GRPO’s +21.3% and PPO’s +17.9%, with ES on one fixed hyperparameter set while RL got a sweep per experiment. Active since Gan and Isola replicated the comparison on six further tasks with the RL arm tuned harder; contested, because Gu et al. argue direct ES at LLM scale does not work — on models at 1.5B and below. | Active |
| SOTA-166 v2 | Set the pretraining data proportions by fitting a mixing law on small runs, not by argument | Ye et al. (2024), LIT-201 — model performance is quantitatively predictable as a function of the mixture proportions, so fitting the law on a sample of mixtures reads off performance on mixtures never trained. Nested with the existing scaling laws, small runs predict a large model. Matched the default mixture trained for 48% more steps. | Proposed |
| SOTA-171 v2 | Repeat a data-constrained corpus for up to about four epochs; past that, added compute stops paying | Muennighoff et al. (2023), LIT-166 — with constrained data at fixed compute, up to four epochs of repeated data changes the loss negligibly against having that much unique data. Past that point the value of adding compute decays toward zero. 400 training runs, up to 900B tokens and 9B parameters, with a scaling law that prices repeated tokens and excess parameters. | Active |
| SOTA-191 v2 | Consider removing LayerNorm’s learnable gain and bias rather than tuning them | Xu et al. (2019), LIT-025 — the bias and gain increase overfitting risk and “do not work in most cases”; LayerNorm-simple, with both removed, beats LayerNorm on four datasets and reaches state of the art on En-Vi. | Proposed |
| SOTA-204 v2 | Anneal a discretisation from coarse to fine over training rather than fixing it | Song et al. (2023), LIT-093. Where a training loss approximates a continuous target through a step count, that count is a bias/variance dial: few steps give a biased but low-variance target early, many steps a faithful but noisy one later. | Proposed |
| SOTA-238 v4 | Set domain weights with a small proxy model under group DRO on excess loss, then transfer them | Xie et al. (2023), LIT-391 — ARXIV-2305.10429. Train a small proxy under group DRO to produce domain weights, then resample and train the real model with them. Optimise worst-case EXCESS loss against a reference model, not worst-case loss — the naive form upweights whichever domain is noisiest, because every domain has a different irreducible entropy. | Active |
| SOTA-333 v4 | For autoregressive generation of continuous sequences, train with an independent noise level per token and condition the rollout on slightly noised history | Chen et al. (2024), LIT-554 — train a causal model to denoise tokens that each carry an independent noise level. At rollout, treat the generated history as slightly noisy, so accumulated errors look like training noise. The same model then samples autoregressively, plans with horizon-wide guidance, and keeps near-future tokens cleaner than far ones. Long-rollout stability is shown on video qualitatively, with a small RNN. Planning and robot gains are measured. | Proposed |
| SOTA-338 v2 | Train the GAN discriminator on frozen multi-scale pretrained features with fixed random channel and scale mixing | Sauer et al. (2021), LIT-562 — project real and generated images through a frozen EfficientNet, mix channels and scales with fixed random convolutions, and train one small discriminator per scale. It reaches StyleGAN2’s best LSUN-Church FID after 1.1M images instead of 88M. The speed-up is measured in FID, and at equal FID on FFHQ a CLIP-space distance and human raters prefer StyleGAN2 (LIT-563). | Proposed |
| SOTA-347 | To write many facts into a model’s weights, update a range of early-to-mid MLP layers jointly in one batched solve (MEMIT), choosing the range by measured edit success rather than causal tracing | Meng et al. (2023), LIT-576 — sequential single-fact edits collapse after tens of facts (ROME scores 50.3 at 10,000). Distributing each update over a range of MLP layers in one least-squares solve holds 85.8 on GPT-J, with neighborhood success falling from 83.5 to 73.7. Choose the layers by editing performance: per-fact causal tracing does not predict it (LIT-574). Evaluate implications too, because weight editors fail them (LIT-577). | Proposed |
| SOTA-374 v4 | When learning embeddings from co-occurrence, subsample very frequent tokens, discarding each occurrence with probability 1 − √(t/f) with t around 10⁻⁵ | Mikolov et al. (2013), LIT-603 — in a Zipfian corpus, the few most frequent words co-occur with everything and dominate the updates while teaching little. Discard each occurrence of word w with probability 1 − √(t/f(w)). This cut skip-gram training time by 2–3× and raised accuracy, only slightly for words (59 → 60% at NEG-5) but a lot for phrases (27 → 42% at NEG-15, 19 → 47% with hierarchical softmax). | Active |
| SOTA-378 | Compare, project and orthogonalize a language model’s concept directions after whitening by the unembedding covariance, not by raw cosine | Park, Choe and Veitch (2024), LIT-606 — training fixes a model’s representation only up to an invertible linear map, so Euclidean cosine between concept directions is not meaningful. Use ⟨u, v⟩ = uᵀ Cov(γ)⁻¹ v, with Cov(γ) the covariance of the unembedding rows, or equivalently transform by Cov(γ)^(−1/2) and then use Euclidean tools. Shown on LLaMA-2 7B and Gemma-2B by heatmap. Contested by a null for cross-lingual transport in 17 models. | Proposed |
| SOTA-392 v3 | To get a one-step sampler from a rectified flow, distil it; treat reflow as an optional extra pass, and keep the pre-reflow model for many steps | Liu, Gong and Liu (2022), LIT-636. On CIFAR-10, one-step FID is 378 from the base model, 6.18 after distillation alone, 12.21 after one reflow, and 4.85 after reflow and distillation (Table 1a). Distillation does most of the work. Reflow adds a further gain at the cost of a second training pass, and worsens many-step quality from 2.58 to 3.36. InstaFlow (LIT-796) contests it at text-to-image scale, starting from a diffusion model rather than a rectified flow. | Proposed |
| SOTA-430 v8 | Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks | Adebayo et al. (2018), LIT-713, read as NOTE-365, with Kokhlikyan et al. (2021), LIT-724. An attribution map that survives re-initializing the model’s weights, or retraining on permuted labels, cannot be telling you about the weights or the labels. Guided Backprop and Guided GradCAM survive the first above the lowest layers, and their maps look as convincing as ever. A rejection rule, not a certificate: passing does not show a method is faithful — GradCAM passes and agrees with occlusion maps at rank correlation 0.254. The verdict is per-modality — the same test on a BERT text classifier reverses it for global Integrated Gradients — and pass or fail is read from curves with no threshold anywhere. | Proposed |