Field consensus — How far THE FIELD has converged on a practice, which is orthogonal to status — this record’s own editorial position. The useful cases are the corners: Deferred + converged is a practice everybody follows that we have not satisfied ourselves is deliberate, and Active + contested is this record being ahead of the field and saying so, which an anthology should be able to do (ADR-015).
One source — one group, one result — nobody has agreed or disagreed yet, which is not the same as contested, where people looked and differed.
145 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-122 v2 | Attach learnable per-row and per-column multipliers to weight matrices so their norms are learned, not set by LR and WD | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Up to 20% relative gain on MMLU, BBH and GSM8K over a Muon baseline at 200 GT. | Proposed |
| SOTA-125 v3 | At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. 24–27 layers × 512 hidden and a large SSM state dimension won every ablation at 90M; 50 layers gained MMLU but halved throughput. | Proposed |
| SOTA-141 v3 | Decay the learning rate linearly all the way to zero | Bergsma et al. (2025), LIT-147 — with the peak tuned, linear decay to zero beats the customary decay to 10% and other shapes at compute-optimal budgets, more so past them; Proposed because contemporary production runs in the record still decay to a floor. | Proposed |
| SOTA-152 | Pack training documents by best fit instead of concatenating and splitting | Ding et al. (2024), LIT-203 — concatenate-then-split truncates documents that would have fit, and the model then learns to continue text whose beginning it never saw. Treating the grouping as bin packing removes those truncations at the same efficiency, no padding added: +4.7% reading comprehension, +16.8% context following, +9.2% program synthesis, and up to 58.3% less closed-domain hallucination. | Proposed |
| SOTA-155 | Train across poorly connected islands: many inner steps per worker, an outer momentum optimizer over the deltas, and a streamed synchronisation | Douillard et al. (2023, 2025), LIT-212 and LIT-214 — federated averaging with the constants pushed hard: many inner AdamW steps per worker, Nesterov momentum as the outer optimizer over the accumulated deltas, synchronised rarely and in streamed subsets. 500× less communication at matched quality, then two more orders off the peak. Filed Proposed: one group, and no frontier report trains this way. | Proposed |
| SOTA-156 v2 | Train without a learning-rate schedule: average the iterates so no stopping time need be fixed | Defazio et al. (2024), LIT-213 — scheduling and iterate averaging are one mechanism, and the averaged form needs no stopping time T, no schedule shape and no hyperparameter beyond what AdamW already has. Filed Proposed: the evidence is a third-party competitive benchmark at workloads far below the scale where this record’s schedule question is open. | Proposed |
| SOTA-157 v3 | Train the language model as a masked diffusion model rather than autoregressively | Nie et al. (2025), LIT-217 — hold the paradigm fixed (pretrain, then SFT) and swap only the factorization: a forward masking process and a reverse process predicting masked tokens, optimizing a likelihood lower bound. Competitive with LLaMA3 8B on in-context learning; past GPT-4o on reversal poem completion. Filed Proposed — one group, one model, 8B. | Proposed |
| SOTA-159 | Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling | Blake et al. (2024), LIT-149 — µP makes activation scale independent of model size; unit scaling makes activations, weights and gradients start at a scale of one. Together the model is size-independent and well-scaled from the first step, the hyperparameters become close to independent so a one-dimensional sweep replaces a joint search, and FP8 works without loss-scaling tricks. | Proposed |
| SOTA-160 | Treat the pretraining token budget and the post-training quantization plan as one decision, not two | Kumar et al. (2024), LIT-186 — degradation from post-training quantization increases with how much data the model was trained on, to the point where additional pretraining data becomes actively harmful if the model will be quantized afterwards. Fit over 465 runs into one precision-aware scaling law covering both pre- and post-training quantization. | Proposed |
| SOTA-161 v4 | Keep the attention output in FP32 during training, because flash attention’s rounding bias compounds instead of cancelling | Qiu et al. (2025), LIT-198 — putting flash attention in BF16 while the FFN goes to FP8 sometimes explodes, and the cause is two things at once: attention produces low-rank updates repeated across steps and tokens, and low-precision addition rounds with a bias. The biased error rides the repeated update and compounds into a systematic gradient bias instead of averaging out. | Active |
| SOTA-167 | Build linear-attention layers from the state-space view: expressive discretisation, complex-valued state, and a MIMO readout | Lahoti et al. (2026), LIT-165 — three changes derived from the SSM view: a more expressive recurrence from the discretisation, a complex-valued state update for state tracking, and a MIMO formulation that adds quality without decode latency. +1.8 downstream at 1.5B over Gated DeltaNet, and Mamba-2’s perplexity at half the state size. | Proposed |
| SOTA-168 | Run Adam in Shampoo’s eigenbasis (SOAP) instead of Shampoo itself | Vyas et al. (2024), LIT-157 — Shampoo at the 1/2 power is Adafactor in the eigenbasis of Shampoo’s preconditioner, so run the better optimizer in that basis. SOAP is Adam there, adding exactly one hyperparameter over Adam. Over 40% fewer iterations and 35% less wall-clock than AdamW at 360M–660M in the large-batch regime. | Proposed |
| SOTA-170 v2 | Choose filtering aggressiveness by the token horizon: at long horizons rephrase what a filter would discard | Su et al. (2024), LIT-178 — aggressive model-based filtering wins at 1T tokens and loses at 15T, because removing 90% of the data optimises mean quality and mean quality is not what binds when the corpus runs out. Ensemble the classifiers, rephrase text synthetically instead of discarding it, and lean less on heuristics. | Active |
| SOTA-172 v2 | Build synthetic pretraining data by editing human text at the token level, not by generating from scratch | Zhu et al. (2024), LIT-205 — pretraining across proportions of synthetic data shows a negative correlation between the proportion and performance, caused by distributional shift and over-concentration of n-gram features. The remedy inverts the usual move: edit human data at the token level, so the result stays anchored to a real distribution and the test error is provably bounded. | Proposed |
| SOTA-173 v2 | Augment the objective to make multi-epoch pretraining productive on a fixed corpus | Chen et al. (2026), LIT-175 — standard autoregressive pretraining overfits severely on a fixed corpus, reaching its optimum early and deteriorating continuously. Three orthogonal augmentation families each delay that and lower validation loss; combining them lowers the minimum further. The claim is hundreds of productive epochs on the same data. | Proposed |
| SOTA-175 | Mask whole syntactic units for code fill-in-the-middle, not random character spans | Gong et al. (2025), LIT-126 — standard FIM masks random character spans, which produces training examples that rarely correspond to an edit anyone makes. AST-FIM masks whole syntactic units at scale, so the middle span is a coherent structure. Up to 5 points over random-character FIM at 1B and 8B, and most useful on real editing. | Proposed |
| SOTA-176 | Hybridise inside the layer: long-term slots, a sliding window, and one softmax over both, with the window size as the only dial | Du et al. (2025), LIT-176 — long-term context in key-value slots updated by a linear RNN, short-term context from a sliding window, and a single softmax attending over both together, so the long/short weighting is per-token, per-head and content-dependent with no fusion parameters. Every layer stays structurally identical and one hyperparameter slides from purely linear to full attention. | Proposed |
| SOTA-177 v2 | Decouple the erase address from the write address in a delta-rule recurrence | Li et al. (2026), LIT-177 — the delta rule corrects what is stored at the current write address before writing there, so stale information at a different address can only decay passively. EDA runs a targeted erase along a learned erase direction first, then the ordinary corrective write. Best at both 2.5B dense and 25B-A2.8B MoE, and the gain persists after 80B tokens of long-context midtraining. | Proposed |
| SOTA-178 | Give the recurrence vector-valued gating and in-context learning rates so it can track state a softmax layer provably cannot | Peng et al. (2025), LIT-173 — a generalised delta rule with vector-valued gating and in-context learning rates, at constant memory and constant time per token. A 2.9B model sets the 3B state of the art on multilingual tasks on dramatically fewer training tokens. The expressivity claim is the sharp one: it can recognise all regular languages, which under standard conjectures exceeds what a softmax stack in TC⁰ can do. | Proposed |
| SOTA-179 | Truncate the rotary encoding’s low frequencies rather than rescaling its base | Barbero et al. (2024), LIT-210 — RoPE’s high frequencies build positional attention heads and its low frequencies carry semantics that provably cannot stay robust over long context. Keeping a fraction p of the frequencies holds performance and at 2B improves it; p=1 is RoPE and p=0 is NoPE, so the practice is a dial between two things the record already holds. | Proposed |
| SOTA-190 v3 | Increase depth before any other dimension when scaling a transformer | Tay et al. (2021), LIT-052 — the DeepNarrow strategy. Small 16L matches T5-Base downstream at 60% of the parameters, 63% of the FLOPs and 40% faster; the limit is parallelism rather than quality. | Proposed |
| SOTA-212 | Post-train by scoring many random weight perturbations in one parallel pass and majority-voting the best of them | Qiu et al. introduced it (LIT-211); Gan and Isola (2026), LIT-233 — ARXIV-2603.12228 — measured this form of it. Sample N Gaussian weight perturbations, score them on a few hundred held-out examples, keep the top K and majority-vote at inference. At equal training FLOPs it matches or beats PPO, GRPO and ES across seven tasks at 0.5B–8B, in one parallel step instead of hundreds of sequential ones — 3.2 minutes for OLMo3-7B on Countdown across 200 GH200s. Filed Proposed: one group, K forward passes per answer, and discrete answers only. | Proposed |
| SOTA-215 | Freeze Adam’s variance term after a warmup before compressing its update, and never compress the update directly | Error-compensated compression assumes the residual it carries forward still means the same thing next step. Adam’s update divides by a running variance, which is a non-linearity the correction does not survive — compressing it directly corrupts the error cancellation and badly harms convergence. The workaround is structural rather than a tuning fix: Adam’s variance stabilises early, so freeze it after a warmup and the update becomes linear enough to compress. | Proposed |
| SOTA-216 | Switch from minibatch SGD to local SGD at the first learning-rate decay, rather than choosing between them | The two options are usually posed as a choice: minibatch SGD generalises and communicates every step, local SGD communicates rarely and generalises worse. Post-local SGD is the observation that they are wanted at different times — run minibatch SGD until the first learning-rate decay, then switch to local SGD with H around 16 to 32 for the rest. The early phase is where the gradient noise is doing work; the late phase is where communication is pure overhead. | Proposed |
| SOTA-218 v2 | Retune the learning rate, momentum and schedule at every batch size you compare, never by a scaling heuristic | Linear scaling and square-root scaling are not batch-size rules; they are guesses about where the optimum moved, and they hold over a narrower range than the comparisons people use them for. A curve of steps-to-target against batch size drawn with transferred metaparameters is not a measurement of batch size at all — it is a comparison between one tuned configuration and several untuned ones, and the point where it bends is the point where the heuristic failed. | Active |
| SOTA-220 | Aggregate worker gradients coordinate-wise by median when any worker may be faulty | Averaging has a breakdown point of zero: one worker returning an arbitrary vector can move the mean anywhere, and adding workers does not help, because it is the worker’s magnitude and not its share of the population that does the moving. The coordinate-wise median is the robust aggregator that needs no estimate of how many workers are faulty; the trimmed mean has better rates and needs one. | Proposed |
| SOTA-221 v2 | To train past the batch size where your optimizer stalls, change the optimizer’s conditioning rather than the scaling rule | You et al. (2019), LIT-265. When a batch size stops reaching your accuracy target, the reflex is to look for a better learning-rate scaling rule. LIT-058 is 168,160 training runs of evidence that there is no such rule to find. What moved the wall instead was the optimizer: normalizing each layer’s update by the ratio of its parameter norm to its update norm, on top of per-coordinate adaptivity, carried BERT from 16K to 32K where a grid-searched AdamW could not follow. This does not license skipping the retuning in SOTA-218 — the two say different things about different activities, and THEORY-012 is why. | Active |
| SOTA-222 v2 | Overlap the synchronisation with one or two local steps instead of communicating every step | Wang et al. (2020), LIT-323. Local SGD is usually argued for as a way to communicate rarely, which costs accuracy at large intervals. The cheaper reading is that it does not take a large interval to win: at tau = 2 the communication-to-computation ratio falls from 34.6% to 1.5% with convergence per iteration roughly unchanged. The interval is not buying infrequency, it is buying somewhere to hide the exchange. | Proposed |
| SOTA-223 | Do not keep the gossip topology static: sample a fresh random neighbourhood every round | de Vos et al. (2023), LIT-315. Decentralized training has spent years choosing good fixed graphs. Epidemic Learning does not choose one: each round, every node sends its update to s peers picked at random. The transient-iteration count falls from the best-known O(n^3) to O(n^3/s^2), and at 96 nodes on CIFAR-10 it converges 1.7x faster and 2.2% more accurately than a static 7-regular graph at equal communication volume. | Proposed |
| SOTA-224 | Ramp the sparsity ratio up over the first few epochs rather than starting at the target | Lin et al. (2017), LIT-056. Gradient sparsification delays the coordinates it drops; error feedback returns them, but it returns them late. Early in training the gradients are large and the trajectory is still choosing where to go, so a delay there costs more than the same delay later. Start at a low sparsity — the paper ramps 75%, 93.75%, 98.4375%, 99.6%, 99.9% over the first few epochs — and reach the target once the run has settled. | Active |
| SOTA-225 | Drop the parameter server for gossip once the network is the bottleneck, and measure where that is rather than inheriting a threshold | Lian et al. (2017), LIT-302. The question is whether a decentralized algorithm can ever beat a centralized one, and the answer is that it can — not by converging better, since the rates match, but by having no node that every other node talks to. The advantage appears where the central node’s link is the constraint, and the paper establishes that by sweeping bandwidth and latency rather than by naming a threshold. | Active |
| SOTA-226 | If the gossip topology is static, make it a directed exponential graph, not a ring | Assran et al. (2018), LIT-254. Having chosen gossip over a parameter server, the next choice is the graph, and it is not a detail: a ring’s consensus rate degrades with the node count badly enough that the speedup regime recedes out of reach. A directed exponential graph — each node sends to peers at distance 1, 2, 4, 8, … in rotation — reaches exact consensus in log2(n) rounds with constant out-degree per round, and spreads the load evenly. | Active |
| SOTA-229 | Scale the draft model’s training data, once nothing constrains it to predict the target’s features | Draft models had stopped improving with more training data, and LIT-185 identifies why: EAGLE trained its draft to predict the target’s hidden features and read tokens off them, which supplies multi-step training signal and caps expressiveness. Remove the constraint — predict tokens directly, fuse features from several depths, and simulate the multi-step draft during training — and the speedup rises with the data instead of flattening. Up to 6.5x, about 1.4x over the previous version, on roughly 8x the data. | Proposed |
| SOTA-231 | Put an adapter on every linear layer, not just the attention query and value projections | Dettmers et al. (2023), LIT-378 — ARXIV-2305.14314. Adapters on the query and value projections alone — the default inherited from the LoRA paper — do not reach full fine-tuning on large base models. The number of adapted matrices is the hyperparameter that decides it; the projection rank r, which is the one people tune, does not affect performance across the paper’s sweep. | Active |
| SOTA-232 | Convert an autoregressive model into a diffusion model by continual pretraining rather than training one from scratch | Gong et al. (2024), LIT-381 — ARXIV-2410.17891. Continual pretraining on under 200B tokens turns an existing autoregressive model into a diffusion language model — GPT2 and LLaMA across 127M-7B — competitive with the AR original. The reason to want one is infilling without prompt re-ordering, not perplexity. | Proposed |
| SOTA-233 | Train self-correction with multi-turn online RL on the model’s own traces, rather than prompting for it | Kumar et al. (2024), LIT-383 — ARXIV-2409.12917. Self-correction is a capability to be trained, not a behaviour to be requested. Prompting a model to revise its own reasoning does not reliably help and can hurt; supervised fine-tuning on correction traces fails by distribution mismatch or behaviour collapse; multi-turn online RL on self-generated traces reaches +15.6% on MATH and +9.1% on HumanEval. | Active |
| SOTA-234 | Train in the target low-bit format from scratch rather than quantizing a finished model | Ma et al. (2024), LIT-380 — ARXIV-2402.17764. Decide the serving format before training and train in it, rather than training in FP16 and compressing afterwards. Ternary weights trained from scratch match an FP16 model of equal size and token budget from 3B upward — but the compute argument assumes hardware built for the format, and only the memory saving is measured on machines that exist. | Proposed |
| SOTA-235 v3 | Update the weights at inference on the test instance when the task is structurally novel | Akyürek et al. (2024), LIT-379 — ARXIV-2411.07279. Build a loss from the test instance’s own in-context examples, take gradient steps at inference, then discard the update. Up to 6x a fine-tuned baseline on ARC-AGI (53.0% at 8B) and +7.3 points on BIG-Bench Hard at 10-shot. The examples are worth several times more as gradient than as context. | Active |
| SOTA-239 | Order training data by skill prerequisite: teach the prerequisite and the dependent skill costs less data | Chen et al. (2023), LIT-392 — ARXIV-2307.14430. Training data has a prerequisite structure, and it is exploitable: train on a prerequisite skill and the dependent skill is reachable with less data. Sample over skill mixtures online rather than fixing proportions in advance. | Proposed |
| SOTA-242 | Select pretraining data by matching a target distribution you can sample from, not by scoring it for quality | Xie et al. (2023), LIT-087 — data selection needs a description of what you want, not a criterion for what is good. Supply a small sample of the target text and resample the raw corpus to match it. The target is an artifact you can produce; a quality rubric is an argument you have to win. | Proposed |
| SOTA-243 v2 | Discard the easy examples when data is abundant and the hard ones when it is scarce | Sorscher et al. (2022), LIT-399 — power-law scaling in dataset size is evidence of redundancy, and given a difficulty ranking it can be beaten toward exponential. But which end of the ranking to discard flips with how much data you started with, and the fraction to discard has to grow with the corpus. | Proposed |
| SOTA-247 | Add a multi-token word to a frozen model’s vocabulary from the model’s own detokenized representation of it | Kaplan et al. (2024), LIT-412 — a model already computes a whole-word representation for a word its tokenizer splits, so initialize the new embedding and unembedding from that rather than from an average of the word’s token embeddings. On Llama2-7B the frozen model then uses the new tokens while keeping its accuracy on existing ones (0.519 against 0.522 on WikiText-103), where mean-embedding initialization both fails to use them and degrades the model. The gain is largest where the tokenizer is worst: Arabic Wiki40B, 0.402 new-token accuracy against 0.117. | Proposed |
| SOTA-248 | Stretch and clip the softmax so an attention head can output exact zeros | Bondarenko et al. (2023), LIT-414 — a head that wants to do nothing has to drive its softmax input to infinity to approximate exact zeros, and that is what creates the activation outliers that break INT8. Stretch the softmax to (γ, ζ) and clip back to (0,1) and zeros become reachable from a finite input. On BERT-base: W8A8 perplexity 1294 → 4.55, max infinity-norm 735 → 20, and the FP16 model gets slightly better rather than worse. Evidence is BERT-base, OPT-125M and ViT-S/16 — small and encoder-heavy. | Proposed |
| SOTA-250 | Pretrain by predicting representations of masked regions, not their pixels | Assran et al. (2023), LIT-216 — ARXIV-2301.08243. Predict the representations of masked target blocks from a context block, with a learned target-encoder, instead of reconstructing pixels. No hand-crafted augmentations, and a ViT-H/14 reaches strong downstream performance in under 1200 GPU-hours — faster than a ViT-S/16 trained with iBOT. | Active |
| SOTA-251 | Train at low resolution and raise it only during the decay phase of a warmup-stable-decay schedule | Assran et al. (2025), LIT-215 — ARXIV-2506.09985. Spend warmup and the constant phase on short, low-resolution clips; raise resolution and clip length only in the final decay. Up to 8x less pretraining compute than training at full resolution throughout, for the same end state. | Proposed |
| SOTA-252 | Make the full-resolution path affordable instead of upsampling, when the shortcut is what breaks correctness | Abdal et al. (2023), LIT-113 — ARXIV-2311.17857. Prior 3D GANs render small and upsample in 2D because volume rendering is too slow at training resolution — but a 2D upsampler is multi-view inconsistent by construction. Efficient Gaussian rendering makes native 512x512 affordable, and the inconsistency leaves with the upsampler rather than being mitigated. | Active |
| SOTA-253 | Add a new instance by reconstructing it, not by training on it | Sun et al. (2022), LIT-111 — ARXIV-2205.12257. Split what the system knows about a specific object from what it has learned in general. Build the object’s structure once by reconstruction; match against it with a network that never saw the object or its category. Adding an object is then a scan, not a training run. | Active |
| SOTA-254 v2 | Train the language model as a masked diffusion model when the corpus is fixed and the compute is not | Prabhudesai et al. (2025), LIT-442 — masked diffusion keeps extracting signal from a repeated corpus for roughly 512 epochs where autoregressive training stops at 32. Below a critical compute threshold, given in closed form from the unique-token count, autoregressive training is better and by a wide margin; above it, diffusion is better. The recommendation is the condition, not the objective. | Proposed |
| SOTA-255 v2 | Tune weight decay upward when pretraining over a repeated corpus, rather than inheriting the customary value | Kim et al. (2025), LIT-441 — with the corpus fixed and parameters past Chinchilla-optimal, the weight decay of 0.1 everyone inherits from GPT-3 is far too small: loss turns upward as parameters or epochs are added. Tuned jointly with learning rate and epoch count, the optimum is roughly 30x larger, and loss becomes monotone in parameter count with a steeper exponent than Chinchilla’s. | Proposed |
| SOTA-256 | Judge a monotone scaling recipe by its asymptote, not by its loss at a compute budget | Kim et al. (2025), LIT-441 — when compute is not the binding constraint, a recipe’s loss at a chosen budget answers a question nobody asked. Fit the scaling law and compare the asymptote instead. The catch is that an asymptote is an extrapolation, and rankings at small scale can invert: the ensembling recipe wins on asymptote while losing at small member counts. | Proposed |
| SOTA-257 v2 | Spend surplus compute on an ensemble of independently seeded models and distil it, rather than on one larger model | Kim et al. (2025), LIT-441 — under a fixed corpus, training K models that differ only in seed and averaging their logits reaches a lower loss asymptote than scaling one model to the same total parameter count. Two 300M models beat one 600M. Members want more epochs and less weight decay than a standalone model, and an 8-member ensemble distils into a 300M student retaining about 83% of the gain. | Proposed |
| SOTA-259 | Hold Adam’s second-moment half-life fixed in tokens when the batch size changes, not beta_2 | Marek et al. (2025), LIT-444 — beta_2 is a decay per optimizer step, and a step is not a fixed amount of data. Changing batch size from B to B' while holding beta_2 fixed changes the second moment’s averaging window in tokens by the same factor. Hold the half-life fixed instead: beta_2' = beta_2^(B'/B). beta_1’s default is fine. | Proposed |
| SOTA-260 | Use the smallest batch size that still saturates the device, and do not gradient-accumulate to avoid it | Marek et al. (2025), LIT-444 — batch size one trains stably once Adam’s second-moment half-life is held fixed in tokens, matches or beats large batches per FLOP, and is far more robust to hyperparameter misspecification. Gradient accumulation costs memory for the accumulated gradient and forgoes optimizer steps, and is worth it only when multiple device replicas are bottlenecked on interconnect bandwidth. | Proposed |
| SOTA-261 | Set AdamW’s weight decay by targeting a timescale that follows a power law in tokens-per-parameter | Bergsma et al. (2025), LIT-443 — under AdamW the weights are an EMA of updates with timescale 1/(eta*lambda), and the fraction of training it averages, tau = 1/(eta*lambda*S), is the quantity that scales. It follows a power law in tokens-per-parameter with exponent about -0.52 across three orders of magnitude of compute, so weight decay can be read off in advance rather than swept. | Proposed |
| SOTA-264 v2 | Fix the noise schedule’s endpoints against the bound and choose its shape to minimise the loss estimator’s variance | Kingma et al. (2021), LIT-446 — in continuous time the bound depends on the schedule only through its endpoints, so the shape is free. Spend it: optimise the endpoints against the VLB and the shape against the variance of the loss estimator. Same bound, faster optimisation, and a low-discrepancy sampler for the time variable cuts the variance further. | Proposed |
| SOTA-267 v3 | Warm up on a subset of the entities before training on all of them, rather than holding one distribution fixed throughout | Zucchet et al. (2025), LIT-450 — the paper argues two quantities move in opposite directions: plateau length is governed by how often the most common entities appear, acquisition speed after the plateau by how often the least common ones do. So concentration buys an early exit from the plateau and costs the tail, and a warm-up that trains on a subset of the entities first, then on all of them, beat the best fixed distribution tried, in single-seed runs on synthetic biographies. | Proposed |
| SOTA-268 | Do not transfer a data mixing ratio across model scales; the threshold where a domain starts being learned moves with size | Gu et al. (2025), LIT-451 — knowledge acquisition from a knowledge-dense dataset mixed into web text has thresholds, not a smooth curve. Below a critical model size, or a critical mixing ratio, the model memorises almost nothing however long it trains; past it, accuracy jumps to over 60%. The critical ratio follows a power law in model size, so a ratio chosen at one scale can be on the wrong side of a transition at another. | Proposed |
| SOTA-271 | Train on a second modality even when the target is single-modality | Huh et al. (2024), LIT-458 — if representations converge on a modality-agnostic model of what generated the data, then data from any modality helps find it. The vision direction is already common practice; the language direction is not, and the paper cites a case where adding image data improved text performance. | Proposed |
| SOTA-282 | Put every matrix and hidden state on the unit hypersphere, and make each block’s residual step size a learned per-dimension parameter | Proposed | |
| SOTA-283 | To fine-tune a model that is already quantized, bank the part of each update that is smaller than the lattice spacing, and rebuild the accumulator from seeds rather than storing it | Proposed | |
| SOTA-284 v2 | Choose the checkpoint you post-train from by how much mass it puts on good rare responses, not by validation cross-entropy | Proposed | |
| SOTA-285 | Put dropout at the start of training if the model underfits, and at the end if it overfits — the schedule decides the sign, not the rate | Proposed | |
| SOTA-286 | To find a model that generalizes badly where its outputs look fine, measure how much of parameter space behaves the way it does | Proposed | |
| SOTA-287 | If you need to train without backpropagation, train each block to denoise a noisy label embedding conditioned on the raw input | Proposed | |
| SOTA-288 | Late in training, replicate a small subset of the weights, train the copies against shared base weights, and average the subset back into one model | Proposed | |
| SOTA-289 v2 | When the sampling budget is small, prefer uniform-state discrete diffusion with consistency distillation — masked diffusion cannot revise what it has already emitted | Proposed | |
| SOTA-290 | To measure what a deployed recommender does, use sock-puppets that never interact, follow sets curated to your question, and exposure weighted by rank | Proposed | |
| SOTA-292 | Check industry ties yourself when reading a literature about deployed platforms — disclosure is unreliable and the ties are mechanically detectable | Proposed | |
| SOTA-294 | Fit the tokenizer to the corpus when the corpus is deliberately narrow | Finke et al. (2025), LIT-485 — holding the dataset and the architecture fixed and changing only the tokenizer, a 4,096-token WordPiece vocabulary seeded with English affixes beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality. It is the one fully controlled comparison in a paper that is mostly about something else, and the effect is larger than the architecture change measured beside it. | Proposed |
| SOTA-295 | Parameterize the generating prompt above the word list, and constrain how each sample opens | Finke et al. (2025), LIT-485 — requiring a story to contain three words drawn from a vocabulary constrains the lexicon and leaves the frame alone, and the frame is where a generator repeats itself: 59.38% of TinyStories begins “once upon a time”. Name a theme, a topic, a style and a narrative feature per sample, and require the sample to open with a given part of speech and letter. Five diversity metrics move; measured simplicity does not. | Proposed |
| SOTA-296 | Add the parallel streams in a short final training stage, not from the start | Chen et al. (2025), LIT-486 — parallel scaling costs P× the training FLOPs, which is what would otherwise make it unaffordable. Train 1T tokens the ordinary way, then switch the streams on for 20B tokens, 2% of the budget. The loss spikes when the random prefixes appear and recovers within 0.0002T tokens, and the logarithmic gains in P are the same ones the from-scratch runs show. | Proposed |
| SOTA-297 | Scale parallel computation with learnable input transforms, not parameters, when inference memory is the binding constraint | Chen et al. (2025), LIT-486 — run the same weights over P learnably-prefixed copies of the input and learn the aggregation. Loss falls as if the parameters had grown by O(log P); a 1.6B model at P = 8 matches a 4.4B one on code. At batch size 1 that costs 22× less added memory and 6× less added latency than the parameter scaling it replaces — and P× the training FLOPs. | Proposed |
| SOTA-298 | Measure a writing assistant by the outcome it was deployed to improve, not by the text features that used to predict it | Corpus et al. (2025), LIT-487 — Change.org’s AI drafting tool made petitions 49% longer, more lexically varied and better rated by human judges on quality and persuasiveness, moving them toward the profile of successful pre-AI petitions. The share reaching 10 signatures fell 5.33 points. The features had been chosen because they predicted success; afterwards they predicted it weakly or negatively. | Proposed |
| SOTA-299 | Finetune the whole generative model, encoder and decoder, for a dense perception task instead of attaching a head to a backbone | Khangaonkar and Pirsiavash (2025), LIT-488 — the freshly-initialized mask head is the part that cannot generalize, because it has only ever seen the finetuning categories. Keep every parameter generatively pretrained, encode the target as an image, and a model finetuned on furniture and cars segments people, x-rays and paintings. The same backbone under a conventional head scores 1.4–2.4 mIoU. | Proposed |
| SOTA-300 | To tell whether a model is doing non-trivial in-context computation, measure how well it predicts its own hidden states, not its next-token loss | Herrmann et al. (2025), LIT-489 — uniform noise maximizes next-token loss and requires no computation; reciting a memorized licence minimizes it and requires none either. Insert a variational bottleneck with a learned autoregressive prior and measure the posterior-prior KL: only the genuinely in-context task scores high, on a Transformer and an LSTM, and the signal survives controlling for next-token loss. | Proposed |
| SOTA-301 | Apply the objective gradient before the denoiser, not after it, when guiding diffusion toward a task objective | Zhang et al. (2026), LIT-490 — the usual recipe adds −η∇f after the denoising step, which walks the sample off the geometry the model learned with nothing left to pull it back. Apply the gradient to the noisy iterate first and let the denoiser follow: it acts as an approximate projection, and the sampler becomes an inexact projected-gradient method. Same cost, one line, and in trajectory planning it is the difference between a plan that looks good and one that survives execution. | Proposed |
| SOTA-302 v2 | Steer a distilled generator by modulating its input noise, not by fine-tuning its weights | Eyring et al. (2025), LIT-491 — train a LoRA hypernetwork to predict an improved initial noise for a frozen step-distilled generator. GenEval on SANA-Sprint goes 0.70 → 0.75 for 0.1 s of added latency, recovering about half of what 30-second test-time optimization buys. Reward fine-tuning the same model instead takes it 0.73 → 0.62: the anchoring KL term is intractable in weight space and tractable in noise space. | Proposed |
| SOTA-303 | Name the phase a time-saving tool accelerates before predicting what it will do to the quality of what people produce with it | Duede, Gross, Crockett and Bergstrom (2026), LIT-492 — in a marginal-value-theorem model, accelerating the phase before value is known makes producers more selective, accelerating the minimum cost of shipping makes them less selective, and both make each item less thoroughly developed — while accelerating discretionary improvement makes each item more thoroughly developed. Opposite signs from the same tool depending on where it acts, so “it saves time” does not determine the direction. | Proposed |
| SOTA-304 | Generate a long program unit by unit, freezing each unit that passes an execution-based check, instead of regenerating the whole artifact on failure | Cheng, Clark and Richardson (2025), LIT-493 — on 100 held-out proposals, the same model produced valid implementations 92% of the time generating unit by unit against an execution-based checker, and 6% of the time one-shot with retries; the one-shot outputs were also trivial, 49 lines against 181. Removing the checker alone costs 62 points, removing the decomposition alone costs 19 — so if you build one, build the checker. | Proposed |
| SOTA-307 v5 | Report generative FID as an error bar over several training seeds, and treat any gap below about 2% of the mean as inconclusive | Dufour, Efros and Pérez (2026), LIT-501 — on several hundred SiT networks, retraining the same recipe moves FID 3.2× more than resampling from a fixed model, and the coefficient of variation holds at 1–2% across four model sizes and every compute budget to 2M steps. Ten times the sampling budget shrinks the small term by √10 and leaves the large one untouched. A lucky training seed is worth up to 2× the compute. | Active |
| SOTA-309 | Score a control interface by the mutual information between the operator’s command and the state change it induces, and choose the horizon of that state change deliberately | Reddy, Levine and Dragan (2022), LIT-503 — whatever the operator is trying to do, a better interface yields less noisy commands, so I(x_t, (s_t, s_{t+Δ})) scores it with no labels, no reward and no task knowledge. Spearman ρ = 0.43 against true task completion across 540K examples in 4 of 5 domains. In the fifth the correlation was strongly negative at Δ = 1 and positive at episode length — so the horizon is not a detail. | Proposed |
| SOTA-310 | Treat a mechanistic explanation that passes circuit error or causal alignment as one of many, and report what you did to rule the others out | Méloux, Maniu, Portet and Peyrard (2025), LIT-504 — by exhaustive enumeration on small MLPs, no network had exactly one circuit interpretation and under 2% had exactly one valid minimal mapping. The median number of explanations satisfying the criteria rises from 38 to 910,000 as width goes 2 → 5, and both counts are lower bounds. Passing circuit error or IIA establishes that an explanation is admissible, not that it is the explanation. | Active |
| SOTA-314 v2 | Absorb quantization outliers into a high-precision low-rank branch taken from the weights, and fuse its kernels into the low-bit ones | Li et al. (2024), LIT-512 — at 4 bits on both weights and activations, smoothing alone is not enough. Shift outliers from activations into weights, peel the dominant singular values into a 16-bit rank-32 branch, quantize the residual. Then fuse: run naïvely the branch costs 57% latency and cancels the win; fused it gives 3.0× over W4A16 and 3.5× memory on 12B FLUX.1. | Active |
| SOTA-315 v2 | Early-stop and tune on validation loss after temperature scaling, then calibrate post hoc, instead of stopping on raw validation loss | Berta et al. (2025), LIT-514 — a proper loss is calibration error plus refinement error, and the two bottom out at different epochs, so the validation-loss minimum is optimal for neither. Fit a temperature on the validation set each time you would have read the loss, read the loss after it, and stop there; then keep the temperature. Measured across 196 classification datasets and a vision benchmark with ten runs per dataset. | Active |
| SOTA-316 | When you are memory-bound rather than time-bound, train all the parameters a block at a time, partitioned on layer boundaries | Liu et al. (2025), LIT-515 — freeze all but one block of layers, train it to local convergence, rotate. Memory falls to roughly the active block, so a 7B trains on one A800 where full-parameter needs two, and perplexity at 2B matches or beats the full-parameter baseline. The price is about three times the iterations, which the paper’s own table shows and its abstract does not. | Proposed |
| SOTA-317 | Do not rank singular directions by magnitude when deciding what to discard; check the small end, and check it after fine-tuning | Staats, Thamm and Rosenow (2024), LIT-517 — in a transformer’s non-square matrices, the smallest singular directions carry data directions and their removal is catastrophic: Llama-3 8B on GSM8K falls from 43.2% to 2.0% when the smallest decile of the Down-Projection goes, against 40.0% for the same decile of the square Attention-Output. And the damage only appears if you prune after fine-tuning. | Active |
| SOTA-318 | Choose the rank per matrix when you compress a transformer, because low-rank structure varies by component and by depth | Jaiswal et al. (2024), LIT-516 — “transformer weights are low rank” is true of some matrices and false of others. Query, Key and MLP Gate converge to low rank; MLP Up, MLP Down and Value do not; middle blocks resist while the first and last few give way. Setting the rank per matrix instead of globally is ~6.4× better in perplexity than uniform reduction at 30% on LLaMA-2 7B and ~47× at 40% on 13B. | Active |
| SOTA-319 | Reparameterize every linear layer by its spectral norm with a learned scalar | Zhai et al. (2023), LIT-523 — replace each linear layer’s W with γ·W/σ(W), γ a learnable scalar initialized to 1 and σ the spectral norm from power iteration. A ViT-B trained this way reaches 82.2% against a DeiT baseline’s 81.8% with no pre-LN, no warmup, no weight decay and LARS in place of Adam. Spectral normalization without the scalar gets 69.81%. | Proposed |
| SOTA-320 | Bound the learning rate by the ratio of the update’s spectral norm to the weight’s, and drop warmup | Qi et al. (2025), LIT-521 — if α_t > τ·σ₁(W_{t−1})/σ₁(∇W_t), truncate the step to that value; otherwise use the schedule. Weyl’s inequality makes this a direct cap on how fast a weight’s largest singular value can grow. ViT-B, ViT-L, GPT-S, Swin-S and Swin-B all train without any warmup and match or beat warmed-up AdamW. | Proposed |
| SOTA-322 | Trade attention heads for depth: more heads conditions the attention block, and the layers you drop cost nothing | Saratchandran, Teney and Lucey (2025), LIT-527 — more heads lower the condition number of the attention block, which is part of what depth was buying. In the five configurations where the MLP width is held fixed, raising head count and cutting layers holds or improves accuracy at 29–53% fewer parameters — TNT-B goes 65.4M → 30.9M at identical Top-1. | Proposed |
| SOTA-326 | Add n-gram embedding memory as one table read early in the network, on top of the expert budget rather than in place of experts | Qiu et al. (2026), LIT-152 — Qwen3.8-Flash-Next’s ablations. One n-gram embedding table at layer 2 takes loss from 1.585 to 1.541 and the benchmark average from 45.44 to 47.94. Splitting the same table across two layers buys nothing, and no depth clearly wins. Trading experts for table parameters at a fixed total lowers loss slightly and moves no downstream benchmark. Grown on top of the budget, the table keeps lowering loss after downstream accuracy has flattened. Layer 2 is a systems choice: the host-memory prefetch overlaps layer 1. | Proposed |
| SOTA-328 | Train one nested model instead of a family of separately trained sizes, and extract the sizes you need from it | Devvrit, Kudugunta, Kusupati et al. (2023), LIT-546 — nest four FFN widths in one Transformer, train one width per step, and extract any size in between by choosing widths per layer. Up to 850M parameters, at the compute of training the four sizes separately, the largest matches its baseline and the smaller ones beat theirs. It pays only if you were going to train the family. Gemma 3n ships it. | Proposed |
| SOTA-329 | For short-output tasks, adapt a frozen model with a low-rank intervention on a few positions of the residual stream, not a weight adapter; do not use it for long chain-of-thought | Wu, Arora et al. (2024), LIT-549 — LoReFT sets an r-dimensional subspace of the hidden state at a few prompt positions to a learned linear function of that state, with the model frozen. At 0.03% of parameters against LoRA’s 0.8%, it leads on commonsense QA (LLaMA-7B 80.2 against 74.7) and is level on GLUE. It loses on arithmetic chain of thought (GSM8K 26.0 against 37.5). All baseline numbers are copied from other papers. | Proposed |
| SOTA-332 | When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network | Sitzmann et al. (2020), LIT-551 — a sine network’s derivative is another sine network, so gradients and Laplacians of the fit are well behaved where a ReLU network’s second derivative is zero. Initialize hidden weights U(±√(6/n)) and scale the first layer by ω₀ = 30 (tune it to the signal), or deep sine networks do not train. It fits images, video, SDFs and PDE solutions from derivative supervision. | Proposed |
| SOTA-334 | In a style-modulated generator, demodulate the convolution weights instead of instance-normalizing the activations | Karras et al. (2019), LIT-560 — AdaIN’s instance normalization discards relative feature magnitudes, so the generator smuggles scale past it with a localized spike, the “water droplet” in every StyleGAN image. Scale each output channel’s modulated weights by 1/√(Σ w’² + ε) instead. It gives the same expected unit variance and style mixing still works. The artifact disappears and FID is unchanged (4.40 → 4.39). | Proposed |
| SOTA-335 | When generated content must move continuously, as in video or animation, build the generator alias-free: treat features as continuous signals and low-pass filter around every nonlinearity | Karras et al. (2021), LIT-559 — fine detail in GAN output sticks to pixel coordinates because the generator aliases. Apply each nonlinearity at 2× temporary resolution and low-pass filter back, start from Fourier features rather than a learned constant, remove per-pixel noise, and handle boundaries properly. On FFHQ-U, translation equivariance goes to 63 dB (and rotation to 40 dB for the -R variant) at StyleGAN2’s FID. | Proposed |
| SOTA-336 | Train high-resolution GANs at a fixed topology with output skips in the generator and a residual discriminator, not by progressive growing | Karras et al. (2019), LIT-560 — progressive growing makes each resolution briefly the output, so it learns output-level detail and leaves features stuck to pixel positions. A fixed generator that sums upsampled RGB outputs from every resolution, with a residual discriminator, keeps the coarse-to-fine emphasis without changing topology. FFHQ FID 4.34 → 3.31, and the phase artifacts are gone. | Proposed |
| SOTA-337 v4 | When an ImageNet classifier is anywhere in the pipeline — training the generator or steering its sampling — confirm FID gains with a Fréchet distance in a non-ImageNet feature space | Kynkäänniemi et al. (2022), LIT-563 — FID’s features are nearly ImageNet class logits, so matching ImageNet-class statistics lowers it without improving images. Resampling a fixed StyleGAN2’s outputs cuts FFHQ FID 5.30 → 1.78, while CLIP-space FD moves 2.76 → 2.64. A model whose discriminator uses ImageNet features exploited this by accident, and the authors suspect data filters and samplers can too. Report a CLIP or SwAV Fréchet distance beside FID whenever that is the case. Classifier guidance (LIT-699) is the sampling-loop instance and the least checked: it maximises an ImageNet classifier’s confidence on every step, and its paper denies producing adversarial examples in one clause without running a test. | Active |
| SOTA-343 | Read a transformer’s intermediate-layer predictions through a tuned lens, not the raw logit lens | Belrose et al. (2023), LIT-569 — decoding a hidden state with the final unembedding (the logit lens, LIT-570) fails outright on BLOOM, OPT and GPT-Neo, and is biased by 4–5 bits where it works. Train one affine translator per layer by KL to the final logits, initialized to the identity, and read the layer through that. It is cheap (d×d per layer), transfers to fine-tunes, and its directions are the model’s (Spearman 0.89). Train it with Muon. | Proposed |
| SOTA-345 | Probe language-model activations with logistic regression on the raw activations, and credit sparse-autoencoder probes only against that baseline under validation-based selection | Kantamneni et al. (2025), LIT-568 — across 113 datasets, adding SAE-latent probes to a toolkit of raw-activation probes, choosing per task by validation AUC, changes test AUC by −0.003. It does not help under data scarcity, class imbalance, label noise or shift. Reported SAE wins came from baselines denied the same pooling or selection. Use logistic regression on activations, and hold any SAE method to it. | Active |
| SOTA-348 | Evaluate a knowledge edit on the facts it implies — logical consequences, multi-hop compositions, aliases and the subject’s other facts — not only on the edited prompt, its paraphrases and unrelated neighbours | Cohen et al. (2023), LIT-577 — the standard edit metrics (efficacy, paraphrase, neighborhood) cannot see whether an edit changed what the model knows or only what it says to one question. On six ripple criteria, ROME, MEMIT and MEND average 38–66, and logical generalization falls as low as 5.5. Prompting with the new fact scores 81–83 on LLaMA-7B. Report ripple metrics beside the standard ones, and compare against in-context editing. | Active |
| SOTA-349 | Before crediting a graph transformer with long-range gains, retune the message-passing baselines under the same budget, including an MLP prediction head, positional encodings and input feature normalization | Tönshoff et al. (2023), LIT-579 — the Long-Range Graph Benchmark’s case that graph transformers are needed rested partly on untuned message-passing baselines. With a basic sweep inside the same 500k-parameter budget, GCN beats GPS on both Peptides tasks. Most of the gain is from replacing a linear prediction head with a 2-layer MLP. Normalizing input features lifts every model on the superpixel tasks, where GPS still leads. Tune both sides before attributing a gap to attention. | Active |
| SOTA-368 | When a model’s latent space is meant to be read by people, measure its interpretability with a human task such as word or topic intrusion, not by held-out likelihood | Chang et al. (2009), LIT-597 — topic models are used for their topics, but they are judged by held-out likelihood, which measures the probability of observations and ignores the representation. On NYT and Wikipedia, CTM had the best likelihood and the worst human scores. Test the latent space directly. Plant an improbable word among a topic’s top words, or an improbable topic among a document’s top topics, and measure how often people find it. | Active |
| SOTA-375 v2 | When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution | Mikolov et al. (2013), LIT-603 — sample negatives in proportion to count^(3/4). That flattens the Zipfian head so frequent words are not nearly the only negatives, while keeping rare words rare. The authors report it “outperformed significantly the unigram and the uniform distributions … on every task we tried”, but give no numbers. | Proposed |
| SOTA-377 | Stop scaling the contrastive batch past about 32k, because the benefit saturates there | Active | |
| SOTA-379 v2 | Before crediting a word-embedding method over count-based ones, give the baselines the same design choices and tune every method alike | Levy, Goldberg and Dagan (2015), LIT-607 — word2vec and GloVe ship with design choices (context smoothing, the negative-sample shift, dynamic windows, subsampling, adding context vectors) that count-based PPMI and SVD can use too. Port them and tune every method the same way before comparing. Done that way across 672 representations and 8 datasets, the prediction-over-count advantage disappears, SGNS beats GloVe on every task, and a single hyperparameter often matters more than the method. | Active |
| SOTA-383 | Fix the sample count before comparing FID values, and settle a close comparison with an unbiased estimator rather than a tighter error bar | Bińkowski et al. (2018), LIT-615 — the FID estimator is biased, the bias depends on the distribution being measured, and no unbiased estimator exists. So two FID numbers are comparable only at equal n, and a small standard deviation is not evidence of a converged estimate: at d=2048 and 50,000 samples their construction reversed a true ordering in 100 trials out of 100, with standard deviations of 0.2 and 0.5. | Active |
| SOTA-385 | Tighten the gradient-norm clip as the model grows rather than carrying one value across scales | Rae et al. (2021), LIT-617 — the Gopher family clipped the global gradient norm at 1.0 for models up to 1.4B and at 0.25 for the 7.1B and 280B models, “for improved stability”. Three held practices say to clip and none names a value; this is the first number in the record, and it is not a constant. | Proposed |
| SOTA-391 | Estimate attention resolution before training when choosing a position encoding, and measure it across the extrapolation boundary | Sun et al. (2022), LIT-638 — attention resolution scores how well an attention pattern distinguishes token distance, and is estimable from the encoding before a run. Measured at the training length and twice it: RoPE 0.91 → 0.08, ALiBi 0.81 → 0.88. The record has four practices about rotary encoding and held no measure of this. | Proposed |
| SOTA-393 | Freeze the image model’s spatial layers when adding time if checkpoint compatibility matters; do not expect a quality gain from it | Girdhar et al. (2023), LIT-635, and Guo et al. (2023), LIT-633. Freezing a text-to-image model’s spatial layers and training only temporal ones keeps the model a drop-in for checkpoints fine-tuned from the same base. Emu Video’s one controlled test finds freezing costs nothing and gains little (55.0 / 58.1 against fine-tuning). The reason to freeze is compatibility, not quality. | Proposed |
| SOTA-394 v3 | When distilling a causal few-step video generator, take the teacher bidirectional, not causal | Yin et al. (2024), LIT-631. A block-causal 4-step student distilled from a bidirectional teacher beats one distilled from a causal teacher, at the same initialization, data and step count (Table 4). The causal teacher is also the weaker model before distillation, so the result shows the arrangement works and does not isolate why. | Proposed |
| SOTA-395 | Post-train an autoregressive video generator on its own rollouts, and score the whole rollout | Huang et al. (2025), LIT-629. Training a causal video generator on context it generated itself, with a distribution-level loss over the whole clip, beats teacher-forced and diffusion-forced training under every objective tried (Table 2). It holds quality as the number of autoregressive steps grows, where they degrade. Evidence is VBench on 5s clips, and the exposure-bias mechanism is inferred, not measured. | Proposed |
| SOTA-396 | When a latent video model reuses an image autoencoder or upsampler, fine-tune them on video so they see time | Blattmann et al. (2023), LIT-621. An image autoencoder decodes each frame independently and flickers. Fine-tuning its decoder on video, with a video discriminator, cuts reconstruction FVD on three datasets: 390.88 to 32.94, 35.82 to 18.66, and 73.78 to 25.55. Making the upsampler temporal cuts FVD from 165.98 to 45.39. Reconstruction FID worsens on one dataset and improves on two. | Proposed |
| SOTA-397 v2 | To condition an unconditionally trained diffusion model on known frames, use reconstruction guidance, not replacement | Ho, Salimans et al. (2022), LIT-627. Extending a video autoregressively from a model trained without conditioning, reconstruction guidance (steering each denoising step toward consistency with the known frames) gives FVD 136 against 451 for replacing the known frames’ noisy latents (Table 6). The model and guidance weight are held fixed. | Proposed |
| SOTA-398 v2 | Pretrain a vision transformer on packed, aspect-preserved images at sampled resolutions, not on fixed square crops | Dehghani et al. (2023), LIT-657. Do not resize or crop images to one square resolution. Keep each image’s aspect ratio and sample its resolution per example, favouring small side lengths. Pack the patches into fixed-length sequences with per-example attention masks. At matched compute this beats square fixed-resolution ViT pretraining, and matches the best ViT with 4x less compute. Most of the gain is throughput. | Proposed |
| SOTA-399 | Choose pre-training weight decay, dropout, clipping and precision for the quantization you intend to ship | Proposed | |
| SOTA-401 | Anchor the patch Gram matrix to an early teacher when training long, because dense features decay while global metrics improve | Proposed | |
| SOTA-402 | To teach a model to reason over facts it holds, spend added data on facts derived from other facts rather than on more atomic facts | Wang et al. (2024), LIT-667. Two sweeps that separate what is normally varied together: with the inferred/atomic ratio φ fixed, scaling the training set changes nothing about how a transformer acquires a two-fact inference rule; with the size effect thus accounted for, raising φ moves the acquisition monotonically, and at φ = 18.0 the delay is gone — 96.7% before training accuracy has even saturated. Synthetic knowledge graphs only, and it buys rule application on facts already seen in compositions, not systematicity. | Proposed |
| SOTA-403 v3 | Share the attention parameters across layers if you need to cut parameters; do not share the feed-forward ones | Lan et al. (2019), LIT-668. A transformer block has two halves and they are not equally compressible across depth. On an ALBERT-base configuration at E = 128, sharing every layer’s attention parameters scores 81.7 average against 81.6 for no sharing while removing 89M parameters to 64M; sharing the feed-forward parameters instead scores 80.2. At E = 768 the same split reads −0.7 against −2.8. One paper, 2019, encoder-only, and nobody has re-run it. | Proposed |
| SOTA-405 | Scale the masked-language-modelling rate with model size rather than holding it at 15% | Wettig et al. (2022), LIT-672. The masking rate has been held at 15% “regardless of model sizes or masking strategies” since 2018, and the number was never swept. Swept, the optimum moves with capacity: 40% at 354M, 20% at 124M, 15% at 51M. At 354M, 40% wins seven of nine GLUE-plus-SQuAD tasks and reaches the 15% model’s QNLI and QQP scores in half the training time — under an efficient recipe, with the advantage shrinking on longer schedules. | Proposed |
| SOTA-406 | Replace every masked token with [MASK]; drop BERT’s 80-10-10 substitution rule | Wettig et al. (2022), LIT-672. BERT replaced 10% of selected positions with the original token and 10% with a random token, to reduce the pretrain/fine-tune mismatch caused by [MASK] never appearing downstream. Measured against an all-[MASK] baseline at a matched 40% rate, 80-10-10 is worse on everything but SST-2 — and the stated motivation does not bite, because “the model can adapt to full, uncorrupted sentences, regardless of the use of alternative corruption strategies in pre-training”. | Proposed |
| SOTA-411 | To let a model spend more compute at inference without more parameters, train a depth-recurrent core on a randomly sampled iteration count, re-injecting the input at every iteration | Geiping et al. (NeurIPS 2025), LIT-683. Put a small core block between a short prelude and coda. At each training step sample how many times to iterate it, from a heavy-tailed distribution with mean 32. Backpropagate through the last few iterations only. Feed the input embedding into every iteration, and start the state from noise. Trained this way, accuracy rises with iterations at test time and saturates per task. A model trained at fixed depth does not gain this. Nothing yet shows it beats a standard transformer of the same FLOPs. | Proposed |
| SOTA-412 | When the target is likelihood, learn a per-dimension noise schedule conditioned on a learned latent; a scalar or unconditioned one buys nothing | Sahoo et al. (NeurIPS 2024), LIT-677. For density estimation with a pixel-space diffusion model, replace the scalar learned schedule with a per-dimension one conditioned on a small discrete latent that an encoder infers from the image. Keep the endpoints fixed. VDM’s bound is reached in 5× fewer steps on CIFAR-10 and improved at equal steps. Each component alone does nothing: a scalar input-conditioned schedule and a time-only multivariate schedule both match VDM. | Proposed |
| SOTA-414 | To make an image diffusion model a dense geometric predictor, fine-tune it end to end as a one-step model at t = T with a task loss | Martin Garcia et al. (WACV 2025), LIT-687. Fix the timestep at T, feed zeros as the noise, decode the prediction through the frozen VAE, and train on the task’s own loss: affine-invariant L1 for depth, angular error for normals. From 20K iterations on 74K synthetic images, the one-step model beats Marigold’s 50-step, 10-member ensemble on depth AbsRel on all five zero-shot sets, and on normals by 2.6–3.0° of mean error. Starting from plain Stable Diffusion is nearly as good. | Proposed |
| SOTA-417 | Filter pretraining text by how well the tokenizer compresses it | Geiping and Goldstein (2023), LIT-681. Drop every document whose token count exceeds t times its character count. Text a tokenizer trained on the corpus compresses badly is mostly markup, code fragments and debris. No model is needed. On C4 it lifts GLUE from 75.9 to 79.3, and on already-clean bookcorpus-wikipedia from 78.1 to 78.7. The gain tracks how dirty the source was. | Proposed |
| SOTA-418 | To drop the tokenizer without losing to BPE, use a two-stage hierarchy with learned chunk boundaries and SSM encoder and decoder layers, not an isotropic byte model | Hwang, Wang and Gu (2025), LIT-679. A byte-level model that must not lose to a BPE Transformer at matched data and FLOPs should be hierarchical. Isotropic byte stacks lose by about 9 downstream points. It should use Mamba-2 layers for the encoder and decoder around the main network, and it should learn its chunk boundaries across two stages. One learned stage only ties BPE and whitespace chunking. Expect about 2× slower training. | Proposed |
| SOTA-419 | Under a fixed compute budget, change the architecture only where it cuts step time at constant parameter count | Geiping and Goldstein (2023), LIT-681. Under a 24-hour, one-GPU budget, every reshape tried (depth 4 to 24, deep-narrow, wide, funnel, FFN-every-k, shared layers) finishes within about 0.1 of the same MLM loss. A smaller model’s extra throughput cancels its slower per-token learning. So the architecture changes that pay are the ones that make each step cheaper without changing how much the model learns per token: dropping biases, predicting only masked tokens, and pre-norm for the learning rate it permits. | Proposed |
| SOTA-423 | When classifier-free guidance over-exposes a latent diffusion model, rescale the guided prediction toward the conditional prediction’s standard deviation | Lin et al. (WACV 2024), LIT-689. Compute x_cfg as usual, rescale it by std(x_pos)/std(x_cfg), and blend with weight φ back toward x_cfg. The source uses φ ≈ 0.7 at w = 7.5. It is the latent-space counterpart of Imagen’s dynamic thresholding, which needs a known pixel range. The evidence is qualitative, and part of the effect may simply be lower effective guidance. | Proposed |
| SOTA-426 | When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either | Lialin et al. (2023), LIT-104 — ReLoRA’s restart discipline, read as NOTE-024. At every merge-and-reinit, prune the adapter’s Adam state (99% by magnitude) and drop the learning rate to zero and re-warm it over 50–100 steps. In the 130M ablation (Table 6) the restart alone does nothing (34.25 vs LoRA’s 34.17), the re-warm without the reset does nothing (34.29), the reset without the re-warm diverges, and only the pair helps (29.77). A rule for how to restart, not a recommendation to restart. | Proposed |
| SOTA-427 | In a batch-normalized network, put dropout after the last batch-norm layer, not upstream of one | Li et al. (2018; CVPR 2019), LIT-705. BN freezes a moving variance accumulated while dropout was scaling activations. At test time dropout stops, the variance changes, and every later BN normalizes by the wrong constant. The cost is large: 77.42% → 68.55% on CIFAR-100 for DenseNet with dropout 0.5 in each bottleneck. The fix, one dropout layer after the last BN, is supported mainly as avoiding that cost. What it adds on its own is about 0.2 top-1 on ImageNet, with no spread reported. | Proposed |
| SOTA-432 | Maintain two power-function weight averages during training and snapshot them, so the averaging length is a post-hoc sweep instead of a guess you have to make before the run | Karras et al. (2023), LIT-720. Keep two power-function averages of the weights during training (σ_rel 0.05 and 0.10) and store both in each periodic snapshot; afterwards, reconstruct the average for any averaging length by a least-squares fit over the stored profiles, with error falling as O(1/n⁴) in the snapshot count. The averaging length stops being a pre-training commitment. Worth it because that commitment is expensive in a way that does not show up in its own column: with the length pinned at 13%, the learning-rate decay moves FID by up to 72% across t_ref ∈ [30k, 160k]; swept post hoc, the same bracket is within 10% of the optimum. | Superseded — by SOTA-428; A duplicate of SOTA-428, which was filed from LIT-714 hours earlier and is the same recommendation from the same paper. SOTA-428 is also the better document: it carries the architecture, learning-rate, guidance and metric dependences, where this one had only the first two. The Active/unreplicated pairing this note argued for is a judgement SOTA-428 declined, at Proposed, and that is the one that stands. |
| SOTA-433 | Guide a diffusion model with a smaller, less-trained copy of itself rather than an unconditional model, and make sure the two models are degraded in the same way | Karras et al. (2024), LIT-721. Replace classifier-free guidance’s unconditional reference with a degraded copy of the same conditional model — smaller and under-trained, same task, same conditioning, same data. The quality gain survives and the diversity loss does not: ImageNet-512 FID 2.56 → 1.34 (EDM2-S), 1.25 (XXL), 1.01 at ImageNet-64, and unconditional EDM2-S 11.67 → 3.86, which CFG cannot do at all. The binding condition is compatibility: degrade the two models differently and the method stops working entirely. | Proposed |
| SOTA-434 | In a vision backbone, carry position with a relative bias inside the attention window and do not add an absolute position embedding on top | Liu et al. (2021), LIT-723, Table 4. One architecture, one recipe, three tasks. A relative position bias inside the attention window is best or tied-best on all three. An absolute position embedding is worth +0.4 ImageNet top-1 and -0.2 COCO box AP and -0.6 ADE20K mIoU, and adding it on top of the relative bias costs 2.1 mIoU while changing ImageNet nothing. The recommendation is the sign, not the size: this is the one design choice in the table that a classification benchmark scores backwards. | Proposed |
| SOTA-435 v2 | Never rank attribution methods by a single faithfulness or stability score, and report the perturbation distribution when you report infidelity | Yeh et al. (2019), LIT-725. Max-sensitivity is minimised by a constant explanation — an explanation that ignores the input entirely wins it — so it can never be a quality score on its own. Infidelity is defined relative to a perturbation distribution you choose, and Propositions 2.2–2.5 show several published methods are each the optimum for some choice, which the paper confirms by giving SHAP the lowest infidelity under SHAP’s own perturbation. Both cautions are free to follow: report the pair, and name the perturbation. | Active |
| SOTA-437 | Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base | Hershcovitch et al. (2024), LIT-750 — ZipNN. In regular trained weights only the float exponent is redundant: about 40 of its 256 values occur and it codes to about a third, while sign and mantissa barely compress. So split the exponent into its own stream and Huffman-code it without an LZ stage. Llama-3.1 BF16 goes to 66.4% against Zstd’s 77.7%, and is faster in both directions. Between checkpoints, XOR deltas against a base up to 10 checkpoints back compress better than standalone files. Sizes and throughputs are measured. Stall time inside a training run is not. | Proposed |
| SOTA-438 | To pretrain a wider language model when a smaller pretrained one of the same depth exists, initialize it by function-preserving symmetric width expansion rather than at random | Samragh et al. (2024), LIT-765 — HyperCloning. Tile each weight matrix of the small model into n×n blocks scaled by 1/n, so the wide model computes exactly the small model’s function at step 0, then train as usual. On three 2× pairs from 350M to 2.9B, it reached the random-init baseline’s final 10-task accuracy 2.2–4× sooner and finished higher. The source model’s pretraining is not charged, so this supports reusing a checkpoint that already exists, not training a small model in order to clone it. | Proposed |
| SOTA-439 | Test a Kronecker-factored curvature implementation against the cases where KFAC is exact before trusting it | Dangel et al. (2025), LIT-768, §5.2 — a tutorial, so the evidence is a derivation, not a training run. KFAC’s averaging step is exact whenever the layer inputs or the backpropagated vectors do not depend on the data point. That gives two tests: a single data point, where KFAC equals the GGN and the empirical Fisher exactly and the Monte-Carlo flavour converges to the GGN; and a deep linear network under square loss on arbitrary data, where the GGN and Monte-Carlo identities still hold and the empirical-Fisher one does not. The second is the one the paper says catches scaling bugs. | Proposed |
| SOTA-440 | Supervise a feed-forward 3D reconstructor with a frozen camera-controlled video model’s renderings instead of captured multi-view data | Bahmani et al. (2025), LIT-743 — Lyra trains a 3D Gaussian Splatting decoder only on a frozen video model’s RGB decodings along six camera paths per image, with no captured multi-view data. In its one controlled arm, training on real RealEstate10K + DL3DV instead scores PSNR 19.08 against 24.77, and adding the real data to the generated data gives 24.74, no better. Both are scored against the teacher’s own renderings on its own prompts, which is the distribution the generated supervision was drawn from — so the arm shows the generated data is sufficient there, not that it beats captured data on real scenes. | Proposed |
| SOTA-441 | When a web-scraped pair corpus is noisy, filter it by self-consistency: keep only the pairs whose passage a model trained on the noisy set ranks near the top | Wang et al. (2022), LIT-744, Table 7. Train a dual encoder on the noisy pairs, rank each pair’s passage against 1M random passages, and keep the pair only if it lands in the top 2. On six BEIR datasets, filtered beat unfiltered at 1M pairs (40.7 against 34.9 nDCG@10) and with all pairs (51.6 against 50.0), though the filtered set is about a quarter the size. One cutoff, chosen by inspection and never varied, and the filtering model’s cost is not counted. | Proposed |
| SOTA-442 | To RL-tune a flow-matching generator, roll it out as a same-marginal SDE built from its own velocity, and anchor each step with the closed-form KL to the reference | Liu et al. (2025), LIT-779. A flow model’s ODE sampler is not a stochastic policy. Swapping it for an SDE with the same marginals, built from the trained velocity alone, makes each step an isotropic Gaussian, so GRPO’s ratio and the KL to the reference model are both closed form. On SD3.5-M, GenEval goes 0.63 → 0.95 either way. Without the KL term, DrawBench aesthetic falls to 4.93 from a base of 5.39, against 5.25 with it (Table 2). Quality is judged by reward models only, one run each. | Proposed |
| SOTA-445 | When the base text-to-image diffusion model is better than the models that generated the preference pairs, train on the pairs with Diffusion-DPO rather than fine-tuning on the preferred images | Wallace et al. (2023), LIT-774, §5.5 and Supp. S6. Pick-a-Pic’s images were generated by SDXL-beta and Dreamlike, and SDXL-1.0 is better than both. Supervised fine-tuning on the preferred images helped SD1.5 (55.5% win rate over its base) and made SDXL worse at every learning rate tried. Diffusion-DPO on the same pairs improved SDXL, preferred to the untuned base 70.0% of the time on PartiPrompts. The SDXL SFT arm has no number, there is one run per setting, and no one else has repeated it. | Proposed |
| SOTA-446 | In consistency training, take the target from the current weights under stop-gradient, not from an EMA teacher | Song and Dhariwal (2023), LIT-786, §3.2. Consistency Models trained the student against an EMA of itself. iCT shows that on a point-mass data distribution the limiting objective with any teacher other than the student carries no information about the data (Prop. 1), and that setting the teacher’s EMA rate to zero improves CIFAR-10 FID with both LPIPS and squared L2 (Fig. 2a, curves). Keep an EMA of the weights for evaluation. Do not use it for the target. | Proposed |
| SOTA-447 | When distilling a large image or video model by continuous-time consistency, add a small DMD distribution-matching term | Zheng et al. (2025), LIT-783. Pure sCM distillation of 2B–14B text-to-image and text-to-video teachers loses small text, texture and temporal coherence. Adding DMD’s loss at weight 0.01, on the student’s own 1–4-step rollouts with a fake-score network updated 5–10 times per step, fixes it in samples. Four-step Wan2.1-1.3B VBench is 84.43 against a re-implemented DMD2’s 84.56. sCM’s own Fig. 7 (LIT-790) is why the consistency term stays primary: distribution matching alone loses recall. | Proposed |
| SOTA-448 | Without a pretrained teacher, train a one-step generator as a MeanFlow average-velocity model rather than by consistency training or a shortcut model | Geng et al. (2025), LIT-784. Train a network for the average velocity over an interval, regressed onto v − (t − r)·du/dt under stop-gradient, with du/dt from a Jacobian-vector product. From scratch on ImageNet-256 with DiT-XL/2, one-step FID is 3.43 after 240 epochs, against 10.60 for a shortcut model and 34.24 for iCT as Inductive Moment Matching re-ran it (Table 2). Those rows are copied, at other budgets. No arm compares distilling a teacher. | Proposed |
| SOTA-449 | When reward-tuning a classifier-free-guided diffusion model over several rounds of sampling, train the guided prediction at the fixed sampling guidance weight, not the conditional prediction alone | Black et al. (2023), LIT-776, App. E.1 and Fig. 10. Reward fine-tuning should not train the unconditional branch, since the reward depends on the prompt. Training the conditional prediction alone made performance deteriorate after the first round of fine-tuning, which the authors attribute to the guidance weight becoming miscalibrated. Fixing the weight (5 in their runs) and training through the guided prediction used for sampling fixed it. One curve per arm, on reward-weighted regression and JPEG compressibility. | Proposed |