Practice statuses — What a status means for a PRACTICE: whether this record says to do the thing today. It is not the standing of the paper underneath — that is lit-statuses, and the two are allowed to disagree — and it is not what the field does, which is consensus (ADR-002).
Promising — early results are good, validation at scale is missing — the old experimental flag.
181 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-122 v2 | Attach learnable per-row and per-column multipliers to weight matrices so their norms are learned, not set by LR and WD | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Up to 20% relative gain on MMLU, BBH and GSM8K over a Muon baseline at 200 GT. | Proposed |
| SOTA-124 v5 | Repeat high-quality data freely when its epoch size exceeds the model’s memorization window | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Roughly 100–500 GT for a 7B model, scaling linearly; the authors call the understanding early. | Proposed |
| SOTA-125 v3 | At a fixed tiny parameter budget, spend parameters on depth and SSM state width before MLP width | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. 24–27 layers × 512 hidden and a large SSM state dimension won every ablation at 90M; 50 layers gained MMLU but halved throughput. | Proposed |
| SOTA-130 v4 | Skip the reasoning SFT stage and run RL with verifiable rewards directly on the base model | Olmo Team (2025), LIT-130 — the RL-Zero track: RLVR on the base model with no reasoning SFT in between, released as an open benchmark for studying RL rather than as the recommended recipe; a variation on SOTA-129 that the record keeps so the family of reasoning recipes can be traced. | Proposed |
| SOTA-136 v4 | Widen the residual stream into several streams mixed by doubly-stochastic matrices (manifold-constrained hyper-connections) | Xie et al. (2025), LIT-140 — hyper-connections with the residual-mixing matrix constrained to be doubly stochastic, so identity mapping survives; 6.7% training overhead at n = 4 in large-scale training, shipped in DeepSeek-V4. | Proposed |
| SOTA-141 v3 | Decay the learning rate linearly all the way to zero | Bergsma et al. (2025), LIT-147 — with the peak tuned, linear decay to zero beats the customary decay to 10% and other shapes at compute-optimal budgets, more so past them; Proposed because contemporary production runs in the record still decay to a floor. | Proposed |
| SOTA-144 v2 | Extend µP’s transfer to depth with CompleteP so one sweep serves deeper models too | Dey et al. (2025), LIT-150 — the depth exponent α = 1 transfers the optimal learning rate across depth and keeps deep layers learning; 11.8% fewer FLOPs than µP at optimal shapes, 34.4% at 179 layers. One group so far. | Proposed |
| SOTA-146 | Correct the GRPO objective before running it — the published form has three independently identified defects | Liu et al. (2025), LIT-167; Yu et al. (2025), LIT-168; Zheng et al. (2025), LIT-180 — three groups, three defects in GRPO’s objective, three published fixes, none of them the same fix. “We ran GRPO” does not say which of these you ran. | Proposed |
| SOTA-148 v3 | Balance mixture-of-experts load with a bias on the routing scores, not an auxiliary loss | Wang et al. (2024), LIT-171 — add a per-expert bias to the routing scores before the top-K decision and update it from that expert’s recent load, so balancing changes which experts are chosen without adding a gradient to the loss. Better balance and better quality than an auxiliary-loss control, and what DeepSeek-V3 runs at 671B. | Proposed |
| SOTA-152 | Pack training documents by best fit instead of concatenating and splitting | Ding et al. (2024), LIT-203 — concatenate-then-split truncates documents that would have fit, and the model then learns to continue text whose beginning it never saw. Treating the grouping as bin packing removes those truncations at the same efficiency, no padding added: +4.7% reading comprehension, +16.8% context following, +9.2% program synthesis, and up to 58.3% less closed-domain hallucination. | Proposed |
| SOTA-155 | Train across poorly connected islands: many inner steps per worker, an outer momentum optimizer over the deltas, and a streamed synchronisation | Douillard et al. (2023, 2025), LIT-212 and LIT-214 — federated averaging with the constants pushed hard: many inner AdamW steps per worker, Nesterov momentum as the outer optimizer over the accumulated deltas, synchronised rarely and in streamed subsets. 500× less communication at matched quality, then two more orders off the peak. Filed Proposed: one group, and no frontier report trains this way. | Proposed |
| SOTA-156 v2 | Train without a learning-rate schedule: average the iterates so no stopping time need be fixed | Defazio et al. (2024), LIT-213 — scheduling and iterate averaging are one mechanism, and the averaged form needs no stopping time T, no schedule shape and no hyperparameter beyond what AdamW already has. Filed Proposed: the evidence is a third-party competitive benchmark at workloads far below the scale where this record’s schedule question is open. | Proposed |
| SOTA-157 v3 | Train the language model as a masked diffusion model rather than autoregressively | Nie et al. (2025), LIT-217 — hold the paradigm fixed (pretrain, then SFT) and swap only the factorization: a forward masking process and a reverse process predicting masked tokens, optimizing a likelihood lower bound. Competitive with LLaMA3 8B on in-context learning; past GPT-4o on reversal poem completion. Filed Proposed — one group, one model, 8B. | Proposed |
| SOTA-158 | Bound the activation’s output range when training in low precision | Jiang et al. (2026), LIT-200 — SwiGLU approximates x² for large positive inputs, which is where its expressive capacity comes from and what enlarges the output range and exacerbates outliers. Two groups independently concluded that is intolerable in low precision. What they agree on is the constraint; the functions they chose differ. | Proposed |
| SOTA-159 | Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling | Blake et al. (2024), LIT-149 — µP makes activation scale independent of model size; unit scaling makes activations, weights and gradients start at a scale of one. Together the model is size-independent and well-scaled from the first step, the hyperparameters become close to independent so a one-dimensional sweep replaces a joint search, and FP8 works without loss-scaling tricks. | Proposed |
| SOTA-160 | Treat the pretraining token budget and the post-training quantization plan as one decision, not two | Kumar et al. (2024), LIT-186 — degradation from post-training quantization increases with how much data the model was trained on, to the point where additional pretraining data becomes actively harmful if the model will be quantized afterwards. Fit over 465 runs into one precision-aware scaling law covering both pre- and post-training quantization. | Proposed |
| SOTA-166 v2 | Set the pretraining data proportions by fitting a mixing law on small runs, not by argument | Ye et al. (2024), LIT-201 — model performance is quantitatively predictable as a function of the mixture proportions, so fitting the law on a sample of mixtures reads off performance on mixtures never trained. Nested with the existing scaling laws, small runs predict a large model. Matched the default mixture trained for 48% more steps. | Proposed |
| SOTA-167 | Build linear-attention layers from the state-space view: expressive discretisation, complex-valued state, and a MIMO readout | Lahoti et al. (2026), LIT-165 — three changes derived from the SSM view: a more expressive recurrence from the discretisation, a complex-valued state update for state tracking, and a MIMO formulation that adds quality without decode latency. +1.8 downstream at 1.5B over Gated DeltaNet, and Mamba-2’s perplexity at half the state size. | Proposed |
| SOTA-168 | Run Adam in Shampoo’s eigenbasis (SOAP) instead of Shampoo itself | Vyas et al. (2024), LIT-157 — Shampoo at the 1/2 power is Adafactor in the eigenbasis of Shampoo’s preconditioner, so run the better optimizer in that basis. SOAP is Adam there, adding exactly one hyperparameter over Adam. Over 40% fewer iterations and 35% less wall-clock than AdamW at 360M–660M in the large-batch regime. | Proposed |
| SOTA-169 | Widen the residual stream into several streams and constrain the mixing between them | Zhu et al. (2024) and three successors, LIT-141 / LIT-140 / LIT-151 / LIT-181 — replace the single residual stream with n parallel streams and learn the mixing. All four agree the width helps and that the mixing needs a constraint; they disagree about which one, and that disagreement is the two practices extending this rather than this itself. | Proposed |
| SOTA-172 v2 | Build synthetic pretraining data by editing human text at the token level, not by generating from scratch | Zhu et al. (2024), LIT-205 — pretraining across proportions of synthetic data shows a negative correlation between the proportion and performance, caused by distributional shift and over-concentration of n-gram features. The remedy inverts the usual move: edit human data at the token level, so the result stays anchored to a real distribution and the test error is provably bounded. | Proposed |
| SOTA-173 v2 | Augment the objective to make multi-epoch pretraining productive on a fixed corpus | Chen et al. (2026), LIT-175 — standard autoregressive pretraining overfits severely on a fixed corpus, reaching its optimum early and deteriorating continuously. Three orthogonal augmentation families each delay that and lower validation loss; combining them lowers the minimum further. The claim is hundreds of productive epochs on the same data. | Proposed |
| SOTA-175 | Mask whole syntactic units for code fill-in-the-middle, not random character spans | Gong et al. (2025), LIT-126 — standard FIM masks random character spans, which produces training examples that rarely correspond to an edit anyone makes. AST-FIM masks whole syntactic units at scale, so the middle span is a coherent structure. Up to 5 points over random-character FIM at 1B and 8B, and most useful on real editing. | Proposed |
| SOTA-176 | Hybridise inside the layer: long-term slots, a sliding window, and one softmax over both, with the window size as the only dial | Du et al. (2025), LIT-176 — long-term context in key-value slots updated by a linear RNN, short-term context from a sliding window, and a single softmax attending over both together, so the long/short weighting is per-token, per-head and content-dependent with no fusion parameters. Every layer stays structurally identical and one hyperparameter slides from purely linear to full attention. | Proposed |
| SOTA-177 v2 | Decouple the erase address from the write address in a delta-rule recurrence | Li et al. (2026), LIT-177 — the delta rule corrects what is stored at the current write address before writing there, so stale information at a different address can only decay passively. EDA runs a targeted erase along a learned erase direction first, then the ordinary corrective write. Best at both 2.5B dense and 25B-A2.8B MoE, and the gain persists after 80B tokens of long-context midtraining. | Proposed |
| SOTA-178 | Give the recurrence vector-valued gating and in-context learning rates so it can track state a softmax layer provably cannot | Peng et al. (2025), LIT-173 — a generalised delta rule with vector-valued gating and in-context learning rates, at constant memory and constant time per token. A 2.9B model sets the 3B state of the art on multilingual tasks on dramatically fewer training tokens. The expressivity claim is the sharp one: it can recognise all regular languages, which under standard conjectures exceeds what a softmax stack in TC⁰ can do. | Proposed |
| SOTA-179 | Truncate the rotary encoding’s low frequencies rather than rescaling its base | Barbero et al. (2024), LIT-210 — RoPE’s high frequencies build positional attention heads and its low frequencies carry semantics that provably cannot stay robust over long context. Keeping a fraction p of the frequencies holds performance and at 2B improves it; p=1 is RoPE and p=0 is NoPE, so the practice is a dial between two things the record already holds. | Proposed |
| SOTA-189 | Set the checkpoint interval from the job’s measured time-to-failure, which shrinks as the job grows | CoreWeave (2025), LIT-222. Expected work lost to a failure is half the inter-checkpoint interval, and mean time to failure falls linearly with GPU count — so the interval belongs to the job’s size and the cluster’s failure rate, not to the write cost alone. | Proposed |
| SOTA-190 v3 | Increase depth before any other dimension when scaling a transformer | Tay et al. (2021), LIT-052 — the DeepNarrow strategy. Small 16L matches T5-Base downstream at 60% of the parameters, 63% of the FLOPs and 40% faster; the limit is parallelism rather than quality. | Proposed |
| SOTA-191 v2 | Consider removing LayerNorm’s learnable gain and bias rather than tuning them | Xu et al. (2019), LIT-025 — the bias and gain increase overfitting risk and “do not work in most cases”; LayerNorm-simple, with both removed, beats LayerNorm on four datasets and reaches state of the art on En-Vi. | Proposed |
| SOTA-193 | Lower Adam’s second-moment decay when the loss spikes, before reaching for the learning rate | Proposed | |
| SOTA-204 v2 | Anneal a discretisation from coarse to fine over training rather than fixing it | Song et al. (2023), LIT-093. Where a training loss approximates a continuous target through a step count, that count is a bias/variance dial: few steps give a biased but low-variance target early, many steps a faithful but noisy one later. | Proposed |
| SOTA-209 | Initialize a mixture-of-experts model from a dense checkpoint rather than training it from scratch | Komatsuzaki et al. (2022), LIT-227 — seed the experts from a dense checkpoint you already paid for. Upcycled T5 and ViT models beat their dense counterparts at ~50% of the dense pretraining sunk cost, and beat sparse models trained from scratch on 100% of it: same architecture, same compute, different initialization. | Proposed |
| SOTA-211 | Spend one fitness evaluation per evolution-strategies direction, not an antithetic pair | Kaya and Hashemi (2026), LIT-232, and Ba et al. (2026), LIT-230 — the antithetic pair’s variance reduction depends on both evaluations sharing randomness, which autoregressive regeneration breaks. Replace the pair with a leave-one-out baseline computed from the population mean the score centering already needs: same expected update, one evaluation per direction, twice as many directions per budget, and half the estimator MSE in transformer blocks. | Proposed |
| SOTA-212 | Post-train by scoring many random weight perturbations in one parallel pass and majority-voting the best of them | Qiu et al. introduced it (LIT-211); Gan and Isola (2026), LIT-233 — ARXIV-2603.12228 — measured this form of it. Sample N Gaussian weight perturbations, score them on a few hundred held-out examples, keep the top K and majority-vote at inference. At equal training FLOPs it matches or beats PPO, GRPO and ES across seven tasks at 0.5B–8B, in one parallel step instead of hundreds of sequential ones — 3.2 minutes for OLMo3-7B on Countdown across 200 GH200s. Filed Proposed: one group, K forward passes per answer, and discrete answers only. | Proposed |
| SOTA-213 | Control evolution-strategies drift with a larger population or an anchor penalty, not by stopping training early | Schweighofer et al. and Liang et al. (2026), LIT-238 and LIT-236, on the scaling Hoy et al. derived in LIT-235 — ES drift is a random walk whose size falls with population size, so raising the population from 30 to 128 halves the update norm and monotonically reduces prior-task degradation. Anchored Weight Decay buys the same reduction at population 30 for 1-2% runtime. Do not stop early instead: the prior-task dip is often transient and recovers by the end of training. | Proposed |
| SOTA-215 | Freeze Adam’s variance term after a warmup before compressing its update, and never compress the update directly | Error-compensated compression assumes the residual it carries forward still means the same thing next step. Adam’s update divides by a running variance, which is a non-linearity the correction does not survive — compressing it directly corrupts the error cancellation and badly harms convergence. The workaround is structural rather than a tuning fix: Adam’s variance stabilises early, so freeze it after a warmup and the update becomes linear enough to compress. | Proposed |
| SOTA-216 | Switch from minibatch SGD to local SGD at the first learning-rate decay, rather than choosing between them | The two options are usually posed as a choice: minibatch SGD generalises and communicates every step, local SGD communicates rarely and generalises worse. Post-local SGD is the observation that they are wanted at different times — run minibatch SGD until the first learning-rate decay, then switch to local SGD with H around 16 to 32 for the rest. The early phase is where the gradient noise is doing work; the late phase is where communication is pure overhead. | Proposed |
| SOTA-217 v4 | Align the hidden-unit permutation before averaging weights from separately trained networks | A neural network’s hidden units can be permuted without changing the function, so two networks trained from different seeds sit in different corners of the same symmetry orbit. Averaging their weights directly averages across that mismatch and traverses a loss barrier that is mostly an artefact of labelling. Match the units first — by weight matching, which needs no data and takes seconds to minutes — and for wide enough networks the barrier is gone. At standard width, and on ImageNet, it shrinks but does not vanish. | Proposed |
| SOTA-220 | Aggregate worker gradients coordinate-wise by median when any worker may be faulty | Averaging has a breakdown point of zero: one worker returning an arbitrary vector can move the mean anywhere, and adding workers does not help, because it is the worker’s magnitude and not its share of the population that does the moving. The coordinate-wise median is the robust aggregator that needs no estimate of how many workers are faulty; the trimmed mean has better rates and needs one. | Proposed |
| SOTA-222 v2 | Overlap the synchronisation with one or two local steps instead of communicating every step | Wang et al. (2020), LIT-323. Local SGD is usually argued for as a way to communicate rarely, which costs accuracy at large intervals. The cheaper reading is that it does not take a large interval to win: at tau = 2 the communication-to-computation ratio falls from 34.6% to 1.5% with convergence per iteration roughly unchanged. The interval is not buying infrequency, it is buying somewhere to hide the exchange. | Proposed |
| SOTA-223 | Do not keep the gossip topology static: sample a fresh random neighbourhood every round | de Vos et al. (2023), LIT-315. Decentralized training has spent years choosing good fixed graphs. Epidemic Learning does not choose one: each round, every node sends its update to s peers picked at random. The transient-iteration count falls from the best-known O(n^3) to O(n^3/s^2), and at 96 nodes on CIFAR-10 it converges 1.7x faster and 2.2% more accurately than a static 7-regular graph at equal communication volume. | Proposed |
| SOTA-229 | Scale the draft model’s training data, once nothing constrains it to predict the target’s features | Draft models had stopped improving with more training data, and LIT-185 identifies why: EAGLE trained its draft to predict the target’s hidden features and read tokens off them, which supplies multi-step training signal and caps expressiveness. Remove the constraint — predict tokens directly, fuse features from several depths, and simulate the multi-step draft during training — and the speedup rises with the data instead of flattening. Up to 6.5x, about 1.4x over the previous version, on roughly 8x the data. | Proposed |
| SOTA-232 | Convert an autoregressive model into a diffusion model by continual pretraining rather than training one from scratch | Gong et al. (2024), LIT-381 — ARXIV-2410.17891. Continual pretraining on under 200B tokens turns an existing autoregressive model into a diffusion language model — GPT2 and LLaMA across 127M-7B — competitive with the AR original. The reason to want one is infilling without prompt re-ordering, not perplexity. | Proposed |
| SOTA-234 | Train in the target low-bit format from scratch rather than quantizing a finished model | Ma et al. (2024), LIT-380 — ARXIV-2402.17764. Decide the serving format before training and train in it, rather than training in FP16 and compressing afterwards. Ternary weights trained from scratch match an FP16 model of equal size and token budget from 3B upward — but the compute argument assumes hardware built for the format, and only the memory saving is measured on machines that exist. | Proposed |
| SOTA-239 | Order training data by skill prerequisite: teach the prerequisite and the dependent skill costs less data | Chen et al. (2023), LIT-392 — ARXIV-2307.14430. Training data has a prerequisite structure, and it is exploitable: train on a prerequisite skill and the dependent skill is reachable with less data. Sample over skill mixtures online rather than fixing proportions in advance. | Proposed |
| SOTA-241 | Rank candidate data selections with a model-free distributional proxy before spending a training run | Xie et al. (2023), LIT-087 — KL reduction in a hashed n-gram space correlates with downstream performance across selection methods, including manual curation, which uses no n-grams at all. A selection can therefore be ranked against its alternatives before anything is trained on it. | Proposed |
| SOTA-242 | Select pretraining data by matching a target distribution you can sample from, not by scoring it for quality | Xie et al. (2023), LIT-087 — data selection needs a description of what you want, not a criterion for what is good. Supply a small sample of the target text and resample the raw corpus to match it. The target is an artifact you can produce; a quality rubric is an argument you have to win. | Proposed |
| SOTA-243 v2 | Discard the easy examples when data is abundant and the hard ones when it is scarce | Sorscher et al. (2022), LIT-399 — power-law scaling in dataset size is evidence of redundancy, and given a difficulty ranking it can be beaten toward exponential. But which end of the ranking to discard flips with how much data you started with, and the fraction to discard has to grow with the corpus. | Proposed |
| SOTA-244 | Score example importance a few epochs into training, averaged over several initializations | Paul et al. (2021), LIT-397 — the importance ranking is available from a single early checkpoint, not only from a finished run. EL2N at epoch 20, averaged over ten initializations, prunes half of CIFAR-10 with accuracy slightly up, and the ranking transfers to other architectures. | Proposed |
| SOTA-245 | State a relation in both orders in the training corpus if you want it usable in both directions | Grosse et al. (2023), LIT-401 — a training sequence influences a completion only when the phrase related to the prompt comes before the phrase related to the completion. Identical content in the reverse order scores barely above an unrelated baseline. | Proposed |
| SOTA-246 | Find mislabelled training data by self-influence, not by training loss | Koh and Liang (2017), LIT-403, with LIT-400 — rank training points by their influence on their own loss and inspect from the top. Mislabelled examples are strong proponents of themselves, so they sort to the front; high training loss does not separate them nearly as well. | Proposed |
| SOTA-247 | Add a multi-token word to a frozen model’s vocabulary from the model’s own detokenized representation of it | Kaplan et al. (2024), LIT-412 — a model already computes a whole-word representation for a word its tokenizer splits, so initialize the new embedding and unembedding from that rather than from an average of the word’s token embeddings. On Llama2-7B the frozen model then uses the new tokens while keeping its accuracy on existing ones (0.519 against 0.522 on WikiText-103), where mean-embedding initialization both fails to use them and degrades the model. The gain is largest where the tokenizer is worst: Arabic Wiki40B, 0.402 new-token accuracy against 0.117. | Proposed |
| SOTA-248 | Stretch and clip the softmax so an attention head can output exact zeros | Bondarenko et al. (2023), LIT-414 — a head that wants to do nothing has to drive its softmax input to infinity to approximate exact zeros, and that is what creates the activation outliers that break INT8. Stretch the softmax to (γ, ζ) and clip back to (0,1) and zeros become reachable from a finite input. On BERT-base: W8A8 perplexity 1294 → 4.55, max infinity-norm 735 → 20, and the FP16 model gets slightly better rather than worse. Evidence is BERT-base, OPT-125M and ViT-S/16 — small and encoder-heavy. | Proposed |
| SOTA-251 | Train at low resolution and raise it only during the decay phase of a warmup-stable-decay schedule | Assran et al. (2025), LIT-215 — ARXIV-2506.09985. Spend warmup and the constant phase on short, low-resolution clips; raise resolution and clip length only in the final decay. Up to 8x less pretraining compute than training at full resolution throughout, for the same end state. | Proposed |
| SOTA-254 v2 | Train the language model as a masked diffusion model when the corpus is fixed and the compute is not | Prabhudesai et al. (2025), LIT-442 — masked diffusion keeps extracting signal from a repeated corpus for roughly 512 epochs where autoregressive training stops at 32. Below a critical compute threshold, given in closed form from the unique-token count, autoregressive training is better and by a wide margin; above it, diffusion is better. The recommendation is the condition, not the objective. | Proposed |
| SOTA-255 v2 | Tune weight decay upward when pretraining over a repeated corpus, rather than inheriting the customary value | Kim et al. (2025), LIT-441 — with the corpus fixed and parameters past Chinchilla-optimal, the weight decay of 0.1 everyone inherits from GPT-3 is far too small: loss turns upward as parameters or epochs are added. Tuned jointly with learning rate and epoch count, the optimum is roughly 30x larger, and loss becomes monotone in parameter count with a steeper exponent than Chinchilla’s. | Proposed |
| SOTA-256 | Judge a monotone scaling recipe by its asymptote, not by its loss at a compute budget | Kim et al. (2025), LIT-441 — when compute is not the binding constraint, a recipe’s loss at a chosen budget answers a question nobody asked. Fit the scaling law and compare the asymptote instead. The catch is that an asymptote is an extrapolation, and rankings at small scale can invert: the ensembling recipe wins on asymptote while losing at small member counts. | Proposed |
| SOTA-257 v2 | Spend surplus compute on an ensemble of independently seeded models and distil it, rather than on one larger model | Kim et al. (2025), LIT-441 — under a fixed corpus, training K models that differ only in seed and averaging their logits reaches a lower loss asymptote than scaling one model to the same total parameter count. Two 300M models beat one 600M. Members want more epochs and less weight decay than a standalone model, and an 8-member ensemble distils into a 300M student retaining about 83% of the gain. | Proposed |
| SOTA-259 | Hold Adam’s second-moment half-life fixed in tokens when the batch size changes, not beta_2 | Marek et al. (2025), LIT-444 — beta_2 is a decay per optimizer step, and a step is not a fixed amount of data. Changing batch size from B to B' while holding beta_2 fixed changes the second moment’s averaging window in tokens by the same factor. Hold the half-life fixed instead: beta_2' = beta_2^(B'/B). beta_1’s default is fine. | Proposed |
| SOTA-260 | Use the smallest batch size that still saturates the device, and do not gradient-accumulate to avoid it | Marek et al. (2025), LIT-444 — batch size one trains stably once Adam’s second-moment half-life is held fixed in tokens, matches or beats large batches per FLOP, and is far more robust to hyperparameter misspecification. Gradient accumulation costs memory for the accumulated gradient and forgoes optimizer steps, and is worth it only when multiple device replicas are bottlenecked on interconnect bandwidth. | Proposed |
| SOTA-261 | Set AdamW’s weight decay by targeting a timescale that follows a power law in tokens-per-parameter | Bergsma et al. (2025), LIT-443 — under AdamW the weights are an EMA of updates with timescale 1/(eta*lambda), and the fraction of training it averages, tau = 1/(eta*lambda*S), is the quantity that scales. It follows a power law in tokens-per-parameter with exponent about -0.52 across three orders of magnitude of compute, so weight decay can be read off in advance rather than swept. | Proposed |
| SOTA-264 v2 | Fix the noise schedule’s endpoints against the bound and choose its shape to minimise the loss estimator’s variance | Kingma et al. (2021), LIT-446 — in continuous time the bound depends on the schedule only through its endpoints, so the shape is free. Spend it: optimise the endpoints against the VLB and the shape against the variance of the loss estimator. Same bound, faster optimisation, and a low-discrepancy sampler for the time variable cuts the variance further. | Proposed |
| SOTA-267 v3 | Warm up on a subset of the entities before training on all of them, rather than holding one distribution fixed throughout | Zucchet et al. (2025), LIT-450 — the paper argues two quantities move in opposite directions: plateau length is governed by how often the most common entities appear, acquisition speed after the plateau by how often the least common ones do. So concentration buys an early exit from the plateau and costs the tail, and a warm-up that trains on a subset of the entities first, then on all of them, beat the best fixed distribution tried, in single-seed runs on synthetic biographies. | Proposed |
| SOTA-268 | Do not transfer a data mixing ratio across model scales; the threshold where a domain starts being learned moves with size | Gu et al. (2025), LIT-451 — knowledge acquisition from a knowledge-dense dataset mixed into web text has thresholds, not a smooth curve. Below a critical model size, or a critical mixing ratio, the model memorises almost nothing however long it trains; past it, accuracy jumps to over 60%. The critical ratio follows a power law in model size, so a ratio chosen at one scale can be on the wrong side of a transition at another. | Proposed |
| SOTA-271 | Train on a second modality even when the target is single-modality | Huh et al. (2024), LIT-458 — if representations converge on a modality-agnostic model of what generated the data, then data from any modality helps find it. The vision direction is already common practice; the language direction is not, and the paper cites a case where adding image data improved text performance. | Proposed |
| SOTA-272 | Expect the loss to be non-monotone at the step size that trains fastest, and do not set the step size from a curvature bound | Cohen et al. (2021), LIT-461 — gradient descent raises the sharpness until its own step size cannot tolerate more, so a rule that anneals the step size to the measured curvature is chasing a number it is producing, and loses to the fixed step it forbids. Demonstrated full-batch. | Proposed |
| SOTA-273 | Choose a pretraining context whose association with the input is neither too strong nor too weak, and mix contexts to get there | Zhai (2025), LIT-459 — a pretraining objective is an association between the input and a context variable, and the useful contexts are the ones whose association is neither too strong nor too weak: the singular values of the expectation operator should decay neither too fast nor too slowly. Contexts that individually miss the range can be mixed into one that does not. | Proposed |
| SOTA-274 v2 | Before adopting a spectral optimizer, measure the stable rank of each block’s incoming activations against its gradient’s nuclear rank | Davis and Drusvyatskiy (2025), LIT-457 — the spectral step’s one-step guarantee beats the Euclidean one by the ratio of gradient nuclear rank to incoming activation stable rank. Both are cheap to compute from a run you are already doing, and the paper never uses them to route. | Proposed |
| SOTA-275 | Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule | Zheng et al. (2026), LIT-462 — Depth-muP’s α = Θ(1/√L) is the rule for a one-transformation residual branch. Attention and FFN branches hold more than one, so Transformers need CompleteP’s α = Θ(1/L), and the looser rule measurably fails to transfer on them. | Proposed |
| SOTA-276 | For a normalized or preconditioned optimizer, width-depth muP is width muP plus a hidden residual multiplier of order 1/L | Zheng et al. (2026), LIT-462 — a normalized or preconditioned update has a norm that does not depend on the residual multiplier, so the depth factor the raw gradient carries is removed and the optimizer’s own muP rule is unchanged. SGD is the exception and needs more. | Proposed |
| SOTA-277 | For a task that is a sequential composition, buy depth rather than width | Chen et al. (2024), LIT-464 — composing k functions costs a constant-depth decoder polynomially many parameters and a log k-layer one polylogarithmically. An exponential separation, proved unconditionally, and about what a shape can express rather than what a run will learn. | Proposed |
| SOTA-282 | Put every matrix and hidden state on the unit hypersphere, and make each block’s residual step size a learned per-dimension parameter | Proposed | |
| SOTA-283 | To fine-tune a model that is already quantized, bank the part of each update that is smaller than the lattice spacing, and rebuild the accumulator from seeds rather than storing it | Proposed | |
| SOTA-284 v2 | Choose the checkpoint you post-train from by how much mass it puts on good rare responses, not by validation cross-entropy | Proposed | |
| SOTA-285 | Put dropout at the start of training if the model underfits, and at the end if it overfits — the schedule decides the sign, not the rate | Proposed | |
| SOTA-286 | To find a model that generalizes badly where its outputs look fine, measure how much of parameter space behaves the way it does | Proposed | |
| SOTA-287 | If you need to train without backpropagation, train each block to denoise a noisy label embedding conditioned on the raw input | Proposed | |
| SOTA-288 | Late in training, replicate a small subset of the weights, train the copies against shared base weights, and average the subset back into one model | Proposed | |
| SOTA-289 v2 | When the sampling budget is small, prefer uniform-state discrete diffusion with consistency distillation — masked diffusion cannot revise what it has already emitted | Proposed | |
| SOTA-290 | To measure what a deployed recommender does, use sock-puppets that never interact, follow sets curated to your question, and exposure weighted by rank | Proposed | |
| SOTA-292 | Check industry ties yourself when reading a literature about deployed platforms — disclosure is unreliable and the ties are mechanically detectable | Proposed | |
| SOTA-293 | Count embedding parameters when you report a tiny model’s size | Finke et al. (2025), LIT-485 — at these sizes the embedding table is not a rounding error, it is most of the model: TinyStories-33M is 68M parameters all-in, and an 8M model in the same paper is “more than half embeddings”. Two papers can each claim the smaller model and both be telling the truth. Print the all-in figure. | Proposed |
| SOTA-294 | Fit the tokenizer to the corpus when the corpus is deliberately narrow | Finke et al. (2025), LIT-485 — holding the dataset and the architecture fixed and changing only the tokenizer, a 4,096-token WordPiece vocabulary seeded with English affixes beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality. It is the one fully controlled comparison in a paper that is mostly about something else, and the effect is larger than the architecture change measured beside it. | Proposed |
| SOTA-295 | Parameterize the generating prompt above the word list, and constrain how each sample opens | Finke et al. (2025), LIT-485 — requiring a story to contain three words drawn from a vocabulary constrains the lexicon and leaves the frame alone, and the frame is where a generator repeats itself: 59.38% of TinyStories begins “once upon a time”. Name a theme, a topic, a style and a narrative feature per sample, and require the sample to open with a given part of speech and letter. Five diversity metrics move; measured simplicity does not. | Proposed |
| SOTA-296 | Add the parallel streams in a short final training stage, not from the start | Chen et al. (2025), LIT-486 — parallel scaling costs P× the training FLOPs, which is what would otherwise make it unaffordable. Train 1T tokens the ordinary way, then switch the streams on for 20B tokens, 2% of the budget. The loss spikes when the random prefixes appear and recovers within 0.0002T tokens, and the logarithmic gains in P are the same ones the from-scratch runs show. | Proposed |
| SOTA-297 | Scale parallel computation with learnable input transforms, not parameters, when inference memory is the binding constraint | Chen et al. (2025), LIT-486 — run the same weights over P learnably-prefixed copies of the input and learn the aggregation. Loss falls as if the parameters had grown by O(log P); a 1.6B model at P = 8 matches a 4.4B one on code. At batch size 1 that costs 22× less added memory and 6× less added latency than the parameter scaling it replaces — and P× the training FLOPs. | Proposed |
| SOTA-298 | Measure a writing assistant by the outcome it was deployed to improve, not by the text features that used to predict it | Corpus et al. (2025), LIT-487 — Change.org’s AI drafting tool made petitions 49% longer, more lexically varied and better rated by human judges on quality and persuasiveness, moving them toward the profile of successful pre-AI petitions. The share reaching 10 signatures fell 5.33 points. The features had been chosen because they predicted success; afterwards they predicted it weakly or negatively. | Proposed |
| SOTA-299 | Finetune the whole generative model, encoder and decoder, for a dense perception task instead of attaching a head to a backbone | Khangaonkar and Pirsiavash (2025), LIT-488 — the freshly-initialized mask head is the part that cannot generalize, because it has only ever seen the finetuning categories. Keep every parameter generatively pretrained, encode the target as an image, and a model finetuned on furniture and cars segments people, x-rays and paintings. The same backbone under a conventional head scores 1.4–2.4 mIoU. | Proposed |
| SOTA-300 | To tell whether a model is doing non-trivial in-context computation, measure how well it predicts its own hidden states, not its next-token loss | Herrmann et al. (2025), LIT-489 — uniform noise maximizes next-token loss and requires no computation; reciting a memorized licence minimizes it and requires none either. Insert a variational bottleneck with a learned autoregressive prior and measure the posterior-prior KL: only the genuinely in-context task scores high, on a Transformer and an LSTM, and the signal survives controlling for next-token loss. | Proposed |
| SOTA-301 | Apply the objective gradient before the denoiser, not after it, when guiding diffusion toward a task objective | Zhang et al. (2026), LIT-490 — the usual recipe adds −η∇f after the denoising step, which walks the sample off the geometry the model learned with nothing left to pull it back. Apply the gradient to the noisy iterate first and let the denoiser follow: it acts as an approximate projection, and the sampler becomes an inexact projected-gradient method. Same cost, one line, and in trajectory planning it is the difference between a plan that looks good and one that survives execution. | Proposed |
| SOTA-302 v2 | Steer a distilled generator by modulating its input noise, not by fine-tuning its weights | Eyring et al. (2025), LIT-491 — train a LoRA hypernetwork to predict an improved initial noise for a frozen step-distilled generator. GenEval on SANA-Sprint goes 0.70 → 0.75 for 0.1 s of added latency, recovering about half of what 30-second test-time optimization buys. Reward fine-tuning the same model instead takes it 0.73 → 0.62: the anchoring KL term is intractable in weight space and tractable in noise space. | Proposed |
| SOTA-303 | Name the phase a time-saving tool accelerates before predicting what it will do to the quality of what people produce with it | Duede, Gross, Crockett and Bergstrom (2026), LIT-492 — in a marginal-value-theorem model, accelerating the phase before value is known makes producers more selective, accelerating the minimum cost of shipping makes them less selective, and both make each item less thoroughly developed — while accelerating discretionary improvement makes each item more thoroughly developed. Opposite signs from the same tool depending on where it acts, so “it saves time” does not determine the direction. | Proposed |
| SOTA-304 | Generate a long program unit by unit, freezing each unit that passes an execution-based check, instead of regenerating the whole artifact on failure | Cheng, Clark and Richardson (2025), LIT-493 — on 100 held-out proposals, the same model produced valid implementations 92% of the time generating unit by unit against an execution-based checker, and 6% of the time one-shot with retries; the one-shot outputs were also trivial, 49 lines against 181. Removing the checker alone costs 62 points, removing the decomposition alone costs 19 — so if you build one, build the checker. | Proposed |
| SOTA-309 | Score a control interface by the mutual information between the operator’s command and the state change it induces, and choose the horizon of that state change deliberately | Reddy, Levine and Dragan (2022), LIT-503 — whatever the operator is trying to do, a better interface yields less noisy commands, so I(x_t, (s_t, s_{t+Δ})) scores it with no labels, no reward and no task knowledge. Spearman ρ = 0.43 against true task completion across 540K examples in 4 of 5 domains. In the fifth the correlation was strongly negative at Δ = 1 and positive at episode length — so the horizon is not a detail. | Proposed |
| SOTA-316 | When you are memory-bound rather than time-bound, train all the parameters a block at a time, partitioned on layer boundaries | Liu et al. (2025), LIT-515 — freeze all but one block of layers, train it to local convergence, rotate. Memory falls to roughly the active block, so a 7B trains on one A800 where full-parameter needs two, and perplexity at 2B matches or beats the full-parameter baseline. The price is about three times the iterations, which the paper’s own table shows and its abstract does not. | Proposed |
| SOTA-319 | Reparameterize every linear layer by its spectral norm with a learned scalar | Zhai et al. (2023), LIT-523 — replace each linear layer’s W with γ·W/σ(W), γ a learnable scalar initialized to 1 and σ the spectral norm from power iteration. A ViT-B trained this way reaches 82.2% against a DeiT baseline’s 81.8% with no pre-LN, no warmup, no weight decay and LARS in place of Adam. Spectral normalization without the scalar gets 69.81%. | Proposed |
| SOTA-320 | Bound the learning rate by the ratio of the update’s spectral norm to the weight’s, and drop warmup | Qi et al. (2025), LIT-521 — if α_t > τ·σ₁(W_{t−1})/σ₁(∇W_t), truncate the step to that value; otherwise use the schedule. Weyl’s inequality makes this a direct cap on how fast a weight’s largest singular value can grow. ViT-B, ViT-L, GPT-S, Swin-S and Swin-B all train without any warmup and match or beat warmed-up AdamW. | Proposed |
| SOTA-321 | Verify a model’s lineage from the singular spectra of its attention products, which survive permutation and rescaling | Wang et al. (2025), LIT-524 — compare the per-layer singular spectra of W_q W_k^T and W_v W_o. Those products absorb the permutation and rescaling that change weights without changing the function, so the spectra identify a model through fine-tuning, pruning, merging and expansion. F1 0.9867 over 55 model pairs, using no data and no queries. | Proposed |
| SOTA-322 | Trade attention heads for depth: more heads conditions the attention block, and the layers you drop cost nothing | Saratchandran, Teney and Lucey (2025), LIT-527 — more heads lower the condition number of the attention block, which is part of what depth was buying. In the five configurations where the MLP width is held fixed, raising head count and cutting layers holds or improves accuracy at 29–53% fewer parameters — TNT-B goes 65.4M → 30.9M at identical Top-1. | Proposed |
| SOTA-323 | Test a claim about what pretraining produces on a model trained with the pretraining objective, not one trained on the task family you are testing | Shen, Mishra and Khashabi (2023), LIT-536 — “there exist weights such that” and “pretraining produces weights such that” are different claims, and training a model on the task family you are about to test collapses them. Their separation of these as Hypothesis 2 and Hypothesis 1 is what let them show that a literature everyone read as evidence about pretrained language models had tested neither the models nor the objective. | Proposed |
| SOTA-324 | Separate rival accounts of an internal algorithm by convergence rate and conditioning, not by how well each fits the output | Fu, Chen, Jia and Sharan (2023), LIT-535 — output similarity cannot tell gradient descent from Newton’s method, because both converge to the same answer. Matching each layer to the best-fitting step count of each candidate gives a linear trend for the right rate and an exponential one for the wrong one; a regime where the rivals must differ — here, condition number 100 — separates them again. Two measurements, both cheap, both decisive where a fit was not. | Proposed |
| SOTA-325 | When the training loss has saturated and runs still differ, measure loss-landscape degeneracy rather than curvature | Lau, Furman, Wang, Murfet and Wei (2023), LIT-542 — on ResNet18/CIFAR10, stronger implicit regularization (higher learning rate, lower batch size, higher momentum) gives a lower local learning coefficient and higher test accuracy, while every training loss has collapsed to zero. The estimator is λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] at β* = 1/log n, run by SGLD, validated against theory on deep linear networks to 100M parameters. Where the loss has stopped distinguishing runs, the geometry has not. | Proposed |
| SOTA-326 | Add n-gram embedding memory as one table read early in the network, on top of the expert budget rather than in place of experts | Qiu et al. (2026), LIT-152 — Qwen3.8-Flash-Next’s ablations. One n-gram embedding table at layer 2 takes loss from 1.585 to 1.541 and the benchmark average from 45.44 to 47.94. Splitting the same table across two layers buys nothing, and no depth clearly wins. Trading experts for table parameters at a fixed total lowers loss slightly and moves no downstream benchmark. Grown on top of the budget, the table keeps lowering loss after downstream accuracy has flattened. Layer 2 is a systems choice: the host-memory prefetch overlaps layer 1. | Proposed |
| SOTA-327 | Train retrieval embeddings with nested losses, so a prefix of the vector can shortlist and the full vector re-rank | Kusupati, Bhatt, Rege et al. (2022), LIT-547 — put the loss on log(d) nested prefixes of the embedding. Each prefix then matches a separately trained model of that width (ResNet50, ImageNet), and one database supports shortlisting on 16 dimensions and re-ranking on 2048: equal mAP@10, 14× faster. Measured on image retrieval. Widely shipped in text embedding models without an independent comparison in the record. | Proposed |
| SOTA-328 | Train one nested model instead of a family of separately trained sizes, and extract the sizes you need from it | Devvrit, Kudugunta, Kusupati et al. (2023), LIT-546 — nest four FFN widths in one Transformer, train one width per step, and extract any size in between by choosing widths per layer. Up to 850M parameters, at the compute of training the four sizes separately, the largest matches its baseline and the smaller ones beat theirs. It pays only if you were going to train the family. Gemma 3n ships it. | Proposed |
| SOTA-329 | For short-output tasks, adapt a frozen model with a low-rank intervention on a few positions of the residual stream, not a weight adapter; do not use it for long chain-of-thought | Wu, Arora et al. (2024), LIT-549 — LoReFT sets an r-dimensional subspace of the hidden state at a few prompt positions to a learned linear function of that state, with the model frozen. At 0.03% of parameters against LoRA’s 0.8%, it leads on commonsense QA (LLaMA-7B 80.2 against 74.7) and is level on GLUE. It loses on arithmetic chain of thought (GSM8K 26.0 against 37.5). All baseline numbers are copied from other papers. | Proposed |
| SOTA-332 | When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network | Sitzmann et al. (2020), LIT-551 — a sine network’s derivative is another sine network, so gradients and Laplacians of the fit are well behaved where a ReLU network’s second derivative is zero. Initialize hidden weights U(±√(6/n)) and scale the first layer by ω₀ = 30 (tune it to the signal), or deep sine networks do not train. It fits images, video, SDFs and PDE solutions from derivative supervision. | Proposed |
| SOTA-333 v4 | For autoregressive generation of continuous sequences, train with an independent noise level per token and condition the rollout on slightly noised history | Chen et al. (2024), LIT-554 — train a causal model to denoise tokens that each carry an independent noise level. At rollout, treat the generated history as slightly noisy, so accumulated errors look like training noise. The same model then samples autoregressively, plans with horizon-wide guidance, and keeps near-future tokens cleaner than far ones. Long-rollout stability is shown on video qualitatively, with a small RNN. Planning and robot gains are measured. | Proposed |
| SOTA-334 | In a style-modulated generator, demodulate the convolution weights instead of instance-normalizing the activations | Karras et al. (2019), LIT-560 — AdaIN’s instance normalization discards relative feature magnitudes, so the generator smuggles scale past it with a localized spike, the “water droplet” in every StyleGAN image. Scale each output channel’s modulated weights by 1/√(Σ w’² + ε) instead. It gives the same expected unit variance and style mixing still works. The artifact disappears and FID is unchanged (4.40 → 4.39). | Proposed |
| SOTA-335 | When generated content must move continuously, as in video or animation, build the generator alias-free: treat features as continuous signals and low-pass filter around every nonlinearity | Karras et al. (2021), LIT-559 — fine detail in GAN output sticks to pixel coordinates because the generator aliases. Apply each nonlinearity at 2× temporary resolution and low-pass filter back, start from Fourier features rather than a learned constant, remove per-pixel noise, and handle boundaries properly. On FFHQ-U, translation equivariance goes to 63 dB (and rotation to 40 dB for the -R variant) at StyleGAN2’s FID. | Proposed |
| SOTA-336 | Train high-resolution GANs at a fixed topology with output skips in the generator and a residual discriminator, not by progressive growing | Karras et al. (2019), LIT-560 — progressive growing makes each resolution briefly the output, so it learns output-level detail and leaves features stuck to pixel positions. A fixed generator that sums upsampled RGB outputs from every resolution, with a residual discriminator, keeps the coarse-to-fine emphasis without changing topology. FFHQ FID 4.34 → 3.31, and the phase artifacts are gone. | Proposed |
| SOTA-338 v2 | Train the GAN discriminator on frozen multi-scale pretrained features with fixed random channel and scale mixing | Sauer et al. (2021), LIT-562 — project real and generated images through a frozen EfficientNet, mix channels and scales with fixed random convolutions, and train one small discriminator per scale. It reaches StyleGAN2’s best LSUN-Church FID after 1.1M images instead of 88M. The speed-up is measured in FID, and at equal FID on FFHQ a CLIP-space distance and human raters prefer StyleGAN2 (LIT-563). | Proposed |
| SOTA-339 | For unpaired image-to-image translation of appearance, constrain both mappings with a cycle-consistency loss | Zhu et al. (2017), LIT-564 — learn X → Y and Y → X together, and penalize ‖F(G(x)) − x‖₁ and ‖G(F(y)) − y‖₁ alongside the two adversarial losses. With GAN losses alone the mapping may ignore its input. With the cycle it keeps content and changes appearance. It is scoped to colour and texture changes, and the authors report failure on geometric ones. | Proposed |
| SOTA-340 | Expect weight-only 4-bit quantization to speed up batched serving only while the batch keeps the matmul memory-bound, and use a kernel built to stay there | Frantar et al. (2024), LIT-567 — weight-only quantization speeds up decoding by loading fewer bytes, so the gain lasts only while the layer is memory-bound. A kernel designed to stay memory-bound (MARLIN) holds about 3.9× over FP16 up to batch 16–32 on an A10, decaying toward 1.5× at 128. End-to-end in vLLM, single-GPU, 2.3–3.2× at batch 16 or below and 1.1–1.2× at 128, on four GPU classes. Pick the format for the batch you actually serve, and pay for it in accuracy only where it buys speed. | Proposed |
| SOTA-341 | Condition an image generator on each training image’s original size instead of discarding or upsampling small images | Podell et al. (2023), LIT-566 — embed each training image’s original height and width, like the timestep, and give it to the model. Then keep the small images that a minimum-resolution filter would drop (39% of SDXL’s pretraining data below 256²) without teaching the model their blur. At inference, set the size you want. On class-conditional ImageNet 512²: FID-5k 43.84 discarding, 39.76 keeping unconditioned, 36.53 keeping with size conditioning. | Proposed |
| SOTA-342 | Report a sparse autoencoder’s fidelity as downstream loss in compute-equivalent terms, not as fraction of loss recovered against zero ablation | Gao et al. (2024), LIT-571 — splice the SAE’s reconstruction into the forward pass and report the resulting loss as the pretraining compute a model of that loss would need. Zero-ablating the residual stream is so destructive that “loss recovered” against it flatters any reconstruction. Their 16M-latent GPT-4 SAE scores 98.2% on it, and equals a GPT-4 trained on 10% of the compute. | Proposed |
| SOTA-343 | Read a transformer’s intermediate-layer predictions through a tuned lens, not the raw logit lens | Belrose et al. (2023), LIT-569 — decoding a hidden state with the final unembedding (the logit lens, LIT-570) fails outright on BLOOM, OPT and GPT-Neo, and is biased by 4–5 bits where it works. Train one affine translator per layer by KL to the final logits, initialized to the identity, and read the layer through that. It is cheap (d×d per layer), transfers to fine-tunes, and its directions are the model’s (Spearman 0.89). Train it with Muon. | Proposed |
| SOTA-344 | Train sparse autoencoders with a TopK activation instead of an L1 penalty, and prevent dead latents with transposed-decoder initialization and an auxiliary loss | Gao et al. (2024), LIT-571 — keep the k largest pre-activations and zero the rest, and train on reconstruction alone. L0 is then set, not tuned through an L1 coefficient that also shrinks the latents that fire. It beats ReLU SAEs on the MSE–L0 frontier by more as SAEs grow. Initialize the encoder as the decoder’s transpose and add a loss reconstructing the residual from dead latents: 7% dead at 16M latents, against up to 90%. | Proposed |
| SOTA-346 | Retune the training timestep shift for each autoencoder latent space, and never rank latent spaces for generation under one shared schedule | Black Forest Labs (2025), LIT-572 — four autoencoders, 30 timestep configurations each. The training shift alone moves FID by 61–86%. An RAE latent that wins when both are tuned loses to a tuned FLUX.2 latent when left unshifted. When you swap the autoencoder under a diffusion or flow model, sweep the shift again, using a logit-normal training distribution. When you compare autoencoders, compare each at its own optimum. | Proposed |
| SOTA-347 | To write many facts into a model’s weights, update a range of early-to-mid MLP layers jointly in one batched solve (MEMIT), choosing the range by measured edit success rather than causal tracing | Meng et al. (2023), LIT-576 — sequential single-fact edits collapse after tens of facts (ROME scores 50.3 at 10,000). Distributing each update over a range of MLP layers in one least-squares solve holds 85.8 on GPT-J, with neighborhood success falling from 83.5 to 73.7. Choose the layers by editing performance: per-fact causal tracing does not predict it (LIT-574). Evaluate implications too, because weight editors fail them (LIT-577). | Proposed |
| SOTA-352 | When a structure predictor is made generative, add a regression model’s predictions to the training data so disordered regions are learned as disorder rather than invented structure | Abramson et al. (2024), LIT-584 — a diffusion head generates plausible compact structure even where the protein has none. AlphaFold- Multimer, a regression model, renders the same regions as extended loops. Mixing its predictions into training teaches the generative model that convention, and “greatly reduced” hallucination. Remaining hallucinations are flagged by low confidence but do not look disordered. | Proposed |
| SOTA-353 | Train a head that predicts the model’s own accuracy on each output, against the metric you care about, and use it to rank samples and filter predictions | Jumper et al. (2021), LIT-583 — regress the per-residue lDDT the prediction will score (pLDDT) and the predicted TM-score. They track truth at r = 0.76 and 0.85 over 10,795 chains, and pLDDT filters the self-distillation set. Abramson et al. (2024), LIT-584, keep the head under diffusion by rolling out a cheap sample during training. They rank seeds by it, and antibody-interface quality keeps rising up to 1,000 seeds. | Proposed |
| SOTA-354 | When unlabelled inputs vastly outnumber labelled ones, retrain from scratch on the labelled data mixed with the model’s own high-confidence predictions for unlabelled inputs, under augmentation that stops it copying them | Jumper et al. (2021), LIT-583 — about 100,000 solved structures and billions of sequences. AlphaFold 2 predicted structures for about 350,000 unlabeled sequences, kept the confident ones, and retrained the same architecture from scratch on those plus the PDB. Cropping and alignment subsampling mean the student cannot just reproduce the teacher. The ablation shows a clear gain. The filter is the model’s own calibrated confidence (SOTA-353). | Proposed |
| SOTA-371 | Score topic coherence automatically as the average pointwise mutual information of a topic’s top-word pairs, estimated on a large external reference corpus | Newman et al. (2010), LIT-600 — when you cannot put every topic in front of people (SOTA-368), approximate their judgment. Take the topic’s top ten words and, for each of the 45 pairs, compute PMI from co-occurrence in 10-word windows over a large general corpus such as Wikipedia. Average the 45 scores. On 237 LDA topics this correlated with nine people’s ratings at ρ ≈ 0.77, far better than WordNet measures. | Proposed |
| SOTA-375 v2 | When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution | Mikolov et al. (2013), LIT-603 — sample negatives in proportion to count^(3/4). That flattens the Zipfian head so frequent words are not nearly the only negatives, while keeping rare words rare. The authors report it “outperformed significantly the unigram and the uniform distributions … on every task we tried”, but give no numbers. | Proposed |
| SOTA-378 | Compare, project and orthogonalize a language model’s concept directions after whitening by the unembedding covariance, not by raw cosine | Park, Choe and Veitch (2024), LIT-606 — training fixes a model’s representation only up to an invertible linear map, so Euclidean cosine between concept directions is not meaningful. Use ⟨u, v⟩ = uᵀ Cov(γ)⁻¹ v, with Cov(γ) the covariance of the unembedding rows, or equivalently transform by Cov(γ)^(−1/2) and then use Euclidean tools. Shown on LLaMA-2 7B and Gemma-2B by heatmap. Contested by a null for cross-lingual transport in 17 models. | Proposed |
| SOTA-385 | Tighten the gradient-norm clip as the model grows rather than carrying one value across scales | Rae et al. (2021), LIT-617 — the Gopher family clipped the global gradient norm at 1.0 for models up to 1.4B and at 0.25 for the 7.1B and 280B models, “for improved stability”. Three held practices say to clip and none names a value; this is the first number in the record, and it is not a constant. | Proposed |
| SOTA-386 v4 | Show a video diffusion model images before and alongside video | Proposed | |
| SOTA-387 | Hold out a slice of the training set for model selection instead of selecting on the validation split | Beyer et al. (2022), LIT-628 — they train on the first 99% of ImageNet-1k and keep 1% as a “minival”, stated as being “to encourage the community to stop selecting design choices on the validation (de-facto test) set.” A convention, asserted rather than measured, and the record held no document on it. | Proposed |
| SOTA-389 v3 | Caption training video with a model that watches the video, not with captions of its frames | Proposed | |
| SOTA-390 v3 | In a video diffusion transformer, attend over space and time jointly rather than factorizing, and budget for the cost | Proposed | |
| SOTA-391 | Estimate attention resolution before training when choosing a position encoding, and measure it across the extrapolation boundary | Sun et al. (2022), LIT-638 — attention resolution scores how well an attention pattern distinguishes token distance, and is estimable from the encoding before a run. Measured at the training length and twice it: RoPE 0.91 → 0.08, ALiBi 0.81 → 0.88. The record has four practices about rotary encoding and held no measure of this. | Proposed |
| SOTA-392 v3 | To get a one-step sampler from a rectified flow, distil it; treat reflow as an optional extra pass, and keep the pre-reflow model for many steps | Liu, Gong and Liu (2022), LIT-636. On CIFAR-10, one-step FID is 378 from the base model, 6.18 after distillation alone, 12.21 after one reflow, and 4.85 after reflow and distillation (Table 1a). Distillation does most of the work. Reflow adds a further gain at the cost of a second training pass, and worsens many-step quality from 2.58 to 3.36. InstaFlow (LIT-796) contests it at text-to-image scale, starting from a diffusion model rather than a rectified flow. | Proposed |
| SOTA-393 | Freeze the image model’s spatial layers when adding time if checkpoint compatibility matters; do not expect a quality gain from it | Girdhar et al. (2023), LIT-635, and Guo et al. (2023), LIT-633. Freezing a text-to-image model’s spatial layers and training only temporal ones keeps the model a drop-in for checkpoints fine-tuned from the same base. Emu Video’s one controlled test finds freezing costs nothing and gains little (55.0 / 58.1 against fine-tuning). The reason to freeze is compatibility, not quality. | Proposed |
| SOTA-394 v3 | When distilling a causal few-step video generator, take the teacher bidirectional, not causal | Yin et al. (2024), LIT-631. A block-causal 4-step student distilled from a bidirectional teacher beats one distilled from a causal teacher, at the same initialization, data and step count (Table 4). The causal teacher is also the weaker model before distillation, so the result shows the arrangement works and does not isolate why. | Proposed |
| SOTA-395 | Post-train an autoregressive video generator on its own rollouts, and score the whole rollout | Huang et al. (2025), LIT-629. Training a causal video generator on context it generated itself, with a distribution-level loss over the whole clip, beats teacher-forced and diffusion-forced training under every objective tried (Table 2). It holds quality as the number of autoregressive steps grows, where they degrade. Evidence is VBench on 5s clips, and the exposure-bias mechanism is inferred, not measured. | Proposed |
| SOTA-396 | When a latent video model reuses an image autoencoder or upsampler, fine-tune them on video so they see time | Blattmann et al. (2023), LIT-621. An image autoencoder decodes each frame independently and flickers. Fine-tuning its decoder on video, with a video discriminator, cuts reconstruction FVD on three datasets: 390.88 to 32.94, 35.82 to 18.66, and 73.78 to 25.55. Making the upsampler temporal cuts FVD from 165.98 to 45.39. Reconstruction FID worsens on one dataset and improves on two. | Proposed |
| SOTA-397 v2 | To condition an unconditionally trained diffusion model on known frames, use reconstruction guidance, not replacement | Ho, Salimans et al. (2022), LIT-627. Extending a video autoregressively from a model trained without conditioning, reconstruction guidance (steering each denoising step toward consistency with the known frames) gives FVD 136 against 451 for replacing the known frames’ noisy latents (Table 6). The model and guidance weight are held fixed. | Proposed |
| SOTA-398 v2 | Pretrain a vision transformer on packed, aspect-preserved images at sampled resolutions, not on fixed square crops | Dehghani et al. (2023), LIT-657. Do not resize or crop images to one square resolution. Keep each image’s aspect ratio and sample its resolution per example, favouring small side lengths. Pack the patches into fixed-length sequences with per-example attention masks. At matched compute this beats square fixed-resolution ViT pretraining, and matches the best ViT with 4x less compute. Most of the gain is throughput. | Proposed |
| SOTA-399 | Choose pre-training weight decay, dropout, clipping and precision for the quantization you intend to ship | Proposed | |
| SOTA-401 | Anchor the patch Gram matrix to an early teacher when training long, because dense features decay while global metrics improve | Proposed | |
| SOTA-402 | To teach a model to reason over facts it holds, spend added data on facts derived from other facts rather than on more atomic facts | Wang et al. (2024), LIT-667. Two sweeps that separate what is normally varied together: with the inferred/atomic ratio φ fixed, scaling the training set changes nothing about how a transformer acquires a two-fact inference rule; with the size effect thus accounted for, raising φ moves the acquisition monotonically, and at φ = 18.0 the delay is gone — 96.7% before training accuracy has even saturated. Synthetic knowledge graphs only, and it buys rule application on facts already seen in compositions, not systematicity. | Proposed |
| SOTA-403 v3 | Share the attention parameters across layers if you need to cut parameters; do not share the feed-forward ones | Lan et al. (2019), LIT-668. A transformer block has two halves and they are not equally compressible across depth. On an ALBERT-base configuration at E = 128, sharing every layer’s attention parameters scores 81.7 average against 81.6 for no sharing while removing 89M parameters to 64M; sharing the feed-forward parameters instead scores 80.2. At E = 768 the same split reads −0.7 against −2.8. One paper, 2019, encoder-only, and nobody has re-run it. | Proposed |
| SOTA-404 v2 | When you ablate an auxiliary loss, ablate the input construction that came with it, or the result is about both | Liu et al. (2019), LIT-671. BERT reported that removing next sentence prediction hurts; RoBERTa observed that the ablation had probably “only removed the loss term while still retaining the segment-pair input format”, separated the two, and found removing the loss matches or slightly improves while the input format was what mattered. LIT-667 is the same defect in another literature: four grokking papers varied the training fraction of a fixed universe, so dataset size and dataset composition moved together, and the conclusion reversed when somebody turned them one at a time. | Proposed |
| SOTA-405 | Scale the masked-language-modelling rate with model size rather than holding it at 15% | Wettig et al. (2022), LIT-672. The masking rate has been held at 15% “regardless of model sizes or masking strategies” since 2018, and the number was never swept. Swept, the optimum moves with capacity: 40% at 354M, 20% at 124M, 15% at 51M. At 354M, 40% wins seven of nine GLUE-plus-SQuAD tasks and reaches the 15% model’s QNLI and QQP scores in half the training time — under an efficient recipe, with the advantage shrinking on longer schedules. | Proposed |
| SOTA-406 | Replace every masked token with [MASK]; drop BERT’s 80-10-10 substitution rule | Wettig et al. (2022), LIT-672. BERT replaced 10% of selected positions with the original token and 10% with a random token, to reduce the pretrain/fine-tune mismatch caused by [MASK] never appearing downstream. Measured against an all-[MASK] baseline at a matched 40% rate, 80-10-10 is worse on everything but SST-2 — and the stated motivation does not bite, because “the model can adapt to full, uncorrupted sentences, regardless of the use of alternative corruption strategies in pre-training”. | Proposed |
| SOTA-408 v3 | Average the weights along the tail of training under a cyclical or high constant learning rate, then re-estimate the normalization statistics | Izmailov et al. (2018), LIT-673. Hold the learning rate high — cyclical or constant — over the last stretch of training and keep a running average of the weights, then ship the average. From pretrained torchvision checkpoints, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at one model’s inference cost. With batch normalization, the running statistics must be recomputed with one pass over the data or the averaged model is broken rather than merely worse. | Proposed |
| SOTA-411 | To let a model spend more compute at inference without more parameters, train a depth-recurrent core on a randomly sampled iteration count, re-injecting the input at every iteration | Geiping et al. (NeurIPS 2025), LIT-683. Put a small core block between a short prelude and coda. At each training step sample how many times to iterate it, from a heavy-tailed distribution with mean 32. Backpropagate through the last few iterations only. Feed the input embedding into every iteration, and start the state from noise. Trained this way, accuracy rises with iterations at test time and saturates per task. A model trained at fixed depth does not gain this. Nothing yet shows it beats a standard transformer of the same FLOPs. | Proposed |
| SOTA-412 | When the target is likelihood, learn a per-dimension noise schedule conditioned on a learned latent; a scalar or unconditioned one buys nothing | Sahoo et al. (NeurIPS 2024), LIT-677. For density estimation with a pixel-space diffusion model, replace the scalar learned schedule with a per-dimension one conditioned on a small discrete latent that an encoder infers from the image. Keep the endpoints fixed. VDM’s bound is reached in 5× fewer steps on CIFAR-10 and improved at equal steps. Each component alone does nothing: a scalar input-conditioned schedule and a time-only multivariate schedule both match VDM. | Proposed |
| SOTA-414 | To make an image diffusion model a dense geometric predictor, fine-tune it end to end as a one-step model at t = T with a task loss | Martin Garcia et al. (WACV 2025), LIT-687. Fix the timestep at T, feed zeros as the noise, decode the prediction through the frozen VAE, and train on the task’s own loss: affine-invariant L1 for depth, angular error for normals. From 20K iterations on 74K synthetic images, the one-step model beats Marigold’s 50-step, 10-member ensemble on depth AbsRel on all five zero-shot sets, and on normals by 2.6–3.0° of mean error. Starting from plain Stable Diffusion is nearly as good. | Proposed |
| SOTA-415 v3 | During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch | Li et al. (2025), LIT-684, extending Hägele et al. (2024), LIT-145. To see where a WSD run would land if annealed now, evaluate the simple average of its last ~10 stable-phase checkpoints. On the one fork measured, that proxy sits within about 1.5 points of a real 250B-token anneal on each benchmark. It costs no training. Still anneal the model you ship: on that fork the anneal is ahead on MMLU at the end. | Proposed |
| SOTA-417 | Filter pretraining text by how well the tokenizer compresses it | Geiping and Goldstein (2023), LIT-681. Drop every document whose token count exceeds t times its character count. Text a tokenizer trained on the corpus compresses badly is mostly markup, code fragments and debris. No model is needed. On C4 it lifts GLUE from 75.9 to 79.3, and on already-clean bookcorpus-wikipedia from 78.1 to 78.7. The gain tracks how dirty the source was. | Proposed |
| SOTA-418 | To drop the tokenizer without losing to BPE, use a two-stage hierarchy with learned chunk boundaries and SSM encoder and decoder layers, not an isotropic byte model | Hwang, Wang and Gu (2025), LIT-679. A byte-level model that must not lose to a BPE Transformer at matched data and FLOPs should be hierarchical. Isotropic byte stacks lose by about 9 downstream points. It should use Mamba-2 layers for the encoder and decoder around the main network, and it should learn its chunk boundaries across two stages. One learned stage only ties BPE and whitespace chunking. Expect about 2× slower training. | Proposed |
| SOTA-419 | Under a fixed compute budget, change the architecture only where it cuts step time at constant parameter count | Geiping and Goldstein (2023), LIT-681. Under a 24-hour, one-GPU budget, every reshape tried (depth 4 to 24, deep-narrow, wide, funnel, FFN-every-k, shared layers) finishes within about 0.1 of the same MLM loss. A smaller model’s extra throughput cancels its slower per-token learning. So the architecture changes that pay are the ones that make each step cheaper without changing how much the model learns per token: dropping biases, predicting only masked tokens, and pre-norm for the learning rate it permits. | Proposed |
| SOTA-420 | When distilling a classifier, match the teacher’s temperature-softened outputs at the same temperature and keep a down-weighted hard-label term | Hinton et al. (2015), LIT-680. Soft-target cross-entropy at temperature T in both teacher and student, plus a hard-label cross-entropy at T = 1 with a considerably lower weight. Multiply the soft term by T², so the balance does not move when T changes. Start low for a small student. The paper’s best realistic setting was T = 2, and a 30-unit MNIST net wanted 2.5–4. | Proposed |
| SOTA-421 | Before fitting a compute-optimal allocation law, tune learning rate, batch size and AdamW beta2 per model size and keep warmup short of the smallest run | Porian et al. (NeurIPS 2024), LIT-690. In a small-scale sweep that will be fitted for N*(C), a warmup longer than the smallest models’ optimal runs moves the exponent by 0.10. So does one batch size, learning rate and β₂ shared across sizes. Tune them per size, with β₂ above 0.95 at small batch, and cap warmup (the source uses warmup tokens = N, at most 20% of the budget). A constant learning rate is enough to fit the exponent, at about half the cost of a per-budget cosine. | Proposed |
| SOTA-422 | Rescale the noise schedule so the last timestep is pure noise, and train with v-prediction so the model can learn there | Lin et al. (WACV 2024), LIT-689, with Emu Video’s comparison, LIT-635. Common VP schedules leave signal at t = T. Stable Diffusion’s leaves √ᾱ_T = 0.068. The model learns to keep the leaked channel mean, and pure-noise inference then limits brightness. Rescale √ᾱ_t so it reaches 0 at T. Then train with v-prediction, because at zero SNR ε-prediction is trivial and v_T = x₀. Sample in the x₀ form, because converting v to ε at zero SNR divides by zero. | Proposed |
| SOTA-423 | When classifier-free guidance over-exposes a latent diffusion model, rescale the guided prediction toward the conditional prediction’s standard deviation | Lin et al. (WACV 2024), LIT-689. Compute x_cfg as usual, rescale it by std(x_pos)/std(x_cfg), and blend with weight φ back toward x_cfg. The source uses φ ≈ 0.7 at w = 7.5. It is the latent-space counterpart of Imagen’s dynamic thresholding, which needs a known pixel range. The evidence is qualitative, and part of the effect may simply be lower effective guidance. | Proposed |
| SOTA-426 | When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either | Lialin et al. (2023), LIT-104 — ReLoRA’s restart discipline, read as NOTE-024. At every merge-and-reinit, prune the adapter’s Adam state (99% by magnitude) and drop the learning rate to zero and re-warm it over 50–100 steps. In the 130M ablation (Table 6) the restart alone does nothing (34.25 vs LoRA’s 34.17), the re-warm without the reset does nothing (34.29), the reset without the re-warm diverges, and only the pair helps (29.77). A rule for how to restart, not a recommendation to restart. | Proposed |
| SOTA-427 | In a batch-normalized network, put dropout after the last batch-norm layer, not upstream of one | Li et al. (2018; CVPR 2019), LIT-705. BN freezes a moving variance accumulated while dropout was scaling activations. At test time dropout stops, the variance changes, and every later BN normalizes by the wrong constant. The cost is large: 77.42% → 68.55% on CIFAR-100 for DenseNet with dropout 0.5 in each bottleneck. The fix, one dropout layer after the last BN, is supported mainly as avoiding that cost. What it adds on its own is about 0.2 top-1 on ImageNet, with no spread reported. | Proposed |
| SOTA-428 | Choose the EMA length after training: store two power-function averages at snapshots and reconstruct any length by least squares | Karras et al. (2023), LIT-714 — EDM2. Track two power-function averages of the weights (σ_rel 0.05 and 0.10), save both every few thousand steps, and synthesize any EMA length after the run from a small linear solve. Then choose it per configuration, per guidance weight and per metric, because on ImageNet-512 the best length moves with all three — at guidance 1.4, FID wants 2% and FD_DINOv2 wants 14%. | Proposed |
| SOTA-429 | Drop half the image patches when training a CLIP-style model, spend the saving on more pairs and a larger batch, and unmask only for a short final tune | Li et al. (2022), LIT-706 — FLIP. Remove 50% of image patches and run the ViT on the rest, with CLIP’s loss and nothing else. The saving buys a 2× larger batch at the same memory and 2× the samples per hour; ViT-L/16 on LAION-400M reaches its unmasked reproduction’s accuracy more than 3× faster. The accuracy comes from what the saving buys: at equal batch, masking is parity. | Proposed |
| SOTA-430 v8 | Before using an attribution map to debug a model or explain what it learned, check that the map changes when the weights are randomized and when the labels are permuted — do not validate it by how it looks | Adebayo et al. (2018), LIT-713, read as NOTE-365, with Kokhlikyan et al. (2021), LIT-724. An attribution map that survives re-initializing the model’s weights, or retraining on permuted labels, cannot be telling you about the weights or the labels. Guided Backprop and Guided GradCAM survive the first above the lowest layers, and their maps look as convincing as ever. A rejection rule, not a certificate: passing does not show a method is faithful — GradCAM passes and agrees with occlusion maps at rank correlation 0.254. The verdict is per-modality — the same test on a BERT text classifier reverses it for global Integrated Gradients — and pass or fail is read from curves with no threshold anywhere. | Proposed |
| SOTA-431 | Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step | Karras et al. (2023), LIT-714 — EDM2. Normalize each output channel’s weight vector on use, re-normalize the stored weights after every step, scale every fixed operation to preserve unit magnitude, and drop the group norms. On the ADM U-Net at ImageNet-512 this takes FID from 6.96 to 2.56 at equal compute, with on-use weight normalization alone worth 6.96 → 3.75. One architecture family; untested on transformers. | Proposed |
| SOTA-433 | Guide a diffusion model with a smaller, less-trained copy of itself rather than an unconditional model, and make sure the two models are degraded in the same way | Karras et al. (2024), LIT-721. Replace classifier-free guidance’s unconditional reference with a degraded copy of the same conditional model — smaller and under-trained, same task, same conditioning, same data. The quality gain survives and the diversity loss does not: ImageNet-512 FID 2.56 → 1.34 (EDM2-S), 1.25 (XXL), 1.01 at ImageNet-64, and unconditional EDM2-S 11.67 → 3.86, which CFG cannot do at all. The binding condition is compatibility: degrade the two models differently and the method stops working entirely. | Proposed |
| SOTA-434 | In a vision backbone, carry position with a relative bias inside the attention window and do not add an absolute position embedding on top | Liu et al. (2021), LIT-723, Table 4. One architecture, one recipe, three tasks. A relative position bias inside the attention window is best or tied-best on all three. An absolute position embedding is worth +0.4 ImageNet top-1 and -0.2 COCO box AP and -0.6 ADE20K mIoU, and adding it on top of the relative bias costs 2.1 mIoU while changing ImageNet nothing. The recommendation is the sign, not the size: this is the one design choice in the table that a classification benchmark scores backwards. | Proposed |
| SOTA-436 | Route a very large or non-grid input through cross-attention into a small learned latent array, so that depth no longer scales with input size | Jaegle et al. (2021), LIT-747 — the Perceiver. A learned array of 512 latents cross-attends to all 50,176 ImageNet pixels, at a cost of O(MN) rather than O(M²), and a 48-block latent transformer then runs at a cost independent of input size. It matched ResNet-50 and ViT-B/16 at 78.0% top-1 without convolutions, did not beat them, and ran unchanged on audio, video and point clouds. The case for doing it is cost and modality-independence, not accuracy, and the cost advantage is argued from complexity, not measured against a baseline. | Proposed |
| SOTA-437 | Store and move model checkpoints losslessly compressed: code the float exponent as its own stream with an entropy coder alone, and store successive checkpoints as deltas against a periodic full base | Hershcovitch et al. (2024), LIT-750 — ZipNN. In regular trained weights only the float exponent is redundant: about 40 of its 256 values occur and it codes to about a third, while sign and mantissa barely compress. So split the exponent into its own stream and Huffman-code it without an LZ stage. Llama-3.1 BF16 goes to 66.4% against Zstd’s 77.7%, and is faster in both directions. Between checkpoints, XOR deltas against a base up to 10 checkpoints back compress better than standalone files. Sizes and throughputs are measured. Stall time inside a training run is not. | Proposed |
| SOTA-438 | To pretrain a wider language model when a smaller pretrained one of the same depth exists, initialize it by function-preserving symmetric width expansion rather than at random | Samragh et al. (2024), LIT-765 — HyperCloning. Tile each weight matrix of the small model into n×n blocks scaled by 1/n, so the wide model computes exactly the small model’s function at step 0, then train as usual. On three 2× pairs from 350M to 2.9B, it reached the random-init baseline’s final 10-task accuracy 2.2–4× sooner and finished higher. The source model’s pretraining is not charged, so this supports reusing a checkpoint that already exists, not training a small model in order to clone it. | Proposed |
| SOTA-439 | Test a Kronecker-factored curvature implementation against the cases where KFAC is exact before trusting it | Dangel et al. (2025), LIT-768, §5.2 — a tutorial, so the evidence is a derivation, not a training run. KFAC’s averaging step is exact whenever the layer inputs or the backpropagated vectors do not depend on the data point. That gives two tests: a single data point, where KFAC equals the GGN and the empirical Fisher exactly and the Monte-Carlo flavour converges to the GGN; and a deep linear network under square loss on arbitrary data, where the GGN and Monte-Carlo identities still hold and the empirical-Fisher one does not. The second is the one the paper says catches scaling bugs. | Proposed |
| SOTA-440 | Supervise a feed-forward 3D reconstructor with a frozen camera-controlled video model’s renderings instead of captured multi-view data | Bahmani et al. (2025), LIT-743 — Lyra trains a 3D Gaussian Splatting decoder only on a frozen video model’s RGB decodings along six camera paths per image, with no captured multi-view data. In its one controlled arm, training on real RealEstate10K + DL3DV instead scores PSNR 19.08 against 24.77, and adding the real data to the generated data gives 24.74, no better. Both are scored against the teacher’s own renderings on its own prompts, which is the distribution the generated supervision was drawn from — so the arm shows the generated data is sufficient there, not that it beats captured data on real scenes. | Proposed |
| SOTA-441 | When a web-scraped pair corpus is noisy, filter it by self-consistency: keep only the pairs whose passage a model trained on the noisy set ranks near the top | Wang et al. (2022), LIT-744, Table 7. Train a dual encoder on the noisy pairs, rank each pair’s passage against 1M random passages, and keep the pair only if it lands in the top 2. On six BEIR datasets, filtered beat unfiltered at 1M pairs (40.7 against 34.9 nDCG@10) and with all pairs (51.6 against 50.0), though the filtered set is about a quarter the size. One cutoff, chosen by inspection and never varied, and the filtering model’s cost is not counted. | Proposed |
| SOTA-442 | To RL-tune a flow-matching generator, roll it out as a same-marginal SDE built from its own velocity, and anchor each step with the closed-form KL to the reference | Liu et al. (2025), LIT-779. A flow model’s ODE sampler is not a stochastic policy. Swapping it for an SDE with the same marginals, built from the trained velocity alone, makes each step an isotropic Gaussian, so GRPO’s ratio and the KL to the reference model are both closed form. On SD3.5-M, GenEval goes 0.63 → 0.95 either way. Without the KL term, DrawBench aesthetic falls to 4.93 from a base of 5.39, against 5.25 with it (Table 2). Quality is judged by reward models only, one run each. | Proposed |
| SOTA-443 | Down-weight large residuals in consistency and MeanFlow training: use a Pseudo-Huber loss or an adaptive inverse-error weight, not plain squared L2 | Song and Dhariwal (2023), LIT-786, and Geng et al. (2025), LIT-784. When the regression target is the model’s own output under stop-gradient, squared L2 trains worse than a loss that limits the pull of large errors. iCT’s Pseudo-Huber beats squared L2 on CIFAR-10 (Fig. 2, curves). MeanFlow’s adaptive weight 1/(‖Δ‖² + c)^p takes one-step ImageNet-256 FID from 79.75 at p = 0 to 61.06 at p = 1 (Table 1e). Two groups, small settings, one run per arm. | Proposed |
| SOTA-444 | When a flow will be sampled with a coarse ODE solver, pair noise and data within each minibatch by exact optimal transport | Pooladian et al. (2023), LIT-789, and Tong et al. (2023), LIT-798. Re-pair each minibatch’s noise and data by exact discrete OT before the flow-matching regression. The marginals stay exact, the learned paths straighten, and on ImageNet-64 reaching FID 20 with Euler takes 12 steps instead of 29, at equal likelihood and 4% more time per iteration. It lowers the step count of an ODE sampler. It does not make one: four-step FID stays near 39 on ImageNet-32. Unconditional, 64 px and below. | Proposed |
| SOTA-445 | When the base text-to-image diffusion model is better than the models that generated the preference pairs, train on the pairs with Diffusion-DPO rather than fine-tuning on the preferred images | Wallace et al. (2023), LIT-774, §5.5 and Supp. S6. Pick-a-Pic’s images were generated by SDXL-beta and Dreamlike, and SDXL-1.0 is better than both. Supervised fine-tuning on the preferred images helped SD1.5 (55.5% win rate over its base) and made SDXL worse at every learning rate tried. Diffusion-DPO on the same pairs improved SDXL, preferred to the untuned base 70.0% of the time on PartiPrompts. The SDXL SFT arm has no number, there is one run per setting, and no one else has repeated it. | Proposed |
| SOTA-446 | In consistency training, take the target from the current weights under stop-gradient, not from an EMA teacher | Song and Dhariwal (2023), LIT-786, §3.2. Consistency Models trained the student against an EMA of itself. iCT shows that on a point-mass data distribution the limiting objective with any teacher other than the student carries no information about the data (Prop. 1), and that setting the teacher’s EMA rate to zero improves CIFAR-10 FID with both LPIPS and squared L2 (Fig. 2a, curves). Keep an EMA of the weights for evaluation. Do not use it for the target. | Proposed |
| SOTA-447 | When distilling a large image or video model by continuous-time consistency, add a small DMD distribution-matching term | Zheng et al. (2025), LIT-783. Pure sCM distillation of 2B–14B text-to-image and text-to-video teachers loses small text, texture and temporal coherence. Adding DMD’s loss at weight 0.01, on the student’s own 1–4-step rollouts with a fake-score network updated 5–10 times per step, fixes it in samples. Four-step Wan2.1-1.3B VBench is 84.43 against a re-implemented DMD2’s 84.56. sCM’s own Fig. 7 (LIT-790) is why the consistency term stays primary: distribution matching alone loses recall. | Proposed |
| SOTA-448 | Without a pretrained teacher, train a one-step generator as a MeanFlow average-velocity model rather than by consistency training or a shortcut model | Geng et al. (2025), LIT-784. Train a network for the average velocity over an interval, regressed onto v − (t − r)·du/dt under stop-gradient, with du/dt from a Jacobian-vector product. From scratch on ImageNet-256 with DiT-XL/2, one-step FID is 3.43 after 240 epochs, against 10.60 for a shortcut model and 34.24 for iCT as Inductive Moment Matching re-ran it (Table 2). Those rows are copied, at other budgets. No arm compares distilling a teacher. | Proposed |
| SOTA-449 | When reward-tuning a classifier-free-guided diffusion model over several rounds of sampling, train the guided prediction at the fixed sampling guidance weight, not the conditional prediction alone | Black et al. (2023), LIT-776, App. E.1 and Fig. 10. Reward fine-tuning should not train the unconditional branch, since the reward depends on the prompt. Training the conditional prediction alone made performance deteriorate after the first round of fine-tuning, which the authors attribute to the guidance weight becoming miscalibrated. Fixing the weight (5 in their runs) and training through the guided prediction used for sampling fixed it. One curve per arm, on reward-weighted regression and JPEG compressibility. | Proposed |