Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Training optimization — optimizers, learning-rate schedules, batch size, training dynamics, scaling laws and scaling strategies.
134 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-001 v2 | Use Adam as the default optimizer absent a reason to choose otherwise | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-002 v2 | Common hyperparameters: β₁=0.9, β₂=0.999, ε=1e-8 | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-003 | Learning rate typically 1e-4 to 1e-3 for most tasks | Kingma et al. (2014), LIT-001 — ARXIV-1412.6980. | Active |
| SOTA-008 v3 | linear warmup of LR stabilizes early training with large batch size. | You et al. (2017), LIT-009 — ARXIV-1708.03888. | Active |
| SOTA-009 v2 | warmup to a large early lr, anneal throughout training to small final lr | Smith et al. (2017), LIT-010 — ARXIV-1708.07120. | Active |
| SOTA-010 v2 | skip connections promote training stability by smoothing out the loss landscape | Li et al. (2017), LIT-014. Moved to THEORY-011 — the claim is a statement about why deep networks became trainable, not an instruction, and belongs in the scheme ADR-031 created for exactly that. This code stays so links into it resolve. | Superseded — by THEORY-011; Not a recommendation, and never was: it states what is true rather than what to do, which is a THEORY under ADR-031 and was one of the eight that decision named and declined to move. The claim is unchanged and is believed — it now lives at THEORY-011, with the figure’s limits and the five practices it underwrites. Superseded rather than Rejected because Rejected on a practice means do not do this, and nobody should read this code as advice against skip connections. ADR-034 is the rule. |
| SOTA-011 v3 | Map the Hessian ratio |lambda_min / lambda_max| to find where the loss surface is non-convex | Li et al. (2017), LIT-014 — ARXIV-1712.09913. | Active |
| SOTA-012 v3 | sharpness in the loss landscape correlates with test error | Li et al. (2017), LIT-014 — ARXIV-1712.09913. | Active |
| SOTA-013 | Use dynamic loss scaling that doubles every 2000 successful steps | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-014 | Maintain master weights in FP32 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-015 v2 | Store optimizer states in FP32 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-016 | Perform forward/backward passes in FP16 | Micikevicius et al. (2017), LIT-011 — ARXIV-1710.03740. | Active |
| SOTA-031 v2 | Keep micro-batch size per GPU as large as memory allows | Rajbhandari et al. (2020), LIT-027 — ARXIV-1910.02054. | Active |
| SOTA-035 | use gradient clipping | Pascanu et al. (2012), LIT-037 — ARXIV-1211.5063. | Active |
| SOTA-040 | larger models are more sample efficient | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. | Active |
| SOTA-041 v2 | lr tuning less important for larger models | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. | Rejected — States the opposite of its source. Kaplan reports that larger models require a smaller learning rate to prevent divergence, and carries an explicit size-dependent rule LR(N) because the right rate is not size-independent |
| SOTA-050 v2 | Scale attention weights by 1/sqrt(head_dim) | Vaswani et al. (2017), LIT-008 — ARXIV-1706.03762. | Active |
| SOTA-061 v3 | Use largest batch that maintains >80% sample efficiency | McCandlish et al. (2018), LIT-017 — ARXIV-1812.06162. The steps/examples trade the rule thresholds; the 80% line itself has no published origin the record can name. | Active |
| SOTA-062 v3 | Scale batch size with model size but sub-linearly | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. Along the compute-optimal frontier the model grows as C^0.73 and the batch as C^0.24: over a billion-fold increase in compute, more than a million-fold in model size against about a hundred-fold in batch. | Superseded — by SOTA-258; The decoupling experiment this practice’s own body invited has been run: holding the token budget fixed, the dependence of batch size on model size very nearly vanishes. What the heuristic was tracking is the data that scaled alongside the model. Superseded rather than Rejected because following it along the Chinchilla line gives roughly the right answer for the wrong reason — the successor is SOTA-258. |
| SOTA-064 v2 | Warmup needed scales sub-linearly with model size | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of warmup. | Rejected — The cited paper does not contain the word “warmup”. Its subject is model shape for downstream fine-tuning; this claim came from a takeaway that belonged to no identifiable source, and no paper in the record supports it |
| SOTA-065 v2 | Initialize layer norm weights closer to 1 for larger models | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of layer normalization, and the claim names no adjustable quantity. | Rejected — The cited paper contains no discussion of layer normalization. The claim is also doubtful on its own terms: LayerNorm’s gain initialises to exactly 1 by convention at every scale, so “closer to 1 for larger models” names no adjustable quantity |
| SOTA-066 v2 | Can use shorter warmup periods for wider models | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of warmup. | Rejected — The same gap as SOTA-064, restated as a width claim. The cited paper contains no discussion of warmup at all |
| SOTA-067 v2 | Monitor loss specifically during first ~5000 steps for instabilities | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of instability, and the 5000 belongs to nobody. | Rejected — The cited paper contains no discussion of training instability. The record’s sourced monitoring practices are SOTA-069 and SOTA-099, which do not depend on a step count |
| SOTA-068 v2 | Use gradient clipping during early training phase | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Superseded by SOTA-035, which cites Pascanu et al. and drops the “early training” qualifier. | Superseded — by SOTA-035; SOTA-035 is the same recommendation with a real source and without the false qualifier — clipping is applied throughout training, not only early |
| SOTA-092 | smaller batch sizes are more sample efficient (i.e., better loss as a function of tokens seen) earlier in training | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-093 | larger batch sizes are beneficial later in training due to better gradient estimates | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-094 | count accelerator efficiency alongside sample efficiency when choosing a batch size: larger batches mean larger matrix multiplications | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. | Active |
| SOTA-096 v2 | num_tokens ~ 20 * num_params | Hoffmann et al. (2022), LIT-068 — ARXIV-2203.15556. | Active |
| SOTA-097 v4 | Optimal batch size scales approximately with compute budget - B ∝ C^(1/4) | Kaplan et al. (2020), LIT-028 — ARXIV-2001.08361. Equation 1.7 gives B ∝ C^(α_C/α_B) = C^0.24, from α_C ≈ 0.050 and α_B ≈ 0.21. | Superseded — by SOTA-258; The exponent is a fit to a projection. Along the Chinchilla line N and D scale together, so C = 6ND moves with D and a power law in either describes the data; two independent groups measuring off that line find optimal and critical batch size scale in the token budget, not in compute. Superseded rather than Rejected because the fit is not an error and the quantity is real — the successor is SOTA-258, and ADR-017’s corrective succession is the shape. |
| SOTA-098 v2 | Monitor the training loss for unexpected spikes, for the whole run | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. The 540B model’s training loss spiked about 20 times, at irregular intervals, sometimes late, despite gradient clipping, and never in the smaller models. | Active |
| SOTA-099 | Track gradient norm statistics to detect training instabilities | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-100 v2 | Use learning rate warmup proportional to model size | Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. | Active |
| SOTA-120 v2 | Prefer AdamW’s decoupled weight decay to L2 regularization added to the loss | Loshchilov et al. (2017), LIT-012 — ARXIV-1711.05101. | Active |
| SOTA-121 v3 | Use Muon with decoupled weight decay and AdamW-matched update RMS in place of AdamW | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Stable at nearly the same optimal LR as AdamW, better evaluations; used for every Falcon-H1-Tiny model. | Active |
| SOTA-122 v2 | Attach learnable per-row and per-column multipliers to weight matrices so their norms are learned, not set by LR and WD | Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost. Up to 20% relative gain on MMLU, BBH and GSM8K over a Muon baseline at 200 GT. | Proposed |
| SOTA-131 v4 | When training with Muon at scale, rescale query and key weights whenever attention logits exceed a threshold (QK-Clip) | Kimi Team (2025), LIT-132 — MuonClip carried a 1T/32B MoE through 15.5T tokens with zero loss spikes where plain Muon let attention logits pass 1000; confirmed at 2.8T in LIT-131. | Active |
| SOTA-139 | Extend the context length in stages during pretraining rather than training at the target length from the start | DeepSeek-AI (2026), LIT-139 — 4K, then 16K, 64K and 1M over 32–33T tokens, with sparse attention introduced at the 64K stage; Kimi K3 (LIT-131) reaches 1M the same way. | Active |
| SOTA-140 v5 | Use a warmup-stable-decay schedule: hold the learning rate, then decay it sharply over the final 10–20% of tokens | Hu et al. (2024), LIT-144, with the controlled comparison in LIT-145 — matches cosine at equal tuning, leaves the token budget open, makes every stable-stage checkpoint a usable branch point; contested at frontier scale by LIT-131, which retuned per schedule and chose cosine. | Active |
| SOTA-141 v3 | Decay the learning rate linearly all the way to zero | Bergsma et al. (2025), LIT-147 — with the peak tuned, linear decay to zero beats the customary decay to 10% and other shapes at compute-optimal budgets, more so past them; Proposed because contemporary production runs in the record still decay to a floor. | Proposed |
| SOTA-142 v3 | Set the peak learning rate by a power law in tokens so it transfers across batch size and training length | Shen et al. (2024), LIT-146 — under WSD the optimal learning rate is a power law in tokens and batch size; the Power scheduler applies it, and with µP the same setting carries across model sizes; adopted independently by Falcon-H1-Tiny. | Active |
| SOTA-143 v2 | Parameterize the model with µP and tune hyperparameters on a narrow proxy, transferring them across width | Yang and Hu (2022), LIT-148 — under the Maximal Update Parametrization the optimal learning rate and friends are stable across width, so tune small and transfer zero-shot; GPT-3 6.7B beaten at 7% of its pretraining cost in tuning. Used by Falcon-H1, MiniCPM and the Power scheduler. | Active |
| SOTA-144 v2 | Extend µP’s transfer to depth with CompleteP so one sweep serves deeper models too | Dey et al. (2025), LIT-150 — the depth exponent α = 1 transfers the optimal learning rate across depth and keeps deep layers learning; 11.8% fewer FLOPs than µP at optimal shapes, 34.4% at 179 layers. One group so far. | Proposed |
| SOTA-156 v2 | Train without a learning-rate schedule: average the iterates so no stopping time need be fixed | Defazio et al. (2024), LIT-213 — scheduling and iterate averaging are one mechanism, and the averaged form needs no stopping time T, no schedule shape and no hyperparameter beyond what AdamW already has. Filed Proposed: the evidence is a third-party competitive benchmark at workloads far below the scale where this record’s schedule question is open. | Proposed |
| SOTA-159 | Combine µP with unit scaling so the hyperparameters decouple and FP8 needs no loss scaling | Blake et al. (2024), LIT-149 — µP makes activation scale independent of model size; unit scaling makes activations, weights and gradients start at a scale of one. Together the model is size-independent and well-scaled from the first step, the hyperparameters become close to independent so a one-dimensional sweep replaces a joint search, and FP8 works without loss-scaling tricks. | Proposed |
| SOTA-160 | Treat the pretraining token budget and the post-training quantization plan as one decision, not two | Kumar et al. (2024), LIT-186 — degradation from post-training quantization increases with how much data the model was trained on, to the point where additional pretraining data becomes actively harmful if the model will be quantized afterwards. Fit over 465 runs into one precision-aware scaling law covering both pre- and post-training quantization. | Proposed |
| SOTA-162 v2 | Train with auxiliary multi-token-prediction heads alongside next-token prediction | Gloeckle et al. (2024), LIT-163 — predict the next n tokens through n independent heads on a shared trunk, as an auxiliary task rather than a replacement. No training-time overhead, the benefit grows with model size and survives multi-epoch training, and the extra heads are a draft model you already trained. | Active |
| SOTA-165 | Precondition the gradient with matrices rather than entrywise scaling | Wen et al. (2025), LIT-156 — under per-optimizer tuning across ten optimizers and four scales, every fastest one multiplies gradients by matrices rather than scaling entrywise. A structural finding that survives a fair comparison, and the class the record’s Muon practice is one member of. The advantage is 1.4× at 0.1B and 1.1× at 1.2B. | Active |
| SOTA-168 | Run Adam in Shampoo’s eigenbasis (SOAP) instead of Shampoo itself | Vyas et al. (2024), LIT-157 — Shampoo at the 1/2 power is Adafactor in the eigenbasis of Shampoo’s preconditioner, so run the better optimizer in that basis. SOAP is Adam there, adding exactly one hyperparameter over Adam. Over 40% fewer iterations and 35% less wall-clock than AdamW at 360M–660M in the large-batch regime. | Proposed |
| SOTA-188 v5 | Parametrize the network so its prediction target has unit variance at every noise level, and sample training noise from a log-normal | Active | |
| SOTA-192 v9 | Normalize the queries and keys before the attention dot product | Active | |
| SOTA-193 | Lower Adam’s second-moment decay when the loss spikes, before reaching for the learning rate | Proposed | |
| SOTA-195 v2 | Predict v rather than the noise when the model will be evaluated at low signal-to-noise | Active | |
| SOTA-198 v2 | Measure the gradient noise scale instead of sweeping batch size, and expect it to grow during the run | Active | |
| SOTA-202 v2 | Clamp the prediction to the training range at every step when sampling from a model’s own output | Active | |
| SOTA-204 v2 | Anneal a discretisation from coarse to fine over training rather than fixing it | Song et al. (2023), LIT-093. Where a training loss approximates a continuous target through a step count, that count is a bias/variance dial: few steps give a biased but low-variance target early, many steps a faithful but noisy one later. | Proposed |
| SOTA-211 | Spend one fitness evaluation per evolution-strategies direction, not an antithetic pair | Kaya and Hashemi (2026), LIT-232, and Ba et al. (2026), LIT-230 — the antithetic pair’s variance reduction depends on both evaluations sharing randomness, which autoregressive regeneration breaks. Replace the pair with a leave-one-out baseline computed from the population mean the score centering already needs: same expected update, one evaluation per direction, twice as many directions per budget, and half the estimator MSE in transformer blocks. | Proposed |
| SOTA-218 v2 | Retune the learning rate, momentum and schedule at every batch size you compare, never by a scaling heuristic | Linear scaling and square-root scaling are not batch-size rules; they are guesses about where the optimum moved, and they hold over a narrower range than the comparisons people use them for. A curve of steps-to-target against batch size drawn with transferred metaparameters is not a measurement of batch size at all — it is a comparison between one tuned configuration and several untuned ones, and the point where it bends is the point where the heuristic failed. | Active |
| SOTA-221 v2 | To train past the batch size where your optimizer stalls, change the optimizer’s conditioning rather than the scaling rule | You et al. (2019), LIT-265. When a batch size stops reaching your accuracy target, the reflex is to look for a better learning-rate scaling rule. LIT-058 is 168,160 training runs of evidence that there is no such rule to find. What moved the wall instead was the optimizer: normalizing each layer’s update by the ratio of its parameter norm to its update norm, on top of per-coordinate adaptivity, carried BERT from 16K to 32K where a grid-searched AdamW could not follow. This does not license skipping the retuning in SOTA-218 — the two say different things about different activities, and THEORY-012 is why. | Active |
| SOTA-243 v2 | Discard the easy examples when data is abundant and the hard ones when it is scarce | Sorscher et al. (2022), LIT-399 — power-law scaling in dataset size is evidence of redundancy, and given a difficulty ranking it can be beaten toward exponential. But which end of the ranking to discard flips with how much data you started with, and the fraction to discard has to grow with the corpus. | Proposed |
| SOTA-251 | Train at low resolution and raise it only during the decay phase of a warmup-stable-decay schedule | Assran et al. (2025), LIT-215 — ARXIV-2506.09985. Spend warmup and the constant phase on short, low-resolution clips; raise resolution and clip length only in the final decay. Up to 8x less pretraining compute than training at full resolution throughout, for the same end state. | Proposed |
| SOTA-254 v2 | Train the language model as a masked diffusion model when the corpus is fixed and the compute is not | Prabhudesai et al. (2025), LIT-442 — masked diffusion keeps extracting signal from a repeated corpus for roughly 512 epochs where autoregressive training stops at 32. Below a critical compute threshold, given in closed form from the unique-token count, autoregressive training is better and by a wide margin; above it, diffusion is better. The recommendation is the condition, not the objective. | Proposed |
| SOTA-255 v2 | Tune weight decay upward when pretraining over a repeated corpus, rather than inheriting the customary value | Kim et al. (2025), LIT-441 — with the corpus fixed and parameters past Chinchilla-optimal, the weight decay of 0.1 everyone inherits from GPT-3 is far too small: loss turns upward as parameters or epochs are added. Tuned jointly with learning rate and epoch count, the optimum is roughly 30x larger, and loss becomes monotone in parameter count with a steeper exponent than Chinchilla’s. | Proposed |
| SOTA-256 | Judge a monotone scaling recipe by its asymptote, not by its loss at a compute budget | Kim et al. (2025), LIT-441 — when compute is not the binding constraint, a recipe’s loss at a chosen budget answers a question nobody asked. Fit the scaling law and compare the asymptote instead. The catch is that an asymptote is an extrapolation, and rankings at small scale can invert: the ensembling recipe wins on asymptote while losing at small member counts. | Proposed |
| SOTA-257 v2 | Spend surplus compute on an ensemble of independently seeded models and distil it, rather than on one larger model | Kim et al. (2025), LIT-441 — under a fixed corpus, training K models that differ only in seed and averaging their logits reaches a lower loss asymptote than scaling one model to the same total parameter count. Two 300M models beat one 600M. Members want more epochs and less weight decay than a standalone model, and an 8-member ensemble distils into a 300M student retaining about 83% of the gain. | Proposed |
| SOTA-258 | Scale batch size with the token budget, not with compute or model size | Zhang et al. (2024), LIT-445, and Bergsma et al. (2025), LIT-443 — critical batch size scales as D^0.46 and D^0.47 respectively, and is nearly flat in model size once the token budget is held fixed. Optimal batch size likewise goes as D^0.38. The compute exponent the field has used since Kaplan fits a projection: along the Chinchilla line C and D move together and only D is doing the work. | Active |
| SOTA-259 | Hold Adam’s second-moment half-life fixed in tokens when the batch size changes, not beta_2 | Marek et al. (2025), LIT-444 — beta_2 is a decay per optimizer step, and a step is not a fixed amount of data. Changing batch size from B to B' while holding beta_2 fixed changes the second moment’s averaging window in tokens by the same factor. Hold the half-life fixed instead: beta_2' = beta_2^(B'/B). beta_1’s default is fine. | Proposed |
| SOTA-260 | Use the smallest batch size that still saturates the device, and do not gradient-accumulate to avoid it | Marek et al. (2025), LIT-444 — batch size one trains stably once Adam’s second-moment half-life is held fixed in tokens, matches or beats large batches per FLOP, and is far more robust to hyperparameter misspecification. Gradient accumulation costs memory for the accumulated gradient and forgoes optimizer steps, and is worth it only when multiple device replicas are bottlenecked on interconnect bandwidth. | Proposed |
| SOTA-261 | Set AdamW’s weight decay by targeting a timescale that follows a power law in tokens-per-parameter | Bergsma et al. (2025), LIT-443 — under AdamW the weights are an EMA of updates with timescale 1/(eta*lambda), and the fraction of training it averages, tau = 1/(eta*lambda*S), is the quantity that scales. It follows a power law in tokens-per-parameter with exponent about -0.52 across three orders of magnitude of compute, so weight decay can be read off in advance rather than swept. | Proposed |
| SOTA-269 | Keep a knowledge base outside the weights; parameters are an expensive and lossy place to memorise facts | Lu et al. (2024), LIT-452 — fact capacity is linear in model size and saturates in epochs, and the extrapolation is damning: all of Wikidata would want about 1000B non-embedding parameters trained for 100 epochs. Derivable facts cost full price unless they happen to share direction and structure. Borgeaud et al., LIT-060, is the record’s evidence that the alternative works at scale. | Active |
| SOTA-270 | Do not read a smooth loss curve as evidence of smooth training; decompose it when the answer matters | Kangaslahti et al. (2025), LIT-455, with Cohen et al. LIT-453 and Kunin et al. LIT-454 — the curve time-averages oscillation, sums over differently-timed abrupt transitions, and reads flat while the weights still travel. Smoothness is what many breakthroughs look like added up, so it is not evidence about the training. | Active |
| SOTA-272 | Expect the loss to be non-monotone at the step size that trains fastest, and do not set the step size from a curvature bound | Cohen et al. (2021), LIT-461 — gradient descent raises the sharpness until its own step size cannot tolerate more, so a rule that anneals the step size to the measured curvature is chasing a number it is producing, and loses to the fixed step it forbids. Demonstrated full-batch. | Proposed |
| SOTA-273 | Choose a pretraining context whose association with the input is neither too strong nor too weak, and mix contexts to get there | Zhai (2025), LIT-459 — a pretraining objective is an association between the input and a context variable, and the useful contexts are the ones whose association is neither too strong nor too weak: the singular values of the expectation operator should decay neither too fast nor too slowly. Contexts that individually miss the range can be mixed into one that does not. | Proposed |
| SOTA-274 v2 | Before adopting a spectral optimizer, measure the stable rank of each block’s incoming activations against its gradient’s nuclear rank | Davis and Drusvyatskiy (2025), LIT-457 — the spectral step’s one-step guarantee beats the Euclidean one by the ratio of gradient nuclear rank to incoming activation stable rank. Both are cheap to compute from a run you are already doing, and the paper never uses them to route. | Proposed |
| SOTA-275 | Pick the depth parameterization from the residual branch: a Transformer branch has more than one transformation, so it needs the stricter rule | Zheng et al. (2026), LIT-462 — Depth-muP’s α = Θ(1/√L) is the rule for a one-transformation residual branch. Attention and FFN branches hold more than one, so Transformers need CompleteP’s α = Θ(1/L), and the looser rule measurably fails to transfer on them. | Proposed |
| SOTA-276 | For a normalized or preconditioned optimizer, width-depth muP is width muP plus a hidden residual multiplier of order 1/L | Zheng et al. (2026), LIT-462 — a normalized or preconditioned update has a norm that does not depend on the residual multiplier, so the depth factor the raw gradient carries is removed and the optimizer’s own muP rule is unchanged. SGD is the exception and needs more. | Proposed |
| SOTA-282 | Put every matrix and hidden state on the unit hypersphere, and make each block’s residual step size a learned per-dimension parameter | Proposed | |
| SOTA-283 | To fine-tune a model that is already quantized, bank the part of each update that is smaller than the lattice spacing, and rebuild the accumulator from seeds rather than storing it | Proposed | |
| SOTA-285 | Put dropout at the start of training if the model underfits, and at the end if it overfits — the schedule decides the sign, not the rate | Proposed | |
| SOTA-287 | If you need to train without backpropagation, train each block to denoise a noisy label embedding conditioned on the raw input | Proposed | |
| SOTA-288 | Late in training, replicate a small subset of the weights, train the copies against shared base weights, and average the subset back into one model | Proposed | |
| SOTA-296 | Add the parallel streams in a short final training stage, not from the start | Chen et al. (2025), LIT-486 — parallel scaling costs P× the training FLOPs, which is what would otherwise make it unaffordable. Train 1T tokens the ordinary way, then switch the streams on for 20B tokens, 2% of the budget. The loss spikes when the random prefixes appear and recovers within 0.0002T tokens, and the logarithmic gains in P are the same ones the from-scratch runs show. | Proposed |
| SOTA-297 | Scale parallel computation with learnable input transforms, not parameters, when inference memory is the binding constraint | Chen et al. (2025), LIT-486 — run the same weights over P learnably-prefixed copies of the input and learn the aggregation. Loss falls as if the parameters had grown by O(log P); a 1.6B model at P = 8 matches a 4.4B one on code. At batch size 1 that costs 22× less added memory and 6× less added latency than the parameter scaling it replaces — and P× the training FLOPs. | Proposed |
| SOTA-309 | Score a control interface by the mutual information between the operator’s command and the state change it induces, and choose the horizon of that state change deliberately | Reddy, Levine and Dragan (2022), LIT-503 — whatever the operator is trying to do, a better interface yields less noisy commands, so I(x_t, (s_t, s_{t+Δ})) scores it with no labels, no reward and no task knowledge. Spearman ρ = 0.43 against true task completion across 540K examples in 4 of 5 domains. In the fifth the correlation was strongly negative at Δ = 1 and positive at episode length — so the horizon is not a detail. | Proposed |
| SOTA-315 v2 | Early-stop and tune on validation loss after temperature scaling, then calibrate post hoc, instead of stopping on raw validation loss | Berta et al. (2025), LIT-514 — a proper loss is calibration error plus refinement error, and the two bottom out at different epochs, so the validation-loss minimum is optimal for neither. Fit a temperature on the validation set each time you would have read the loss, read the loss after it, and stop there; then keep the temperature. Measured across 196 classification datasets and a vision benchmark with ten runs per dataset. | Active |
| SOTA-316 | When you are memory-bound rather than time-bound, train all the parameters a block at a time, partitioned on layer boundaries | Liu et al. (2025), LIT-515 — freeze all but one block of layers, train it to local convergence, rotate. Memory falls to roughly the active block, so a 7B trains on one A800 where full-parameter needs two, and perplexity at 2B matches or beats the full-parameter baseline. The price is about three times the iterations, which the paper’s own table shows and its abstract does not. | Proposed |
| SOTA-319 | Reparameterize every linear layer by its spectral norm with a learned scalar | Zhai et al. (2023), LIT-523 — replace each linear layer’s W with γ·W/σ(W), γ a learnable scalar initialized to 1 and σ the spectral norm from power iteration. A ViT-B trained this way reaches 82.2% against a DeiT baseline’s 81.8% with no pre-LN, no warmup, no weight decay and LARS in place of Adam. Spectral normalization without the scalar gets 69.81%. | Proposed |
| SOTA-320 | Bound the learning rate by the ratio of the update’s spectral norm to the weight’s, and drop warmup | Qi et al. (2025), LIT-521 — if α_t > τ·σ₁(W_{t−1})/σ₁(∇W_t), truncate the step to that value; otherwise use the schedule. Weyl’s inequality makes this a direct cap on how fast a weight’s largest singular value can grow. ViT-B, ViT-L, GPT-S, Swin-S and Swin-B all train without any warmup and match or beat warmed-up AdamW. | Proposed |
| SOTA-324 | Separate rival accounts of an internal algorithm by convergence rate and conditioning, not by how well each fits the output | Fu, Chen, Jia and Sharan (2023), LIT-535 — output similarity cannot tell gradient descent from Newton’s method, because both converge to the same answer. Matching each layer to the best-fitting step count of each candidate gives a linear trend for the right rate and an exponential one for the wrong one; a regime where the rivals must differ — here, condition number 100 — separates them again. Two measurements, both cheap, both decisive where a fit was not. | Proposed |
| SOTA-325 | When the training loss has saturated and runs still differ, measure loss-landscape degeneracy rather than curvature | Lau, Furman, Wang, Murfet and Wei (2023), LIT-542 — on ResNet18/CIFAR10, stronger implicit regularization (higher learning rate, lower batch size, higher momentum) gives a lower local learning coefficient and higher test accuracy, while every training loss has collapsed to zero. The estimator is λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] at β* = 1/log n, run by SGLD, validated against theory on deep linear networks to 100M parameters. Where the loss has stopped distinguishing runs, the geometry has not. | Proposed |
| SOTA-328 | Train one nested model instead of a family of separately trained sizes, and extract the sizes you need from it | Devvrit, Kudugunta, Kusupati et al. (2023), LIT-546 — nest four FFN widths in one Transformer, train one width per step, and extract any size in between by choosing widths per layer. Up to 850M parameters, at the compute of training the four sizes separately, the largest matches its baseline and the smaller ones beat theirs. It pays only if you were going to train the family. Gemma 3n ships it. | Proposed |
| SOTA-338 v2 | Train the GAN discriminator on frozen multi-scale pretrained features with fixed random channel and scale mixing | Sauer et al. (2021), LIT-562 — project real and generated images through a frozen EfficientNet, mix channels and scales with fixed random convolutions, and train one small discriminator per scale. It reaches StyleGAN2’s best LSUN-Church FID after 1.1M images instead of 88M. The speed-up is measured in FID, and at equal FID on FFHQ a CLIP-space distance and human raters prefer StyleGAN2 (LIT-563). | Proposed |
| SOTA-354 | When unlabelled inputs vastly outnumber labelled ones, retrain from scratch on the labelled data mixed with the model’s own high-confidence predictions for unlabelled inputs, under augmentation that stops it copying them | Jumper et al. (2021), LIT-583 — about 100,000 solved structures and billions of sequences. AlphaFold 2 predicted structures for about 350,000 unlabeled sequences, kept the confident ones, and retrained the same architecture from scratch on those plus the PDB. Cropping and alignment subsampling mean the student cannot just reproduce the teacher. The ablation shows a clear gain. The filter is the model’s own calibrated confidence (SOTA-353). | Proposed |
| SOTA-358 v3 | Drop the domain inductive bias once pre-training data is large enough, and keep it when it is not | Active | |
| SOTA-359 v2 | Supervise vision from the caption, and match image to caption rather than predicting its words | Active | |
| SOTA-360 | Score a density ratio against sampled negatives instead of reconstructing the target | Active | |
| SOTA-363 | Decouple the negative set from the batch with a queue, and keep the encoder that fills it slow | Active | |
| SOTA-372 v2 | Run the encoder on visible tokens only and push the mask tokens into a small decoder you discard | Active | |
| SOTA-375 v2 | When training with sampled negatives, draw them from the unigram distribution raised to the 3/4 power, not from the unigram or uniform distribution | Mikolov et al. (2013), LIT-603 — sample negatives in proportion to count^(3/4). That flattens the Zipfian head so frequent words are not nearly the only negatives, while keeping rare words rare. The authors report it “outperformed significantly the unigram and the uniform distributions … on every task we tried”, but give no numbers. | Proposed |
| SOTA-376 | Score each image-text pair independently with a sigmoid, so the loss needs no global normalization | Active | |
| SOTA-377 | Stop scaling the contrastive batch past about 32k, because the benefit saturates there | Active | |
| SOTA-381 v2 | Sample a handful of negatives instead of normalizing over the vocabulary, and let the count fall as the data grows | Active | |
| SOTA-384 | Calibrate a trained classifier with a single temperature fitted on held-out data, not with a richer map | Guo et al. (2017), LIT-616 — rescale the logits by a single learned 1/T fitted for NLL on a held-out set. It cannot change the argmax, so accuracy is unchanged by construction, and it beats vector scaling, matrix scaling, histogram binning, isotonic regression and BBQ — including the two that strictly contain it. Typical uncalibrated ECE is 4–10%. | Active |
| SOTA-385 | Tighten the gradient-norm clip as the model grows rather than carrying one value across scales | Rae et al. (2021), LIT-617 — the Gopher family clipped the global gradient norm at 1.0 for models up to 1.4B and at 0.25 for the 7.1B and 280B models, “for improved stability”. Three held practices say to clip and none names a value; this is the first number in the record, and it is not a constant. | Proposed |
| SOTA-386 v4 | Show a video diffusion model images before and alongside video | Proposed | |
| SOTA-388 v2 | Train a plain ViT on ImageNet-1k with average pooling, fixed 2D sin-cos positions, a 1024 batch and light augmentation | Beyer et al. (2022), LIT-628 — five changes, none novel, take ViT-S/16 on ImageNet-1k from 66.8% to 76.5% at 90 epochs and 80.0% at 300. Global average pooling, fixed 2D sin-cos position embeddings, batch 1024, RandAugment at level 10 and Mixup at p = 0.2. 6h30 on a TPUv3-8 for the 90-epoch run. | Active |
| SOTA-395 | Post-train an autoregressive video generator on its own rollouts, and score the whole rollout | Huang et al. (2025), LIT-629. Training a causal video generator on context it generated itself, with a distribution-level loss over the whole clip, beats teacher-forced and diffusion-forced training under every objective tried (Table 2). It holds quality as the number of autoregressive steps grows, where they degrade. Evidence is VBench on 5s clips, and the exposure-bias mechanism is inferred, not measured. | Proposed |
| SOTA-398 v2 | Pretrain a vision transformer on packed, aspect-preserved images at sampled resolutions, not on fixed square crops | Dehghani et al. (2023), LIT-657. Do not resize or crop images to one square resolution. Keep each image’s aspect ratio and sample its resolution per example, favouring small side lengths. Pack the patches into fixed-length sequences with per-example attention masks. At matched compute this beats square fixed-resolution ViT pretraining, and matches the best ViT with 4x less compute. Most of the gain is throughput. | Proposed |
| SOTA-399 | Choose pre-training weight decay, dropout, clipping and precision for the quantization you intend to ship | Proposed | |
| SOTA-401 | Anchor the patch Gram matrix to an early teacher when training long, because dense features decay while global metrics improve | Proposed | |
| SOTA-402 | To teach a model to reason over facts it holds, spend added data on facts derived from other facts rather than on more atomic facts | Wang et al. (2024), LIT-667. Two sweeps that separate what is normally varied together: with the inferred/atomic ratio φ fixed, scaling the training set changes nothing about how a transformer acquires a two-fact inference rule; with the size effect thus accounted for, raising φ moves the acquisition monotonically, and at φ = 18.0 the delay is gone — 96.7% before training accuracy has even saturated. Synthetic knowledge graphs only, and it buys rule application on facts already seen in compositions, not systematicity. | Proposed |
| SOTA-403 v3 | Share the attention parameters across layers if you need to cut parameters; do not share the feed-forward ones | Lan et al. (2019), LIT-668. A transformer block has two halves and they are not equally compressible across depth. On an ALBERT-base configuration at E = 128, sharing every layer’s attention parameters scores 81.7 average against 81.6 for no sharing while removing 89M parameters to 64M; sharing the feed-forward parameters instead scores 80.2. At E = 768 the same split reads −0.7 against −2.8. One paper, 2019, encoder-only, and nobody has re-run it. | Proposed |
| SOTA-404 v2 | When you ablate an auxiliary loss, ablate the input construction that came with it, or the result is about both | Liu et al. (2019), LIT-671. BERT reported that removing next sentence prediction hurts; RoBERTa observed that the ablation had probably “only removed the loss term while still retaining the segment-pair input format”, separated the two, and found removing the loss matches or slightly improves while the input format was what mattered. LIT-667 is the same defect in another literature: four grokking papers varied the training fraction of a fixed universe, so dataset size and dataset composition moved together, and the conclusion reversed when somebody turned them one at a time. | Proposed |
| SOTA-405 | Scale the masked-language-modelling rate with model size rather than holding it at 15% | Wettig et al. (2022), LIT-672. The masking rate has been held at 15% “regardless of model sizes or masking strategies” since 2018, and the number was never swept. Swept, the optimum moves with capacity: 40% at 354M, 20% at 124M, 15% at 51M. At 354M, 40% wins seven of nine GLUE-plus-SQuAD tasks and reaches the 15% model’s QNLI and QQP scores in half the training time — under an efficient recipe, with the advantage shrinking on longer schedules. | Proposed |
| SOTA-406 | Replace every masked token with [MASK]; drop BERT’s 80-10-10 substitution rule | Wettig et al. (2022), LIT-672. BERT replaced 10% of selected positions with the original token and 10% with a random token, to reduce the pretrain/fine-tune mismatch caused by [MASK] never appearing downstream. Measured against an all-[MASK] baseline at a matched 40% rate, 80-10-10 is worse on everything but SST-2 — and the stated motivation does not bite, because “the model can adapt to full, uncorrupted sentences, regardless of the use of alternative corruption strategies in pre-training”. | Proposed |
| SOTA-408 v3 | Average the weights along the tail of training under a cyclical or high constant learning rate, then re-estimate the normalization statistics | Izmailov et al. (2018), LIT-673. Hold the learning rate high — cyclical or constant — over the last stretch of training and keep a running average of the weights, then ship the average. From pretrained torchvision checkpoints, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at one model’s inference cost. With batch normalization, the running statistics must be recomputed with one pass over the data or the averaged model is broken rather than merely worse. | Proposed |
| SOTA-409 | Average the fine-tuned models from your hyperparameter sweep instead of keeping only the best, adding each one in validation order and only if it helps | Wortsman et al. (2022), LIT-675. A hyperparameter sweep ends by discarding every model but one. Average them instead — sorted by held-out accuracy, keeping each only if held-out accuracy improves, which makes the result no worse than the best individual model on that set by construction. O(1) inference against an ensemble’s O(k). +0.7 pp on CLIP and +0.5 pp on ALIGN over the best sweep member; +0.16 on ImageNet and +0.34 under distribution shift for ViT-G. Averaging all of them uniformly instead can come out worse than the best one. | Active |
| SOTA-411 | To let a model spend more compute at inference without more parameters, train a depth-recurrent core on a randomly sampled iteration count, re-injecting the input at every iteration | Geiping et al. (NeurIPS 2025), LIT-683. Put a small core block between a short prelude and coda. At each training step sample how many times to iterate it, from a heavy-tailed distribution with mean 32. Backpropagate through the last few iterations only. Feed the input embedding into every iteration, and start the state from noise. Trained this way, accuracy rises with iterations at test time and saturates per task. A model trained at fixed depth does not gain this. Nothing yet shows it beats a standard transformer of the same FLOPs. | Proposed |
| SOTA-412 | When the target is likelihood, learn a per-dimension noise schedule conditioned on a learned latent; a scalar or unconditioned one buys nothing | Sahoo et al. (NeurIPS 2024), LIT-677. For density estimation with a pixel-space diffusion model, replace the scalar learned schedule with a per-dimension one conditioned on a small discrete latent that an encoder infers from the image. Keep the endpoints fixed. VDM’s bound is reached in 5× fewer steps on CIFAR-10 and improved at equal steps. Each component alone does nothing: a scalar input-conditioned schedule and a time-only multivariate schedule both match VDM. | Proposed |
| SOTA-413 v2 | In a scaling-law study, count the output head in parameters and FLOPs, and fit loss against compute with an irreducible-loss offset | Pearce and Song (TMLR 2024), LIT-688. At small scale the embedding and output head are a large share of the model. Excluding them from N, and from the FLOPs in C = 6ND, biases the fitted allocation exponent: 0.49 becomes 0.74 when the same runs are relabelled. An offset-free loss–compute fit biases that exponent too. Do not compare exponents across studies that count differently. | Active |
| SOTA-415 v3 | During a warmup-stable-decay run, estimate the annealed score from a uniform average of recent stable-phase checkpoints instead of launching a decay branch | Li et al. (2025), LIT-684, extending Hägele et al. (2024), LIT-145. To see where a WSD run would land if annealed now, evaluate the simple average of its last ~10 stable-phase checkpoints. On the one fork measured, that proxy sits within about 1.5 points of a real 250B-token anneal on each benchmark. It costs no training. Still anneal the model you ship: on that fork the anneal is ahead on MMLU at the end. | Proposed |
| SOTA-417 | Filter pretraining text by how well the tokenizer compresses it | Geiping and Goldstein (2023), LIT-681. Drop every document whose token count exceeds t times its character count. Text a tokenizer trained on the corpus compresses badly is mostly markup, code fragments and debris. No model is needed. On C4 it lifts GLUE from 75.9 to 79.3, and on already-clean bookcorpus-wikipedia from 78.1 to 78.7. The gain tracks how dirty the source was. | Proposed |
| SOTA-419 | Under a fixed compute budget, change the architecture only where it cuts step time at constant parameter count | Geiping and Goldstein (2023), LIT-681. Under a 24-hour, one-GPU budget, every reshape tried (depth 4 to 24, deep-narrow, wide, funnel, FFN-every-k, shared layers) finishes within about 0.1 of the same MLM loss. A smaller model’s extra throughput cancels its slower per-token learning. So the architecture changes that pay are the ones that make each step cheaper without changing how much the model learns per token: dropping biases, predicting only masked tokens, and pre-norm for the learning rate it permits. | Proposed |
| SOTA-420 | When distilling a classifier, match the teacher’s temperature-softened outputs at the same temperature and keep a down-weighted hard-label term | Hinton et al. (2015), LIT-680. Soft-target cross-entropy at temperature T in both teacher and student, plus a hard-label cross-entropy at T = 1 with a considerably lower weight. Multiply the soft term by T², so the balance does not move when T changes. Start low for a small student. The paper’s best realistic setting was T = 2, and a 30-unit MNIST net wanted 2.5–4. | Proposed |
| SOTA-421 | Before fitting a compute-optimal allocation law, tune learning rate, batch size and AdamW beta2 per model size and keep warmup short of the smallest run | Porian et al. (NeurIPS 2024), LIT-690. In a small-scale sweep that will be fitted for N*(C), a warmup longer than the smallest models’ optimal runs moves the exponent by 0.10. So does one batch size, learning rate and β₂ shared across sizes. Tune them per size, with β₂ above 0.95 at small batch, and cap warmup (the source uses warmup tokens = N, at most 20% of the budget). A constant learning rate is enough to fit the exponent, at about half the cost of a per-budget cosine. | Proposed |
| SOTA-422 | Rescale the noise schedule so the last timestep is pure noise, and train with v-prediction so the model can learn there | Lin et al. (WACV 2024), LIT-689, with Emu Video’s comparison, LIT-635. Common VP schedules leave signal at t = T. Stable Diffusion’s leaves √ᾱ_T = 0.068. The model learns to keep the leaked channel mean, and pure-noise inference then limits brightness. Rescale √ᾱ_t so it reaches 0 at T. Then train with v-prediction, because at zero SNR ε-prediction is trivial and v_T = x₀. Sample in the x₀ form, because converting v to ε at zero SNR divides by zero. | Proposed |
| SOTA-424 v4 | Train one network for both conditional and unconditional scores by dropping the condition on 10% of examples, then pick the guidance weight by which metric you are willing to lose | Ho and Salimans (2022), LIT-693. Replace the conditioning with a null token on a fraction of training examples — 0.1 is the measured best, with 0.2 and 0.5 worse — so one network learns both scores, then extrapolate away from the unconditional estimate at sampling. The weight is not a quality dial: on ImageNet 64×64, w from 0.1 to 4.0 takes FID from 1.55 to 26.22 and IS from 66.11 to 260.2. Choose it by which of those you are prepared to sacrifice, and never compare two models at different weights. | Active |
| SOTA-426 | When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either | Lialin et al. (2023), LIT-104 — ReLoRA’s restart discipline, read as NOTE-024. At every merge-and-reinit, prune the adapter’s Adam state (99% by magnitude) and drop the learning rate to zero and re-warm it over 50–100 steps. In the 130M ablation (Table 6) the restart alone does nothing (34.25 vs LoRA’s 34.17), the re-warm without the reset does nothing (34.29), the reset without the re-warm diverges, and only the pair helps (29.77). A rule for how to restart, not a recommendation to restart. | Proposed |
| SOTA-428 | Choose the EMA length after training: store two power-function averages at snapshots and reconstruct any length by least squares | Karras et al. (2023), LIT-714 — EDM2. Track two power-function averages of the weights (σ_rel 0.05 and 0.10), save both every few thousand steps, and synthesize any EMA length after the run from a small linear solve. Then choose it per configuration, per guidance weight and per metric, because on ImageNet-512 the best length moves with all three — at guidance 1.4, FID wants 2% and FD_DINOv2 wants 14%. | Proposed |
| SOTA-429 | Drop half the image patches when training a CLIP-style model, spend the saving on more pairs and a larger batch, and unmask only for a short final tune | Li et al. (2022), LIT-706 — FLIP. Remove 50% of image patches and run the ViT on the rest, with CLIP’s loss and nothing else. The saving buys a 2× larger batch at the same memory and 2× the samples per hour; ViT-L/16 on LAION-400M reaches its unmasked reproduction’s accuracy more than 3× faster. The accuracy comes from what the saving buys: at equal batch, masking is parity. | Proposed |
| SOTA-431 | Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step | Karras et al. (2023), LIT-714 — EDM2. Normalize each output channel’s weight vector on use, re-normalize the stored weights after every step, scale every fixed operation to preserve unit magnitude, and drop the group norms. On the ADM U-Net at ImageNet-512 this takes FID from 6.96 to 2.56 at equal compute, with on-use weight normalization alone worth 6.96 → 3.75. One architecture family; untested on transformers. | Proposed |
| SOTA-432 | Maintain two power-function weight averages during training and snapshot them, so the averaging length is a post-hoc sweep instead of a guess you have to make before the run | Karras et al. (2023), LIT-720. Keep two power-function averages of the weights during training (σ_rel 0.05 and 0.10) and store both in each periodic snapshot; afterwards, reconstruct the average for any averaging length by a least-squares fit over the stored profiles, with error falling as O(1/n⁴) in the snapshot count. The averaging length stops being a pre-training commitment. Worth it because that commitment is expensive in a way that does not show up in its own column: with the length pinned at 13%, the learning-rate decay moves FID by up to 72% across t_ref ∈ [30k, 160k]; swept post hoc, the same bracket is within 10% of the optimum. | Superseded — by SOTA-428; A duplicate of SOTA-428, which was filed from LIT-714 hours earlier and is the same recommendation from the same paper. SOTA-428 is also the better document: it carries the architecture, learning-rate, guidance and metric dependences, where this one had only the first two. The Active/unreplicated pairing this note argued for is a judgement SOTA-428 declined, at Proposed, and that is the one that stands. |
| SOTA-438 | To pretrain a wider language model when a smaller pretrained one of the same depth exists, initialize it by function-preserving symmetric width expansion rather than at random | Samragh et al. (2024), LIT-765 — HyperCloning. Tile each weight matrix of the small model into n×n blocks scaled by 1/n, so the wide model computes exactly the small model’s function at step 0, then train as usual. On three 2× pairs from 350M to 2.9B, it reached the random-init baseline’s final 10-task accuracy 2.2–4× sooner and finished higher. The source model’s pretraining is not charged, so this supports reusing a checkpoint that already exists, not training a small model in order to clone it. | Proposed |
| SOTA-439 | Test a Kronecker-factored curvature implementation against the cases where KFAC is exact before trusting it | Dangel et al. (2025), LIT-768, §5.2 — a tutorial, so the evidence is a derivation, not a training run. KFAC’s averaging step is exact whenever the layer inputs or the backpropagated vectors do not depend on the data point. That gives two tests: a single data point, where KFAC equals the GGN and the empirical Fisher exactly and the Monte-Carlo flavour converges to the GGN; and a deep linear network under square loss on arbitrary data, where the GGN and Monte-Carlo identities still hold and the empirical-Fisher one does not. The second is the one the paper says catches scaling bugs. | Proposed |
| SOTA-444 | When a flow will be sampled with a coarse ODE solver, pair noise and data within each minibatch by exact optimal transport | Pooladian et al. (2023), LIT-789, and Tong et al. (2023), LIT-798. Re-pair each minibatch’s noise and data by exact discrete OT before the flow-matching regression. The marginals stay exact, the learned paths straighten, and on ImageNet-64 reaching FID 20 with Euler takes 12 steps instead of 29, at equal likelihood and 4% more time per iteration. It lowers the step count of an ODE sampler. It does not make one: four-step FID stays near 39 on ImageNet-32. Unconditional, 64 px and below. | Proposed |