Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Model stability — initialization, normalization, gradient handling, regularization, loss-landscape behaviour.
57 of 449 SOTA documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| SOTA-004 | Place BatchNorm after linear/conv layers but before activation functions | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-005 | Use running statistics for inference | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-006 | Consider alternatives like LayerNorm for transformers | Ioffe et al. (2015), LIT-002 — ARXIV-1502.03167. | Active |
| SOTA-008 v3 | linear warmup of LR stabilizes early training with large batch size. | You et al. (2017), LIT-009 — ARXIV-1708.03888. | Active |
| SOTA-009 v2 | warmup to a large early lr, anneal throughout training to small final lr | Smith et al. (2017), LIT-010 — ARXIV-1708.07120. | Active |
| SOTA-010 v2 | skip connections promote training stability by smoothing out the loss landscape | Li et al. (2017), LIT-014. Moved to THEORY-011 — the claim is a statement about why deep networks became trainable, not an instruction, and belongs in the scheme ADR-031 created for exactly that. This code stays so links into it resolve. | Superseded — by THEORY-011; Not a recommendation, and never was: it states what is true rather than what to do, which is a THEORY under ADR-031 and was one of the eight that decision named and declined to move. The claim is unchanged and is believed — it now lives at THEORY-011, with the figure’s limits and the five practices it underwrites. Superseded rather than Rejected because Rejected on a practice means do not do this, and nobody should read this code as advice against skip connections. ADR-034 is the rule. |
| SOTA-020 | Use larger learning rates with batch normalization | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Active |
| SOTA-021 | Monitor loss landscape smoothness during training | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Rejected — states no threshold and no response, and the smoothing it refers to is a mechanism from a paper the record had conflated with this one. Measuring landscape smoothness during training is not something the practice’s source does or recommends; the finding it rests on is Santurkar et al.’s explanation of why batch normalization works, which is a fact about the technique rather than an instruction |
| SOTA-022 | Place BN after linear/conv layers but before activation functions | Santurkar et al. (2018), LIT-015 — ARXIV-1806.02375. | Superseded — by SOTA-004; a duplicate of SOTA-004, which states the same placement rule from batch normalization’s own paper. The two arrived in the same import from different notes and have coexisted since; SOTA-004 is the one with the originating source |
| SOTA-025 v2 | Initialize the LayerNorm gain to 1 | Ba et al. (2016), LIT-005 — ARXIV-1607.06450. Identity at initialisation: the normalised signal passes through unchanged. A convention rather than a published result, and the record says so. | Active |
| SOTA-026 v2 | Initialize LayerNorm bias to 0 | Ba et al. (2016), LIT-005 — ARXIV-1607.06450. Zero is the identity; a convention rather than a published result, and moot in any model using RMSNorm. | Active |
| SOTA-027 v2 | Use a smaller learning rate for LayerNorm parameters | Xu et al. (2019), LIT-025 — ARXIV-1911.07013. | Rejected — The cited paper contains no discussion of learning rates and argues the parameters in question should be removed rather than tuned. The durable claim — treat norm parameters as their own group — is elsewhere in the record, and is about weight decay rather than rate |
| SOTA-032 v5 | Put the layer normalization inside the residual block, before the sublayer | Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. Pre-LN: normalize the input to each sublayer rather than the sum after it, so the gradients near the output are well behaved at initialization. | Active |
| SOTA-035 | use gradient clipping | Pascanu et al. (2012), LIT-037 — ARXIV-1211.5063. | Active |
| SOTA-050 v2 | Scale attention weights by 1/sqrt(head_dim) | Vaswani et al. (2017), LIT-008 — ARXIV-1706.03762. | Active |
| SOTA-051 v5 | Initialize a residual or adapter branch to exactly zero, not merely near zero | Bachlechner et al. (2020), LIT-047 — ARXIV-2003.04887. Start an added branch at exactly zero so it is the identity at initialisation, and let training raise it. Exact zero rather than small-random is the claim: a small random branch is PROBABLY harmless, a zero one is PROVABLY the identity — which is what matters when the thing being protected is a residual stack deep enough to attenuate, or a pretrained model expensive enough that adapter noise is not worth risking. | Active |
| SOTA-052 | Use smaller variance for deep networks | Wang et al. (2022), LIT-084 — ARXIV-2203.00555. | Active |
| SOTA-053 | Special handling for gated architectures | Bachlechner et al. (2020), LIT-047 — ARXIV-2003.04887. | Active |
| SOTA-060 v4 | Initialize weights from N(0, 0.02) and scale the residual-output projections by 1/sqrt(2N) | Shoeybi et al. (2019), LIT-022 — ARXIV-1909.08053. | Active |
| SOTA-067 v2 | Monitor loss specifically during first ~5000 steps for instabilities | Tay et al. (2021), LIT-052 — ARXIV-2109.10686. Rejected: the source contains no discussion of instability, and the 5000 belongs to nobody. | Rejected — The cited paper contains no discussion of training instability. The record’s sourced monitoring practices are SOTA-069 and SOTA-099, which do not depend on a step count |
| SOTA-069 | Monitor exp(loss) for stability | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-070 | Track gradient norm ratios between layers | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-071 | Use gradient clipping with dynamic threshold | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-072 | Implement early warning system for NaNs | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-098 v2 | Monitor the training loss for unexpected spikes, for the whole run | Chowdhery et al. (2022), LIT-069 — ARXIV-2204.02311. The 540B model’s training loss spiked about 20 times, at irregular intervals, sometimes late, despite gradient clipping, and never in the smaller models. | Active |
| SOTA-099 | Track gradient norm statistics to detect training instabilities | Zeng et al. (2022), LIT-054 — ARXIV-2210.02414. | Active |
| SOTA-100 v2 | Use learning rate warmup proportional to model size | Xiong et al. (2020), LIT-114 — ARXIV-2002.04745. | Active |
| SOTA-161 v4 | Keep the attention output in FP32 during training, because flash attention’s rounding bias compounds instead of cancelling | Qiu et al. (2025), LIT-198 — putting flash attention in BF16 while the FFN goes to FP8 sometimes explodes, and the cause is two things at once: attention produces low-rank updates repeated across steps and tokens, and low-precision addition rounds with a bias. The biased error rides the repeated update and compounds into a systematic gradient bias instead of averaging out. | Active |
| SOTA-182 v2 | Compute the normalization statistic without centering (RMSNorm) | Zhang and Sennrich (2019), LIT-023 — ARXIV-1910.07467. Drop the mean subtraction from layer normalization and rescale by the root mean square alone. | Active |
| SOTA-191 v2 | Consider removing LayerNorm’s learnable gain and bias rather than tuning them | Xu et al. (2019), LIT-025 — the bias and gain increase overfitting risk and “do not work in most cases”; LayerNorm-simple, with both removed, beats LayerNorm on four datasets and reaches state of the art on En-Vi. | Proposed |
| SOTA-192 v9 | Normalize the queries and keys before the attention dot product | Active | |
| SOTA-193 | Lower Adam’s second-moment decay when the loss spikes, before reaching for the learning rate | Proposed | |
| SOTA-217 v4 | Align the hidden-unit permutation before averaging weights from separately trained networks | A neural network’s hidden units can be permuted without changing the function, so two networks trained from different seeds sit in different corners of the same symmetry orbit. Averaging their weights directly averages across that mismatch and traverses a loss barrier that is mostly an artefact of labelling. Match the units first — by weight matching, which needs no data and takes seconds to minutes — and for wide enough networks the barrier is gone. At standard width, and on ImageNet, it shrinks but does not vanish. | Proposed |
| SOTA-240 v4 | Apply dropout where the model can memorize what it is shown, and not where it cannot | Srivastava et al. (2014), LIT-395 — dropout has a “sweet spot” in dataset size and reports both of its edges: no gain at all on data small enough to memorize through the noise, and a declining gain once the data is large enough that overfitting is not the problem. It costs 2-3x training time and it is not a drop-in — the paper’s own recipe pairs it with n/p units, 10-100x the learning rate, momentum 0.95-0.99 and a max-norm constraint. Introduced by Hinton et al. (2012), LIT-394. | Active |
| SOTA-255 v2 | Tune weight decay upward when pretraining over a repeated corpus, rather than inheriting the customary value | Kim et al. (2025), LIT-441 — with the corpus fixed and parameters past Chinchilla-optimal, the weight decay of 0.1 everyone inherits from GPT-3 is far too small: loss turns upward as parameters or epochs are added. Tuned jointly with learning rate and epoch count, the optimum is roughly 30x larger, and loss becomes monotone in parameter count with a steeper exponent than Chinchilla’s. | Proposed |
| SOTA-282 | Put every matrix and hidden state on the unit hypersphere, and make each block’s residual step size a learned per-dimension parameter | Proposed | |
| SOTA-285 | Put dropout at the start of training if the model underfits, and at the end if it overfits — the schedule decides the sign, not the rate | Proposed | |
| SOTA-286 | To find a model that generalizes badly where its outputs look fine, measure how much of parameter space behaves the way it does | Proposed | |
| SOTA-288 | Late in training, replicate a small subset of the weights, train the copies against shared base weights, and average the subset back into one model | Proposed | |
| SOTA-319 | Reparameterize every linear layer by its spectral norm with a learned scalar | Zhai et al. (2023), LIT-523 — replace each linear layer’s W with γ·W/σ(W), γ a learnable scalar initialized to 1 and σ the spectral norm from power iteration. A ViT-B trained this way reaches 82.2% against a DeiT baseline’s 81.8% with no pre-LN, no warmup, no weight decay and LARS in place of Adam. Spectral normalization without the scalar gets 69.81%. | Proposed |
| SOTA-320 | Bound the learning rate by the ratio of the update’s spectral norm to the weight’s, and drop warmup | Qi et al. (2025), LIT-521 — if α_t > τ·σ₁(W_{t−1})/σ₁(∇W_t), truncate the step to that value; otherwise use the schedule. Weyl’s inequality makes this a direct cap on how fast a weight’s largest singular value can grow. ViT-B, ViT-L, GPT-S, Swin-S and Swin-B all train without any warmup and match or beat warmed-up AdamW. | Proposed |
| SOTA-325 | When the training loss has saturated and runs still differ, measure loss-landscape degeneracy rather than curvature | Lau, Furman, Wang, Murfet and Wei (2023), LIT-542 — on ResNet18/CIFAR10, stronger implicit regularization (higher learning rate, lower batch size, higher momentum) gives a lower local learning coefficient and higher test accuracy, while every training loss has collapsed to zero. The estimator is λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] at β* = 1/log n, run by SGLD, validated against theory on deep linear networks to 100M parameters. Where the loss has stopped distinguishing runs, the geometry has not. | Proposed |
| SOTA-332 | When an implicit representation will be supervised through its derivatives, use sine activations with the SIREN initialization rather than a ReLU network | Sitzmann et al. (2020), LIT-551 — a sine network’s derivative is another sine network, so gradients and Laplacians of the fit are well behaved where a ReLU network’s second derivative is zero. Initialize hidden weights U(±√(6/n)) and scale the first layer by ω₀ = 30 (tune it to the signal), or deep sine networks do not train. It fits images, video, SDFs and PDE solutions from derivative supervision. | Proposed |
| SOTA-364 | Break batch normalization’s cross-sample leak before it solves your contrastive task for you | Active | |
| SOTA-365 v2 | Learn without negatives by breaking the symmetry: a predictor on one branch and a stop-gradient on the other | Active | |
| SOTA-366 | Monitor the standard deviation of the l2-normalized embedding; healthy is one over root d and collapsed is zero | Active | |
| SOTA-367 | Or put the anti-collapse constraint in the loss, as a variance floor and a decorrelation term | Active | |
| SOTA-385 | Tighten the gradient-norm clip as the model grows rather than carrying one value across scales | Rae et al. (2021), LIT-617 — the Gopher family clipped the global gradient norm at 1.0 for models up to 1.4B and at 0.25 for the 7.1B and 280B models, “for improved stability”. Three held practices say to clip and none names a value; this is the first number in the record, and it is not a constant. | Proposed |
| SOTA-407 v2 | Interpolate the zero-shot and fine-tuned weights at about half way rather than shipping the fine-tuned model | Wortsman et al. (2021), LIT-674. Fine-tuning a zero-shot model spends the robustness that made it worth starting from. Ship (1 − α)·θ_zero-shot + α·θ_fine-tuned at α = 0.5 instead: +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model, with reference accuracy falling by at most 0.3 pp and usually rising, at no cost in training or inference. Requires that the fine-tuned weights were obtained from the zero-shot weights. | Active |
| SOTA-408 v3 | Average the weights along the tail of training under a cyclical or high constant learning rate, then re-estimate the normalization statistics | Izmailov et al. (2018), LIT-673. Hold the learning rate high — cyclical or constant — over the last stretch of training and keep a running average of the weights, then ship the average. From pretrained torchvision checkpoints, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at one model’s inference cost. With batch normalization, the running statistics must be recomputed with one pass over the data or the averaged model is broken rather than merely worse. | Proposed |
| SOTA-426 | When you merge and reinitialise a low-rank adapter mid-training, prune the optimizer state and re-warm the learning rate from zero — both, not either | Lialin et al. (2023), LIT-104 — ReLoRA’s restart discipline, read as NOTE-024. At every merge-and-reinit, prune the adapter’s Adam state (99% by magnitude) and drop the learning rate to zero and re-warm it over 50–100 steps. In the 130M ablation (Table 6) the restart alone does nothing (34.25 vs LoRA’s 34.17), the re-warm without the reset does nothing (34.29), the reset without the re-warm diverges, and only the pair helps (29.77). A rule for how to restart, not a recommendation to restart. | Proposed |
| SOTA-427 | In a batch-normalized network, put dropout after the last batch-norm layer, not upstream of one | Li et al. (2018; CVPR 2019), LIT-705. BN freezes a moving variance accumulated while dropout was scaling activations. At test time dropout stops, the variance changes, and every later BN normalizes by the wrong constant. The cost is large: 77.42% → 68.55% on CIFAR-100 for DenseNet with dropout 0.5 in each bottleneck. The fix, one dropout layer after the last BN, is supported mainly as avoiding that cost. What it adds on its own is about 0.2 top-1 on ImageNet, with no spread reported. | Proposed |
| SOTA-431 | Make every layer of a diffusion denoiser preserve activation magnitude, and force each weight vector back to fixed norm after every step | Karras et al. (2023), LIT-714 — EDM2. Normalize each output channel’s weight vector on use, re-normalize the stored weights after every step, scale every fixed operation to preserve unit magnitude, and drop the group norms. On the ADM U-Net at ImageNet-512 this takes FID from 6.96 to 2.56 at equal compute, with on-use weight normalization alone worth 6.96 → 3.75. One architecture family; untested on transformers. | Proposed |
| SOTA-432 | Maintain two power-function weight averages during training and snapshot them, so the averaging length is a post-hoc sweep instead of a guess you have to make before the run | Karras et al. (2023), LIT-720. Keep two power-function averages of the weights during training (σ_rel 0.05 and 0.10) and store both in each periodic snapshot; afterwards, reconstruct the average for any averaging length by a least-squares fit over the stored profiles, with error falling as O(1/n⁴) in the snapshot count. The averaging length stops being a pre-training commitment. Worth it because that commitment is expensive in a way that does not show up in its own column: with the length pinned at 13%, the learning-rate decay moves FID by up to 72% across t_ref ∈ [30k, 160k]; swept post hoc, the same bracket is within 10% of the optimum. | Superseded — by SOTA-428; A duplicate of SOTA-428, which was filed from LIT-714 hours earlier and is the same recommendation from the same paper. SOTA-428 is also the better document: it carries the architecture, learning-rate, guidance and metric dependences, where this one had only the first two. The Active/unreplicated pairing this note argued for is a judgement SOTA-428 declined, at Proposed, and that is the one that stands. |
| SOTA-438 | To pretrain a wider language model when a smaller pretrained one of the same depth exists, initialize it by function-preserving symmetric width expansion rather than at random | Samragh et al. (2024), LIT-765 — HyperCloning. Tile each weight matrix of the small model into n×n blocks scaled by 1/n, so the wide model computes exactly the small model’s function at step 0, then train as usual. On three 2× pairs from 350M to 2.9B, it reached the random-init baseline’s final 10-task accuracy 2.2–4× sooner and finished higher. The source model’s pretraining is not charged, so this supports reusing a checkpoint that already exists, not training a small model in order to clone it. | Proposed |
| SOTA-443 | Down-weight large residuals in consistency and MeanFlow training: use a Pseudo-Huber loss or an adaptive inverse-error weight, not plain squared L2 | Song and Dhariwal (2023), LIT-786, and Geng et al. (2025), LIT-784. When the regression target is the model’s own output under stop-gradient, squared L2 trains worse than a loss that limits the pull of large errors. iCT’s Pseudo-Huber beats squared L2 on CIFAR-10 (Fig. 2, curves). MeanFlow’s adaptive weight 1/(‖Δ‖² + c)^p takes one-step ImageNet-256 FID from 79.75 at p = 0 to 61.06 at p = 1 (Table 1e). Two groups, small settings, one run per arm. | Proposed |
| SOTA-446 | In consistency training, take the target from the current weights under stop-gradient, not from an EMA teacher | Song and Dhariwal (2023), LIT-786, §3.2. Consistency Models trained the student against an EMA of itself. iCT shows that on a point-mass data distribution the limiting objective with any teacher other than the student carries no information about the data (Prop. 1), and that setting the teacher’s EMA rate to zero improves CIFAR-10 FID with both LPIPS and squared L2 (Fig. 2a, curves). Keep an EMA of the weights for evaluation. Do not use it for the target. | Proposed |