Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Model stability — initialization, normalization, gradient handling, regularization, loss-landscape behaviour.
88 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-002 | Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift | Ioffe et al. (2015), ARXIV-1502.03167. Introduces normalization of layer inputs during training. | Active |
| LIT-005 | Layer Normalization | Ba et al. (2016), ARXIV-1607.06450. Introduces normalization across features instead of batch. | Active |
| LIT-014 v2 | Visualizing the Loss Landscape of Neural Nets | Li et al. (2017), ARXIV-1712.09913. Visualizes loss landscape geometry. | Active |
| LIT-015 v2 | Understanding Batch Normalization | Bjorck et al. (2018), ARXIV-1806.02375. Argues that batch normalization’s main benefit is that it permits substantially larger learning rates, and studies what goes wrong without it at initialisation. | Active |
| LIT-022 v2 | Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | Shoeybi et al. (2019), ARXIV-1909.08053. Model parallel transformers. | Active |
| LIT-023 v3 | Root Mean Square Layer Normalization | Zhang et al. (2019), ARXIV-1910.07467. Simplified layer normalization. | Active |
| LIT-025 v2 | Understanding and Improving Layer Normalization | Xu et al. (2019), ARXIV-1911.07013. LayerNorm’s benefit comes from the derivatives of the mean and variance, not from forward normalization — and its bias and gain increase overfitting risk and do not work in most cases. | Active |
| LIT-029 v2 | On Layer Normalization in the Transformer Architecture | Xiong et al. (2020), ARXIV-2002.04745. A duplicate: the migration filed this paper twice, and this copy’s one-line reading attributed RMSNorm to it. LIT-114 is the surviving note. | Rejected — by LIT-114; a duplicate of LIT-114, which holds the same paper and reads it correctly |
| LIT-032 v2 | PowerNorm: Rethinking Batch Normalization in Transformers | Shen et al. (2020), ARXIV-2003.07845. Alternative to LayerNorm. | Active |
| LIT-037 v2 | On the difficulty of training Recurrent Neural Networks | Pascanu et al. (2012), ARXIV-1211.5063. The origin of gradient-norm clipping: exploding gradients are a geometric fact about error surfaces with cliffs, and rescaling the gradient when its norm exceeds a threshold is the cheap, correct response. | Active |
| LIT-043 | Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM | Narayanan et al. (2021), ARXIV-2104.04473. 3D parallel training strategy. | Active |
| LIT-047 v3 | ReZero is All You Need: Fast Convergence at Large Depth | Bachlechner et al. (2020), ARXIV-2003.04887. Gate every residual branch with a single zero-initialized scalar: the network starts as the identity, satisfies dynamical isometry at initialization, and trains at depths where more elaborate schemes are needed otherwise. | Active |
| LIT-054 v2 | GLM-130B: An Open Bilingual Pre-trained Model | Zeng et al. (2022), ARXIV-2210.02414. A 130B training run reported with its failures intact: the paper’s stated purpose is to unveil how a model at this scale can be successfully pre-trained, and most of what it has to say is about loss spikes and divergence. | Active |
| LIT-084 | DeepNet: Scaling Transformers to 1,000 Layers | Wang et al. (2022), ARXIV-2203.00555. Scale-aware normalization. | Active |
| LIT-088 v3 | Scaling Vision Transformers to 22 Billion Parameters | Zhai et al. (2023), ARXIV-2302.05442. Vision model scaling. | Active |
| LIT-114 v2 | On Layer Normalization in the Transformer Architecture | Xiong et al. (2020), ARXIV-2002.04745. Warmup is not a universal law but a fix for where the normalization sits: Post-LN has large gradients near the output at initialization and needs it, Pre-LN does not. | Active |
| LIT-119 | Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale | Falcon-LLM Team (2026), technical blogpost. Holds the parameter budget at 90M and pulls the other three levers — data strategy, architecture, optimizer — one at a time: pretraining from scratch on the target SFT or reasoning mix beats pretrain-then-finetune at this scale, Muon with learnable multipliers beats the AdamW recipe, and depth and SSM width beat MLP width. | Active |
| LIT-139 v2 | DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence | DeepSeek-AI (2026), ARXIV-2606.19348. Two MoE models — V4-Pro at 1.6T/49B and V4-Flash at 284B/13B — with a 1M-token context reached by a hybrid of compressed sparse attention and heavily compressed dense attention, manifold-constrained hyper-connections widening the residual stream, and Muon as the optimizer for most modules over 32–33T tokens; at 1M tokens V4-Pro needs an estimated 27% of V3.2’s per-token FLOPs and 10% of its KV cache. Multi-token prediction is carried over from V3 unchanged, as a training objective; the report does not mention speculative decoding. | Active |
| LIT-154 | 2 OLMo 2 Furious | OLMo Team (2025), ARXIV-2501.00656. A fully open 7B–32B recipe whose stated focus is training stability and per-token efficiency, plus the late-stage specialized data mix that the decay phase is for. | Active |
| LIT-155 | Small-scale proxies for large-scale Transformer training instabilities | Wortsman et al. (2023), ARXIV-2309.14322. Instabilities reported only at frontier scale reappear in small models trained at high learning rate, and the large-scale mitigations work there too — which makes instability something a small budget can study. | Active |
| LIT-198 v2 | Why Low-Precision Transformer Training Fails: An Analysis on Flash Attention | Qiu et al. (2025), ARXIV-2510.04212. The first mechanistic account of why training with flash attention in low precision explodes. Two things have to coincide: attention representations that are low-rank and similar across steps and tokens, and rounding error in BF16 addition that is biased rather than symmetric. The bias acts as a coefficient on the repeated low-rank update, so the errors accumulate instead of cancelling and the spectral norm of weights runs away. A minimal change to flash attention that de-biases the rounding stops it. | Active |
| LIT-200 v3 | PowLU: An Activation Function for Stable Pre-Training of LLMs | Jiang et al. (2026), ARXIV-2605.25704. For large positive inputs SwiGLU approximates x², and that quadratic amplification is what enlarges the output range and produces outliers, particularly in low-precision training. PowLU replaces it with a rational power function giving adaptive nonlinearity and bounded growth. Scaling-law experiments plus 7.9B and 124B Ling models against SwiGLU, and against SwiGLU-Clip at 7.9B only. | Active |
| LIT-223 v2 | How Does Batch Normalization Help Optimization? | Santurkar et al. (2018), ARXIV-1805.11604. “Distributional stability of layer inputs has little to do with the success of BatchNorm” — what it does instead is make the optimization landscape significantly smoother. | Active |
| LIT-251 v2 | The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks | Entezari et al. (2021), ARXIV-2110.06296. Conjectures that most of the loss barrier between two independently trained networks is permutation, not genuine disagreement: account for the hidden-unit symmetry and there would likely be no barrier. The evidence is indirect, and the paper’s own permutation search did not remove the barrier for deep networks. | Active |
| LIT-333 v2 | Git Re-Basin: Merging Models modulo Permutation Symmetries | Ainsworth et al. (2022), ARXIV-2209.04836. Git Re-Basin: three algorithms for finding the permutation that aligns one trained network to another. After alignment their linear interpolation has zero barrier on MNIST and on sufficiently wide ResNets on CIFAR-10. It does not at standard width, and not on ImageNet. | Active |
| LIT-393 | Understanding Dropout | Baldi and Sadowski (2013), NIPS 26:2814-2822. The mathematics behind the weight-scaling rule. For a single logistic unit the normalized weighted geometric mean over all dropout configurations is exactly the unit evaluated at the expected input, so one forward pass with scaled weights computes it — and logistic and constant functions are the only functions with that property. In deep networks the same argument holds through three recursive equations of which only one is an approximation, good to second order and exact when activities are consistent across subnetworks. The same paper derives the adaptive weight-decay term, which is the other account. | Active |
| LIT-394 | Improving neural networks by preventing co-adaptation of feature detectors | Hinton et al. (2012), ARXIV-1207.0580. The three-page note that first states dropout: randomly omit half the feature detectors on each training case, and the overfitting a large network shows on a small training set is greatly reduced. The mechanism it names is co-adaptation — a detector that is only useful in the company of several specific others — and the fix is to make that company unreliable. Everything the record holds about dropout descends from this; the evidence is in LIT-395 two years later. | Active |
| LIT-395 | Dropout: A Simple Way to Prevent Neural Networks from Overfitting | Srivastava et al. (2014), JMLR 15:1929-1958. The definitive dropout paper, and the one that carries the conditions rather than just the technique: a “sweet spot” in dataset size outside which dropout buys nothing in either direction, a 2-3x training-time cost, the weight-scaling rule that makes a single test-time pass approximate the ensemble, and a practical guide (p = 0.5-0.8 hidden, 0.8 input, 10-100x the learning rate, momentum 0.95-0.99, max-norm c = 3-4) whose coupling between those knobs is the part most citations drop. | Active |
| LIT-396 | Dropout Training as Adaptive Regularization | Wager et al. (2013), ARXIV-1307.1493. For generalized linear models, dropout is first-order equivalent to L2 applied after scaling the features by an estimate of the inverse diagonal Fisher information — so the penalty it imposes is data-dependent rather than isotropic, which is what distinguishes it from weight decay. The paper also connects the result to AdaGrad, and turns it into a semi-supervised method that builds the regularizer from unlabeled data. | Active |
| LIT-472 | nGPT: Normalized Transformer with Representation Learning on the Hypersphere | Loshchilov et al. (2024), ARXIV-2410.01131. Normalize every embedding-dimension vector in every matrix to the unit hypersphere after each step, delete all normalization layers, weight decay and LR warmup, and make each block’s residual contribution a learnable per-dimension step size. 4× / 10× / 20× fewer tokens to the same loss at 1k / 4k / 8k context, at 0.5B and 1B on OpenWebText — and 60–80% more time per step. | Active |
| LIT-475 | Dropout Reduces Underfitting | Liu, Xu, Jin, Shen and Darrell (2023), ARXIV-2303.01500. Dropout applied only for the first stretch of training and then switched off lowers training loss and raises test accuracy on models too small to overfit — the regime where standard dropout costs up to 6 points. The mechanism is measured: early dropout reduces the angle between mini-batch gradients and the whole-dataset gradient. | Active |
| LIT-476 | Estimating the Probability of Sampling a Trained Neural Network at Random | Scherlis and Belrose (2025), ARXIV-2501.18812. An estimator for the measure, under the initialization distribution, of the region around a trained network whose behaviour matches it. Negative log of that measure is a description length. A ConvNeXt deliberately trained to generalize badly has a smaller such region — detectable on clean held-out data where its behaviour is indistinguishable. | Active |
| LIT-478 v3 | Neural networks with late-phase weights | von Oswald, Kobayashi, Meulemans, Henning, Grewe and Sacramento (2020), ARXIV-2007.12927. Late in training, replicate a small subset of the weights — BatchNorm’s scale and shift will do — train K copies that share everything else, and average them into one model at the end. Inference cost unchanged. WRN 28-10 on CIFAR-100: 81.35 → 82.87. Starting the same procedure at initialization instead fails to beat the baseline. | Active |
| LIT-519 | Complexity-Guided Component-wise Initialization for Language Model Pretraining | Garbers and Oh (2026), ARXIV-2607.09204 — eleven GPT-2-style checkpoints differing in size, language, tokenizer and corpus share a depth profile: effective-rank entropy falls in the last few blocks, most sharply in the residual-writing matrices W_O and W_down, whose Frobenius norms rise almost linearly with depth. Initializing to imitate that profile changes the spectra and does not improve results. Read as NOTE-262. | Active |
| LIT-520 | The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training | Zhang, Zhou, Jia, Chen and Cheng (2026), ARXIV-2605.26489 — the trace-normalized singular spectrum stops moving long before the weight matrices do. Variation peaks near 10⁻² in the first ~1000 steps, while validation loss falls from ~10 to ~4.0, then settles at a 10⁻⁴ floor that coincides with the slow-descent phase. Holds on GPT-2 and LLaMA, across schedules, weight decays, AdamW and Muon. Read as NOTE-261. | Active |
| LIT-521 | Taming Transformer Without Using Learning Rate Warmup | Qi et al. (2025), ARXIV-2505.21910 — attention entropy collapse comes in two modes, and only one crashes the model: sparse and low-rank is fatal, sparse alone is benign. The driver is spectral energy concentration in W_q^T W_k. Suppressing it by bounding the learning rate with Weyl’s inequality trains ViT, Swin and GPT without warmup, matching warmed-up AdamW on all five configurations. Read as NOTE-266. | Active |
| LIT-522 | Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation | Shi et al. (2026), ARXIV-2605.12492 — instead of bounding or renormalizing a weight matrix’s spectrum, never change it: update W by left and right orthogonal transformations so the singular values are invariant and only the singular vectors rotate. On a 60M LLaMA with every normalization layer removed, AdamW and Muon produce NaNs and Pion trains 9.6B tokens. Read as NOTE-267. | Active |
| LIT-523 | Stabilizing Transformer Training by Preventing Attention Entropy Collapse | Zhai et al. (2023), ARXIV-2303.06296 — proves a tight lower bound on attention entropy that falls like Ω(Tσe^{-σ}) in the spectral norm of W_K W_Q^T, then removes the growth by reparameterizing every linear layer as γ·W/σ(W). A ViT trained this way reaches 82.2% on ImageNet without pre-LN, warmup, weight decay or an adaptive optimizer. Read as NOTE-265. | Active |
| LIT-525 | Transformers Learn Low Sensitivity Functions: Investigations and Implications | Vasudeva et al. (2024), ARXIV-2403.06925 — sensitivity to token-wise random input perturbations is one number that separates transformers from MLPs, CNNs, ConvMixers and LSTMs across vision and language. It also predicts robustness, tracks flatness, can be regularized for, and keeps falling through a grokking plateau while the loss does not move. Read as NOTE-271. | Active |
| LIT-530 | Attention is not all you need: pure attention loses rank doubly exponentially with depth | Dong, Cordonnier and Loukas (2021), ARXIV-2103.03404 — strip the skip connections and MLPs out of a transformer and its output converges doubly exponentially to a rank-1 matrix with identical rows. Skip connections stop it, MLPs slow it, and layer normalization provably does nothing — it is a right-multiplication, and right-multiplication cannot raise rank. Read as NOTE-274. | Active |
| LIT-537 | Grokking as the Transition from Lazy to Rich Training Dynamics | Kumar, Bordelon, Gershman and Pehlevan (2023), ARXIV-2310.06110, ICLR 2024 — the counterexample. A two-layer MLP groks on modular arithmetic with no weight decay at all, and its parameter norm rises through the transition, which the weight-norm accounts cannot produce. The proposed replacement: grokking is the network leaving the lazy (linearised) regime and starting to learn features, controlled by an output-scale parameter α and by the alignment between the initial neural tangent kernel and the target. Read as NOTE-284. | Active |
| LIT-538 | Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets | Power, Burda, Edwards, Babuschkin and Misra (2022), ARXIV-2201.02177 — the paper that named grokking. A small transformer on division mod 97 at 50% data reaches near-perfect training accuracy in under 10³ steps and near-perfect validation accuracy around 10⁶, with almost no generalization before 10⁵. The regime dependence is measured here, not later: near 25–30% data on S₅, removing 1% of the training set raises median time to generalization by 40–50%, while time to fit stays at 10³–10⁴ throughout. Weight decay more than halves the data needed. Read as NOTE-281. | Active |
| LIT-539 | Explaining grokking through circuit efficiency | Varma, Shah, Kenton, Kramár and Kumar (2023), ARXIV-2309.02390, DeepMind — the account that earns its place by predicting things nobody had seen. Memorising and generalising circuits both fit the training set; weight decay prefers whichever produces more logit per unit parameter norm; memorisation gets less efficient as the dataset grows and generalisation does not, so there is a critical dataset size D_crit where the preference flips. From that the paper predicts ungrokking and semi-grokking, and then observes both. Read as NOTE-283. | Active |
| LIT-540 | Omnigrok: Grokking Beyond Algorithmic Data | Liu, Michaud and Tegmark (2022), ARXIV-2210.01117, ICLR 2023 — the paper that answers whether grokking is specific to algorithmic data. No: it is induced on MNIST, IMDb and QM9 by shrinking the training set and scaling the initialization, and eliminated on algorithmic data by constraining the weight norm. The proposed mechanism is the “LU” mismatch — reduced training loss is L-shaped in weight norm and test loss U-shaped, so above the critical norm the two disagree. Read as NOTE-282. | Active |
| LIT-541 | Deep Learning is Singular, and That’s Good | Murfet, Wei, Gong, Li, Gell-Redman and Quella (2020), ARXIV-2010.11560 — the invitation, and the trunk under a school the record has been citing without holding. Neural networks are singular statistical models: the set of optimal parameters is a real analytic variety with singularities, so the Laplace approximation and anything that divides by the determinant of the Hessian do not apply. The correct effective parameter count is the real log canonical threshold λ, which counts how many directions change the model, not how sharply the loss rises in them. Read as NOTE-286. | Active |
| LIT-542 | The Local Learning Coefficient: A Singularity-Aware Complexity Measure | Lau, Furman, Wang, Murfet and Wei (2023), ARXIV-2308.12108 — the instrument. It localizes Watanabe’s learning coefficient to a single minimum, defines it as the exponent in V(ε) = c ε^λ (−log ε)^{m−1} + …, and gives an SGLD-based estimator λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] with β* = 1/log n. Validated against known theoretical values on deep linear networks up to 100M parameters. On ResNet18/CIFAR10 it separates training configurations whose training losses have all collapsed to zero. Read as NOTE-285. | Active |
| LIT-544 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei and Murfet (2023), ARXIV-2310.06301 — the paper that separates two things everyone calls a phase transition. In the Toy Model of Superposition at two hidden dimensions, regular k-gons are critical points with derivable local learning coefficients; the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α] then predicts a Bayesian 5 → 6 transition at n_cr ≈ 600, observed at 600 ≤ n ≤ 700. The same critical points govern SGD plateaus — and the 5 → 6 transition is not one of the dynamical ones. Read as NOTE-288. | Active |
| LIT-548 | Feature Learning in Infinite-Width Neural Networks | Yang and Hu (2020), ARXIV-2011.14522 — Tensor Programs IV, where µP is introduced. Among parametrizations that scale initialization, multipliers and learning rate with width, every stable, nontrivial one either learns features in the wide limit or trains as a kernel, never both. The standard parametrization is stable only at an O(1/width) learning rate, which puts it in the kernel regime. µP is the one that updates every layer maximally, and its maximum stable learning rate does not move with width. | Active |
| LIT-551 | Implicit Neural Representations with Periodic Activation Functions | Sitzmann, Martel et al. (2020), ARXIV-2006.09661. An MLP with sin activations fits images, video, audio and signed distance functions, and, unlike ReLU networks, their derivatives, because a SIREN’s derivative is itself a SIREN. It trains only with a specific initialization: weights U(±√(6/n)) keep every pre-activation near N(0,1), and the first layer is scaled by ω₀ = 30. The initialization’s activation statistics are verified. Its role in earlier periodic networks’ failures is asserted, not ablated. | Active |
| LIT-556 | Tensor Programs III: Neural Matrix Laws | Yang (2020), ARXIV-2009.10685. At random initialization and infinite width, a network’s pre-activations become asymptotically free (in the random-matrix sense) of its weights, for any architecture: the Free Independence Principle. This rigorously justifies the Jacobian singular-value calculations behind dynamical isometry, which had assumed it, and gives a second justification of the gradient independence assumption. The general Master Theorem is also what TP-IV uses. | Active |
| LIT-557 | Tensor Programs II: Neural Tangent Kernel for Any Architecture | Yang (2020), ARXIV-2006.14548. The neural tangent kernel of a randomly initialized network of any standard architecture converges to a deterministic limit as width grows, and the paper shows how to compute it. It also gives a checkable condition (Simple GIA Check) under which the common heuristic that backprop’s Wᵀ is independent of forward W is correct, and cases where that heuristic gives wrong answers. | Active |
| LIT-558 | Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes | Yang (2019), ARXIV-1910.12478. At random initialization, any network built from MLPs, RNNs, convolutions, pooling, skip connections, attention, batchnorm or layernorm converges to a Gaussian process as width grows. This is the NNGP correspondence extended from MLPs and CNNs to all standard architectures, proved by expressing the forward pass in the NETSOR language. The first paper of the series and the tutorial for its machinery. A statement about initialization, not training. | Active |
| LIT-590 | Momentum Contrast for Unsupervised Visual Representation Learning | He et al. (2019), ARXIV-1911.05722. Reframes contrastive learning as dictionary look-up and asks for a dictionary that is both large and consistent: a queue makes it large independently of the batch, and an EMA-updated key encoder makes it consistent. The momentum ablation is the result that propagated — at m=0 it does not train at all. | Active |
| LIT-591 | A Simple Framework for Contrastive Learning of Visual Representations | Chen et al. (2020), ARXIV-2002.05709. Strips contrastive learning to augment, encode, project, NT-Xent — no memory bank, no special architecture — and then ablates each piece. The two findings that outlived the method: the augmentation pair is the pretext task, and the projection head is a sacrificial layer you throw away. | Active |
| LIT-593 | Exploring Simple Siamese Representation Learning | Chen & He (2020), ARXIV-2011.10566. Strips a joint-embedding method to nothing: no negatives, no large batch, no momentum encoder — shared weights, a predictor on one side, a stop-gradient on the other. It works, which makes stop-gradient the candidate for the load-bearing piece, and contradicts BYOL’s account of its own method. | Active |
| LIT-594 | Bootstrap your own latent: A new approach to self-supervised Learning | Grill et al. (2020), ARXIV-2006.07733. Learns a representation with no negatives at all: an online network predicts a slow EMA copy of itself under a different augmentation. 74.3% ImageNet linear. Why it does not collapse is a hypothesis in the paper, not a result, and LIT-593 disputes which part is doing the work. | Active |
| LIT-595 | VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learning | Bardes et al. (2021), ARXIV-2105.04906. Three explicit terms — a hinge keeping each embedding dimension’s standard deviation above a threshold, an invariance term, and a decorrelation term — applied to each branch separately. Needs no weight sharing, batch norm, stop-gradient or memory bank, and the variance term transplants into other methods and improves them. | Active |
| LIT-596 | Barlow Twins: Self-Supervised Learning via Redundancy Reduction | Zbontar et al. (2021), ARXIV-2103.03230. Removes every asymmetry — no predictor, no stop-gradient, no moving average — and prevents collapse in the objective instead: drive the cross-correlation matrix between the two branches’ outputs toward the identity. 73.2% ImageNet linear, and unlike its neighbours it wants a very high-dimensional output. | Active |
| LIT-598 | Unsupervised Learning of Visual Features by Contrasting Cluster Assignments | Caron et al. (2020), ARXIV-2006.09882. Predict one view’s cluster code from another view’s representation instead of comparing features pairwise, with an equipartition constraint doing the anti-collapse work. 75.3% ImageNet linear — and multi-crop, which is the part that transplants into other methods for 2-4 points. | Active |
| LIT-599 v3 | DINOv2: Learning Robust Visual Features without Supervision | Oquab et al. (2023), ARXIV-2304.07193. The thesis is that existing self-supervised objectives already produce general-purpose features if the data is curated enough, so the contribution is a retrieval-based curation pipeline and the engineering to scale it — not a new objective. Frozen features, no finetuning, competitive with the best open weakly-supervised models. | Active |
| LIT-639 | ResiDual: Transformer with Dual Residual Connections | Xie et al. (2023), ARXIV-2304.14802. Gives the representation-collapse argument a rate: in a Pre-LN transformer the per-layer change in the normalised hidden state decays as O(1/√k), and adding a layer to an N−1 layer model moves the output by O(1/√N). Also measures what Pre-LN’s no-warmup claim costs — 32.28 against 35.12 BLEU at E6D6, and 31.82 against 35.18 at E12D12. | Active |
| LIT-640 v3 | Query-Key Normalization for Transformers | Henry, Dachapally, Pawar and Chen (2020), ARXIV-2010.04245. The origin of QK-norm, which thirty documents in this record name and SOTA-192 recommends while sourcing an adopter. ℓ₂-normalize each query and key along the head dimension before the dot product, then scale by a learnable parameter instead of dividing by √d — so the temperature becomes a trained quantity rather than a constant. +0.928 BLEU averaged over five low-resource pairs. | Active |
| LIT-651 | Transformers without Tears: Improving the Normalization of Self-Attention | Nguyen and Salazar (2019), ARXIV-1910.05895. The first systematic evaluation of pre-norm in the base Transformer regime — and it reports a result the record’s universal pre-norm practice does not carry: on high-resource WMT’14 English-German, post-norm wins, 27.58 to 26.83. Also introduces ScaleNorm, the ℓ₂-normalization-with-one-learned-scalar that LIT-640’s QK-norm is an application of. | Active |
| LIT-664 | Emerging Properties in Self-Supervised Vision Transformers | Caron et al. (2021), ARXIV-2104.14294. The origin of the DINO line, which 31 documents here reference and 12 practices touch. Self-supervised ViT features carry explicit semantic segmentation that supervised ViTs and convnets do not show as clearly, and reach 78.3% ImageNet top-1 with a plain k-NN — no finetuning, no linear probe, no augmentation. Collapse is avoided by centering and sharpening the teacher output alone, with no predictor. | Active |
| LIT-668 v2 | ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | Lan, Chen, Goodman, Gimpel, Sharma and Soricut (2019), ARXIV-1909.11942. The measurement behind cross-layer parameter sharing, which this record started recommending — via the fix THEORY-103 endorses — without holding a single paper on it. The sharp result is an asymmetry: sharing the attention parameters across every layer costs +0.1 average at E = 128, and sharing the feed-forward parameters costs −1.4. Also carries a priority claim the record’s dropout practice should have: the first report that dropout hurts a large transformer. | Active |
| LIT-673 | Averaging Weights Leads to Wider Optima and Better Generalization | Izmailov, Podoprikhin, Garipov, Vetrov and Wilson (2018), ARXIV-1803.05407. Average the weights visited along the tail of an SGD trajectory under a cyclical or high constant learning rate. The paper LIT-478 named as the first thing missing from this record’s weight-space ensembling coverage, and the baseline SOTA-288’s promotion condition asks for. From pretrained torchvision models, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at the inference cost of one model. One gotcha, stated in the paper: batch normalization statistics have to be recomputed with a pass over the data. | Active |
| LIT-674 | Robust fine-tuning of zero-shot models | Wortsman, Ilharco, Kim, Li, Kornblith, Roelofs, Gontijo-Lopes, Hajishirzi, Farhadi, Namkoong and Schmidt (2021), ARXIV-2109.01903. Fine-tuning a zero-shot model buys target accuracy and costs robustness. Linearly interpolating the zero-shot and fine-tuned weights recovers both: at α = 0.5, +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model while reference accuracy falls by at most 0.3 pp and usually rises. No extra cost at training or inference. And it names the precondition: averaging all the layers of two unrelated networks “typically fails, achieving no better accuracy than a randomly initialized neural network” — this works because the trajectory is shared. | Active |
| LIT-675 | Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time | Wortsman, Ilharco, Gadre, Roelofs, Gontijo-Lopes, Morcos, Namkoong, Farhadi, Carmon, Kornblith and Schmidt (2022), ARXIV-2203.05482. The dependency SOTA-407’s consensus note named as unheld. A hyperparameter sweep throws away every model but one; average them instead. The greedy recipe — sort by held-out accuracy, add each model only if held-out accuracy improves — is “no worse than the best individual model on the held-out validation set” by construction, at O(1) inference cost against an ensemble’s O(k). +0.7 pp on CLIP and +0.5 pp on ALIGN over the best sweep member. It does not move that practice’s consensus, because it is the same first author. | Active |
| LIT-680 | Distilling the Knowledge in a Neural Network | Hinton, Vinyals and Dean (NIPS 2014 Deep Learning Workshop), ARXIV-1503.02531. This is where the word “distillation” in 112 files of this record comes from. It trains a student on the teacher’s softmax at a raised temperature, adds a down-weighted hard-label term, and scales the soft term by T². On speech, a 10-model ensemble distils into one model of the same size as each member, keeping 86% of the frame-accuracy gain and all of the 0.2-point WER gain. Three things usually quoted from it are not in it: the phrase “dark knowledge”, a high temperature on the realistic task (the best was T = 2), and an ablation of T². | Active |
| LIT-681 | Cramming: Training a Language Model on a Single GPU in One Day | Geiping and Goldstein (ICML 2023), ARXIV-2212.14034. A BERT-style MLM trained from scratch for 24 hours on one consumer GPU, which reaches 79.2–80.4 GLUE against BERT-base’s 80.9. The transferable finding is negative: model shape does not beat the scaling law at this budget. Every reshape it tried lands near the same final loss, because a smaller model’s extra throughput cancels its slower learning. So the only architectural lever is to cut step time at constant parameter count. Its data result is the cheapest filter in the record: drop text the tokenizer compresses badly. | Active |
| LIT-682 | ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models | Defazio (2026), ARXIV-2605.19095, a single-author tech report from the author of LIT-213. Plain Schedule-Free fails at large batch (“a scaling ‘cliff’ at 2M tokens per batch”). Its small weight decay was “compensating for gradient norm drift, in a somewhat hacky way”. ScheduleFree+ fixes both: it restores inner momentum and adds Polyak step sizes with AdamC-style weight decay. From 120M to 2B it beats WSD at every size, and it beats tuned linear decay except at the largest short runs. The ladder changes six things at once, and the one quantified margin, 31%, is for the 120M model only. | Active |
| LIT-684 | Model Merging in Pre-training of Large Language Models | Li et al. (ByteDance Seed, 2025), ARXIV-2505.12082. A uniform average of about ten recent checkpoints on a constant-learning-rate (WSD stable-phase) run beats the latest checkpoint at every size tested, up to 20B/200B MoE and 70B dense. It roughly tracks where an anneal would land. The travelling claim, that constant LR plus merging “can effectively match the performance of an annealed model at any point”, rests on one fork of one 1.3B/13B model. At the end of that fork’s 250B-token decay, the annealed model is about 1.5 MMLU ahead. The baselines in the big tables are unannealed checkpoints. | Active |
| LIT-686 | On the distance between two neural networks and the stability of learning | Bernstein, Vahdat, Yue and Liu (NeurIPS 2020), ARXIV-2002.03432. A deep network’s trust region is not quadratic. The change in its function and Jacobian is bounded by a product of per-layer relative perturbations (“deep relative trust”), and Figure 1 shows loss and gradient breaking down quasi-exponentially in that quantity at depth 16. The optimizer it derives, Fromage, is LARS plus a norm-correcting prefactor. It is the Frobenius antecedent of the modular-norm line. LIT-437, which shares an author, names its conditioning assumption as the flaw and replaces Frobenius with spectral. | Active |
| LIT-700 | Non-Euclidean Gradient Descent Operates at the Edge of Stability | Islamov, Crawshaw, Cohen and Gower (ICML 2026), ARXIV-2603.05002. Steepest descent under any norm — Spectral GD, ℓ∞-descent, Block CD, and the normalized forms SignGD and normalized Spectral GD — also sharpens to 2/η and trains there, if sharpness is measured in the update’s own norm: max_{‖d‖=1} dᵀ∇²L d. The ordinary top Hessian eigenvalue stays well below 2/η for Spectral GD and ℓ∞-descent, so the Euclidean diagnostic misses the regime entirely. Full-batch, CIFAR-10 and Tiny Shakespeare scale; the non-Euclidean sharpness is a multi-restart Frank-Wolfe estimate, not an exact value. | Active |
| LIT-704 | Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning | Gal and Ghahramani (2015; ICML 2016), ARXIV-1506.02142. With dropout before every weight layer and L2 weight decay, the training objective is, for a suitable precision τ and length-scale l, a variational objective whose approximate posterior is Bernoulli-masked weight matrices under a deep Gaussian process. Averaging T stochastic forward passes (“MC dropout”) then gives a predictive mean, and their spread plus τ⁻¹ gives a predictive variance. The equivalence is exact only up to an approximated KL term, and the derivation is in a separate appendix paper. The benchmark comparisons use one-hidden-layer, 50-unit networks on UCI regression. | Active |
| LIT-705 | Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift | Li, Chen, Hu and Yang (2018; CVPR 2019), ARXIV-1801.05134. Dropout changes a unit’s variance between train and test mode, and a batch-norm layer downstream keeps normalizing by the train-mode variance. Dropout 0.5 in every DenseNet bottleneck costs 77.42% → 68.55% on CIFAR-100, and the model then misclassifies its own training data in eval mode. The damage is measured and large. The remedy’s gain is not: one dropout layer after the last BN gives about 0.2 top-1 on ImageNet and mixed, mostly sub-0.3 changes on CIFAR, with no spread reported. | Active |
| LIT-707 | Analysis of dropout learning regarded as ensemble learning | Hara, Saitoh and Shouno (2017), ARXIV-1706.06859. In a teacher-student soft-committee machine (erf units, N = 1000, 100 student units, one fixed set of N inputs reused), dropout at p = 0.5 reaches lower test MSE than an ensemble of two independently trained 50-unit halves. Its residual error is about that of SGD with a tuned L2 penalty. The title says analysis, but no analytic result is derived. Everything is simulation curves averaged over 10 trials, with no numbers or spread reported. | Active |
| LIT-708 | A Controlled Study of Attention-Only Transformers | Ndubuaku et al. (2026), ARXIV-2607.18363. Removing every feed-forward layer from a decoder costs 0.47 nats at matched depth and 0.26 at matched training FLOPs, but only 0.0055 nats (0.27% of loss) at matched parameters once the freed budget goes into attention depth: 20 attention-only layers against 4 standard blocks, 24M parameters, 105B tokens of a synthetic reasoning corpus, with a separate learning-rate sweep for each arm. The remaining gap sits on low-context tokens, where the answer must come from the weights. A pre-registered test on fineweb-edu predicted 0.02–0.05 nats and measured 0.040. Removing QK-norm made the attention-only stack diverge; residual gates and ReZero changed nothing. Everything is at or below 87M parameters. | Active |
| LIT-711 | Do Transformer Modifications Transfer Across Implementations and Applications? | Narang et al. (2021), ARXIV-2102.11972. About fifty published transformer modifications reimplemented in one codebase (T5 / Mesh TensorFlow, 223M encoder-decoder) with hyperparameters held fixed, scored on C4 pre-training loss, SuperGLUE, XSum, WebQuestions and WMT’14 En-De. Most do not help. The ones that do are small (GLU-variant activations, RMSNorm, untied embeddings), cost parameters or time (Switch/MoE, product-key memory, mixture of softmaxes), or were invented in the same codebase. The Universal Transformer, ReZero, Fixup, cross-layer sharing, the Evolved Transformer and lightweight/dynamic convolutions all come out worse than the vanilla baseline. The caveat that travels least: the positive results are also mostly same-codebase — GLU Variants (LIT-030) was run in this codebase at this exact configuration — so the paper is a strong negative test and a weak positive one. | Active |
| LIT-714 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras et al. (2023), ARXIV-2312.02696 — EDM2. Two contributions that do not depend on each other. Rebuilding the ADM U-Net so every layer preserves activation, weight and update magnitude takes ImageNet-512 FID from 8.00 to 2.56 at equal compute, the largest single step (6.96 → 3.75) being weight normalization on the learned layers. And post-hoc EMA: store two power-function averages at snapshots and reconstruct any EMA length after training, which shows the optimal length moving with architecture, training time, learning rate, guidance weight and — by a factor of seven under guidance — with which metric you score it. | Active |
| LIT-733 | Language Models are Unsupervised Multitask Learners | Radford et al. (2019) — GPT-2. Language models trained on WebText begin to perform tasks zero-shot when conditioned on a suitable context, and the performance rises log-linearly with capacity up to the 1.5B model, which still underfits. Its §2.3 carries two architecture changes the field kept: layer norm moved to the input of each sub-block, and residual-layer weights scaled at initialization by 1/sqrt(N), N the number of residual layers, to account for accumulation on the residual path with depth. | Active |
| LIT-749 | bert2BERT: Towards Reusable Pretrained Language Models | Chen et al. (2021), ARXIV-2110.07143 — bert2BERT. Initialize a larger Transformer language model from a smaller pretrained one by Net2Net-style width expansion, then stack layers for depth. Growing a 54M BERT (12 layers, width 512) into BERT-base (width 768) with function-preserving initialization saves 30.4% of pre-training FLOPs at equal GLUE and SQuAD scores. Its recommended variant breaks function preservation by borrowing the next layer’s weights and adds a two-stage schedule, which saves 45.2% on BERT-base and 47% on a GPT-base decoder. | Active |
| LIT-758 | Net2Net: Accelerating Learning via Knowledge Transfer | Chen, Goodfellow and Shlens (2015), ARXIV-1511.05641 — Net2Net, ICLR 2016. Initialize a larger network so it computes exactly the function of a trained smaller one, then train it normally. Net2WiderNet replicates randomly chosen units and divides their outgoing weights by the replication count, and Net2DeeperNet inserts identity layers. On ImageNet Inception-BN, a widened student reaches the randomly initialized network’s final validation accuracy roughly 2×10⁶ minibatches sooner, with no loss of final accuracy. | Active |
| LIT-764 | A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima | Xie et al. (2020), ARXIV-2002.03495 (ICLR 2021). Near a minimum, SGD’s gradient-noise covariance is approximately the Hessian divided by the batch size; feed that into a Kramers escape-time calculation and the mean time to leave a valley is exponential in B/η times the inverse Hessian eigenvalue along the escape direction. So SGD favours flat minima exponentially, where gradient descent with injected white noise favours them only polynomially. Validated as escape rates on test functions and networks of ten hidden units; generalization is not measured. | Active |
| LIT-765 | Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization | Samragh et al. (2024), ARXIV-2409.12903 — HyperCloning. Initialize a wider language model from a smaller pretrained one of the same depth by tiling each weight matrix into n×n blocks scaled by 1/n, so every hidden vector is n stacked copies of the small model’s and the logits match exactly. Against random initialization with every other setting identical, the cloned models reach the random baseline’s final 10-task accuracy 2.2–4× sooner and finish higher, on OPT, Pythia and OLMo targets of 1.3–2.9B. The small model’s own pretraining is not charged to the comparison. | Active |
| LIT-767 | Understanding Deep Learning via Notions of Rank | Razin (2024), ARXIV-2408.02111 — PhD thesis. Gradient descent’s implicit regularization is better read as rank minimization than norm minimization: on a 2×2 matrix completion problem, a depth L ≥ 2 linear network trained by gradient flow drives every norm and quasi-norm to infinity, with probability at least 0.5 over arbitrarily small initialization, while rank falls to its minimum. The same bias toward low (tensor, hierarchical-tensor) rank is proved for polynomial-activation networks equivalent to tensor factorizations; it predicts convolutional networks’ locality, which a regularizer derived from it partly undoes on long-range tasks. | Active |
| LIT-786 | Improved Techniques for Training Consistency Models | Song and Dhariwal, OpenAI (2023), ARXIV-2310.14189. Consistency training from random weights, without a teacher or LPIPS. The teacher network takes no EMA, because with one the limiting objective carries no information about the data. A Pseudo-Huber loss replaces LPIPS, whose ImageNet features can leak into FID. The step count doubles on a schedule, and noise levels are drawn log-normally. One-step FID is 2.51 on CIFAR-10 and 3.25 on ImageNet-64, two-step 2.24 and 2.77, beating consistency distillation. Ablations are CIFAR-10 curves, one run each. | Active |
| LIT-790 | Simplifying, Stabilizing and Scaling Continuous-Time Consistency Models | Lu and Song, OpenAI (2024), ARXIV-2410.11081. Continuous-time consistency models had been unstable. A trigonometric flow parameterization, identity time input with positional embeddings, normalized tangents, learned loss weights and a Flash Attention JVP make them train to 1.5B parameters. Continuous time beats every discrete step count tried. Two-step FID is 2.06 on CIFAR-10, 1.48 on ImageNet-64 and 1.88 on ImageNet-512, against teachers at 2.01, 1.33 and 1.73. Every model, “consistency training” included, starts from the pretrained diffusion model. | Active |