model-stability.
37 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-001 | Understanding and Improving Layer Normalization | LayerNorm’s benefit is in the backward pass — the derivatives of the mean and variance re-center and re-scale the gradients — not in forward normalization. Its bias and gain increase overfitting risk and “do not work in most cases”. | Read |
| NOTE-008 | Visualizing the Loss Landscape of Neural Nets | Filter normalization makes loss-surface plots comparable across architectures, and under it sharpness correlates with generalization error. Deep networks transition from nearly convex to chaotic; skip connections prevent that transition, which is why they are needed at depth. | Read |
| NOTE-013 | DeepNet | Scale the residual by α and initialise the residual branch with gain β, both constants determined only by depth. Bounds the model update theoretically, combines Post-LN’s quality with Pre-LN’s stability, and reaches 1,000 layers. | Read |
| NOTE-022 | RMSNorm | LayerNorm gives re-centering and re-scaling invariance; the hypothesis is that re-centering is dispensable. Normalize by root mean square alone — re-scaling invariance and implicit learning-rate adaptation kept, 7–64% less time per step. | Read |
| NOTE-030 | Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism | Splits a transformer layer across GPUs by choosing the partition that needs no communication in the middle — column-parallel then row-parallel — so a whole layer costs two all-reduces forward and two backward. No compiler, no library changes, a few lines of PyTorch. 8.3B parameters on 512 GPUs at 76% scaling efficiency. | Read |
| NOTE-056 | PowerNorm: Rethinking Batch Normalization in Transformers | Diagnoses why batch normalization underperforms layer normalization in NLP — the batch mean and variance fluctuate far more than in vision — and fixes it by dropping the zero-mean step and normalizing by a running quadratic mean instead of a batch variance. Beats LN by 0.4/0.6 BLEU and 5.6/3.0 perplexity with no hyperparameter changes. | Read |
| NOTE-058 | Scaling Vision Transformers to 22 Billion Parameters | Three architectural changes carry ViT from 4B to 22B — parallel attention/MLP blocks, QK normalization, and removed biases. QK-norm is the load-bearing one: training diverged around 8B from attention logits growing until the softmax was almost one-hot with near-zero entropy, and LayerNorm on the queries and keys before the dot product fixed it. | Read |
| NOTE-116 v2 | Git Re-Basin: Merging Models modulo Permutation Symmetries | Entezari et al. conjecture that SGD solutions can be permuted so that their linear interpolation has no barrier. Git Re-Basin makes this actionable. Given two trained networks θ_A and θ_B, it finds permutations π that minimize ‖vec(θ_A) − vec(π(θ_B))‖², then interpolates θ_A with π(θ_B). The barrier disappears on MNIST and on wide ResNets on CIFAR-10. It does not at 1× width, and not on ImageNet. | Read |
| NOTE-131 v2 | The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks | Conjectures that, for SGD solutions, the loss landscape has essentially one basin modulo permutation symmetry: permute the hidden units of one independently trained network appropriately and there would likely be no barrier on the linear path to another. The support is a theorem for a one-hidden-layer network at random initialization and indirect empirical evidence. The paper’s own search did not find such permutations for deep networks. | Read |
| NOTE-221 | nGPT | Reading it: the headline is 4–20× and the footnote is 60–80% more time per step, so the honest wall-clock figure is roughly 5.5× at 4k. Still large. The part that generalises beyond this architecture is smaller and more interesting — that a matrix’s norm is currently an accident of the learning rate and weight decay, and does not have to be. | Read |
| NOTE-224 | Dropout Reduces Underfitting | Reading it: the persuasive number is the training loss, not the accuracy. Early dropout lowers ViT-T’s training loss from 3.443 to 3.394 while standard dropout raises it to 3.885 — so the same operator is doing two opposite things depending only on when it runs, and one of them is not regularization. | Read |
| NOTE-225 | Local Volume | Reading it: the useful result is the audit, not the hypothesis. A network trained to generalize badly is detected on clean held-out data where its behaviour matches a clean model’s, in a small number of forward passes. The volume hypothesis itself is left where it was found — “broadly consistent”, by the authors’ own conclusion. | Read |
| NOTE-227 | Late-phase weights | Reading it: three ablations constrain the recommendation more than the headline does. Starting at initialization fails outright, the fanciest variant is worse than doing nothing, and replicating every weight instead of a few is worse than replicating a few. All three are reported. | Read |
| NOTE-261 | The singular distribution stops moving long before the weights do | Read for one question: is the spectral shape that three other papers measure a stable property or a snapshot? It is stable, and early. The empirical section was read and the theory that derives the stability bound was not, which is stated rather than implied. | Read |
| NOTE-262 | Eleven checkpoints agree on where the spectrum concentrates, and imitating it does not help | Read for breadth rather than depth of claim. Eleven GPT-2-style checkpoints differing in size, language, tokenizer and corpus share a depth profile in effective-rank entropy, which is the right instrument for a claim about where a matrix’s magnitude sits. The paper’s own headline — that you can initialize from that profile — is a negative result, reported as one. | Read |
| NOTE-265 | A proved bound under the entropy-collapse observation, and a reparameterization that removes four crutches | Read on its own merits after being triaged past once. The record already held the failure this paper is about and two remedies for it, and had neither the inequality that makes the failure inevitable nor this third remedy. The accuracy numbers are nearly flat; what changes is how many hyperparameters have to be right. | Read |
| NOTE-266 | Not all entropy collapse crashes a model, and the discriminator is low-rankness | Read after the paper it contradicts. The contribution is a distinction the record did not have: an attention map can be sparse with near-zero entropy and the run is fine, provided it is not also low-rank. That makes entropy the symptom and spectral energy concentration in the query-key product the cause, and it makes a warmup-free optimizer fall out. | Read |
| NOTE-267 | Freezing the spectrum by construction, and what one 60M run can and cannot show | Read as the third position on one axis. If the problem is that weight spectra drift during training, the options are renormalize them, bound how fast they move, or make them constant. This does the third. The idea is clean and the evidence for the claim worth recommending is one model at 60M, which is why no practice is filed. | Read |
| NOTE-271 | One metric that separates transformers from every other architecture, and what it buys | Read for whether the record wants an inductive-bias claim. It does, because the claim comes with a computable scalar rather than an adjective, and the scalar does four jobs — separates architectures, predicts robustness, correlates with flatness, and moves during a grokking plateau. The fourth is the one this record has an existing use for. | Read |
| NOTE-274 | The trunk under a term the record was using for two different things | Read because four documents in this record lean on “rank collapse” and none of them held its source. Reading it produced a finding that was not the point of the unit: the record carries that term for two distinct phenomena, and the conflation is the field’s rather than ours. | Read |
| NOTE-281 | The founding paper already measured the regime dependence | Read because the record staked a claim on grokking being a data-starved-regime phenomenon while holding only the paper that put a number on it. The founding paper measured the dependence too, and reported it as a headline: converged accuracy is flat across a range of training-set sizes while time-to-generalize explodes as the set shrinks. It also names weight decay as the strongest intervention it found, which is the thread every later mechanism pulls on. | Read |
| NOTE-282 | Induced on images, text and molecules, and eliminated on demand | Read to settle whether grokking is specific to algorithmic data. It is not — but producing it on MNIST, IMDb and QM9 takes two deliberate departures from standard practice: a much smaller training set and an inflated initialization scale. At standard initialization there is no grokking on any of them. The control runs both ways: constraining the weight norm nearly eliminates grokking on algorithmic data. | Read |
| NOTE-283 | Two novel phenomena predicted from the theory and then observed | Read as the mechanistic account with the best evidential shape in this cluster. From “weight decay prefers the circuit with more logit per unit norm, and memorisation gets less efficient as the dataset grows” it derives a critical dataset size, and from that derives ungrokking — a grokked network losing its generalization when trained on less data — and semi-grokking. Both were then observed. The mechanism needs weight decay, which is where the counterexample lands. | Read |
| NOTE-284 | A counterexample with no weight decay and a rising weight norm | Read because it is the reason this cluster cannot be filed as a single account. A two-layer MLP groks on modular arithmetic with no weight decay and a parameter norm that goes up through the transition — which neither the LU mechanism nor circuit efficiency can produce, since both turn on the norm coming down. The replacement offered is a transition out of lazy dynamics, with two measurable knobs; what it does not address is ungrokking. | Read |
| NOTE-285 | A complexity measure that discriminates where the training loss is identically zero | Read as the instrument that makes the school usable. The LLC is the volume scaling exponent of the near-optimal set around one minimum, estimated by SGLD at inverse temperature 1/log n. Validated against known theoretical values on deep linear networks to 100M parameters. The result that makes it a practice: on ResNet18/CIFAR10 it separates learning rate, batch size and momentum settings whose training losses have all collapsed to zero. | Read |
| NOTE-286 | The effective parameter count is a volume exponent, and the Hessian is measuring the wrong thing | Read as the trunk under a school the record was already citing. Its single most consequential sentence for this record is that the RLCT matters more than the curvature of the directions it counts — a published argument that the quantity a sharpness visualisation displays is not the quantity that governs generalisation, which is what SOTA-012 rests on. | Read |
| NOTE-288 | The transition the posterior makes and the transition SGD makes are not the same transition | Read for the distinction in its title, which the record needed and did not have. Bayesian phase transitions are changes in where the posterior concentrates as sample size grows, and are well defined; dynamical transitions are changes in the SGD trajectory, and are not. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600 and observed at 600–700 — has no dynamical counterpart at all. | Read |
| NOTE-292 | Tensor Programs IV | Classifies width-scaling parametrizations (abc-parametrizations) and proves that every stable, nontrivial one either learns features in the wide limit or trains as a kernel, not both. The standard parametrization is stable only at an O(1/width) learning rate, which is the kernel side. Introduces µP as the unique member that updates every layer maximally. Read end to end through §10 and Appendices A, C.1 and D. The proofs in Appendix H were not checked line by line. | Read |
| NOTE-295 | SIREN | Sine-activated MLPs whose derivatives are well behaved, trainable because weights U(±√(6/n)) keep pre-activations standard normal and outputs arcsine at every depth, with ω₀ = 30 on the first layer. They fit signals and derivatives that ReLU, tanh and positional encoding do not. Read §1–4.2 and supplement §1; the later experiments were skimmed. | Read |
| NOTE-359 | Non-Euclidean Edge of Stability | Steepest descent under the ℓ∞, block and spectral norms, and SignGD and normalized Spectral GD, sharpen to 2/η and train there — measured in the update’s own norm, where the ordinary top Hessian eigenvalue stays far below the threshold. Reading it: the identity is exact and the empirics are consistent, but the sharpness is a Frank-Wolfe estimate, the divergence theorem covers one line of initializations, and everything is full-batch. | Read |
| NOTE-361 | DeepSeek-V4 | Two MoE models (1.6T/49B and 284B/13B) reach a native 1M-token context by interleaving two compressed attentions — 4x-compressed KV with top-k sparse selection (CSA) and 128x-compressed dense KV (HCA), each with a 128-token sliding-window branch — at 27% of V3.2’s per-token FLOPs and 10% of its KV cache for Pro at 1M. Multi-token prediction is kept “without modification” from V3 as a depth-1 training objective; the report never mentions speculative decoding or a draft model. | Read |
| NOTE-363 | Dropout as ensemble learning | In a teacher-student soft-committee simulation, dropout at p = 0.5 beats a fixed two-way split of the same network trained as an ensemble, and matches a tuned L2 penalty. That is the whole result: curves averaged over 10 trials, with no derivation, no numbers, one p and one toy model. | Read |
| NOTE-364 | Dropout and BN variance shift | Dropout upstream of batch norm changes a unit’s variance between train and test mode, and BN’s frozen moving variance does not follow. The measured cost is large: CIFAR-100 DenseNet 77.42% → 68.55%, and the model misclassifies its own training data in eval mode. It shrinks with a lower drop rate and with a wider fan-in. Moving dropout after the last BN avoids the cost, but adds only about 0.2 top-1 on ImageNet by itself. | Read |
| NOTE-367 | Narang et al., transformer modifications | About fifty transformer modifications reimplemented in one T5/Mesh TensorFlow codebase at 223M parameters, with hyperparameters fixed, on four task families. Most do not beat the vanilla pre-norm, relative-attention baseline. The winners are GLU-variant activations, RMSNorm, untied embeddings, and sparse experts or other variants that cost parameters or time. The Universal Transformer, ReZero, Fixup, block sharing and the Evolved Transformer all lose. The positives are largely same-codebase re-runs, GLU Variants included, so the paper is strong evidence against the losers and weak evidence that the winners transfer. | Read |
| NOTE-368 | Attention-only transformers | A necessity test of the feed-forward layer with three matchings. Deleting it in place costs 0.47 nats. At matched FLOPs the cost is 0.26 nats. At matched parameters, with the budget moved into attention depth, the cost is 0.0055 nats, and it shrinks with tokens. What remains is concentrated on low-context tokens, where recall has to come from the weights. QK-norm is what keeps the attention-only stack trainable. At or below 87M parameters, on a synthetic reasoning corpus. | Read |
| NOTE-370 | EDM2 | Magnitude-preserving layers plus forced weight normalization take the ADM U-Net from FID 8.00 to 2.56 on ImageNet-512 at equal compute, in a cumulative single-model ladder whose steps are all far larger than FID’s noise. Post-hoc EMA reconstructs any averaging length from two stored power-function averages, and what it shows is that the best length depends on architecture, learning rate, guidance weight and the metric — the last by 2% against 14% under guidance. | Read |
| NOTE-372 | Dropout as a Bayesian approximation | Dropout before every weight layer, with L2, is shown to optimize (up to an approximated KL term) a variational objective for a deep Gaussian process, so running T stochastic forward passes at test time yields a predictive mean and variance for free. The evidence is qualitative figures, a UCI regression table on 50-unit one-layer networks, and one RL run; the MC-versus-weight-scaling comparison is stated without numbers. | Read |