training-optimization.
97 of 378 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-004 | Scale Efficiently | Model shape, not only model size, determines downstream fine-tuning quality — and the widely adopted T5-Base and T5-Large configurations are Pareto-inefficient. Read in full for #114 after its LIT note was found to describe a different paper. | Read |
| NOTE-012 | LARS: Large Batch Training of Convolutional Networks | Linear LR scaling with warm-up is “not general enough and training may diverge”. Layer-wise Adaptive Rate Scaling sets a per-layer rate from the ratio of weight norm to gradient norm, reaching AlexNet at batch 8K and ResNet-50 at batch 32K without accuracy loss. | Read |
| NOTE-015 | Decoupled Weight Decay Regularization | L2 regularization and weight decay are equivalent for SGD and not for adaptive methods. Common Adam implementations do L2 while calling it weight decay; decoupling it recovers the real thing, separates the weight-decay and learning-rate choices, and closes Adam’s generalization gap to SGD with momentum. | Read |
| NOTE-017 v2 | Scaling Laws for Neural Language Models | Loss is a power law in model size, dataset size and compute over seven orders of magnitude, and shape barely matters. Its learning-rate finding is the opposite of what the record recorded: larger models require a smaller rate to avoid divergence, and the paper carries an explicit LR(N) rule. | Read |
| NOTE-024 v2 | ReLoRA: High-Rank Training Through Low-Rank Updates | Restart LoRA repeatedly during pretraining — merge the adapter, reinitialise it, prune the optimizer state, re-warm the learning rate — so a sequence of low-rank updates sums to a high-rank one. The ablation is the finding: the full-rank warm start it also requires accounts for most of the measured gain, and the restart machinery adds 0.42 perplexity on top of it. | Read |
| NOTE-025 | AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs | Keeps an EMA of the gradient’s scalar L2 norm and, when the current gradient is smaller than that history, scales it up to match — a floor under the gradient norm, applied to Adam’s first moment only. The exact mirror image of gradient clipping. Evidence is VGG/ResNet on CIFAR-10, CIFAR-100 and TinyImageNet; there is no transformer and no language result. | Read |
| NOTE-033 | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B | The 3D-parallelism report — 8-way tensor, 35-way pipeline, data parallel over 560 DGX A100 nodes — and an unusually complete training recipe, including a batch-size ramp from 32 to 1920 and β₂ lowered to 0.95 to suppress loss spikes. Also the corpus’s cleanest artefact of pre-Chinchilla misallocation: 530B parameters trained on 270B tokens. | Read |
| NOTE-039 | PaLM 2 Technical Report | Independently re-derives compute-optimal scaling at larger compute than Chinchilla and lands on the same answer — grow data and parameters roughly 1:1, against the earlier 3:1 model-first trend. States the inference-economics corollary plainly: for a fixed training and inference budget, train a smaller model on more tokens rather than making the architecture inference-efficient. | Read |
| NOTE-041 | Population Based Training of Neural Networks | Train a population of models in parallel; periodically let a poor performer copy a good one’s weights (exploit) and perturb its hyperparameters (explore). The output is a hyperparameter schedule rather than a fixed setting, discovered inside a single training run at roughly the cost of the population. | Read |
| NOTE-062 | An Empirical Model of Large-Batch Training | Defines the gradient noise scale — the ratio of the gradient’s variance to its squared norm — and shows it predicts, to an order of magnitude, the batch size past which data parallelism stops buying speed. Also the source of “critical batch size” as a measurable quantity, and of the hyperbola relating steps taken to examples processed. | Read |
| NOTE-066 | Contrastive Learning with Hard Negative Samples | Samples hard negatives without labels by reweighting the contrastive objective rather than changing the sampling procedure — two extra lines of code, no computational overhead — with a tunable hardness knob. The knob exists because harder negatives are more likely to be false negatives, so hardness trades learning signal against contamination. | Read |
| NOTE-072 v2 | Scaling Laws for Autoregressive Generative Modeling | Extends the language scaling laws to image, video, multimodal and mathematics, finding the same power-law-plus-constant form and a nearly domain-independent exponent for optimal model size, N_opt(C) ∝ C^0.7. That exponent implies D ∝ N^0.4 — sub-linear data scaling, which Chinchilla later overturned. The paper also flags the inconsistency that overturned it. | Read |
| NOTE-073 | On the Convergence of Adam and Beyond | Constructs a one-dimensional convex problem on which Adam provably converges to the worst point in the feasible set, locating the error in the original convergence proof and in the exponential moving average itself. Proposes AMSGrad, which keeps the running maximum of the second moment so the effective learning rate is non-increasing. ICLR 2018 best paper; almost nobody runs AMSGrad. | Read |
| NOTE-075 | Hyper-ES | Argues from two lemmas that full-parameter ES cannot work at LLM scale — random perturbations concentrate near orthogonality to any useful descent direction, and the leftover orthogonal components accumulate as a random walk away from the pretrained weights — and replaces the search space with the span of a few cheap GRPO-derived LoRA directions, searched by CMA-ES. | Read |
| NOTE-076 | Understanding Evolution Strategies for LLM Reasoning | Three questions about what ES post-training does rather than whether it wins: it raises pass@1 and pass@k together where GRPO trades one for the other, its 40x larger parameter drift is functionally sparse and does not produce broad forgetting, and the population size needed for stable training falls as the model grows. | Read |
| NOTE-077 | Beyond the Best Guess | Reinforcement learning raises pass@1 and lowers pass@k, eventually below the base model, because it increases the number of prompts solved in none of k samples; evolution strategies raise both and never fall below base at any k or scale tested, up to 32B against published RL checkpoints. | Read |
| NOTE-078 | Neural Thickets | The neighbourhood of a pretrained weight vector is dense with task-improving Gaussian perturbations, the density rises monotonically with model scale, and the perturbations that help are specialists rather than uniform improvements — which together are why a single parallel round of random guessing plus majority voting can match PPO, GRPO and ES at equal training FLOPs. | Read |
| NOTE-079 | EGGROLL, Unrolled | Works out what EGGROLL’s low-rank update converges to — a resolvent-filtered gradient, which can be nonconservative and can turn an attracting optimum into a repelling one, while being exact on every quadratic objective — and turns the finite-population half of the analysis into LOO-ROLL, which replaces two antithetic evaluations per direction with one leave-one-out evaluation at unchanged expected field. | Read |
| NOTE-080 | Evolution Strategies at the Hyperscale | Structuring each ES worker’s perturbation as a rank-r Gaussian product instead of a dense matrix raises arithmetic intensity by enough for a hundredfold throughput gain at billion scale, while the population average stays high-rank and the low-rank update provably approaches the full-rank one at a rate faster than the usual parametric O(1/sqrt(r)). | Read |
| NOTE-081 | Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies | Prior-task degradation under ES is transient drift rather than irreversible forgetting, is not specific to ES, and is driven by the random walk Hoy et al. predicted — whose size falls with population exactly as their equation says. Anchored Weight Decay buys a population-128 reduction at population 30 for 1-2% runtime. | Read |
| NOTE-082 | Evolutionary Strategies lead to Catastrophic Forgetting in LLMs | The first independent replication attempt of ES-beats-GRPO, which fails — ES lands close but GRPO stays ahead on three of four settings at 1B and 1.5B — and the first measurement of what ES costs on a held-out capability: HellaSwag declines steadily as Countdown training continues past the point Countdown itself converged. | Read |
| NOTE-083 | Matching Accuracy, Different Geometry | ES and GRPO reach the same accuracy by travelling in nearly orthogonal directions two orders of magnitude apart in length, and land in the same basin with no loss barrier between them — because most of the ES displacement is a loss-invariant random walk whose squared norm grows as sigma^2 d T / N. | Read |
| NOTE-084 | The Blessing of Dimensionality in LLM Fine-tuning | One geometric property — a fine-tuning landscape whose curvature lives in a few stiff directions amid a near-zero bulk — accounts for both why a population of thirty keeps working as models grow and why the training reward rises, peaks and decays under fixed hyperparameters, in GRPO as well as ES. | Read |
| NOTE-085 | ESSA | The earliest ES-for-LLM paper in this record, and the one that accepted the dimensionality objection instead of overturning it: SFT-train LoRA adapters, take their SVD, and run CMA-ES over the top singular values only — reaching 0.80 on GSM8K in under 25 minutes against GRPO’s 50+, from 100 examples against 5,978, and aligning a 32B model at INT4. | Read |
| NOTE-086 | ESSAM | Transposes sharpness-aware minimization into zeroth order — step against the reward-weighted direction, recompute the update at that neighbouring point, apply it at the original — and closes most of standard ES’s gap to GRPO on GSM8K while keeping inference-level GPU memory. | Read |
| NOTE-087 | Evolution Strategies as a Scalable Alternative to Reinforcement Learning | Shared random seeds let every worker reconstruct every other worker’s perturbation, so an ES update costs one scalar per worker instead of a gradient — which turns 11 hours on 18 cores into 10 minutes on 1,440. The paper also argues that the policy-gradient estimator’s variance grows with episode length while ES’s does not, and that ambient parameter count is not the dimension that matters. | Read |
| NOTE-088 | Proximal Policy Optimization Algorithms | Clip the probability ratio to [1-eps, 1+eps] and take the minimum of the clipped and unclipped surrogate, so the objective is a pessimistic lower bound — which gives trust-region behaviour from first-order optimization, permits several epochs of minibatch SGD on one batch of collected data, and fits in a few lines of change to vanilla policy gradient. | Read |
| NOTE-106 | A Bayesian Perspective on Generalization and Stochastic Gradient Descent | SGD implicitly samples from a Bayesian posterior over parameters, and the effective “temperature” of that posterior is controlled by the noise scale g = εN/B. Increasing batch size B while holding ε fixed cools the posterior, biasing the optimizer toward sharper, less-generalizing minima. | Read |
| NOTE-108 | Neural Tangent Kernel: Convergence and Generalization in Neural Networks | At infinite width, gradient descent on neural networks behaves identically to kernel gradient descent with the NTK. The Jacobian (∂f/∂θ) becomes deterministic and constant, so the NTK Θ = J^T J is a fixed PSD matrix. Loss decays as e^{-λ_min(Θ) t}. This is the “lazy training” or “kernel regime”: parameters barely move from initialization, and the network functions as a linear model in parameter space. | Read |
| NOTE-123 | Bayesian Learning via Stochastic Gradient Langevin Dynamics | SGD with a specific noise injection schedule performs approximate Langevin Monte Carlo. The noise term eps_t ~ N(0, eps_t) counteracts the bias from using stochastic (minibatch) gradients, so the iterates explore the posterior rather than collapsing to a mode. At large step sizes the algorithm behaves like standard SGD (optimization); as step size → 0 the Markov chain mixes and samples from the posterior. | Read |
| NOTE-132 v2 | Large Batch Optimization for Deep Learning: Training BERT in 76 minutes | Scaling the learning rate layerwise by the ratio of the parameter norm to the update norm (inspired by LARS) stabilizes large-batch training across heterogeneous layer curvatures, allowing Adam-like adaptivity to compose with layerwise scaling and generalize across both vision and language models where LARS alone fails. | Read |
| NOTE-144 | Measuring the Effects of Data Parallelism on Neural Network Training | The relationship between batch size and training steps to a goal error always follows the same three-regime shape — perfect scaling (b-fold benefit), diminishing returns, and a maximum useful batch size beyond which adding more parallelism helps nothing — but where these transitions occur is workload-specific and cannot be reliably predicted from simple properties of the model, data, or optimizer. | Read |
| NOTE-176 | Beyond neural scaling laws: beating power law scaling via data pruning | Power-law scaling of error in dataset size is a symptom of redundancy, not a ceiling. With a ranking of example difficulty and a pruning fraction that grows with the corpus, error can fall exponentially instead. Which end of the ranking to discard inverts with data abundance: keep hard examples when data is plentiful, easy ones when it is scarce. | Read |
| NOTE-189 | Modular Duality in Deep Learning | Gradients live in the dual space and weights live in the primal one, so subtracting one from the other is a type error that ordinary gradient descent commits every step. Fix it with a duality map built recursively from per-layer operator norms. Two consequences the record cares about: muP and Shampoo turn out to be partial approximations of one such map, and the map for Linear layers is a Newton-Schulz iteration — which is Muon. | Read |
| NOTE-190 | How much do language models memorize? | Defines memorization as compression rate against the model, separates it from generalization by training on uniform random bitstrings, and measures GPT-style transformer capacity at ~3.6 bits per parameter, linear in parameter count and nearly indifferent to precision. Double descent begins exactly where the data’s information content crosses that capacity. | Read |
| NOTE-191 | Pre-training under infinite compute | Under a fixed 200M-token corpus and no compute limit, the standard recipe overfits; tuning weight decay to ~30x the customary 0.1 makes loss monotone in parameter count. Ensembling independently trained models then beats parameter scaling on asymptote, and distils back down at ~83% retention. | Read |
| NOTE-193 | Power Lines | Scales the AdamW timescale rather than the weight decay, and finds it follows a power law in tokens-per-parameter with exponent about -0.52 over three orders of magnitude of compute. Optimal and critical batch size both scale as power laws in the token budget, not in compute. | Read |
| NOTE-194 | Small Batch Size Training for Language Models | Small batches look unstable because beta_2 is held fixed while the batch changes. Hold the second moment’s half-life fixed in tokens instead and batch size one trains stably, matches larger batches per FLOP, is far more robust to hyperparameters, and lets momentum-free SGD match AdamW at 1.3B. | Read |
| NOTE-195 | How Does Critical Batch Size Scale in Pre-training? | Decouples model size from data size — which every prior batch-size study had scaled together — and finds critical batch size tracks the token budget and is nearly flat in model size. Supported by a muP argument for the model half and a least-squares analysis for the data half. | Read |
| NOTE-199 v2 | How do language models learn facts? | Factual recall is learned in three phases, and the plateau between generic statistics and individual knowledge is the attention recall circuit being built — patching in a trained model’s attention patterns removes the plateau. Imbalanced data shortens the plateau and, the paper argues, slows acquisition after it; a warm-up on a subset of individuals before training on all of them beat the fixed distributions tried, in single-seed runs. | Read |
| NOTE-200 | Scaling Laws for Fact Memorization | Fact capacity is linear in model size and saturates in epochs along a negative exponential; all of Wikidata would want ~1000B non-embedding parameters at 100 epochs. Redundant facts cost full price unless they share direction and structure. Generalisation to unseen facts exists and scales like ordinary pretraining. | Read |
| NOTE-201 | Data Mixing Can Induce Phase Transitions in Knowledge Acquisition | Linear scaling of acquired knowledge in model size holds when a knowledge-dense dataset is trained on alone and breaks once it is mixed into web text: below a critical model size, or a critical mixing ratio, the model memorises almost nothing. Attributed to capacity allocation as a knapsack, with the critical ratio a power law in model size. | Read |
| NOTE-202 | The Limiting Dynamics of SGD | Long after the loss converges the weights keep travelling, distance growing as a power law with a non-trivial exponent. Derives SGD at finite learning rate and batch size as an underdamped Langevin equation and shows the driver is a modified loss plus probability currents, with the motion incoherent oscillation in the Hessian’s top eigensubspace rather than diffusion. | Read |
| NOTE-203 | Central Flows | Models the time-averaged trajectory of an oscillating optimizer as a differential equation that predicts the real path numerically. Reading it: at the edge of stability sharpness sets the step size, not the learning rate; and adaptive optimizers steer away from curvature so they can step further later, which ablating makes them slower. | Read |
| NOTE-204 | Hidden Breakthroughs in Language Model Training | A smooth loss curve is what many differently-timed abrupt transitions look like when averaged. POLCA decomposes loss change per example and along a curvature-derived low-rank basis; on synthetic arithmetic it recovers the carrying skill, which clustering the exact loss curves does not. | Read |
| NOTE-205 | Edge of Stability | Full-batch gradient descent raises the loss Hessian’s top eigenvalue until it hits 2/eta and then trains there. Reading it: the three standard analytical assumptions fail at every reasonable step size, and the curvature-based step-size rule loses to the fixed step size it forbids. | Read |
| NOTE-207 | Contextures: The Mechanism of Representation Learning | A representation is the association between the input and a context variable; capturing its maximum information is optimal on tasks compatible with that context. A context is most useful when the association is neither too strong nor too weak, and mixing contexts is offered as the cheap way to get there. Skimmed: a 313k-character dissertation, read at summary level. | Skimmed |
| NOTE-208 | Muon is Not That Special | An optimizer that replaces the gradient’s singular values with chaotic noise matches Muon on NanoGPT. Reading it: the LMO account of spectral optimizers fails its own control, and what the control leaves standing is that Muon’s optimal step size is constant where Euclidean descent’s is not. | Read |
| NOTE-209 | When Spectral Updates Help | A spectral step beats a Euclidean one on a block when the gradient’s nuclear rank exceeds the stable rank of that block’s incoming activations. Reading it: both sides are measurable, the activation side is provably small in transformers, and the gap grows with dimension. | Read |
| NOTE-211 | Spectral muP under Width-Depth Scaling | Derives the muP spectral condition for joint width-depth scaling and shows the rule is indexed by residual-branch depth. Reading it: Depth-muP and CompleteP are k = 1 and k ≥ 2 of one family, Transformers are the second, and for preconditioned optimizers the whole depth correction is a 1/L residual multiplier. | Read |
| NOTE-219 | Emergent Abilities | The paper that named the phenomenon, read for what it actually argues rather than what it is cited for. It raises the metric explanation itself, in §5.1, and declines it on two grounds — one of which was later answered and one of which still stands. | Read |
| NOTE-220 | Emergence as a Mirage | Reading it: the strongest evidence is not the rebuttal of GPT-3’s arithmetic curve but the vision experiments, where emergence is manufactured in models that had never shown it. The weakest part is what “mirage” gets read as — the paper explicitly declines the claim that emergence is impossible. | Read |
| NOTE-221 | nGPT | Reading it: the headline is 4–20× and the footnote is 60–80% more time per step, so the honest wall-clock figure is roughly 5.5× at 4k. Still large. The part that generalises beyond this architecture is smaller and more interesting — that a matrix’s norm is currently an accident of the learning rate and weight decay, and does not have to be. | Read |
| NOTE-222 | Quantized Evolution Strategies | Reading it: the memory trick is the contribution, not the error feedback. Error feedback is 1-bit SGD’s, correctly credited; what is new is refusing to store the accumulator at all and rebuilding it from seeds, which is the only reason the method fits in the memory budget that motivated it. | Read |
| NOTE-223 | The Coverage Principle | Reading it: the strongest thing here is not the theorem but the Bernoulli example, which shows KL going to infinity while coverage stays fine on a two-outcome model. That is the whole argument in four lines, and it explains why a metric everybody watches can be anti-correlated with what they want. | Read |
| NOTE-224 | Dropout Reduces Underfitting | Reading it: the persuasive number is the training loss, not the accuracy. Early dropout lowers ViT-T’s training loss from 3.443 to 3.394 while standard dropout raises it to 3.885 — so the same operator is doing two opposite things depending only on when it runs, and one of them is not regularization. | Read |
| NOTE-225 | Local Volume | Reading it: the useful result is the audit, not the hypothesis. A network trained to generalize badly is detected on clean held-out data where its behaviour matches a clean model’s, in a small number of forward passes. The volume hypothesis itself is left where it was found — “broadly consistent”, by the authors’ own conclusion. | Read |
| NOTE-226 | NoProp | Reading it: the interesting number is in the train column, not the test one. NoProp matches backprop on test while sitting 8-15 points behind on train, which on MNIST and CIFAR costs nothing and would be the whole story anywhere fitting is the binding constraint. The paper does not mention it. | Read |
| NOTE-227 | Late-phase weights | Reading it: three ablations constrain the recommendation more than the headline does. Starting at initialization fails outright, the fanciest variant is worse than doing nothing, and replicating every weight instead of a few is worse than replicating a few. All three are reported. | Read |
| NOTE-232 | Mixture-of-Transformers | Read to test whether it satisfies SOTA-262’s promotion condition. It does, as that condition was written — second group, per-modality weights against a shared-weight backbone, FLOPs held identical. The component ablation is a bonus the condition did not ask for and the practice needed. | Read |
| NOTE-235 | ParScale | Read as a candidate third scaling axis. The scaling law is fitted honestly and the inference-cost analysis is better instrumented than most; the claim that carries furthest is the asymmetry — code benefits more than general text, on two independent measurements — and it is the claim the paper labels a conjecture. | Read |
| NOTE-244 | SQ hardness for semiautomata, and the regime it needs | Read from the #180 worklist. The theorem is clean and the regime it needs — an alphabet cubic in the number of states — is what a citing reader will drop. Filed for the mental model underneath it, which the record had nothing for: longer random sequences carry less information about a hidden transition structure, not more. | Read |
| NOTE-248 | The FID Lottery: which lottery, and how big | Read from the #180 worklist, taken ahead of its dwell rank because SOTA-305 had just landed on the adjacent question. The finding that matters is not that FID is noisy but which noise dominates: the training run, by 3.2×, in a term no amount of extra sampling touches — and a control that rules out the boring explanation. | Read |
| NOTE-250 | MIMI: the objective that needs the knowledge it assumes away | Read from the #180 worklist in rank order. The headline is a task-agnostic interface score; the durable finding is its failure case, where an assistant doing its job drives the metric negative and the fix is a horizon hyperparameter that requires exactly the task knowledge the objective was built to avoid. | Read |
| NOTE-259 | Refine, then calibrate: the validation-loss minimum is a compromise nobody chose | Read from the #180 worklist and filed because the record had no calibration document at all. A proper loss is calibration error plus refinement error; the two bottom out at different epochs, so stopping on validation loss lands between them. Fit a temperature before reading the loss and the compromise goes away. 196 tabular datasets, three model families, ten runs per vision dataset. | Read |
| NOTE-260 | Block coordinate descent: a smaller machine bought with three times the steps | Read from the #180 worklist as the last of the 5-revisit tier. The method is sound and the accounting needs separating: the headline 2.6% mixes the method with the price of a 4090, the same-device figure is 33%, and the iteration counts in the paper’s own Table 2 show BCD paying about 3× the steps for the memory it saves. | Read |
| NOTE-261 | The singular distribution stops moving long before the weights do | Read for one question: is the spectral shape that three other papers measure a stable property or a snapshot? It is stable, and early. The empirical section was read and the theory that derives the stability bound was not, which is stated rather than implied. | Read |
| NOTE-262 | Eleven checkpoints agree on where the spectrum concentrates, and imitating it does not help | Read for breadth rather than depth of claim. Eleven GPT-2-style checkpoints differing in size, language, tokenizer and corpus share a depth profile in effective-rank entropy, which is the right instrument for a claim about where a matrix’s magnitude sits. The paper’s own headline — that you can initialize from that profile — is a negative result, reported as one. | Read |
| NOTE-264 | Low-rank structure in a transformer is non-uniform, and a global rank is measurably the wrong choice | Read second in the spectral cluster, and it is the one that carries a recommendation. “Transformer weights are low rank” turns out to be true of some matrices and false of others, with the split systematic by component and by depth. The comparison against a uniform rank at matched compression was run and the margins are large. | Read |
| NOTE-265 | A proved bound under the entropy-collapse observation, and a reparameterization that removes four crutches | Read on its own merits after being triaged past once. The record already held the failure this paper is about and two remedies for it, and had neither the inequality that makes the failure inevitable nor this third remedy. The accuracy numbers are nearly flat; what changes is how many hyperparameters have to be right. | Read |
| NOTE-266 | Not all entropy collapse crashes a model, and the discriminator is low-rankness | Read after the paper it contradicts. The contribution is a distinction the record did not have: an attention map can be sparse with near-zero entropy and the run is fine, provided it is not also low-rank. That makes entropy the symptom and spectral energy concentration in the query-key product the cause, and it makes a warmup-free optimizer fall out. | Read |
| NOTE-267 | Freezing the spectrum by construction, and what one 60M run can and cannot show | Read as the third position on one axis. If the problem is that weight spectra drift during training, the options are renormalize them, bound how fast they move, or make them constant. This does the third. The idea is clean and the evidence for the claim worth recommending is one model at 60M, which is why no practice is filed. | Read |
| NOTE-278 | Linear in Newton steps, exponential in gradient steps: a rate is what separates the rivals | Read as the challenge from inside the gradient-descent line’s own setup. Matching each transformer layer to the best number of steps of a candidate algorithm gives a linear trend against Iterative Newton — about 3 iterations per middle layer — and an exponential one against gradient descent; on data with condition number 100 the transformer is unchanged while GD needs 2,000 steps a 12-layer model cannot hold. The method is the contribution: fits cannot separate these rivals and rates can. | Read |
| NOTE-281 | The founding paper already measured the regime dependence | Read because the record staked a claim on grokking being a data-starved-regime phenomenon while holding only the paper that put a number on it. The founding paper measured the dependence too, and reported it as a headline: converged accuracy is flat across a range of training-set sizes while time-to-generalize explodes as the set shrinks. It also names weight decay as the strongest intervention it found, which is the thread every later mechanism pulls on. | Read |
| NOTE-282 | Induced on images, text and molecules, and eliminated on demand | Read to settle whether grokking is specific to algorithmic data. It is not — but producing it on MNIST, IMDb and QM9 takes two deliberate departures from standard practice: a much smaller training set and an inflated initialization scale. At standard initialization there is no grokking on any of them. The control runs both ways: constraining the weight norm nearly eliminates grokking on algorithmic data. | Read |
| NOTE-283 | Two novel phenomena predicted from the theory and then observed | Read as the mechanistic account with the best evidential shape in this cluster. From “weight decay prefers the circuit with more logit per unit norm, and memorisation gets less efficient as the dataset grows” it derives a critical dataset size, and from that derives ungrokking — a grokked network losing its generalization when trained on less data — and semi-grokking. Both were then observed. The mechanism needs weight decay, which is where the counterexample lands. | Read |
| NOTE-284 | A counterexample with no weight decay and a rising weight norm | Read because it is the reason this cluster cannot be filed as a single account. A two-layer MLP groks on modular arithmetic with no weight decay and a parameter norm that goes up through the transition — which neither the LU mechanism nor circuit efficiency can produce, since both turn on the norm coming down. The replacement offered is a transition out of lazy dynamics, with two measurable knobs; what it does not address is ungrokking. | Read |
| NOTE-285 | A complexity measure that discriminates where the training loss is identically zero | Read as the instrument that makes the school usable. The LLC is the volume scaling exponent of the near-optimal set around one minimum, estimated by SGLD at inverse temperature 1/log n. Validated against known theoretical values on deep linear networks to 100M parameters. The result that makes it a practice: on ResNet18/CIFAR10 it separates learning rate, batch size and momentum settings whose training losses have all collapsed to zero. | Read |
| NOTE-286 | The effective parameter count is a volume exponent, and the Hessian is measuring the wrong thing | Read as the trunk under a school the record was already citing. Its single most consequential sentence for this record is that the RLCT matters more than the curvature of the directions it counts — a published argument that the quantity a sharpness visualisation displays is not the quantity that governs generalisation, which is what SOTA-012 rests on. | Read |
| NOTE-288 | The transition the posterior makes and the transition SGD makes are not the same transition | Read for the distinction in its title, which the record needed and did not have. Bayesian phase transitions are changes in where the posterior concentrates as sample size grows, and are well defined; dynamical transitions are changes in the SGD trajectory, and are not. In the one model where both are computed, the sharpest Bayesian transition — predicted at n_cr ≈ 600 and observed at 600–700 — has no dynamical counterpart at all. | Read |
| NOTE-289 | MatFormer | One Transformer with four nested FFN widths, trained one width per step, gives four sizes plus Mix’n’Match sizes nobody trained. At the compute of training the four separately, it matches the largest and beats the smaller ones, which see up to 4× the data through shared weights. Read §1–5 and Appendices B, D.1, E.1 and F. | Read |
| NOTE-292 | Tensor Programs IV | Classifies width-scaling parametrizations (abc-parametrizations) and proves that every stable, nontrivial one either learns features in the wide limit or trains as a kernel, not both. The standard parametrization is stable only at an O(1/width) learning rate, which is the kernel side. Introduces µP as the unique member that updates every layer maximally. Read end to end through §10 and Appendices A, C.1 and D. The proofs in Appendix H were not checked line by line. | Read |
| NOTE-303 | Projected GAN | A GAN discriminator on frozen, randomly mixed, multi-scale pretrained features reaches prior-best FIDs up to 40× faster and sets new FIDs on 22 datasets. Everything is measured in FID or ImageNet-derived metrics, and the discriminator is ImageNet-pretrained, which LIT-563 later shows distorts FID. Read §1–5; the appendix was not read. | Read |
| NOTE-326 | word2vec (negative sampling, subsampling, phrases) | The tricks that made word2vec the standard: negative sampling with a unigram^(3/4) noise distribution, subsampling of frequent words, and data-driven phrase tokens, plus additive compositionality. The 3/4 exponent is asserted without numbers. Read in full. | Read |
| NOTE-334 | Movie Gen | At 5B, flow matching beats v-prediction diffusion on quality by 3.3 annotation σ, but on text alignment by only 1.9σ. Video-native captions beat captions stitched from three frames on alignment by 2.9σ. Every σ in the paper is rater variance, and each ablation arm was trained once. The Llama 3 block beats a DiT block that differs only in norm, activation and bias. The scaling claim rests on image-only runs covering less than a quarter of a decade of compute. | Read |
| NOTE-335 | Self Forcing | Training a few-step autoregressive video model on its own rollouts, with a distribution-matching loss on the whole clip, beats teacher- and diffusion-forced training in matched few-step comparisons (VBench 84.31 against 82.76 and 82.32 chunk-wise), and does not drop when moved to frame-wise generation, where the baselines do. It runs at 17 fps with 0.69s latency on one H100, and its parity with the slower 1.3B base rests on a 14B teacher: the SiD variant, with a 1.3B teacher, scores 84.07 against the base’s 84.26. | Read |
| NOTE-344 | HunyuanVideo | A 13B open text-to-video model whose one quantitative design argument is a pair of compute-optimal fits on a proxy family: the image fit puts more of each added FLOP into parameters (0.5634 against 0.4325), and the video fit, initialized from the image envelope, reverses it (0.3618 against 0.6289). No architectural or data choice is ablated, and the proxy differs from the shipped model in objective, text encoder and conditioning. | Read |
| NOTE-345 | Open-Sora 2.0 | An 11B video model initialized from FLUX, trained mostly at 256px, and adapted to 768px as text/image-to-video. The “$200k” is 4,160 H200 GPU-days for one final run on HunyuanVideo’s VAE. The 4×32×32 Video DC-AE is a separate, unconverged experiment whose cost is outside that figure. Every training-strategy choice that makes the run cheap is asserted, not ablated. | Read |
| NOTE-357 | SEDD | Score entropy makes the ratio view of discrete diffusion trainable and gives it a likelihood bound; with it, absorbing-state diffusion reaches a near-tie with a retrained autoregressive model on LM1B. “Beats GPT-2” is 3 of 5 zero-shot sets against a model trained on different data. The controlled result that lasts is that masking beats uniform corruption on every table. | Read |
| NOTE-359 | Non-Euclidean Edge of Stability | Steepest descent under the ℓ∞, block and spectral norms, and SignGD and normalized Spectral GD, sharpen to 2/η and train there — measured in the update’s own norm, where the ordinary top Hessian eigenvalue stays far below the threshold. Reading it: the identity is exact and the empirics are consistent, but the sharpness is a Frank-Wolfe estimate, the divergence theorem covers one line of initializations, and everything is full-batch. | Read |
| NOTE-360 | MDLM | Masked diffusion’s ELBO is a schedule-weighted average of masked-LM losses, and with a modern recipe it is the best discrete diffusion language model at 110M parameters. It does not beat autoregression in-domain, even with the AR baseline given half the optimizer steps; the paper’s own checklist names that as its limitation. | Read |
| NOTE-361 | DeepSeek-V4 | Two MoE models (1.6T/49B and 284B/13B) reach a native 1M-token context by interleaving two compressed attentions — 4x-compressed KV with top-k sparse selection (CSA) and 128x-compressed dense KV (HCA), each with a 128-token sliding-window branch — at 27% of V3.2’s per-token FLOPs and 10% of its KV cache for Pro at 1M. Multi-token prediction is kept “without modification” from V3 as a depth-1 training objective; the report never mentions speculative decoding or a draft model. | Read |
| NOTE-366 | Primer | Squaring the feed-forward ReLU and adding a causal width-3 depthwise convolution after each per-head Q/K/V projection makes decoder-only LMs reach a vanilla baseline’s quality in 1.75–2.3× less compute at 35M, at least 1.8× at 110M and 4.2× at 537M. That last figure is at fixed size, where the factor grows with training length. At 1.9B, against a GELU baseline, it takes ⅓ of the compute. Both changes cost step time and are paid for in the accounting. Squared ReLU beats SwiGLU in exactly one figure, at 110M. Against the SwiGLU + RMSNorm baseline the saving is about 2×, and it vanishes on encoder–decoder masked LM. | Read |
| NOTE-368 | Attention-only transformers | A necessity test of the feed-forward layer with three matchings. Deleting it in place costs 0.47 nats. At matched FLOPs the cost is 0.26 nats. At matched parameters, with the budget moved into attention depth, the cost is 0.0055 nats, and it shrinks with tokens. What remains is concentrated on low-context tokens, where recall has to come from the weights. QK-norm is what keeps the attention-only stack trainable. At or below 87M parameters, on a synthetic reasoning corpus. | Read |
| NOTE-369 | Min-SNR weighting | Capping the x0-space loss weight at min(SNR, 5) speeds diffusion training under x0, ε and v prediction, and makes ε output trainable under weightings that otherwise diverge. The 3.4× headline is against constant-weighted x0-prediction. Against plain ε-MSE on a UNet, FID goes 8.55 → 7.32 at 200K iterations and 4.21 → 4.14 at 1M, single runs. The multi-task “conflicting gradients” account is motivated by one probe and a proxy objective, and is not tested against FID. | Read |
| NOTE-370 | EDM2 | Magnitude-preserving layers plus forced weight normalization take the ADM U-Net from FID 8.00 to 2.56 on ImageNet-512 at equal compute, in a cumulative single-model ladder whose steps are all far larger than FID’s noise. Post-hoc EMA reconstructs any averaging length from two stored power-function averages, and what it shows is that the best length depends on architecture, learning rate, guidance weight and the metric — the last by 2% against 14% under guidance. | Read |
| NOTE-371 | FLIP | Removing 50% of image patches in CLIP training halves the image encoder’s cost and lets the contrastive batch double at the same memory; ViT-L/16 on LAION-400M then matches its unmasked reproduction more than 3× faster and beats it by about a point at equal epochs. At equal batch the masking gains nothing, and at ViT-B/16 the full-schedule model is 0.2 behind — the improvement is the batch and the extra samples that masking pays for. | Read |