Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Training optimization — optimizers, learning-rate schedules, batch size, training dynamics, scaling laws and scaling strategies.
193 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-001 | Adam: A Method for Stochastic Optimization | Kingma et al. (2014), ARXIV-1412.6980. Introduces adaptive moment estimation for gradient-based optimization. | Active |
| LIT-006 v2 | SGDR: Stochastic Gradient Descent with Warm Restarts | Loshchilov and Hutter (2016), ARXIV-1608.03983. A duplicate: the migration filed this paper twice, and the other copy carries the reading and the supersession. LIT-042 is the surviving note. | Rejected — by LIT-042; a duplicate of LIT-042, which holds the same paper and the record’s judgement on it |
| LIT-007 | Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour | Goyal et al. (2017), ARXIV-1706.02677. Linear scaling rule for learning rates with batch size. | Active |
| LIT-009 v2 | Large Batch Training of Convolutional Networks | You et al. (2017), ARXIV-1708.03888. Layer-wise adaptive rate scaling (LARS). | Active |
| LIT-010 | Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates | Smith et al. (2017), ARXIV-1708.07120. One-cycle learning rate policy (1cp). | Active |
| LIT-011 | Mixed Precision Training | Micikevicius et al. (2017), ARXIV-1710.03740. Enables FP16 training with FP32 master weights. | Active |
| LIT-012 v2 | Decoupled Weight Decay Regularization | Loshchilov et al. (2017), ARXIV-1711.05101. Fixes weight decay implementation in Adam. | Active |
| LIT-013 v2 | Population Based Training of Neural Networks | Jaderberg et al. (2017), ARXIV-1711.09846. Automated hyperparameter optimization. | Active |
| LIT-017 v2 | An Empirical Model of Large-Batch Training | McCandlish et al. (2018), ARXIV-1812.06162. Introduces gradient noise scale. | Active |
| LIT-018 v2 | On the Convergence of Adam and Beyond | Reddi et al. (2018), ARXIV-1904.09237. Identifies convergence issues in Adam. | Active |
| LIT-028 v5 | Scaling Laws for Neural Language Models | Kaplan et al. (2020), ARXIV-2001.08361. Power-law scaling relationships. | Active |
| LIT-040 v2 | Scaling Laws for Autoregressive Generative Modeling | Henighan et al. (2020), ARXIV-2010.14701. Universal scaling behaviors. | Active |
| LIT-042 v2 | SGDR: Stochastic Gradient Descent with Warm Restarts | Loshchilov and Hutter (2016), ARXIV-1608.03983. Warm restarts: anneal the learning rate on a cosine to a floor, then jump it back up and repeat, with each cycle longer than the last. | Superseded — by LIT-035; single-cycle cosine decay proved better at scale |
| LIT-044 v2 | Contrastive Learning with Hard Negative Samples | Robinson et al. (2020), ARXIV-2010.04592. Importance of negative sample selection. | Rejected — needs per-case tuning, no consistent cross-domain benefit |
| LIT-052 v2 | Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers | Tay et al. (2021), ARXIV-2109.10686. Model shape, not just model size, determines downstream fine-tuning quality; the DeepNarrow strategy reaches T5-Base quality with 50% fewer parameters and 40% faster training. | Active |
| LIT-058 v3 | Measuring the Effects of Data Parallelism on Neural Network Training | Shallue et al. (2018), ARXIV-1811.03600. An exhaustive empirical study of batch size: across models, datasets and optimizers, increasing the batch shortens training in a regime of perfect scaling, then hits diminishing returns and finally a wall, and where those transitions fall is workload-dependent. | Active |
| LIT-061 v3 | Efficient Large Scale Language Modeling with Mixtures of Experts | Artetxe et al. (2021), ARXIV-2112.10684. “A detailed empirical study of how autoregressive MoE language models scale in comparison with dense models” — across in- and out-of-domain language modelling, zero- and few-shot priming, and full fine-tuning. | Active |
| LIT-065 v2 | Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model | Smith et al. (2022), ARXIV-2201.11990. Large scale training techniques. | Active |
| LIT-068 | Training Compute-Optimal Large Language Models | Hoffmann et al. (2022), ARXIV-2203.15556. Compute-optimal scaling laws. | Active |
| LIT-069 | PaLM: Scaling Language Modeling with Pathways | Chowdhery et al. (2022), ARXIV-2204.02311. Pathways system architecture. | Active |
| LIT-090 v2 | Symbolic Discovery of Optimization Algorithms | Chen et al. (2023), ARXIV-2302.06675. Program search over an infinite, sparse space of optimizer programs, which turns up Lion — sign of an interpolated momentum, one state tensor instead of Adam’s two. | Rejected — retired at migration on a description that was not this paper; the reason has not been re-examined |
| LIT-099 v2 | PaLM 2 Technical Report | Anil et al. (2023), ARXIV-2305.10403. Architecture improvements. | Active |
| LIT-101 v3 | AdaNorm: Adaptive Gradient Norm Correction based Optimizer for CNNs | Dubey et al. (2022), ARXIV-2210.06364. Adaptive gradient normalization. | Proposed |
| LIT-104 v4 | ReLoRA: High-Rank Training Through Low-Rank Updates | Lialin et al. (2023), ARXIV-2307.05695. Restart a LoRA adapter repeatedly during pretraining so low-rank updates sum to a high-rank one. The restart needs an optimizer reset and a re-warm together (SOTA-426); the paper’s own ablation shows the full-rank warm start carries most of the gain. | Active |
| LIT-105 | Symbolic Discovery of Optimization Algorithms | Chen et al. (2023), ARXIV-2302.06675. A duplicate: the migration filed this paper twice, once under its real title and once under the name of the optimizer it discovers. LIT-090 is the surviving note. | Rejected — by LIT-090; a duplicate of LIT-090, which holds the same paper |
| LIT-119 | Falcon-H1-Tiny: A series of extremely small, yet powerful language models redefining capabilities at small scale | Falcon-LLM Team (2026), technical blogpost. Holds the parameter budget at 90M and pulls the other three levers — data strategy, architecture, optimizer — one at a time: pretraining from scratch on the target SFT or reasoning mix beats pretrain-then-finetune at this scale, Muon with learnable multipliers beats the AdamW recipe, and depth and SSM width beat MLP width. | Active |
| LIT-121 | Learnable Multipliers: Freeing the Scale of Language Model Matrix Layers | Velikanov et al. (2026), ARXIV-2601.04890. Weight decay and gradient noise settle a matrix layer at an equilibrium norm set by the hyperparameters, not the data; a learnable scalar per matrix and per row and column frees it, generalises µP, and improves downstream results with both Adam and Muon. | Active |
| LIT-122 | Muon is Scalable for LLM Training | Liu et al. (2025), ARXIV-2502.16982. Two changes make Muon work at scale without re-tuning: decoupled weight decay, and scaling each update so its RMS matches what AdamW would produce; scaling-law runs show about 2× the compute efficiency of AdamW, demonstrated on a 3B/16B MoE trained for 5.7T tokens. | Active |
| LIT-127 v2 | DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models | Shao et al. (2024), ARXIV-2402.03300. A 7B model continued-pretrained on 120B math tokens mined from Common Crawl reaches 51.7% on MATH without tools or voting, and introduces Group Relative Policy Optimization — PPO without a critic, with the baseline taken from a group of sampled outputs — the RL algorithm most later reasoning recipes use. | Active |
| LIT-132 | Kimi K2: Open Agentic Intelligence | Kimi Team (2025), ARXIV-2507.20534. A 1T-parameter, 32B-active MoE pretrained on 15.5T tokens with zero loss spikes under MuonClip — Muon plus QK-Clip, which rescales query and key projections whenever the largest attention logit crosses a threshold — then post-trained with large-scale agentic data synthesis and a joint RL stage. | Active |
| LIT-139 v2 | DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence | DeepSeek-AI (2026), ARXIV-2606.19348. Two MoE models — V4-Pro at 1.6T/49B and V4-Flash at 284B/13B — with a 1M-token context reached by a hybrid of compressed sparse attention and heavily compressed dense attention, manifold-constrained hyper-connections widening the residual stream, and Muon as the optimizer for most modules over 32–33T tokens; at 1M tokens V4-Pro needs an estimated 27% of V3.2’s per-token FLOPs and 10% of its KV cache. Multi-token prediction is carried over from V3 unchanged, as a training objective; the report does not mention speculative decoding. | Active |
| LIT-144 | MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategies | Hu et al. (2024), ARXIV-2404.06395. Introduces the Warmup-Stable-Decay schedule: hold the learning rate constant for most of training and decay it sharply over roughly the last 10% of tokens, where most of the loss improvement arrives. Because the decay can start at any point, the token budget need not be fixed in advance, intermediate checkpoints stay useful, and new data can be mixed into the decay stage. | Active |
| LIT-145 | Scaling Laws and Compute-Optimal Training Beyond Fixed Training Durations | Hägele et al. (2024), ARXIV-2405.18392. A constant learning rate with a cooldown scales as predictably as cosine and matches it in quality, so scaling laws can be fitted from checkpoints along one run rather than from a run per duration; stochastic weight averaging along the trajectory improves further at no training cost. | Active |
| LIT-146 | Power Scheduler: A Batch Size and Token Number Agnostic Learning Rate Scheduler | Shen et al. (2024), ARXIV-2408.13359. Under a WSD schedule the optimal learning rate follows a power law in the number of training tokens and the batch size; the Power scheduler sets the rate from that law so one value transfers across token budgets and batch sizes, and combined with µP across model sizes too. | Active |
| LIT-147 | Straight to Zero: Why Linearly Decaying the Learning Rate to Zero Works Best for LLMs | Bergsma et al. (2025), ARXIV-2502.15938. The common recipe decays the learning rate to 10% of its peak; a large empirical study finds that, with the peak tuned, linear decay all the way to zero consistently beats it and other schedules at compute-optimal token budgets, and the paper explains why by reading AdamW as an exponential moving average of updates. | Active |
| LIT-148 | Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfer | Yang and Hu (2022), ARXIV-2203.03466. In the Maximal Update Parametrization the optimal learning rate and other hyperparameters stay put as width grows, so they can be tuned on a small proxy and transferred to the full model zero-shot: BERT-large beaten from a 13M-parameter sweep, GPT-3 6.7B beaten at a tuning cost of 7% of one pretraining run. | Active |
| LIT-149 | u-µP: The Unit-Scaled Maximal Update Parametrization | Blake et al. (2024), ARXIV-2407.17465. Combine µP with Unit Scaling so that activations, weights and gradients start at unit scale regardless of size: lower loss than plain µP, FP8 training out of the box, and hyperparameters that can be swept one at a time, with a learning-rate sweep alone reaching near-optimal loss. | Active |
| LIT-150 | Don’t be lazy: CompleteP enables compute-efficient deep transformers | Dey et al. (2025), ARXIV-2505.01618. Some parametrizations, µP among them, fail to transfer the optimal learning rate across depth and leave deep layers learning lazily; CompleteP (the depth exponent α = 1) transfers across depth and keeps every layer non-lazy, saving 11.8% of FLOPs over µP at optimal shapes and 34.4% at 179 layers, and widening the range of compute-efficient width-to-depth ratios. | Active |
| LIT-153 | Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization | Wen et al. (2026), ARXIV-2606.16899. The follow-up that diagnoses why matrix optimizers’ gains shrink with scale: constant decoupled weight decay fixes the equilibrium weight norm and so fixes the angular learning rate. Pinning the Frobenius norms instead recovers 20–30% over the weight-decay baseline. | Active |
| LIT-155 | Small-scale proxies for large-scale Transformer training instabilities | Wortsman et al. (2023), ARXIV-2309.14322. Instabilities reported only at frontier scale reappear in small models trained at high learning rate, and the large-scale mitigations work there too — which makes instability something a small budget can study. | Active |
| LIT-156 | Fantastic Pretraining Optimizers and Where to Find Them | Wen et al. (2025), ARXIV-2509.02046. Ten optimizers, four scales, tuned fairly and judged at the end of training rather than mid-run: the matrix preconditioners really are fastest, and their advantage over a well-tuned AdamW shrinks with scale — 1.4× at 0.1B to 1.1× at 1.2B. | Active |
| LIT-157 | SOAP: Improving and Stabilizing Shampoo using Adam | Vyas et al. (2024), ARXIV-2409.11321. Shampoo at the 1/2 power is Adafactor run in the eigenbasis of Shampoo’s preconditioner; running Adam there instead gives one extra hyperparameter over Adam and beats both. | Active |
| LIT-158 | A Distributed Data-Parallel PyTorch Implementation of the Distributed Shampoo Optimizer for Training Neural Networks At-Scale | Shi et al. (2023), ARXIV-2309.06497. The engineering that made a second-order optimizer usable at scale: block-diagonal Kronecker preconditioners sharded across GPUs, at most 10% per-step wall-clock over diagonal adaptive methods. | Active |
| LIT-159 | Muon: An optimizer for hidden layers in neural networks | Jordan et al. (2024), blog post. Where Muon comes from: take the momentum matrix for a hidden weight, orthogonalise it with a few Newton–Schulz iterations, and step in that direction — leaving embeddings, the classifier head and the gains and biases to AdamW. | Active |
| LIT-163 v2 | Better & Faster Large Language Models via Multi-token Prediction | Gloeckle et al. (2024), ARXIV-2404.19737. Predict the next n tokens from n heads on a shared trunk: better sample efficiency at no training cost, growing with model size, and up to 3× faster inference as a side-effect. | Active |
| LIT-186 | Scaling Laws for Precision | Kumar et al. (2024), ARXIV-2411.04330. Training in low precision reduces a model’s effective parameter count, and post-training quantization degrades more the longer a model was trained — so past some point extra pretraining data is actively harmful. | Active |
| LIT-211 | Evolution Strategies at Scale: LLM Fine-Tuning Beyond Reinforcement Learning | Qiu et al. (2025), ARXIV-2509.24372. Evolution strategies applied to the full parameter space of an LLM, at billion scale, without dimensionality reduction — which the field had assumed impossible. Against PPO and GRPO on Countdown across Qwen2.5 (0.5B–7B) and LLaMA3 (1B–8B), ES improves over the base model by 36.4% on average against 21.3% for GRPO and 17.9% for PPO, and does it with one fixed hyperparameter set while RL got a sweep per experiment. | Active |
| LIT-213 | The Road Less Scheduled | Defazio et al. (2024), ARXIV-2405.15682. A third answer to the question the schedule chain is about: do not schedule. Iterate averaging and scheduling turn out to be the same mechanism, and the averaged form needs no stopping time T, no schedule shape and no hyperparameter beyond what AdamW already has — while matching schedules that do know T. Won the MLCommons 2024 AlgoPerf Self-Tuning track. | Active |
| LIT-221 | Train With Mixed Precision — NVIDIA deep learning performance guide | NVIDIA, the Train With Mixed Precision user guide. Where the dynamic loss scaling constants everyone runs actually come from: “we successfully trained networks with N = 2000, increasing scaling factor by 2, decreasing scaling factor by 0.5” — reported as one tested configuration, with “many other settings are valid as well.” | Active |
| LIT-229 | Evolution Strategies at the Hyperscale | Sarkar et al. (2025), ARXIV-2511.16652. EGGROLL — Evolution Guided GeneRal Optimisation via Low-rank Learning. Replaces the full-rank Gaussian perturbation each ES worker draws with a rank-r outer product, cutting auxiliary memory per layer from O(mn) to O(r(m+n)) and raising arithmetic intensity enough for a hundredfold throughput gain at billion scale, at up to 91% of pure batch-inference throughput. The population average is still high-rank, and the low-rank update converges to the full-rank one at O(1/r). | Active |
| LIT-230 v3 | Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO | Ba et al. (2026), ARXIV-2608.27351. The first study of what ES post-training DOES rather than whether it wins. Three findings: ES raises pass@1 and pass@k together while GRPO raises pass@1 and falls below the base model on pass@16 and pass@32 in 15 of 18 comparisons; ES’s 40x larger parameter drift is functionally sparse, concentrated in LayerNorm and attention projections, and does not produce broad forgetting; and the population size needed for stable ES falls as the model grows. | Active |
| LIT-231 v2 | Hyper-ES: Effective Evolution Strategies for LLM Reasoning via Descent Direction Merging | Gu et al. (2026), ARXIV-2608.05541. Argues that full-parameter ES on billion-parameter LLMs is ineffective because almost all random perturbations are near-orthogonal to a useful descent direction, and replaces the search space rather than the optimizer: a handful of cheap GRPO runs supply LoRA descent directions, and CMA-ES then searches layer-wise DARE-TIES merging coefficients over their span. Beats GRPO-LoRA by about 1% with 10% fewer gradient updates — on models at 1.5B and below. | Active |
| LIT-232 | EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale | Kaya and Hashemi (2026), ARXIV-2609.10980. Works out what EGGROLL’s low-rank update actually converges to and finds it is a resolvent-filtered gradient, which can be nonconservative and can turn a local optimum into a repelling one — while being exact on every quadratic objective at every rank. The practical yield is LOO-ROLL, which replaces the two antithetic evaluations per direction with one leave-one-out evaluation, halving estimator MSE in transformer blocks at equal cost. | Active |
| LIT-233 v3 | Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights | Gan and Isola (2026), ARXIV-2603.12228. Measures the neighbourhood of pretrained weights and finds it dense with task-improving Gaussian perturbations, with the density rising monotonically with model scale — 0% of perturbations match base accuracy on GSM8K at 0.5B against 64% at 32B — and finds those perturbations to be specialists rather than generalists. The method built on it, RandOpt, post-trains in one parallel step and is competitive with PPO, GRPO and ES at equal training FLOPs, at the cost of K forward passes at inference. | Active |
| LIT-234 v2 | Beyond the Best Guess: Improving LLM Solution Coverage with Evolution Strategies | Hayes et al. (2026), ARXIV-2608.12679. Takes evolution strategies up to 32B and asks one question: what does post-training do to the breadth of the output distribution? RL raises pass@1 and lowers pass@k, eventually below the base model; ES raises both, and never falls below base at any k or scale tested. The mechanism is visible in the accuracy histogram — RL increases the number of prompts solved in none of k samples, putting a hard ceiling on pass@k that no extra sampling can lift. | Active |
| LIT-235 v2 | Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training | Hoy et al. (2026), ARXIV-2604.01499. Resolves the drift dispute by decomposing the ES update: an on-manifold component that changes the loss, and an off-manifold component that is loss-invariant and therefore performs a random walk whose squared norm grows as sigma^2 d T / N. In a landscape with many flat directions the second dominates — which is why ES moves two orders of magnitude further than GRPO, why the two update directions are nearly orthogonal, and why the two solutions are nevertheless linearly connected with no loss barrier. | Active |
| LIT-236 v2 | The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective | Liang et al. (2026), ARXIV-2602.00170. One geometric property accounts for two puzzles at once: fine-tuning landscapes are low-dimensional in curvature, so a handful of stiff directions carry the improvement while a near-zero bulk absorbs variance. That gives rise-then-decay dynamics under fixed hyperparameters — in GRPO as well as ES — and it gives degeneracy, where many ambient perturbations share the same projection onto the stiff subspace, so a population of about 30 keeps finding improvements from 0.5B to 7B with no rightward shift. | Active |
| LIT-237 v2 | Evolutionary Strategies lead to Catastrophic Forgetting in LLMs | Abdi et al. (2026), ARXIV-2601.20861. The first independent attempt to reproduce ES-beats-GRPO, and it does not: ES lands close but GRPO stays ahead on every task but one, at 1B and 1.5B. Its larger contribution is the forgetting curve — held-out HellaSwag falls steadily as Countdown training continues past the point where Countdown itself has converged, tracing a convex Pareto front, with ES parameter drift roughly 1000x GRPO’s and ES updates dense where GRPO’s are about 95% sparse. | Active |
| LIT-238 v2 | Overcoming Forgetting in LLM Fine-Tuning with Evolution Strategies | Schweighofer et al. (2026), ARXIV-2605.30148. Answers the forgetting result with three findings: the degradation is transient drift rather than irreversible forgetting — HellaSwag falls 8% over 300 iterations and returns to baseline by the end — it is not specific to ES, since GRPO forgets too on ProofWriter, and it is driven by the random walk Hoy et al. predicted, whose size falls with population as their equation says. Anchored Weight Decay, an L1 or L2 pull toward the initial weights applied in the update rule, buys a population-128 reduction in drift at population 30 for 1-2% runtime. | Active |
| LIT-239 v2 | ESSA: Evolutionary Strategies for Scalable Alignment | Korotyshova et al. (2025), ARXIV-2507.04453. The earliest paper in this record’s ES line, and it takes the opposite route to LIT-211: shrink the search space rather than search the full one. SFT-train LoRA adapters, take their SVD, and let CMA-ES optimize only the top singular values. On GSM8K with Qwen2.5-Math-7B it passes 0.80 accuracy in under 25 minutes against GRPO’s 50+, on 100 training examples against GRPO’s 5,978 — and it aligns a 32B model at INT4 and INT8, where backpropagation cannot run at all. | Active |
| LIT-240 v2 | ESSAM: A Novel Competitive Evolution Strategies Approach to Reinforcement Learning for Memory Efficient LLMs Fine-Tuning | Sun et al. (2026), ARXIV-2602.01003. Adds a sharpness-aware step to ES: move the parameters against the reward-weighted direction first, recompute the update at that neighbouring point, and apply it at the original — steering the solution toward flat regions. On GSM8K across Qwen2.5 at 0.5B-7B it averages 78.27% against standard ES’s 75.97%, PPO’s 77.72% and GRPO’s 78.34%, at inference-level GPU memory. | Active |
| LIT-241 | Proximal Policy Optimization Algorithms | Schulman, Wolski, Dhariwal, Radford and Klimov (2017), ARXIV-1707.06347. The baseline every paper in this record’s evolution-strategies line measures against, and the algorithm GRPO is a modification of. Its contribution is one objective: clip the probability ratio to [1-eps, 1+eps], take the minimum of clipped and unclipped so the result is a pessimistic lower bound, and you get trust-region behaviour from first-order optimization and a few lines of change. | Active |
| LIT-242 | Evolution Strategies as a Scalable Alternative to Reinforcement Learning | Salimans, Ho, Chen, Sidor and Sutskever (2017), ARXIV-1703.03864. The paper ten of the twelve papers in this record’s evolution-strategies line cite, and the source of the mechanism they all inherit: synchronize random seeds across workers and each one can reconstruct every other’s perturbation, so a parameter update costs one scalar per worker instead of a gradient. 3D humanoid walking in 10 minutes on 1,440 cores, linear speedup, and competitive Atari results in an hour. | Active |
| LIT-245 | Bayesian Learning via Stochastic Gradient Langevin Dynamics | Welling and Teh (2011), Bayesian Learning via Stochastic Gradient Langevin Dynamics. Adding correctly scaled Gaussian noise to a decaying-step-size SGD update turns the optimizer into a posterior sampler, with no accept-reject step. | Active |
| LIT-265 | Large Batch Optimization for Deep Learning: Training BERT in 76 minutes | You et al. (2019), ARXIV-1904.00962. LAMB: normalize each layer’s update by the ratio of weight norm to update norm, which lets BERT train at batch size 32k without per-batch-size retuning. | Active |
| LIT-305 | A Bayesian Perspective on Generalization and Stochastic Gradient Descent | Smith and Le (2017), ARXIV-1710.06451. Generalization is set by a noise scale g = eps*N/B rather than by batch size alone, which is why the learning rate must rise with the batch, and the same quantity predicts an optimal batch size. | Active |
| LIT-360 v2 | Neural Tangent Kernel: Convergence and Generalization in Neural Networks | Jacot et al. (2018), ARXIV-1806.07572. In the infinite-width limit under standard parametrization, training follows a fixed kernel — the network is linear in its parameters around initialization, and no features are learned. | Active |
| LIT-399 | Beyond neural scaling laws: beating power law scaling via data pruning | Sorscher et al. (2022), ARXIV-2206.14486. Power-law scaling in dataset size can be beaten — reduced toward exponential — if you have a good ranking of which examples to discard and you prune harder as the corpus grows. Which end of the ranking to discard inverts with data abundance. | Active |
| LIT-436 | Scalable Optimization in the Modular Norm | Large et al. (2024), ARXIV-2405.14813. Generalises the per-layer natural norm to the whole weight space: the modular norm is defined recursively alongside the architecture. Normalising any base optimizer’s updates in it makes the learning rate transferable across width and depth, with no optimizer-specific scale factors. | Active |
| LIT-437 | A Spectral Condition for Feature Learning | Yang et al. (2023), ARXIV-2310.17813. Feature learning at large width follows from scaling the spectral norm of weights and their updates like sqrt(fan-out/fan-in) — not from the Frobenius-norm and entry-size heuristics in common use. It also yields an elementary derivation of muP. | Active |
| LIT-438 | Modular Duality in Deep Learning | Bernstein and Newhouse (2024), ARXIV-2410.21265. The gradient is a dual vector and should not be subtracted from the weights without a duality map. Modular dualization builds that map recursively from per-layer operator norms; the Linear case is a rectangular Newton-Schulz iteration, and a variant of it set NanoGPT speed records — which is where Muon comes from. | Active |
| LIT-440 | How much do language models memorize? | Morris et al. (2025), ARXIV-2505.24832. Separates what a model stores about a specific dataset from what it stores about the data-generating process, and measures the first by training on uniform random bitstrings where the second is impossible. GPT-style transformers hold about 3.6 bits per parameter, near-independent of precision, and the capacity is linear in parameter count across hundreds of models from 500K to 1.5B. | Active |
| LIT-441 | Pre-training under infinite compute | Kim et al. (2025), ARXIV-2509.14786. Under a fixed corpus and no compute limit, the standard recipe of more epochs and more parameters overfits — and the fix is regularization, at roughly 30x the customary weight decay. Ensembling independently trained models then beats parameter scaling outright, and distilling the ensemble keeps most of the gain. The paper also proposes judging such recipes by their scaling law’s asymptote rather than by loss at a budget. | Active |
| LIT-443 | Power Lines: Scaling Laws for Weight Decay and Batch Size in LLM Pre-training | Bergsma et al. (2025), ARXIV-2505.13738. Hundreds of muP-trained models say the AdamW timescale — not the weight decay itself — is the quantity that scales, and it follows a power law in tokens-per-parameter with exponent about -0.52. Optimal and critical batch size both scale as power laws in the token budget D, independently of model size, which is not what Kaplan’s compute exponent says. | Active |
| LIT-444 | Small Batch Size Training for Language Models: When Vanilla SGD Works, and Why Gradient Accumulation Is Wasteful | Marek et al. (2025), ARXIV-2507.07101. Batch size one trains stably, and the reason small batches looked unstable is that beta_2 was held fixed while the batch changed. Hold the second moment’s half-life fixed in tokens instead and small batches match or beat large ones per FLOP, are far more robust to hyperparameter choice, and let plain SGD without momentum match AdamW at 1.3B. | Active |
| LIT-445 | How Does Critical Batch Size Scale in Pre-training? | Zhang et al. (2024), ARXIV-2410.21676. Decouples model size from data size, which prior batch-size studies had scaled together, and finds critical batch size tracks the token budget and is almost flat in model size. Backed by an argument from muP — past a width, more width does not raise it — and by a least-squares analysis giving the same shape. 85M to 1.2B on C4. | Active |
| LIT-450 v3 | How do language models learn facts? Dynamics, curricula and hallucinations | Zucchet et al. (2025), ARXIV-2503.21676. Factual recall is learned in three phases with a long plateau between generic statistics and individual knowledge, and the plateau is the attention recall circuit forming — shown by patching a reference model’s attention patterns in, at which point the plateau disappears. Imbalanced data shortens the plateau, up to a point, and the paper argues it slows acquisition afterwards; a warm-up on a subset of individuals, then all of them, beat the fixed distributions tried (single seed). | Active |
| LIT-451 | Data Mixing Can Induce Phase Transitions in Knowledge Acquisition | Gu et al. (2025), ARXIV-2505.18091. Knowledge acquired from a knowledge-dense dataset scales linearly with model size when that dataset is trained on alone — and stops doing so once it is mixed into web text. Below a critical mixing ratio the model memorises almost nothing however long it trains; below a critical model size, likewise. The critical ratio follows a power law in model size, so the transitions are predictable. | Active |
| LIT-452 | Scaling Laws for Fact Memorization of Large Language Models | Lu et al. (2024), ARXIV-2406.15720. Fact capacity scales linearly with model size and saturates in training epochs along a negative exponential. Extrapolated, memorising all 15B Wikidata triples wants about 1000B non-embedding parameters trained for 100 epochs. Redundant facts cost full price unless they share direction and structure, so parameters are an expensive place to keep a knowledge base. | Active |
| LIT-453 v2 | Understanding Optimization in Deep Learning with Central Flows | Cohen et al. (2024), ARXIV-2410.24206. The trajectory an optimizer actually takes oscillates at the edge of stability; its time-average is a differential equation — the central flow — that predicts the real trajectory numerically. Reading it: at the edge of stability the learning rate does not set the step size, and adaptive optimizers do not merely adapt to curvature, they steer away from it so they can take larger steps. | Active |
| LIT-454 | The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion | Kunin et al. (2021), ARXIV-2107.09133. Long after the loss has converged the weights keep moving, with distance travelled growing as a power law with a non-trivial exponent. Derives SGD at finite learning rate and batch size as an underdamped Langevin equation and shows what drives the motion is not the training loss but a modified loss plus probability currents that make the trajectory oscillate rather than diffuse randomly. | Active |
| LIT-455 | Hidden Breakthroughs in Language Model Training | Kangaslahti et al. (2025), ARXIV-2506.15872. A smooth loss curve is not evidence of smooth training. POLCA decomposes the change in loss along a low-rank basis of the training subspace and per example, and the breakthroughs the aggregate hides come back — on synthetic arithmetic it recovers the skill of carrying a digit, which clustering the exact loss curves does not. | Active |
| LIT-456 | Muon is Not That Special: Random or Inverted Spectra Work Just as Well | Shumaylov et al. (2026), ARXIV-2605.11181. Replace the gradient’s singular values with chaotic noise and the optimizer still matches Muon. The geometric account of why spectral optimizers work does not survive its own control; what survives is that Muon’s optimal step size is constant. | Active |
| LIT-457 v2 | When do spectral gradient updates help in deep learning? | Davis and Drusvyatskiy (2025), ARXIV-2512.04299. A spectral update beats a Euclidean one on a block exactly when the gradient’s squared nuclear-to-Frobenius ratio exceeds the stable rank of that block’s incoming activations — a measurable condition, and they show it is satisfied throughout transformer training. | Active |
| LIT-459 | Contextures: The Mechanism of Representation Learning | Zhai (2025), ARXIV-2504.19792 — a CMU dissertation. A representation is learned from the association between the input and a context variable; an encoder capturing the maximum information of that association is optimal on tasks compatible with the context. A context is most useful when the association is neither too strong nor too weak, and mixing contexts is a cheap way to get there. | Active |
| LIT-461 | Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability | Cohen et al. (2021), ARXIV-2103.00065. Full-batch gradient descent drives the maximum Hessian eigenvalue up until it reaches 2/eta, then trains there: the loss goes non-monotone over short timescales and keeps falling over long ones. The paper the record has been treating as established premise without holding it. | Active |
| LIT-462 v2 | Spectral Condition for μP under Width-Depth Scaling | Zheng et al. (2026), ARXIV-2603.00541. Extends the spectral condition for muP from width to width-and-depth, and finds the rule is set by how many transformations a residual branch contains: one gives Depth-muP, two or more gives CompleteP. Transformers are the second case. | Active |
| LIT-470 | Emergent Abilities of Large Language Models | Wei et al. (2022), ARXIV-2206.07682. Names and defines emergent abilities — present in larger models, absent in smaller ones, and therefore not predictable by extrapolating a scaling law. A survey of existing curves rather than new experiments: eight few-shot examples across five model families, plus prompting strategies that only help past a threshold. Section 5.1 raises the metric explanation itself and argues against it on two specific grounds. | Active |
| LIT-471 | Are Emergent Abilities of Large Language Models a Mirage? | Schaeffer et al. (2023), ARXIV-2304.15004. A sharp capability curve can be produced from a smooth one by the metric alone. Rescoring fixed GPT-3 outputs under Token Edit Distance removes the emergence; >92% of hand-annotated BIG-Bench emergent abilities sit under two metrics, one nonlinear and one discontinuous; and emergence is induced on demand in vision models that had never shown it. | Active |
| LIT-472 | nGPT: Normalized Transformer with Representation Learning on the Hypersphere | Loshchilov et al. (2024), ARXIV-2410.01131. Normalize every embedding-dimension vector in every matrix to the unit hypersphere after each step, delete all normalization layers, weight decay and LR warmup, and make each block’s residual contribution a learnable per-dimension step size. 4× / 10× / 20× fewer tokens to the same loss at 1k / 4k / 8k context, at 0.5B and 1B on OpenWebText — and 60–80% more time per step. | Active |
| LIT-473 | Quantized Evolution Strategies: High-precision Fine-tuning of Quantized LLMs at Low-precision Cost | Xu, Miikkulainen and Qiu (2026), ARXIV-2602.03120. Fine-tune a quantized model in its own integer space: accumulate the part of each evolution-strategies update that is smaller than the lattice spacing until it crosses a grid point, and rematerialize that accumulator from stored seeds so it costs kilobytes instead of an FP16 copy of the weights. Countdown, Qwen2.5 at 1.5B and 3B, INT4/INT8/W8A8. | Active |
| LIT-474 | The Coverage Principle: How Pre-Training Enables Post-Training | Chen, Huang, Golowich, Malladi, Block, Ash, Krishnamurthy and Foster (2025), ARXIV-2510.15020. Names the quantity cross-entropy is a bad proxy for: the coverage profile, the probability mass a model puts on rare high-quality responses. Proves it is necessary and sufficient for Best-of-N to succeed, that next-token prediction optimizes it faster than it optimizes cross-entropy, and that sequence-level KL carries a linear dependence on sequence length that coverage does not. | Active |
| LIT-475 | Dropout Reduces Underfitting | Liu, Xu, Jin, Shen and Darrell (2023), ARXIV-2303.01500. Dropout applied only for the first stretch of training and then switched off lowers training loss and raises test accuracy on models too small to overfit — the regime where standard dropout costs up to 6 points. The mechanism is measured: early dropout reduces the angle between mini-batch gradients and the whole-dataset gradient. | Active |
| LIT-476 | Estimating the Probability of Sampling a Trained Neural Network at Random | Scherlis and Belrose (2025), ARXIV-2501.18812. An estimator for the measure, under the initialization distribution, of the region around a trained network whose behaviour matches it. Negative log of that measure is a description length. A ConvNeXt deliberately trained to generalize badly has a smaller such region — detectable on clean held-out data where its behaviour is indistinguishable. | Active |
| LIT-477 | NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation | Li, Teh and Pascanu (2025), ARXIV-2503.24322. Train each block independently to denoise a noisy label embedding given the raw input — diffusion machinery pointed at a classifier. No forward or backward pass across the network at training time, about half the GPU memory, and test accuracy matching backprop on MNIST and CIFAR-10. Every block sees the input, which is what makes independence possible and what makes the comparison awkward. | Active |
| LIT-478 v3 | Neural networks with late-phase weights | von Oswald, Kobayashi, Meulemans, Henning, Grewe and Sacramento (2020), ARXIV-2007.12927. Late in training, replicate a small subset of the weights — BatchNorm’s scale and shift will do — train K copies that share everything else, and average them into one model at the end. Inference cost unchanged. WRN 28-10 on CIFAR-100: 81.35 → 82.87. Starting the same procedure at initialization instead fails to beat the baseline. | Active |
| LIT-483 | Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models | Liang et al. (2024), ARXIV-2411.04996. Untie every non-embedding parameter by modality — feed-forward, attention projections, layer norms — and keep global self-attention over the whole sequence. Matches a dense baseline at 55.8% of the FLOPs in the Chameleon setting and 37.2% with speech added. The component ablation is the useful part: the feed-forward carries most of it, attention adds less, layer norms add nothing. | Active |
| LIT-486 | Parallel Scaling Law for Language Models | Chen et al. (2025), ARXIV-2505.10475. Run the same weights over P learnably-prefixed copies of the input and aggregate the outputs: the loss falls as if the parameter count had risen by O(log P). Fitted across 0.5B–4.4B and P = 1…8 on two corpora. At batch size 1, matching a parameter-scaled model costs 22× less added memory and 6× less added latency — and P× the training FLOPs, which is the real price. | Active |
| LIT-495 | On the Statistical Query Complexity of Learning Semiautomata: a Random Walk Approach | Giapitzakis, Fountoulakis, Nichani and Lee (2025), ARXIV-2510.04115. The first Statistical Query hardness result for semiautomata, and the first where the hardness comes from the transition structure alone rather than from a hard language embedded in the machine. Two distinct machines in the constructed family agree at chance after O(N² ln N) random symbols, so the signal separating them vanishes exponentially in sequence length. The regime it needs is unusual and belongs beside the result: an alphabet of size Ω(N³ ln N). | Active |
| LIT-501 | The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation | Dufour, Efros and Pérez (2026), ARXIV-2606.20536. Several hundred SiT networks trained on class-conditional ImageNet 256×256, with FID treated as a random variable over a panel of training seeds × sampling seeds. Retraining moves FID 3.2× more than resampling does, the coefficient of variation holds at 1–2% across four model sizes and ten compute budgets, and a lucky training seed reaches the same FID with up to 2× less compute. | Active |
| LIT-503 v2 | First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization | Reddy, Levine and Dragan (2022), ARXIV-2205.12381. Proposes scoring a control interface by the mutual information between the operator’s command and the state transition it induces — no labels, no reward, no knowledge of the task. Validated on 540K examples (Spearman ρ = 0.43, in 4 of 5 domains) and used to learn an interface from scratch in under 30 minutes. The fifth domain correlated negatively, and fixing it needs a horizon hyperparameter whose setting requires the task knowledge the method assumes away. | Active |
| LIT-508 | Opening the Black Box of Deep Neural Networks via Information | Shwartz-Ziv and Tishby (2017), ARXIV-1703.00810 — the claim the information-bottleneck theory of deep learning rests on. Training is said to have two phases, a short fitting phase and a long compression phase in which I(X;T) falls; compression is said to explain generalization; and the compression is attributed to the diffusion-like behaviour of SGD. Measured by binning arctan activations into 30 equal intervals between -1 and 1 over 4096 input patterns. Held as the trunk of a three-way dispute — see LIT-509 and LIT-507. | Active |
| LIT-514 | Rethinking Early Stopping: Refine, Then Calibrate | Berta, Holzmüller, Jordan and Bach (2025), ARXIV-2501.19195 — the record’s first document on probability calibration. A proper loss splits into calibration error and refinement error; the two are not minimized at the same epoch, so stopping on validation loss lands at a compromise optimal for neither. Stop on validation loss after temperature scaling instead, then calibrate. 196 tabular datasets across three model families, plus a vision benchmark of ten runs per dataset. Read as NOTE-259. | Active |
| LIT-515 | Exploiting Block Coordinate Descent for Cost-Effective LLM Model Training | Liu et al. (2025), ARXIV-2506.12037 — full-parameter training one block at a time, with blocks cut on layer boundaries so the optimized kernels still apply. Reaches comparable or better perplexity at 2B on about three times the iterations, using one A800 where full-parameter needs two. Read as NOTE-260. | Active |
| LIT-516 | From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications | Jaiswal et al. (2024), ARXIV-2407.11239 — how nearly low-rank a transformer weight matrix is depends on which matrix and how deep: Query/Key and MLP Gate converge to low rank, MLP Up/Down and Value do not, and middle blocks resist while the first and last few give way. Picking the rank per matrix instead of globally is ~6.4× better in perplexity than uniform reduction at 30% on LLaMA-2 7B and ~47× better at 40% on 13B. Read as NOTE-264. | Active |
| LIT-519 | Complexity-Guided Component-wise Initialization for Language Model Pretraining | Garbers and Oh (2026), ARXIV-2607.09204 — eleven GPT-2-style checkpoints differing in size, language, tokenizer and corpus share a depth profile: effective-rank entropy falls in the last few blocks, most sharply in the residual-writing matrices W_O and W_down, whose Frobenius norms rise almost linearly with depth. Initializing to imitate that profile changes the spectra and does not improve results. Read as NOTE-262. | Active |
| LIT-520 | The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training | Zhang, Zhou, Jia, Chen and Cheng (2026), ARXIV-2605.26489 — the trace-normalized singular spectrum stops moving long before the weight matrices do. Variation peaks near 10⁻² in the first ~1000 steps, while validation loss falls from ~10 to ~4.0, then settles at a 10⁻⁴ floor that coincides with the slow-descent phase. Holds on GPT-2 and LLaMA, across schedules, weight decays, AdamW and Muon. Read as NOTE-261. | Active |
| LIT-521 | Taming Transformer Without Using Learning Rate Warmup | Qi et al. (2025), ARXIV-2505.21910 — attention entropy collapse comes in two modes, and only one crashes the model: sparse and low-rank is fatal, sparse alone is benign. The driver is spectral energy concentration in W_q^T W_k. Suppressing it by bounding the learning rate with Weyl’s inequality trains ViT, Swin and GPT without warmup, matching warmed-up AdamW on all five configurations. Read as NOTE-266. | Active |
| LIT-522 | Pion: A Spectrum-Preserving Optimizer via Orthogonal Equivalence Transformation | Shi et al. (2026), ARXIV-2605.12492 — instead of bounding or renormalizing a weight matrix’s spectrum, never change it: update W by left and right orthogonal transformations so the singular values are invariant and only the singular vectors rotate. On a 60M LLaMA with every normalization layer removed, AdamW and Muon produce NaNs and Pion trains 9.6B tokens. Read as NOTE-267. | Active |
| LIT-523 | Stabilizing Transformer Training by Preventing Attention Entropy Collapse | Zhai et al. (2023), ARXIV-2303.06296 — proves a tight lower bound on attention entropy that falls like Ω(Tσe^{-σ}) in the spectral norm of W_K W_Q^T, then removes the growth by reparameterizing every linear layer as γ·W/σ(W). A ViT trained this way reaches 82.2% on ImageNet without pre-LN, warmup, weight decay or an adaptive optimizer. Read as NOTE-265. | Active |
| LIT-535 | Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression | Fu, Chen, Jia and Sharan (2023), ARXIV-2310.17086, NeurIPS 2024 — the challenge from inside the same setup. In Garg et al.’s exact training regime, successive transformer layers track Iterative Newton linearly (roughly 3 iterations per middle layer) and gradient descent only exponentially; on data with condition number 100 the transformer is unaffected while GD needs 2,000 steps, which twelve layers cannot hold. Theorem 5.1 builds k Newton iterations in k + 8 layers with O(d) width. Read as NOTE-278. | Active |
| LIT-537 | Grokking as the Transition from Lazy to Rich Training Dynamics | Kumar, Bordelon, Gershman and Pehlevan (2023), ARXIV-2310.06110, ICLR 2024 — the counterexample. A two-layer MLP groks on modular arithmetic with no weight decay at all, and its parameter norm rises through the transition, which the weight-norm accounts cannot produce. The proposed replacement: grokking is the network leaving the lazy (linearised) regime and starting to learn features, controlled by an output-scale parameter α and by the alignment between the initial neural tangent kernel and the target. Read as NOTE-284. | Active |
| LIT-538 | Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets | Power, Burda, Edwards, Babuschkin and Misra (2022), ARXIV-2201.02177 — the paper that named grokking. A small transformer on division mod 97 at 50% data reaches near-perfect training accuracy in under 10³ steps and near-perfect validation accuracy around 10⁶, with almost no generalization before 10⁵. The regime dependence is measured here, not later: near 25–30% data on S₅, removing 1% of the training set raises median time to generalization by 40–50%, while time to fit stays at 10³–10⁴ throughout. Weight decay more than halves the data needed. Read as NOTE-281. | Active |
| LIT-539 | Explaining grokking through circuit efficiency | Varma, Shah, Kenton, Kramár and Kumar (2023), ARXIV-2309.02390, DeepMind — the account that earns its place by predicting things nobody had seen. Memorising and generalising circuits both fit the training set; weight decay prefers whichever produces more logit per unit parameter norm; memorisation gets less efficient as the dataset grows and generalisation does not, so there is a critical dataset size D_crit where the preference flips. From that the paper predicts ungrokking and semi-grokking, and then observes both. Read as NOTE-283. | Active |
| LIT-540 | Omnigrok: Grokking Beyond Algorithmic Data | Liu, Michaud and Tegmark (2022), ARXIV-2210.01117, ICLR 2023 — the paper that answers whether grokking is specific to algorithmic data. No: it is induced on MNIST, IMDb and QM9 by shrinking the training set and scaling the initialization, and eliminated on algorithmic data by constraining the weight norm. The proposed mechanism is the “LU” mismatch — reduced training loss is L-shaped in weight norm and test loss U-shaped, so above the critical norm the two disagree. Read as NOTE-282. | Active |
| LIT-541 | Deep Learning is Singular, and That’s Good | Murfet, Wei, Gong, Li, Gell-Redman and Quella (2020), ARXIV-2010.11560 — the invitation, and the trunk under a school the record has been citing without holding. Neural networks are singular statistical models: the set of optimal parameters is a real analytic variety with singularities, so the Laplace approximation and anything that divides by the determinant of the Hessian do not apply. The correct effective parameter count is the real log canonical threshold λ, which counts how many directions change the model, not how sharply the loss rises in them. Read as NOTE-286. | Active |
| LIT-542 | The Local Learning Coefficient: A Singularity-Aware Complexity Measure | Lau, Furman, Wang, Murfet and Wei (2023), ARXIV-2308.12108 — the instrument. It localizes Watanabe’s learning coefficient to a single minimum, defines it as the exponent in V(ε) = c ε^λ (−log ε)^{m−1} + …, and gives an SGLD-based estimator λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] with β* = 1/log n. Validated against known theoretical values on deep linear networks up to 100M parameters. On ResNet18/CIFAR10 it separates training configurations whose training losses have all collapsed to zero. Read as NOTE-285. | Active |
| LIT-544 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei and Murfet (2023), ARXIV-2310.06301 — the paper that separates two things everyone calls a phase transition. In the Toy Model of Superposition at two hidden dimensions, regular k-gons are critical points with derivable local learning coefficients; the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α] then predicts a Bayesian 5 → 6 transition at n_cr ≈ 600, observed at 600 ≤ n ≤ 700. The same critical points govern SGD plateaus — and the 5 → 6 transition is not one of the dynamical ones. Read as NOTE-288. | Active |
| LIT-546 | MatFormer: Nested Transformer for Elastic Inference | Devvrit, Kudugunta, Kusupati et al. (2023), ARXIV-2310.07707. Nest four FFN widths inside one Transformer, so each smaller FFN is the first neurons of the larger. Train one width per step, sampled uniformly, and pick a different width per layer at inference to extract sizes nobody trained. Up to 850M parameters it matches four separately trained sizes at the same total compute. The small submodels beat theirs partly because shared weights see up to 4× the tokens. | Active |
| LIT-548 | Feature Learning in Infinite-Width Neural Networks | Yang and Hu (2020), ARXIV-2011.14522 — Tensor Programs IV, where µP is introduced. Among parametrizations that scale initialization, multipliers and learning rate with width, every stable, nontrivial one either learns features in the wide limit or trains as a kernel, never both. The standard parametrization is stable only at an O(1/width) learning rate, which puts it in the kernel regime. µP is the one that updates every layer maximally, and its maximum stable learning rate does not move with width. | Active |
| LIT-556 | Tensor Programs III: Neural Matrix Laws | Yang (2020), ARXIV-2009.10685. At random initialization and infinite width, a network’s pre-activations become asymptotically free (in the random-matrix sense) of its weights, for any architecture: the Free Independence Principle. This rigorously justifies the Jacobian singular-value calculations behind dynamical isometry, which had assumed it, and gives a second justification of the gradient independence assumption. The general Master Theorem is also what TP-IV uses. | Active |
| LIT-557 | Tensor Programs II: Neural Tangent Kernel for Any Architecture | Yang (2020), ARXIV-2006.14548. The neural tangent kernel of a randomly initialized network of any standard architecture converges to a deterministic limit as width grows, and the paper shows how to compute it. It also gives a checkable condition (Simple GIA Check) under which the common heuristic that backprop’s Wᵀ is independent of forward W is correct, and cases where that heuristic gives wrong answers. | Active |
| LIT-558 | Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes | Yang (2019), ARXIV-1910.12478. At random initialization, any network built from MLPs, RNNs, convolutions, pooling, skip connections, attention, batchnorm or layernorm converges to a Gaussian process as width grows. This is the NNGP correspondence extended from MLPs and CNNs to all standard architectures, proved by expressing the forward pass in the NETSOR language. The first paper of the series and the tutorial for its machinery. A statement about initialization, not training. | Active |
| LIT-562 | Projected GANs Converge Faster | Sauer, Chitta, Müller, Geiger (2021), ARXIV-2111.01007. Feed real and generated images through a frozen ImageNet-pretrained EfficientNet, mix its multi-scale features with fixed random 1×1 and 3×3 convolutions, and put a small discriminator on each scale. FID reaches StyleGAN2’s best on LSUN-Church after 1.1M images instead of 88M, and beats it on 22 datasets. A later analysis (LIT-563) shows part of the gain is in FID’s perceptual null space: at equal FID, humans and a CLIP-feature distance prefer StyleGAN2. | Active |
| LIT-590 | Momentum Contrast for Unsupervised Visual Representation Learning | He et al. (2019), ARXIV-1911.05722. Reframes contrastive learning as dictionary look-up and asks for a dictionary that is both large and consistent: a queue makes it large independently of the batch, and an EMA-updated key encoder makes it consistent. The momentum ablation is the result that propagated — at m=0 it does not train at all. | Active |
| LIT-603 v3 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov, Sutskever, Chen, Corrado and Dean (NIPS 2013), ARXIV-1310.4546. Makes skip-gram fast and better. Negative sampling replaces the softmax with logistic regression against k noise words drawn from the unigram distribution raised to the 3/4 power. Frequent words are subsampled, with discard probability 1 − √(t/f) at t ≈ 10⁻⁵, which runs 2–10× faster and helps rare words. High-PMI bigrams become single tokens. Vector sums compose meanings, for example Russian + river ≈ Volga River. | Active |
| LIT-605 v2 | Sigmoid Loss for Language Image Pre-Training | Zhai et al. (2023), ARXIV-2303.15343. Replace CLIP’s softmax with a pairwise sigmoid, so the loss never needs a global view of the similarity matrix — no all-gathers, only a per-device block in memory, and better results below 16k batch. Then trains at batch size one million and finds the benefit saturated at 32k. | Active |
| LIT-609 v2 | Distributed Representations of Words and Phrases and their Compositionality | Mikolov et al. (2013), ARXIV-1310.4546. Where negative sampling comes from — and it is NCE with the consistency deliberately dropped, kept only because the vectors stay good. Also: the number of negatives that helps falls as the dataset grows, which is the opposite of what the vision contrastive line later assumed. | Superseded — by LIT-603; a duplicate of LIT-603, filed concurrently by |
| LIT-610 v2 | Efficient Estimation of Word Representations in Vector Space | Mikolov et al. (2013), ARXIV-1301.3781. CBOW and Skip-gram: drop the non-linear hidden layer that made neural language models expensive, and word vectors become cheap enough to train on 1.6 billion words in under a day. The linear-analogy finding is here, and so is the test set built to stop it being an anecdote. | Superseded — by LIT-604; a duplicate of LIT-604, filed concurrently by |
| LIT-616 | On Calibration of Modern Neural Networks | Guo et al. (2017), ARXIV-1706.04599. Modern networks are badly calibrated where their 1998-era ancestors were not, and depth, width, BatchNorm and less weight decay each make it worse while improving accuracy. The mechanism is that NLL overfits while 0/1 loss keeps improving. The remedy is one scalar: temperature scaling beats every richer method tried, including the ones that contain it. | Active |
| LIT-617 | Scaling Language Models: Methods, Analysis & Insights from Training Gopher | Rae et al. (2021), ARXIV-2112.11446. Six models on one dataset at one token budget, 44M to 280B, evaluated on 152 tasks — a controlled study of parameter scale. The gains are nonuniform: 51.2% of tasks improve by over 25%, 10.5% not at all, and Gopher is worse than its own smaller models on Abstract Algebra, Temporal Sequences and High School Mathematics. Also the allocation LIT-068 corrects: 280B parameters on 300B tokens is 1.07 tokens per parameter against Chinchilla’s 20. | Active |
| LIT-620 v2 | HunyuanVideo: A Systematic Framework For Large Video Generative Models | Kong et al., Tencent Hunyuan (2024), ARXIV-2412.03603. A 13B open video model: FLUX-style dual-then-single-stream transformer, a multimodal LLM as text encoder, a from-scratch causal 3D VAE. It fits the line’s only explicit video compute-optimal scaling law, on a proxy model that differs from the shipped one in objective, text encoder and conditioning. It reports no controlled ablations. | Active |
| LIT-626 v2 | Movie Gen: A Cast of Media Foundation Models | The Movie Gen team, Meta (2024), ARXIV-2410.13720. A 30B video model built from a Llama 3-style transformer, flow matching and an 8×8×8 temporal autoencoder, with the most controlled ablations in the video line. At 5B, flow matching beats diffusion and video captions beat frame captions. The “LLaMa3 scaling laws” claim rests on image-only runs at four sizes. | Active |
| LIT-628 | Better plain ViT baselines for ImageNet-1k | Beyer, Zhai and Kolesnikov (2022), ARXIV-2205.01580. Four pages that move a plain ViT-S/16 on ImageNet-1k from 66.8% to 80.0% with five changes, none of them novel and none of them a regulariser. The original recipe scores 66.8 / 67.2 / 67.1 at 90 / 150 / 300 epochs — it cannot use more training at all. The belief the abstract names is that ViT needs sophisticated regularisation at this scale; the answer is that standard augmentation is sufficient. | Active |
| LIT-629 v3 | Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion | Huang et al., Adobe Research and UT Austin (2025), ARXIV-2506.08009. Self Forcing trains an autoregressive video model on its own rollouts, conditioning on self-generated context and scoring the whole clip with a distribution-level loss. On Wan2.1-1.3B it beats teacher-forced and diffusion-forced training under every objective tried, and holds quality where they degrade frame by frame. It reaches 17 fps at 0.69s latency on one H100. The comparison against slower models is confounded by a 14B teacher. | Active |
| LIT-634 v2 | Open-Sora 2.0: Training a Commercial-Level Video Generation Model in $200k | The Open-Sora team, HPC-AI Tech (2025), ARXIV-2503.09642. An 11B video model initialized from FLUX, with most of its compute at 256px and a 768px text/image-to-video stage that is short in iterations but takes 37% of the compute. The “$200k” is one final diffusion-model run at an assumed H200 rental rate. It excludes data, captioning, failed runs, and the FLUX and HunyuanVideo-VAE weights it starts from. A 4×32×32 autoencoder is a further efficiency experiment, and the model adapted to it is unconverged and weaker. | Active |
| LIT-640 v3 | Query-Key Normalization for Transformers | Henry, Dachapally, Pawar and Chen (2020), ARXIV-2010.04245. The origin of QK-norm, which thirty documents in this record name and SOTA-192 recommends while sourcing an adopter. ℓ₂-normalize each query and key along the head dimension before the dot product, then scale by a learnable parameter instead of dividing by √d — so the temperature becomes a trained quantity rather than a constant. +0.928 BLEU averaged over five low-resource pairs. | Active |
| LIT-642 | Professor Forcing: A New Algorithm for Training Recurrent Networks | Lamb, Goyal et al., MILA (2016), ARXIV-1610.09038. A discriminator is trained to tell a recurrent network’s hidden-state dynamics under teacher forcing from its dynamics when free-running, and the generator is trained to fool it alongside the usual likelihood loss. Human raters prefer its samples for handwriting generated 20 times past the training length and for raw audio. The likelihood gains are small, the sequential-MNIST claim has no teacher-forcing baseline, and word-level language modeling showed no difference. | Active |
| LIT-646 | Improved Distribution Matching Distillation for Fast Image Synthesis | Yin et al., MIT and Adobe (2024), ARXIV-2405.14867. DMD2 drops DMD’s regression loss. It stabilizes training by updating the fake score five times per generator step, adds a GAN loss on real images, and trains multi-step students on their own simulated inputs. The ImageNet ablation is clean: removing the loss costs 2.62 → 3.48 FID, and the update rule restores 2.61. The student is slightly less diverse than its teacher (LPIPS diversity 0.61 against 0.64). The paper does not attribute this to the reverse KL. | Active |
| LIT-647 | Sequence Level Training with Recurrent Neural Networks | Ranzato, Chopra, Auli and Zaremba, Facebook AI Research (2015), ARXIV-1511.06732. The paper that named exposure bias. MIXER anneals a cross-entropy-trained RNN toward REINFORCE on the test metric over its own samples, and beats cross-entropy by 1 to 3 points on three tasks with greedy decoding. Its baselines that avoid exposure bias without a sequence-level reward do not reliably help, so the gain is not shown to come from fixing exposure bias. | Active |
| LIT-649 | Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks | Bengio, Vinyals, Jaitly and Shazeer, Google (2015), ARXIV-1506.03099. During training, each previous-token input is replaced by the model’s own prediction with a probability that grows on a schedule. On captioning, parsing and frame-level speech recognition with small LSTMs this beats training on ground truth only, and training only on the model’s own output does badly. The gains are single runs with no variance reported. Feeding uniform random tokens recovers part of the captioning gain, and the paper never measures how errors grow with sequence length. | Active |
| LIT-651 | Transformers without Tears: Improving the Normalization of Self-Attention | Nguyen and Salazar (2019), ARXIV-1910.05895. The first systematic evaluation of pre-norm in the base Transformer regime — and it reports a result the record’s universal pre-norm practice does not carry: on high-resource WMT’14 English-German, post-norm wins, 27.58 to 26.83. Also introduces ScaleNorm, the ℓ₂-normalization-with-one-learned-scalar that LIT-640’s QK-norm is an application of. | Active |
| LIT-654 | Muon Outperforms Adam in Tail-End Associative Memory Learning | Wang et al. (2025), ARXIV-2509.26030. Ninety-four documents in this record name Muon and seventeen practices recommend or condition on it; five theories mention it and none says why it works. This one ablates Muon block by block: applying it to the value-output weights and the FFN alone almost recovers the full-Muon trajectory, and applying it to query-key contributes little. The account is that Muon’s update matches the outer-product structure of associative memory, giving isotropic spectra and better learning of rare classes. | Active |
| LIT-656 | Intriguing Properties of Quantization at Scale | Ahmadian et al. (2023), ARXIV-2305.19268. A controlled study from 410M to 52B, same architecture, varying weight decay, dropout, gradient clipping and training dtype: outlier dimensions are not an inherent product of scale but of the optimization conditions during pre-training. Their 52B model gains 0.08% under plain INT8 (77.31 → 77.39 average) where OPT-66B loses about 42%. The record’s largest cluster had the emergence framing in an Active practice’s title. | Active |
| LIT-657 | Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution | Dehghani, Mustafa et al., Google DeepMind (2023), ARXIV-2307.06304. Packs patches from several images into each fixed-length sequence, with per-example attention masks and pooling. A ViT can then take any aspect ratio and a sampled resolution per image. It leads ViT at each of 12 compute-matched budgets, and matches the best ViT with 4x less compute. The paper attributes the gain chiefly to seeing about 5x more images, not to aspect ratio. The one test that isolates aspect ratio is a linear probe on fairness attributes. | Active |
| LIT-663 | DINOv3 | Siméoni et al. (2025), ARXIV-2508.10104. The finding the record needs is not the model: train a self-supervised ViT long enough and its dense features rot while its global metrics keep improving. Segmentation mIoU declines after ~200k iterations and for the 7B falls below its early levels, while ImageNet classification rises monotonically throughout. Gram anchoring — pull the student’s patch Gram matrix towards an early teacher’s — repairs it, and it is a different defect from the register artifacts. | Active |
| LIT-668 v2 | ALBERT: A Lite BERT for Self-supervised Learning of Language Representations | Lan, Chen, Goodman, Gimpel, Sharma and Soricut (2019), ARXIV-1909.11942. The measurement behind cross-layer parameter sharing, which this record started recommending — via the fix THEORY-103 endorses — without holding a single paper on it. The sharp result is an asymmetry: sharing the attention parameters across every layer costs +0.1 average at E = 128, and sharing the feed-forward parameters costs −1.4. Also carries a priority claim the record’s dropout practice should have: the first report that dropout hurts a large transformer. | Active |
| LIT-671 | RoBERTa: A Robustly Optimized BERT Pretraining Approach | Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov (2019), ARXIV-1907.11692. A replication study that finds BERT was significantly undertrained and, more usefully here, that BERT’s own NSP ablation could not have shown what it claimed: it removed the loss while keeping the paired-segment input format. Separate the two and removing NSP matches or slightly improves. The paper is careful about its own margins — dynamic masking is “comparable or slightly better”, and byte-level BPE is adopted despite being slightly worse. | Active |
| LIT-672 | Should You Mask 15% in Masked Language Modeling? | Wettig, Gao, Zhong and Chen (2022), ARXIV-2202.08005. The sweep BERT never ran, and the answer the record has been openly missing: 15% is not universally optimal, and the optimum tracks model size — 40% at 354M, 20% at 124M, 15% at 51M. It also decomposes the masking rate into a corruption rate and a prediction rate whose effects are antagonistic, and uses that to find BERT’s 80-10-10 substitution rule worse than plain [MASK]. | Active |
| LIT-673 | Averaging Weights Leads to Wider Optima and Better Generalization | Izmailov, Podoprikhin, Garipov, Vetrov and Wilson (2018), ARXIV-1803.05407. Average the weights visited along the tail of an SGD trajectory under a cyclical or high constant learning rate. The paper LIT-478 named as the first thing missing from this record’s weight-space ensembling coverage, and the baseline SOTA-288’s promotion condition asks for. From pretrained torchvision models, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at the inference cost of one model. One gotcha, stated in the paper: batch normalization statistics have to be recomputed with a pass over the data. | Active |
| LIT-675 | Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time | Wortsman, Ilharco, Gadre, Roelofs, Gontijo-Lopes, Morcos, Namkoong, Farhadi, Carmon, Kornblith and Schmidt (2022), ARXIV-2203.05482. The dependency SOTA-407’s consensus note named as unheld. A hyperparameter sweep throws away every model but one; average them instead. The greedy recipe — sort by held-out accuracy, add each model only if held-out accuracy improves — is “no worse than the best individual model on the held-out validation set” by construction, at O(1) inference cost against an ensemble’s O(k). +0.7 pp on CLIP and +0.5 pp on ALIGN over the best sweep member. It does not move that practice’s consensus, because it is the same first author. | Active |
| LIT-677 | Diffusion Models With Learned Adaptive Noise | Sahoo, Gokaslan, De Sa and Kuleshov (NeurIPS 2024), ARXIV-2312.13236. MuLAN replaces VDM’s scalar learned schedule with a per-pixel schedule conditioned on a learned auxiliary latent. At equal steps and architecture the bound moves 2.65 → 2.60 bits/dim on CIFAR-10, and it reaches VDM’s 2.65 in 2M steps instead of 10M. Its own ablations confirm VDM’s invariance where VDM claimed it. A scalar schedule conditioned on the input buys nothing, and a multivariate schedule conditioned on time only is “comparable to that of VDM”. The headline 2.55 is a different estimator. | Active |
| LIT-678 v2 | Diffusion Meets Flow Matching: Two Sides of the Same Coin | Gao, Hoogeboom, Heek, De Bortoli, Murphy and Salimans (2024), a blog post at diffusionflow.github.io. With a Gaussian source, flow matching is a diffusion model with the schedule α_t = 1−t, σ_t = t, and the flow-matching Euler sampler is DDIM. What separates the named recipes is three choices: the loss weighting, the network output and the sampling schedule. “Straight paths” holds only for a single-point target, and a variance-preserving schedule can be straighter for wide data. The content is derivation and toy illustration only: no trained models and no benchmarks. | Active |
| LIT-680 | Distilling the Knowledge in a Neural Network | Hinton, Vinyals and Dean (NIPS 2014 Deep Learning Workshop), ARXIV-1503.02531. This is where the word “distillation” in 112 files of this record comes from. It trains a student on the teacher’s softmax at a raised temperature, adds a down-weighted hard-label term, and scales the soft term by T². On speech, a 10-model ensemble distils into one model of the same size as each member, keeping 86% of the frame-accuracy gain and all of the 0.2-point WER gain. Three things usually quoted from it are not in it: the phrase “dark knowledge”, a high temperature on the realistic task (the best was T = 2), and an ablation of T². | Active |
| LIT-681 | Cramming: Training a Language Model on a Single GPU in One Day | Geiping and Goldstein (ICML 2023), ARXIV-2212.14034. A BERT-style MLM trained from scratch for 24 hours on one consumer GPU, which reaches 79.2–80.4 GLUE against BERT-base’s 80.9. The transferable finding is negative: model shape does not beat the scaling law at this budget. Every reshape it tried lands near the same final loss, because a smaller model’s extra throughput cancels its slower learning. So the only architectural lever is to cut step time at constant parameter count. Its data result is the cheapest filter in the record: drop text the tokenizer compresses badly. | Active |
| LIT-682 | ScheduleFree+: Scaling Learning-Rate-Free & Schedule-Free Learning to Large Language Models | Defazio (2026), ARXIV-2605.19095, a single-author tech report from the author of LIT-213. Plain Schedule-Free fails at large batch (“a scaling ‘cliff’ at 2M tokens per batch”). Its small weight decay was “compensating for gradient norm drift, in a somewhat hacky way”. ScheduleFree+ fixes both: it restores inner momentum and adds Polyak step sizes with AdamC-style weight decay. From 120M to 2B it beats WSD at every size, and it beats tuned linear decay except at the largest short runs. The ladder changes six things at once, and the one quantified margin, 31%, is for the 120M model only. | Active |
| LIT-683 | Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach | Geiping et al. (NeurIPS 2025), ARXIV-2502.05171. Huginn has a 2-layer prelude, a 4-layer core iterated r times with the input re-injected every iteration, and a 2-layer coda: 3.5B parameters trained on 800B tokens with r sampled per step (mean 32) and backpropagation through only the last 8 iterations. Accuracy rises with test-time iterations and saturates per task. Recurrence is shown to buy accuracy per parameter, not per FLOP. The only controlled baseline has the same parameters and about 1/16 of the compute. “Equivalent to 50B parameters” is a FLOP count, not a comparison. | Active |
| LIT-684 | Model Merging in Pre-training of Large Language Models | Li et al. (ByteDance Seed, 2025), ARXIV-2505.12082. A uniform average of about ten recent checkpoints on a constant-learning-rate (WSD stable-phase) run beats the latest checkpoint at every size tested, up to 20B/200B MoE and 70B dense. It roughly tracks where an anneal would land. The travelling claim, that constant LR plus merging “can effectively match the performance of an annealed model at any point”, rests on one fork of one 1.3B/13B model. At the end of that fork’s 250B-token decay, the annealed model is about 1.5 MMLU ahead. The baselines in the big tables are unannealed checkpoints. | Active |
| LIT-685 | Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs | Chen, Wang, Xiao, Wang, Chen, Cai, He and Wang (ACL 2025), ARXIV-2507.10613, published at ACL as “Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies”. It claims that gains decelerate faster than a power law when data is “dense” or when tokens per parameter run high, and fits a modified Chinchilla law over more than 400 models from 20M to 7B. None of it should be leaned on. The density metric’s embedding, dimension and cluster count are never stated. The fitted law has no density term, only two logistic multipliers on D/N bounded in [1.5, 2). And the paper’s own Table 4 contradicts its “across all model sizes”. | Active |
| LIT-686 | On the distance between two neural networks and the stability of learning | Bernstein, Vahdat, Yue and Liu (NeurIPS 2020), ARXIV-2002.03432. A deep network’s trust region is not quadratic. The change in its function and Jacobian is bounded by a product of per-layer relative perturbations (“deep relative trust”), and Figure 1 shows loss and gradient breaking down quasi-exponentially in that quantity at depth 16. The optimizer it derives, Fromage, is LARS plus a norm-correcting prefactor. It is the Frobenius antecedent of the modular-norm line. LIT-437, which shares an author, names its conditioning assumption as the flaw and replaces Frobenius with spectral. | Active |
| LIT-688 v2 | Reconciling Kaplan and Chinchilla Scaling Laws | Pearce and Song (TMLR 2024), ARXIV-2406.12907. Kaplan’s N_opt ∝ C^0.73 against Chinchilla’s C^0.50 is mostly a counting artifact. Kaplan counted non-embedding parameters and fit at small scale, where embeddings are much of the model. Simulating the Chinchilla study under those conditions gives 0.74–0.78. The result is a simulation from Chinchilla’s fitted surface, extrapolated four or more orders of magnitude below its range. The experiments are five models of 0.8–4.6M parameters at context length 16. They also find that the learning-rate schedule, which Chinchilla blamed, is not the main cause. | Active |
| LIT-689 | Common Diffusion Noise Schedules and Sample Steps are Flawed | Lin, Liu, Li and Yang (WACV 2024), ARXIV-2305.08891. Common schedules leave signal at the last timestep. Stable Diffusion’s x_T is 0.068·x₀ + 0.998·ε. The model learns to keep the leaked per-channel mean, and pure-noise inference then pins samples to “medium brightness”. Many samplers also never start at t = T. It proposes four fixes: zero terminal SNR, v-prediction, sampling from the last timestep, and a guidance rescale. The only numeric result applies all four at once (COCO FID 22.96 → 21.66). Every single-fix ablation is qualitative, on one to three prompts. It credits the trailing discretization to DPM-Solver, and it never calls the spacing fix “slight”. | Active |
| LIT-690 | Resolving Discrepancies in Compute-Optimal Scaling of Language Models | Porian, Wortsman, Jitsev, Schmidt and Carmon (NeurIPS 2024), ARXIV-2406.19146. It reproduces Kaplan’s allocation exponent (0.835 on RefinedWeb, 0.864 on OpenWebText2) over 900+ runs from 5M to 901M, then removes it in steps. Counting the head’s FLOPs takes 0.129 off, a warmup short enough for small models takes 0.104, and tuning learning rate, batch and β₂ per size, with a constant learning rate, takes 0.105, landing on 0.497. Chinchilla’s proposed cause, matching the cosine decay to each run, moves it 0.031 on a side branch. “careful learning rate decay is not essential for the validity of their scaling law.” | Active |
| LIT-692 | Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation | Kingma and Gao (NeurIPS 2023), ARXIV-2303.00848. Every standard diffusion loss is ½∫ w(λ) E‖ε̂ − ε‖² dλ for an implied weighting w. Given w, the loss is invariant to the schedule’s interior shape, and the schedule is only an importance sampler. If w falls monotonically in log-SNR, the loss is the expected ELBO of Gaussian-noise-augmented data. App. D.3 derives that flow matching’s weighting equals v-MSE’s under a cosine schedule. In its 64×64 ablations the training schedule makes no difference, and the weighting and the sampler each move FID by 0.1–0.3. Monotone weightings tie EDM. They do not beat it. | Active |
| LIT-693 v3 | Classifier-Free Diffusion Guidance | Ho and Salimans (2022), ARXIV-2207.12598. The technique SOTA-202, SOTA-203 and SOTA-410 all turn on, which this record had never held. Drop the conditioning on a fraction of training examples and one network learns both scores; at sampling, extrapolate away from the unconditional one. The trade it buys is severe and measured: on ImageNet 64×64, w from 0.1 to 4.0 moves FID from 1.55 to 26.22 while IS goes 66.11 to 260.2. Any FID or IS comparison across guidance scales is therefore meaningless, which is the missing source for SOTA-307’s instruction to search the scale per cell. | Active |
| LIT-700 | Non-Euclidean Gradient Descent Operates at the Edge of Stability | Islamov, Crawshaw, Cohen and Gower (ICML 2026), ARXIV-2603.05002. Steepest descent under any norm — Spectral GD, ℓ∞-descent, Block CD, and the normalized forms SignGD and normalized Spectral GD — also sharpens to 2/η and trains there, if sharpness is measured in the update’s own norm: max_{‖d‖=1} dᵀ∇²L d. The ordinary top Hessian eigenvalue stays well below 2/η for Spectral GD and ℓ∞-descent, so the Euclidean diagnostic misses the regime entirely. Full-batch, CIFAR-10 and Tiny Shakespeare scale; the non-Euclidean sharpness is a multi-restart Frank-Wolfe estimate, not an exact value. | Active |
| LIT-701 | Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution | Lou, Meng and Ermon (2023), ARXIV-2310.16834. SEDD: learn the ratios p_t(y)/p_t(x) of a discrete diffusion with score entropy, a Bregman loss that keeps them positive and yields an ELBO. The first discrete diffusion model to reach GPT-2-level zero-shot perplexity, and the absorbing (mask) process beats the uniform one on every table. The GPT-2 comparison is against the released model, trained on different data and re-evaluated without a sliding window. | Active |
| LIT-702 | Simple and Effective Masked Diffusion Language Models | Sahoo et al. (2024), ARXIV-2406.07524. MDLM: the masked-diffusion training objective reduces to a weighted average of BERT-style masked-LM losses, a continuous-time ELBO whose value is invariant to the noise schedule. It is the strongest discrete diffusion language model at its scale — beating SEDD at matched tokens — and still trails a retrained autoregressive baseline in-domain (LM1B 27.04 vs 22.32; OWT 23.21 vs 17.54), which the authors state as the limitation. | Active |
| LIT-706 | Scaling Language-Image Pre-training via Masking | Li et al. (2022), ARXIV-2212.00794 — FLIP. Train CLIP with 50–75% of image patches removed and the image encoder run on the rest: the saved compute buys more pairs per hour and a 2–4× larger batch at the same memory, which outweighs the information lost per image. ViT-L/16 on LAION-400M reaches its CLIP reproduction’s accuracy more than 3× faster. The accuracy gain at a fixed schedule is 1–2 points at L and nothing at B/16, and at equal batch size masking 50% buys nothing — the batch is doing the work. | Active |
| LIT-708 | A Controlled Study of Attention-Only Transformers | Ndubuaku et al. (2026), ARXIV-2607.18363. Removing every feed-forward layer from a decoder costs 0.47 nats at matched depth and 0.26 at matched training FLOPs, but only 0.0055 nats (0.27% of loss) at matched parameters once the freed budget goes into attention depth: 20 attention-only layers against 4 standard blocks, 24M parameters, 105B tokens of a synthetic reasoning corpus, with a separate learning-rate sweep for each arm. The remaining gap sits on low-context tokens, where the answer must come from the weights. A pre-registered test on fineweb-edu predicted 0.02–0.05 nats and measured 0.040. Removing QK-norm made the attention-only stack diverge; residual gates and ReZero changed nothing. Everything is at or below 87M parameters. | Active |
| LIT-709 | Primer: Searching for Efficient Transformers for Language Modeling | So et al. (NeurIPS 2021), ARXIV-2109.08668. An evolutionary search over TensorFlow primitives, at a fixed 24-hour training budget, finds two changes that carry most of the gain. One squares the feed-forward ReLU. The other adds a causal width-3 depthwise convolution after each per-head Q, K and V projection (MDHA). The only direct activation comparison has squared ReLU ahead of SwiGLU and ReGLU, in one 110M C4 figure. “4.2×” is against the original ReLU T5 at 537M. Against the SwiGLU + RMSNorm baseline the saving is 2× at equal final quality. On encoder–decoder masked LM, Primer-EZ does no better than that baseline. | Active |
| LIT-710 | Efficient Diffusion Training via Min-SNR Weighting Strategy | Hang et al. (ICCV 2023), ARXIV-2303.09556. Weight each timestep’s x0-space loss by min(SNR(t), γ), γ = 5, which is min(γ/SNR, 1) on an ε loss and min(SNR, γ)/(SNR+1) on a v loss. The headline “3.4× faster” is against an x0-prediction baseline with constant weight, on one ViT-B run. Against the ε-prediction default most people train, the UNet gain is 8.55 → 7.32 FID at 200K iterations and 4.21 → 4.14 at 1M. It speeds early training much more than it moves the end point. The “conflicting gradients between timesteps” explanation rests on one fine-tuning probe and a proxy objective. | Active |
| LIT-714 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras et al. (2023), ARXIV-2312.02696 — EDM2. Two contributions that do not depend on each other. Rebuilding the ADM U-Net so every layer preserves activation, weight and update magnitude takes ImageNet-512 FID from 8.00 to 2.56 at equal compute, the largest single step (6.96 → 3.75) being weight normalization on the learned layers. And post-hoc EMA: store two power-function averages at snapshots and reconstruct any EMA length after training, which shows the optimal length moving with architecture, training time, learning rate, guidance weight and — by a factor of seven under guidance — with which metric you score it. | Active |
| LIT-720 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras, Aittala, Lehtinen, Hellsten, Aila and Laine (2023), ARXIV-2312.02696. Post-hoc EMA: keep two power-function weight averages during training, snapshot them periodically, and afterwards reconstruct the average for any decay length by least squares — reconstruction error falling as O(1/n⁴) in the snapshot count. That makes the averaging length a sweep instead of a pre-training guess, and the consequence is the headline number here: with the length fixed at 13%, varying the learning-rate decay over t_ref ∈ [30k, 160k] moves FID by up to 72%; sweeping the length post hoc brings the whole bracket within 10% of the optimum. Also EDM2’s magnitude-preserving architecture, reaching FID 1.91 on ImageNet-512 without guidance. | Superseded — by LIT-714; A duplicate. 2312.02696 was already held as LIT-714, filed hours earlier in #375 and merged while this note was being written; LIT-714 is the fuller reading — the whole ablation ladder, the metric-disagreement finding, and NOTE-370 with the appendices — and ten documents cite it against this note’s four, all of which were its own. Retired rather than deleted so the mistake stays visible and the formerly: alias keeps any stray citation resolving. Nothing here is lost: everything this note said, LIT-714 or SOTA-428 says already, including the learning-rate bracket. |
| LIT-721 | Guiding a Diffusion Model with a Bad Version of Itself | Karras, Aittala, Kynkäänniemi, Lehtinen, Aila and Laine (2024), ARXIV-2406.02507. CFG’s quality gain is not the class emphasis. It comes from the unconditional reference model being worse — a harder task on a smaller training slice — so the density ratio p₁/p₀ falls off away from the manifold and guidance points inward. Replace the unconditional model with a smaller, less-trained copy of the same conditional model and the quality gain survives without the diversity loss: ImageNet-512 FID 2.56 → 1.34 on EDM2-S, 1.25 on XXL, 1.01 at ImageNet-64, and unconditional EDM2-S 11.67 → 3.86, which CFG cannot do at all. The mechanism is tested with a negative control: degrade the two models differently and guidance stops helping entirely. | Active |
| LIT-738 | Shortformer: Better Language Modeling Using Shorter Inputs | Press, Smith and Lewis (2020), ARXIV-2012.15832. Training a transformer language model on short subsequences first and then at the full length is faster and better than training at the full length throughout: on WikiText-103 (247M parameters, final length 3,072), 128 tokens for the first 50 epochs gives 17.52 dev perplexity against the baseline’s 18.65. The paper credits the two-stage routine to BERT, which used it only for speed. Also introduces position-infused attention. | Active |
| LIT-744 | Text Embeddings by Weakly-Supervised Contrastive Pre-training | Wang et al. (2022), ARXIV-2212.03533 — E5. A BERT-initialised bi-encoder trained with InfoNCE and in-batch negatives (batch 32,768) on ~270M web text pairs, cut from ~1.3B by a consistency filter that keeps a pair only if a model trained on the noisy set ranks its passage in the top 2 of 1M. With no labelled data the base model averages 42.9 nDCG@10 on 15 BEIR datasets against BM25’s 41.7, the first unsupervised model reported above BM25; after fine-tuning on MS-MARCO, NQ and NLI the 330M large model averages 61.4 on 56 MTEB English datasets, above the 4.8B GTR-xxl and Sentence-T5-xxl. | Active |
| LIT-746 | Position: Curvature Matrices Should Be Democratized via Linear Operators | Dangel et al. (2025), ARXIV-2501.19183 — a position paper with a library behind it: expose the Hessian, GGN, Fisher flavours and (E)KFAC as linear operators, so applications ask only for matrix-vector products and the batching, scaling and determinism pitfalls live in one tested place. Its measurements are costs in units of a gradient, on ResNet-50 and nanoGPT: computing KFAC’s factors takes 1.5–2.5 gradients and applying them 0.1–0.2, while a Hessian-vector product takes 4.5–5.5 gradients and 3–4.5× the memory in PyTorch. | Active |
| LIT-749 | bert2BERT: Towards Reusable Pretrained Language Models | Chen et al. (2021), ARXIV-2110.07143 — bert2BERT. Initialize a larger Transformer language model from a smaller pretrained one by Net2Net-style width expansion, then stack layers for depth. Growing a 54M BERT (12 layers, width 512) into BERT-base (width 768) with function-preserving initialization saves 30.4% of pre-training FLOPs at equal GLUE and SQuAD scores. Its recommended variant breaks function preservation by borrowing the next layer’s weights and adds a two-stage schedule, which saves 45.2% on BERT-base and 47% on a GPT-base decoder. | Active |
| LIT-753 | Rigorous dynamical mean field theory for stochastic gradient descent methods | Gerbelot et al. (2022), ARXIV-2210.06591 — a proof that the discrete-time dynamical mean-field equations of statistical physics are exact for a family of first-order methods (multi-pass SGD, heavy-ball and Nesterov momentum, Langevin noise, learning-rate schedules) learning a shallow estimator on Gaussian data, in the limit where samples and dimension grow together and the mini-batch is a fixed fraction of the data. On a teacher-student perceptron the solved equations match simulation at d = 1000 across learning rates and batch fractions. | Active |
| LIT-758 | Net2Net: Accelerating Learning via Knowledge Transfer | Chen, Goodfellow and Shlens (2015), ARXIV-1511.05641 — Net2Net, ICLR 2016. Initialize a larger network so it computes exactly the function of a trained smaller one, then train it normally. Net2WiderNet replicates randomly chosen units and divides their outgoing weights by the replication count, and Net2DeeperNet inserts identity layers. On ImageNet Inception-BN, a widened student reaches the randomly initialized network’s final validation accuracy roughly 2×10⁶ minibatches sooner, with no loss of final accuracy. | Active |
| LIT-761 | Synaptic Field Theory for Neural Networks | Lee, Lee and Yi (2025), ARXIV-2503.08827 — a hep-th proposal with no experiments. Gradient descent with momentum, Ẅ + γẆ + ∂C/∂W = 0, follows from an action weighted by e^{γt}, the volume factor of de Sitter space with Hubble rate H = γ/d. Treat the weight indices as spatial coordinates and take the continuum limit, and training becomes a field theory with the data as external sources. Only toy networks give a local one: periodic, nearest-neighbour wiring with linear or quadratic activations. | Active |
| LIT-764 | A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima | Xie et al. (2020), ARXIV-2002.03495 (ICLR 2021). Near a minimum, SGD’s gradient-noise covariance is approximately the Hessian divided by the batch size; feed that into a Kramers escape-time calculation and the mean time to leave a valley is exponential in B/η times the inverse Hessian eigenvalue along the escape direction. So SGD favours flat minima exponentially, where gradient descent with injected white noise favours them only polynomially. Validated as escape rates on test functions and networks of ten hidden units; generalization is not measured. | Active |
| LIT-765 | Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization | Samragh et al. (2024), ARXIV-2409.12903 — HyperCloning. Initialize a wider language model from a smaller pretrained one of the same depth by tiling each weight matrix into n×n blocks scaled by 1/n, so every hidden vector is n stacked copies of the small model’s and the logits match exactly. Against random initialization with every other setting identical, the cloned models reach the random baseline’s final 10-task accuracy 2.2–4× sooner and finish higher, on OPT, Pythia and OLMo targets of 1.3–2.9B. The small model’s own pretraining is not charged to the comparison. | Active |
| LIT-767 | Understanding Deep Learning via Notions of Rank | Razin (2024), ARXIV-2408.02111 — PhD thesis. Gradient descent’s implicit regularization is better read as rank minimization than norm minimization: on a 2×2 matrix completion problem, a depth L ≥ 2 linear network trained by gradient flow drives every norm and quasi-norm to infinity, with probability at least 0.5 over arbitrarily small initialization, while rank falls to its minimum. The same bias toward low (tensor, hierarchical-tensor) rank is proved for polynomial-activation networks equivalent to tensor factorizations; it predicts convolutional networks’ locality, which a regularizer derived from it partly undoes on long-range tasks. | Active |
| LIT-768 | Kronecker-factored Approximate Curvature (KFAC) From Scratch | Dangel et al. (2025), ARXIV-2507.05127 — a tutorial, math and PyTorch side by side, for the original KFAC of Martens and Grosse: fully-connected layers, no weight sharing. One scaffold covers the GGN, the Monte-Carlo Fisher and the empirical Fisher, and the only thing that changes between them is which vectors are backpropagated. Its practical content is the tests: KFAC is exact on a single data point, and on a deep linear network with square loss for the GGN and MC Fisher but not for the empirical Fisher, which is how to catch the scaling and factor-order bugs it says are easy to ship. | Active |
| LIT-770 v2 | DMAD: Distribution Matching as Adversarial Distillation for Fast Visual Generation | Yu et al., Texas A&M and ByteDance (2026), ARXIV-2610.02188. DMAD replaces DMD’s fake-score critic with a discriminator and trains the student on a linear loss on its logit. At the unrestricted discriminator optimum that gradient equals DMD’s idealized one (App. A, checked); nothing bounds it off the optimum, where training runs with one discriminator step per student step. One-step ImageNet-64 FID is 1.24 against DMD2’s 1.28, at 570K iterations, and 1.04 with an ImageNet-pretrained discriminator scored only by FID. Four-step SDXL reaches 14.47 FID at lower CLIP. Wan2.1 VBench margins are 0.1–0.3 points. No diversity metric and no seeds are reported. | Active |
| LIT-776 | Training Diffusion Models with Reinforcement Learning | Black, Janner, Du, Kostrikov and Levine, UC Berkeley and MIT (2023), ARXIV-2305.13301. Treat each denoising step of a stochastic sampler as an action. Each step is then an isotropic Gaussian with an exact likelihood, so policy gradient (DDPO) can fine-tune a diffusion model on any black-box reward of the final image. On Stable Diffusion v1.4, with JPEG compressibility, LAION aesthetic score and a LLaVA + BERTScore alignment reward, both DDPO estimators beat reward-weighted regression per reward query, and the PPO-clipped one is slightly ahead. There is no KL term. Over-optimization is shown and handled by picking checkpoints by hand. Prompt sets are narrow (45–398 animals), there is no human evaluation, and the run count is not stated. | Active |
| LIT-789 | Multisample Flow Matching: Straightening Flows with Minibatch Couplings | Pooladian et al., Meta AI, NYU and Weizmann (2023), ARXIV-2304.14772. Flow matching with noise and data paired within each minibatch by OT (BatchOT) or a cheaper stable matching keeps the marginals exact. As batch size grows the paths straighten toward the OT map. On ImageNet-64 the FID-20 point drops from 29 Euler steps to 12, and adaptive FID from 13.93 to 12.37, at equal likelihood and 4% more time per iteration. Low-NFE numbers stay far from one-step quality: 38.86 FID at 4 Euler steps on ImageNet-32. One run per row. | Active |
| LIT-791 | Improving the Training of Rectified Flows | Lee, Lin and Fanti, CMU and Microsoft Research (2024), ARXIV-2405.20320. One reflow round is enough, argued informally, and the 2-rectified flow then trains better with a U-shaped timestep distribution and an LPIPS-Huber loss. Started from EDM, with no separate distillation stage, one-step FID is 3.07 on CIFAR-10 and 4.31 on ImageNet-64. Its own table lists DMD at 2.62 on ImageNet-64. There is no distillation-alone arm from the same teacher, and many-step quality is not compared with the pre-reflow model. | Active |
| LIT-795 | Scaling Laws for Reward Model Overoptimization | Gao, Schulman and Hilton, OpenAI (2022), ARXIV-2210.10760. InstructGPT’s 6B reward model stands in for people and labels comparisons for proxy reward models of 3M to 3B parameters. Optimizing a 1.2B policy against a proxy, the gold reward rises and then falls as a function of d = √KL from the initial policy: d(α − βd) for best-of-n and d(α − β log d) for PPO. The coefficients move smoothly with the proxy’s size, so the peak is predictable. Larger proxies and more comparison data push the peak out, and a 6B policy peaks at the same KL as a 1.2B one. A KL penalty in the reward did not improve gold reward at a given KL; its effect was like early stopping, a result the authors flag as possibly hyperparameter-sensitive. RL spends far more KL than best-of-n, so KL does not compare optimization across methods. Synthetic labels, one environment. | Active |
| LIT-798 | Improving and generalizing flow-based generative models with minibatch optimal transport | Tong et al., Mila (2023), ARXIV-2302.00482. Flow matching trains on any coupling of source and target samples. Pairing each minibatch by exact optimal transport (OT-CFM) gives near-OT paths in 2-D (normalized path energy 0.02–0.09 against 0.2–2.7 for the independent coupling, five seeds) at under 1% overhead. On CIFAR-10 its gain is small: adaptive-solver FID 3.58 against 3.66 at 134 against 146 NFE, and the independent coupling is better at 1,000 Euler steps. Its rerun of the path comparison, at a recipe that beats Flow Matching’s own, has the straight path ahead of VP at 100 Euler steps (4.64 against 7.77). | Active |
| LIT-799 | Olmo Hybrid: From Theory to Practice and Back | Merrill et al., Ai2 and Lambda (2026), ARXIV-2604.03444. Olmo 3 7B with its sliding-window layers replaced by Gated DeltaNet (negative eigenvalues), at 3:1 with full attention, trained to 6T tokens. It reaches Olmo 3’s MMLU in 49% fewer tokens and beats it in every domain after mid-training, though data mix, schedule and mid-training batch also changed. Scaling fits from 60M to 1B give the hybrid a lower data coefficient (83.7 against 94.9, non-overlapping CIs); the projected 1.3–1.9× token savings have CIs that include 1 at every size but 3B. A theorem shows hybrids express a task neither parent can. Removing negative eigenvalues barely changes the scaling, which weakens the expressivity explanation. | Active |