Topics — The twenty-eight kinds of claim this anthology files against — the primary axis of both the practice registry and the reading list, and the same twenty-eight for each (ADR-026). The filing rule is the point: take a domain topic when the claim is ABOUT the domain; a claim merely discovered in one still takes its kind. The axis organizes the record, it does not bound it: content the list cannot place is evidence the list is short (ADR-059).
Analysis and evaluation — how to find out whether something worked — what to measure, what a measurement cannot tell you, and which comparisons are unsound; theory, interpretability and debugging belong here too.
238 of 737 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-014 v2 | Visualizing the Loss Landscape of Neural Nets | Li et al. (2017), ARXIV-1712.09913. Visualizes loss landscape geometry. | Active |
| LIT-019 v2 | The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks | Frankle and Carbin (2018), ARXIV-1803.03635. A dense network contains a sparse subnetwork that trains to the same accuracy in the same number of steps — but only from its original initialization, which is the part that makes it a claim about initialization rather than about architecture. | Active |
| LIT-031 v2 | Pruned Neural Networks are Surprisingly Modular | Filan et al. (2020), ARXIV-2003.04881. Analysis of neural network modularity. | Superseded — by LIT-041, ARXIV-2110.08058; by later work on clusterability and mechanistic interpretability |
| LIT-039 v3 | Gradient Flow in Sparse Neural Networks and How Lottery Tickets Win | Evci et al. (2020), ARXIV-2010.03533. Sparse networks trained from scratch underperform because they have poor gradient flow, and a lottery ticket wins by re-learning the pruned solution it came from rather than by having found a better sparse architecture. | Active |
| LIT-041 | Clusterability in Neural Networks | Filan et al. (2021), ARXIV-2103.03386. Larger models develop more modular features. | Rejected — the result does not look interesting enough to carry |
| LIT-052 v2 | Scale Efficiently: Insights from Pre-training and Fine-tuning Transformers | Tay et al. (2021), ARXIV-2109.10686. Model shape, not just model size, determines downstream fine-tuning quality; the DeepNarrow strategy reaches T5-Base quality with 50% fewer parameters and 40% faster training. | Active |
| LIT-057 v2 | What Makes for Good Views for Contrastive Learning? | Tian et al. (2020), ARXIV-2005.10243. Analysis of view generation. | Rejected — theoretical, with no concrete training recommendation |
| LIT-060 v2 | Improving Language Models by Retrieving from Trillions of Tokens | Borgeaud et al. (2021), ARXIV-2112.04426. Retrieval-augmented language models. | Active |
| LIT-072 v2 | Simple Open-Vocabulary Object Detection with Vision Transformers | Minderer et al. (2022), ARXIV-2205.06230. Open vocabulary detection. | Active |
| LIT-077 v2 | Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models | Srivastava et al. (2022), ARXIV-2206.04615. Comprehensive evaluation framework. | Active |
| LIT-085 v3 | Progress measures for grokking via mechanistic interpretability | Nanda et al. (2023), ARXIV-2301.05217. Grokking measurement. | Active — measurement rather than actionable practice |
| LIT-092 v2 | Understanding Contrastive Learning Requires Incorporating Inductive Biases | Saunshi et al. (2022), ARXIV-2202.14037. Role of inductive biases. | Rejected — theoretical analysis without concrete recommendations |
| LIT-095 v3 | The Unreasonable Effectiveness of Deep Features as a Perceptual Metric | Zhang et al. (2018), ARXIV-1801.03924. Deep features from almost any pretrained network, compared layer-wise, match human perceptual judgements far better than L2, SSIM or any hand-designed metric — and the effect is robust to which network and which supervision produced the features. | Active |
| LIT-123 | Wait, Wait, Wait… Why Do Reasoning Models Loop? | Pipis et al. (2025), ARXIV-2512.12895. Reasoning models loop — repeat the same text — at low temperature; larger models loop less and distilled students loop far more than their teachers, which the paper traces to errors in learning and, on a synthetic task, to risk aversion: when the progress-making action is hard to learn and a cyclic one is easy, probability shifts to the cycle. | Active |
| LIT-167 | Understanding R1-Zero-Like Training: A Critical Perspective | Liu et al. (2025), ARXIV-2503.20783. Two corrections to the R1-Zero story: the base model may already carry what looks like emergent reasoning, and GRPO has a length bias that inflates wrong answers. Dr. GRPO removes the bias. | Active |
| LIT-190 | Massive Activations in Large Language Models | Sun et al. (2024), ARXIV-2402.17762. A handful of activations in an LLM run up to 100,000× larger than the rest, at fixed feature dimensions and a few token positions, and they act as fixed biases rather than input-dependent features: setting them to their own mean leaves the model intact, setting them to zero destroys it. They are how the model smuggles an attention bias into a mechanism that has none, and giving attention an explicit learnable bias makes them disappear at no cost in quality. | Active |
| LIT-228 | Emergent Modularity in Pre-trained Transformers | Zhang et al. (2023), ARXIV-2305.18390. Neurons in a pre-trained transformer are functionally specialized, the specialization clusters into mixture-of-experts-shaped groups, and perturbing a functional expert damages the corresponding function. The modular structure stabilizes early in pre-training — earlier than the neurons themselves do. | Active |
| LIT-230 v3 | Understanding Evolution Strategies for LLM Reasoning: Broader Reasoning Coverage than GRPO | Ba et al. (2026), ARXIV-2608.27351. The first study of what ES post-training DOES rather than whether it wins. Three findings: ES raises pass@1 and pass@k together while GRPO raises pass@1 and falls below the base model on pass@16 and pass@32 in 15 of 18 comparisons; ES’s 40x larger parameter drift is functionally sparse, concentrated in LayerNorm and attention projections, and does not produce broad forgetting; and the population size needed for stable ES falls as the model grows. | Active |
| LIT-233 v3 | Neural Thickets: Diverse Task Experts Are Dense Around Pretrained Weights | Gan and Isola (2026), ARXIV-2603.12228. Measures the neighbourhood of pretrained weights and finds it dense with task-improving Gaussian perturbations, with the density rising monotonically with model scale — 0% of perturbations match base accuracy on GSM8K at 0.5B against 64% at 32B — and finds those perturbations to be specialists rather than generalists. The method built on it, RandOpt, post-trains in one parallel step and is competitive with PPO, GRPO and ES at equal training FLOPs, at the cost of K forward passes at inference. | Active |
| LIT-235 v2 | Matching Accuracy, Different Geometry: Evolution Strategies vs GRPO in LLM Post-Training | Hoy et al. (2026), ARXIV-2604.01499. Resolves the drift dispute by decomposing the ES update: an on-manifold component that changes the loss, and an off-manifold component that is loss-invariant and therefore performs a random walk whose squared norm grows as sigma^2 d T / N. In a landscape with many flat directions the second dominates — which is why ES moves two orders of magnitude further than GRPO, why the two update directions are nearly orthogonal, and why the two solutions are nevertheless linearly connected with no loss barrier. | Active |
| LIT-236 v2 | The Blessing of Dimensionality in LLM Fine-tuning: A Variance-Curvature Perspective | Liang et al. (2026), ARXIV-2602.00170. One geometric property accounts for two puzzles at once: fine-tuning landscapes are low-dimensional in curvature, so a handful of stiff directions carry the improvement while a near-zero bulk absorbs variance. That gives rise-then-decay dynamics under fixed hyperparameters — in GRPO as well as ES — and it gives degeneracy, where many ambient perturbations share the same projection onto the stiff subspace, so a population of about 30 keeps finding improvements from 0.5B to 7B with no rightward shift. | Active |
| LIT-248 | Topics in propagation of chaos | Sznitman (1991), DOI:10.1007/BFb0085169. The standard treatment of propagation of chaos: N interacting particles become asymptotically independent as N grows, with the empirical measure converging at rate 1/sqrt(N). | Active |
| LIT-261 | The mean field analysis of the Kuramoto model on graphs II. Asymptotic stability of the incoherent state, center manifold reduction, and bifurcations | Chiba and Medvedev (2017), ARXIV-1709.08305. The companion stability analysis: the incoherent state of the graph Kuramoto model, its centre- manifold reduction, and the bifurcations at the transition. | Active |
| LIT-262 | Consensus Based Sampling | Carrillo et al. (2021), ARXIV-2106.02519. Consensus-based sampling: an interacting-particle method that produces Gaussian posterior approximations without derivatives, tuned by an interpolation parameter between optimization and sampling. | Active |
| LIT-267 | Mean Field Analysis of Neural Networks: A Law of Large Numbers | Sirignano and Spiliopoulos (2018), ARXIV-1805.01053. The law-of-large- numbers half of the mean-field limit for single-hidden-layer networks: the empirical measure of parameters converges to a deterministic limit. | Active |
| LIT-271 | A Mean Field View of the Landscape of Two-Layers Neural Networks | Mei et al. (2018), ARXIV-1804.06561. The distribution of neurons in a wide two-layer network evolves by a Wasserstein gradient flow, so SGD on the finite network is a noisy discretization of a deterministic PDE. | Active |
| LIT-275 | The mean field analysis for the Kuramoto model on graphs I. The mean field equation and transition point formulas | Chiba and Medvedev (2016), ARXIV-1612.06493. The mean-field limit of the Kuramoto model on a graph is an equation in the graphon, and the synchronization threshold is set by the extreme eigenvalues of that operator rather than by average degree. | Active |
| LIT-282 | Convex Analysis of the Mean Field Langevin Dynamics | Nitanda et al. (2022), ARXIV-2201.10469. Mean-field Langevin dynamics read as convex optimization in measure space, with a duality gap that can be monitored and a log-Sobolev inequality giving linear convergence. | Active |
| LIT-285 v2 | Coordination of groups of mobile autonomous agents using nearest neighbor rules | Jadbabaie, Lin and Morse (2003), DOI:10.1109/TAC.2003.812781. Nearest-neighbour heading averaging on undirected graphs converges to a common value for any switching sequence whose graphs are jointly connected across contiguous, bounded time intervals. The proof is via Wolfowitz’s ergodic-product theorem. It is asymptotic only, with no rate, and its priority is contested by Bertsekas and Tsitsiklis against their 1980s agreement results (LIT-325). The system is shown stable although no common quadratic Lyapunov function exists. | Active |
| LIT-286 | The Kuramoto model: A simple paradigm for synchronization phenomena | Acebron et al. (2005), DOI:10.1103/RevModPhys.77.137. The standard review of the Kuramoto model: the synchronization transition, the order parameter, and what noise does to the critical coupling. | Active |
| LIT-293 | Emergent Behavior in Flocks | Cucker and Smale (2007), DOI:10.1109/TAC.2007.895842. Flocking emerges unconditionally when the interaction between agents decays slowly enough with distance, and only conditionally when it decays faster. | Active |
| LIT-296 | The Variational Formulation of the Fokker-Planck Equation | Jordan et al. (1998), DOI:10.1137/S0036141096303359. The Fokker-Planck equation is the gradient flow of free energy in the Wasserstein metric, realized by an implicit variational scheme. | Active |
| LIT-297 | Reconciling modern machine learning practice and the bias-variance trade-off | Belkin et al. (2018), ARXIV-1812.11118. The double-descent curve: test error rises to a peak at the interpolation threshold and then falls again past it, so more parameters past the point of fitting the training set help rather than hurt. | Active |
| LIT-298 | On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport | Chizat and Bach (2018), ARXIV-1805.09545. Gradient descent on the neurons of an overparameterized two-layer network converges to a global optimum of the measure problem, given a separation condition on the initialization. | Active |
| LIT-301 | Products of indecomposable, aperiodic, stochastic matrices | Wolfowitz (1963), DOI:10.1090/S0002-9939-1963-0154756-3. Products of indecomposable aperiodic stochastic matrices converge to a rank-one limit — the theorem every gossip and consensus convergence proof reduces to. | Active |
| LIT-304 | Consensus Problems in Networks of Agents With Switching Topology and Time-Delays | Olfati-Saber and Murray (2004), DOI:10.1109/TAC.2004.834113. Consensus on directed graphs: average consensus iff the graph is balanced, convergence rate the Fiedler value of the mirror graph, switching among balanced strongly connected graphs, and an exact bound on tolerable uniform delay. | Active |
| LIT-310 | Phase transitions, logarithmic Sobolev inequalities, and uniform-in-time propagation of chaos for weakly interacting diffusions | Delgadino et al. (2021), ARXIV-2112.06304. Uniform-in-N log-Sobolev inequalities for weakly interacting diffusions, which is what makes propagation of chaos hold uniformly in time rather than only on finite horizons. | Active |
| LIT-311 | An effective theory of collective deep learning | Arola-Fernández and Lacasa (2023), ARXIV-2310.12802. An ensemble of networks coupled through their weights undergoes a phase transition into a collective-learning regime, in which each learns classes it never saw. | Active |
| LIT-312 | Exact Solution for On-Line Learning in Multilayer Neural Networks | Saad and Solla (1995), DOI:10.1103/PhysRevLett.74.4337. The companion Letter: a closed set of order-parameter equations for online learning in a two-layer network, solved exactly. | Active |
| LIT-324 | Broken Neural Scaling Laws | Caballero et al. (2022), ARXIV-2210.14891. Broken neural scaling laws: a functional form with breaks that fits and extrapolates the scaling behaviours a single power law cannot, including non-monotone and inflected curves. | Active |
| LIT-326 | A Necessary and Sufficient Condition for Consensus Over Random Networks | Tahbaz-Salehi and Jadbabaie (2008), DOI:10.1109/TAC.2008.917743. Consensus over random networks holds if and only if the expected topology has a spanning tree — the condition is on the average graph, not on each round. | Active |
| LIT-328 v2 | Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networks | Bordelon et al. (2020), ARXIV-2002.02561. Learning curves for kernel regression follow a power law whose exponent is set by the decay of the kernel-target spectrum, which connects spectral structure to observed scaling exponents. | Active |
| LIT-340 | Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup | Goldt et al. (2019), ARXIV-1906.08632. Exact ODEs for the order parameters of a two-layer network trained by online SGD in the teacher-student setup, extending the Saad-Solla analysis to the modern setting. | Active |
| LIT-346 | On-line learning in soft committee machines | Saad and Solla (1995), DOI:10.1103/PhysRevE.52.4225. The exact learning dynamics of a soft committee machine under online gradient descent, including the symmetric plateau and the specialization that ends it. | Active |
| LIT-353 | Emergence of stochastic flocking for the discrete Cucker-Smale model with randomly switching topologies | Dong et al. (2019), ARXIV-1912.11949. Cucker-Smale flocking under randomly switching topologies: almost-sure velocity consensus given a union-spanning- tree condition on the switching sequence. | Active |
| LIT-360 v2 | Neural Tangent Kernel: Convergence and Generalization in Neural Networks | Jacot et al. (2018), ARXIV-1806.07572. In the infinite-width limit under standard parametrization, training follows a fixed kernel — the network is linear in its parameters around initialization, and no features are learned. | Active |
| LIT-364 | Classifying high-dimensional Gaussian mixtures: Where kernel methods fail and neural networks succeed | Refinetti et al. (2021), ARXIV-2102.11742. A two-layer network with a few hidden units beats kernel methods on a Gaussian-mixture classification task in the high-dimensional limit, which separates feature learning from lazy training. | Deferred |
| LIT-365 | Trainability and Accuracy of Neural Networks: An Interacting Particle System Approach | Rotskoff and Vanden-Eijnden (2018), ARXIV-1805.00915. The same mean-field limit reached as an interacting particle system, with approximation error scaling as 1/n in width rather than 1/sqrt(n). | Active |
| LIT-379 v2 | The Surprising Effectiveness of Test-Time Training for Few-Shot Learning | Akyürek et al. (2024), ARXIV-2411.07279. Temporarily update the model’s parameters at inference, using a loss built from the test instance’s own in-context examples. On ARC-AGI this is up to 6x a fine-tuned baseline — 53.0% with an 8B model, 61.9% ensembled with program synthesis, which the paper puts at average human performance. Read as evidence about in-context learning’s limits as much as about the technique. | Active |
| LIT-382 | Large Language Models Cannot Self-Correct Reasoning Yet | Huang et al. (2023), ARXIV-2310.01798. The negative result the self-correction literature is answering: asked to revise its own reasoning with no external feedback, a model does not reliably improve, and often gets worse. Names the thing being tested — INTRINSIC self-correction — and that distinction is most of the paper’s lasting value. | Active |
| LIT-388 v2 | Evaluating Large Language Models Trained on Code | Chen et al. (2021), ARXIV-2107.03374. Introduces HumanEval, which measures FUNCTIONAL correctness of generated programs against unit tests rather than textual similarity — and defines pass@k along with the unbiased estimator the record already recommends without citing. The benchmark is the better-known half; the metric is the half this anthology was leaning on. | Active |
| LIT-389 | Training Verifiers to Solve Math Word Problems | Cobbe et al. (2021), ARXIV-2110.14168. Introduces GSM8K — 8.5K grade-school math word problems, chosen because they are conceptually simple and models still failed them, which isolates multi-step reasoning from mathematical difficulty. Also introduces verifier-ranked best-of-n sampling, the ancestor of the reranking the record recommends elsewhere. | Active |
| LIT-390 | Measuring Massive Multitask Language Understanding | Hendrycks et al. (2020), ARXIV-2009.03300. MMLU: 57 subjects from elementary mathematics to law, measuring breadth of acquired knowledge rather than reasoning. Its own abstract names the insensitivity that matters most — models “frequently do not know when they are wrong”, so a high MMLU score is not evidence of calibration. | Active |
| LIT-398 | DataComp-LM: In search of the next generation of training sets for language models | Li et al. (2024), ARXIV-2406.11794. A controlled testbed for data curation — a fixed 240T-token corpus, fixed training recipes at 412M–7B, and 53 evaluations — whose baseline finding is that model-based filtering is what makes a high-quality training set. The position two documents in this record already argue with. | Active |
| LIT-400 | Estimating Training Data Influence by Tracing Gradient Descent | Pruthi et al. (2020), ARXIV-2002.08484. Attributes a prediction to training data by tracing the actual optimization: how much did the loss on the test point move on the steps where this training example was used? Needs only gradients, saved checkpoints and a loss — no Hessian, no convexity. | Active |
| LIT-401 | Studying Large Language Model Generalization with Influence Functions | Grosse et al. (2023), ARXIV-2308.03296. Scales influence functions to language models up to 52B parameters with an EK-FAC approximation to the inverse-Hessian-vector product, then uses them to ask what training data a behaviour came from. Generalization gets more abstract with scale — and stays startlingly sensitive to word order. | Active |
| LIT-402 | If Influence Functions are the Answer, Then What is the Question? | Bae et al. (2022), ARXIV-2209.05364. Decomposes the gap between influence estimates and leave-one-out retraining into five terms, finds that three of them are not errors at all, and identifies what influence functions on neural networks do estimate: the proximal Bregman response function. | Active |
| LIT-403 | Understanding Black-box Predictions via Influence Functions | Koh and Liang (2017), ARXIV-1703.04730. Traces a prediction back through the learning algorithm to the training points responsible for it, using a classical technique from robust statistics made tractable by needing only gradients and Hessian-vector products. The origin of training-data attribution in deep learning. | Active |
| LIT-404 | HellaSwag: Can a Machine Really Finish Your Sentence? | Zellers et al. (2019), ARXIV-1905.07830. Commonsense sentence completion whose wrong answers are generated and then filtered against the models of the day until those models fail — so the benchmark’s difficulty is defined relative to a specific generation of systems, which is the caveat to carry. | Active |
| LIT-405 | On the Measure of Intelligence | Chollet (2019), ARXIV-1911.01547. Argues that measuring skill measures the wrong thing, and introduces ARC — a puzzle benchmark built so that every task is structurally novel, which is what makes it resistant to the memorization that saturates ordinary benchmarks. | Active |
| LIT-407 | Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge | Clark et al. (2018), ARXIV-1803.05457. Grade-school science questions split into an Easy set and a Challenge set, the latter selected because retrieval and word-co-occurrence baselines both fail on it. The “ARC” that appears in this record’s evaluation suites — and not the one SOTA-235 reports on. | Active |
| LIT-408 | Program Synthesis with Large Language Models | Austin et al. (2021), ARXIV-2108.07732. Introduces MBPP — Mostly Basic Python Problems — short crowd-sourced tasks each with three test cases, the entry-level counterpart to HumanEval and the other half of the pair this record’s code-generation results are reported on. | Active |
| LIT-411 | Measuring Mathematical Problem Solving With the MATH Dataset | Hendrycks et al. (2021), ARXIV-2103.03874. 12,500 competition mathematics problems, each with a full step-by-step solution — which is why the dataset became a training signal as well as a benchmark, and why a result on it needs to say which it was used as. | Active |
| LIT-415 | Stream of Search (SoS): Learning to Search in Language | Gandhi et al. (2024), ARXIV-2404.03683. Filed for its task, not its method. Countdown — combine input numbers with arithmetic operations to reach a target — is the verifiable puzzle twenty-three documents here report results on, and this is the paper that put it in front of this literature. | Active |
| LIT-416 | GPQA: A Graduate-Level Google-Proof Q&A Benchmark | Rein et al. (2023), ARXIV-2311.12022. 448 multiple-choice questions written by domain PhDs, validated so that skilled non-experts with unrestricted web access reach 34% against experts’ 65%. The gap is the instrument — and it is small enough that the benchmark’s own noise floor is worth knowing. | Active |
| LIT-419 | OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems | He et al. (2024), ARXIV-2402.14008. 8,476 olympiad-level maths and physics problems, bilingual and partly multimodal, on which the best model at publication scored 17.97% — a floor-effect instrument that the record uses to separate sub-8B models, which is the caveat to carry. | Active |
| LIT-420 | Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them | Suzgun et al. (2022), ARXIV-2210.09261. The 23 BIG-bench tasks where models had not yet beaten the average human rater, split out as BBH — a suite selected by a model generation’s shortfall, and one whose headline finding is that the shortfall was partly a prompting artifact. | Active |
| LIT-421 | Gradient-based learning applied to document recognition | LeCun et al. (1998), Proceedings of the IEEE. The paper MNIST comes from. In this record it is cited for the dataset alone, and almost always in one role: the small convex-ish problem where a claim about influence, attribution or retraining can be checked against ground truth. | Active |
| LIT-422 | ImageNet: A large-scale hierarchical image database | Deng et al. (2009), CVPR. The database itself — millions of images organised on the WordNet noun hierarchy. Almost nothing in this record is measured on it: an “ImageNet” number here is nearly always ILSVRC (LIT-423), the thousand-class challenge built from a slice of it. | Active |
| LIT-423 | ImageNet Large Scale Visual Recognition Challenge | Russakovsky et al. (2014), ARXIV-1409.0575. The thousand-class challenge built on a slice of LIT-422’s database, and the instrument almost every “ImageNet top-1” number in this record is actually measured on — including the human-error figure those numbers are read against. | Active |
| LIT-424 | Learning Multiple Layers of Features from Tiny Images | Krizhevsky (2009), University of Toronto technical report. The source of CIFAR-10 and CIFAR-100 — 32×32 labelled images drawn from the Tiny Images collection. Sixty-nine documents here measure something on it, which makes it the record’s most-used instrument after ILSVRC. | Active |
| LIT-426 v2 | The Pile: An 800GB Dataset of Diverse Text for Language Modeling | Gao et al. (2020), ARXIV-2101.00027. 800GB assembled from 22 named sub-corpora with declared weights. The named domains and the default mixture are why the data-mixing line in this record is measured on it — and every “over The Pile’s default weights” number is relative to a choice its authors made, not to an optimum. | Active |
| LIT-427 | SmolLM2: When Smol Goes Big — Data-Centric Training of a Small Language Model | Ben Allal et al. (2025), ARXIV-2502.02737. The model report behind SmolLM2-135M-Instruct, which is the base LIT-129’s whole experiment runs on — and the source of the overtraining ratio that makes a 135M model worth fine-tuning at all. | Active |
| LIT-433 | Learning to summarize from human feedback | Stiennon et al. (2020), ARXIV-2009.01325. The step where the preference loop crosses into language: human comparisons between summaries, a reward model fit to them, PPO against that reward. The summaries beat the human reference summaries, and beat much larger models fine-tuned supervised — which is the result that made the loop look like a general answer rather than a control technique. | Active |
| LIT-440 | How much do language models memorize? | Morris et al. (2025), ARXIV-2505.24832. Separates what a model stores about a specific dataset from what it stores about the data-generating process, and measures the first by training on uniform random bitstrings where the second is impossible. GPT-style transformers hold about 3.6 bits per parameter, near-independent of precision, and the capacity is linear in parameter count across hundreds of models from 500K to 1.5B. | Active |
| LIT-452 | Scaling Laws for Fact Memorization of Large Language Models | Lu et al. (2024), ARXIV-2406.15720. Fact capacity scales linearly with model size and saturates in training epochs along a negative exponential. Extrapolated, memorising all 15B Wikidata triples wants about 1000B non-embedding parameters trained for 100 epochs. Redundant facts cost full price unless they share direction and structure, so parameters are an expensive place to keep a knowledge base. | Active |
| LIT-455 | Hidden Breakthroughs in Language Model Training | Kangaslahti et al. (2025), ARXIV-2506.15872. A smooth loss curve is not evidence of smooth training. POLCA decomposes the change in loss along a low-rank basis of the training subspace and per example, and the breakthroughs the aggregate hides come back — on synthetic arithmetic it recovers the skill of carrying a digit, which clustering the exact loss curves does not. | Active |
| LIT-458 v2 | The Platonic Representation Hypothesis | Huh et al. (2024), ARXIV-2405.07987. Representations are converging: across architectures, objectives and modalities, and more so as models get larger. The conjecture is that they are converging on a statistical model of whatever generated the data. The mathematical argument holds only for bijective observations, which the authors say plainly and test at the boundary. | Active |
| LIT-463 v2 | Comment on The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity | Lawsen (2025), ARXIV-2506.09250. Argues the reported accuracy collapse is the evaluation: output-token limits, a scorer that cannot tell refusal from failure, and River Crossing instances that have no solution yet count as failures. One claim is a fact; the rest is argued. | Active |
| LIT-464 | Theoretical limitations of multi-layer Transformer | Chen et al. (2024), ARXIV-2412.02975. The first unconditional lower bound for a decoder-only transformer with more than one layer: any constant-depth decoder needs polynomially many parameters to compose functions sequentially. Depth, an encoder, or chain of thought each buy what width cannot. | Active |
| LIT-465 | Auxiliary task demands mask the capabilities of smaller language models | Hu and Frank (2024), ARXIV-2404.02418. Across four capacities, the higher-demand way of asking scores lower than the lower-demand way of asking the same question — and the gap is widest for the smallest and least-trained models, so demands are mistaken for missing capability. COLM 2024. | Active |
| LIT-466 | The Illusion of Thinking: Understanding the Strengths and Limitations of Reasoning Models via the Lens of Problem Complexity | Shojaee et al. (2025), ARXIV-2506.06941. Controllable puzzles instead of contaminated benchmarks, and reasoning models collapse to zero accuracy past a complexity threshold while reducing their thinking tokens. NeurIPS 2025; the collapse reading is publicly contested and the three-regime finding is not. | Active |
| LIT-470 | Emergent Abilities of Large Language Models | Wei et al. (2022), ARXIV-2206.07682. Names and defines emergent abilities — present in larger models, absent in smaller ones, and therefore not predictable by extrapolating a scaling law. A survey of existing curves rather than new experiments: eight few-shot examples across five model families, plus prompting strategies that only help past a threshold. Section 5.1 raises the metric explanation itself and argues against it on two specific grounds. | Active |
| LIT-471 | Are Emergent Abilities of Large Language Models a Mirage? | Schaeffer et al. (2023), ARXIV-2304.15004. A sharp capability curve can be produced from a smooth one by the metric alone. Rescoring fixed GPT-3 outputs under Token Edit Distance removes the emergence; >92% of hand-annotated BIG-Bench emergent abilities sit under two metrics, one nonlinear and one discontinuous; and emergence is induced on demand in vision models that had never shown it. | Active |
| LIT-474 | The Coverage Principle: How Pre-Training Enables Post-Training | Chen, Huang, Golowich, Malladi, Block, Ash, Krishnamurthy and Foster (2025), ARXIV-2510.15020. Names the quantity cross-entropy is a bad proxy for: the coverage profile, the probability mass a model puts on rare high-quality responses. Proves it is necessary and sufficient for Best-of-N to succeed, that next-token prediction optimizes it faster than it optimizes cross-entropy, and that sequence-level KL carries a linear dependence on sequence length that coverage does not. | Active |
| LIT-476 | Estimating the Probability of Sampling a Trained Neural Network at Random | Scherlis and Belrose (2025), ARXIV-2501.18812. An estimator for the measure, under the initialization distribution, of the region around a trained network whose behaviour matches it. Negative log of that measure is a description length. A ConvNeXt deliberately trained to generalize badly has a smaller such region — detectable on clean held-out data where its behaviour is indistinguishable. | Active |
| LIT-480 | Auditing Political Exposure Bias: Algorithmic Amplification on Twitter/X During the 2024 U.S. Presidential Election | Ye, Luceri and Ferrara (2024), ARXIV-2411.01852. 120 sock-puppet accounts, six weeks, 9.79M tweets from Twitter/X’s “For You” timeline. The method is the transferable part: puppets that never interact, follow sets controlled to the subject, and exposure weighted by rank rather than counted. Accounts following nobody get the most diverse recommendations, so personalization concentrates rather than broadens. | Active |
| LIT-481 | Industry Influence in High-Profile Social Media Research | Bak-Coleman, West, O’Connor and Bergstrom (2026), ARXIV-2601.11507. 49% of social media papers in Science, Nature and PNAS have a disclosable industry tie and most go undisclosed — from 21% of authors, so a concentrated group rather than broad engagement. Ties are over-represented in research on what users share and sparse in research on platform dynamics. | Active |
| LIT-482 | Moving towards informative and actionable social media research | Bak-Coleman, Lewandowsky, Lorenz-Spreen, Narayanan, Orben and Oswald (2025), ARXIV-2505.09254. Individual-level randomized trials do not estimate collective effects, for four named reasons — non-linearity across scale, hysteresis, feedback in time, and SUTVA violation through the network. A null from such a trial is not evidence of no effect, and the positive recommendation is to evaluate specific interventions instead of net effects. | Active |
| LIT-484 v2 | TinyStories: How Small Can Language Models Be and Still Speak Coherent English? | Eldan and Li (2023), ARXIV-2305.07759. Restrict the training corpus to what a three-year-old’s vocabulary can say and a model below 10M parameters — or one with a single transformer block — writes fluent, grammatical, multi-paragraph stories. The finding is about the corpus, not the architecture: what stops small models speaking is the breadth of what they are asked to learn, not the complexity of language. | Active |
| LIT-485 | Parameterized Synthetic Text Generation with SimpleStories | Finke et al. (2025), ARXIV-2504.09184. Sampling three words from a child’s vocabulary was not enough: 59.38% of TinyStories begins “once upon a time”. Parameterize the generating prompt by theme, topic, style, narrative feature, grammar feature and author persona, and constrain the opening part of speech and letter, and every diversity metric moves. The one clean ablation in the paper is about something else — a 4,096-token domain tokenizer beats GPT-2’s 50,257 by +26.7 coherence and +27.5 quality at fixed data and architecture. | Active |
| LIT-486 | Parallel Scaling Law for Language Models | Chen et al. (2025), ARXIV-2505.10475. Run the same weights over P learnably-prefixed copies of the input and aggregate the outputs: the loss falls as if the parameter count had risen by O(log P). Fitted across 0.5B–4.4B and P = 1…8 on two corpora. At batch size 1, matching a parameter-scaled model costs 22× less added memory and 6× less added latency — and P× the training FLOPs, which is the real price. | Active |
| LIT-487 | Introducing AI to an Online Petition Platform Changed Outputs but not Outcomes | Corpus et al. (2025), ARXIV-2511.13949. Change.org’s “write with AI” tool reached the US, GB and Canada eleven weeks before Australia. Across 1.5M petitions, access made petitions 49% longer, more lexically varied, 1.5 grades harder to read, and rated higher on quality and persuasiveness — moving them toward the profile of successful pre-AI petitions — while raising inter-petition similarity 23% and leaving outcomes flat or worse. The features that used to predict success stopped predicting it. | Active |
| LIT-489 | Measuring In-Context Computation Complexity via Hidden State Prediction | Herrmann, Csordás and Schmidhuber (2025), ARXIV-2503.13431. Next-token loss cannot tell hard from interesting: random noise scores high and requires no computation, memorized recitation scores low. Insert a variational bottleneck with a learned autoregressive prior mid-network and measure the KL between posterior and prior — the nats of hidden-state information the past did not predict. That quantity separates in-context learning from memorization and noise, and tracks formal-language description length and maths difficulty after controlling for next-token loss. | Active |
| LIT-490 | Denoising as Projection: Constrained Optimization with Gradient-Guided Diffusion | Zhang et al. (2026), ARXIV-2608.29507. Standard gradient guidance adds the objective gradient after the denoising step, which can push the sample off the learned data geometry. Swap the order — gradient first, then one pretrained-denoiser call — and the denoiser acts as an approximate projection, making the sampler an inexact projected-gradient method with convergence guarantees on linear subspaces, compact convex sets and compact Riemannian submanifolds. One gradient and one denoiser call per step; no score Jacobians, no fine-tuning, no extra samples. | Active |
| LIT-492 | The unintended consequences of large language models as a labor-augmenting technology in science | Duede, Gross, Crockett and Bergstrom (2026), ARXIV-2607.17397. A marginal-value-theorem model of how a time-saving tool changes what its users produce, assuming the tool is perfect — faster with no added error and no cost. Removing time raises the long-run rate of return, which is the opportunity cost of time, so developed work is developed less thoroughly. Which work gets developed at all moves in opposite directions depending on the phase accelerated, and thoroughness reverses sign in the third phase — a split the paper’s own closing sentence does not carry. | Active |
| LIT-493 | Language Modeling by Language Models | Cheng, Clark and Richardson (2025), ARXIV-2506.20249. An LLM agent system that proposed, implemented and pretrained 1,062 verified language-model block designs at 14M–350M parameters. The result that survives its own controls is the code generation: decomposing a long program into units and freezing each one that passes an execution-based checker takes valid-design rate from 6% to 92%. The architecture-discovery headline does not survive as well — the five best designs average 0.6 points below the five human seeds they were bred from. | Active |
| LIT-495 | On the Statistical Query Complexity of Learning Semiautomata: a Random Walk Approach | Giapitzakis, Fountoulakis, Nichani and Lee (2025), ARXIV-2510.04115. The first Statistical Query hardness result for semiautomata, and the first where the hardness comes from the transition structure alone rather than from a hard language embedded in the machine. Two distinct machines in the constructed family agree at chance after O(N² ln N) random symbols, so the signal separating them vanishes exponentially in sequence length. The regime it needs is unusual and belongs beside the result: an alphabet of size Ω(N³ ln N). | Active |
| LIT-501 | The FID Lottery: Quantifying Hidden Randomness in Generative-Model Evaluation | Dufour, Efros and Pérez (2026), ARXIV-2606.20536. Several hundred SiT networks trained on class-conditional ImageNet 256×256, with FID treated as a random variable over a panel of training seeds × sampling seeds. Retraining moves FID 3.2× more than resampling does, the coefficient of variation holds at 1–2% across four model sizes and ten compute budgets, and a lucky training seed reaches the same FID with up to 2× less compute. | Active |
| LIT-502 | MemGraphRAG: Memory-based Multi-Agent System for Graph Retrieval-Augmented Generation | Wu, Xiang, Tang, Chen, Zhang and Su (2026), ARXIV-2606.00610. A multi-agent GraphRAG system, filed for its pilot study rather than its system: across three published GraphRAG pipelines, graph expansion raises retrieval recall (GFM-RAG 84.3% against vanilla RAG’s 71.8%) and drops relevance (38.5% against 62.9%), and end-task accuracy falls with it. Their own filtering probe finds 40% of extracted triples can be deleted without cost. | Active |
| LIT-503 v2 | First Contact: Unsupervised Human-Machine Co-Adaptation via Mutual Information Maximization | Reddy, Levine and Dragan (2022), ARXIV-2205.12381. Proposes scoring a control interface by the mutual information between the operator’s command and the state transition it induces — no labels, no reward, no knowledge of the task. Validated on 540K examples (Spearman ρ = 0.43, in 4 of 5 domains) and used to learn an interface from scratch in under 30 minutes. The fifth domain correlated negatively, and fixing it needs a horizon hyperparameter whose setting requires the task knowledge the method assumes away. | Active |
| LIT-504 | Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable? | Méloux, Maniu, Portet and Peyrard (2025), ARXIV-2502.20914. Asks whether mechanistic interpretability’s own criteria pick out a unique explanation, and answers no at every stage by exhaustive enumeration on small MLPs. No network had exactly one circuit interpretation, under 2% had exactly one valid minimal mapping, and the median count rises from 38 to 910,000 as width goes from 2 to 5 — with both figures explicitly lower bounds. | Active |
| LIT-506 v2 | Interpretable RNA Foundation Model from Unannotated Data for Highly Accurate RNA Structure and Function Predictions | Chen et al. (2022), ARXIV-2204.00300 — a 12-layer BERT encoder trained by masked-token prediction on 23.7 million unannotated ncRNA sequences from RNAcentral, then used as a frozen feature extractor for secondary structure, 3D contact, protein-RNA binding and UTR regulation. Beats twelve secondary-structure methods (F1 0.941 on ArchiveII600, 0.704 on bpRNA TS0). Read as NOTE-252, which records that the paper’s one clean input-swap control moves R² by 0.002 on the generalization set. | Active |
| LIT-507 | Adaptive Estimators Show Information Compression in Deep Neural Networks | Chelombiev, Houghton and O’Donnell (2019), ARXIV-1902.09037 — the third voice, and the one whose body is weaker than its abstract. Argues that LIT-509’s ReLU result is an artifact of non-adaptive binning and that with adaptive estimators saturation is not required for compression. Its own Figure 6 shows that averaged over 50 initializations a ReLU network has no distinct phase at all, and it agrees with the rebuttal that hidden-layer compression does not correlate with generalization. Read as NOTE-254. | Active |
| LIT-508 | Opening the Black Box of Deep Neural Networks via Information | Shwartz-Ziv and Tishby (2017), ARXIV-1703.00810 — the claim the information-bottleneck theory of deep learning rests on. Training is said to have two phases, a short fitting phase and a long compression phase in which I(X;T) falls; compression is said to explain generalization; and the compression is attributed to the diffusion-like behaviour of SGD. Measured by binning arctan activations into 30 equal intervals between -1 and 1 over 4096 input patterns. Held as the trunk of a three-way dispute — see LIT-509 and LIT-507. | Active |
| LIT-509 v2 | On the Information Bottleneck Theory of Deep Learning | Saxe et al. (2018), ICLR — the rebuttal. Takes the three claims of LIT-508 and shows none holds in general. The compression phase tracks the nonlinearity: tanh compresses, ReLU does not, replicated with the original authors’ own code, and consistent across binning, a kernel density estimator and — for layer entropy, on the original task only — the Kraskov estimator. Compression and generalization dissociate in all four combinations. Full-batch gradient descent compresses as much as SGD. Read as NOTE-253. | Active |
| LIT-510 | Gemini: A Family of Highly Capable Multimodal Models | Gemini Team, Google (2023), ARXIV-2312.11805 — held for its evaluation methodology rather than its models. Its appendices publish three things the record wanted: an MMLU sweep in which the ordering against GPT-4 flips depending on the inference procedure; a measurement of how cheaply a benchmark can be inflated (100 fine-tuning steps on HellaSwag-adjacent web extracts take Ultra to 96.0% at 1-shot); and a FLEURS ablation retraining without the benchmark’s own training set. Read as NOTE-255. | Active |
| LIT-514 | Rethinking Early Stopping: Refine, Then Calibrate | Berta, Holzmüller, Jordan and Bach (2025), ARXIV-2501.19195 — the record’s first document on probability calibration. A proper loss splits into calibration error and refinement error; the two are not minimized at the same epoch, so stopping on validation loss lands at a compromise optimal for neither. Stop on validation loss after temperature scaling instead, then calibrate. 196 tabular datasets across three model families, plus a vision benchmark of ten runs per dataset. Read as NOTE-259. | Active |
| LIT-517 | Small Singular Values Matter: A Random Matrix Analysis of Transformer Models | Staats, Thamm and Rosenow (2024), ARXIV-2410.17770 — the singular spectrum of a trained transformer weight matrix is a Marchenko-Pastur bulk plus outliers, and in non-square matrices the outliers appear at both ends. Removing the smallest decile of the Down-Projection drops Llama-3 8B on GSM8K from 43.2% to 2.0%, while the smallest decile of the square Attention-Output matrix costs almost nothing. Magnitude order is not importance order. Read as NOTE-263. | Active |
| LIT-518 | The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reduction | Sharma, Ash and Misra (2023), ARXIV-2312.13558 — replacing one weight matrix by a low-rank approximation, especially an MLP matrix in a later layer, can improve accuracy: GPT-J on CounterFact goes from 13.3% to 24.1% off a single layer, with no retraining. The gains land on facts rare in the training corpus. Filed for the debate it anchors rather than read in full. | Active |
| LIT-519 | Complexity-Guided Component-wise Initialization for Language Model Pretraining | Garbers and Oh (2026), ARXIV-2607.09204 — eleven GPT-2-style checkpoints differing in size, language, tokenizer and corpus share a depth profile: effective-rank entropy falls in the last few blocks, most sharply in the residual-writing matrices W_O and W_down, whose Frobenius norms rise almost linearly with depth. Initializing to imitate that profile changes the spectra and does not improve results. Read as NOTE-262. | Active |
| LIT-520 | The Stability of Singular Distribution: A Spectral Perspective on the Two-Phase Dynamics of Language Model Pre-training | Zhang, Zhou, Jia, Chen and Cheng (2026), ARXIV-2605.26489 — the trace-normalized singular spectrum stops moving long before the weight matrices do. Variation peaks near 10⁻² in the first ~1000 steps, while validation loss falls from ~10 to ~4.0, then settles at a 10⁻⁴ floor that coincides with the slow-descent phase. Holds on GPT-2 and LLaMA, across schedules, weight decays, AdamW and Muon. Read as NOTE-261. | Active |
| LIT-524 | Ghost in the Transformer: Detecting Model Reuse with Invariant Spectral Signatures | Wang, Ma, Xinyi and Li (2025), ARXIV-2511.06390 — take the singular spectra of W_q W_k^T and W_v W_o per layer. Those products are invariant to the permutation and scaling transforms that hide derivation while preserving function, so the spectra fingerprint the model. F1 0.9867 over 55 model pairs, data-free, against 0.9610 for the best prior method. Read as NOTE-268. | Active |
| LIT-525 | Transformers Learn Low Sensitivity Functions: Investigations and Implications | Vasudeva et al. (2024), ARXIV-2403.06925 — sensitivity to token-wise random input perturbations is one number that separates transformers from MLPs, CNNs, ConvMixers and LSTMs across vision and language. It also predicts robustness, tracks flatness, can be regularized for, and keeps falling through a grokking plateau while the loss does not move. Read as NOTE-271. | Active |
| LIT-526 | Concepts Whisper: Spectral Anti-Concentration and the Dual Geometry of Transformer Representations | Acharya, Rimal and Dhakal (2026), ARXIV-2605.01609 — contextual concept directions sit in the low-variance tail of the unembedding covariance in 17 of 17 models, while static vocabulary contrasts sit at the top. The paper also reports a robust null against the hypothesis it set out to test, and labels each of its own claims robust, partial or suggestive. Read as NOTE-270. | Active |
| LIT-528 | The emergence of clusters in self-attention dynamics | Geshkovski, Letrouit, Polyanskiy and Rigollet (2023), ARXIV-2305.05465 — treat tokens as interacting particles and self-attention as their dynamics. Tokens cluster, and the spectrum of the value matrix decides the limiting geometry: V = I gives the vertices of a convex polytope, a simple positive leading eigenvalue gives at most three parallel hyperplanes, V = −I gives one cluster at the origin. In one dimension the attention matrix provably converges to a low-rank Boolean matrix. Read as NOTE-272. | Active |
| LIT-531 | Deep Learning and the Information Bottleneck Principle | Tishby and Zaslavsky (2015), ARXIV-1503.02406 — the five-page position paper the whole information-bottleneck line descends from. It proposes quantifying every layer by I(X;h) and I(Y;h), draws the information plane as an explicitly qualitative figure with a hypothesized layer path, and states that approaching the IB limit requires stochastic mapping between the layers — the condition its successors were later refuted for dropping. Read as NOTE-275. | Active |
| LIT-532 | What learning algorithm is in-context learning? Investigations with linear models | Akyürek, Schuurmans, Andreas, Ma and Zhou (2022), ARXIV-2211.15661, ICLR 2023 — the independent construction, and the finding the title undersells. A transformer needs constant depth and O(d) hidden size for a gradient step and O(d²) for a Sherman–Morrison ridge update; but the trained model’s behaviour moves through three algorithmic phases as depth grows — gradient descent, then ridge, then ordinary least squares — and at every noise level the ridge parameter that best fits it is the one that minimizes Bayes risk. Read as NOTE-277. | Active |
| LIT-533 | Transformers learn in-context by gradient descent | von Oswald, Niklasson, Randazzo, Sacramento, Mordvintsev, Zhmoginov and Vladymyrov (2022), ARXIV-2212.07677, ICML 2023 — the paper that named the mesa-optimizer. Proposition 1 constructs key, query, value and projection matrices for which one linear self-attention layer is exactly one gradient-descent step on an implicit linear model, and a single trained layer is shown to find approximately those weights. Deeper trained models do not match plain gradient descent; they match GD++, gradient descent with an iterative curvature correction — a result of this paper’s own. Read as NOTE-280. | Active |
| LIT-534 | What Can Transformers Learn In-Context? A Case Study of Simple Function Classes | Garg, Tsipras, Liang and Valiant (2022), ARXIV-2208.01066, NeurIPS 2022 — the setup that turned “how does in-context learning work” into a question with a checkable answer. A 9.5M-parameter GPT-2 trained from scratch on sequences of (x, f(x)) pairs in-context-learns unseen linear functions at the accuracy of the optimal least-squares estimator, and sparse linear functions, two-layer ReLU networks and depth-4 decision trees at or above their task-specific algorithms. Read as NOTE-279. | Active |
| LIT-535 | Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression | Fu, Chen, Jia and Sharan (2023), ARXIV-2310.17086, NeurIPS 2024 — the challenge from inside the same setup. In Garg et al.’s exact training regime, successive transformer layers track Iterative Newton linearly (roughly 3 iterations per middle layer) and gradient descent only exponentially; on data with condition number 100 the transformer is unaffected while GD needs 2,000 steps, which twelve layers cannot hold. Theorem 5.1 builds k Newton iterations in k + 8 layers with O(d) width. Read as NOTE-278. | Active |
| LIT-536 | Do pretrained Transformers Learn In-Context by Gradient Descent? | Shen, Mishra and Khashabi (2023), ARXIV-2310.08540, ICML 2024 — the challenge from outside. It separates two hypotheses the field had been conflating: that ICL in a naturally pretrained model is gradient descent, and that transformer weights exist that simulate it. Theorem 1 says an algorithm equivalent to ICL must share its order sensitivity, and gradient descent is order-stable while ICL is not; on LLaMA-7B, ICL and fine-tuning disagree on accuracy, top-10 token overlap and overlap cosine similarity across four datasets. Read as NOTE-276. | Active |
| LIT-537 | Grokking as the Transition from Lazy to Rich Training Dynamics | Kumar, Bordelon, Gershman and Pehlevan (2023), ARXIV-2310.06110, ICLR 2024 — the counterexample. A two-layer MLP groks on modular arithmetic with no weight decay at all, and its parameter norm rises through the transition, which the weight-norm accounts cannot produce. The proposed replacement: grokking is the network leaving the lazy (linearised) regime and starting to learn features, controlled by an output-scale parameter α and by the alignment between the initial neural tangent kernel and the target. Read as NOTE-284. | Active |
| LIT-538 | Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets | Power, Burda, Edwards, Babuschkin and Misra (2022), ARXIV-2201.02177 — the paper that named grokking. A small transformer on division mod 97 at 50% data reaches near-perfect training accuracy in under 10³ steps and near-perfect validation accuracy around 10⁶, with almost no generalization before 10⁵. The regime dependence is measured here, not later: near 25–30% data on S₅, removing 1% of the training set raises median time to generalization by 40–50%, while time to fit stays at 10³–10⁴ throughout. Weight decay more than halves the data needed. Read as NOTE-281. | Active |
| LIT-539 | Explaining grokking through circuit efficiency | Varma, Shah, Kenton, Kramár and Kumar (2023), ARXIV-2309.02390, DeepMind — the account that earns its place by predicting things nobody had seen. Memorising and generalising circuits both fit the training set; weight decay prefers whichever produces more logit per unit parameter norm; memorisation gets less efficient as the dataset grows and generalisation does not, so there is a critical dataset size D_crit where the preference flips. From that the paper predicts ungrokking and semi-grokking, and then observes both. Read as NOTE-283. | Active |
| LIT-540 | Omnigrok: Grokking Beyond Algorithmic Data | Liu, Michaud and Tegmark (2022), ARXIV-2210.01117, ICLR 2023 — the paper that answers whether grokking is specific to algorithmic data. No: it is induced on MNIST, IMDb and QM9 by shrinking the training set and scaling the initialization, and eliminated on algorithmic data by constraining the weight norm. The proposed mechanism is the “LU” mismatch — reduced training loss is L-shaped in weight norm and test loss U-shaped, so above the critical norm the two disagree. Read as NOTE-282. | Active |
| LIT-541 | Deep Learning is Singular, and That’s Good | Murfet, Wei, Gong, Li, Gell-Redman and Quella (2020), ARXIV-2010.11560 — the invitation, and the trunk under a school the record has been citing without holding. Neural networks are singular statistical models: the set of optimal parameters is a real analytic variety with singularities, so the Laplace approximation and anything that divides by the determinant of the Hessian do not apply. The correct effective parameter count is the real log canonical threshold λ, which counts how many directions change the model, not how sharply the loss rises in them. Read as NOTE-286. | Active |
| LIT-542 | The Local Learning Coefficient: A Singularity-Aware Complexity Measure | Lau, Furman, Wang, Murfet and Wei (2023), ARXIV-2308.12108 — the instrument. It localizes Watanabe’s learning coefficient to a single minimum, defines it as the exponent in V(ε) = c ε^λ (−log ε)^{m−1} + …, and gives an SGLD-based estimator λ̂(w*) = n β* [E_{w|w*,β*,γ} L_n(w) − L_n(w*)] with β* = 1/log n. Validated against known theoretical values on deep linear networks up to 100M parameters. On ResNet18/CIFAR10 it separates training configurations whose training losses have all collapsed to zero. Read as NOTE-285. | Active |
| LIT-543 | Loss Landscape Degeneracy and Stagewise Development in Transformers | Hoogland, Wang, Farrugia-Roberts, Carroll, Wei and Murfet (2024), ARXIV-2402.02364, TMLR 2025 — the instrument applied. Tracking the LLC through training divides two transformers into five stages each, and the boundaries coincide with structural changes the loss curve does not show: bigrams, then n-grams, then previous-token heads, then the induction circuit. In the in-context regression transformer, in-context learning is acquired in LR2 and then deteriorates across LR3 and LR4 as the model specializes to its pre-training distribution. Read as NOTE-287. | Active |
| LIT-544 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei and Murfet (2023), ARXIV-2310.06301 — the paper that separates two things everyone calls a phase transition. In the Toy Model of Superposition at two hidden dimensions, regular k-gons are critical points with derivable local learning coefficients; the free energy formula F_n ≈ min_α [n L_n(w*_α) + λ_α log n + c_α] then predicts a Bayesian 5 → 6 transition at n_cr ≈ 600, observed at 600 ≤ n ≤ 700. The same critical points govern SGD plateaus — and the 5 → 6 transition is not one of the dynamical ones. Read as NOTE-288. | Active |
| LIT-548 | Feature Learning in Infinite-Width Neural Networks | Yang and Hu (2020), ARXIV-2011.14522 — Tensor Programs IV, where µP is introduced. Among parametrizations that scale initialization, multipliers and learning rate with width, every stable, nontrivial one either learns features in the wide limit or trains as a kernel, never both. The standard parametrization is stable only at an O(1/width) learning rate, which puts it in the kernel regime. µP is the one that updates every layer maximally, and its maximum stable learning rate does not move with width. | Active |
| LIT-549 | ReFT: Representation Finetuning for Language Models | Wu, Arora et al. (2024), ARXIV-2404.03592. Freeze the model and train a low-rank edit to the hidden state at a few prompt positions: h + Rᵀ(Wh + b − Rh), with R having orthonormal rows. It uses 15–65× fewer parameters than LoRA. It leads on commonsense QA (LLaMA-7B, 80.2 against LoRA’s 74.7), leads on GPT-4-judged instruction following, and is level on GLUE. It loses on arithmetic chain of thought (GSM8K 26.0 against 37.5). Baseline numbers are copied from earlier papers. | Active |
| LIT-550 | Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains | Tancik, Srinivasan, Mildenhall et al. (2020), ARXIV-2006.10739. A plain MLP on low-dimensional coordinates learns high frequencies so slowly that in practice it never does. In the NTK picture this is because the kernel’s eigenvalues fall off fast with frequency. Mapping the input through sinusoids makes the kernel stationary, with a bandwidth set by the frequencies. Random Gaussian frequencies beat NeRF’s positional encoding on all seven tasks, and only their scale matters. | Active |
| LIT-553 | Cold Diffusion: Inverting Arbitrary Image Transforms Without Noise | Bansal et al. (2022), ARXIV-2208.09392. Train a network to invert blur, masking, downsampling or snow at every severity, then sample by alternating restoration and re-degradation, and something generative happens without Gaussian noise. The durable result is the sampler, x_{s−1} = x_s − D(x̂₀, s) + D(x̂₀, s−1), which is exact for degradations linear in severity whatever the restorer’s errors. The headline, that noise is not needed, is contradicted by its own Table 5: noiseless blur generation is FID 97.00 on CelebA against 23.11 with noise, and a σ = 0.002 noise sprinkle brings it to 49.45. | Active |
| LIT-555 v2 | Warm Diffusion: Recipe for Blur-Noise Mixture Diffusion Models | Hsueh et al. (2025), ARXIV-2511.16904. Degrade with blur and noise together, and predict the blurry image and the missing high frequencies with two heads. Blur lets the model use the dependency of fine detail on coarse structure. Noise keeps intermediate states on the data manifold. Sweeping the blur-to-noise ratio on CIFAR-10 at 35 steps gives FID 1.97 at BNR = 0 (EDM), 1.85 at BNR = 0.5 and 11.97 at BNR = 10. A small gain, and a large loss as the process goes cold. | Active |
| LIT-556 | Tensor Programs III: Neural Matrix Laws | Yang (2020), ARXIV-2009.10685. At random initialization and infinite width, a network’s pre-activations become asymptotically free (in the random-matrix sense) of its weights, for any architecture: the Free Independence Principle. This rigorously justifies the Jacobian singular-value calculations behind dynamical isometry, which had assumed it, and gives a second justification of the gradient independence assumption. The general Master Theorem is also what TP-IV uses. | Active |
| LIT-557 | Tensor Programs II: Neural Tangent Kernel for Any Architecture | Yang (2020), ARXIV-2006.14548. The neural tangent kernel of a randomly initialized network of any standard architecture converges to a deterministic limit as width grows, and the paper shows how to compute it. It also gives a checkable condition (Simple GIA Check) under which the common heuristic that backprop’s Wᵀ is independent of forward W is correct, and cases where that heuristic gives wrong answers. | Active |
| LIT-558 | Tensor Programs I: Wide Feedforward or Recurrent Neural Networks of Any Architecture are Gaussian Processes | Yang (2019), ARXIV-1910.12478. At random initialization, any network built from MLPs, RNNs, convolutions, pooling, skip connections, attention, batchnorm or layernorm converges to a Gaussian process as width grows. This is the NNGP correspondence extended from MLPs and CNNs to all standard architectures, proved by expressing the forward pass in the NETSOR language. The first paper of the series and the tutorial for its machinery. A statement about initialization, not training. | Active |
| LIT-563 | The Role of ImageNet Classes in Fréchet Inception Distance | Kynkäänniemi, Karras, Aittala, Aila, Lehtinen (2022), ARXIV-2203.06026. FID’s feature space sits one affine map from ImageNet class logits, so it attends to whatever ImageNet class an image evokes. That is often not the subject: seat belts and bow ties on FFHQ faces. Resampling a fixed generator’s outputs to match those statistics cuts FFHQ FID from 5.30 to 1.78 (−66%) while a CLIP-space distance moves 2.76 → 2.64. A Projected FastGAN with an ImageNet-pretrained discriminator matches StyleGAN2’s FID and loses to it in CLIP space and with humans. | Active |
| LIT-565 | FLUX-Reason-6M & PRISM-Bench: A Million-Scale Text-to-Image Reasoning Dataset and Comprehensive Benchmark | Fang et al. (2025), ARXIV-2509.09680. 6 million FLUX-generated images with 20 million bilingual captions, organized by six characteristics (imagination, entity, text rendering, style, affection, composition), and with “Generation Chain-of-Thought” breakdowns. Curation cost 15,000 A100 GPU-days. Released with PRISM-Bench, seven tracks scored by vision-language models, used to evaluate 19 text-to-image models. A dataset and a benchmark, with no training ablation read here. | Active |
| LIT-568 | Are Sparse Autoencoders Useful? A Case Study in Sparse Probing | Kantamneni, Engels et al. (2025), ARXIV-2502.16681. On 113 probing datasets over Gemma-2-9B, and replicated on Llama-3.1-8B, logistic-regression probes are compared with probes on SAE latents. Selection is done the way a practitioner would, by validation AUC over a “quiver” of methods. Adding SAE probes does not help in standard conditions (−0.003 AUC) or under data scarcity, class imbalance, label noise or covariate shift. Earlier positive results for SAE probes disappear against stronger baselines, including some of the authors’ own. | Active |
| LIT-569 | Eliciting Latent Predictions from Transformers with the Tuned Lens | Belrose et al. (2023), ARXIV-2303.08112. The logit lens is unreliable: it fails on BLOOM, OPT-125M and GPT-Neo, and even where it works it is biased, 4–5 bits of KL from the final marginal on GPT-Neo-2.7B. The tuned lens trains one affine translator per layer by distillation toward the final logits. It has lower perplexity and bias on every model family tested up to 20B, and features it relies on also matter to the model (Spearman 0.89). Version 6 (2025) adds that every lens in the paper was undertrained, and that Muon trains them much better. | Active |
| LIT-570 | interpreting GPT: the logit lens | nostalgebraist (2020), a LessWrong post. Apply GPT-2’s final layer norm and unembedding to each intermediate layer’s residual stream and read the result as a next-token distribution. On GPT-2 those distributions make sense and converge on the final prediction well before the last layer, and they almost never resemble the input tokens. The post calls it a partial view. | Active |
| LIT-571 | Scaling and evaluating sparse autoencoders | Gao et al. (2024), ARXIV-2406.04093. TopK sparse autoencoders keep the k largest latents, which sets L0 directly and replaces the L1 penalty. They beat ReLU SAEs on the sparsity-reconstruction frontier, and the gap widens with scale. Encoder-as-decoder-transpose initialization plus an auxiliary dead-latent loss leave 7% dead latents in a 16M-latent SAE on GPT-4, against up to 90% without them. Clean scaling laws in latents and k. That SAE scores 98.2% “loss recovered”, yet substituted into GPT-4 it matches a model trained on 10% of GPT-4’s compute. | Active |
| LIT-572 | FLUX.2: Analyzing and Enhancing the Latent Space of FLUX – Representation Comparison | Black Forest Labs (2025), technical report. DiT-XL flow models on ImageNet 256² in four latent spaces (SD-VAE, FLUX.1-VAE, FLUX.2-VAE, RAE), 30 configurations each: training timestep distribution, training shift and sampling shift, with and without REPA. The training shift moves FID by 61–86%, and without tuning it per space the rankings flip. FLUX.1’s VAE reconstructs far better than SD’s and is less learnable (gFID 10.1 against 7.7). FLUX.2’s VAE, with 8× SD’s dimensionality and semantic regularization, gets both (LPIPS 0.27, gFID 3.7). FLUX.2 is that latent plus Mistral-3 24B and a rectified-flow transformer. | Active |
| LIT-574 | Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models | Hase, Bansal, Kim, Ghandeharioun (NeurIPS 2023), ARXIV-2301.04213. On 652 facts known to GPT-J, causal tracing often peaks outside the layers ROME and MEMIT edit. Yet editing at those layers works regardless. In a regression of ROME’s rewrite score, the edit layer explains 94.7% of the variance, and adding the fact’s tracing effect raises it to 94.8%. Robust across editing methods (ROME, MEMIT, fine-tuning), metrics, models and datasets. Localization, as causal tracing measures it, does not tell you where to edit. | Active |
| LIT-575 | Transformer Feed-Forward Layers Are Key-Value Memories | Geva, Schuster, Berant, Levy (2021), ARXIV-2012.14913. Read each FFN hidden unit as a memory. Its input weight is a key that fires on human-recognizable prefix patterns: shallow n-grams in lower layers, semantic topics in upper ones. Its output weight is a value that, projected through the output embedding, favors next tokens. The evidence is 160 sampled keys in one 16-layer WikiText-103 model. Top value predictions agree with the keys’ triggers at most 3.5% of the time, in upper layers only. Layer outputs are mostly compositions, matching no single memory’s prediction. | Active |
| LIT-576 | Mass-Editing Memory in a Transformer | Meng, Sen Sharma, Andonian, Belinkov, Bau (2023), ARXIV-2210.07229. MEMIT writes thousands of facts at once by spreading each update over a range of mid-layer MLPs (layers 3–8 in GPT-J) in one batched least-squares solve. At 10,000 CounterFact edits it scores 85.8 on GPT-J (ROME 50.3, fine-tuning 67.6 with generation collapse). Neighborhood success falls from 83.5 to 73.7. | Active |
| LIT-577 | Evaluating the Ripple Effects of Knowledge Editing in Language Models | Cohen, Biran, Yoran, Globerson, Geva (2023), ARXIV-2307.12976. An edit implies other facts: logical consequences, multi-hop compositions, aliases. RippleEdits tests 5K edits on six such criteria. ROME, MEMIT and MEND average 38–66 across GPT-2, GPT-J, GPT-NeoX and LLaMA. Logical generalization can be as low as 5.5–7.0. Stating the new fact in the prompt (in-context editing) scores best, beating ROME by more than 10 points on GPT-NeoX and by 29 on LLaMA. | Active |
| LIT-578 | Locating and Editing Factual Associations in GPT | Meng, Bau, Andonian, Belinkov (2022), ARXIV-2202.05262. Causal tracing: corrupt the subject tokens, then restore single hidden states to see which carry the fact. The strongest effects are at mid-layer MLPs on the last subject token. ROME writes a new fact as a rank-one update to one such MLP, treated as a linear associative memory. On CounterFact it scores 89.2 (GPT-2 XL) and 91.5 (GPT-J) against at most 68.7 for the baselines. It edits one fact at a time and degrades after about 10. | Active |
| LIT-579 | Where Did the Gap Go? Reassessing the Long-Range Graph Benchmark | Tönshoff, Ritzert, Rosenbluth, Grohe (2023), ARXIV-2309.00367. On the Long-Range Graph Benchmark, graph transformers were reported to beat message-passing GNNs. After a basic hyperparameter sweep within the same 500k-parameter budget, GCN, GINE and GatedGCN all beat GPS on Peptides-func and Peptides-struct. Most of the gain comes from a 2-layer prediction head instead of a linear one. On the superpixel datasets, input feature normalization accounts for at least half the gain of most models, and GPS stays ahead there. The link-prediction metric’s code computed raw MRR where the paper specified filtered MRR. | Active |
| LIT-580 | How Powerful are Graph Neural Networks? | Xu, Hu, Leskovec, Jegelka (ICLR 2019), ARXIV-1810.00826. Message-passing GNNs are at most as discriminative as the 1-WL isomorphism test, and they reach that bound when aggregation and readout are injective on multisets. Sum aggregation with an MLP update (GIN) is; mean (GCN) and max (GraphSAGE) are not. On featureless Reddit graphs, mean-aggregation GNNs are at chance (50.0, 20.0) and GIN scores 92.4 and 57.5. On the other seven benchmarks, most differences are within a standard deviation. | Active |
| LIT-581 | Pitfalls of Graph Neural Network Evaluation | Shchur, Mumme, Bojchevski, Günnemann (R2L workshop, NeurIPS 2018), ARXIV-1811.05868. GCN, GAT, MoNet and GraphSAGE were run with one shared training and tuning procedure, over 100 random splits × 20 seeds on 8 datasets. GCN has the best average rank (2.3) and relative accuracy (99.4%). On Planetoid’s single split GAT wins Cora and Citeseer; on another random split of the same sizes GCN wins them. GAT’s lower average on Amazon comes from 138 of 2,000 runs collapsing. | Active |
| LIT-586 v2 | LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale | Dettmers et al. (2022), ARXIV-2208.07339. Int8 inference for the feed-forward and attention projections, at no measured cost in quality up to 175B, by isolating a small set of systematically emergent outlier feature dimensions into a 16-bit multiplication and quantizing the other 99.9% vector-wise. | Active |
| LIT-588 v2 | Learning Transferable Visual Models From Natural Language Supervision | Radford et al. (2021), ARXIV-2103.00020. Train an image encoder and a text encoder to match 400M (image, text) pairs contrastively, then build a classifier for any label set by embedding its class names. The most durable result is not the zero-shot accuracy — it is that supervised adaptation which raises ImageNet by 9.2% buys no average robustness. | Active |
| LIT-589 | Representation Learning with Contrastive Predictive Coding | Oord et al. (2018), ARXIV-1807.03748. Predict the future in latent space and score it as a density ratio against sampled negatives, rather than reconstructing the observation. The loss is named InfoNCE here, and it is the objective CLIP, SimCLR and MoCo all run; the bound it maximizes is capped at log N, which the folklore about negatives leaves out. | Active |
| LIT-597 | Reading Tea Leaves: How Humans Interpret Topic Models | Chang, Boyd-Graber, Gerrish, Wang and Blei (NIPS 2009). Two Mechanical Turk tasks measure whether a topic model’s latent space means anything to people. In word intrusion, subjects find the odd word out among a topic’s top five. In topic intrusion, they find the topic that does not belong to a document. On pLSI, LDA and CTM at 50, 100 and 150 topics over NYT and Wikipedia, CTM has the best held-out likelihood and the worst human scores. LDA is usually best. The paper concludes that likelihood does not track interpretability, and it reads the trend as a negative correlation. | Active |
| LIT-600 | Automatic Evaluation of Topic Coherence | Newman, Lau, Grieser and Baldwin (NAACL 2010). Humans rated 237 LDA topics from news and books for coherence. The paper then tested automated scores built from WordNet, Wikipedia and Google. The simplest measure, the average pointwise mutual information of a topic’s top-word pairs counted in 10-word windows over Wikipedia, reaches a Spearman correlation of about 0.77 with the human mean. Inter-annotator agreement, measured differently, is 0.73–0.82. WordNet measures are erratic, and Google drops from 0.80 on news to 0.51 on books. | Active |
| LIT-604 | Efficient Estimation of Word Representations in Vector Space | Mikolov, Chen, Corrado and Dean (2013), ARXIV-1301.3781. Two log-linear architectures drop the hidden layer of a neural language model. CBOW predicts a word from its averaged context, and skip-gram predicts the context from the word. The paper also introduces the Semantic-Syntactic Word Relationship test (8,869 semantic and 10,675 syntactic analogy questions), answered by vector arithmetic. On it, skip-gram at 300 dimensions reaches 53.3% on 783M words in about 3 days on one CPU, far ahead of earlier public embeddings. The comparisons are not controlled for data size or dimension. | Active |
| LIT-606 | The Linear Representation Hypothesis and the Geometry of Large Language Models | Park, Choe and Veitch (ICML 2024), ARXIV-2311.03658. “Concepts are directions” has three readings: a shared difference between word pairs, a linear probe, and a steering vector. The paper formalizes the first with counterfactual pairs in the unembedding and embedding spaces, and proves the first is a probe and the second a steering vector. Training identifies the representation only up to an invertible linear map, so Euclidean cosine has no special meaning. An inner product under which causally separable concepts are orthogonal unifies the two spaces, and one such product is the inverse covariance of the unembedding rows. Evidence is 27 concepts on LLaMA-2 7B, with a Gemma-2B comparison in the appendix. | Active |
| LIT-607 v3 | Improving Distributional Similarity with Lessons Learned from Word Embeddings | Levy, Goldberg and Dagan (TACL 2015). Takes the design choices bundled into word2vec and GloVe (dynamic windows, subsampling, shifted PMI, context-distribution smoothing, adding context vectors, eigenvalue weighting), transfers them to count-based PPMI and SVD, and compares all four methods under the same tuning: 672 representations, 8 similarity and analogy datasets, and cross-validated selection. Once tuned alike, no method wins consistently. A single hyperparameter often matters more than the choice of method. SGNS beats GloVe on every task, reversing GloVe’s headline comparison. Smoothing the context distribution by 0.75 is the one setting that helps everywhere. | Active |
| LIT-608 | Don’t count, predict! A systematic comparison of context-counting vs. context-predicting semantic vectors | Baroni, Dinu and Kruszewski (ACL 2014). The first broad comparison of count-based distributional vectors (36 PPMI/LMI models, full, SVD and NMF) against word2vec CBOW vectors (48 models), on 14 lexical-semantics benchmarks and a 2.8B-token corpus. Predict wins on almost every task, often by 10 points or more, and is far more robust to its worst settings. The count models were not given word2vec’s context smoothing or negative shift, and SVD kept its singular-value weighting. LIT-607 traces the result to those differences and finds no consistent winner when both are tuned alike. | Active |
| LIT-611 v2 | GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium | Heusel et al. (2017), ARXIV-1706.08500. Where FID comes from — and it comes from a subsection of a paper about two time-scale update rules and Nash equilibria. Fit a Gaussian to Inception activations for each distribution and take the Wasserstein-2 distance between them. Validated by monotonicity under six synthetic corruptions. | Active |
| LIT-614 | Improved Techniques for Training GANs | Salimans et al. (2016), ARXIV-1606.03498. Where the Inception Score comes from — exp(E_x KL(p(y|x) ‖ p(y))), proposed as a cheap stand-in for a Mechanical Turk study it correlates with, on 50k samples because part of what it measures is diversity. The incumbent LIT-611 displaced, and the paper that first tied generative evaluation to ImageNet class structure on purpose. | Active |
| LIT-615 | Demystifying MMD GANs | Bińkowski et al. (2018), ARXIV-1801.01401. Where KID comes from, and where the case against comparing FID numbers comes from. The plug-in FID estimator is biased, no unbiased estimator of FID exists, and at d=2048 with 50,000 samples it reliably returns the wrong ordering between two models with a standard deviation small enough to hide it. KID is the squared MMD in the same Inception features, with a cubic kernel and a simple unbiased estimator. | Active |
| LIT-616 | On Calibration of Modern Neural Networks | Guo et al. (2017), ARXIV-1706.04599. Modern networks are badly calibrated where their 1998-era ancestors were not, and depth, width, BatchNorm and less weight decay each make it worse while improving accuracy. The mechanism is that NLL overfits while 0/1 loss keeps improving. The remedy is one scalar: temperature scaling beats every richer method tried, including the ones that contain it. | Active |
| LIT-617 | Scaling Language Models: Methods, Analysis & Insights from Training Gopher | Rae et al. (2021), ARXIV-2112.11446. Six models on one dataset at one token budget, 44M to 280B, evaluated on 152 tasks — a controlled study of parameter scale. The gains are nonuniform: 51.2% of tasks improve by over 25%, 10.5% not at all, and Gopher is worse than its own smaller models on Abstract Algebra, Temporal Sequences and High School Mathematics. Also the allocation LIT-068 corrects: 280B parameters on 300B tokens is 1.07 tokens per parameter against Chinchilla’s 20. | Active |
| LIT-628 | Better plain ViT baselines for ImageNet-1k | Beyer, Zhai and Kolesnikov (2022), ARXIV-2205.01580. Four pages that move a plain ViT-S/16 on ImageNet-1k from 66.8% to 80.0% with five changes, none of them novel and none of them a regulariser. The original recipe scores 66.8 / 67.2 / 67.1 at 90 / 150 / 300 epochs — it cannot use more training at all. The belief the abstract names is that ViT needs sophisticated regularisation at this scale; the answer is that standard augmentation is sufficient. | Active |
| LIT-638 | A Length-Extrapolatable Transformer | Sun et al. (2022), ARXIV-2212.10554. Defines attention resolution — a scalar for how well an attention pattern distinguishes token distance, estimable before training — and measures it across the extrapolation boundary. RoPE collapses 0.91 → 0.08 at twice its training length; ALiBi holds 0.81 → 0.88 and is the only incumbent that does. The record holds both papers and four practices about RoPE, and had no measure of this. | Active |
| LIT-639 | ResiDual: Transformer with Dual Residual Connections | Xie et al. (2023), ARXIV-2304.14802. Gives the representation-collapse argument a rate: in a Pre-LN transformer the per-layer change in the normalised hidden state decays as O(1/√k), and adding a layer to an N−1 layer model moves the output by O(1/√N). Also measures what Pre-LN’s no-warmup claim costs — 32.28 against 35.12 BLEU at E6D6, and 31.82 against 35.18 at E12D12. | Active |
| LIT-641 | Attention Approximates Sparse Distributed Memory | Bricken and Pehlevan (2021), ARXIV-2111.05498. Attention’s update rule is Kanerva’s sparse distributed memory read, given two conditions: L² normalized vectors and a fitted softmax temperature β. The retrodiction is the interesting part — those two conditions are QK-norm, published a year earlier for unrelated reasons. Trained QK-norm heads learn β ∈ [10, 25], the range that interpolates between SDM’s optimality criteria. | Active |
| LIT-644 v2 | Building Normalizing Flows with Stochastic Interpolants | Albergo and Vanden-Eijnden (2022), ARXIV-2209.15571. A continuous normalizing flow can be trained without ODE solves by regressing a velocity onto the time derivative of any interpolant between two densities. It is one of three concurrent origins of this simulation-free objective, with Flow Matching and Rectified Flow. Its path is not the straight line: every experiment uses the trigonometric cos/sin interpolant, which SiT later calls GVP. The image results arrived only in v3, and they are quoted against diffusion baselines, not matched against them. | Active |
| LIT-645 | Stochastic Interpolants: A Unifying Framework for Flows and Diffusions | Albergo, Boffi and Vanden-Eijnden (2023), ARXIV-2303.08797. The earlier interpolant objective is extended with a latent Gaussian term. One learned velocity and score then give an ODE and a family of SDEs, all sharing the interpolant’s marginals, and the SDE’s diffusion coefficient can be chosen after training. For SDEs the regression losses bound the KL, and for the ODE they do not. This is the framework SiT instantiates. The evidence that SDE sampling beats ODE sampling is a 2-D checkerboard and a 128-D Gaussian mixture. The paper reports no image metric and no diffusion baseline. | Active |
| LIT-648 | Improving Image Generation with Better Captions | Betker et al., OpenAI and Microsoft (2023), no arXiv id. Identical T5-conditioned latent diffusion models are trained on the same images with alt-text, short synthetic or descriptive synthetic captions. The descriptive ones give higher CLIP score, and more synthetic in the blend is better up to the 95% tested. The controlled evidence is CLIP score alone, at 256px, and on ground-truth prompts the gain is inside the checkpoint noise. The paper does not describe the DALL-E 3 model, and it says its DALL-E 3 comparisons are not a measurement of recaptioning. | Active |
| LIT-650 | Competition-Level Code Generation with AlphaCode | Li et al. (2022), ARXIV-2203.07814. Top 54.3% on ten Codeforces contests, by sampling up to a million programs per problem and filtering 99% of them away on the example tests. Two things the record needs are incidental to that: HumanEval’s false-positive rate, hand-measured at 30%, and the fact that pass@k assumes every sample can be submitted and is therefore an upper bound on what a system with a submission budget achieves. | Active |
| LIT-653 | Softmax is not Enough (for Sharp Size Generalisation) | Veličković, Perivolaropoulos, Barbero and Pascanu (2024), ARXIV-2410.01104. Twenty-three practices in this record name the softmax and no note held it. This one proves a limit on it: in a Transformer over a finite vocabulary, attention coefficients must decay towards zero as the number of items grows, so a head that is sharp in distribution cannot stay sharp on larger inputs. Max retrieval falls from 98.6% at 16 items to 12.4% at 16,384. | Active |
| LIT-665 | Transformer Language Models without Positional Encodings Still Learn Positional Information | Haviv, Ram, Press, Izsak and Levy (2022), ARXIV-2203.16634. The result the record’s position-encoding cluster rests on, credited here to a paper a year younger. A decoder-only LM with no positional encoding at all is competitive; probing shows it acquires absolute position within four layers. The mechanism is the causal mask, and the control that shows it is a bidirectional MLM, where NoPos perplexity is 147.18 against 4.00. | Active |
| LIT-667 | Grokked Transformers are Implicit Reasoners: A Mechanistic Journey to the Edge of Generalization | Wang, Yue, Su and Sun (2024), ARXIV-2405.15071. Grokking outside algorithmic data, on reasoning over stored facts — and a named correction to the critical-data-size hypothesis the record holds as THEORY-071. With the inferred/atomic ratio φ held at 9.0, varying the training-set size changes nothing; varying φ at fixed size moves grokking speed monotonically and at φ = 18.0 all but removes it. Also traces the generalizing circuit for two tasks and explains from its layout why composition never generalizes out of distribution while comparison does. | Active |
| LIT-669 | Sparse Distributed Memory and Related Models | Kanerva (1992), RIACS TR 92.10, published in 1993 in Hassoun (ed.), Associative Neural Memories (Oxford). SDM restated as a matrix memory: a fixed random address matrix A, a wide layer of hard locations, and a modifiable contents matrix C trained by the outer-product rule. Stored words are weighted by the overlaps of activation sets. The activation probability that maximizes signal-to-noise is p = (2MT)^(-1/3), and capacity is about 10% of the number of locations at one bit error in a thousand. This is the source of two of the three optimal radii LIT-641 fits β against. The third, critical distance, is not in it. | Active |
| LIT-670 | BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding | Devlin, Chang, Lee and Toutanova (2018), ARXIV-1810.04805. The model 89 files in this record referred to and none held. Two things it is cited for here turn out not to be in it: the 15% masking rate was never swept — “in all of our experiments, we mask 15%”, and the one appendix table headed “Masking Rates” varies the 80/10/10 replacement split at a fixed 15% — and its NSP ablation removes the loss while keeping the paired-segment input format, which LIT-671 later separates and reverses. | Active |
| LIT-671 | RoBERTa: A Robustly Optimized BERT Pretraining Approach | Liu, Ott, Goyal, Du, Joshi, Chen, Levy, Lewis, Zettlemoyer and Stoyanov (2019), ARXIV-1907.11692. A replication study that finds BERT was significantly undertrained and, more usefully here, that BERT’s own NSP ablation could not have shown what it claimed: it removed the loss while keeping the paired-segment input format. Separate the two and removing NSP matches or slightly improves. The paper is careful about its own margins — dynamic masking is “comparable or slightly better”, and byte-level BPE is adopted despite being slightly worse. | Active |
| LIT-672 | Should You Mask 15% in Masked Language Modeling? | Wettig, Gao, Zhong and Chen (2022), ARXIV-2202.08005. The sweep BERT never ran, and the answer the record has been openly missing: 15% is not universally optimal, and the optimum tracks model size — 40% at 354M, 20% at 124M, 15% at 51M. It also decomposes the masking rate into a corruption rate and a prediction rate whose effects are antagonistic, and uses that to find BERT’s 80-10-10 substitution rule worse than plain [MASK]. | Active |
| LIT-673 | Averaging Weights Leads to Wider Optima and Better Generalization | Izmailov, Podoprikhin, Garipov, Vetrov and Wilson (2018), ARXIV-1803.05407. Average the weights visited along the tail of an SGD trajectory under a cyclical or high constant learning rate. The paper LIT-478 named as the first thing missing from this record’s weight-space ensembling coverage, and the baseline SOTA-288’s promotion condition asks for. From pretrained torchvision models, ten further epochs gives ImageNet top-1 76.15 → 76.97 on ResNet-50 and 78.31 → 78.94 on ResNet-152, at the inference cost of one model. One gotcha, stated in the paper: batch normalization statistics have to be recomputed with a pass over the data. | Active |
| LIT-674 | Robust fine-tuning of zero-shot models | Wortsman, Ilharco, Kim, Li, Kornblith, Roelofs, Gontijo-Lopes, Hajishirzi, Farhadi, Namkoong and Schmidt (2021), ARXIV-2109.01903. Fine-tuning a zero-shot model buys target accuracy and costs robustness. Linearly interpolating the zero-shot and fine-tuned weights recovers both: at α = 0.5, +3.5 to +23.2 pp under six distribution shifts against the fine-tuned model while reference accuracy falls by at most 0.3 pp and usually rises. No extra cost at training or inference. And it names the precondition: averaging all the layers of two unrelated networks “typically fails, achieving no better accuracy than a randomly initialized neural network” — this works because the trajectory is shared. | Active |
| LIT-675 | Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time | Wortsman, Ilharco, Gadre, Roelofs, Gontijo-Lopes, Morcos, Namkoong, Farhadi, Carmon, Kornblith and Schmidt (2022), ARXIV-2203.05482. The dependency SOTA-407’s consensus note named as unheld. A hyperparameter sweep throws away every model but one; average them instead. The greedy recipe — sort by held-out accuracy, add each model only if held-out accuracy improves — is “no worse than the best individual model on the held-out validation set” by construction, at O(1) inference cost against an ensemble’s O(k). +0.7 pp on CLIP and +0.5 pp on ALIGN over the best sweep member. It does not move that practice’s consensus, because it is the same first author. | Active |
| LIT-676 v4 | DPM-Solver++: Fast Solver for Guided Sampling of Diffusion Probabilistic Models | Lu, Zhou, Bao, Chen, Li and Zhu (2022), ARXIV-2211.01095. The regime SOTA-203 does not mention. At a classifier guidance scale of 8.0 on ImageNet 256×256, FID at 10 function evaluations is 13.04 for first-order DDIM, 114.62 for DPM-Solver-2 and 164.74 for DPM-Solver-3 — higher order is monotonically catastrophic, and the practice’s own framing of DDIM as “the least accurate member of the family” inverts. Two causes, one of which the record already holds as SOTA-202. The fix is order 2, multistep, on the data-prediction parameterization. | Active |
| LIT-681 | Cramming: Training a Language Model on a Single GPU in One Day | Geiping and Goldstein (ICML 2023), ARXIV-2212.14034. A BERT-style MLM trained from scratch for 24 hours on one consumer GPU, which reaches 79.2–80.4 GLUE against BERT-base’s 80.9. The transferable finding is negative: model shape does not beat the scaling law at this budget. Every reshape it tried lands near the same final loss, because a smaller model’s extra throughput cancels its slower learning. So the only architectural lever is to cut step time at constant parameter count. Its data result is the cheapest filter in the record: drop text the tokenizer compresses badly. | Active |
| LIT-684 | Model Merging in Pre-training of Large Language Models | Li et al. (ByteDance Seed, 2025), ARXIV-2505.12082. A uniform average of about ten recent checkpoints on a constant-learning-rate (WSD stable-phase) run beats the latest checkpoint at every size tested, up to 20B/200B MoE and 70B dense. It roughly tracks where an anneal would land. The travelling claim, that constant LR plus merging “can effectively match the performance of an annealed model at any point”, rests on one fork of one 1.3B/13B model. At the end of that fork’s 250B-token decay, the annealed model is about 1.5 MMLU ahead. The baselines in the big tables are unannealed checkpoints. | Active |
| LIT-685 | Sub-Scaling Laws: On the Role of Data Density and Training Strategies in LLMs | Chen, Wang, Xiao, Wang, Chen, Cai, He and Wang (ACL 2025), ARXIV-2507.10613, published at ACL as “Revisiting Scaling Laws for Language Models: The Role of Data Quality and Training Strategies”. It claims that gains decelerate faster than a power law when data is “dense” or when tokens per parameter run high, and fits a modified Chinchilla law over more than 400 models from 20M to 7B. None of it should be leaned on. The density metric’s embedding, dimension and cluster count are never stated. The fitted law has no density term, only two logistic multipliers on D/N bounded in [1.5, 2). And the paper’s own Table 4 contradicts its “across all model sizes”. | Active |
| LIT-688 v2 | Reconciling Kaplan and Chinchilla Scaling Laws | Pearce and Song (TMLR 2024), ARXIV-2406.12907. Kaplan’s N_opt ∝ C^0.73 against Chinchilla’s C^0.50 is mostly a counting artifact. Kaplan counted non-embedding parameters and fit at small scale, where embeddings are much of the model. Simulating the Chinchilla study under those conditions gives 0.74–0.78. The result is a simulation from Chinchilla’s fitted surface, extrapolated four or more orders of magnitude below its range. The experiments are five models of 0.8–4.6M parameters at context length 16. They also find that the learning-rate schedule, which Chinchilla blamed, is not the main cause. | Active |
| LIT-690 | Resolving Discrepancies in Compute-Optimal Scaling of Language Models | Porian, Wortsman, Jitsev, Schmidt and Carmon (NeurIPS 2024), ARXIV-2406.19146. It reproduces Kaplan’s allocation exponent (0.835 on RefinedWeb, 0.864 on OpenWebText2) over 900+ runs from 5M to 901M, then removes it in steps. Counting the head’s FLOPs takes 0.129 off, a warmup short enough for small models takes 0.104, and tuning learning rate, batch and β₂ per size, with a constant learning rate, takes 0.105, landing on 0.497. Chinchilla’s proposed cause, matching the cosine decay to each run, moves it 0.031 on a side branch. “careful learning rate decay is not essential for the validity of their scaling law.” | Active |
| LIT-692 | Understanding Diffusion Objectives as the ELBO with Simple Data Augmentation | Kingma and Gao (NeurIPS 2023), ARXIV-2303.00848. Every standard diffusion loss is ½∫ w(λ) E‖ε̂ − ε‖² dλ for an implied weighting w. Given w, the loss is invariant to the schedule’s interior shape, and the schedule is only an importance sampler. If w falls monotonically in log-SNR, the loss is the expected ELBO of Gaussian-noise-augmented data. App. D.3 derives that flow matching’s weighting equals v-MSE’s under a cosine schedule. In its 64×64 ablations the training schedule makes no difference, and the weighting and the sampler each move FID by 0.1–0.3. Monotone weightings tie EDM. They do not beat it. | Active |
| LIT-693 v3 | Classifier-Free Diffusion Guidance | Ho and Salimans (2022), ARXIV-2207.12598. The technique SOTA-202, SOTA-203 and SOTA-410 all turn on, which this record had never held. Drop the conditioning on a fraction of training examples and one network learns both scores; at sampling, extrapolate away from the unconditional one. The trade it buys is severe and measured: on ImageNet 64×64, w from 0.1 to 4.0 moves FID from 1.55 to 26.22 while IS goes 66.11 to 260.2. Any FID or IS comparison across guidance scales is therefore meaningless, which is the missing source for SOTA-307’s instruction to search the scale per cell. | Active |
| LIT-694 | Rigor with Machine Learning from Field Theory to the Poincaré Conjecture | Gukov et al. (2024), ARXIV-2402.13321. Stochastic, black-box ML can still yield rigorous results in mathematics and physics along two routes. The first is conjecture generation followed by human proof, or RL searches whose outputs can be checked exactly; this is how an RL/Bayesian-optimized ribbon verifier ruled out over 800 candidate counterexamples to the smooth 4D Poincaré conjecture. The second is importing ML theory, such as NNGP/NN-field-theory and gradient-descent metric flows that include Perelman’s Ricci flow. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-699 v2 | Diffusion Models Beat GANs on Image Synthesis | Dhariwal and Nichol (2021), ARXIV-2105.05233. The classifier-guidance paper, the ablated U-Net the diffusion literature then treated as the default, and the source of the guidance-scale trade. Its title claim is metric-dependent and the paper says so: ADM-G beats BigGAN-deep on FID and recall at 128, 256 and 512, and loses on precision at all three — 0.78 against 0.86, 0.82 against 0.87, 0.87 against 0.88. It also asserts, once, in the introduction, that the gradient scale can be raised “by an order of magnitude without obtaining adversarial examples”, and runs no experiment anywhere in the paper that tests it. | Active |
| LIT-700 | Non-Euclidean Gradient Descent Operates at the Edge of Stability | Islamov, Crawshaw, Cohen and Gower (ICML 2026), ARXIV-2603.05002. Steepest descent under any norm — Spectral GD, ℓ∞-descent, Block CD, and the normalized forms SignGD and normalized Spectral GD — also sharpens to 2/η and trains there, if sharpness is measured in the update’s own norm: max_{‖d‖=1} dᵀ∇²L d. The ordinary top Hessian eigenvalue stays well below 2/η for Spectral GD and ℓ∞-descent, so the Euclidean diagnostic misses the regime entirely. Full-batch, CIFAR-10 and Tiny Shakespeare scale; the non-Euclidean sharpness is a multi-restart Frank-Wolfe estimate, not an exact value. | Active |
| LIT-703 | Improved Precision and Recall Metric for Assessing Generative Models | Kynkäänniemi, Karras, Laine, Lehtinen and Aila (2019), ARXIV-1904.06991. The definitions SOTA-425 rests on: build a k-NN hypersphere approximation of each distribution’s manifold in a feature space, then precision is the fraction of generated samples inside the real manifold and recall the fraction of real samples inside the generated one. The features are VGG-16 fc2, not Inception — the record had this wrong. Its Figure 4 is the demonstration: two StyleGAN setups at FID 16.9 and 16.7 where one produces good samples at low variety and the other is “practically all images broken”, and the FID-optimised configuration has visible facial distortion. | Active |
| LIT-704 | Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning | Gal and Ghahramani (2015; ICML 2016), ARXIV-1506.02142. With dropout before every weight layer and L2 weight decay, the training objective is, for a suitable precision τ and length-scale l, a variational objective whose approximate posterior is Bernoulli-masked weight matrices under a deep Gaussian process. Averaging T stochastic forward passes (“MC dropout”) then gives a predictive mean, and their spread plus τ⁻¹ gives a predictive variance. The equivalence is exact only up to an approximated KL term, and the derivation is in a separate appendix paper. The benchmark comparisons use one-hidden-layer, 50-unit networks on UCI regression. | Active |
| LIT-705 | Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift | Li, Chen, Hu and Yang (2018; CVPR 2019), ARXIV-1801.05134. Dropout changes a unit’s variance between train and test mode, and a batch-norm layer downstream keeps normalizing by the train-mode variance. Dropout 0.5 in every DenseNet bottleneck costs 77.42% → 68.55% on CIFAR-100, and the model then misclassifies its own training data in eval mode. The damage is measured and large. The remedy’s gain is not: one dropout layer after the last BN gives about 0.2 top-1 on ImageNet and mixed, mostly sub-0.3 changes on CIFAR, with no spread reported. | Active |
| LIT-708 | A Controlled Study of Attention-Only Transformers | Ndubuaku et al. (2026), ARXIV-2607.18363. Removing every feed-forward layer from a decoder costs 0.47 nats at matched depth and 0.26 at matched training FLOPs, but only 0.0055 nats (0.27% of loss) at matched parameters once the freed budget goes into attention depth: 20 attention-only layers against 4 standard blocks, 24M parameters, 105B tokens of a synthetic reasoning corpus, with a separate learning-rate sweep for each arm. The remaining gap sits on low-context tokens, where the answer must come from the weights. A pre-registered test on fineweb-edu predicted 0.02–0.05 nats and measured 0.040. Removing QK-norm made the attention-only stack diverge; residual gates and ReZero changed nothing. Everything is at or below 87M parameters. | Active |
| LIT-711 | Do Transformer Modifications Transfer Across Implementations and Applications? | Narang et al. (2021), ARXIV-2102.11972. About fifty published transformer modifications reimplemented in one codebase (T5 / Mesh TensorFlow, 223M encoder-decoder) with hyperparameters held fixed, scored on C4 pre-training loss, SuperGLUE, XSum, WebQuestions and WMT’14 En-De. Most do not help. The ones that do are small (GLU-variant activations, RMSNorm, untied embeddings), cost parameters or time (Switch/MoE, product-key memory, mixture of softmaxes), or were invented in the same codebase. The Universal Transformer, ReZero, Fixup, cross-layer sharing, the Evolved Transformer and lightweight/dynamic convolutions all come out worse than the vanilla baseline. The caveat that travels least: the positive results are also mostly same-codebase — GLU Variants (LIT-030) was run in this codebase at this exact configuration — so the paper is a strong negative test and a weak positive one. | Active |
| LIT-712 | Axiomatic Attribution for Deep Networks | Sundararajan, Taly and Yan (ICML 2017), ARXIV-1703.01365. Integrated Gradients: attribute a prediction to input features by integrating the gradient along the straight line from a baseline x′ to the input, times (x − x′). Its case is axiomatic, not empirical. It satisfies Sensitivity and Implementation Invariance, which gradients, DeconvNets and Guided Backprop (Sensitivity) and DeepLift and LRP (Implementation Invariance) each break, and its attributions sum to F(x) − F(x′). It is the unique symmetry-preserving path method. No experiment measures whether its maps are faithful. The only comparison with another method is visual (Figure 2). The baseline is a free choice that changes the answer. | Active |
| LIT-713 v2 | Sanity Checks for Saliency Maps | Adebayo, Gilmer, Muelly, Goodfellow, Hardt and Kim (NeurIPS 2018), ARXIV-1810.03292. Two randomization tests for any explanation method. Compare its maps on the trained model against the same architecture with weights re-initialized (top-down, or one layer at a time), and against a model trained on permuted labels. A method whose maps survive either cannot support model debugging or explain what the model learned from data. Gradients and GradCAM pass. Guided Backprop and Guided GradCAM are invariant to the higher layers’ weights. Integrated Gradients and gradient⊙input get no verdict. Their maps keep the input’s structure (rank correlation with absolute values and SSIM stay high), while their sign decorrelates at once. The lesson that travels is the negative one: judging a saliency map by how it looks is not a test. | Active |
| LIT-714 | Analyzing and Improving the Training Dynamics of Diffusion Models | Karras et al. (2023), ARXIV-2312.02696 — EDM2. Two contributions that do not depend on each other. Rebuilding the ADM U-Net so every layer preserves activation, weight and update magnitude takes ImageNet-512 FID from 8.00 to 2.56 at equal compute, the largest single step (6.96 → 3.75) being weight normalization on the learned layers. And post-hoc EMA: store two power-function averages at snapshots and reconstruct any EMA length after training, which shows the optimal length moving with architecture, training time, learning rate, guidance weight and — by a factor of seven under guidance — with which metric you score it. | Active |
| LIT-718 | Automated decision-making as domination | Burrell (2024), DOI-10.5210/fm.v29i4.13630. Defining algorithmic injustice as allocative unfairness is a product of computer science’s status and funding, not of the harms themselves. Redefining it as domination, following Iris Marion Young, widens the remedies from code fixes to law, participation, resistance and bans. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-719 | Discovering cognitive strategies with tiny recurrent neural networks | Ji-An et al. (2023), DOI-10.1038/s41586-025-09142-4. GRUs with only 1–4 units predict individual animals’ and humans’ choices in six reward-learning tasks better than 30+ classical cognitive models of the same dimensionality. Because they are so small, they can be read as dynamical systems, exposing strategies (state-dependent learning rates, perseveration, reward-induced indifference) that no classical model contains. | Deferred — seeded from the abstract and a skim on 2026-09-25; not read in full |
| LIT-721 | Guiding a Diffusion Model with a Bad Version of Itself | Karras, Aittala, Kynkäänniemi, Lehtinen, Aila and Laine (2024), ARXIV-2406.02507. CFG’s quality gain is not the class emphasis. It comes from the unconditional reference model being worse — a harder task on a smaller training slice — so the density ratio p₁/p₀ falls off away from the manifold and guidance points inward. Replace the unconditional model with a smaller, less-trained copy of the same conditional model and the quality gain survives without the diversity loss: ImageNet-512 FID 2.56 → 1.34 on EDM2-S, 1.25 on XXL, 1.01 at ImageNet-64, and unconditional EDM2-S 11.67 → 3.86, which CFG cannot do at all. The mechanism is tested with a negative control: degrade the two models differently and guidance stops helping entirely. | Active |
| LIT-722 | Score-Based Generative Modeling through Stochastic Differential Equations | Song, Sohl-Dickstein, Kingma, Kumar, Ermon and Poole (2020), ARXIV-2011.13456. The framework this record has been standing on without holding. Classifier guidance is Eq. 14 here, six months before ADM: the conditional reverse-time SDE adds ∇ₓ log pₜ(y|x) to the unconditional score, and §5 gives the recipe for the noise-conditioned classifier that supplies it — sample (x(0), y), noise it through the tractable forward SDE, train with a mixture of cross-entropy losses over timesteps. Also the probability flow ODE with exact likelihoods, predictor-corrector sampling, CIFAR-10 IS 9.89 / FID 2.20 / 2.99 bits/dim, and the first 1024×1024 samples from a score model. Its §I.2 imputation approximation is the “replacement” SOTA-397 exists to reject. | Active |
| LIT-724 v2 | Investigating Sanity Checks for Saliency Maps with Image and Text Classification | Kokhlikyan, Miglani, Alsallakh, Martin and Reblitz-Richardson (2021), ARXIV-2106.07475. The follow-up SOTA-430 and NOTE-365 both named as the first thing to read against the sanity checks, unread until now. Three findings, all narrowing what a failed randomization test means: the failure is attributable to the input multiplier rather than to the attribution; the verdict does not transfer from image to text, where global Integrated Gradients becomes parameter-sensitive; and SSIM and Spearman rank correlation return opposite verdicts on the same experiment. | Active |
| LIT-725 | On the (In)fidelity and Sensitivity for Explanations | Yeh, Hsieh, Suggala, Inouye and Ravikumar (2019), ARXIV-1901.09392, NeurIPS 2019. The definitions behind the two numbers LIT-724 reports and this record had started quoting: infidelity and max-sensitivity. Two properties a reader needs and neither citing document carried — max-sensitivity is minimised by a constant explanation, and infidelity is defined relative to a chosen perturbation distribution for which several existing methods are each the optimum. It also reproduces SOTA-430’s metric split on a third model family, and builds the planted-feature setting SOTA-430’s promote_when asks for while pointing it at a different question. | Active |
| LIT-726 | Talagrand Meets Talagrand: Upper and Lower Bounds on Expected Soft Maxima of Gaussian Processes with Finite Index Sets | Chu et al. (2025), ARXIV-2502.06709. Take a centered Gaussian process on a finite set T with max variance σ² and minimum separation a. The expected softmax-weighted average g(β) = E[Σ_t X_t e^{βX_t}/Z] satisfies g(β) ≤ √(2σ²·E D(ν_β‖ν₀)) for every β (Thm 1, D = KL from uniform). For β ≥ β*, g(β) ≥ c·a·√(E D(ν_β‖ν₀)), with c the Sudakov constant (1/17 via Marcus–Rosen) (Thm 2). For i.i.d. N(0,σ²) logits there is an exact identity, g(β) = βσ²(1 − E‖ν_β‖²₂) (Lemma 2). The quenched free energy is bounded above by √(2σ²·E D_{1/2}(ν_β‖ν₀)), with an i.i.d. lower bound of (cσ/2)√(E D_{1/2}) (Thms 4–5). | Active |
| LIT-727 v4 | Striving for Simplicity: The All Convolutional Net | Springenberg, Dosovitskiy, Brox and Riedmiller (2014), ARXIV-1412.6806, ICLR 2015 workshop. Guided backpropagation’s defining paper, and so the referent for SOTA-430’s headline failure case — the method whose maps stay “visually and quantitatively similar” after the weights above the lowest layers are destroyed. The masking rule is stated here in one sentence: zero the gradient wherever either the top gradient or the bottom activation is negative. The paper is about an architecture; guided backprop arrives in §4.2 as a tool for analysing it, and the property later found to be its defect is presented here as its advantage. | Active |
| LIT-728 v2 | Visualizing and Understanding Convolutional Networks | Zeiler and Fergus (2013), ARXIV-1311.2901. Two attribution families start here and the record depends on both. The deconvnet is the approach guided backprop (LIT-727) is explicitly “a new variant of”, and so is the antecedent of SOTA-430’s headline failure case. Occlusion sensitivity — slide a grey square, watch the class probability — is the perturbation-based family LIT-725 names as its own origin, and LIT-725’s infidelity is its quantitative descendant. Also an early depth-against-parameters ablation: removing the fully connected layers costs little, removing them and two middle conv layers is “dramatically worse”. | Active |
| LIT-729 v2 | When Explanations Lie: Why Many Modified BP Attributions Fail | Sixt, Granz and Landgraf (2019), ARXIV-1912.09818, ICML 2020. The account SOTA-430’s failing row was missing — for one family, and not the family this record said it was for. Theorem 1: a backpropagation rule that keeps only non-negative relevance gives a product of non-negative matrices, which converges to a rank-1 matrix, and C v = c γᵀ v = λc — once converged the later layers can only change the map’s sign. That covers DTD, LRP-α1β0, Excitation BP and PatternAttribution. It explicitly does not cover Guided Backprop or Deconv, whose ReLU on the gradient makes the backward pass non-linear and which it defers to Nie et al. (2018), now LIT-730. | Active |
| LIT-730 | A Theoretical Explanation for Perplexing Behaviors of Backpropagation-based Visualizations | Nie, Zhang and Patel (2018), ARXIV-1805.07039, ICML 2018. The third mechanism, and the one SOTA-430’s headline failure row needed. Theorem 1: in a random three-layer CNN, guided backpropagation recovers the input, s_k^GBP(x) ≈ x, regardless of the class label. Theorem 2: saliency map and DeconvNet in the same network are N(0, I) — noise. So GBP and DeconvNet are doing (partial) image recovery, “unrelated to the decision-making of neural networks”, and the two causes are the backward ReLU and a CNN’s local connections. Four arms test it, including removing local connections and an adversarial attack that changes the class while barely changing the input. | Active |
| LIT-731 | SmoothGrad: removing noise by adding noise | Smilkov, Thorat, Kim, Viégas and Wattenberg (2017), ARXIV-1706.03825. The smoothing eight held documents name — averaging the gradient over n noisy copies of the input, with 10–20% noise and n ≈ 50, past which “there was little apparent change”. Its own evidence is entirely qualitative and it says so: “since quantitative evaluation of a map remains an unsolved problem, we again focus on qualitative evaluation.” And its §3.1 visualization lessons contain informal statements of two things this record spent two days building accounts for: the signed-versus-absolute ambiguity, and the input multiplier putting the input’s own edges into the map. | Active |
| LIT-732 | Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localization | Selvaraju, Cogswell, Das, Vedantam, Parikh and Batra (2016; ICCV 2017, IJCV 2019), ARXIV-1610.02391. The referent behind both halves of SOTA-430’s verdict table — GradCAM, which passes the randomization test, and Guided GradCAM, which fails it, are both defined here, and the second is defined as the pointwise product of the first with guided backpropagation. Grad-CAM reads one layer: gradients of the class logit into the last conv feature maps, global-average-pooled into per-channel weights. Also carried: a 2016 human study in which guided backprop, the best-looking method, comes last of four — 44.44% against Deconvolution’s 53.33% — a number its authors did not gloss until v4 in December 2019, fourteen months after LIT-713 made the same point another way. | Active |
| LIT-734 | Fine-Tuning Language Models from Human Preferences | Ziegler et al. (2019), ARXIV-1909.08593. The first time the human-preference loop is run on a pretrained language model: a 774M GPT-2 fine-tuned with RL against a reward model fit to human choices, with a KL penalty holding it near the pretrained model. On stylistic continuation, 5,000 comparisons give a policy humans prefer 86% of the time over zero-shot. On TL;DR and CNN/DM summarization, 60,000 comparisons produce “smart copiers” that labelers like but that may be exploiting the labelers’ heuristics, the problem LIT-433 set out to fix. | Active |
| LIT-740 | Role-Play with Large Language Models | Shanahan, McDonell and Reynolds (2023), ARXIV-2305.16367 — a conceptual paper, with no experiments. A base LLM behind a dialogue prompt is best described as role-playing the character the context implies. More precisely, it keeps a superposition of characters consistent with the conversation so far and commits to none of them, and every reply narrows the set. The framing lets folk-psychological words be used without ascribing beliefs or goals to the model. It also gives a behavioural test for kinds of falsehood: invented answers vary under regeneration, “good faith” errors do not, and “deliberate” deception shows when the same question is asked from different contexts. | Active |
| LIT-746 | Position: Curvature Matrices Should Be Democratized via Linear Operators | Dangel et al. (2025), ARXIV-2501.19183 — a position paper with a library behind it: expose the Hessian, GGN, Fisher flavours and (E)KFAC as linear operators, so applications ask only for matrix-vector products and the batching, scaling and determinism pitfalls live in one tested place. Its measurements are costs in units of a gradient, on ResNet-50 and nanoGPT: computing KFAC’s factors takes 1.5–2.5 gradients and applying them 0.1–0.2, while a Hessian-vector product takes 4.5–5.5 gradients and 3–4.5× the memory in PyTorch. | Active |
| LIT-752 | Position: AI/ML Influencers Have a Place in the Academic Process | Weissburg, Arora, Wang, Pan and Wang (2024), ICML 2024 position paper. Papers tweeted by the two ML paper-sharing accounts @_akhaliq and @arankomatsuzaki have median citations of 24 and 31 against 14 and 12 for controls matched on venue, year, topic embedding and author prominence, while OpenReview scores of tweeted papers and their controls are indistinguishable. A negative-outcome-control estimate keeps a positive causal effect on being highly cited at every threshold tried, largest for less-cited papers. | Active |
| LIT-753 | Rigorous dynamical mean field theory for stochastic gradient descent methods | Gerbelot et al. (2022), ARXIV-2210.06591 — a proof that the discrete-time dynamical mean-field equations of statistical physics are exact for a family of first-order methods (multi-pass SGD, heavy-ball and Nesterov momentum, Langevin noise, learning-rate schedules) learning a shallow estimator on Gaussian data, in the limit where samples and dimension grow together and the mini-batch is a fixed fraction of the data. On a teacher-student perceptron the solved equations match simulation at d = 1000 across learning rates and batch fractions. | Active |
| LIT-759 | From monoliths to modules: Decomposing transducers for efficient world modelling | Boyd et al. (2025), ARXIV-2512.02193 — a formal paper with no experiments. A world model is a transducer (a stochastic input-output machine with latent memory, generalizing POMDPs), and composing transducers is a Kronecker product of their operators. The paper inverts this. Two information measures, Intransducibility (with latents) and Acausality (observables only), are zero exactly when one block of variables can be a causal module driven by the rest. Peeling such modules off factors a monolithic model into a DAG of sub-transducers that can be learned and inferred separately. Minimal predictive models (ε-transducers) are closed under composition. | Active |
| LIT-761 | Synaptic Field Theory for Neural Networks | Lee, Lee and Yi (2025), ARXIV-2503.08827 — a hep-th proposal with no experiments. Gradient descent with momentum, Ẅ + γẆ + ∂C/∂W = 0, follows from an action weighted by e^{γt}, the volume factor of de Sitter space with Hubble rate H = γ/d. Treat the weight indices as spatial coordinates and take the continuum limit, and training becomes a field theory with the data as external sources. Only toy networks give a local one: periodic, nearest-neighbour wiring with linear or quadratic activations. | Active |
| LIT-764 | A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima | Xie et al. (2020), ARXIV-2002.03495 (ICLR 2021). Near a minimum, SGD’s gradient-noise covariance is approximately the Hessian divided by the batch size; feed that into a Kramers escape-time calculation and the mean time to leave a valley is exponential in B/η times the inverse Hessian eigenvalue along the escape direction. So SGD favours flat minima exponentially, where gradient descent with injected white noise favours them only polynomially. Validated as escape rates on test functions and networks of ten hidden units; generalization is not measured. | Active |
| LIT-767 | Understanding Deep Learning via Notions of Rank | Razin (2024), ARXIV-2408.02111 — PhD thesis. Gradient descent’s implicit regularization is better read as rank minimization than norm minimization: on a 2×2 matrix completion problem, a depth L ≥ 2 linear network trained by gradient flow drives every norm and quasi-norm to infinity, with probability at least 0.5 over arbitrarily small initialization, while rank falls to its minimum. The same bias toward low (tensor, hierarchical-tensor) rank is proved for polynomial-activation networks equivalent to tensor factorizations; it predicts convolutional networks’ locality, which a regularizer derived from it partly undoes on long-range tasks. | Active |
| LIT-768 | Kronecker-factored Approximate Curvature (KFAC) From Scratch | Dangel et al. (2025), ARXIV-2507.05127 — a tutorial, math and PyTorch side by side, for the original KFAC of Martens and Grosse: fully-connected layers, no weight sharing. One scaffold covers the GGN, the Monte-Carlo Fisher and the empirical Fisher, and the only thing that changes between them is which vectors are backpropagated. Its practical content is the tests: KFAC is exact on a single data point, and on a deep linear network with square loss for the GGN and MC Fisher but not for the empirical Fisher, which is how to catch the scaling and factor-order bugs it says are easy to ship. | Active |
| LIT-769 | Structural Decoupling: A Scaffold-Flow Theory of Generalization and Alignment | Li (2025), ARXIV-2506.20699. A theory paper with no experiments. It defines a problem’s width, the minimum number of locally feasible contexts needed to cover it, and states that width is incomparable with VC dimension and that a learner with fewer contexts than the width has an error floor no amount of within-context training removes. From the additive split of complexity into a routing term and a within-context term it argues that the parts of a model that keep context structure should not be trained by the gradients that fit predictions inside a context, and it reads hallucination, reward hacking and deceptive alignment as failures of that structure. The current version (v2, 2026) replaced a v1 about a “Context-Content Uncertainty Principle” for brains and machines. | Active |
| LIT-773 | Improving Video Generation with Human Feedback | Liu, Liu, Liang et al., CUHK, Tsinghua and Kuaishou (2025), ARXIV-2501.13918. Three parts. A 182k-pair human preference set over videos from 12 text-to-video systems, rated separately for visual quality, motion and text alignment. VideoReward, a Qwen2-VL-2B reward model trained with a Bradley–Terry loss that models ties. Flow-DPO, Diffusion-DPO rewritten for rectified flow. Carried over exactly, the loss weights the KL by β(1 − t)², which goes to zero at the noise end; that variant raises the reward’s quality scores and drops its text alignment below the base model’s. A constant β fixes it. On an internal 1B video model with LoRA, constant-β Flow-DPO beats SFT and Flow-RWR, and humans prefer it to the base 44% to 27% with 29% ties. Training labels and automatic win rates come from the same reward model, and VBench barely moves. | Active |
| LIT-782 | Pick-a-Pic: An Open Dataset of User Preferences for Text-to-Image Generation | Kirstain, Polyak, Singer, Matiana, Penna and Levy, Tel Aviv University and Stability AI (2023), ARXIV-2305.01569. A web app that generates two images per prompt and logs which one the user keeps, or a tie. The paper’s version holds 583,747 training pairs over 37,523 prompts from 4,375 users. PickScore is CLIP-H fine-tuned on those pairs with a reward-model loss. It predicts held-out user choices at 70.5% under a tie-aware accuracy where chance is 56.8%, against 68.0% for the authors’ expert annotators. Its ranking of 45 model configurations correlates 0.790 with real users’ Elo. On MS-COCO captions FID ranks nine configurations at −0.900 against human experts, because people prefer higher guidance. Every validation is in-distribution or uses colleague annotators. Diffusion-DPO trains on this data and Flow-GRPO optimizes PickScore as a reward. | Active |
| LIT-786 | Improved Techniques for Training Consistency Models | Song and Dhariwal, OpenAI (2023), ARXIV-2310.14189. Consistency training from random weights, without a teacher or LPIPS. The teacher network takes no EMA, because with one the limiting objective carries no information about the data. A Pseudo-Huber loss replaces LPIPS, whose ImageNet features can leak into FID. The step count doubles on a schedule, and noise levels are drawn log-normally. One-step FID is 2.51 on CIFAR-10 and 3.25 on ImageNet-64, two-step 2.24 and 2.77, beating consistency distillation. Ablations are CIFAR-10 curves, one run each. | Active |
| LIT-788 | ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generation | Xu, Liu, Wu, Tong, Li, Ding, Tang and Dong, Tsinghua, Zhipu AI and BUPT (2023), ARXIV-2304.05977. A reward model for text-to-image preference: a BLIP backbone trained on 136,892 ranked pairs over 8,878 DiffusionDB prompts. It picks the rater’s choice 65.14% of the time, against 54.82% for CLIP similarity, about as often as one annotator agrees with another (65.3%). ReFL, added in v3, backpropagates the reward through one denoising step taken late in the chain. On Stable Diffusion v1.4 it is ranked over the base model 58.79% of the time, ahead of filtering, reward weighting and RAFT. Best-of-64 under the same reward reaches 73.33%. Its premise, that a diffusion model has no likelihood for RLHF to use, is true of the whole chain and not of a step. | Active |
| LIT-793 | Flow map matching with stochastic interpolants: A mathematical framework for consistency models | Boffi, Albergo and Vanden-Eijnden, Courant Institute (2024), ARXIV-2406.07507. Learn the two-time flow map X_{s,t} of a probability-flow ODE. Distillation from a velocity field can use a Lagrangian loss (new) or an Eulerian one, which is the continuous-time limit of consistency distillation; both bound a Wasserstein distance. Consistency training’s update is shown not to descend any loss. On CIFAR-10 the Lagrangian loss distils far better than the Eulerian: FID 7.13 against 48.32 at two steps, teacher 5.53. Direct training on ImageNet-32 gives 16.90 FID at four steps. Small models; no one-step image table. | Active |
| LIT-795 | Scaling Laws for Reward Model Overoptimization | Gao, Schulman and Hilton, OpenAI (2022), ARXIV-2210.10760. InstructGPT’s 6B reward model stands in for people and labels comparisons for proxy reward models of 3M to 3B parameters. Optimizing a 1.2B policy against a proxy, the gold reward rises and then falls as a function of d = √KL from the initial policy: d(α − βd) for best-of-n and d(α − β log d) for PPO. The coefficients move smoothly with the proxy’s size, so the peak is predictable. Larger proxies and more comparison data push the peak out, and a 6B policy peaks at the same KL as a 1.2B one. A KL penalty in the reward did not improve gold reward at a given KL; its effect was like early stopping, a result the authors flag as possibly hyperparameter-sensitive. RL spends far more KL than best-of-n, so KL does not compare optimization across methods. Synthetic labels, one environment. | Active |
| LIT-797 | Aligning Text-to-Image Models using Human Feedback | Lee, Liu, Ryu, Watkins, Du, Boutilier, Abbeel, Ghavamzadeh and Gu, Google Research and UC Berkeley (2023), ARXIV-2302.12192. RLHF’s recipe carried to a text-to-image model, without the RL. Two labellers mark 27,528 Stable Diffusion v1.5 images good or bad on templated count, colour and background prompts. A small head on frozen CLIP embeddings learns that label. One offline round of reward-weighted likelihood plus a pre-training loss then fine-tunes the model. Alignment wins 50% of 120 prompts by a two-thirds vote against 3% for the base. Fidelity loses: FID 13.97 to 16.76, and 3% against 20% when compared with best-of-16 sampling from the base under the same reward. One run, no diversity measure. | Active |
| LIT-799 | Olmo Hybrid: From Theory to Practice and Back | Merrill et al., Ai2 and Lambda (2026), ARXIV-2604.03444. Olmo 3 7B with its sliding-window layers replaced by Gated DeltaNet (negative eigenvalues), at 3:1 with full attention, trained to 6T tokens. It reaches Olmo 3’s MMLU in 49% fewer tokens and beats it in every domain after mid-training, though data mix, schedule and mid-training batch also changed. Scaling fits from 60M to 1B give the hybrid a lower data coefficient (83.7 against 94.9, non-overlapping CIs); the projected 1.3–1.9× token savings have CIs that include 1 at every size but 3B. A theorem shows hybrids express a task neither parent can. Removing negative eigenvalues barely changes the scaling, which weakens the expressivity explanation. | Active |
| LIT-800 | Why Gated DeltaNet Survives 4-Bit Quantization: NVFP4 W4A4 for the Recurrent Half of a Hybrid 27B LLM | Kozyrev and Maiboroda, Minima (2026), ARXIV-2609.04098. Public 4-bit builds of Qwen3.8-27B kept its 48 Gated DeltaNet layers in 8 or 16 bits for fear that errors in a recurrence compound. Quantizing all 496 linear layers to NVFP4 W4A4 by calibration alone stays within seed noise of BF16 on five task suites (average −0.52) at 17.5 GiB. The perplexity cost is the largest of the recipes compared, +0.72 at 4K, but it shrinks along a 32K window. In replayed layers the gate projections everyone protected are the least sensitive. State error holds at a plateau of about 12.6% over 32K tokens, and a state impulse is erased within hundreds to a few thousand steps. One model, one format. | Active |
| LIT-801 | Cracks in the Foundation: Seemingly Minor Architectural Choices Impact Long Context Extension | Bertsch, Soldaini, Gormley, Neubig, Hajishirzi, Lo and Groeneveld, Ai2 and CMU (2026), ARXIV-2608.10296. Twenty-six 7–8B models, 140B tokens each on identical data, extended identically to 64K. QK norm, GQA, sliding-window layers and a 4K pretraining length each cost long-context ability, and together they compound: HELMET at 32K runs from 56.4 down to 29.9. Counting how many of the four a model has predicts that score with in-sample R² 0.67 (0.61 leave-one-out). Pretraining loss predicts it at R² 0.29. QK norm is the largest single cost, about 4 to 6 points. GQA plus sliding windows costs about 9, against 1.1 for windows alone. One seed per configuration, and initialization alone moves scores by 7.7% of the range on average. | Active |