Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Learning theory — why learning generalizes and how it proceeds — compression and description length, Kolmogorov complexity and sufficient statistics, rate–distortion and PAC-style bounds, model selection; and training dynamics: solvable models of learning trajectories, stages, plateaus and phase transitions in training, and the implicit bias of optimisation. A work whose subject is how to train a model better is ML practice and goes to the anthology (ADR-036).
142 of 2004 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-224 v2 | Shannon Information and Kolmogorov Complexity | Grünwald & Vitányi (2004), arXiv:cs/0410002. A survey pairing Shannon’s and Kolmogorov’s notions: entropy and complexity, probabilistic and algorithmic mutual information, data processing, sufficient statistics, and rate–distortion and the structure function. For a computable source f, expected complexity is within K(f) + O(1) of entropy, and expected algorithmic mutual information is within K(f) and 2K(f) of Shannon’s. An algorithmic sufficient statistic is probabilistically nearly sufficient in expectation, one way only. The expected structure function tends to the distortion-rate function per symbol. | Active — read 2026-10-14 (NOTE-776) in full; worth reading as the one place that sets each Shannon quantity beside its Kolmogorov counterpart with the gap stated. Expected complexity exceeds entropy by at most K(f) + O(1), expected algorithmic mutual information is within K(f) and 2K(f) of Shannon’s, and the expected structure function brackets the distortion-rate function with a K(f, d, m, R) shift in rate. The two new results are Theorems 5.13 and 6.27. Read it knowing that every bridge costs the complexity of the source and says nothing about one string, that sufficiency links only one way (algorithmic to probabilistic, in expectation), that a step in Theorem 6.27 is asserted, and that the authors warn the draft contains errors. |
| LIT-225 v2 | Minimum Description Length Induction, Bayesianism, and Kolmogorov Complexity | Vitányi & Li (1999), arXiv:cs/9901014. Ideal MDL selects the H minimising K(H) + K(D|H), Bayes’s rule with the universal distribution in place of prior and likelihood. By the Fundamental Inequality, when the data are random for H and H is random for the prior, Bayes code lengths and Kolmogorov code lengths agree to within K(Pr(·|H)) + K(P). So ideal MDL and MAP agree to that factor among such hypotheses, and can diverge outside them. For finite-set models they coincide with Kolmogorov’s minimal sufficient statistic. For random sequences, shortest-program prediction converges to the true conditional. | Active — read 2026-10-14 (NOTE-801) in full; worth reading as the stated condition under which minimum description length and Bayes coincide: by the Fundamental Inequality (Theorem 3), Bayes code lengths and shortest-program lengths agree to within K(Pr(·|H)) + K(P) when the data are Martin-Löf random for the hypothesis and the hypothesis is random for the prior, so ideal MDL and MAP agree to a factor 2^{−α} among such hypotheses (Theorem 4), and for finite-set models both reduce to Kolmogorov’s minimal sufficient statistic. Read it knowing that α can be as large as the hypothesis’s own complexity, that admissibility cannot be checked, that “with high probability” (Corollary 1) and the overfitting claim are unproved, and that the arXiv text is the submitted version. |
| LIT-226 v2 | The Conditional Entropy Bottleneck | Fischer (2020), ARXIV-2002.05379. The Conditional Entropy Bottleneck minimizes I(X;Z|Y) − γI(Y;Z), which is the Information Bottleneck with γ = β − 1, aimed at the “Minimum Necessary Information” point I(X;Y) = I(X;Z) = I(Y;Z). On Fashion MNIST it improves accuracy, adversarial robustness, OoD detection and calibration over VIB and deterministic baselines, and it does not memorize random labels. On CIFAR-10 it improves accuracy and robustness. That excess retained information causes these failures is stated as a hypothesis and supported only empirically. | Active — read 2026-10-14 (NOTE-754) in full; worth reading as the paper that names the Minimum Necessary Information point I(X;Y) = I(X;Z) = I(Y;Z), shows that its objective is the Information Bottleneck with γ = β − 1, and gives variational bounds (the residual information ReX) that make the distance to that point observable. Its CatGen decoder is InfoNCE with the backward encoder as critic, capped by log K. Read it knowing that the causal claim, that retained information causes adversarial vulnerability, miscalibration, poor OoD detection and memorisation, is a hypothesis supported only by Fashion MNIST and one selected CIFAR-10 model per setting, and that the arXiv v1 prints the CatGen inequality (Eq. 30) with the wrong sign. |
| LIT-229 v2 | Meaningful Information | Vitányi (2001; IEEE Trans. Inf. Theory 2006), arXiv:cs/0111053. With total recursive functions as models, the algorithmic minimal sufficient statistic, sophistication, splits a string’s information into structure and noise. The split collapses to zero for every string if partial recursive models are allowed. Random strings have sophistication about 0, and some strings of every length are almost all structure. The finite-set, probabilistic and total-function versions agree to within logarithmic terms. Sophistication is not computable, and a sufficient-statistic oracle would compute K. | Active — read 2026-10-14 (NOTE-790) in full; worth reading as the short proof that splitting an individual string’s information into structure and noise works only in a model class without a universal member: with partial recursive models every string’s minimal sufficient statistic is trivial (Lemma 6.2), while with total recursive functions sophistication runs from about 0 for random strings to above n − log n − 2 log log n for some string of every length (Theorem 6.5), and agrees with the finite-set and probabilistic versions to within O(log n) (§VII). Read it knowing that everything holds only to within additive constants or logarithmic terms, that sophistication is uncomputable and not usefully approximable, that two steps rest on cited work (Gács–Tromp–Vitányi; Vereshchagin–Vitányi), and that the structure-function characterisation of §IX is stated, not proved. |
| LIT-233 v3 | Data-Dependent Generalization Bounds via Variable-Size Compressibility | Sefidgaran & Zaidi (2023), ARXIV-2303.05369. Letting the coding rate of a block of training runs vary with the realised sample and output gives generalisation bounds that depend on the sample at hand, not only on its distribution. With the rate set to log dP_{W|S}/dP(W) for a fixed prior, the bound is the disintegrated PAC-Bayes bound, and general change-of-measure versions recover most PAC-Bayes and mutual-information bounds. A new trajectory bound is data-dependent except for a coupling term that carries the trajectory’s dependence on the sample, and its experiment drops that term. | Active — read 2026-10-14 (NOTE-806) in full; worth reading as the paper that makes rate-distortion generalisation bounds data-dependent by letting the coding rate vary with the realised sample and output (Theorem 1), and that shows the disintegrated PAC-Bayes bound to be the case where the rate is log dP_{W|S}/dP(W) against a fixed prior, with Xu–Raginsky, Rivasplata et al. and LIT-236’s bounds as other instances (Theorems 3–5). It adds a lossy PAC-Bayes bound finite for deterministic continuous-parameter algorithms (Proposition 1) and a trajectory bound (Theorem 7). Read it knowing that the coding construction is proved only for finite alphabets and the general proofs are change of measure, that the rate is a Shannon code length against a prior and not Kolmogorov complexity, that Theorem 7 is data-dependent except for a coupling term M that carries the trajectory’s dependence on the sample, and that the CIFAR-10 experiment drops M and shows only that two curves fall together, with a vacuous value for FCN4. |
| LIT-234 v3 | The Effects of Regularization and Data Augmentation are Class Dependent | Balestriero et al. (2022), ARXIV-2204.03632. On ImageNet with ResNet-50, strengthening random crop or changing weight decay raises mean test accuracy while some classes fall sharply (e.g. “barn spider” about 68%→46% as the crop lower bound goes from 100% to 8%). Different pretrained architectures agree on which classes crops make unrecognisable. Pretraining crop strength also moves per-class linear-probe accuracy on iNaturalist in opposite directions. The evidence is selected classes and an uncorrected test. | Active — read 2026-10-14 (NOTE-845) in full; worth reading as a clear demonstration, on standard ImageNet ResNet-50 training, that the regularization strength best on average is not best for every class: stronger random crop and changed weight decay raise mean accuracy while some named classes lose 20 points or more, and the shift survives a frozen-backbone transfer to iNaturalist. Read it knowing that the evidence is per-class curves for hand-picked classes over 10–20 runs, that its significance test is uncorrected for 1,000 classes (4.4% rejections at the 5% level), that the weight-decay claim rests on one figure with no numbers, that Theorem 1 does not reach the class-level claim, and that the abstract’s iNaturalist example (class #8889, 70% to 30%) is a drop from moderate to strong cropping, not from none. |
| LIT-235 v2 | When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysis | Sun et al. (2023), ARXIV-2308.05017. A spectral contrastive loss over an augmentation graph that adds labelled same-class pairs (NSCL) is equivalent to factorizing its adjacency. At that optimum, a novel labelling’s least-squares residual is exactly its component on the discarded eigenvectors that their labelled rows cannot absorb, which can be zero. In a toy, known classes sharing the target attribute remove the residual, and ones sharing a rival attribute raise it. | Active — read 2026-10-14 (NOTE-833) in full; worth reading as an exact account, within HaoChen et al.’s spectral contrastive framework, of how labelled data from other classes changes which partition of unlabelled data the learned features express: the novel labelling’s residual is its part outside the top eigenspace that the labelled rows of the discarded eigenvectors cannot absorb, and a toy shows known classes can help or hurt. Read it knowing that Theorem 5.5 is printed as ≤ though its proof gives equality, that everything is at the exact optimum of a population graph, that the general bound (Theorem 5.6) is approximate, pools all known classes into one averaged block and has notational slips, and that the help-or-harm examples have one sample per kind. |
| LIT-236 v3 | Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithms | Sefidgaran et al. (COLT 2022), ARXIV-2203.02474. The expected generalisation error of any algorithm is at most √(2σ²·R/n) + ε, where R is the least I(S;Ŵ) of a surrogate output within ε in expected generalisation error. With ε = 0 this is Xu–Raginsky’s bound, which for finite alphabets is shown to be a lossless compression rate. With ε > 0 it stays finite where I(S;W) is infinite, and for a Lipschitz loss it becomes a rate-distortion function, or dimension, of the output’s law. A tail version takes a supremum over a KL ball. All bounds depend on the unknown data distribution. | Active — read 2026-10-14 (NOTE-780) in full; worth reading as the source of the lossy generalisation bound |E gen| ≤ √(2σ²·R_E(ε)/n) + ε, where R_E(ε) is the least I(S;Ŵ) over surrogate outputs within ε in expected generalisation error (Theorem 4), of which Xu–Raginsky’s bound is the lossless case, and of its Lipschitz, rate-distortion-dimension and tail forms (Corollaries 6–7, Theorem 10). Read it knowing that the block-coding meaning is proved only for finite alphabets, that in general spaces Theorem 4 is Xu–Raginsky applied to a surrogate, that every bound depends on the unknown data distribution, that no bound is evaluated for a learned model, and that the one worked example mixes squared and unsquared distortion, so its d/n rate is unsupported. |
| LIT-238 v2 | Minimum Description Length Revisited | Grünwald & Roos (2019), ARXIV-1908.08484. A review of MDL since 2007, written without coding theory. Each model is replaced by a single universal distribution with small worst-case regret against its best fit. NML is the minimax choice, with complexity (k/2) log(n/2π) + log ∫ v√|I| + o(1). Luckiness functions generalise priors and penalties. The switch distribution nearly resolves the AIC–BIC dilemma, RIPr tests survive optional continuation, general penalised estimators inherit MDL convergence, and MDL and Bayes can be inconsistent under misspecification. Results are cited, not proved. | Active — read 2026-10-14 (NOTE-760) in full; worth reading as the current map of MDL. A model is replaced by one universal distribution with small worst-case regret, NML is the minimax one with complexity (k/2) log(n/2π) + log ∫ v√|I| + o(1), estimation is penalised ML with a luckiness function, and the decade’s results are collected: the switch distribution, RIPr tests, convergence of general MDL estimators, and inconsistency under misspecification. Read it knowing that it is a review whose results are cited rather than proved, that it sets aside Kolmogorov-complexity MDL, and that its “unification” of AIC with BIC and of cross-validation with Bayes is, in its own words, partial. |
| LIT-239 v2 | Kolmogorov’s Structure Functions and Model Selection | Vereshchagin & Vitányi (2002), arXiv:cs/0204037. For a string x of length n and every complexity bound α, Kolmogorov’s structure function h_x(α) + α, the minimal two-part code length λ_x(α) and the best achievable randomness deficiency β_x(α) + K(x) are one function to within O(log n). A set minimising code length at α is a best-fitting set there, though not conversely. Every non-increasing shape occurs for some x. h_x and λ_x can be approximated from above but not computed; β_x cannot be approximated at all. Finite-set models, extended to computable distributions at O(log n) cost. | Active — read 2026-10-14 (NOTE-787) in full; worth reading as the theorem that, for a single string and every bound on model complexity, the set with the shortest two-part code is within O(log n) of the best-fitting set in randomness deficiency (Theorem IV.8), with the complete catalogue of shapes the structure functions can take and the proof that fit cannot be approximated while code length can. Read it knowing that every result holds only to O(log n), that the models are complexity-bounded finite sets or computable distributions and not statistical families, which the authors say the results do not transfer to directly, and that the construction proving Theorem D.2 was not checked line by line here. |
| LIT-240 v3 | The Role of the Information Bottleneck in Representation Learning | Vera et al. (2018), DOI-10.1109/ISIT.2018.8437679. The generalization gap of the cross-entropy loss for an encoder is bounded by a term in the information the representation keeps about the input, so minimizing empirical risk plus that bound is equivalent to the Information Bottleneck on the empirical distribution — offered as an explanation of why noise injection such as dropout helps. | Unavailable — searched 2026-10-14 for a lawful full text of the ISIT 2018 paper and none was reached, so it is not read and no NOTE is filed. OpenAlex: closed, with one other location, the CentraleSupélec HAL record hal-01756003, which the HAL API gives as submitType “notice” (metadata only, no file); Unpaywall: not open access; IEEE Xplore returned a bot check (202), failing on access, not availability, though the paper is subscription-only there; arXiv: a search of Piantanida and Vera lists no paper with this title, only the related, distinct arXiv:1802.05355; CONICET’s repository, which lists an event-paper record for it, returned 502 and 503, so whether it holds a file is unknown; a web search for the title found no author-posted copy. It stays Unavailable until a copy is supplied. |
| LIT-242 v3 | On the Stepwise Nature of Self-Supervised Learning | Simon et al. (2023), ARXIV-2303.15438. For a simplified Barlow Twins loss on a linear or fixed-kernel model, gradient flow from small initialization learns the top eigenvectors of a symmetrized cross-correlation operator one at a time and reaches the minimum-norm zero-loss solution, kernel PCA on K_Γ. Exact from aligned initialization; informal for generic small initialization. ResNets trained with Barlow Twins, SimCLR and VICReg show the same stepwise rank growth, as an experimental finding. | Active — read 2026-10-14 (NOTE-741) in full; worth reading for the dynamics that the optimum-based spectral accounts lack. For a simplified Barlow Twins loss (λ = 1, no batch norm, symmetrized cross-correlation) on a linear or fixed-kernel model, gradient flow from small initialization learns the top eigenvectors of Γ, or of K_Γ, one at a time and reaches the minimum-norm zero-loss solution: kernel PCA. Read it knowing three things. The step for generic initialization (Result 4.6) is an informal derivation. Its operator is built from the model’s kernel and the sample pairs, not from K⁺. And its evidence for SimCLR, VICReg and ResNets is experimental: the initial NTK predicts them poorly, and the final empirical NTK captures about half the embedding subspace. |
| LIT-245 v4 | Minimum Description Length and Generalization Guarantees for Representation Learning | Sefidgaran et al. (2023), ARXIV-2402.03254. The expected generalization error of an encoder–decoder classifier is bounded by 2√((2·MDL + K + 2)/n) + ε for any decoder. Here MDL is the KL divergence between the encoder’s joint law of train-and-ghost latents and a prior symmetric under label-preserving permutations, not I(X;U). The quantity reflects the encoder’s structure and is finite for deterministic encoders under lossy compression. It reduces to sample-wise mutual information for a product prior, and the predicted-label version contains the f-CMI and VC bounds. | Active — read 2026-10-14 (NOTE-742) in full; worth reading as the theorem that bounds the generalisation error of an encoder–decoder learner by the encoder alone: 2√((2·MDL + K + 2)/n) + ε, where MDL is the KL divergence of the joint train-and-ghost latents to a prior symmetric under label-preserving permutations (Theorem 4). The bound holds for any decoder and stays finite for deterministic continuous encoders under lossy compression. With a product prior it reduces to sample-wise mutual information, which is why it can be tighter than IB-style bounds. Read it knowing that the bound is never evaluated numerically, that the prior may depend on the unseen ghost data and is only approximated in practice, that a K + 2 term sets a floor, that the proofs other than Theorem 4’s were not checked here, and that the experiment is one CIFAR-10 CNN at 62–65% accuracy. |
| LIT-249 v4 | Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss | HaoChen et al. (2021), ARXIV-2106.04156. Minimizing the population spectral contrastive loss L(f) = −2E[f(x)ᵀf(x⁺)] + E[(f(x)ᵀf(x⁻))²] is, up to a constant, a rank-k factorization of the normalized adjacency matrix of the augmentation graph, so its minimizers are the top-k eigenvectors (eigenfunctions, in the infinite case) of that graph up to a row scaling and an invertible linear map — neither of which changes linear-probe accuracy. | Active — read 2026-10-14 (NOTE-740) in full; worth reading as the founding statement that a contrastive loss is a low-rank factorization of the normalized augmentation graph. Lemma 3.2 makes the spectral contrastive loss equal to ‖D^(−1/2)AD^(−1/2) − FFᵀ‖²_F up to a constant, so its population minimizers are the top-k eigenvectors up to a row scaling and a right factor, which Lemma 3.1 shows a linear probe cannot see. Theorem 3.8 then bounds linear-probe error by Õ(α/ρ²_⌊k/2⌋). Read it knowing that everything is proved on a finite augmentation space at the population optimum of the spectral loss alone. The infinite case (App. F) assumes a uniform bound on the density ratio, not only Hilbert–Schmidt, and asserts rather than proves the extension of the main theorems. The symmetry of the pair distribution is used throughout and never stated. |
| LIT-251 v2 | Contrastive learning, multi-view redundancy, and linear models | Tosh, Krishnamurthy & Hsu (2021), ARXIV-2008.10150. The logistic contrastive problem of telling a same-datum pair of views from an independent pair is solved over all functions by the pointwise mutual information, whose exponential g*(x,z) = p(x,z)/(p(x)p(z)) is the change of measure from p(z) to p(z|x). When the two views are redundant for a label, the infer-then-predict function E[E[Y|Z]|X=x] = ∫ E[Y|Z=z] g*(x,z) p(z) dz is within εX + 2√(εXεZ) + εZ of the Bayes predictor, so linear functions of the landmark embedding (g*(x,Z_1), …, g*(x,Z_m)) are near-optimal, with error O(1/m). | Active — read 2026-10-14 (NOTE-841) in full; worth reading as the redundancy account of contrastive learning: when two views are redundant for a label, the infer-then-predict function µ(x) = E[E[Y|Z]|X=x] is within εX + 2√(εXεZ) + εZ of the Bayes predictor (Lemma 1), and µ is linear in the density ratio g*(x,·) = p(x,z)/(p(x)p(z)), “the change-of-measure from the marginal distribution of Z to the conditional distribution of Z given X = x”. Random landmark evaluations of g* then give near-Bayes linear predictors at rate 1/m. Read it knowing that it is population-level, that the logistic problem has a single negative, that its error propagation assumes a uniform bound on the ratio, and that its low-dimensional direct embeddings are shown to exist, not to be reached. It never assumes the views are exchangeable. |
| LIT-253 v2 | When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspective | Nielsen et al. (2025), ARXIV-2506.03784. For softmax models p(y|x) ∝ exp(f(x)ᵀg(y)) under a diversity condition, the model distribution determines the embeddings and unembeddings up to an invertible linear map (f = Af′, g₀ = A⁻ᵀg′₀). The determination is not stable under KL divergence, though: two models can be ε-close in KL, even both near maximum likelihood, while their embeddings are far from linearly equivalent. Under a log-likelihood variance distance it is stable: d_LLV ≤ ε implies a representational dissimilarity of at most 2Mε. | Active — read 2026-10-14 (NOTE-738) in full; worth reading as the exact statement that softmax models are identified only up to the paired map f = Af′, g₀ = A⁻ᵀg′₀, and as the proof that this identification is not stable under KL divergence but is stable under a log-likelihood variance distance (d_LLV ≤ ε ⇒ d_f,g ≤ 2Mε). Read it knowing that the KL counterexample needs the unembedding norm to grow without bound, that how far apart its embeddings are is shown numerically rather than proved, that everything concerns the final embedding and unembedding only, and that the experiments use 2- to 5-dimensional representations. |
| LIT-258 v2 | Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functions | Johnson et al. (2022), ARXIV-2210.01883. On a finite augmentation space, the NT-Logistic and spectral contrastive losses are minimized over unconstrained kernels by the positive-pair kernel K⁺(a₁,a₂) = p⁺(a₁,a₂)/(p(a₁)p(a₂)), and InfoNCE by K⁺ up to a constant on each communicating class. Kernel PCA under K⁺ returns exactly the eigenfunctions of the positive-pair Markov chain, and the top-d of those span the linear-predictor subspace that minimizes worst-case approximation error over approximately view-invariant targets. | Active — read 2026-10-14 (NOTE-751) in full; worth reading as the cleanest statement linking contrastive losses, one Mercer kernel and a minimax-optimal eigenbasis. The NT-Logistic and spectral losses are minimized over unconstrained kernels by K⁺ = p⁺/(p⊗p), and InfoNCE by K⁺ up to a constant per communicating class (over symmetric functions, with the minimizer form taken from Van den Oord et al.). Kernel PCA under K⁺ returns λ_i^(1/2) times the positive-pair chain’s eigenfunctions (Thm 3.2). The top-d span minimizes worst-case L²(p) error over approximately view-invariant targets, at ε/(2(1 − λ_(d+1))), provided d is at least the number of communicating classes (Thm 4.1). Read it knowing that everything is proved on finite spaces at the unconstrained population optimum. The supporting experiments are synthetic, and the authors report that they could not extract the eigenfunctions from real-data models. |
| LIT-260 v2 | Kernel Mean Embedding of Distributions: A Review and Beyond | Muandet et al. (2016), ARXIV-1605.09522. Point evaluation is bounded on an RKHS, so the Riesz theorem gives the reproducing kernel k_x. Likewise, when E√k(X,X) < ∞ the expectation functional f ↦ E_P f is bounded, so Riesz gives a unique mean embedding μ_P with E_P f = ⟨f, μ_P⟩. MMD is then the RKHS distance ‖μ_P − μ_Q‖, and it separates distributions exactly when the kernel is characteristic. | Active — read 2026-10-14 (NOTE-838) in full; worth reading as the one held text that runs the chain bounded evaluation → Riesz → k_x → Moore–Aronszajn → μ_P → MMD/HSIC in numbered statements, with Lemma 3.1 (μ_P exists and represents expectation when E_P√k(X,X) < ∞) proved in the text, and as a map of conditional mean embeddings and their limits. Read it knowing that it is a survey whose proofs are mostly sketched or cited, that every result assumes a fixed kernel, that it defines μ_P as a Bochner integral as well as by Riesz, so Riesz is one route to existence and not shown to be the only one, and that the unregularized conditional embedding C_YX C_XX⁻¹ often fails to exist. |
| LIT-261 v2 | A Generalized Representer Theorem | Schölkopf et al. (2001), DOI-10.1007/3-540-44581-1_27. For any nonempty set X, real positive-definite kernel k, strictly increasing g and any cost c on the training triples, even coupled, non-convex or infinite-valued, every minimizer of c + g(‖f‖) over the countable kernel expansions of finite RKHS norm is a finite expansion Σ α_i k(·,x_i) at the training points. The proof needs only the reproducing property and orthogonal decomposition onto span{k(·,x_i)}, not the Riesz theorem. Existence of a minimizer is assumed. | Active — read 2026-10-14 (NOTE-836) in full; worth reading as the short proof that any minimizer of an arbitrary cost on the training outputs plus a strictly increasing function of the RKHS norm is a kernel expansion on the training points, by the reproducing property and orthogonal decomposition, with no convexity and no Riesz theorem. Read it knowing that the theorem is stated over countable kernel expansions of finite norm rather than the completed RKHS, that it assumes a minimizer exists, and that its semiparametric Theorem 2 is stated without proof. |
| LIT-271 v2 | Neural Tangent Kernel of Matrix Product States: Convergence and Applications | Guo & Draper (2021), ARXIV-2111.14046. In the sequential infinite-bond-dimension limit, with tensor variances σ_i²/√(|α_i||α_{i+1}|) and per-tensor learning rates (|α_i||α_{i+1}|)^{-1/2}, the NTK of a periodic MPS tends to K(x,x′)=Σ_k φ(x_k)·φ(x′k) Π{l≠k} σ_l² φ(x_l)·φ(x′_l). By my reduction this is (Σ_k σ_k⁻²) times the MPS’s own GP covariance, so gradient flow is kernel regression with a fixed product kernel. The Born-machine solution P_x(t)=1/m−(1/m−P_x(0))e^{−4mKt/Z} is correct (I checked it; Z is in fact conserved). The lazy-training lemma is heuristic, the positive-definiteness “proof” shows at most semi-definiteness, several constants are off by factors of n or 2^{−n}, and there are no experiments. | Rejected — read in full 2026-09-27 (NOTE-244); not worth a reader’s time as a source. Its one sound calculation (the infinite-bond MPS is a linear model in a fixed product-kernel feature space, so its NTK is its GP kernel up to a constant) follows in a line from multilinearity. Tensor Programs II (ANTH-LIT-557) covers such architectures rigorously. The paper’s distinctive claims are the lazy-training proof, positive-definiteness “without any extra assumptions of the data set”, and “over-fitting is naturally avoided”. The first is circular, the second is false for the finite local dimensions it uses, and the third is contradicted by its own memorisation result. |
| LIT-308 v2 | Stochastic Thermodynamics of Learning | Goldt & Seifert (2017), DOI-10.1103/PhysRevLett.118.010601. For a perceptron whose weights follow overdamped Langevin dynamics and whose predicted labels are fast two-state processes, the information learnt about fixed random labels is bounded by the weights’ total entropy production: Σ_μ I(σ_T^μ:σ^μ) ≤ Σ_n [ΔS(ω_n) + ΔQ_n] (Eq. 16, k_B = T = 1). This defines a learning efficiency η ≤ 1. The cost side includes the change in the weights’ own Shannon entropy, and in the paper’s toy model η → 0.99 as the heat ΔQ → 0 (slow driving). So the bound is not a heat floor of k_BT ln 2 per bit learnt. | Active — read in full 2026-09-29 (NOTE-296); worth reading as the founding letter of the “stochastic thermodynamics of learning” line, and as the precise statement of what learning costs in that framework: entropy production, of which heat is only one part. Read the main-text claim “for arbitrary learning algorithm … at all times” alongside its supplement. The derivation uses a γ ≫ 1 (fast-label) limit and a weight-independence step that holds for batch but is not argued for online learning. |
| LIT-324 v2 | Opening the Black Box of Deep Neural Networks via Information | Shwartz-Ziv & Tishby (2017), ARXIV-1703.00810. The paper studies small tanh networks (layer widths down from 12 to 2) trained by SGD on a 12-bit, O(3)-symmetric synthetic rule, with mutual information estimated by binning each neuron into 30 bins over all 4,096 inputs. It reports two phases: a short “ERM” phase of a few hundred epochs in which I(T;Y) rises, and a long “compression” phase in which I(X;T) falls. The phases coincide with a drop in per-layer gradient SNR (from high-SNR drift to low-SNR diffusion, at about 350 epochs), and the converged layers sit on the IB curve at fitted β. Everything is shown on small synthetic tasks, with the explanations argued informally. | Active — read in full 2026-09-29 (NOTE-298); worth reading as the source of the IB account of deep-learning dynamics and of the drift/diffusion gradient-SNR phase picture, which McCandlish et al. cite. Its compression and generalisation claims are contested: the anthology’s ANTH-LIT-509 reports them refuted, which I have not checked. Read it as the claim, not the result, and never without the critique. |
| LIT-327 v2 | Thermodynamics of Prediction | Still et al. (2012), DOI-10.1103/PhysRevLett.109.120604. For a system driven by a stochastic signal through a fixed Markov kernel, the average work dissipated as the signal steps x_t → x_{t+1} equals, in units of k_BT, the system’s instantaneous nonpredictive information, I[s_t;x_t] − I[s_t;x_{t+1}] (Eq. 14). Summed over a protocol, this total “nostalgia” lower-bounds dissipation and excess work (Eq. 18), and it adds to Landauer’s bound: −β⟨Q⟩ ≥ I_e + I_mem − I_pred (Eq. 21). All three are averages over paths and protocols. None of them is a per-operation or per-bit floor on the cost of storing or using predictive information. | Active — read in full 2026-09-29 (NOTE-295); worth reading as the short, exact source of “memory that does not predict costs dissipation” and of the nostalgia-augmented Landauer bound. Read it knowing three things. The bound prices only the nonpredictive part of memory. The nonnegativity of the per-step nostalgia needs a Markov drive, which the paper does not say. And nothing in it supplies a k_BT ln 2 floor per judgment. |
| LIT-338 v2 | The information bottleneck method | Tishby et al. (1999), ARXIV-physics/0004057. The paper defines relevant information as what X tells about a second variable Y and poses the information-bottleneck problem, min over p(x̃|x) of L = I(X̃;X) − β·I(X̃;Y), under the Markov chain X̃ ← X ← Y (eq. 15). Theorem 4 gives the exact formal solution p(x̃|x) = p(x̃)/Z(x,β)·exp(−β·D_KL[p(y|x)‖p(y|x̃)]), so the KL divergence emerges as the effective distortion. Theorem 5 gives a Blahut–Arimoto-style alternating iteration that converges but, being non-convex jointly, not necessarily to a unique solution. | Active — read in full 2026-09-29 (NOTE-300); worth reading as the short, exact source of the IB functional, its self-consistent equations, the information plane and β as the one trade-off parameter. The phase-transition claim is cited out, not shown here. Nothing in it concerns gradients, SGD or batch size, so it is not the principle behind the gradient-as-channel narrative in the sense row 12 suggests. |
| LIT-341 v2 | Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets | Power et al. (2022), ARXIV-2201.02177. Small decoder-only transformers (2 layers, width 128, 4 heads, ~4·10⁵ non-embedding parameters) trained on binary-operation tables over abstract symbols can reach 100% validation accuracy long after fitting the training set. In the headline run (division mod 97, 50% data, Adam with no weight decay, a budget of 10⁶ steps), training accuracy is near-perfect before 10³ steps, validation accuracy only near 10⁶, with little progress before 10⁵. The optimisation time to generalise rises steeply as the training fraction falls (on S₅ near 25–30% data, −1% data costs +40–50% median steps), and weight decay is the most data-efficient intervention tried. | Active — read in full 2026-09-29 (NOTE-287); worth reading as the paper that named grokking and measured its data and optimisation dependence. It is a testbed paper with ablations and no mechanism, so it supplies the phenomenon row 15 cites, not the resolvability or singular-learning-theory reading. |
| LIT-345 v2 | Progress measures for grokking via mechanistic interpretability | Nanda et al. (2023), ARXIV-2301.05217. A one-layer ReLU transformer (P = 113, 30% of pairs, full-batch AdamW with λ = 1, 40k epochs) that groks modular addition implements a “Fourier multiplication” algorithm. The embedding maps a and b to cos/sin(w_k·a), cos/sin(w_k·b) at five key frequencies k ∈ {14, 35, 41, 42, 52}, with w_k = 2πk/113. Attention and MLP form cos/sin(w_k(a+b)), and the neuron-logit map W_L (≈ rank 10) reads out Σ_k α_k·cos(w_k(a+b−c)). Progress measures built from this (restricted and excluded loss) show training splits into memorisation (0–1.4k epochs), circuit formation (1.4k–9.4k) and cleanup (9.4k–14k), with the test-accuracy jump inside cleanup. | Active — read in full 2026-09-29 (NOTE-283); worth reading as the fullest mechanistic account of a grokking network. Its learned structure is exactly the character basis of ℤ/113 (five real 2-dimensional irreps), which makes it the nearest existing empirical instance of the map’s row-7 “predicate harmonics = characters” prediction, in a setting where the group was known in advance. |
| LIT-346 v2 | Analysis of Boolean Functions | O’Donnell (2014), DOI-10.1017/CBO9781139814782. Every f: {−1,1}ⁿ → ℝ has a unique multilinear (Fourier–Walsh) expansion f = Σ_S f̂(S)χ_S over the parities χ_S(x) = ∏{i∈S} x_i, which are an orthonormal basis (Thm 1.1, Thm 1.5) and are the characters of the group F₂ⁿ, χ_S(x+y) = χ_S(x)χ_S(y) (eq. 1.5; §3.2; §8.5). The central operators are all diagonal in this basis with eigenvalues depending only on |S|: the noise operator T_ρχ_S = ρ^{|S|}χ_S (Prop. 2.47) and the Laplacian Lχ_S = |S|χ_S (Prop. 2.37). Influence, noise stability and Arrow’s Theorem (via Stab{−1/3}) become statements about the Fourier weight at each level (Thms 2.20, 2.38, 2.49, 2.56). | Active — read in part 2026-09-29 (NOTE-291); worth reading as the standard, freely available text on Fourier analysis of functions on the Boolean cube. Chapters 1–2 alone give the character basis, Plancherel and Parseval, convolution, and the level-by-level diagonalisation of the noise operator and Laplacian. Read it as abelian harmonic analysis only: it has no nonabelian representation theory, no irreps of dimension > 1 and no isotypic decomposition. |
| LIT-347 v2 | Information-theoretic analysis of generalization capability of learning algorithms | Xu & Raginsky (2017), ARXIV-1705.07809. If the loss ℓ(w,Z) is σ-subgaussian for every w, then the expected generalisation error of any learning algorithm P_{W|S} obeys |E[L_μ(W) − L_S(W)]| ≤ √(2σ²·I(S;W)/n) (Theorem 1). This extends Russo–Zou to uncountable hypothesis spaces, and the paper adds a high-probability version, n = (8σ²/α²)(ε/β + log(2/β)) (Theorem 3). Minimising empirical risk plus I(S;W), relaxed to a KL term against a prior Q, yields exactly the Gibbs algorithm (Theorem 5). Noisy ERM and adaptive composition are analysed through the same mutual-information quantity. | Active — read in full 2026-09-29 (NOTE-306); worth reading as the short, clean source of the input–output mutual-information generalisation bound, and of “a learning algorithm is a channel P_{W|S}”, whose capacity is sup_μ I(S;W). The bound is in expectation and is vacuous for deterministic algorithms on continuous hypotheses, which is what the later compression and rate–distortion refinements (LIT-236, LIT-233) address. |
| LIT-348 v2 | An Empirical Model of Large-Batch Training | McCandlish et al. (2018), ARXIV-1812.06162. The paper derives a local model in which the loss progress one step can make at batch size B is ΔL_max/(1 + B_noise/B), with B_noise = tr(HΣ)/GᵀHG (eqs. 2.7–2.8). Averaged over a run at fixed B, the model gives the steps/examples hyperbola (S/S_min − 1)(E/E_min − 1) = 1, which defines the critical batch B_crit = E_min/S_min (eqs. 2.11–2.12). Across eight tasks, B_crit spans about six orders of magnitude (from 20 to over 10 million, §3), and the Hessian-free B_simple = tr(Σ)/|G|² tracks it to about an order of magnitude (Table 1: run-averaged B_simple/B_crit runs from 0.05 for the SVHN VAE and autoencoder to 8 for SVHN). | Active — read in full 2026-09-29 (NOTE-293); worth reading as the origin of the batch-size/gradient-SNR/critical-batch/time–compute result that row 12 of the owner’s map attributes to it. The theory is elementary and local, and the evidence is empirical. It has nothing on information, channels or energy, so it owns row 12’s formulas for progress and trade-off and none of its information-theoretic framing. |
| LIT-354 v4 | Algebraic Geometry and Statistical Learning Theory | Watanabe (2009), DOI-10.1017/CBO9780511800474. Per publisher description: develops statistical learning theory for singular models (neural networks, mixtures, hidden Markov models) in which the Fisher information is degenerate, using resolution of singularities to show that the real log canonical threshold governs Bayesian free energy and generalization error. | Unavailable — seeded 2026-09-29 from metadata and published descriptions, not read; no lawful full text could be reached, and it stays Unavailable until a copy is supplied. Its core results are now read in a lawful substitute by the same author, Watanabe’s A Widely Applicable Bayesian Information Criterion (LIT-616; arXiv 1208.6338, JMLR 2013). That paper restates the book’s standard representation theorem, the real log canonical threshold and the free-energy asymptotics F = nL_n(w0) + λ log n − (m − 1) log log n, citing the book for each. Read it for the results; the book holds their proofs, the generalization-error theory and WAIC. |
| LIT-356 v2 | How much does your data exploration overfit? Controlling bias via information usage | Russo & Zou (2015), arXiv:1511.05219. Suppose an analyst reports statistic φ_T, chosen by any data-dependent rule T from m candidates each σ-sub-Gaussian. Then the selection bias obeys |E[φ_T − µ_T]| ≤ σ√(2·I(T;φ)) (Prop. 1), where “information usage” I(T;φ) ≤ H(T) ≤ log m. For argmax selection over φ ~ N(µ,I) the squared error is Θ(1 + H(T)) (Prop. 3). Adaptive analyses compose additively, with I(T_{k+1};φ) ≤ Σ_i I(Y_{T_i};φ_{T_i} | H_{i−1},T_i) (Lemma 1). Answering each query with Gaussian noise of variance σ²√j/n therefore keeps the k-th answer’s error at O(σk^{1/4}/√n) (Prop. 7). | Active — read in full 2026-09-29 (NOTE-308); Worth reading as the origin of the mutual-information bias and generalisation bound that LIT-347 (Xu & Raginsky) extends, and as the first place its adaptive-composition chain rule and its per-query “information budget” with Gaussian noise (½·log(1+SNR) per query) appear. Read it knowing that it is stated for a finite set of m candidate statistics, and that “the bound is tight” is shown only for Gaussian argmax and threshold selections, as squared-error (not bias) bounds except in the exponential case. |
| LIT-369 v2 | Grokking and Generalization Collapse: Insights from HTSR theory | Prakash & Martin (2025), arXiv:2506.04434. A depth-3, width-200 ReLU MLP trained on 1,000 MNIST images (the Omnigrok recipe, with weights and biases scaled ×8, MSE loss, AdamW, WD = 0) for 10⁷ steps goes through three phases read off the accuracy curves. In pre-grokking (to ~10⁵ steps) training accuracy saturates while test accuracy stays low. In grokking (~10⁵–10⁶) test accuracy rises to ~0.85. In “anti-grokking” (~10⁶–10⁷) it falls to ~0.45–0.5 while training accuracy stays near 1. The average HTSR power-law exponent α of the two hidden weight matrices goes 4.0 → 2.9 → 1.1 at the phase ends (Table 1). In the collapse phase, correlation traps (outlier eigenvalues of the elementwise-shuffled weight matrix) appear. The paper’s own weight-decay control shows FC2’s α below 2, and traps in both layers, without a collapse (Fig. 6, Table 2). That undercuts its headline claim that α < 2 “invariably” heralds collapse. | Proposed — read in full 2026-09-30 (NOTE-317); worth knowing as the report of a late generalisation collapse after grokking at WD = 0, but it is one architecture on one 1k-sample MNIST subset, with the seed count unstated, and its central diagnostic (α < 2 ⇒ collapse) is contradicted by its own WD = 0.01 control. What would settle it: several seeds and architectures, with α and traps reported for runs that reach α < 2 without collapsing, and a comparison against the ℓ² norm under the Omnigrok LU account, which the paper does not cite. |
| LIT-370 v2 | New Evidence of the Two-Phase Learning Dynamics of Neural Networks | Zhou et al. (2025), arXiv:2505.13900. VGG-16 and ResNet-20 on CIFAR-10 (SGD with momentum, lr 0.1, 160 epochs) are compared across pairs of training times. Before an early “inflection point” (~2,500 iterations for VGG-16, ~100–500 for ResNet-20), a tiny parameter perturbation under identical SGD noise sends the run to a different basin by the end: loss barrier up to ~2, test disagreement ~0.15–0.5. After it, the same perturbation leaves the runs linearly connected. Also after it, the empirical NTK keeps changing step to step but stays within a bounded cosine distance of any later reference kernel (the “cone effect”, e.g. S ≈ 0.05–0.1 for VGG-16 vs ≈ 0.15–0.2 from initialisation). Switching to linearised training later gives better test accuracy (VGG-16 0.13 → 0.82 as the switch moves from ≤200 to 5,000 iterations). | Proposed — read in full 2026-09-30 (NOTE-318); a compact, readable demonstration that an early, short chaotic phase gives way to a confined, stable one in both basin and kernel terms, but it rests on two architectures on one dataset, one run each with no error bars. The chaos half largely re-measures Frankle et al.’s (2020) spawning / linear-mode-connectivity result with a parameter perturbation in place of fresh SGD noise, and the claimed role of the cone in generalisation is asserted. What would settle it: seeds and error bars, the batch size, a stated perturbation norm, other domains, and a quantitative test of confinement (e.g. an angular radius that stops growing) rather than a 3-D picture. |
| LIT-371 v2 | How do language models learn facts? Dynamics, curricula and hallucinations | Zucchet et al. (2025), arXiv:2503.21676. ‘An 8-layer, 44M-parameter decoder-only transformer trained with AdamW on synthetic biographies (64k individuals by default) learns factual recall in three phases. It first fits the marginal attribute-value distribution, then sits on a plateau exactly at the no-knowledge baseline, then acquires individual-specific associations. Plateau end grows with population as 0.43·N^0.81 (R² = 0.998, N = 4k–256k, 5 seeds). Attention patching from post-plateau checkpoints removes the plateau, and attention from the recall position to name tokens rises through it, so the plateau is when the attention extraction circuit forms. Nothing in the three phases is a compression of input information: the last phase adds individual-specific information.‘ | Active — read in full 2026-09-30 (NOTE-319); worth reading as a clean, intervention-backed decomposition of a loss plateau into circuit formation followed by knowledge storage, in a setting where knowledge is countable; its curriculum and fine-tuning results are single-seed and synthetic, and it measures no information quantity, so it bears on phases of training but not on the IB account. |
| LIT-372 v2 | On the Information Bottleneck Theory of Deep Learning | Saxe et al. (2018), DOI-10.1088/1742-5468/ab3985. Replicating Shwartz-Ziv & Tishby with their own code, the paper shows the information-plane “compression phase” appears only for double-saturating nonlinearities measured through an imposed binning/noise model: tanh compresses, ReLU/softplus/linear do not (binning, KDE, Kraskov; SZT task and MNIST), and re-binning the same tanh run evenly in net input removes it. Compression and generalisation dissociate in all four combinations, full-batch GD compresses as much as SGD, and the drift→diffusion gradient-SNR transition occurs in networks that cannot compress (ReLU, MNIST, a 1-1-1 linear net), so it is real but unrelated to compression. | Active — read in full 2026-09-30 (NOTE-320); worth reading as the direct experimental test of all three claims of the IB account of deep learning (two phases; compression causes generalisation; SGD diffusion causes compression), each refuted as a general statement with the original code, and for its Appendix C argument that I(h;X) is infinite in a deterministic network so every finite information-plane value belongs to the analyst’s noise model. |
| LIT-373 v4 | The two dragons of cognition: recursive condensation for predictive processing | Li (2026), DOI-10.3389/fncom.2026.1778902. A “Hypothesis and Theory” essay proposing that the neocortex is a “biological Savitch machine”. Alternating expansion (“odd parity”) and contraction (“even parity”) phases manufacture the disjoint closed sets of Urysohn’s Lemma, and storing (“memoizing”) the condensed tokens converts exponential search into polynomial “navigation”. Its one non-standard formal result, Theorem 2, R(N) ∝ b^{αN} (§5.2.2, p. 11), is stated without proof. There is no data, simulation or model. The two mathematical pillars are misdescribed: Savitch’s theorem saves space at the cost of time and does not memoize, and Urysohn’s Lemma gives a continuous separator, not a linear one, and is not an “if and only if”. | Rejected — re-read jointly with LIT-374 and arXiv 2602.07974 on 2026-10-01 (NOTE-321); Read jointly with LIT-374 and the companion StrLT preprint (arXiv 2602.07974), the essay is the cognitive motivation of a coherent programme: structure is discovered by costly search, condensed into frozen tokens, and reused cheaply. It is not just a misuse of two theorems. Read charitably, its mathematical slips are repairable and are not the reason to set it aside: the Urysohn “iff” and linear readout, Savitch as memoisation, the Definition 2 clash. The reason is what is left after repair: an amortised-inference and complementary-learning-systems picture already in the literature it cites; a neural mapping whose predictions do not discriminate it from standard consolidation accounts; one original quantitative claim, Theorem 2, that is not argued. A reader who wants the programme should read LIT-374 and the companion, where it is made precise. |
| LIT-374 v3 | Structural Decoupling: A Scaffold-Flow Theory of Generalization and Alignment | Li (2026), arXiv:2506.20699. ‘Proposes “width”, the least number of cells in an open cover on each of which some predictor is γ-contractive with conditional risk ≤ δ, as a structural complexity separate from VC dimension. From an additive Rademacher bound, √(2d_str log(e(K−1)n/d_str)/n) + LK·R_n(G) (Thm 4.5), it argues that context routing (the “scaffold”) should be trained by different signals from within-context prediction (the “flow”), and that alignment failures are scaffold failures. The routine results are proved: coupon-collector Ω(w log w), Lipschitz composition, penalised model selection, multiclass Sauer–Shelah. The new ones are not. The phase transition at width is assumed in its proof. The bouquet witness of width/VC separation contradicts its own construction. The “fundamental theorem” is false as stated, because infinite pseudo-dimension does not prevent learnability. No theorem links the CS-Laplacian estimator to width. There are no experiments.‘ | Proposed — re-read jointly with LIT-373 and arXiv 2602.07974 on 2026-10-01 (NOTE-322); promising and unproven. Read jointly with the companion arXiv 2602.07974, which carries the fuller statements and several definitions this paper leaves implicit. The formal core holds in repaired form: width as a structural complexity incomparable with the per-cell VC dimension; an error floor below width; the Ω(w log w) discovery cost; an additive Rademacher bound for routed predictors. The flagged slips are repairable and not load-bearing: the bouquet cover, the non-degeneracy assumption in the appendix, Massart, and pseudo-dimension against fat-shattering in Thm 4.6. What remains unproven are the claims that would make it matter: that the CS estimator converges to width itself and not to a predictor-relative count; that a monolithic model cannot match K ≥ w routing; above all, that router/scaffold training should be decoupled from task loss. This is presented as derived from the additive bound, but the bound holds for joint training too. The alignment section is asserted. What would settle it: on synthetic benchmarks of known width, (i) a K-sweep showing the floor at K < w for routed learners and its absence or presence for capacity-matched monolithic models; (ii) CS width recovery from a cold start; (iii) decoupled against joint router training, compared on forgetting and boundary error. |
| LIT-375 v2 | Structural Learning Theory: A Metric-Topology Factorization Approach | Li (2026), arXiv:2602.07974. >- The formal core of Xin Li’s structural-learning programme. Width w(P; γ, δ) is the least number of open cells on each of which some predictor is γ-contractive with conditional risk ≤ δ. The paper proves that width is incomparable with the per-cell VC dimension (Thm 2.2). It shows that with K < w cells there is an n-independent error floor η(w, K), under a non-degeneracy assumption stated in the body (Thm 2.4, Assumption 2.3). Width is at least c·β₁·L/D₀ under a bounded-overlap assumption (Thm 2.5), and discovering all basins costs Ω(w log w) samples (Thm 2.6). The CS-Laplacian eigenvalue count converges uniformly to the predictor-relative count w_G(P) when n ≳ (H_G + d_X log(1/r_x) + log(1/δ′))/(r_x^{d_X} g_eff²) (Thm 3.5); no result shows that w_G equals width. There are no experiments. | Proposed — read in full 2026-10-01 (NOTE-324); >- Promising and unproven. It is the programme’s fullest and most careful formal statement, and it carries the learning theory of LIT-374 §3 with the assumptions in the body. Its routine results hold, and its bouquet slip is repairable. What would settle it: - a theorem that the CS count for a learned predictor equals w(P; γ, δ), not w_G(P); - a K-sweep on benchmarks of known width, showing the floor at K < w for routed learners, and testing whether a capacity-matched monolithic model shows it; - the decoupled-versus-joint training comparison it motivates but does not run. |
| LIT-376 v2 | Local Urysohn Width: A Topological Complexity Measure for Classification | Li (2026), arXiv:2603.15412. >- It defines local Urysohn width uw_{D₀}(P, γ): the least number of connected sets of diameter ≤ D₀, each carrying a continuous classifier correct on the margin-γ safe region it meets, needed to cover that safe region. It proves: - uw = w on a bouquet of w circles with safe balls at the antipodes, when 3γ/2 ≤ D₀ < L/2 − 3γ/4 (Thm 4.1); - uw ≥ w·m, with m = Θ(L/D₀) safe balls per loop, so uw = Ω(w·L/D₀) (Cor 5.1); - two-way non-determination with VC dimension (Thm 6.1); - an Ω(w log w) sample lower bound via label-permuted coupon collection (Thm 7.1); - uw ≥ 2β₁/Δ₀ for good, convex, bounded-adjacency covers (Thm 8.5). The proofs are correct. The obstruction is metric, as the paper concedes in Remark 8.11: patches cannot reach across the wedge point. Because Urysohn’s lemma makes local correctness automatic on any patch, the measure is a connected covering number of the safe region that does not depend on the labels. | Proposed — read in full 2026-10-01 (NOTE-323); >- Promising and unproven. It is the cleanest and most honest paper in Li’s programme: definitions are precise, the main theorems are correctly proved, a dependency table separates proved from conjectural results, and earlier errors are corrected openly. It is also the consistent construction that LIT-374’s and arXiv 2602.07974’s bouquet lacked. What would settle it: a result showing that this width governs error or sample cost for learners not built from local experts (for example local-learning or nearest-neighbour rates), or a proof of Conj 8.9 under conditions its motivating examples actually satisfy. Without one of these, it is a covering number with a topological headline. |
| LIT-607 | Natural Gradient Works Efficiently in Learning | Amari (1998), Neural Computation 10(2):251–276. When the parameter space carries a Riemannian metric G, the steepest descent direction of a loss is the natural gradient G⁻¹∇L, not the ordinary gradient. For a statistical model the metric is the Fisher information, the only invariant one. Online natural gradient learning with step 1/t is Fisher efficient: it attains the Cramér–Rao bound asymptotically, as well as the best batch estimator. The paper computes the metric and its inverse for a simple perceptron, for the group of nonsingular matrices (blind source separation, giving the update ∇L·WᵀW) and for linear systems, proposes an adaptive learning rate with 1/t convergence, and conjectures that the natural gradient escapes the plateaus of backpropagation. | Active — read in full 2026-10-03 (NOTE-472); worth reading as the source of the natural gradient: steepest descent in a Riemannian parameter space is G⁻¹∇L (Theorem 1), the Fisher information is the metric on a statistical model, and online learning with the natural gradient and a 1/t rate is Fisher efficient (Theorem 2). Its efficiency proof needs the Fisher matrix at the optimum to be invertible, which is exactly what singular learning theory says fails for neural networks. |
| LIT-608 | Deep Equals Shallow for ReLU Networks in Kernel Regimes | Bietti & Bach (2021), ICLR 2021. Dot-product kernels on the sphere are diagonal in spherical harmonics, with one eigenvalue µ_k per degree k shared by N(d, k) ~ k^(d−2) harmonics. Theorem 1 reads the asymptotic decay of µ_k off the kernel function’s expansion at ±1: a non-integer exponent ν there gives µ_k ~ k^(−d−2ν+1). The arc-cosine kernels have ν = 1/2 and 3/2, and composing them preserves ν, so the NTK of a deep ReLU network decays as k^(−d) and its random-feature kernel as k^(−d−2), exactly as for two layers. Deep and shallow ReLU kernels therefore have the same RKHS, the same as the Laplace kernel’s for the NTK. | Active — read 2026-10-03 (NOTE-475): main text and Appendix A in full, the proofs in Appendices B and C followed at the level of their lemmas. Worth reading as the general rule behind the NTK eigenvalue decays the record holds: for a dot-product kernel on the sphere, the decay of its spherical-harmonic eigenvalues is fixed by the kernel function’s non-integer exponent at the endpoints ±1 (Theorem 1). Deep ReLU kernels keep the shallow exponent, so depth leaves the spectrum’s shape unchanged: µ_k ~ C(d, L) k^(−d) for the NTK and k^(−d−2) for the random-feature kernel, with only the constant growing in L (Corollaries 2–3). |
| LIT-612 | The Convergence Rate of Neural Networks for Learned Functions of Different Frequencies | Basri, Jacobs, Kasten & Kritchman (2019), NeurIPS 2019. In the linearised dynamics of a wide two-layer ReLU network, on uniform data on a sphere, the Gram matrix H∞ is a convolution, so its eigenvectors are Fourier modes or spherical harmonics. Its eigenvalues are computed in closed form per frequency. Without bias, every odd frequency k ≥ 3 has eigenvalue zero, so the network cannot represent or learn it (Theorems 2 and 4). With a zero-initialised bias all frequencies pass. The eigenvalues fall as 1/k² on S¹ and, numerically, about 1/k^(d+1) on S^d, so the time to learn frequency k grows as k² and k³ on the circle and the 2-sphere. Deep and residual networks show the same exponents. | Active — read 2026-10-03 (NOTE-481): main text in full, the supplementary derivations of the S^d eigenvalues skimmed. Worth reading for the closed-form NTK eigenvalues per frequency and the exact null space they reveal. On the circle the bias-free two-layer ReLU kernel has eigenvalue 2(k²+1)/(π²(k²−1)²) on even k ≥ 2 and exactly zero on odd k ≥ 3 (Eq. 12), so such a network cannot represent cos 3θ at all (Theorems 2 and 4). A zero-initialised bias fills the null space, with 1/(π²k²) on odd k (Eq. 13). Convergence time then grows as k² on S¹ and about k³ on S², and shallow, deep and residual networks match: fitted exponents 1.93–2.15 on S¹ and 2.74–3.13 on S² (Figs. 6–7). |
| LIT-613 | Traces of Class/Cross-Class Structure Pervade Deep Learning Spectra | Papyan (2020), JMLR 21(252). Cross-entropy indexes every deepnet quantity by example, class and would-be class, a two-way MANOVA layout. In the spectra of features, backpropagated errors, gradients, the Fisher, the Hessian and weights, knockouts attribute the top outlier to the global mean, C−1 outliers to between-class second moment, a mini-bulk of C(C−1) to between-cross-class covariance, and the main bulk to within-cross-class covariance. Feature and error class means separate from the bulk and become orthogonal with depth. Gradient class means are approximately feature class means ⊗ error class means, layer by layer; K-FAC’s single Kronecker product misaligns the Fisher’s outliers and the class-wise CFAC aligns them. For multinomial logistic regression on a Gaussian class model the expected Fisher spectrum is derived exactly. | Active — read 2026-10-03 (NOTE-474); worth reading as the general account of class/cross-class structure in deepnet spectra: features, backpropagated errors, gradients, Fisher, Hessian and weights all show one outlier from the global mean, C−1 from class means, a mini-bulk of about C(C−1) from cross-class means, and a main bulk from within-class variation. Per layer, gradient class means are approximately Kronecker products of feature class means and error class means, which is why K-FAC, pairing every class with every class, misaligns the Fisher’s outliers, and CFAC, pairing each class with itself, aligns them. Proved only for multinomial logistic regression on a Gaussian model. |
| LIT-616 | A Widely Applicable Bayesian Information Criterion | Watanabe (2012), arXiv:1208.6338; JMLR 14:867–897 (2013). A model is singular when the parameter-to-distribution map is not one-to-one or the Fisher information is not positive definite: neural networks, mixtures, reduced-rank regression, HMMs. Its Bayes free energy is nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the real log canonical threshold (RLCT), not d/2. WBIC, the posterior average of nL_n(w) at inverse temperature 1/log n, has the same leading terms even for singular, unrealizable models, equals BIC for regular ones, and needs no knowledge of the true distribution. Two temperatures give an estimate of λ. In reduced-rank regression WBIC picked the true rank in 100 of 100 trials and recovered theoretical RLCTs closely. | Active — read in full 2026-10-03 (NOTE-473); worth reading as the record’s lawful, short statement of the core of singular learning theory, standing in for Watanabe’s unread book (LIT-354). It states the Bayes free energy asymptotics F = nL_n(w0) + λ log n − (m − 1) log log n + O_p(1), with λ the real log canonical threshold and m its multiplicity (Theorem 2), and defines λ through resolution of singularities (Theorem 1, Eq. 18). Its own result is WBIC, the posterior mean of nL_n(w) at inverse temperature 1/log n, which matches F to the λ log n term even for singular and unrealizable models (Theorem 4), equals BIC for regular ones (Theorem 5), and estimates λ from two temperatures (Corollary 3). The prior results are stated with citations to the book, not re-proved. |
| LIT-617 | The Full Spectrum of Deepnet Hessians at Scale: Dynamics with SGD Training and Sample Size | Papyan (2018), arXiv. Stochastic Lanczos without reorthogonalization (O(Mnp) time, O(p) memory) plus subspace-iteration deflation measures the full Hessian spectrum of VGG11 and ResNet18 on MNIST, Fashion-MNIST and CIFAR-10/100, over 16,000 trained models. It confirms a bulk with about C outliers, attributes the outliers to the Gauss–Newton term G and most of the bulk to the residual H, finds H’s spectrum symmetric with a power-law tail rather than Marchenko–Pastur, and finds H never negligible. Version 2 decomposes G into class means (outliers), cross-class variation and within-class variation (two bulks) and tracks them across epochs and sample sizes. | Active — read 2026-10-03 (NOTE-478); worth reading as the measurement that the bulk-and-C-outliers spectrum of the Hessian persists at full scale (VGG11, 28M parameters), that the outliers belong to the Gauss–Newton/Fisher term G and the bulk mostly to the residual H, and, in its second version, that G splits into class means (outliers) and two bulks (cross-class and within-class variation). Its tools are stochastic Lanczos spectral density estimation without reorthogonalization, plus subspace-iteration deflation. |
| LIT-618 | Prevalence of Neural Collapse during the terminal phase of deep learning training | Papyan, Han & Donoho (2020), PNAS 117(40). Training past zero error (the terminal phase, TPT) drives four linked phenomena in the last layer: within-class variability collapses (Tr(Σ_WΣ_B†)/C → 0), centred class means become equinorm and equiangular with cosines −1/(C−1), a simplex ETF; classifier rows become proportional to the means; and the decision becomes nearest class mean. Measured on MNIST, Fashion-MNIST, SVHN, CIFAR-10/100, STL10 and ImageNet with VGG, ResNet and DenseNet, 480 models. Test accuracy and DeepFool robustness keep improving during TPT. Theorems: NC1–NC2 imply NC3–NC4 for the Webb–Lowe and Soudry max-margin classifiers, and the simplex ETF uniquely maximizes the small-noise error exponent. | Active — read in full 2026-10-03 (NOTE-476); worth reading as the source of neural collapse: in the terminal phase of training, last-layer features collapse to their class means (NC1), the centred means form a simplex equiangular tight frame (NC2), the classifier aligns with them (NC3) and the decision becomes nearest class centre (NC4). Measured on 7 datasets × 3 architectures; NC1–NC2 imply NC3–NC4 for the Webb–Lowe and max-margin classifiers, and the simplex ETF is the unique optimal codebook in a vanishing-noise Gaussian channel. It explains Papyan’s Hessian outliers as class means outgrowing collapsing within-class variation. |
| LIT-619 | Measurements of Three-Level Hierarchical Structure in the Outliers in the Spectrum of Deepnet Hessians | Papyan (2019), ICML. For a C-class network the Gauss–Newton term of the Hessian is G = (1/n)ΔΔᵀ, a second moment (not covariance) of logit derivatives δᵢ,c,c′ indexed by example, true class and logit. Grouping them into C² group means δc,c′ and C clusters with centres δc gives G = G₀ + G₁ + G₂ + G₃ (Eq. 28). The top-C outliers of G are those of G₁ = (C−1)Ave_c δcδcᵀ, because clusters are far apart and tight. Early in training the group means cluster by logit c′, later by true class c. Holds across MNIST, Fashion-MNIST, CIFAR-10 and ImageNet with VGG, ResNet and DenseNet; the outliers exceed their approximations slightly, as random matrix theory would predict. | Active — read in full 2026-10-03 (NOTE-479); worth reading as the paper that explains the C outliers of the deepnet Hessian: the Gauss–Newton/Fisher term G is a second moment, not a covariance, of logit derivatives indexed by (example, class, logit), and the C outliers are the eigenvalues of G₁ = (C−1)·Ave_c δ_c δ_cᵀ, the C class-cluster centres, computable from a C×C Gram matrix without high-dimensional eigenanalysis. The match is empirical, over 18,000 trained models. |
| LIT-623 v2 | Bayesian Interpolation | MacKay (1992), Neural Computation 4(3):415–447. Two levels of inference: fit each model, then rank models by their evidence P(D|H). Under a Gaussian approximation the evidence is the best-fit likelihood times an Occam factor, the ratio of posterior to prior accessible parameter volume, so complex or finely tuned models penalise themselves (Eqs. 5–6). For linear interpolation with quadratic regularisers the evidence sets the regularising constant and the noise level through γ, the number of well-determined parameters (2αE_W = γ, 2βE_D = N − γ). It ranks polynomial, radial-basis, spline, Hermite and neural-network models on two data sets. In expectation, no fixed model’s log evidence beats the true model’s. | Active — read in full 2026-10-03 (NOTE-471); worth reading as the source of the evidence framework and the Occam factor in the form the field uses: evidence ≈ best-fit likelihood × Occam factor, the Occam factor being the ratio of posterior to prior accessible parameter volume, (2π)^{k/2} det^{-1/2}A · P(w_MP|H) under a Gaussian approximation (Eqs. 5–6). It also gives the “number of good parameter measurements” γ = Σ λ_a/(λ_a + α), which sets the regularising constant (2αE_W = γ) and the noise level (2βE_D = N − γ) at the evidence maximum. The framework is Gull and Skilling’s, as MacKay says, and every demonstration is one-dimensional interpolation. Flagged anthology-candidate for its regularisation and early-stopping arguments. |
| LIT-624 | Towards Understanding the Spectral Bias of Deep Learning | Cao, Fang, Wu, Zhou & Gu (2019). For a two-layer ReLU network trained by gradient descent on squared loss, the training residual projected onto the eigenfunctions of the NTK integral operator with the r_k largest eigenvalues decays like (1 − λ_(r_k))^T (Theorem 4.2). This holds for any labelling, provided the sample size beats the eigengap and the width is polynomial in λ_(r_k)^(−1). On uniform data on S^d the eigenfunctions are spherical harmonics, with µ_k = Ω(k^(−d−1)) for k ≫ d and Ω(d^(−k+1)) for d ≫ k on even degrees, and µ_k = 0 on odd k ≥ 3 (Theorem 4.3). Lower degrees are learned first, faster, and with fewer samples and narrower networks. | Active — read 2026-10-03 (NOTE-483): main text and Appendix A in full, the proofs in Appendix B followed at the level of their lemmas. Worth reading as the theorem that makes “spectral bias” a statement about NTK eigendirections: for a two-layer ReLU network trained by gradient descent, the residual projected onto the top r_k eigenfunctions of the NTK integral operator shrinks like (1 − λ_(r_k))^T (Theorem 4.2), for any target, with sample size governed by the eigengap λ_(r_k) − λ_(r_k+1). On uniform spherical data the eigenfunctions are spherical harmonics, the odd degrees above 1 have eigenvalue zero, and µ_k = Ω(max(k^(−d−1), d^(−k+1))) for even k (Theorem 4.3). |
| LIT-657 | The Generalization Ridge: Information Flow in Natural Language Generation | Chang, Deng & Chen (2025), COLM 2026. InfoRidge estimates, layer by layer and through fine-tuning, the matrix-based mutual information between a transformer’s last-token hidden state and the embedding of the target token, and between each block’s residual update and that target. In GPT-2 Small and Medium, Qwen-2.5 0.5B and Llama 3.1 8B on CLUTRR, ECQA and a synthetic arithmetic task, it rises, peaks in the upper-middle layers and falls. On the synthetic task the peak coincides with the best OOD early-exit accuracy while ID accuracy keeps rising. Middle blocks add the most information, attention to task tokens peaks there, and residual scales fitted on OOD data downweight the last layers. The profile persists over 50 decoding steps on CNN/DailyMail. | Active — read 2026-10-03 (NOTE-536); worth reading as a layer-wise information profile of fine-tuned language models: a matrix-based Rényi estimate of I(Zℓ; Y), between the last-token hidden state at layer ℓ and the input embedding of the true next token, rises through the early layers, peaks in the upper-middle layers and falls in the last few, the “generalization ridge”. On a synthetic arithmetic task with GPT-2 Small the peak (0.27 at layer 10 of 12) is where OOD early-exit accuracy is best (56.9%) while ID accuracy keeps rising to 93.2% at layer 12. Learned residual scales put less weight on the final layers when fitted on OOD data. The estimator is a kernel-alignment quantity, not Shannon mutual information: at layer 12 it falls to 0.019, near its layer-6 value, where the same layer decodes the target with 93% ID accuracy. |
| LIT-663 | Taxonomizing local versus global structure in neural network loss landscapes | Yang, Hodgkinson, Theisen, Zou, Gonzalez, Ramchandran & Mahoney (2021), NeurIPS. Thousands of trained networks are placed on a load-like by temperature-like grid and measured three ways: Hessian top eigenvalue and trace (local sharpness), Bezier-curve mode connectivity between two independently trained models (global connectivity), and CKA similarity of their outputs on mixup points. Two transitions give four phases: I sharp and poorly connected, II sharp and well connected, III flat and poorly connected, IV flat and well connected, with IV split by CKA into IV-A and IV-B. Test accuracy is best in IV-B. Sharpness alone mispredicts accuracy across phases, and in phase III training to zero loss does worse than stopping in the sharp phases. Double descent appears as a dark band at the phase boundary under label noise. | Active — read 2026-10-03 (NOTE-516); worth reading as the empirical taxonomy of loss landscapes by local and global structure: Hessian top eigenvalue and trace separate locally sharp from locally flat, Bezier-curve mode connectivity between independently trained models separates globally poorly-connected from well-connected, and CKA similarity of their outputs splits the well-connected, flat phase in two. Across a load-like by temperature-like grid (width, data amount or label noise against batch size, learning rate or weight decay), the best test accuracy sits in the flat, well-connected, high-similarity phase IV-B; the Hessian alone mispredicts, and where connectivity is poor, training to zero loss can lower test accuracy. Empirical only, mostly ResNet18 on CIFAR-10. |
| LIT-667 | Does Data Scaling Lead to Visual Compositional Generalization? | Uselis, Dittadi & Oh (2025), ICML 2025. Two labelled concepts with n values each form an n × n grid; training shows k combinations per value and testing uses the rest. On dSprites, 3DShapes, PUG, colored MNIST and a new FSprites, ResNet-50s trained from scratch reach near 100% on seen pairs and fall sharply on unseen ones; 4× more in-distribution data does not close the gap, while more values and more combinations do. Features pass from spurious, to discriminative but entangled, to linearly factored and near-orthogonal as combinatorial coverage grows. Exact linear factorization makes k = 2 sufficient for perfect generalization. Frozen DINO, DINOv2 and CLIP features are above chance but far from that ideal. | Active — read 2026-10-03 (NOTE-525); worth reading as a controlled test of whether more data buys visual compositional generalization: in an (n, k) grid where each of n values of two concepts is seen in only k combinations, ResNet-50s trained from scratch fail on unseen pairs whatever the data volume (4× more in-distribution data leaves 60–80% drops), and succeed only as the fraction of combinations seen rises, when their features become linearly factored (a pair’s embedding is the sum of per-concept vectors; R² > 0.8). Under exact linear factorization, k = 2 combinations per value suffice for a linear classifier to get all unseen pairs right (Proposition 4.1). DINO, DINOv2 and CLIP features are partly factored. Model selection is oracle, on the test set, by design. |
| LIT-671 | Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets | Kuditipudi, Wang, Lee, Zhang, Li, Hu, Arora & Ge (2019), NeurIPS 2019. Mode connectivity follows from generic robustness properties of trained nets. Two ε-dropout-stable ReLU networks, each able to zero half its units per layer at loss cost ε, are connected by a piecewise-linear path of loss at most max endpoint + ε; ε-noise-stable networks are connected by 10 segments with barrier Õ(ε). Conversely, for any width there is a dataset on which global minima of a two-layer student are disconnected, so connectivity is a property of the solutions training finds. Checked on MNIST convnets and VGG-11 on CIFAR-10. | Active — read in full 2026-10-03 (NOTE-544); the first explanation of mode connectivity that applies to realistic multilayer nets. If two solutions can each lose half their units per layer at a cost of at most ε in loss (dropout stability), they are joined by a piecewise-linear path whose loss never exceeds the worse endpoint by more than ε; a stronger noise-stability condition gives 10 segments. A counterexample shows that connectivity fails for some global minima of arbitrarily overparametrized two-layer nets, so the property belongs to the solutions training finds, not to the landscape. The conditions are checked on small MNIST convnets and one VGG-11 trained with channel-wise dropout. |
| LIT-672 | Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior | Martin & Mahoney (2017), arXiv preprint. Zhang et al.’s observations, that deep networks fit noisy labels and that only early stopping regularizes them, are read through the statistical mechanics of learning. A Very Simple Deep Learning model has two control parameters: a load α = m_eff/N, lowered by randomizing labels, and a temperature τ, raised by stopping early. In the thermodynamic limit (m, N → ∞, α fixed), generalization error can drop discontinuously at a critical αc, and a two-dimensional (α, τ) phase diagram has a spin-glass-like overtraining phase. The mechanism is entropy–energy competition, shown on the continuous and Ising perceptrons: the Ising entropy vanishes near zero error, so the first-order condition loses its interior solution. | Active — read in full 2026-10-03 (NOTE-542); worth reading as the position paper that brought the statistical-mechanics theory of generalization back to deep learning: a network is idealized as a black box governed by a load-like parameter α (effective data per capacity, lowered by label noise) and a temperature-like parameter τ (raised by early stopping), and in the thermodynamic limit m, N → ∞ with α = m/N fixed, generalization has phases with sharp transitions. The mechanism is shown on the continuous versus Ising perceptron: where the entropy of near-optimal hypotheses vanishes (s(ε) ~ −(π²/2)ε² ln ε), the error drops discontinuously at αc, which PAC/VC bounds do not even qualitatively describe. A qualitative argument with no new theorem and no experiment. |
| LIT-676 | Unveiling Mode Connectivity in Graph Neural Networks | Li, Chen, Han, Zeng, Liu & Tang (2025), preprint. GCN solutions for node classification from different seeds are connected by quadratic Bezier curves but often separated on the straight line, more so on co-authorship and heterophilous graphs than on citation graphs. GNN variants (GCN, GraphSAGE, GAT) give similar barriers, MLPs different ones. On CSBM graphs, barriers fall with density and feature separability and are highest at middling homophily. The authors bound the barrier through the spectral gap, bound the generalisation gap by the barrier, and propose a Wasserstein distance between graphs’ interpolation-loss curves that correlates with transfer gaps. | Proposed — read in full 2026-10-03 (NOTE-529); watching, as the first study of mode connectivity in graph neural networks. Two GCNs trained on the same graph with different seeds are usually joined by a low-loss quadratic Bezier curve but often not by a straight line, and the barrier depends on the graph more than on the GNN variant: it falls with density, with feature separability, and with homophily far from one half, in synthetic CSBM graphs and, by the authors’ reading, in 12 real ones. A loss-barrier bound, a generalisation bound in terms of the barrier, and a barrier-based distance between graphs are stated, but the appendix that should hold the proofs does not; the v1 text still carries an editor’s placeholder. What would settle it: the proofs, and numbers for the barriers the figures show. |
| LIT-678 | Intermediate Layer Classifiers for OOD generalization | Uselis & Oh (2025), ICLR 2025. Linear probes on the frozen features of an intermediate layer (ILCs) are compared with last-layer retraining for out-of-distribution classification on CMNIST, Waterbirds, CelebA, MultiCelebA, CIFAR-10C, CIFAR-100C and ImageNet variants, with ResNets and ViTs. Few-shot (probe trained on OOD data), the best intermediate layer usually wins, by more when OOD data are scarce. Zero-shot (probe trained on ID data, OOD only for selecting the layer), it beats retraining the last layer, sharply under subpopulation shift. Layer L − 2 beats L − 1 in every dataset shown. A distance-ratio sensitivity score finds intermediate features less displaced by the shift than penultimate ones, most of all for minority groups. | Active — read 2026-10-03 (NOTE-541); worth reading as the measurement that, under distribution shift, a trained classifier’s best linear probe is usually not on the penultimate layer: Intermediate Layer Classifiers (linear probes on frozen layer-l features, l ≤ L − 2) beat last-layer retraining in the few-shot setting on most of nine shifted datasets, and, trained only on in-distribution data, raise worst-group accuracy from 79.4% to 87.1% on Waterbirds and 56.0% to 82.0% on CelebA. The proposed reason is that penultimate features move further under shift, especially for minority groups, while intermediate ones stay put. Selection of the layer uses OOD validation data, and the gains for ViTs are small. |
| LIT-679 | A Random Matrix Analysis of Random Fourier Features: Beyond the Gaussian Kernel, a Precise Phase Transition, and the Corresponding Double Descent | Liao, Couillet & Mahoney (2020), NeurIPS 2020 and J. Stat. Mech. (2021). For random Fourier features ΣX = [cos(WX); sin(WX)] with n, p, N large and comparable, the expected resolvent of ΣXᵀΣX/n is asymptotically Q̄ = ((N/n)(Kcos/(1+δcos) + Ksin/(1+δsin)) + λI)⁻¹, with δσ = (1/n)tr KσQ̄, not the Gaussian-kernel prediction. Closed forms for training and test MSE follow and match MNIST, Fashion-MNIST and Kannada-MNIST. In the ridgeless limit the δ jump at 2N = n, separating an under- and an over-parameterized phase; the test MSE diverges there unless test data sit within about λ of the training data, which is the double-descent peak. A tuned λ removes it. | Active — read 2026-10-03 (NOTE-519); worth reading as an exact random-matrix account of double descent in random Fourier feature ridge regression when n, p and N are large together: the RFF Gram matrix does not converge to the Gaussian kernel in spectral norm, but its resolvent has a deterministic equivalent built from the cos and sin kernels with corrections δcos, δsin solving a fixed point (Theorem 1). Training and test MSE follow in closed form (Theorems 2–3). As λ → 0 the δ jump from O(1) to order 1/λ across 2N = n, so the model has two phases, and the test error diverges at the boundary when test data differ from training data; that singularity is the double-descent peak. Data need only be concentrated random vectors, not Gaussian. |
| LIT-683 | Exact expressions for double descent and implicit regularization via surrogate random design | Dereziński, Liang & Mahoney (2019), NeurIPS 2020. For the Moore–Penrose estimator X⁺y in linear regression, a determinantal surrogate design (density ∝ pdet(XXᵀ), Poisson sample size with mean n) gives exact MSE: σ² tr((Σ + λnI)⁻¹)(1 − αn)/(d − n) + wᵀ(Σ + λnI)⁻¹w·(d − n)/tr((Σ + λnI)⁻¹) for n < d, σ² tr(Σ⁻¹) at n = d, σ² tr(Σ⁻¹)(1 − βn)/(n − d) for n > d, where n = tr(Σ(Σ + λnI)⁻¹). For n < d, E[X⁺y] equals the population ridge solution with that λn: over-parameterization is implicit ridge regularization whose strength makes n the effective dimension. The surrogate agrees with the i.i.d. design as d, n → ∞ for sub-Gaussian data. | Active — read 2026-10-03 (NOTE-518); worth reading as the exact, non-asymptotic account of double descent and implicit regularization for minimum-norm least squares: replacing the i.i.d. design by a determinantal “surrogate” design gives closed-form MSE in every regime (Theorem 1) and shows that, below the interpolation threshold n < d, the expected minimum-norm solution equals the population ridge solution with the λn for which the ridge effective dimension tr(Σ(Σ + λnI)⁻¹) equals n (Theorem 2). The surrogate matches the i.i.d. design asymptotically for sub-Gaussian data (Theorem 3), at an empirically O(1/d) rate. The tool is a class of random matrices for which determinant and expectation commute. |
| LIT-792 | Transformers learn in-context by gradient descent | von Oswald, Niklasson, Randazzo, Sacramento, Mordvintsev, Zhmoginov and Vladymyrov (2022), ICML 2023. Proposition 1 constructs weights for which one linear self-attention layer transforms each token (x_j, y_j) to (x_j, y_j − ΔW x_j), with ΔW one gradient step on the context’s squared loss, so the query token’s read-out equals the prediction of the updated linear model. A trained single layer matches the construction up to scale; deeper models outperform K steps of gradient descent and match GD++, gradient descent on inputs transformed by I − γXXᵀ. Synthetic regression only. | Active — read 2026-10-09 (NOTE-621); worth reading here for one exact identity: with constructed key, query, value and projection matrices, one linear self-attention layer applied to tokens (x_j, y_j) changes every token exactly as one gradient-descent step on the context’s squared regression loss would change the targets (Proposition 1), so reading a context and updating an implicit model’s weights on it produce the same prediction. A trained single layer finds those weights up to scale; deeper trained models match a preconditioned variant (GD++) rather than plain gradient descent. Everything is on synthetic regression with small attention-only models. |
| LIT-796 | The Surprising Effectiveness of Test-Time Training for Few-Shot Learning | Akyürek, Damani, Zweiger, Qiu, Guo, Pari, Kim and Andreas (2024), ICML 2025. Test-time training here is a temporary, per-task LoRA update of a language model on a loss built from the task’s own demonstrations, discarded afterwards. On BIG-Bench Hard (10-shot, Llama 3.1 8B) it raises accuracy from 50.5% (in-context learning) to 57.8%; on ARC it takes a fine-tuned 1B model from 5% to 29% on an 80-task subset and an 8B system to 53.0% on the public validation set. Training on leave-one-out in-context tasks beats training on the bare pairs. | Active — read 2026-10-09 (NOTE-637); worth reading here as a controlled comparison of two uses of the same demonstrations: held as context (in-context learning) or turned into a temporary update of the model’s weights (test-time training, per-task LoRA, discarded after the task). With nothing added but the update, BIG-Bench Hard 10-shot accuracy goes from 50.5% to 57.8% and fine-tuned ARC accuracy roughly sixfold on an 80-task subset (5% to 29%). The update works best when its training data keep the in-context format. The comparison is empirical, at 1B–8B parameters, on two benchmarks with known leakage risk. |
| LIT-797 | An Explanation of In-context Learning as Implicit Bayesian Inference | Xie, Raghunathan, Liang and Ma (2021), ICLR 2022. Models pretraining text as a mixture of HMMs indexed by a latent concept θ and assumes the LM fits it exactly; in-context prediction is then the posterior predictive, and Theorem 1 shows the in-context predictor reaches the Bayes-optimal 0-1 risk as the number of examples grows, when each example’s KL signal about θ exceeds the error from the low-probability transitions between examples. Without that distinguishability, excess risk falls roughly as 1/k in the example length. Small Transformers and LSTMs trained on the synthetic GINC dataset reproduce it, and also show order sensitivity and zero-shot beating few-shot. | Active — read 2026-10-09 (NOTE-633); worth reading as the cleanest formal statement of in-context learning as posterior inference over a latent concept: if pretraining documents are a mixture of HMMs and the model fits that mixture exactly, then conditioning on a prompt of concatenated examples selects the prompt’s concept by posterior concentration, despite the prompt being low-probability under pretraining, provided each example carries more KL signal about the concept than the error from its unnatural boundaries (Theorem 1). The guarantees are asymptotic in the number of examples and say nothing about how a trained network computes the posterior; the experiments are on a synthetic dataset (GINC) built to satisfy the assumptions. |
| LIT-801 | Learning to (Learn at Test Time): RNNs with Expressive Hidden States | Sun, Li, Dalal, Xu, Vikram, Zhang, Dubois, Chen, Wang, Koyejo, Hashimoto and Guestrin (2024), ICML 2025. Makes a sequence layer’s hidden state the weights W of an inner model f, updated per token by a gradient step on a learned self-supervised reconstruction loss, with output f(θ_Q x_t; W_t). With a linear f and batch gradient descent this is exactly linear attention (Theorem 1); with the Nadaraya–Watson estimator as inner learner it is exactly softmax self-attention (Theorem 2). TTT-Linear and TTT-MLP match or beat Mamba at 125M–1.3B and keep lowering perplexity to 32k tokens of context, where Mamba plateaus after 16k. | Active — read 2026-10-09 (NOTE-620); worth reading for its reframing of a sequence layer as a learner trained on its own context: the hidden state is the weights of an inner model, the update rule a gradient step on a self-supervised loss, the output rule the inner model’s prediction. Two exact identities follow: a linear inner model with batch gradient descent is linear attention (Theorem 1), and the Nadaraya–Watson estimator as inner learner is softmax self-attention (Theorem 2). The empirical part, TTT-Linear and TTT-MLP against Mamba and a Transformer at 125M–1.3B, is an architecture result that belongs to the anthology’s question. |
| LIT-855 | Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models | Karkada, Simon, Bahri and DeWeese (2025), ARXIV-2502.09863, NeurIPS 2025. The quartic approximation of word2vec’s loss, under symmetric reweighting with constant aggregate weight, is exactly unweighted factorisation of M*, the co-occurrence matrix’s relative deviation from independence, and from small initialisation gradient flow learns M*‘s eigenvectors one at a time in sigmoidal steps. The embeddings this predicts match word2vec’s features and benchmark scores far better than truncated PMI or PPMI factorisations do. | Active — read 2026-10-09 (NOTE-660); worth reading as the first closed-form account of what a contrastive word-embedding algorithm learns under a rank constraint, and in what order. Replacing word2vec’s logistic loss by its quartic Taylor approximation, and choosing symmetric reweightings whose aggregate weight is constant, turns the objective into unweighted symmetric matrix factorisation of M*, the matrix of relative deviations of word co-occurrence from independence (Theorem 1, proved). From vanishing random initialisation, gradient flow then learns the eigenvectors of M* one at a time, each by a sigmoidal step in time (ln(λ_k/σ²))/λ_k (Result 3, a derivation that drops terms, not a proof). The resulting embeddings track word2vec’s features and benchmark scores closely, and much more closely than truncated factorisations of PMI or positive PMI. The analogy part is empirical: difference vectors of semantic pairs look like a spiked random matrix, and the spike’s strength tracks analogy accuracy. |
| LIT-857 | Single-Head Attention in High Dimensions: A Theory of Generalization, Weights Spectra, and Scaling Laws | Boncoraglio, Erba, Troiani, Xu, Krzakala and Zdeborová (2025), arXiv. For a tied single-head softmax attention layer trained by ridge-penalized ERM on Gaussian token sequences labelled by an attention-indexed target, in the limit n/d², p/d fixed, a six-parameter variational problem gives the exact train and test error, interpolation and perfect-recovery thresholds, and the singular-value law of the learned query–key map: η·ReLU(S0 + δZ − ε), Z a GOE matrix. Weight decay is a nuclear-norm penalty on that map, so the factorized model beats direct training of the product. Power-law targets give sequential mode recovery and LASSO- rate scaling laws. Non-rigorous (replica/AMP), checked against Adam. | Active — read 2026-10-09 (NOTE-661); worth reading as a solvable model in which the spectrum of trained attention weights is derived, not just measured: for empirical risk minimization of one tied softmax attention layer on Gaussian sequences, with n of order d², the learned query–key map is, in law, a rescaled ReLU of the target plus a semicircle noise bulk shifted by a threshold, so its outliers are recovered target directions and its bulk is finite-sample noise. Weight decay on the factor is exactly a nuclear-norm penalty on the product. With a power-law target, modes are recovered one by one as the threshold moves, which gives heavy-tailed learned spectra and power-law learning curves whose exponents match the LASSO rates. The main results are replica/AMP “Claims”, not theorems, checked against Adam runs at d = 100–400. |
| LIT-858 | Towards Understanding Grokking: An Effective Theory of Representation Learning | Liu, Kitouni, Nolte, Michaud, Tegmark and Williams (2022), NeurIPS 2022. In a toy model that decodes the sum of two learned embeddings, grokking is representation learning: an unseen pair is answered correctly when its embedding sum equals a trained pair’s, and the critical training fraction (about 0.4 for addition with p = 10) is where the training set’s “parallelogram” equations first fix the embedding up to translation and scale. A posited effective loss on the normalised embeddings predicts that threshold and, through its third Hessian eigenvalue, the time to structure. Phase diagrams over decoder learning rate and weight decay place grokking between comprehension and memorization in the toy, a transformer and MNIST. | Active — read 2026-10-09 (NOTE-662); worth reading as an early mechanistic account of when grokking networks generalise, in a toy model that sums two learned embeddings and decodes the sum. An unseen pair is predicted correctly when its embedding sum coincides with a trained pair’s (a “parallelogram”), and the critical training fraction is where the training set’s parallelogram equations first fix the embedding up to translation and scale. The account is exact for the toy (1D embeddings, addition hard-coded in the architecture, an effective loss posited and derived only for a linear decoder); the transformer and MNIST evidence is qualitative (PCA pictures and phase diagrams), and the four “phases” are defined by accuracy thresholds, not by any order parameter. |
| LIT-859 | Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformers | Mainali and Teixeira (2025), ARXIV-2504.12916. For a single linear attention layer trained by small-step SGD on in-context regression ℝᵈ → ℝᵈ, with the input covariance and every task matrix sharing one eigenbasis and the weights assumed to stay in it, the averaged dynamics decouple into one logistic equation per eigenmode: mode α is learned on a timescale τ_α = (ηP s_α²)⁻¹, giving plateaus and cliffs in the loss, and settles where p_α q_α = N / ((N+1)s_α + Tr S), so the trained predictor applies an inverse, finite-N-corrected training covariance to the context’s empirical covariance. A scaling symmetry conserves ‖p₂‖² − ‖q₁‖². Effective-rank and subspace measures applied to attention-only transformers and a grokking model are offered as qualitative parallels. | Active — read 2026-10-09 (NOTE-658); worth reading as an exactly solved case of the slow learning that produces a fast in-context learner: for one linear-attention layer trained by gradient flow on in-context vector regression, with tasks and inputs sharing one eigenbasis, each eigenmode of the input covariance learns by its own logistic curve on a timescale ∝ 1/s_α², the trained product of the two weight blocks settles at the inverse of a finite-context-corrected training covariance, so the learned in-context predictor is a preconditioned estimator (the fixed point Ahn et al. and Zhang et al. had found), and ‖p₂‖² − ‖q₁‖² is conserved. The decoupling rests on an assumed spectral alignment and on large N and many tasks. The second half, measures applied to attention-only transformers and a grokking model, is qualitative and correlational: no seeds, error bars or controls are reported, and “mechanistic explanation” overstates it. |
| LIT-862 | A mathematical theory of semantic development in deep neural networks | Saxe, McClelland and Ganguli (2019), PNAS 116(23):11537–11546, ARXIV-1810.10531. A two-layer linear network trained from small weights learns the singular modes of its input–output correlations one at a time, each in a sigmoidal step at a time of order (τ/s)ln(s/ε), where a shallow network learns them all at once. With data generated on a tree this gives coarse-to-fine differentiation in stages, transient illusory correlations and narrowing generalisation; the SVD also defines typicality, prototypes and category coherence, and minimum-norm weights fix the hidden similarity matrix across networks. | Active — read 2026-10-09 (NOTE-666); worth reading as the paper that turns the exactly solved learning dynamics of a two-layer linear network into a theory of semantic development. Trained by gradient descent from small weights on whitened inputs, the network learns the singular modes of the item–feature correlation matrix one at a time, each by a sigmoid at a time of order (τ/s)ln(s/ε), while a shallow network learns every mode on one exponential timescale. Hierarchically generated data have singular values that fall with depth in the tree, so broad distinctions are learned first in stage-like transitions, individual predictions can rise and fall between stages, and inductive generalisation narrows with development. The same SVD gives typicality, prototypes and a coherence threshold (a BBP transition), rings give Fourier modes, and minimum-norm weights, which small initialisation approximates, give identical hidden similarity matrices across networks. The trajectory is exact only from decoupled, balanced initial weights with white inputs; from random small weights it is checked by simulation, and the match to children and cortex is qualitative. |
| LIT-864 | Exact solutions to the nonlinear dynamics of learning in deep linear neural networks | Saxe, McClelland and Ganguli (2014), ARXIV-1312.6120, ICLR 2014. Solves gradient-descent learning in deep linear networks with whitened inputs from decoupled initial weights: each singular mode of the input–output correlation is learned by a sigmoid at a time of order (τ/s)ln(s/ε), with conserved differences of squared layer strengths. For any depth, the delay depth adds is set by the initial end-to-end mode strength, finite as depth grows when that strength is of order one and the step size is scaled with depth. Argues, with simulations, that scaled Gaussian layers are not isometric in product while orthogonal ones are, and that tanh networks with orthogonal weights near gain 1 stay close to isometric. | Active — read 2026-10-09 (NOTE-667); worth reading as the paper in which the learning dynamics of deep linear networks were solved exactly from decoupled starts at any depth (Fukumizu had solved one hidden layer for another class of starts) and the stepwise, mode-by-mode picture was set out that later work calls Saxe-style. Gradient descent on a linear network with one hidden layer and whitened inputs is a cubic system whose modes cooperate within a singular mode of the input–output correlations and repel across modes; from decoupled initial weights each mode’s strength follows a sigmoid reaching s at a time of about (τ/s)ln(s/ε), with a conserved difference of squared layer strengths and, in the appendix, the unbalanced case. Beyond what the 2019 restatement keeps, it solves arbitrary depth from decoupled starts: depth slows learning through the initial end-to-end mode strength, so from strength of order one the number of iterations stays finite as depth grows once the step size is scaled down with depth, while from strength ε the delay grows like s/ε. It argues that scaled Gaussian layers preserve norms but not isometry, that random orthogonal layers give depth-independent learning times on MNIST, and that tanh networks with orthogonal weights near gain 1 keep the end-to-end Jacobian’s singular values of order one. Random initial weights, the role of isometry in learning time and everything nonlinear rest on simulation, not proof. |
| LIT-877 | How Deep Neural Networks Learn Compositional Data: The Random Hierarchy Model | Cagnetta, Petrini, Tomasini, Favero and Wyart (2023; Phys. Rev. X 14, 031001, 2024). Introduces the Random Hierarchy Model, a random context-free grammar of depth L with m synonymic expansions per symbol. Shallow and lazy networks need a fixed fraction of all data, exponential in the input dimension; deep feature-learning CNNs learn from about n_c m^L examples, polynomial in it, which is where hidden layers become invariant to swapping synonyms and where the class statistics of input patches, equal for synonyms, emerge from sampling noise. Without those correlations deep networks fail too. The law is measured; the link to correlations is a scaling argument plus a one-step-gradient model. | Active — read 2026-10-09 (NOTE-677); worth reading as the paper that introduces the Random Hierarchy Model, a random context-free grammar of fixed depth L and branching s in which each class and each higher-level feature has m interchangeable (“synonymic”) lower-level representations, and as a measured and partly derived account of how many examples a deep network needs to learn it. Two-layer networks and lazy (kernel-regime) deep networks need a fixed fraction of all the data, which is exponential in the input dimension d = s^L; deep convolutional networks in a feature-learning parametrisation need P* ≈ n_c m^L, a power of d. That number is measured, not proved. It coincides with the training-set size at which the networks’ hidden layers become insensitive to swapping synonyms, and with the size at which the class-conditional frequencies of input s-tuples, which are equal for synonyms, rise above sampling noise; a one-step gradient argument for a two-layer network on one patch links the last two. Removing the input–label correlations, with the hierarchy kept, makes deep networks fail again. The hierarchy is one of constituency (parts composing wholes), not of attributes implying attributes. |
| LIT-880 | How Deep Networks Learn Sparse and Hierarchical Data: the Sparse Random Hierarchy Model | Tomasini and Wyart (2024; ICML 2024, PMLR 235). Pads each informative feature of the Random Hierarchy Model with uninformative positions, so the label is unchanged by discrete displacements of features, a stand-in for smooth deformations. Deep networks learn this sparse model from a number of examples polynomial in the input dimension, measured as (s0 + 1)^L n_c m^L (times s^(L/2)) without weight sharing and (s0 + 1)^2 n_c m^L with it, and become insensitive to synonym swaps and to feature displacements at the same training-set size as they learn the task, which the paper offers as the reason deformation stability tracks test error. Fitted laws; the image case is by analogy. | Active — read 2026-10-09 (NOTE-684); worth reading as the sparse extension of the Random Hierarchy Model (LIT-877) and as a model-based account of why a network’s insensitivity to small smooth deformations of its input tracks its test error. Each informative feature of the RHM is padded with s0 uninformative “empty” positions and may sit anywhere in its sub-patch, so the data become very sparse (s^L informative positions among (s(s0 + 1))^L) and the label is unchanged by discrete displacements of the informative features, a stand-in for diffeomorphisms of an image. On this Sparse Random Hierarchy Model, deep locally connected and convolutional networks learn from a number of examples polynomial in the input dimension: measured, P* ≈ s^(L/2)(s0 + 1)^L n_c m^L without weight sharing and P* ≈ C(s0 + 1)^2 n_c m^L with it. At about that training-set size the hidden layers become insensitive both to swapping synonymous tuples and to displacing features, so the two invariances and good performance arrive together; the paper offers this as the explanation of the correlation between deformation stability and test error measured on CIFAR-10. The laws are fitted, the (s0 + 1)^L factor has a one-step-gradient argument for locally connected networks, the CNN’s (s0 + 1)^2 is unexplained, and the image case is an analogy: no synonym sensitivity was measured on images. |
| LIT-883 | Towards a theory of how the structure of language is acquired by deep neural networks | Cagnetta and Wyart (2024), NeurIPS 2024. In sequences generated by the Random Hierarchy Model, the correlation between two tokens shrinks by a factor of about m per level of their lowest common ancestor, and a sample of P sequences resolves it only above a noise floor of (v²P)^(−1/2), so a finite training set acts as an effective context window t*(P). Deep next-token learners show loss steps at P_ℓ ≈ v m^(2ℓ−1), where their representations come to encode the level-ℓ hidden symbol, and the steps add up to a power law. On character-level text the measured correlation decay predicts where the loss for a given context length levels off. | Active — read 2026-10-09 (NOTE-687); worth reading as the self-supervised sequel to the Random Hierarchy Model (LIT-877), and the first of that line in which the latent hierarchy is read off a co-occurrence matrix of observable tokens. For sequences generated by the RHM, the covariance between two token positions has zero mean over draws of the grammar and a root-mean-square size set by the depth ℓ of their lowest common ancestor, about ((1 − m/v^(s−1))/(v³ m^(2ℓ−1)))^(1/2) (derived for large v and m); the estimate from P samples carries noise of size (v²P)^(−1/2) (proved by a central-limit argument), so a finite training set resolves correlations only out to an effective distance t*(P). Deep transformers and CNNs trained to predict the last token show loss steps at about P_ℓ = v m^(2ℓ−1), where their hidden layers become insensitive to resampling the subtree below a level-ℓ symbol. That the networks use the correlations to build the hidden variables is argued, not proved. On character-level Shakespeare and WikiText-103 the measured correlation decays as t^(−β), and loss curves for different context lengths collapse under the t*(P) ~ P^(1/(2β)) it implies. No embedding, PMI or linear direction is studied. |
| LIT-887 v2 | Why does deep and cheap learning work so well? | Lin, Tegmark and Rolnick (2016; Journal of Statistical Physics 168, 1223–1247, 2017). Argues that networks are cheap because physical data have low-order, local, symmetric Hamiltonians, and deep because data come from Markov generative hierarchies whose minimal sufficient statistics compose. Proves that a single hidden layer needs exactly 2^n neurons to multiply n inputs, against about 4n in a deep tree, and gives linear no-flattening examples. Holds that RG is supervised feature extraction, not unsupervised learning; arXiv v1–v2 carried a counterexample against LIT-882 that hit a typo in its Eq. 8, and was dropped later. | Active — read 2026-10-09 (NOTE-688); worth reading as the third position in the dispute over whether deep learning performs renormalization, and for two results that stand on their own. On the dispute it holds that RG is “essentially a feature extractor for supervised learning”: a coarse-graining is a renormalization only once the variables to keep are specified, and fitting the data distribution does not specify them. That holds, and it agrees with what LIT-882 concedes and LIT-873 shows; its sharper form was in an appendix of arXiv v1 and v2 only, dropped from v3 on and from the journal text, showing that a matched partition function does not give a matched distribution and (v2) that the exact trace condition is met by hidden variables that do not interact with the system at all. Its conclusion that RG has “little to do” with unsupervised learning is too strong: LIT-873 gets the relevant variables without labels, from the far environment. On its own merits: a two-variable product needs four neurons and an n-variable product needs exactly 2^n in one hidden layer against about 4n in a deep tree (proved, in a Taylor-matching sense, with weights that grow without bound); the minimal sufficient statistics of a Markov generative hierarchy compose level by level (proved, and nearly immediate); that physics favours low-order, local, symmetric Hamiltonians, and that this is why cheap networks work, is argued, not shown. |
| LIT-888 | Deriving Neural Scaling Laws from the statistics of natural language | Cagnetta, Raventós, Ganguli and Wyart (2026), ICML 2026. The data-limited scaling exponent of a language model’s test loss is predicted from two measured statistics of the corpus: β, the decay of token–token covariance with lag, which sets the usable context n*(P) ~ P^(1/(2β)), and γ, the decay of next-token conditional entropy with context length. When learning within that context is fast, the loss falls as P^(−γ/(2β)); on TinyStories and WikiText-103 transformers match the predicted 0.185 and 0.141 up to about 10^8 tokens. | Active — read 2026-10-09 (NOTE-689); worth reading as the sequel to LIT-883 that takes its effective-context argument from character-level toy runs to subword-tokenised text and a predicted exponent. Two statistics of a corpus are measured: β, the decay with lag n of the largest singular value of the token–token covariance matrix (‖C(n)‖op ~ n^(−β)), and γ, the decay of the next-token conditional entropy with context length (H_n − H∞ ~ n^(−γ), a differential form of Hilberg’s hypothesis). A signal-to-noise argument gives the horizon n*(P) ~ P^(1/(2β)) a sample of P tokens can use; if learning within that horizon is fast, the test loss falls as P^(−γ/(2β)), and in general as P^(−min(δ, γ/(2β))), with δ the decay of the within-horizon excess loss (App. A, an ansatz-based derivation, not a theorem). Measured on TinyStories (β ≈ 0.88, γ ≈ 0.325, predicted α ≈ 0.185) and WikiText-103 (β ≈ 0.94, γ ≈ 0.265, α ≈ 0.141) with GPT-2- and LLaMA-style transformers up to about 10^8 tokens; the empirical exponents fall inside the predicted intervals and the n-gram losses collapse under P/n^(2β), L_n n^γ. γ is read off the n-gram losses of the largest trained models, not computed from the text alone. Usable horizons reached are a few tens of tokens. No embedding, PMI or hierarchy is studied. |
| LIT-890 | The power of deeper networks for expressing natural functions | Rolnick and Tegmark (2017; ICLR 2018). Extends LIT-887’s 2^n result: one hidden layer of smooth, bias-free units needs exactly ∏(rᵢ+1) neurons to Taylor-approximate x₁^r₁⋯xₙ^rₙ, a deep network O(Σ log rᵢ); sparse polynomials inherit the gap up to a factor 1/c; at depth k a product of n inputs costs at most about 2^(n^(1/k)), conjectured tight. The uniform-approximation lower bounds rest on an unproved step, and the gap grows with degree, not with the number of inputs. | Active — read 2026-10-09 (NOTE-693); worth reading as the sequel that turns LIT-887’s one depth-separation theorem into a family, and for where the family stops. Proved, for networks without biases and smooth activations with nonzero Taylor coefficients: one hidden layer needs exactly ∏(rᵢ+1) neurons to Taylor-approximate the monomial x₁^r₁⋯xₙ^rₙ, against Σ(7⌈log₂ rᵢ⌉+4) for a deep network; a sparse polynomial with c monomials costs at least 1/c of its dearest monomial; xᵈ costs d+1 shallow against 7⌈log₂ d⌉ deep; and a product of n inputs costs O(n^((k−1)/k)·2^(n^(1/k))) at depth k. The move to uniform approximation, its headline over LIT-887, rests for the lower bounds on a step the proof asserts and that is false in general (a network can be uniformly within ε of a polynomial while its Taylor coefficients diverge), so only the Taylor-sense lower bounds are established. That depth k needs 2^Θ(n^(1/k)) neurons is a conjecture, backed by one trained-network figure at n = 20 with tanh, outside the theorems’ hypotheses. The gap is exponential in a monomial’s degree, not in the number of inputs: polynomials of bounded degree, such as the low-order Hamiltonians LIT-887 calls natural, show none. |
| LIT-940 v2 | Optimal scaling laws in learning hierarchical multi-index models | Defilippis, Krzakala, Loureiro and Maillard (2026). Take Gaussian inputs, n = αd samples, and a hierarchical multi-index target with even links and weights a_k ~ k^(−γ). Direction k becomes weakly recoverable at α_k = Θ(a_k^(−2)), so features emerge in sequence as the sample size grows, through sharp spectral transitions. The Bayes-optimal subspace error is Θ(α^(−1+1/(2γ))) until α ≍ m⋆^(2γ), then Θ(m⋆/α). A target-agnostic spectral estimator attains it. So does a two-layer network built on that estimator with a ridge readout. No training trajectory is analysed. | Active — read 2026-10-14 (NOTE-805) in full; worth reading as an exact theory of when a feature becomes learnable. Take Gaussian inputs and a target Σ a_k g_k(⟨w_k, x⟩) with even links and a_k ~ k^(−γ). Direction k becomes weakly recoverable at n = Θ(a_k^(−2) d), so features emerge one by one as spikes leaving a random-matrix bulk. A target-agnostic spectral estimator attains the Bayes-optimal rates, Θ(α^(−1+1/(2γ))) and then Θ(m⋆/α). Read it knowing that the cascade is in the number of samples, not in training time. The link to gradient descent is a remark citing other work. The network theorem is for a spectral first layer plus a ridge readout on split data. Held as THEORY-283. |
| LIT-941 v2 | A Theory of Unsupervised Translation Motivated by Understanding Animal Communication | Goldwasser et al. (2022; NeurIPS 2023). A theoretical framework for unsupervised translation with no parallel data and no assumption that the corpora share domain or structure: maximum likelihood against a prior that scores how plausible a target text is. In two stylised random models it proves error bounds that fall with common ground, source-language complexity and samples, with a matching lower bound in one, err = O(log|Θ| / (α·min(m,|T|))), and checks them on synthetic data. Information-theoretic, lossless and uniform. | Active — read 2026-10-14 (NOTE-804) in full; worth reading as the first theory of translation without parallel data: maximum likelihood against a plausibility prior learns the translator with an Occam-type bound divided by the common ground, err = O(log|Θ| / (α·min(m,|T|))) in its common-nonsense model, tight up to a constant. Read it knowing that its models are uniform, random and lossless, that the bounds are information-theoretic, and that the prior substitutes for parallel data in learning a decoder, not for information in a message. Filed THEORY-270; bears on QUESTION-020 without answering it. |
| LIT-943 v2 | From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networks | Dominé, Anguita, Proca, Braun, Kunin, Mediano & Saxe (2025), ICLR. For a two-layer linear network with whitened inputs, trained by full-batch gradient flow from a λ-balanced start, W₂ᵀW₂ − W₁W₁ᵀ = λI, the dynamics of the network function, representations and finite-width NTK are solved exactly. The solution is a matrix Riccati equation. The relative scale λ is conserved and sets the regime. As λ → 0 learning is rich: sigmoidal, stage-like, with task-specific representations. As |λ| → ∞ it is lazy: exponential, with a static NTK. Its sign, together with the architecture, decides which layer stays task-agnostic, and in funnel networks it gives a delayed rich phase. Wide LeCun-initialized networks are λ-balanced in probability. | Active — read 2026-10-13 (NOTE-715): main text in full, and the statement of Appendix A.3 (random initializations are λ-balanced) without its proof; the proofs in Appendices B–C not checked. Its control parameter is the relative scale of the layers, not the absolute scale the seed entry named. The account filed from it is THEORY-212, and the question it raises under SGD is QUESTION-081. |
| LIT-951 v2 | Physics of Skill Learning | Liu, Liu, Michaud, Gore and Tegmark (2025, preprint). In a toy model of independent skills, the Geometry model, gradient descent learns orthogonal skills together, each in time ∝ 1/p. Sign descent makes them compete for coordinate signs, so each starts as the one before finishes (the Domino effect), and the n-th takes time ∝ n. A one-parameter Resource ODE fits this and conserves u_i^(1/p_i). In a small MLP, a parity built from two others waits for them. The evidence is simulation and small experiments. | Active — read 2026-10-14 (NOTE-794) in full; worth reading for one contrast in a toy model of independent skills, the Geometry model. Under gradient descent, orthogonal skills are learned together, each in time ∝ 1/p. Under sign descent they compete for coordinate signs, and each starts as the one before finishes. That is the Domino effect, and it makes the n-th skill take time ∝ n. A separate MLP experiment shows a second cause of sequential onset, dependency between skills. Read it knowing that the Domino effect is shown in simulation, that the Resource and Domino models are fitted to it rather than derived, that the order is frequency order under every optimizer, and that the real-network evidence is a few small MLPs trained with Adam. Held as THEORY-253. |
| LIT-952 v2 | Neural Networks Learn Statistics of Increasing Complexity | Belrose, Pope, Quirke, Mallen, Dury and Fern (ICML 2024). Vision networks trained on real images, probed during training with images whose class means or covariances are grafted from another class and with maximum-entropy samples keeping only those statistics, act on them early (peaks between about 2^4 and 2^12 steps) and lose that later. First-order sensitivity peaks before second-order. Pythia fits unigram-sampled text before bigram-sampled text. The n-gram/moment equivalence is an identity; the ordering is empirical, for orders one and two. | Active — read 2026-10-14 (NOTE-777) in full; worth reading for its probes of what a network is using at a given checkpoint, which are maximum-entropy samples and optimal-transport grafts of class statistics, and for the result they give. Vision networks act on class means and covariances early in training and lose that later, with first-order sensitivity peaking before second-order. Pythia fits unigram text before bigram text. Read it knowing that the ordering is empirical and covers orders one and two, that the Taylor argument for it is a heuristic, and that its n-gram/moment theorem is an identity of one-hot indicators. Held as THEORY-279. |
| LIT-954 v2 | The semantic landscape paradigm for neural networks | Gokhale (2023). A conceptual proposal with no experiment and no proof. Training is pictured as a walk on a graph whose nodes are the algorithms (“heuristic models”) a network can implement, ranked by a fallibility F, the KL divergence from the truth over all inputs. Grokking’s critical dataset size and emergence with scale are argued to be connectivity thresholds of that graph, called percolation; power-law scaling in training time, data and size is derived from an assumed trap model with exponentially distributed barriers and F ∝ 1/N_b. Thresholds and scaling laws are two kinds of event in one vocabulary. | Active — read 2026-10-14 (NOTE-807) in full; worth reading only as a vocabulary: training pictured as a walk on a graph of the algorithms a network can implement, with thresholds as connectivity changes and scaling laws as trap-model waiting times. Read it knowing that it has no experiment and no proof, that its percolation is asserted without a percolation model, and that it treats thresholds and scaling laws as two kinds of event, not one. For QUESTION-039 it is a vote for several kinds in a common language; THEORY-181 holds a sharper account of grokking’s critical data size. No THEORY filed. |
| LIT-964 v2 | The reusable holdout: Preserving validity in adaptive data analysis | Dwork et al. (2015). Adaptive reuse of data inflates false discoveries. By the Supplementary Material: if the max-information between the data and the function an analyst picks is at most log₂e·τ²n, that function’s empirical mean exceeds its true mean by τ with probability at most e^{−τ²n}, and ε-differential privacy bounds max-information by log₂e·εn. Thresholdout answers m adaptively chosen [0, 1]-valued queries on a holdout to accuracy τ with probability 1 − β for n = O(ln(m/β)/τ²·min{B, √(B ln(m/β))/τ}), B the budget of detected overfits. The main text was not read. | Active — read 2026-10-14 (NOTE-848) in part; worth reading for QUESTION-040 as a high-probability bound on overfitting under adaptive reuse: bounded max-information between the data and the analyst’s choices bounds the chance a chosen statistic overfits, differential privacy bounds max-information, and Thresholdout validates m adaptively chosen statistical queries with a holdout growing as ln m. Read it knowing that only the Supplementary Material was reached, not the main text, that four of its six bounds are proved in companion papers, and that it covers [0, 1]-valued queries on a discrete universe. |
| LIT-970 v2 | Symmetry in Neural Network Parameter Spaces | Zhao, Walters & Yu (2025), TMLR. A survey of parameter-space symmetries: transformations of a network’s parameters that leave its function, or its loss, unchanged. It catalogues them for common components (linear layers, homogeneous and other activations, normalization, softmax, attention, LoRA). It surveys when they are complete, so that parameters are identifiable up to them. Continuous symmetries stretch minima into connected manifolds and discrete ones replicate them. Under gradient flow, continuous symmetries give conserved quantities, such as layer imbalance, that label trajectories and locate the endpoint among the minima. Finite steps, gradient noise and weight decay break these conservation laws. | Active — read 2026-10-13 (NOTE-712): §§1–5 and 7–8 in full, §6 in part. A survey, so its claims about the works it cites are secondhand here. It bears on QUESTION-058 and QUESTION-069, and on the null space that THEORY-209 leaves out. The account filed from it is THEORY-210. |
| LIT-971 v2 | Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data | Gerstgrasser et al. (2024). Small language models (9M–126M) and a molecular diffusion model trained for five to eight rounds on their predecessors’ samples degrade when each round’s data replace the last, and hold their test loss when the samples accumulate alongside the real data; a VAE’s error still rises under accumulation, more slowly. In a fixed-design linear regression it proves that accumulation gives test error σ²d/(T−d−1)·Σ1/i², bounded by π²/6 times the first round’s, where replacement gives σ²d/(T−d−1)·n. The error is bounded, not constant. | Active — read 2026-10-14 (NOTE-815) in full; worth reading for QUESTION-016 as an exact, bounded case of degradation along a reconstruction chain: in a fixed-design linear regression refit to its own labels, keeping all earlier data gives test error σ²d/(T−d−1)·Σ1/i², below π²/6 times the first fit’s, where replacing the data gives linear growth. Read it knowing that the error still rises every round, that what is retained is the source sample in the training pool, not an interpreter’s side information, and that one squared-error task is measured, not sufficiency. Filed as THEORY-289. |
| LIT-974 v2 | Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path | Han, Papyan & Donoho (2021; ICLR 2022). Networks trained with MSE loss show neural collapse, on five datasets with VGG, ResNet and DenseNet. The MSE loss splits exactly into a term where the classifier is the least-squares classifier for the current features and a non-negative deviation term. The deviation is measured to be negligible from early in training: the “central path”. On that path, for unconstrained features with no weight decay and re-whitened to identity within-class covariance after every step, gradient flow gives closed-form dynamics. Every singular value of Σ_W^(−1/2)M̃ grows as (4t/N)^(1/4), and they tend to equality, which yields NC1–NC4. | Active — read 2026-10-14 (NOTE-783) in full; worth reading as the record’s one derivation of neural-collapse dynamics in closed form. Under MSE loss, a trained classifier stays near the least-squares classifier for its own features (measured), and on that central path a whitened gradient flow of unconstrained features drives every class-separation singular value to infinity as (4t/N)^(1/4) and to equality, which is NC1 and NC2. It also measures collapse under MSE on five datasets and three architectures. Read it knowing that the flow is not gradient descent (features are re-whitened after every step), that the theory has no weight decay while every experiment has it, that it is MSE and not cross-entropy, and that the experiments are LIT-618’s protocol with one run per cell. |
| LIT-975 v3 | Contextures: The Mechanism of Representation Learning | Zhai (2025), CMU PhD dissertation. A representation is learned from the joint law of an input X and a context variable A. The top-d left singular functions of the operator g ↦ E[g(A)|X], which runs between two different spaces and need not be symmetric, are proved to attain the minimum worst-case linear-probe approximation error over the tasks compatible with the context. The thesis says they are the unique such span, which is false (THEORY-239). Several squared-error pretraining objectives reach that span at their optimum. That a context is most useful at moderate association, and that scaling model size gives diminishing returns, are argued and measured on small datasets, not proved. | Active — read 2026-10-14 (NOTE-770) in full; worth reading for QUESTION-052 as the singular-function result for a pair operator that need not be symmetric: for any joint law of an input and a context, the top left singular functions of the conditional-expectation operator attain the minimum worst-case linear-probe approximation error over compatible tasks (Theorem 3.4), though, contrary to the thesis, they are not the unique span that does (THEORY-239). Read it knowing that every result is at the population optimum of squared-error objectives, that its only graph example is undirected, and that its claims about scaling and about moderate association are argued from a few experiments, not proved. |
| LIT-978 v3 | Neural Collapse: A Review on Modelling Principles and Generalization | Kothapalli (2022; TMLR 2023). A survey of neural collapse in one notation. It reviews the unconstrained-features model and local elasticity as ways of modelling how training reaches the collapsed state. Every derivation it covers works with features cut loose from the data. On generalization it finds the literature in conflict: collapse on the training set does not fix test error, and collapse on test data is weaker and depends on the dataset. No agreed threshold says when a network is collapsed. It reports Kornblith et al.’s finding that more collapsed ImageNet networks transfer worse. Skimmed. | Active — read 2026-10-14 (NOTE-772) in part; worth reading as the map of the first two years of models of neural collapse, sorted into unconstrained-features models and local elasticity, with their assumptions side by side, and as the place where the conflicting claims about collapse and generalization are set next to each other: collapse on the training set reached by different optimizers with different test error, weaker and dataset-dependent collapse on test data, and worse transfer from more collapsed ImageNet networks. Read it knowing that it is a survey whose results are all second-hand, that every derivation it covers removes the data, and that it predates the imbalance geometry of LIT-994 and the kernel analysis of LIT-985. Skimmed: the equations were not checked. |
| LIT-984 v2 | Data Mixing Can Induce Phase Transitions in Knowledge Acquisition | Gu et al. (2025), NeurIPS 2025. Synthetic biographies mixed into web text are recalled almost not at all below a critical model size and below a critical mixing ratio, and 4–16 times more training barely helps; past the threshold recall rises steeply. An idealised learner with capacity measured as mutual information explains it: such a learner takes the facts only once the web data’s marginal loss reduction per bit falls below theirs (Theorem 4.3). If the web loss is a power law in size, the critical size scales as (1/rp)^(1/(α+1)). Pythia models from 14M to 6.9B; the link from optimal learner to trained network is assumed. | Active — read 2026-10-14 (NOTE-796) in full; worth reading for QUESTION-039 as a threshold of a different kind from the record’s others: whether a set of facts is learned at all is set by competition for capacity with the rest of the data, through the ratio of their marginal loss reductions, not by the facts’ own statistics. Read it knowing that the theorem is about an idealised optimal learner, says nothing about when in training the facts arrive, and predicts an onset over a width of Htot rather than a jump; that trained networks follow it rests on the shape of the curves and one exponent. Its practical advice belongs to the anthology’s reading, ANTH-LIT-451. Filed as THEORY-293. |
| LIT-985 v2 | Can Kernel Methods Explain How the Data Affects Neural Collapse? | Kothapalli and Tirer (2024; TMLR 2025). The paper writes NC1, as tr Σ_W / tr Σ_B, for the features of any kernel. It proves, for one-dimensional two-class Gaussian data on opposite sides of the origin with ReLU and zero bias, that the NTK of a wide two-layer net has the same expected NC1 as the NNGP kernel at initialization. In higher dimensions it measures the NTK no more collapsed, and with ReLU less. A data-aware adaptive kernel lowers NC1 and approximates finite networks on separable data. It departs from them under class overlap in high dimension and under imbalance. Erf activation gives lower NC1 than ReLU. | Active — read 2026-10-14 (NOTE-774) in full; worth reading as the record’s one account of neural collapse in which the data enter, and for its negative result. In the case it can solve, lazy-regime training of a wide two-layer net leaves the within-class variability of the features exactly where initialization put it, because there the NTK is a within-class rescaling of the NNGP kernel. It also gives NC1 as a closed function of any kernel matrix. Read it knowing that the theorem covers one-dimensional data with two classes on opposite sides of the origin and zero bias, that the higher-dimensional claim is simulation (and for ReLU the NTK is less collapsed, not equal), that its NC1 is tr Σ_W / tr Σ_B on shallow MSE-trained binary nets, and that the data-aware kernel it proposes fails under class overlap and imbalance. Its printed Eq. 16 appears to keep a cross-class term the kernel sets to zero. |
| LIT-994 v2 | Imbalance Trouble: Revisiting Neural-Collapse Geometry | Thrampoulidis, Kini, Vakilian & Behnia (2022, NeurIPS). In the unconstrained-features model under STEP class imbalance, every solution of the max-margin problem, the limit of cross-entropy with vanishing regularization, has logits equal to the simplex-encoded label matrix Ẑ. Its classifier and embedding Gram matrices come from Ẑ’s SVD (SELI). That is the simplex ETF only for balanced or binary data. With any finite ridge the geometry differs and depends on its strength; it tends to SELI as the ridge vanishes. In the unconstrained-features model and in ResNet-18 and VGG-13 runs, training ends closer to SELI than to the ETF, not at it, and the approach slows as imbalance grows. | Active — read 2026-10-14 (NOTE-764) in full; worth reading as the statement of what neural collapse becomes under class imbalance. In the unconstrained-features max-margin problem, the logits equal the simplex-encoded labels at any imbalance. The classifier and embedding geometries come from that matrix’s SVD, and they are the simplex ETF only for balanced or binary data. Majority classifiers come out longer and minority classifiers closer together, and the minority-collapse threshold is given in closed form. Read it knowing that gradient descent is only conjectured to reach this geometry, that under any finite weight decay the geometry differs and depends on its strength, that convergence is slow and slower with imbalance, and that the deep-net runs are one seed per cell with weight decay on. |
| LIT-995 v2 | Deep sequence models tend to memorize geometrically; it is unclear why | Noroozizadeh et al. (2025; ICML 2026). Transformers and Mamba trained on a graph’s edge bigrams embed its nodes so that multi-hop relations are readable, and they solve an in-weights path-finding task built to defeat next-token learning. The authors argue that straightforward supervisory, capacity, optimization and succinctness pressures do not explain why this geometry wins over a lookup the same models can store. In a wide two-layer weight-tied model, unregularized cross-entropy drives the embeddings into the top Laplacian eigenvectors without a rank bottleneck. This is shown empirically, with an informal partial proof. Why deeper models prefer the geometry is left open. | Active — read 2026-10-14 (NOTE-800) in full; worth reading for QUESTION-060 as a case where neither the supervision nor what the model can represent fixes the learned variables. Sequence models trained only on a graph’s edges embed it as a global geometry, although a lookup is expressible, found faster and as succinct. In a two-layer model the geometry is the top Laplacian eigenspace, reached by cross-entropy training alone. Read it knowing that every argument is empirical or informal by the authors’ own account, that the spectral account covers only the two-layer model, and that initialization scale can tip deeper models into a lookup. |
| LIT-1011 | The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion | Kunin, Sagastuy-Brena, Gillespie, Margalit, Tanaka, Ganguli & Yamins (2021). Long after the loss converges, networks trained with SGD keep moving at a constant speed, and their distance from where they were grows as a power law in steps. Modelling SGD with momentum at finite learning rate and batch size as an underdamped Langevin equation, the paper solves linear regression exactly: an Ornstein–Uhlenbeck process whose stationary distribution is a Gibbs distribution on a modified loss, not the training loss, in broken detailed balance, so probability currents drive oscillations in phase space. Under a quadratic approximation, a pre-trained ResNet-18 on ImageNet shows the predicted isotropic exploration, velocity regularization, oscillations and speed. | Active — read 2026-10-13 (NOTE-711): main text, and Appendices A–C and H; the derivations in Appendices D–G were not checked. Held also in the Anthology of the SOTA as ANTH-LIT-454, read there for what the loss curve hides; read here for what a converged run is, under ADR-013. |
| LIT-1012 | Noise Balance and Stationary Distribution of Stochastic Gradient Descent | Ziyin, Li & Ueda (2023; v2 2025). Under the SDE model of SGD with temperature T = η/S, a rescaling symmetry between parameter groups u and w gives d(‖u‖² − ‖w‖²)/dt = −T(uᵀC₁u − wᵀC₂w): the Brownian term vanishes, and minibatch noise drives the quantity to the point where the two groups’ gradient noise balances. Gradient descent conserves it, and gradient descent with isotropic noise diffuses along it. At stationarity SGD therefore differs from gradient descent by a finite amount however small T is, and is not ergodic. For a diagonal linear network of arbitrary depth and width, the paper solves the stationary distribution exactly. It finds a phase transition to the zero solution above a critical T, sign coherence, fluctuation inversion and a tail set by depth alone. | Active — read 2026-10-13 (NOTE-714): §§I–IV in full; the proofs in Appendix A not checked. Cited by LIT-970 as Ziyin et al. (2023a), “Law of balance and stationary distribution of stochastic gradient descent”; retitled in v2. The source of THEORY-211. |
| LIT-1013 | Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics | Kunin, Sagastuy-Brena, Ganguli, Yamins & Tanaka (2021), ICLR. A differentiable symmetry of the loss makes the gradient orthogonal to the symmetry’s generator and constrains the Hessian. Under gradient flow each symmetry then conserves a quantity: translation (softmax) the parameter sum, scale (batchnorm) the norm, and rescale (homogeneous activations) the difference of squared norms. Weight decay and finite learning rates break these laws, and a modified gradient flow gives exact integral expressions for the broken quantities, matched on VGG-16 trained on Tiny ImageNet. Gradient noise is orthogonal to the generators, so it is low-rank along symmetry directions. | Active — read 2026-10-13 (NOTE-713): main text in full, and Appendices F, I and J; Appendices A–E, G and H not checked. The first-hand source for the conservation laws and their breaking that THEORY-210 took from the LIT-970 survey. |
| LIT-1014 | Symmetry Induces Structure and Constraint of Learning | Ziyin (2024), ICML. A loss has an O-mirror symmetry if it is invariant under the reflection I − 2OOᵀ. Rescaling, rotation and permutation symmetries all imply mirror symmetries. Every such symmetry makes Oᵀθ = 0 a stationary condition: once there, (S)GD cannot leave. It also splits the Hessian into blocks. Above a threshold weight decay, all minima satisfy the constraint (Theorem 1). Strong gradient noise makes it attractive even at a saddle (§2.4). As corollaries, rescaling gives sparsity, rotation gives low rank, and permutation gives identical, merged neurons. The paper ties these to collapse phenomena and to loss of plasticity. | Active — read 2026-10-13 (NOTE-716): main text in full; the proofs in Appendix C and the experimental details in Appendix B not checked. Cited by LIT-970 as Ziyin (2024). The source of THEORY-213. |
| LIT-1044 v2 | Why Do Better Loss Functions Lead to Less Transferable Features? | Kornblith, Chen, Lee & Norouzi (2020; NeurIPS 2021). Nine training objectives for ResNet-50 on ImageNet: seven beat softmax cross-entropy significantly, by up to 0.9 points top-1, and all eight alternatives yield fixed penultimate features that transfer worse to eight new-class datasets, by linear and k-NN readout. Each raises penultimate-layer class separation R² (one minus within-class over total cosine distance), and mean linear transfer falls with R² across losses (Spearman ρ = −0.93) and along a cosine-softmax temperature sweep. The losses differ only in the last third of the network, and full fine-tuning removes the gap. | Active — read 2026-10-14 (NOTE-762) in full; worth reading as the record’s one direct measurement of what class separation in the last layers costs off the trained task. Across nine objectives for ResNet-50 on ImageNet, those that beat softmax cross-entropy all raise penultimate-layer class separation (R² 0.349 → up to 0.845), and mean linear transfer to eight new-class datasets falls with it (Spearman ρ = −0.93; 75.9% for softmax, 48.3% for squared error). Cosine-softmax temperature moves both on one knob. The losses differ only in the last third of the network, and full fine-tuning erases the transfer gap. Read it knowing that the networks are early-stopped at 91.6–93.6% training accuracy, so this is relative within-class variance and not terminal-phase neural collapse; that the cost is for new classes, while on shifted ImageNet test sets with the same classes the more separated losses do slightly better (Table C.2); and that it is one dataset, two architectures and a correlation across losses, with one within-loss sweep. Filed as THEORY-246. |
| LIT-1052 | Calibration for the (Computationally-Identifiable) Masses | Hébert-Johnson, Kim, Reingold and Rothblum (2018), ICML; arXiv 1711.08513. A predictor is α-multicalibrated on a collection C of subpopulations if, within every set S in C, the individuals given (roughly) the same prediction v have true probabilities averaging within α of v, except on an α-fraction of S. For any C whose sets have size at least γN, a simple iterative patching algorithm reaches multicalibration from O(log|C|/α^{11/2}γ^{3/2}) samples in time linear in |C|, and the result is a circuit only O(s/α⁴γ) in size when the sets of C are computed by size-s circuits. Learning it is equivalent, up to polynomial factors, to weak agnostic learning of C, so it is hard for rich C under cryptographic assumptions. Calibrating any predictor h on its own level sets as well as on C costs at most 6α in squared error. All guarantees are relative to the benchmark p*, which is itself defined relative to the features available, and they hold only for large sets; the paper offers no uniqueness for the predictor and does not state a theorem in indistinguishability form. | Active — read in full 2026-10-15 from the authors’ arXiv preprint, latest version v2 of 16 March 2018 (arxiv.org/pdf/1711.08513, 40 pp.); the ICML version (PMLR 80:1939–1948, proceedings.mlr.press/v80/hebert-johnson18a/hebert-johnson18a.pdf, with a supplement) was fetched and its definitions and theorem statements compared, not read in full. Worth reading as the source definition of multicalibration: calibration required on every set in a collection C of large, efficiently identifiable subpopulations; a patching algorithm that reaches it from O(log|C|/α^{11/2}γ^{3/2}) samples in time linear in |C|, with a circuit of size O(s/α⁴γ); learning it equivalent to weak agnostic learning of C, and so hard for rich C under cryptographic assumptions; and calibration on any predictor’s level sets at almost no cost in squared error. Its multi-AE (accuracy in expectation on every S in C) is what later work calls multiaccuracy; the paper does not use that word. For QUESTION-103 it supplies adequacy relative to a class of observers in correlation form (a violation is a set in C correlating with the residual), but it states no indistinguishability theorem, no distinguishing advantage and nothing from signal detection; that framing is Dwork et al.’s (LIT-1068). For CLAIM-152 it is consistent and adds a second relativity: adequacy is relative to the outcome, to the features through p*, and to the class C, and even with all three fixed the adequate predictor is not unique. For QUESTION-104 it gives a formal instance of budget-relative sufficiency in which representing an adequate predictor is cheap (only modestly larger than the observer class) while finding one can be hard, but it has no notion of an optimal partition and does not show one changing with the budget. |
| LIT-1063 | Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters | Snell, Lee, Xu and Kumar (2024, arXiv 2408.03314; ICLR 2025). On MATH with PaLM 2-S*, which way of spending test-time compute works best (beam search against a process reward model versus best-of-N; sequential self-revision versus parallel sampling) depends on how hard the question is for the base model, and choosing the method per difficulty bin matches best-of-N with up to about 4× less test-time compute (the discussion says 2–4×). In a FLOPs-matched comparison with a ~14× larger model, the extra test-time compute a small model may spend is set by R, the ratio of total inference tokens to pretraining tokens: when R is small the larger model’s pretraining cost buys many samples per query and test-time compute wins on easy and medium questions; when R is large the budget per query falls toward the parameter ratio and the larger model wins, always on the hardest questions. Difficulty estimation (2048 samples per question) and verifier and revision-model finetuning are not counted in the compute, R is an aggregate token ratio with no term for which queries recur, and the evidence is one model family on one benchmark of 500 test questions. | Active — read in full 2026-10-15 from the arXiv preprint (arxiv.org/pdf/2408.03314, v1 of 2024-08-06, the only arXiv version) and from the ICLR 2025 conference version, open access in the ICLR proceedings (proceedings.iclr.cc, retitled ”… than Scaling Parameters for Reasoning”), which is a revision not posted to arXiv: it charges verifier calls to inference FLOPs, which halves the stated R values (0.16/0.79/22 become 0.08/0.40/11) at unchanged budgets, and adds a verifier-quality check. Worth reading as the standard empirical case that the best use of inference compute depends on how hard a question is for the base model, and that choosing the method per difficulty bin matches best-of-N with up to about 4× less test-time compute on MATH with PaLM 2-S*, before counting the 2048-sample difficulty estimate. For QUESTION-095 it shows a FLOPs-matched amortization model: a ~14× larger model’s extra pretraining is a fixed cost spread over all inference, so the per-query test-time budget it buys scales as M + 3(M − 1)/R, with R the ratio of inference to pretraining tokens; test-time compute wins at low R on easy and medium questions and loses at high R and on the hardest questions at every R. R is aggregate inference volume, not a count of recurring queries; the larger model’s per-token cost is a retrieval-cost term; query predictability appears as difficulty but its cost is not counted; staleness and erosion are absent. |
| LIT-1068 | Outcome Indistinguishability | Dwork, Kim, Reingold, Rothblum and Yona (2021), STOC; arXiv 2011.13426. A predictor p̃ is outcome indistinguishable (OI) relative to a class A of distinguishers when no A ∈ A can tell an individual with Nature’s outcome from the same individual with an outcome drawn from Ber(p̃_i). The paper sets out four levels by how much access the distinguisher has to p̃: none, the prediction on the sampled individual, oracle queries, or the code. The lower two levels are equivalent to multi-accuracy and multi-calibration over closely related classes. At those levels OI predictors exist with size O(s/ε²), whatever Nature’s complexity. At the oracle level, under fine-grained hardness of clique counting or BPP ≠ PSPACE, some efficient distinguisher classes force any OI predictor to be nearly as expensive as Nature’s own predictor. That gives a strict separation between oracle access and sample access. OI never requires p̃ ≈ p*. A predictor is good only relative to A, and with the paper’s construction a weak A gives a coarse predictor that a richer A would refute. The paper covers binary outcomes, a fixed representation of individuals and single samples (multi-sample OI is left mostly to future work). | Active — read in full 2026-10-15 from the arXiv preprint, arXiv:2011.13426v1 (arxiv.org/pdf/2011.13426, 61 pp.), the only arXiv version. The STOC 2021 version of record (doi:10.1145/3406325.3451064) was not read; its only open copies are submitted versions (arXiv, MIT DSpace). The recurrence algebra of Theorems 5.9–5.10 and the proof of Proposition 6.3 were checked for structure only. Worth reading as the source of outcome indistinguishability. A predictor is adequate relative to a class A of distinguishers when none can tell its sampled outcomes from Nature’s. The paper sets out four levels by the distinguisher’s access to the predictor (none, the sampled prediction, oracle queries, the code). It proves the lower two equivalent to multi-accuracy and multi-calibration and gives OI predictors of size O(s/ε²) whatever Nature’s complexity. At the oracle level it gives conditional hardness (clique counting, or BPP ≠ PSPACE): cheap distinguishers with query access can force any OI predictor to be about as costly as Nature’s own. For QUESTION-103 it supplies the formal half: distinguishing advantage is the ROC quantity, and with all subset distinguishers it becomes statistical distance. It says nothing about SDT, criteria or ideal-observer efficiency, and adds that access as well as budget sets an observer’s power. For CLAIM-152 it is an instance of adequacy relative to the observer class with the target held fixed, with no uniqueness or minimality of the predictor. For QUESTION-104 it shows a restricted class accepting a coarser model (p̃ ≡ 1/2 where Nature’s two halves are computationally indistinguishable), but at the oracle and code levels a bounded observer need not be satisfied by a cheap model. |
| LIT-1069 | Scaling Scaling Laws with Board Games | Jones (2021), arXiv 2104.03113. Training AlphaZero on Hex at board sizes 3×3 to 9×9, the best Elo reachable for a given training compute follows one change-point curve across sizes: an incline of about 500 Elo per tenfold of training compute, with the compute for perfect play rising about 7× and the compute to leave random play about 4× per added board row, and fits on small boards predicting larger ones. On 9×9 only, train-time and test-time (tree-search) compute substitute at a fixed rate: at equal Elo, each 10× of train-time compute removes about 15× of test-time compute (fit: log test = −1.2·log train + 0.004·Elo + 29), down to a one-node search, over roughly 1e14–1e16 FLOPS-seconds of training. The trade is an iso-performance curve for a single search; the paper never counts how many games or moves the trained agent will play, so it does not price amortization, and it does not state the unit of test-time compute. One game, small boards, Elo anchored to a perfect player, and no uncertainty on the fitted trade-off coefficients. | Active — read in full 2026-10-15 from the author’s arXiv preprint, version 2 of 2021-04-15, the latest (arxiv.org/pdf/2104.03113v2; v1 diffed against it, same methods and numbers). Worth reading as a clean small-scale scaling law in problem size as well as compute: AlphaZero on Hex, boards 3×3 to 9×9, one change-point frontier of about 500 Elo per tenfold of training compute, with the compute for perfect play rising about 7× per added board row and fits on small boards predicting larger ones. For QUESTION-095 it shows that, on 9×9 at fixed Elo, training and tree-search compute substitute along a straight log–log line, about 15× less search per 10× more training (fit: log test = −1.2·log train + 0.004·Elo + 29), over roughly 1e14–1e16 FLOPS-seconds of training and down to a one-node search; it does not count the games or moves the trained agent will play, or define the unit of test-time compute, so it prices the iso-performance substitution but not amortization, and has no terms for query shift, staleness or erosion of search. |
| LIT-1070 | Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Laws | Sardana, Portes, Doubov and Frankle (2024, ICML; arXiv 2401.00448). Fixing a target pre-training loss and adding inference FLOPs (2N per token over a known lifetime demand D_inf) to training FLOPs (6N per token), the compute-minimising model is smaller and trained on more data than Chinchilla-optimal, increasingly so as D_inf approaches and exceeds the training-set size; the optimum has no closed form and is found numerically. The savings are modest in FLOPs (for a 30B-Chinchilla-quality model with 10^13 inference tokens, 28% by training 13.6B on 2.84× the data) and larger in dollars once inference’s low hardware utilisation is priced in (17% to 58% in the paper’s examples). 47 training runs (150M–6B parameters, 10–10,000 tokens per parameter) find loss still falling at extreme ratios, but more slowly than Chinchilla-fitted laws predict, so the formula overstates the gain from long training. Demand is assumed known before training and fixed given quality; there is no term for uncertainty, discounting, obsolescence or a model being replaced. | Active — read in full 2026-10-15 from the authors’ arXiv preprint v3 (arxiv.org/pdf/2401.00448v3, 14 Apr 2025), which is the ICML 2024 paper (PMLR 235, also fetched from proceedings.mlr.press/v235/sardana24a.html and diffed: same equations, tables and numbers, small wording edits); v1 (2023-12-31) is an earlier two-author NeurIPS 2023 workshop version without the experiments. Worth reading as the worked case of choosing how much to precompute by expected use: fixing quality, it adds lifetime inference FLOPs (2N per token) to training FLOPs (6N per token), finds the optimum numerically (no closed form), and shows the optimal model shrinking and training longer as demand grows (30B-Chinchilla quality with 10^13 inference tokens: 13.6B on 2.84× the data, 28% fewer FLOPs; in dollars, with inference at 1% utilisation, savings of 17% to 58%), with 47 training runs up to 10,000 tokens per parameter showing quality still improving but more slowly than the fitted law predicts. For QUESTION-095 it supplies expected reuse and a non-zero per-use cost as explicit terms, and nothing for staleness, uncertainty about demand (assumed known before training) or erosion; for CLAIM-303 it instantiates the break-even n > F/(c_r − c_a − u) with u = 0, and shows the choice as a continuous dial tied by a production function rather than a file-or-not decision. |
| LIT-1071 | Resource-rational analysis: Understanding human cognition as the optimal use of limited computational resources | Lieder and Griffiths (2020; online 2019), BBS 43. Resource rationality defines the right cognitive mechanism as the one, among those a brain can execute, that maximizes expected utility of its result minus the expected opportunity cost of the time and resources it uses (Eq. 3), averaged over the environments consistent with what is known (Eq. 4); resource-rational analysis is a five-step iterate-and-refine method for deriving process models from that objective. The objective is per episode. It has no separate terms for storing a result, retrieving it later, or its going stale: the value of what a computation leaves behind is folded into the utility of the episode’s result, and the cost of finding the heuristic in the first place is assigned to evolution and learning and left out. Reuse appears only in the review, qualitatively: habits as reused action sequences, Dasgupta et al. 2018’s memorized hypotheses, Howes et al. 2016’s memory-load choice (the review’s “memorize-or-look-up”), and “cognitive prostheses”. The framework is offered as a methodological device, not a claim that people are optimal. | Active — read in full 2026-10-15 from the publisher’s typeset PDF of the whole BBS treatment (target article pp. 1–16, Authors’ Response pp. 45–51, and the commentaries of Bates, Sims & Jacobs, Dimov, Dingemanse, and Ma & Woodford; the other 21 commentaries searched by keyword only), posted by the Griffiths lab at cocosci.princeton.edu/papers/lieder_resource.pdf. Unpaywall and OpenAlex list it as closed. Worth reading as the standard statement of resource rationality: the best mechanism a brain can run is the one that maximizes expected utility of its result minus the expected opportunity cost of the time and resources it takes (Eqs. 3–4), and the five-step method that derives process models from that. For QUESTION-095 it shows that one objective can cover computing, remembering and retrieving, and it groups habits, memorized inferences (Dasgupta et al. 2018) and how much to hold in memory (Howes et al. 2016, which the review calls a memorize-or-look-up trade) under it. But the objective is per episode and has no separate terms for precompute, storage, retrieval or staleness. It leaves the cost of finding a heuristic out on purpose, as already paid by evolution and learning. Offloading to the world appears only in Dingemanse’s commentary, which the authors accept as a frontier. For CLAIM-303 it supplies the general benefit-minus-cost form the break-even condition fits, and no support for the claim’s staleness term. |
| LIT-1232 v2 | On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transport | Chizat & Bach (2018). Training a sum of many parameterized units by gradient descent is, in the many-particle limit, a Wasserstein gradient flow on the measure over unit parameters. A measure is a global minimizer iff the first variation of the objective is nonnegative everywhere and zero on its support; under homogeneity and an initialization whose support separates the domain, limits of the flow are global minimizers. NeurIPS 2018. | Active — read 2026-10-29 (NOTE-1367), the main paper in full and the supplement in part, from arXiv 1805.09545v2; worth reading here for its optimality condition on measures, first variation nonnegative everywhere and zero on the support, which is uninvadability, and for its escape argument, in which mass where the first variation is negative grows exponentially. Read it knowing that its theorems assume the flow converges, need an initialization that seeds every direction and so infinitely many particles, and give no rates; the 1-homogeneous proofs and the many-particle-limit proofs were not read. |
| LIT-1235 v2 | What can ecosystems learn? Expanding evolutionary ecology with learning theory | Power et al. (2015). In a competitive Lotka–Volterra community, individual selection lowers species i’s sensitivity to species j at a rate proportional to x_i x_j, which with a normalising trade-off on each species’ total competition is Hebbian learning. Simulated with symmetric interactions, the community comes to hold attractors for two past forcing patterns, completes them from fragments and switches with hysteresis; in the long run the memory degrades. | Active — read 2026-10-29 (NOTE-1368) in full, from the publisher’s version in PubMed Central (PMC4672551) with its erratum; worth reading as the model in which individual selection on competitive interactions acts like Hebbian learning, so that a community with no community-level selection comes to hold and recall several past compositions. Read it knowing that the memory needs an imposed trade-off that fixes each species’ total competition, that simulations keep interactions symmetric, that the two-memory state collapses in the long run, and that its printed equations (1) and (10)–(11) are wrong as printed. |
| LIT-1258 v2 | Exact and Soft Successive Refinement of the Information Bottleneck | Charvin, Catenacci Volpi & Polani (Entropy 2023). Defines successive refinement for the information bottleneck in single-letter form: a coarse optimum T₁ can be extended to a fine optimum (T₁, S). This holds iff some pair of optima forms the chain X → T₂ → T₁, iff the coarse encoder is a garbling of the fine one, and, when the fine decoder is injective, iff the coarse decoder points lie in the convex hull of the fine ones. That last form gives a linear-programming test. Refinability is proved for binary X and Y, for Y a function of X, and (from a cited semigroup result) for jointly Gaussian variables. Numerically, for 3×3 and larger random sources, the hull test on computed optima often fails, especially below the second bifurcation. The loss, measured by unique information, stayed under 5.4% of I(X; T₁, T₂). The failures are not certified, since global optimality and uniqueness of the optima are left as a conjecture. | Active — read 2026-10-20 (NOTE-981) in full, from the published version in the University of Hertfordshire Research Archive; worth reading for the single-letter definition of successive refinement for the information bottleneck and its equivalent forms (Proposition 1: some pair of optimal encoders forms the Markov chain X → T₂ → T₁; Proposition 8: the coarse encoder is a Blackwell garbling of the fine one); for the convex-hull test (Proposition 4: with an injective fine decoder, refinement holds iff every decoder point q(X|t₁) lies in the convex hull of the q(X|t₂)), checkable by a linear programme; for proofs of refinability when X and Y are both binary, when Y is a function of X, and for jointly Gaussian variables (the last resting on Kline and Palmer’s semigroup result); and for soft refinement measured by unique information. Read it knowing that the linear programme tests the particular optima a solver returns, so that a failed test certifies nothing unless the optima are globally optimal and unique, which the paper leaves as Conjecture 1 and calls its own figures “mere numerical proxies” (Appendix D); that its evidence that refinement fails is therefore numerical and uncertified, with no count of how many sampled distributions failed; that the unique information computed is an upper bound on the soft measure for the same reason; that the deep-learning reading of §4.2 is speculative; and that the commentary on Proposition 8 swaps its points (ii) and (iii). |
| LIT-1261 v2 | On the Spectral Bias of Neural Networks | Rahaman et al. (2019; arXiv 2018). Proves, for compactly supported ReLU networks, that the Fourier transform falls as k^(−d−1) in almost every direction and no slower than k^(−2). Shows by experiment that a deep ReLU network fits low frequencies before high ones, even when the high ones are ten times stronger, and that its high-frequency components are the least robust to parameter noise. Shows that on a curve embedded in the plane, high latent frequencies are fitted more easily as the embedding gets longer and more convoluted. | Active — read 2026-10-20 (NOTE-983) in full; worth reading for the experiment in which a ReLU network fits low frequencies before high ones even when the high ones are ten times stronger (THEORY-759), for the Fourier analysis of piecewise-linear networks (THEORY-761), and for the observation that the frequencies that are hard to learn depend on how the data’s manifold is embedded. Read it knowing that its theorem is proved only for compactly supported networks, that the link from the theorem to the order of learning is an upper bound on speed and not a proof of order, that the volume result in Appendix D does not follow, and that its “manifold complexity” is confounded with the curve’s length. |
| LIT-1309 v2 | A Theory of Usable Information Under Computational Constraints | Xu, Zhao, Song, Stewart & Ermon (2020). Predictive V-information, H_V(Y) − H_V(Y | X) with the conditional entropies taken as the least expected log loss over a predictive family V, is the information about Y that a predictor restricted to V can use. It recovers Shannon mutual information when V is unrestricted and the coefficient of determination for linear-Gaussian V, is non-negative and zero under independence, can be asymmetric, can be raised by computing on X, and can be estimated with PAC guarantees from the Rademacher complexity of V. | Active — read 2026-10-21 (NOTE-1048) in full, from arXiv:2002.10689v1, the ICLR 2020 camera-ready; worth reading as the usable-information literature the record lacked: it defines predictive V-information, the information a predictor restricted to a family V can extract, shows that computation can create it (so the data-processing inequality fails for it), and bounds its estimation error by the Rademacher complexity of V (THEORY-846, THEORY-840). Read it knowing that the failure of the data-processing inequality is shown by informal examples (decryption, feature learning), not by a theorem, that V-information is not bounded by Shannon information unless V’s input-free predictions include the true marginal (the record’s counterexample, THEORY-846), that Proposition 1.2’s “mean absolute deviation” is the deviation about the geometric median, and that the experiments are small. |
| LIT-1336 v2 | Performative Prediction | Perdomo, Zrnic, Mendler-Dünner & Hardt (2020). When a deployed prediction changes the distribution of the outcomes it predicts, define performative stability (a model optimal for the distribution it induces) and performative optimality (least risk on the distribution it induces); show they can differ; prove that with a smooth, strongly convex loss and a distribution map ε-Lipschitz in Wasserstein distance, retraining converges to a unique stable point when ε < γ/β, that each condition is needed, and that stable points lie within 2L_zε/γ of optima. | Active — read 2026-10-21 (NOTE-1044) in full, from arXiv:2002.06673v4; worth reading for the split between performative stability (optimal for the distribution one’s own deployment induces, the fixed point of retraining) and performative optimality, for the exact conditions under which retraining converges to a unique stable point (smooth, strongly convex loss and ε < γ/β, each shown necessary; THEORY-861), and for the bound 2L_zε/γ between stable points and optima (THEORY-839). Read it knowing that its conditions hold in its own simulation only at the smallest ε, that the distribution map depends on the current model alone, and that it says nothing about how to reach the optimum. |
| LIT-1411 v2 | Theory on Mixture-of-Experts in Continual Learning | Li, Lin, Duan, Liang & Shroff (2024; ICLR 2025). In continual learning of overparameterized linear regression tasks, drawn uniformly from N ground truths each carrying a feature signal a linear gate can read, a top-1 MoE whose gate is trained on a locality loss plus a Switch-style balancing loss sorts identically initialized experts into one set per task (or task cluster) within about M/η rounds. If gate training continues, the balancing loss eventually erases the gaps between gate outputs and routing fails, so the gate is frozen once the gaps settle; forgetting then decays as 1/t, and extra experts only lengthen exploration. Small deep-network runs show the same pattern. | Active — read 2026-10-29 (NOTE-1380), the main paper in full and the experimental appendix, from arXiv v3; worth reading as a solvable model in which identically initialized experts specialize into task sets under a locality loss, and in which a Switch-style balancing loss, left running, erases the gate gaps that encode the specialization, so the gate must be frozen. Read it knowing that the tasks are linear and carry a feature signal that makes routing separable, that task arrival is uniform, that the necessity of termination comes from its own balancing term, and that the proofs (Appendices B–J) were not checked in this reading. |
| LIT-1412 v2 | Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity | Fedus, Zoph & Shazeer (2021; JMLR 2022). Simplifies mixture-of-experts routing to a single expert per token (the Switch Transformer), with a fixed expert capacity and an auxiliary load-balancing loss α·N·Σf_iP_i, and with training techniques (router-local float32, smaller initialization, expert dropout) that allow large sparse models to be trained in bfloat16. At equal FLOPs per token, up to 7x faster pre-training than T5-Base by wall-clock, gains across 101 languages over mT5-Base, and a 1.6-trillion-parameter model about 4x faster than T5-XXL to a fixed perplexity. | Active — read 2026-10-29 (NOTE-1363) in full, from arXiv v3 (the JMLR version); worth reading as the paper that made top-1 expert routing with a fixed capacity and the auxiliary balancing loss α·N·Σf_iP_i the standard for sparse Transformers, with the stability fixes that let it scale to 1.6 trillion parameters at T5’s FLOPs per token. Read it knowing that nearly every comparison is a single run, that it never measures what experts specialize in, that its balancing loss is not minimized only by uniform routing as §2.2 says, and that one quoted margin over T5-XXL (Switch-C’s) is 0.051, not “over 0.061”. |
| LIT-1414 v2 | Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts | Wang, Gao, Zhao, Sun & Dai (2024). Instead of an auxiliary balancing loss, whose gradients the authors hold interfere with language modelling, Loss-Free Balancing adds a per-expert bias to the routing scores used only for top-K selection and after each batch moves it by u·sign of the expert’s load deviation. On MoE models of 1B and 3B parameters trained on 100B and 200B tokens it gives slightly lower perplexity (9.50 vs 9.56; 7.92 vs 7.97) and much better global balance (maximal violation 0.04 vs 0.72 and 0.52) than an auxiliary loss with α = 0.001, one run each. | Active — read 2026-10-29 (NOTE-1388) in full, from arXiv v1; worth reading as the method that takes load balancing out of the gradient: a per-expert bias, used only to choose the top-K experts and moved by the sign of each expert’s load deviation on the previous batch. Read it knowing that the control responds to how often an expert is used, not to what it contributes, that the evidence is one run per configuration at 1B and 3B with perplexity gains of about 0.05, and that the conclusion’s 300B tokens for the 3B model should be 200B. |
| LIT-1484 | The Hive Mind is a Single Reinforcement Learning Agent | Soma, Bouteiller, Hamann and Beltrame show that in an infinite, well-mixed population, imitation of success has the expected update of a single Cross Learning bandit agent, and the weighted voter model of the honey-bee waggle dance the expected update of “Maynard-Cross Learning”, π_a ← π_a + (r_k / v^π)(1[a = k] − π_a), which is the Maynard-Smith replicator dynamic. Each member is one parallel action sample of the population’s policy, and the members themselves do not learn. Simulations, qualitative by the authors’ account, show the result surviving moderate swarm and neighbourhood sizes, and finite populations losing options for good. | Active — read in full 2026-10-24 (NOTE-1132) from arXiv v6 (21 July 2026, CC BY 4.0); worth reading as a proved aggregation result: in an infinite, well-mixed population, two imitation rules with no individual learning have the expected update of a single bandit-learning agent. Imitation of success gives Cross Learning (the Taylor replicator dynamic), the honey-bee weighted voter model gives what the authors call Maynard-Cross Learning (the Maynard-Smith replicator dynamic). The macro-agent is a stateless bandit policy over the members’ types. The biological model is deliberately partial, the simulations qualitative, and pure imitation has no way back once a type is lost or the environment changes. |
| LIT-1501 | A selection-mutation model for Q-learning in multi-agent systems | Tuyls, Verbeeck & Lenaerts (2003). Derives the continuous-time dynamics of Boltzmann Q-learning in two-player stateless matrix games, as later restated by Kaisers & Tuyls: ẋ_i = x_i α(τ⁻¹[(Ay)_i − x·Ay] − log x_i + Σ_k x_k log x_k), a replicator (selection) term plus an entropy (mutation) term, linking exploration in reinforcement learning to mutation in evolutionary game theory. Summary drawn from the abstract and from Kaisers & Tuyls 2010, not from the paper. | Unavailable — filed 2026-10-24 for the owner’s question how the record stands on multi-agent reinforcement learning; a reading was attempted the same day and failed on access. The ACM Digital Library (doi:10.1145/860575.860687) answered every request from this client with a Cloudflare challenge (403). Unpaywall and OpenAlex list no open copy. The repository records OpenAlex names hold no file this client could reach: the VUB repository (biblio.vub.ac.be) page is a script-only catalogue entry, ULB’s DI-fusion record shows no attached full text, the Hasselt document server returned 503 and a search result describes its record as “No Fulltext”, and CiteSeerX reset the connection. Web searches found no author copy. The model the paper derives is read instead through Kaisers & Tuyls 2010 (LIT-1511, NOTE-1140), which restates its equation and shows that it describes frequency-adjusted, not plain, Q-learning. It stays Unavailable until a copy is supplied. No NOTE is filed. |
| LIT-1503 | QMIX: Monotonic Value Function Factorisation for Deep Multi-Agent Reinforcement Learning | Rashid et al. represent a team’s joint action-value Q_tot as a mixing of per-agent values Q_a(τ^a, u^a) constrained by ∂Q_tot/∂Q_a ≥ 0, with non-negative mixing weights produced from the global state by hypernetworks. Monotonicity makes the joint argmax the profile of individual argmaxes, so execution is decentralised and maximisation is linear in the number of agents. It represents more than a sum (VDN), learning the optimal policy of a two-step game VDN misses, and beat VDN and independent Q-learning on six StarCraft II maps; it cannot represent joint values in which an agent’s best action depends on the others’. | Active — read in full 2026-10-24 (NOTE-1154) from arXiv v2 (6 June 2018, the ICML camera-ready, arXiv non-exclusive licence); worth reading for one structural condition: if the joint action-value is non-decreasing in each agent’s own value, the profile of individually greedy actions is a greedy joint action, so a collective can act on a common value while each member consults only its own. The condition is sufficient, not necessary, and it cannot represent a joint value in which one agent’s best action depends on what the others do at the same step. Its evidence is a two-step matrix game and StarCraft II win rates on six small maps. |
| LIT-1504 | Real World Games Look Like Spinning Tops | Czarnecki, Gidel, Tracey, Tuyls, Omidshafiei, Balduzzi & Jaderberg (NeurIPS 2020). Games in which players can signal n bits before the outcome is settled contain every win–loss pattern on 2ⁿ strategies, so cycles of length 2ⁿ (Go: at least 2¹⁰⁰⁰). Nash clustering orders a game’s strategies into clusters weakly ordered by population performance. Empirical games sampled from small board games and the AlphaStar league show cluster size peaking at middling strength, the “spinning top”, and population-based training converges once the population is large enough. | Active — read in full 2026-10-24 (NOTE-1146) from arXiv v2 (17 June 2020); worth reading for one clean theorem, one general lemma and a hypothesis with evidence. The theorem: where players can signal n bits before the result is settled, every pattern of wins and losses among 2ⁿ strategies, a cycle through all of them included, is realized. The lemma: Nash clusters are weakly ordered by population performance, and beating a population that contains a whole cluster lifts a strategy above it. The hypothesis, that real games are shaped like a spinning top, with cycles thickest at middling strength, rests on sampled empirical games. The abstract says it is proved and the body says it is not. The random-game theorem’s proof is defective, and the fixed-memory fictitious play result needs a replacement rule it does not state. |
| LIT-1507 | Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer | Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton & Dean (2017). A layer of up to 131,072 feed-forward experts with a trainable noisy top-k gate, inserted between LSTM layers, raises capacity over 1000-fold at roughly constant computation and beat the best published language-modelling and translation results of the time. Without balancing losses the gate concentrates on a few experts, a self-reinforcing imbalance; losses penalizing unequal summed gate weight and unequal load remove it and, in one ablation, cut test perplexity from 39.8 to 35.6. | Active — read in full 2026-10-24 (NOTE-1145) from arXiv v1, the only version; worth reading here for one observation and its fix, not for its scale results. The gate of a sparsely-gated mixture of experts “tends to converge to a state where it always produces large weights for the same few experts”, which the authors call self-reinforcing because favoured experts are trained more. Two auxiliary losses, the squared coefficient of variation of each expert’s summed gate weight and of a smooth estimate of its load, remove the imbalance. In the one ablation (a 256-expert model, 10 epochs) no loss gives CV 3.04 and a peak-to-mean load of 17.8 at test perplexity 39.8, any loss gives 35.6–35.7. The fix equalizes use regardless of merit; the noise in the gate is there mainly to make the load estimate differentiable. |
| LIT-1509 | Open-ended Learning in Symmetric Zero-sum Games | Balduzzi, Garnelo, Bachrach, Czarnecki, Pérolat, Jaderberg & Graepel (ICML 2019). Every symmetric zero-sum game between parametrized agents splits into a transitive part and a cyclic part relative to a measure on agents. In cyclic games individual performance is meaningless, so the paper measures populations, by the value of the game between them and by the convex hull of their evaluation rows (the gamescape). Response to the Nash mixture enlarges the gamescape, and response to the rectified Nash, which trains each supported agent on what it beats, grows it in Blotto-type games where self-play does not. | Active — read in full 2026-10-24 (NOTE-1143) from arXiv v2 (13 May 2019, the authors’ “ICML 2019, final version”); worth reading for one invariant and one reframing. The invariant: the value of the zero-sum game between two populations, which does not depend on the choice of Nash, ignores members that are behavioural mixtures of others, and cannot be positive for a population inside another’s hull. The reframing: in a cyclic game an individual agent’s performance means nothing, so progress is the growth of a population’s gamescape. Its functional Hodge decomposition is right but is stated with a degenerate inner product, and it depends on a measure over agents. Experiments are two small resource-allocation games. |
| LIT-1510 | QTRAN: Learning to Factorize with Transformation for Cooperative Multi-Agent Reinforcement Learning | Son, Kim, Kang, Hostallero & Yi define Individual-Global-Max (IGM): a joint action-value is factorised by per-agent values when the joint argmax is the profile of individual argmaxes. Additivity (VDN) and monotonicity (QMIX) are sufficient for it; QTRAN instead trains the sum of individual values plus a state value to equal the joint value at the greedy profile and to bound it elsewhere (Theorem 1), a condition also necessary up to positive rescaling. On 310 random 3×3 matrix games it learned the optimum in all, VDN and QMIX failed in 177; it also did better on Gaussian squeeze and a penalised predator-prey task. | Active — read in full 2026-10-24 (NOTE-1141) from arXiv v1 (14 May 2019, the ICML version with its supplement, arXiv non-exclusive licence); worth reading for the Individual-Global-Max condition, that the agents’ individually greedy actions form a greedy joint action, and for showing it strictly weaker than additive or monotonic factorisation. The authors’ own supplement notes that every single-state matrix game satisfies it, so IGM has bite only where each agent sees part of the state; it is a condition on whether members acting on local information can jointly reach the collective optimum, not a condition for a common currency. Evidence is matrix games, Gaussian squeeze and a predator-prey variant. |
| LIT-1511 | Frequency adjusted multi-agent Q-learning | Kaisers & Tuyls (AAMAS 2010). The replicator-plus-entropy model of Boltzmann Q-learning derived by Tuyls et al. (2003, 2005), ẋ_i = x_i α(τ⁻¹[(Ay)_i − x·Ay] − log x_i + Σ_k x_k log x_k), assumes that every Q-value changes at rate α. In Q-learning only the chosen action’s value is updated, so E[Q̇_i] carries a factor x_i, and trajectories in the prisoners’ dilemma, battle of the sexes and matching pennies deviate from the model, depending on initialization. Frequency Adjusted Q-learning scales each update by min(β/x_i, 1) and follows the model wherever x_i ≥ β. | Active — read in full 2026-10-24 (NOTE-1140) from the AAMAS 2010 proceedings PDF on the IFAAMAS site; read in place of Tuyls, Verbeeck & Lenaerts 2003 (LIT-1501), whose model it restates and corrects. Worth reading as the correction to “Q-learning is replicator dynamics plus mutation”: the selection–mutation equation describes Q-learning with Boltzmann exploration only if every action’s estimate is updated at the same rate. In Q-learning the chosen action is updated more often, so the expected change of Q_i carries a factor x_i, and the learning traces depart from the model, depending on how the Q-values are initialized. Frequency Adjusted Q-learning divides the step by x_i and matches the model. Evidence is three 2×2 games, plotted, with no quantitative measure of fit. |
| LIT-1512 | Mean Field Multi-Agent Reinforcement Learning | Yang, Luo, Li, Zhou, Zhang & Wang (2018). Factorizing each agent’s Q-function into pairwise terms and expanding about the neighbours’ mean action reduces many-agent interaction to an agent against the average of its neighbours. Mean field Q-learning and actor-critic follow; MF-Q is proved to converge to the Nash Q-value under strong assumptions. With 500 and 1000 agents in Gaussian Squeeze the mean field methods learn where four baselines fail; MF-Q reproduces the order parameter of a 20 × 20 Ising model against MCMC; and it wins a 64-versus-64 battle game. | Active — read in full 2026-10-24 (NOTE-1148) from arXiv v5 (15 December 2020), the ICML 2018 paper with appendices; worth reading as the standard way multi-agent RL reaches hundreds of agents: each agent’s Q-function takes its own action and the mean action of its neighbours, so many-body interaction becomes a two-body one between an agent and the average of the rest. Mean field Q-learning is proved to converge to a Nash Q-value under strong assumptions, every stage game met in training having a global optimum or a saddle point among them, and learns where four baselines fail with 500 and 1000 agents. The approximation needs agents that are homogeneous and interchangeable within a neighbourhood; the bound on its error is loose and the cancellation of errors argued, not proved; and the contraction step needs a high Boltzmann temperature while the convergence assumptions need it to go to zero. |
| LIT-1513 | Multi-agent Reinforcement Learning in Sequential Social Dilemmas | Leibo, Zambaldi, Lanctot, Marecki & Graepel (2017). Defines a sequential social dilemma as a Markov game whose cooperative and defecting policies, played against each other, give payoffs satisfying the social dilemma inequalities. Independent deep Q-learners in a fruit-Gathering game learn aggression when apples are scarce or a tagged rival is out long; in Wolfpack they hunt together when the joint-capture bonus and radius are high. Both games embed Prisoner’s Dilemmas, but network size raises defection in Gathering and lowers it in Wolfpack, which no matrix game captures. | Active — read in full 2026-10-24 (NOTE-1151) from arXiv v1, the AAMAS 2017 paper; worth reading as the paper that moved social dilemmas from matrix games to learned policies in Markov games. Two two-player games, Gathering (apples, and a beam that removes the rival for a while) and Wolfpack (lone or joint capture of prey), both yield Prisoner’s Dilemma payoff matrices when the learned policies are played off, yet respond in opposite directions to agent capacity: a larger network makes Gathering agents more aggressive and Wolfpack agents more cooperative, because aggression is the harder policy to implement in one and cooperation in the other. Agents learn aggression where apples are scarce. It is a descriptive simulation with independent learners, no figures of spread, and no protective institution or depletable commons. |
| LIT-1514 | Adaptive Mixtures of Local Experts | Jacobs, Jordan, Nowlan & Hinton (1991). A system of expert networks and a gating network, trained with the error of a single expert selected stochastically by the gate (or the negative log-likelihood of a mixture of Gaussians), so that each expert produces the whole output and an expert better than the gate-weighted average gains responsibility for the case. On speaker-independent vowel discrimination, 4 or 8 linear experts learned twice as fast as matched backpropagation networks at equal accuracy and divided the task by class pair; all but 2 or 3 experts went unused. | Active — read in full 2026-10-24 (NOTE-1147) from the scan of the published article on Geoffrey Hinton’s University of Toronto page; worth reading as the origin of the mixture of experts and as a learning rule in which authority over a case goes to whichever expert does better than the gate-weighted average. Redefining the error as that of a single expert chosen stochastically by the gate makes the experts compete rather than cooperate, and the log-likelihood form adapts the best-fitting expert fastest. On a four-class vowel task, mixtures of 4 or 8 linear experts reached criterion in about half the epochs of backpropagation networks with matched parameters, at the same accuracy, and in every run all but 2 or 3 experts ended with mixing proportions effectively 0. The authors read that as economy; nothing in the paper tests whether the idle experts could recover. |
| LIT-1517 | Counterfactual Multi-Agent Policy Gradients | Foerster, Farquhar, Afouras, Nardelli & Whiteson train decentralised actors with one centralised critic Q(s, τ, u) and credit agent a with A^a = Q(s, τ, u) − Σ_{u′a} π^a(u′a | τ^a) Q(s, τ, (u^−a, u′a)): the value of the joint action against the critic’s average over a’s own alternatives under a’s policy, the others’ actions fixed. The baseline does not depend on a’s action, so it does not change the expected gradient (Lemma 1). On decentralised StarCraft micromanagement with a narrow field of view, it beat independent and central-V/QV critics on four maps. | Active — read in full 2026-10-24 (NOTE-1152) from arXiv v3 (11 December 2024, arXiv non-exclusive licence), the corrected version; worth reading as the standard statement of counterfactual credit in cooperative multi-agent learning: each agent’s action is scored against the critic’s average over that agent’s own alternatives, the others’ actions held fixed, and that baseline provably leaves the expected gradient unchanged. The counterfactual values are the learned critic’s estimates, which exist only for actions the agents sometimes take, so the method credits agents that all act on every step and does not recover the value of one that is never heeded. Results are StarCraft win rates on four small maps. |
| LIT-1519 | Learning with Opponent-Learning Awareness | Foerster et al. replace an agent’s objective V¹(θ¹, θ²) with its value after the opponent’s anticipated naive gradient step, V¹(θ¹, θ² + Δθ²), and differentiate through Δθ² = η∇θ²V², adding (∇θ²V¹)ᵀ∇θ¹∇θ²V² · δη to the naive gradient. In the memory-one iterated prisoner’s dilemma, two exact-gradient LOLA learners played tit-for-tat-like policies in 81% of runs (return −1.06 against −1.98 for naive learners); in iterated matching pennies they settled at the mixed Nash equilibrium. A policy-gradient version cooperated in the Coin Game, and with opponent modelling did so less reliably. | Active — read in full 2026-10-24 (NOTE-1150) from arXiv v4 (19 September 2018, the AAMAS 2018 text, arXiv non-exclusive licence); worth reading as the first learning rule that models another learner’s update and climbs its own return through it: a LOLA agent adds to its gradient the effect of its own parameters on the opponent’s next naive step. Two exact-gradient LOLA learners reached tit-for-tat-like reciprocity in the iterated prisoner’s dilemma where naive learners defected. Shaping is one step deep, assumes access to the opponent’s parameters (or a behavioural clone of them), and is shown only in two-player games; its numbers have one internal inconsistency (see the NOTE). |
| LIT-1520 | Coupled replicator equations for the dynamics of learning in multiagent systems | Sato & Crutchfield (2002; PRE 2003). Two agents choose by softmax over reward memories Q_i that decay at rate α. In the limit where learning is slow relative to play, each agent’s choice distribution follows ẋ_i/x_i = β[(Ay)_i − x·Ay] + α Σ_j x_j log(x_j/x_i): a coupled replicator dynamic plus an entropy-like term towards uniform play. With α = 0 it is the conservative two-population replicator equation; with α > 0 it is dissipative. In rock–scissors–paper, simulations show quasiperiodic tori and Hamiltonian chaos without decay, and limit cycles, intermittency and chaotic attractors with it. | Active — read in full 2026-10-24 (NOTE-1144) from arXiv:nlin/0204057v2 (18 August 2002); worth reading as a four-page derivation of the learning dynamics of softmax reinforcement learners with forgetting: two such agents follow coupled replicator equations plus a term α_X Σ_j x_j log(x_j/x_i), set by the rate at which reward memories decay, that pushes each agent towards uniform play. Without forgetting the system is conservative, Hamiltonian in zero-sum games, and in rock–scissors–paper it is chaotic for nonzero tie rewards; with forgetting it is dissipative, with limit cycles and chaotic attractors. The memory update is not Watkins’s Q-learning: every action’s memory is updated with its average reward at the same rate. |
| LIT-1521 | A Simple Adaptive Procedure Leading to Correlated Equilibrium | Hart & Mas-Colell (Econometrica 2000; handout 1996). Regret matching: each period a player who last played j switches to k with probability R_t(j, k)/μ, the positive part of its average gain had it played k whenever it played j, and otherwise repeats j. If all players do so, the empirical distribution of play converges almost surely to the set of correlated equilibria. The proof runs through Blackwell’s approachability theorem; regrets vanish exactly when the empirical distribution approaches the correlated-equilibrium set. Inertia, a positive probability of repeating, is needed for the result. | Active — read in full 2026-10-24 (NOTE-1142) from the published article as posted on Sergiu Hart’s homepage; worth reading as the theorem that correlated equilibrium is reachable by simple adaptive play: if every player switches from its last action to each alternative with probability proportional to its positive regret for not having made that switch in the past, the empirical joint distribution of play converges almost surely to the set of correlated equilibria. No device, no forecasts and no best replies are needed; the common history is the correlating device. The limits matter: convergence is to the set, not to a point; correlated equilibria that are interior and not pure Nash cannot be the limit of the empirical distribution; and the players must observe one another’s past play. |
| LIT-1524 v2 | Cleaning up the neighborhood: A full classification for adversarial partial monitoring | Lattimore & Szepesvári (2018; ALT 2019). Completes the classification of finite adversarial partial-monitoring games by minimax regret to all games, including the degenerate ones (with actions optimal only on a lower-dimensional part of the simplex) that Bartók et al. (2014) left open: 0 with no pair of neighbouring actions, Θ̃(√n) when locally observable, Θ(n^2/3) when globally but not locally observable, and Ω(n) otherwise. A new NeighborhoodWatch2 attains √n with high probability by redistributing mass onto degenerate actions in proportion to the neighbours they lie between. It corrects a slip in Bartók et al.’s T^2/3 lower-bound proof and, in the preprint’s words, an exchanged expectation and maximum in the earlier localisation argument. A second algorithm, RelExp3, gives point-locally observable games an asymptotic √n bound free of arbitrarily large game constants, with a matching linear dependence on the number of feedback symbols. | Active — read 2026-10-25 (NOTE-1162) in full, from the arXiv preprint v1 of 23 May 2018 (arxiv.org/pdf/1805.09247v1, 24 pp.); the ALT 2019 version (PMLR 98:529–556) was fetched and its theorem statements, footnote and contribution list compared, not read in full. Worth reading for the completed classification of finite adversarial partial-monitoring games, degenerate and duplicate actions included (THEORY-963); for how actions that are optimal only on a boundary enter it, through the neighbourhood set and not as neighbours, and for its correction of the lower-bound proof of Bartók et al. (2014) (THEORY-961); for NeighborhoodWatch2, a simpler algorithm with a high-probability √n bound; and for RelExp3 on point-locally observable games (THEORY-950). Read it knowing that the arXiv text says Foster and Rakhlin and Bartók et al. “inadvertently exchanged an expectation and maximum” in their localisation argument and the published text no longer says so, though both abstracts say “correct errors in previous analyses”; that it proves only the two pieces of the classification that were missing or flawed and cites the rest; that its constants are game-dependent and can be large; and that a few small typos survive in the arXiv text. |
| LIT-1526 v2 | Online Learning and Blackwell Approachability with Partial Monitoring: Optimal Convergence Rates | Kwon & Perchet (2017). Blackwell approachability generalizes regret minimization (multi-criteria optimization, online optimization of a convex loss, online linear optimization under a cumulative constraint). Under partial monitoring the decision maker does not see outcomes but a random signal correlated with the decision–outcome pair, or with the outcome only. The paper gives the first approachability algorithms for polytope target sets with convergence rate O(T^-1/2) when the signal does not depend on the decision and O(T^-1/3) for general signals, and shows that these rates cannot be improved without further assumptions on the objectives or the signals (abstract). | Active — read 2026-10-25 (NOTE-1159) in full, from the PMLR copy and its supplementary material; worth reading for the reduction of approachability under partial monitoring to full-information approachability in a lifted space of flags, for the optimal worst-case rates, O(T^-1/3) for general signals and O(T^-1/2) when the signal does not depend on the decision (THEORY-947), and for the exploration floor that decision-dependent signals require (THEORY-957). Read it knowing that approachability is assumed rather than decided, that the guarantees concern the average payoff and so let violations be compensated over time, that optimality is worst case over games, that the constants grow with a simplicial subdivision, and that the proof for outcome-dependent signals is omitted. |
| LIT-1528 v3 | Partial Monitoring—Classification, Regret Bounds, and Algorithms | Bartók, Foster, Pál, Rakhlin & Szepesvári (2014). In a partial-monitoring game the learner picks an action, the environment an outcome, and the learner suffers a loss and receives a feedback signal, both fixed functions of action and outcome. For every game with finitely many actions and outcomes (against adversarial opponents, every non-degenerate one) the minimax regret is zero or scales as T^1/2, T^2/3 or T, up to constants and logarithmic factors; computationally efficient algorithms attain it within a logarithmic factor, with individual bounds on expected regret for i.i.d. outcomes (abstract). | Active — read 2026-10-25 (NOTE-1164) in full, from the authors’ preprint of March 2013 on Alexander Rakhlin’s MIT page; worth reading for the complete classification of finite partial-monitoring games by minimax regret (0, Θ̃(√T), Θ(T^2/3), Θ(T); THEORY-964), for the linear-algebra conditions, global and local observability, that decide the class (THEORY-962), and for the algorithms CBP and NeighborhoodWatch. Read it knowing that the adversarial classification covers non-degenerate games only, that Lattimore and Szepesvári (2019, LIT-1524) completed it to degenerate games, corrected the kernel in its T^2/3 lower-bound proof, and, in their preprint, an exchanged expectation and maximum in NeighborhoodWatch’s localisation (THEORY-963, THEORY-961), that the classification concerns the best policy in a fixed, known game, and that the copy read is the preprint, not the journal text. |
| LIT-1530 v2 | Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks | Musat, Pimentel, Zucchet & Hofmann (2026), arXiv:2607.11875. For a class of block-list inductive tasks that includes associative recall, k-hop induction and in-context n-grams, it proves that population gradient descent keeps an idealized attention-only transformer’s key–query and output weights on a low-dimensional invariant manifold fixed by the data’s symmetries, whose coordinates are interpretable head actions. On it, it proves that a saturated in-weights circuit suppresses the in-context circuit’s gradient by the fraction of data it does not solve and that burstiness multiplies it, shows empirically that initialization decides which of several induction circuits wins, and detects circuits in trained models by projecting onto the coordinates. | Active — read 2026-10-26 (NOTE-1177) in full, from the arXiv copy (v2); worth reading for its proof that symmetries of the data confine population gradient descent to a subspace of interpretable head actions (THEORY-981), for its theorem that a saturated memorized shortcut starves the in-context circuit of gradient on the data it solves (THEORY-1009), and for its scan in which initialization picks one of several induction circuits, read by the authors as cooperation through shared units and competition for layers (THEORY-1001). Read it knowing that the architecture is idealized (fixed orthogonal write maps, merged key-query, orthonormal embeddings), that invariance is proved and attraction is not, that the results are for gradient descent while its circuit-detection experiment uses AdamW, and that the competition scan was run in training projected onto seven coordinates. |
| LIT-1535 v2 | Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient | Wang, Hoogland, van Wingerden, Furman & Murfet (2024), arXiv:2410.02984, ICLR 2025. Refined variants of the local learning coefficient, a complexity measure from singular learning theory, are applied to individual components of a two-layer attention-only transformer, tracing how attention heads progressively differentiate into distinct functional roles during training, what data they specialize to, and a previously unidentified multigram circuit (abstract). | Active — read 2026-10-26 (NOTE-1188) in full, from the arXiv PDF of v1; worth reading as the record’s first developmental use of the learning coefficient, for per-head trajectories that separate attention heads by function (THEORY-975), and for its observations of heads changing role as circuits form (THEORY-967, Proposed). Read it knowing that it is one two-layer attention-only model with three appendix seeds, that the estimates are trusted only for their ordering, that cluster counts were chosen by eye, and that nothing is predicted before it happens. |
| LIT-1538 v2 | In-context Learning and Induction Heads | Olsson et al. (2022), arXiv:2209.11895. Induction heads, attention heads that complete [A][B] … [A] → [B], develop at precisely the point of a sudden sharp increase in in-context learning, visible as a bump in the training loss. Six lines of evidence argue that they may be the mechanism of most in-context learning in large transformers: strong and causal for small attention-only models, correlational for larger models with MLPs (abstract). | Active — read 2026-10-26 (NOTE-1173) in full, from the arXiv print of the Transformer Circuits article; worth reading as the first mechanistic account of a transition in training, for the in-context learning score, and for the way it grades its own evidence by model class. Read it knowing that the case is causal only for small attention-only models (THEORY-973), that the large-model thesis is correlational and by analogy (THEORY-990), that every model was trained once on the same data order, and that its dates for the window disagree between sections. |
| LIT-1649 v2 | Compositional Dynamics in Learning and Mechanics | Spivak (2026). An operad ArrSm of “smooth adaptive arrangements” (a responsive parameter space, a lens of output and input maps between manifolds, and a real potential) with two operad functors into polynomial coalgebras, discrete-time input-output machines, built by lens internalization (Corollary 3.3.4). The functors differ only in the integrator: Φconf stores position and steps down the sharpened gradient, Φphase stores position and momentum and takes a symplectic-Euler step. Φconf gives gradient descent with backpropagation, weight decay and Newton’s method. One arrangement of harmonic particles on a finite directed graph gives the discrete wave equation under Φphase and the discrete heat equation under Φconf, both governed by the symmetrized graph Laplacian, and for harmonic closed systems the phase step preserves the symplectic pairing exactly. | Active — read in full 2026-10-27 (NOTE-1225), from arXiv v2; worth reading as a compositional account in which the coupling data (wiring, potential, response) and the dynamical regime are separate choices: one arrangement of harmonic particles gives a discrete wave equation or a discrete heat equation depending only on whether the integrator stores momentum, and gradient descent with backpropagation is the same construction applied to a parameterized map. Rigorous for its categorical core and its worked examples; many further examples are reported from Claude-written code the author did not audit. |
| LIT-1879 v2 | Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent | Hennick & De Baerdemacker (2025; ICLR 2026). Models late-training SGD as sub-diffusion on a porous landscape: a time-fractional Fokker–Planck equation, the local learning coefficient as the medium’s mass dimension, and a spectral dimension d_s fitted from weight displacement, R(t) ∝ t^(d_s/2λ) with d_s ≤ λ. Where an effective scalar diffusion D_ξ is constant on a region, the steady state is exp(−L/D_ξ), the posterior tempered to effective sample size 1/D_ξ. How mass is shared between regions is not derived; in one toy test the cluster masses of SGD solutions match an SGLD posterior after a fitted tempering scale. Displacement fits cover MNIST MLPs, TinyStories models and fine-tuned ImageNet convnets. | Active — read in full 2026-10-31 (NOTE-1467); its “tempered posterior” is a temperature, the posterior at effective sample size 1/D_ξ within a region of constant diffusion, and the “accessibility” that would make SGD differ from Bayes across regions is not derived but fitted (THEORY-1258); its sub-diffusion fits are filed as Proposed (THEORY-1272). |
| LIT-1880 v2 | When Can Safe Controllers Adapt? Information before Commitment | Saligrama (2026). A safe adaptive controller must stay safe under every model not yet ruled out. Its first action that leaves no safe continuation under an alternative is a commitment, and the observation that action produces comes too late to justify it. With the information before commitment measured as a KL divergence between stopped records, every uniformly safe policy forfeits at least about ½e^{−I} of the value reachable only by committing. Acting on an exclusion at risk α needs about log(1/(2α)) nats beforehand. Bounded information and a linear gap give linear regret, shown for a headroom-limited device and for constrained linear regulation. Finite information can still certify a smaller candidate list. Safe recovery at O(√(T log T)) holds when a safe experiment is repeatable, informative and returns to a common state. | Active — read in full 2026-10-31 (NOTE-1466), with the main proofs; worth reading for the precise sense in which evidence must precede commitment: the observation a committing action produces cannot justify it, and bounded information before commitment leaves a fixed fraction of the oracle gap to every uniformly safe controller (THEORY-1254). It also proves the positive case, safe recovery from a repeatable informative experiment, and shows certainty equivalence failing at an active constraint (THEORY-1256). |
| LIT-1883 v2 | Overcoming Scaling Limits in On-Policy Self-Distillation for LLM Reasoning | Hossain, Kousar & Tourni (2026). In on-policy self-distillation the frozen initial model, given a reference solution, teaches the student along the student’s own rollouts. Under forward KL the best student is that teacher averaged over the contexts the student cannot see, with the conditional mutual information as irreducible loss and higher entropy (Proposition 1). The teacher’s advantage lies mostly on failed trajectories, where standard OPSD puts about 70% of its supervision, and is smaller at 4B than at 1.7B. OASIS supervises only verified rollouts and gives the teacher the student’s own failed attempt in place of the reference. It keeps a 3.2–3.8-point gain over the base model on AIME and HMMT from Qwen3-1.7B to 8B, where OPSD’s gain falls to 0.14, in single runs. | Active — read in full 2026-10-31 (NOTE-1479); worth reading as the published statement of the forward-KL limit on distilling from a better-informed teacher, with its entropy consequence (THEORY-1259), and for the finding that such supervision helps only where it is imitable (THEORY-1263). The trained objective clips each entry and is not a forward KL. |
| LIT-1884 v2 | Why Self-Training Helps and Hurts: Denoising vs. Signal Forgetting | Wu, Yang & Sun (2026). In overparameterized linear regression, iterated self-training with fresh covariates, noiseless pseudo-labels and minimum-norm students projects the estimate each round onto the row space of a new design. Noise from the first fit decays at (1 + τ)⁻¹ per round, and signal along a direction of variance s at s/(s + τ). Risk is U-shaped under anisotropy, monotone under isotropy, and tends to that of predicting zero. Deterministic equivalents concentrate uniformly over ridge parameters, and an iterated GCV picks the stopping round. One CIFAR-10 experiment shows the U-shape. | Active — read 2026-10-31 (NOTE-1471), body and appendices, with the proofs skimmed; worth reading for an exact account of iterated reconstruction through fresh low-rank views, in which forgetting the ancestor first improves and then destroys fidelity to the world, and every student fits its teacher exactly on its own sample (THEORY-1269). |
| LIT-1891 v2 | General agents contain world models | Richens, Abel, Bellot & Everitt (2025), ICML. In a fully observed, finite, communicating, stationary controlled Markov process, a deterministic goal-conditioned policy whose success probability is within a factor 1 − δ of optimal on every composite goal of depth up to n determines each transition probability to within √(2P(1−P)/((n−1)(1−δ))). An algorithm reads the estimate off the policy alone, from its first action on either-or goals. A policy competent only at one-step goals determines nothing. “Contains a world model” means this recoverability from the input-output map, “not its specific use”, so it holds of any implementation of the same map. The experiments use a model-based agent in one 20-state environment. | Active — read in full 2026-10-31 (NOTE-1469); worth reading for the cleanest theorem that competence across a family of goals fixes the environment’s dynamics, and for exactly what “contains a world model” means in it: the transition function is computable from the goal-conditioned input-output map (THEORY-1249), which holds of a lookup table as much as of a planner (THEORY-1260). It is the objection to the functional reading of CLAIM-519, and it does not meet that claim’s defeated_if. |
| LIT-1894 v2 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei & Murfet (2023). In Elhage et al.’s toy model of superposition with two hidden dimensions, in the high-sparsity limit, the population loss has a closed form and regular k-gons are critical points with derived local learning coefficients. The free energy formula predicts a Bayesian phase transition from the 5-gon to the 6-gon near n = 600, which MCMC confirms. SGD trajectories plateau at the same critical points and move to lower loss and higher estimated complexity, but the 5 → 6 transition was not observed in training. The paper states that there is no necessary relation between the two kinds of transition, and conjectures only that training’s transitions have Bayesian antecedents. | Active — read 2026-10-31, the derivation appendices skimmed (NOTE-1462); it does not argue that SGD and the posterior trace one journey: it states “no necessary relation” between their transitions, proposes only that training’s transitions have Bayesian antecedents, and its main Bayesian transition, 5-gon to 6-gon, was not seen in training (THEORY-1275). |
| LIT-1895 v2 | Self-Distillation Amplifies Regularization in Hilbert Space | Mobahi, Farajtabar & Bartlett (2020). In ℓ2-regularized regression in a Hilbert space held to a fixed training-error tolerance, refitting to one’s own predictions on the same inputs shrinks each eigencomponent of the labels by d_k/(c_t + d_k). The multiplier c_t must rise as the labels shrink. Rounds therefore amplify regularization, use progressively fewer basis functions (Theorem 5) and collapse to the zero function after finitely many. A few rounds can reduce overfitting and more underfit, in a spline example and, by fractions of a point, in ResNet and VGG runs on CIFAR. | Active — read 2026-10-31 (NOTE-1478), body and appendices, with the proofs of Appendix C skimmed; worth reading as the clean case of iterated self-distillation losing everything through the regularizer alone, on the same inputs with nothing withheld (THEORY-1252). |
| LIT-1896 v2 | Multi-Token Prediction Done Right | Teoh (2026). With a shared linear output head, the cross-entropy gradient at each multi-token-prediction hidden state is exactly the gradient of a forward KL toward the model’s own position-aligned next-token prediction, held fixed, plus the next-token cross-entropy gradient at the aligned state. A one-line straight-through routing sends only the KL part to the MTP heads and trunk, keeping cross-entropy at the head. The residual carries about 80% of the MTP gradient reaching the trunk at about a quarter of the KL part’s signal-to-noise ratio. Removing it (“Soft-MTP”) improves next-token loss, speculative decoding and zero-shot accuracy at 167M–1.2B by small margins in single runs. | Active — read in full 2026-10-31 (NOTE-1470); worth reading for its exact split of the multi-token-prediction gradient into self-distillation toward the model’s own next-token head and a data-target residual (THEORY-1274), and for what the split does not carry: in the DeepSeek-style heads of its main results the student sees the same tokens as its teacher, so it is not the information projection the exchange read into it. |
| LIT-1898 v2 | Stochastic Gradient Descent as Approximate Bayesian Inference | Mandt, Hoffman & Blei (2017). Constant-rate SGD near a minimum with positive-definite Hessian A and constant gradient-noise covariance BBᵀ is modelled as an Ornstein–Uhlenbeck process whose Gaussian stationary covariance solves AΣ + ΣA = (ε/S)BBᵀ. Minimising KL to the Gaussian posterior gives the optimal constant rate 2SD/(N tr BBᵀ) and the preconditioner (2S/εN)(BBᵀ)⁻¹, which matches the posterior exactly; momentum only rescales the covariance; constant-rate SGLD overestimates the posterior covariance; and averaging T iterates gives covariance A⁻¹BBᵀA⁻¹/(TS), the posterior’s for TS observations in a well-specified model. Tested on linear, logistic and softmax regression. The search phase, singular minima and multiple basins are outside the analysis. | Active — read in full 2026-10-31 (NOTE-1475); the paper the owner recalled at U4, and the right one, but what it shows is narrower than “the same process”: near one non-degenerate minimum, the optimiser’s noise scale ε/S plays the part of 1/N, and training time enters only as the wait for stationarity (THEORY-1255, THEORY-1257). |
| LIT-1899 v2 | A Variational Analysis of Stochastic Gradient Algorithms | Mandt, Hoffman & Blei (2016). The first version of the 2017 JMLR paper. Constant-rate SGD near a quadratic optimum is modelled as an Ornstein–Uhlenbeck process; its Gaussian stationary law is fitted to the Gaussian posterior by minimising KL over the learning rate, batch size and preconditioner, giving the optimal rate 2DS/(N tr BBᵀ) and the exact preconditioner (2S/εN)(BBᵀ)⁻¹. It shows SGFS’s preconditioner is KL-optimal, quantifies its diagonal truncation, and uses constant SGD for variational EM over hyperparameters. The later paper adds momentum, constant-rate SGLD and iterate averaging. | Active — read in full 2026-10-31 (NOTE-1473); every result reappears in the 2017 JMLR paper (LIT-1898), which is the one to cite. Worth keeping as the first statement and as the version Hennick and De Baerdemacker cite. |