Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Anthology candidate — a curation flag, not a subject — somebody judged the Anthology of the SOTA may want this work as well; it stays here unless a transfer is decided, and is never a primary topic (ADR-005, ADR-048; group: flags).
63 of 1352 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-078 v2 | The top C eigenvalues of a trained C-class classifier’s Fisher are carried by the second moment of its class-mean logit derivatives, so a cut at the bulk edge keeps the span of the class means, and they stand clear of the bulk to the extent between-class separation exceeds within-class variation | Papyan (2019), LIT-619: for softmax cross-entropy the Gauss–Newton matrix G, which is the model’s Fisher averaged over inputs, is an uncentred second moment of logit derivatives indexed by class and logit. It decomposes exactly into four terms (Eq. 28), and its top C eigenvalues match those of the class-mean term G₁. Papyan (2020), LIT-613, splits those C into one global-mean outlier and C − 1 between-class ones, finds a C(C − 1) cross-class mini-bulk beneath them, and proves the whole pattern for a Gaussian toy model. Papyan (2018), LIT-617, supplies the measurements and the attribution of the Hessian’s outliers to G. The decomposition is algebra. That G₁ carries the outliers is measured, by eye, under a training protocol narrower than practice. So this is Proposed. | Proposed |
| THEORY-080 v3 | In a linearised network with a Gaussian prior on its parameters, the Occam factor is a sum over the NTK’s eigenvalues, so on spherical data each harmonic pays ½ log(1 + βnµ_k/α) and the kernel’s eigenvalue decay schedules the complexity penalty across frequencies | The filer’s synthesis, stated by no source. Its exact half is filed as the record’s own claim, CLAIM-384; what stays here is the spherical account. A network linearised at initialisation, with an isotropic Gaussian prior on its parameters and Gaussian noise, is MacKay’s linear-in-parameters model (LIT-623). Its Occam factor is −½ Σ log(1 + βλ_a/α) over the eigenvalues of JᵀJ. These are the empirical NTK’s, which on uniform spherical data form Bietti & Bach’s degree blocks, nµ_k with multiplicity N(d, k) (LIT-608). Degrees whose βnµ_k exceeds α pay about ½ log n per harmonic, and degrees below pay almost nothing. With µ_k ~ k^(−d), the number of paid directions γ grows as n^((d−1)/d), not as the parameter count. Basri et al.’s bound (LIT-612, Eq. 14) is the matching data-fit term. The kernel’s decay exponent is the schedule of the penalty. | Proposed |
| THEORY-082 | Steepest descent is defined only relative to a metric, and Amari’s proof that natural-gradient learning is Fisher efficient needs the Fisher information at the optimum to be invertible, the condition whose failure WBIC calls singular; in a multilayer perceptron it fails wherever an output weight is zero | Amari (1998), LIT-607. Theorem 1 is a Lagrangian derivation: under |dw|² = Σg_ij dw_i dw_j the steepest-descent direction is −G⁻¹∇L, so it changes with G. Theorem 2 shows that online learning with step 1/t along the Fisher natural gradient attains the Cramér–Rao bound G⁻¹/t, and its proof uses G(w*)⁻¹. Watanabe, LIT-616, Eq. 14, calls a truth regular when the optimum is one point and its Fisher is positive definite, and singular otherwise. From Amari’s own Eq. 6.12, the score for hidden unit i’s input weights carries the factor v_i. So the Fisher of a multilayer perceptron loses rank wherever an output weight is zero. That set is part of the optimum whenever the teacher needs fewer hidden units than the student has. Active because each step is checkable in the two texts. The claim that the Fisher is the only invariant metric is Chentsov’s, cited and not held. | Active |
| THEORY-083 v3 | In Papyan, Han and Donoho’s runs of VGG, ResNet and DenseNet trained with cross-entropy on seven balanced datasets without augmentation, training past zero error shrinks last-layer within-class variability against between-class variability, and moves the centred class means and the classifier toward a common simplex equiangular tight frame and the decision toward nearest class mean, nearly to completion on the four easier datasets and only partly on ImageNet | Papyan, Han & Donoho (2020), LIT-618, measured 480 trained classifiers on seven datasets with VGG, ResNet and DenseNet. Through the terminal phase, within-class variability of last-layer features falls (NC1), and the centred class means become equinorm and equiangular with cosine −1/(C − 1) (NC2). The classifier aligns with them (NC3), and the decision becomes nearest class mean (NC4). They prove that NC1 and NC2 imply NC3 and NC4 for the MSE-optimal and max-margin classifiers, and that the simplex ETF is the unique error-exponent-optimal codebook. Read from the plots, collapse is close to complete on the four easier datasets and partial on ImageNet at the last epoch. Papyan (2020), LIT-613, sees class means separate and grow more orthogonal with depth. The measurements cover one protocol, and the dynamics are unexplained outside free-features models. Within them, THEORY-278 (LIT-974) derives collapse for MSE under a re-whitened flow, and THEORY-286 (LIT-994) gives the cross-entropy endpoint under class imbalance. The paper’s further claim, that the Hessian’s class outliers are a symptom of collapse, is argued and not measured, and is left to THEORY-078. | Active |
| THEORY-084 v3 | Because the Gauss–Newton Fisher JᵀJ and the NTK Gram matrix JJᵀ share their non-zero spectrum, the lazy-regime Fisher of a ReLU network on spherical data inherits the NTK’s harmonic-degree blocks, and a spectral threshold on it is well posed only at gaps between blocks | The filer’s synthesis, stated by no source. The identity is linear algebra: JᵀJ (parameters × parameters) and JJᵀ (samples × samples) have the same non-zero eigenvalues. Under squared loss JᵀJ is the Gauss–Newton matrix and the Fisher. JJᵀ is the empirical NTK Gram matrix, which in the kernel regime on uniform spherical data has the degree-block spectrum of Bietti & Bach (2021), LIT-608: µ_k with multiplicity N(d, k). The threshold claim draws on Cao et al. (2019), LIT-624. Their guarantees are stated only for whole eigenspaces, with sample costs that grow as the gap below them shrinks. It also draws on Basri et al. (2019), LIT-612, where a structural zero block shows up in a finite sample as small non-zero eigenvalues. No paper filed computes a Fisher spectrum in this regime. | Proposed |
| THEORY-085 v2 | A classifier’s class-driven Fisher outliers pair each class’s feature mean with that class’s own error mean, so K-FAC’s single Kronecker product of class-averaged factors adds every cross-class pairing and misplaces them | Papyan (2020), LIT-613: per layer, the Fisher is a weighted sum of Kronecker products of features and backpropagated errors (Eq. 7.5), and its class outliers come from class means of gradients that are approximately h_c ⊗ δ_c. K-FAC (Martens & Grosse 2015, LIT-622) multiplies the class-averaged feature covariance by the class-averaged error covariance. Papyan finds its outliers and mini-bulks “drastically misaligned”, and the class-wise CFAC aligns much better at deep layers. The identity relating the two, CFAC − K-FAC = the class-weighted between-class covariance of the factors, is the record’s own and is filed as CLAIM-391. The evidence for the mechanism is one MLP, spectra only, judged by eye. | Proposed |
| THEORY-086 v2 | In the kernel regime gradient descent fits the target eigenspace by eigenspace of the neural tangent kernel, at rates set by their eigenvalues; on uniform spherical data the eigenspaces are the harmonic degrees, and for fully connected ReLU networks the non-zero eigenvalues decay as k^(−d) at every depth | Three derived results that compose. Cao et al. (2019), LIT-624, Theorem 4.2: for a wide two-layer ReLU network, the training residual’s component in the top-r_k NTK eigenspace falls as (1 − λ_{r_k})^T, for any target. Basri et al. (2019), LIT-612: on uniform spherical data the Gram matrix is a convolution, so its eigenvectors are harmonics, with closed-form eigenvalues per frequency. The bias-free kernel is zero on odd frequencies k ≥ 3, and a bias restores them. Bietti & Bach (2021), LIT-608, Corollary 3: the L-layer ReLU NTK decays as k^(−d) on S^(d−1), the same exponent at every depth. Active for the kernel regime on uniform spherical data, where every step is proved. It says nothing about networks whose kernel moves. | Active |
| THEORY-087 v3 | The Bayesian complexity penalty grows as (d/2) log n in a regular model, where the curvature at the optimum sets the Occam factor, and as λ log n in a singular one, where λ is the real log canonical threshold; with the prior positive on the optimal set, λ falls below d/2 when that set is infinite or has a point of degenerate curvature, and equals d/2 when it is finitely many points of non-degenerate curvature | MacKay (1992), LIT-623, writes the evidence as best-fit likelihood times an Occam factor, the ratio of posterior to prior accessible volume, which under a Gaussian approximation is P(w_MP)(2π)^{k/2} det^{-1/2}A. When the likelihood dominates, det A grows as n^d, so the penalty is (d/2) log n. Watanabe’s WBIC paper, LIT-616, recovers that case (Lemma 3, Theorem 5) and states what replaces it when the Fisher information is degenerate at the optimum: λ log n − (m − 1) log log n, with λ the RLCT (Theorem 2). The regular half is re-derived by Laplace’s method with its conditions. The singular half is Watanabe’s cited theorem, checked by exact free energies in two singular models of known λ. The volume reading of λ is in Watanabe’s book, which is unread. | Active |
| THEORY-092 v3 | Behavioural capacity does not fix intelligence: a system with the capacity to respond sensibly to every possible input can lack it, so whether behaviour is intelligent, or the mark of any mental property, depends partly on how it is produced | Block (1981), LIT-642: the strongest behaviourist conception of intelligence, a capacity to give sensible responses to any sequence of inputs, “whatever they may be”, is met by a machine that stores every sensible conversation of a test’s length and looks up its next reply, yet “the machine has the intelligence of a toaster”. So psychologism is true: two systems alike in all actual and counterfactual behaviour can differ in intelligence through the processing that mediates it. Block (1995), LIT-630, n. 1: the same machine refutes the wider claim that there is no mental difference without a behavioural one. It is a claim about constitution, not evidence: performance stays evidence for competence. It names no required kind of processing, and its machine is a lookup table, not a trained network. | Proposed |
| THEORY-104 v2 | Under distribution shift the best layer for a linear readout lies below the top: across the last layers of a trained network, in-distribution accuracy rises while out-of-distribution accuracy falls | Uselis & Oh (2025), LIT-678: linear probes on frozen intermediate layers of image classifiers beat last-layer retraining under shift, with zero-shot worst-group accuracy 79.4 → 87.1 (Waterbirds) and 56.0 → 82.0 (CelebA), layer 7 of 8 beating layer 8 on every subpopulation dataset, while in-distribution accuracy is best at the last layer. Chang, Deng & Chen (2025), LIT-657: in GPT-2 Small on a synthetic task, early-exit out-of-distribution accuracy peaks at layer 10 (56.90%) and falls to 50.00% at layer 12 while in-distribution accuracy climbs to 93.17%. The layer is chosen on out-of-distribution data throughout; ViT gains are small; the language-model evidence is one synthetic task; and Chang et al.’s information estimate is questioned in its NOTE. | Proposed |
| THEORY-105 v2 | For RoBERTa-base fine-tuned on English classification data, with loss measured by a linear probe refitted on the frozen encoder, points sampled from the convex hull of models fine-tuned on one dataset, one task family or twelve GLUE and SuperGLUE datasets lost no more than any model fine-tuned elsewhere and beat the group’s own members in 88% to 96.7% of comparisons, and the centroid of the general-level models was a better starting point for BitFit than the pretrained model | Gueta et al. (2023), LIT-658: RoBERTa-base models fine-tuned on the same dataset, the same task family, or any of 12 GLUE and SuperGLUE datasets cluster in weight space. Sampled points in the convex hull of each group beat the group’s own members in 88% to 96.7% of comparisons. Extrapolating past the members raises the loss quickly, so the members sit near the region’s edge and averaging moves inward. The paper calls the region convex, but shows only that the hull lies inside it, by a probe-based loss. Its reading of model soups and SWA as picking points inside the region is argued in its §8, not tested. Moving from the centroid towards the origin stays low-loss far past the members, which the authors cannot explain. Promoted 2026-10-20 on the probe-loss comparisons measured (ADR-047 v3); the geometric thesis is THEORY-828. | Active |
| THEORY-106 | In random-features ridge regression the double-descent peak is a divergence of the test error at the interpolation threshold 2N = n as the ridge goes to zero, which a positive ridge keeps finite and a ridge of the tuned size removes; minimum-norm linear regression has the same peak at n = d in exact finite-sample form | Liao, Couillet & Mahoney (2020), LIT-679: for random Fourier features with n, p and N large together, the test MSE has a closed form through a deterministic equivalent of the resolvent. As λ → 0 its corrections jump from O(1) to order 1/λ across 2N = n, det(Ω⁻¹) scales as λ, and the test error diverges at 2N = n unless the test data lie within about λ of the training data. At λ = 10⁻⁷ and 10⁻³ the curve has a singular peak; at the tuned λ ≈ 0.2 and at 10 it has none. Dereziński, Liang & Mahoney (2019), LIT-683: exact MSE of the minimum-norm estimator under a surrogate design, peaking at n = d with value σ² tr(Σ⁻¹), and below the threshold equal in expectation to ridge regression at the λn that makes the ridge effective dimension n. Proportional-limit theorems for one shallow model each; nothing here reaches trained deep networks. | Proposed |
| THEORY-107 | Merging models fine-tuned from one pretrained start loses accuracy through interference in parameter coordinates: redundant small updates dilute the influential ones, and influential updates of different models conflict in sign | Yadav et al. (2023), LIT-675: keeping only the top 20% of each task vector by magnitude loses almost nothing, so most of an update is redundant. Among the entries that remain, signs conflict between models, more so as more models are merged. Merging (IA)³ models with the multitask model’s signs reaches 72.0 against 73.1 for multitask training and 66.4 for TIES with elected signs. That is the strongest evidence that sign is the bottleneck. But the oracle signs come from training on every task’s data, the ablation finds rescaling and the disjoint mean matter more than election, and conflicts are as frequent among checkpoints of one task as across tasks. | Proposed |
| THEORY-108 | Across load and temperature, the connectivity and output similarity of independently trained solutions predict test accuracy where endpoint curvature does not: a flat but poorly connected landscape generalizes worse than a connected one of the same curvature | Yang, Hodgkinson, Theisen, Zou, Gonzalez, Ramchandran & Mahoney (2021), LIT-663: thousands of trained networks on a load by temperature grid, measured by the endpoint Hessian, by mode connectivity on a trained Bezier curve between two independent runs, and by CKA of their outputs. Phases III (flat, poorly connected) and IV-A (flat, connected) have almost the same Hessian and different accuracy; with 10% noisy labels the sharp, unconverged phases beat phase III; label noise barely moves the Hessian while connectivity and CKA fall. The best accuracy is in the flat, connected, high-similarity phase IV-B. The counterexamples are to the anthology’s ANTH-SOTA-012. One architecture family carries most of it, phases are read by eye, and connectivity alone also fails when the load is data amount. | Proposed |
| THEORY-109 v2 | The solutions SGD finds in a deep network lie in one connected low-loss set, joined by simple curves and spanning volumes of measurable dimension, and this is a property of the solutions training finds, not of every minimum | Garipov et al. (2018), LIT-673, and Draxler et al. (2018), LIT-653, found independently that minima of deep networks trained from different initializations are joined by paths of near-constant loss, a one-bend curve or an AutoNEB path, while the straight line between them climbs to near-chance error. Benton et al. (2021), LIT-656, widened curves to simplicial complexes joining up to seven modes and a low-loss region of at least 10 dimensions. Kuditipudi et al. (2019), LIT-671, proved that dropout-stable solutions are always joined, and constructed global minima that are not, so connectivity belongs to found solutions. Proposed: the evidence is pairs and small complexes of CIFAR classifiers. | Proposed |
| THEORY-110 v2 | A set of trained solutions has a centre that is linearly connected, after alignment, to the rest of them: the solution set is a star domain even where it is not convex | Sonthalia et al. (2024), LIT-659, conjecture that below the width at which SGD solutions become convex modulo permutation they are still a star domain, and train a star model whose barriers to held-out solutions, after weight matching, are about 2–5 times lower than those between solutions (ResNet-18 CIFAR-10 0.078 against 0.383; ImageNet 2.794 against 5.948), and not zero. Lin et al. (2024), LIT-682, prove star-shaped linear connectivity for wide two-layer teacher–student and deep linear networks, and find centres for five trained image classifiers with no permutation. Proposed: reduced, not zero, and no held-out test without alignment. | Proposed |
| THEORY-111 v3 | In image classifiers, between two networks linearly connected by spawning from a shared checkpoint or by permutation alignment (MLPs, and VGG-16, ResNet-20 and ResNet-50 on CIFAR-10 and Tiny-ImageNet), nearly every layer’s features at three points on the line were proportional to the interpolation of the endpoints’ features, and pairs that were not linearly connected were far from it | Zhou et al. (2023), LIT-674: whenever two image classifiers are linearly mode connected, by spawning or by permutation, the features of nearly every layer of the interpolated network are proportional to the interpolation of the endpoints’ features. This is layerwise linear feature connectivity, and two conditions, weak additivity of ReLU and a commutativity of weights with features, imply it exactly. Lubana et al.’s Lemma 2, LIT-669, gives the exact one-hidden-layer case: linear connectivity between interpolating minima forces shared activation patterns, and with shared patterns the hidden features are linear along the path. Measured on pairs at three interpolation points, image classification only. The regularity outside image classification, and the step to many-model averages such as soups and SWA, are QUESTION-318. | Active |
| THEORY-112 v2 | Aligning independently trained solutions over the architecture’s symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path’s midpoint, took it to zero or near zero except on BookCorpus (0.42) | Entezari et al. (2022), LIT-652, conjectured that SGD solutions are one basin modulo permutation. Git Re-Basin (2023), LIT-661, aligned MNIST MLPs and 32×-wide ResNet-20s to zero barrier, but not 1×-width models or ImageNet ResNet-50. Ferbach et al. (2024), LIT-680, proved it for wide nets with independent neurons, with a width requirement exponential in the previous layer’s width. Theus et al. (2025), LIT-664, took transformers to zero only with an orthogonal map on the residual stream added; permutations alone left 0.29–1.60. Bo Zhao et al. (2025), LIT-655, place permutations inside the full symmetry group. Lubana et al.’s barriers between models that compute differently are the boundary. Whether most of the barrier goes at standard width on ImageNet or a language corpus under data-free alignment, and whether the barrier left rises with depth, is QUESTION-317. | Active |
| THEORY-113 v2 | Merging ResNets trained from different initialisations on disjoint halves of a label set by permuting one onto the other and averaging fell far short of the ensemble; their cross-model feature correlation fell with depth, merging only the early layers recovered most of the ensemble’s accuracy, and merging features within each model as well as across both helped at 4× width and above; Stoica et al. attribute the failure to features with no counterpart in the other model | Stoica et al. (2023), LIT-665, merge ResNets trained from different initialisations on disjoint label sets. Permute-then-average fails, which they explain by the permuted model lying in a basin for its own task only. Mean cross-model feature correlation falls with depth (0.50, 0.37, 0.27 at three depths). Merging only the early layers and keeping later layers as task heads recovers most of the ensemble’s accuracy. Merging features within one model as well as across both helps once the models are wide enough to hold redundant features. With full merging on ImageNet-1k, ZipIt! is no better than permutation (8.6%). No same-task control separates the effect of task difference from that of depth; that is QUESTION-316. | Active |
| THEORY-114 v2 | In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training | Frankle et al. (2020), LIT-654: two copies trained from step k under different SGD noise are linearly connected only once k is a little way in, at 1.5–3% of training for VGG-16 and ResNet-20 on CIFAR-10 and 16–20% for Inception-v3 and ResNet-50 on ImageNet. Git Re-Basin (2023), LIT-661: between independently trained MLPs, the barrier after permutation alignment is high at initialization and falls gradually over training. Entezari et al.’s theorem proves alignment-LMC at random initialization; Ferbach et al. (2024), LIT-680, show the width that needs grows recursively exponentially with depth, far past practical widths, which is why the two are consistent. | Active |
| THEORY-115 v2 | Linear connectivity from a shared, already-stable start holds only when the objectives the runs optimise share structure, and breaks as that structure is removed | Mirzadeh et al. (2020), LIT-651: from a task-1 solution, a model trained on tasks 1 and 2 together stays on a low task-1 loss line from the start, and one trained on task 2 alone does not. Adding noise to task 2’s inputs, corrupting its labels, or removing classes separates the solutions progressively. The authors explain it by curvature: the multitask direction avoids the earlier task’s top Hessian eigenvectors. Gueta et al. (2023), LIT-658: fine-tuning runs from one pretrained model are linearly connected within a dataset and a task family. Wen et al. (2023), LIT-681: in class-incremental learning the line between adjacent continual minima crosses a ridge in the old tasks’ loss. Limits: much of the measured breakage is forgetting at the endpoint, not a barrier, and FedAvg models from one random start under one global objective are separated by a barrier that grows with heterogeneity. | Proposed |
| THEORY-116 v2 | A linear barrier between two minima that survives the removal of symmetries marks a difference in the input attributes the two models rely on: they compute by different mechanisms | Lubana et al. (2022), LIT-669: models that rely on a synthetic cue and models that ignore it are separated by a linear barrier even after permutation matching. Fine-tuned models that keep a linear path to their parent keep its cue, and those that lose the path lose the cue. Their Conjecture 1 states this claim’s direction. What their Appendix F.3 proves, for a one-hidden-layer ReLU network with interpolating minima, is the converse: linear connectivity forces identical activation patterns, so models relying on attributes of different complexity cannot be linearly connected under any permutation. Zhou et al. (2023), LIT-674, support that converse in deep image classifiers, where linearly connected pairs have features that blend layer by layer. The stated direction rests on experiments with synthetic cues and on a one-layer sufficiency argument. | Proposed |
| THEORY-117 v3 | In Uselis, Dittadi and Oh’s two-concept grids, ResNet-50s trained from scratch, with checkpoints chosen on the unseen pairs, reach unseen combinations as the share of combinations seen grows and not as up to four times more data of the same combinations is added; and when embeddings are exactly linearly factored, seen pairs that connect every value of both concepts, as two per value arranged in one cycle do, determine the embedding of every unseen pair | Uselis, Dittadi & Oh (2025), LIT-667. In an n × n grid of two concepts with k combinations seen per value, ResNet-50s trained from scratch fit seen pairs and fail on unseen ones; four times the data of the same combinations leaves 60–80% drops, while more values and more combinations help. Features pass from spurious, to decodable but entangled, to linearly factored as coverage grows. Model selection was on the test set, by design, so the results bound what is possible. The record re-derived that exactly factored embeddings of every unseen pair are fixed by seen pairs connecting every value, as the paper’s k = 2 cycle does. Honest selection, other architectures and more concepts are QUESTION-207. | Active |
| THEORY-175 v3 | Cosine similarities to matched anchor points are unchanged by exactly the maps that preserve each sample’s cosines to the anchors, among them every orthogonal map times a positive scalar and every per-sample positive rescaling but not translations; in Moschella et al.’s experiments independently trained encoders agreed in them only approximately, enough for a decoder reading them to transfer untrained between seeds, architectures and languages with loss, while their nearest neighbours mostly differed | Moschella et al. (2022; ICLR 2023), LIT-849, as read in NOTE-652. The invariance is by construction; its class (the maps preserving each sample’s cosines to the anchors, which include every orthogonal map times a positive scalar and every per-sample positive rescaling, but not translations) is the reader’s statement, re-derived in a dated note. The approximate agreement is the paper’s evidence: relative vectors match at cosine 0.86–0.97 across encoders, while ten-nearest-neighbour overlap is 0.10–0.39. It does not say the maps between trained spaces are isometries, which the source never measures (QUESTION-211), nor that the transfer needs no paired data: the anchors are paired data. | Active |
| THEORY-180 v2 | In a tied single-head attention layer trained by ridge-penalized empirical risk minimization on high-dimensional Gaussian sequences, weight decay is a nuclear-norm penalty on the query–key map, and the learned map is a soft-thresholded noisy copy of the target: its outliers are recovered target directions and its bulk is finite-sample noise | Boncoraglio, Erba, Troiani, Xu, Krzakala & Zdeborová (2025), LIT-857: with S = WWᵀ, λ‖W‖²_F is exactly λ‖S‖_* (Appendix A), and in the limit n/d², p/d fixed the trained WᵀW/√(pd) has the law η·ReLU(S0 + δZ − εI), Z ∼ GOE, with δ a noise level that falls with the data and ε a threshold set by λ (Claim 4.1, replica/AMP, checked against Adam at d ≤ 400). It does not say that real transformers’ spectra arise this way: the data are isotropic Gaussian, the target is inside the model class, and the agreement with measured spectra is qualitative. | Proposed |
| THEORY-181 v3 | In a model that decodes the sum of two learned embeddings, at zero training loss with an injective decoder, a training set fixes the generalising representation of addition when its same-answer pairs leave only translation and scale free; for non-modular addition with p = 10 and one-dimensional embeddings, generalisation sets in near the training fraction, about 0.4, at which that becomes likely | Liu et al. (2022), LIT-858, in a toy built for it: same-answer training pairs force equal embedding sums, equal sums carry answers to unseen pairs, and for addition with p = 10 the fraction at which the forced equations first pin the embedding to a + kb (about 0.4) matches the onset of generalisation and Power et al.’s critical fraction (LIT-341). It is shown for one size, 1D embeddings and an operation built into the architecture. It is not shown that larger networks generalise by this mechanism, and for S₃ the networks generalise beyond it. | Active |
| THEORY-209 v2 | A converged network trained with SGD is not at a point and does not sample its training loss: under a quadratic approximation it samples a Gibbs distribution on a modified loss, isotropic in the Hessian’s column space, with probability currents that keep it oscillating, so its speed and spread are set by the learning rate, momentum and batch size | Kunin et al. (2021), LIT-1011, read in NOTE-711. Modelled as an underdamped Langevin equation with noise covariance proportional to the Hessian, SGD at the end of training converges to a nonequilibrium stationary state: Gibbs on a modified loss whose curvature undoes the Hessian’s anisotropy, with inverse temperature S(1 − β²), in broken detailed balance. A ResNet-18 on ImageNet explores isotropically in a plane where the loss is anisotropic, oscillates at Hessian-dependent rates, and moves at the predicted speed across a hyperparameter sweep. For the learned-structure line, a trained solution is in part a property of the regime that trained it. Proposed: the deep-learning evidence is one network, under assumptions the paper concedes are strained. | Proposed |
| THEORY-210 v2 | Continuous parameter symmetries stretch a network’s minima into connected manifolds along which the loss is flat, and give gradient flow conserved quantities that fix where on such a manifold training ends; finite steps, gradient noise and weight decay break them, so the position along a symmetry orbit is set by initialization and optimizer, not by the loss | Zhao, Walters & Yu (2025), LIT-970, read in NOTE-712: a survey, so the dynamical results are secondhand. Continuous parameter-space symmetries (GL, scaling, orthogonal, translation) make minima positive-dimensional and connected, which is one source of mode connectivity. Under gradient flow each gives a conserved quantity, such as layer imbalance, that is fixed by initialization and locates the endpoint on the manifold. Under SGD these are not conserved. Read first-hand (Neural Mechanics; Ziyin et al.), weight decay drives the conserved quantities toward zero, and minibatch noise drives a rescaling symmetry’s toward the point where the layers’ gradient noise balances. The survey had reported this as imbalance decaying with large learning rates, and a stationary distribution biased toward balanced solutions that a Langevin model does not predict. This is the null-space half that THEORY-209’s limiting dynamics leave out. Proposed: one survey, with the SGD results unread here. | Proposed |
| THEORY-211 v3 | In the stochastic-differential-equation model of SGD, with a gradient-noise covariance that depends on the parameters, minibatch noise drives the difference of squared norms along a rescaling symmetry, at a rate proportional to learning rate over batch size, to the point on the symmetry orbit where the gradient-noise traces of the two parameter groups are equal, whenever both groups carry noise; so at stationarity SGD neither keeps the difference fixed, as gradient flow does, nor lets it diffuse, as isotropic noise does, and differs from gradient descent by a finite amount however small the noise | Ziyin, Li & Ueda (LIT-1012, read in NOTE-714), with the gradient flow conservation law and the Itô term from Neural Mechanics (LIT-1013). If ℓ(u, w) = ℓ(λu, w/λ) for every sample, then under the SDE model of SGD d(‖u‖² − ‖w‖²)/dt = −T(uᵀC₁u − wᵀC₂w), with T = η/S. The noise has no component along the symmetry, but its dependence on the parameters produces a deterministic drift through Itô’s lemma. The drift points to the noise-balanced point, and weight decay adds to its rate. In a diagonal linear network the stationary distribution follows exactly: it is non-ergodic, has a memory of initialization, and collapses to zero above a critical temperature. The law is re-derived here (2026-10-19) within the SDE model; the tests are small synthetic networks, and whether it holds in deep networks on real data is QUESTION-164. | Active |
| THEORY-212 | In a two-layer linear network trained by gradient flow, the relative scale of the two layers, which gradient flow conserves, sets the learning regime: balanced layers learn in sigmoidal stages and both build task-specific representations, while strongly unbalanced layers learn exponentially with a static kernel and leave the larger layer task-agnostic | Dominé et al. (2025), LIT-943, read in NOTE-715. With whitened inputs and a λ-balanced start, W₂ᵀW₂ − W₁W₁ᵀ = λI, the full gradient flow is solved exactly. λ is conserved and sets the regime. As λ → 0, singular values are learned in sigmoidal stages, as in Saxe et al., and both layers become task-specific. As |λ| → ∞, learning is exponential, the NTK is static, and the larger layer stays near the identity while the smaller carries the task at scale ∝ 1/|λ|. The paper analyses gradient flow only. Under SGD the conserved quantity drifts (THEORY-211), which QUESTION-081 takes up. | Proposed |
| THEORY-213 v3 | Every mirror symmetry of a network’s per-sample loss makes its symmetric subspace invariant under gradient descent and SGD, and when the loss’s Hessian is bounded below by −c, weight decay γ‖θ‖² with γ > c/2 puts every minimum in that subspace; rescaling symmetries that hold at λ = −1, rotation symmetries and permutation symmetries contain such mirrors, whose subspaces are zeroed weights, reduced rank and identical neurons | Ziyin (2024), LIT-1014, read in NOTE-716. If a per-sample loss is invariant under the reflection I − 2OOᵀ, then Oᵀθ = 0 is a stationary condition for GD and SGD. The Hessian is block diagonal there. Above a critical weight decay all minima satisfy the constraint. Under SGD with strong noise the subspace is attractive even at a strict saddle, by a Lyapunov-exponent argument. Rescaling, rotation and permutation symmetries imply mirrors, so they yield sparsity, low rank and merged neurons, with collapse and loss of plasticity as consequences. The SGD half is approximate and the network evidence is limited; the title holds only the exact half. Promoted on 2026-10-20 on the exact half, re-derived. The SGD half is QUESTION-206. | Active |
| THEORY-246 v2 | Among ResNet-50s trained on ImageNet with nine objectives and stopped early, seven of which beat softmax cross-entropy’s top-1 by up to 0.9 points, fixed penultimate features with more class separation transfer worse by linear probe to datasets whose classes are mostly not ImageNet’s (Spearman ρ = −0.93 across the nine; mean accuracy 75.9% for softmax to 48.3% for squared error), and within cosine softmax a higher temperature raises separation and lowers that accuracy; the objectives’ representations differ mainly in the last third of the network, and full fine-tuning brings mean transfer to within 1.3 points across them | Kornblith, Chen, Lee & Norouzi (2021), LIT-1044, read in NOTE-762. Across nine objectives for ResNet-50 on ImageNet, penultimate class separation R² runs from 0.349 (softmax) to 0.845 (squared error), Papyan’s Tr(Σ_W Σ_B†)/C from 21.2 to 0.33, and mean linear transfer to eight new-class datasets falls with R² (Spearman ρ = −0.93), from 75.9% to 48.3%. Cosine-softmax temperature raises R² from 0.24 to 0.77 and lowers transfer from 76.0% to 58.9%. CKA puts the differences in the last third of the network; fine-tuning erases them. The cost is for new classes: relearning the training classes, more separation helps. Networks early-stopped near 92% training accuracy; one dataset; correlational across losses. Promoted 2026-10-20 on this one study (ADR-047 v3), narrowed to what it measured. | Active |
| THEORY-331 | Counting the inference a model will serve as well as its training, the least-compute way to reach a fixed quality shifts toward up-front compute as expected inference volume grows: a smaller model trained on more data in Sardana et al., a larger pretrained model over test-time search in Snell et al.; test-time compute won only at low volume and on questions the small model already sometimes solved | Sardana et al. (2024; arXiv 2023), LIT-1070, read in NOTE-881, and Snell et al. (2024), LIT-1063, read in NOTE-869. Sardana et al. minimise training plus lifetime inference FLOPs at a fixed loss under a Chinchilla-fitted law: the optimum moves to smaller models trained longer as known demand grows (28% fewer FLOPs at 30B quality and 10^13 inference tokens), and 47 runs show loss still falling at extreme ratios but more slowly than the law predicts. Snell et al. match FLOPs between a small model with test-time search or revision and a ~14× larger model on MATH: test-time compute wins at a low ratio of inference to pretraining tokens on easier questions and loses at a high ratio, and on the hardest at every ratio. Both take expected use as a known aggregate; neither has staleness or uncertainty about demand, and Snell’s evidence is one model family on 500 questions. | Proposed |
| THEORY-342 v2 | In AlphaZero on 9×9 Hex, at fixed playing strength, training compute and tree-search compute substitute along a straight line in log–log space, about 15× less search for each 10× more training over about two decades of training compute, down to a one-node search; the trade is measured for one search and counts no games played | Jones (2021), LIT-1069, read in NOTE-884. Across about 200 AlphaZero models on Hex boards 3×3 to 9×9, the best Elo for a given training compute follows one change-point curve, shifted by a fixed factor of compute per board row. On 9×9 only, iso-Elo contours of training against test-time search compute are straight in log–log space (fit: log test = −1.2·log train + 0.004·Elo + 29) over roughly 1e14–1e16 FLOPS-seconds of training, ending at a one-node search. One game, one nonstandard implementation, no uncertainty on the fitted coefficients, and no count of how many searches the trained agent will run, so it gives an exchange rate, not an amortization. | Active |
| THEORY-421 v2 | In HGCN embeddings of two synthetic 3-ary trees trained for link prediction or node classification, distance from the origin is only a partial reading of hierarchical level: the root is not innermost (distance 1.8–3.3 against a minimum of 1.2–2.4) and level order is recovered for 69–75% of node pairs; a loss built only on pairwise distances leaves the origin arbitrary | Yang, Zhou, Ying, Chen & King (2023), LIT-871, read in NOTE-669. On two synthetic 3-ary trees of eight levels, HGCN leaves the root at hyperbolic distance 1.8–3.3 from the origin where the minimum is 1.2–2.4, the distances are roughly normal rather than leaf-heavy, and radius orders true levels for 69–75% of 5,000 random pairs. Recentring on the centroid plus an outward push raises that to 75–84%. The diagnosis rests on one architecture, two synthetic trees and one table without variance, and the paper does not separate a real failure to order levels from the origin’s arbitrariness under losses that see only pairwise distances. | Active |
| THEORY-425 | A connecting path trained only on clean data keeps only the behaviour the clean data ask for: between two backdoored or error-injected image classifiers, models near the ends of such a path keep most clean accuracy and lose the attack, and a path flat in ordinary loss can carry a barrier in adversarial loss that tracks the input Hessian’s top eigenvalue | Zhao, Chen, Das, Ramamurthy & Lin (2020), LIT-668, read in NOTE-530. A quadratic Bezier path trained on 50–2,500 clean samples between two backdoored VGG-16s on CIFAR-10 has, at t = 0.1, 88% clean accuracy and 1.1% backdoor success with 2,500 samples, and 63% and 2.5% with 50; injected weight errors fail at every point of such paths. Paths trained with the standard loss between regular and adversarially trained models are flat in standard loss with a PGD-loss barrier correlated with the input Hessian’s top eigenvalue (0.71–0.88). The proposition offered for that correlation rests on an assumption that would remove the barrier. | Active |
| THEORY-481 v4 | Under gradient flow averaged over tasks and data, a linear-attention layer trained on in-context regression learns as a two-layer linear network from an aligned start does: when tasks share the input covariance’s eigenbasis and the weights start and stay aligned with it, then in the long-context limit each input mode is learned on its own logistic curve whose timescale falls with the square of its variance, so learning is staged by the input spectrum, and the layer converges to the estimator preconditioned by the inverse input covariance; at finite context length the modes decouple exactly only when inputs are isotropic and the modes start equal, and otherwise couple at order 1/N | Mainali & Teixeira (2025), LIT-859, read in NOTE-658. The prediction of one linear-attention layer reduces to p₂ W Σ̂_x q₁ x_q, a product of two learned factors, so averaged gradient flow decouples into modes with τ_α dp_α/dt = q_α(1 − p_α q_α s^∞_α) and τ_α = (ηP s_α²)⁻¹, logistic from a balanced start. The fixed point is ŷ = W Σ̂_x Σ_x⁻¹ x_q as N → ∞. The paper’s finite-N form, the one Ahn et al. and Zhang, Frei & Bartlett found, holds at finite N only for isotropic inputs with equal modes. The trajectory is new. The decoupling into independent modes is exact as N → ∞. At finite N it holds only for isotropic inputs with equal modes: the paper commutes the non-diagonal empirical covariance at (A71) and (A73), and the correct Wishart moment couples the modes at order 1/N. Alignment of the weights with the data’s eigenbasis is assumed throughout, the null blocks vanish only in expectation over tasks, and the transfer to non-linear transformers is timing coincidences. | Active |
| THEORY-496 | In few-shot classification and multiple choice by language models up to 175B parameters, whether each demonstration’s label is correct contributes little to the gain over zero-shot, while the distribution of the inputs, the label space and the paired format carry most of it | Min et al. (2022), LIT-814, read in NOTE-607. With k = 16 demonstrations, labels drawn at random from the correct label set cost 0–5 points against gold labels (2.6 on classification, 1.7 on multiple choice on average) across twelve model–inference combinations and 26 datasets. Out-of-distribution inputs, random English words as labels, or breaking the input–label pairing each cost far more. The random labels always come from the right label set, gaps reach about 14 points on single datasets, and the authors’ reading that the models learn no new correspondence at test time is untested on tasks pretraining cannot have taught. | Active |
| THEORY-504 v2 | In the Gemma Scope sparse autoencoders on Gemma 2, the two fidelity measures disagree by site: residual-stream autoencoders cost more next-token loss than MLP or attention ones at comparable fraction of variance unexplained, and base-model autoencoders transfer to the instruction-tuned model by loss more than by variance explained | Lieberum et al. (2024), LIT-611, read in NOTE-482. Across every layer and sublayer of Gemma 2 2B and 9B, JumpReLU autoencoders on the residual stream raise the language model’s loss more, when spliced in, than MLP and attention autoencoders whose FVU is about the same; and autoencoders trained on the base model lose little loss on instruction-tuned rollouts while showing a larger FVU gap. The authors’ explanations (the residual stream is the bottleneck; fine-tuning re-weights old features) are conjectures, and the authors say no agreed metric of autoencoder quality exists. | Active |
| THEORY-509 | Under genericity conditions, two dense or top-k (k ≥ 2) mixture-of-experts layers compute the same function only if they differ by a permutation of the experts with their gates and a common translation of the gate weights, and the translation leaves every interpolation barrier unchanged, so expert order is the only gate symmetry alignment must undo | Tran, Trinh, Bui & Nguyen (2025), LIT-670, read in NOTE-513. Softmax is invariant to a common shift of scores and the sum to the order of terms, so G(n) = ℝ^d × ℝ × S_n preserves the layer; Theorems 4.1–4.2 show that generically it is all of the gate’s symmetry, for dense gating and for top-k with k ≥ 2 (top-1 has an extra scaling). Aligning expert order and then each expert’s hidden units gives barriers of 0.0044–0.0066 between GPT-2 MoE layers fine-tuned from different seeds, against 0.14–0.86 without the order step. Every experiment shares and freezes the backbone around the inserted layer, so it is not evidence about independently trained MoE models. | Active |
| THEORY-510 | How much of an ImageNet image a crop must keep for its label to survive varies widely by class, with architectures agreeing on which classes are fragile, and strengthening random crop raises mean accuracy while some classes lose 20 points or more; how many classes lose is not established | Balestriero, Bottou & LeCun (2022), LIT-234, read in NOTE-845. Six pretrained models on cropped test images keep some classes recognisable at an 8% crop and lose others below about 50%, and agree on which. Over 20 ResNet-50 runs, moving the crop lower bound from 100% to 8% raises mean accuracy from about 70% to 76% while “academic gown” falls from about 62% to 40%. The classes shown are hand-picked; the per-class test rejects for 4.4% of classes at the 5% level, uncorrected for 1,000 tests, which alone is not more than chance; and evaluation-set noise is not counted. Theorem 1 does not reach which classes pay. | Proposed |
| THEORY-521 | In the thermodynamic limit, with examples and capacity growing together at fixed load, generalization error can jump discontinuously with the load when the entropy of hypotheses just above the best error vanishes faster than the training-error energy grows: a spherical perceptron’s error falls smoothly as 1/α, an Ising perceptron’s drops to zero at a critical load | Martin & Mahoney (2017), LIT-672, read in NOTE-542, restating Seung, Sompolinsky & Tishby (1992) and Haussler, Kearns, Seung & Tishby (1996), which the record does not hold. Generalization is set by maximizing s(ε) − α ln(1/(1 − ε)). For weights on a sphere s(ε) ~ ln ε and ε ~ 1/α; for ±1 weights s(ε) ~ −(π²/2)ε² ln ε, the first-order condition has no solution past a critical load, and the error drops to zero. Uniform bounds that see a class only through its size or VC dimension are smooth and blind to this. That deep networks have such a phase diagram is the authors’ conjecture, and their mapping of label noise to load and early stopping to temperature is an analogy. | Proposed |
| THEORY-557 | Low-loss paths between independently trained solutions are found outside supervised image classifiers: graph neural networks trained from different seeds on one graph are joined by quadratic Bezier curves where the straight line has a barrier, and one pair each of diffusion and contrastive models by chains of low-loss checkpoints | Li et al. (2025), LIT-676, read in NOTE-529: GCN node classifiers from different seeds, on 12 real graphs and synthetic CSBM graphs, show linear barriers that a fitted quadratic Bezier curve removes. Yao, Zhang & Tian (2026), LIT-660, read in NOTE-534: one pair of DDPMs and one pair of small CLIP models joined by chains of checkpoints whose training loss stays under a threshold, where AutoNEB’s paths rise steeply on CLIP. The evidence is plots without barrier numbers in the first, one pair per model and retrained anchors in the second. Zhang & Blei’s claim of flat paths between LDA’s ELBO maxima (LIT-677) reports no results. | Proposed |
| THEORY-566 v2 | In Ganea et al.’s hyperbolic neural networks, linear maps and nonlinearities are Euclidean ones read through the chart at the origin, so a stack of them without biases is a Euclidean network conjugated by that chart, and the curvature acts only through the bias translations and the distance-based output layer | Ganea, Bécigneul & Hofmann (2018), LIT-879, read in NOTE-686, which checked the closed forms numerically. Möbius layers are defined as exp₀ ∘ f ∘ log₀, so any composition of them is exp₀ ∘ (the Euclidean composition) ∘ log₀, as the paper notes; only the Möbius bias, which is parallel transport from the origin followed by the exponential map, and the hyperbolic logistic regression’s distances to geodesic hyperplanes are not absorbed by the chart. The structural fact is proved; that the networks’ gains, confined to a synthetic task and to separating WordNet subtrees in low-dimensional Poincaré embeddings, come from those two operations is this record’s inference. NOTE-686 filed no THEORY. | Active |
| THEORY-578 | Between two trained networks that compute nearly the same function, a chain of models minimising the summed Jensen–Shannon divergence between neighbours, a discretised Fisher–Rao geodesic found without labels, can be a low-loss path where the straight line in weights has a barrier | Tan, Long, Zhao & Laine (2023), LIT-662, read in NOTE-521. Two ResNet-20s at 4× width with LayerNorm, CIFAR-10, permuted by Git Re-Basin’s weight matching: the straight line between them still has a large loss barrier, and a chain of 25 models optimised from it to minimise Σ JSD between neighbours on unlabelled training images keeps train and test loss low. One pair, one figure, no numbers in the text. The hypothesis that every geodesic between SGD solutions is mode-connecting is stated, not tested, and a chain whose neighbours agree in output may be low-loss almost by construction. | Proposed |
| THEORY-675 | Without a shared stable start, the linear barrier between solutions of one objective depends on the structure of the training data: between GCNs it falls with graph density and feature separability and is highest at middling homophily, and between federated-averaging models it grows with client heterogeneity and falls with width | Li et al. (2025), LIT-676, read in NOTE-529: on synthetic CSBM graphs the barrier between GCNs from different seeds falls with density and feature separability and peaks at middling homophily; GCN, GraphSAGE and GAT barriers are similar, an MLP’s differ. Zhou, Zhang & Tsang (2023), LIT-666, read in NOTE-517: FedAvg global models from one random start under more heterogeneous clients are separated by a larger line barrier, joined by a one-bend chain, and the barrier falls with width. Both readings named this account and held it back. The GNN proofs are missing and its barriers are plots; the federated bound treats heterogeneity as isotropic noise, and its lemma and theorem conditions conflict as written. | Proposed |
| THEORY-683 | Permuting a fixed set of few-shot examples moves a GPT-family model’s classification accuracy between near chance and near the supervised state of the art, at every model size tested; the good orders are specific to the model, the spread survives more examples and calibration, and bad orders mostly fail by predicting one label | Lu et al. (2021, ACL 2022), LIT-833, read in NOTE-626. All 24 orders of four balanced SST-2 examples give accuracies from about 50% to above 85%, in GPT-2 and GPT-3 models of eight sizes; Spearman correlations of order accuracy between sizes lie between −0.35 and 0.27 (175B against 2.7B: 0.05). The variance stays high up to 32 examples and after calibration. It is a measurement on causal GPT models and text classification, with the transfer analysis resting on one example set, and the paper finds no property of an order that predicts its effect. | Active |
| THEORY-718 | In a linear-in-parameters model with Gaussian noise and a Gaussian prior, the evidence is exactly the best-fit likelihood times an Occam factor set by the posterior curvature, and when the design grows linearly with n that factor costs (k/2) log n | MacKay (1992), LIT-623, Eqs. 5–6 with Eqs. 16 and 19–20, read in NOTE-471: for y = Φw + noise with Gaussian noise of precision β and the prior w ~ N(0, α⁻¹I) on k weights, the posterior is exactly Gaussian, so the evidence is exactly P(D|w_MP) times the Occam factor P(w_MP)(2π)^{k/2} det^{-1/2}A, with A = αI + βΦᵀΦ. If ΦᵀΦ/n converges to a positive-definite matrix and w_MP stays bounded, the log Occam factor is −(k/2) log n + O(1). Exact for this model class only. The general regular case and the singular case are THEORY-087, which is Proposed. | Active |
| THEORY-736 | Two ε-dropout-stable solutions of a multilayer ReLU net, under a loss convex in the output, are joined by a piecewise-linear path whose loss never exceeds the worse endpoint’s by more than ε, and joined pairwise paths compose with no higher barrier than their worst | Kuditipudi et al. (2019), LIT-671, Theorem 1, read in full in NOTE-544, with Draxler et al.’s (2018) concatenation bound from LIT-653. Dropout stability gives every solution a half-width copy with the same loss up to ε. Half the units are then free, so working units can be swapped in and permuted at no cost. That joins any two such solutions. It is a statement about dropout-stable solutions only: whether training produces them is THEORY-109’s open question, and Theorem 4 of the same paper builds disconnected global minima. | Active |
| THEORY-741 | A two-layer ReLU student of width at least kM, fitting an orthonormal teacher of M neurons on the sphere, has for any k global minima a global minimum joined to each by a two-piece linear path of minima; and in a deep linear network of depth L and width above 1 + r(L − 1), any r global minima in general position share a global minimum linearly connected to each | Lin, Li and Wu (2024), LIT-682, Theorems 6, 11 and 18 with Lemmas 7 and 25, read in NOTE-535 and re-derived by the promotion audit. Two toy models whose minima can be written down: a two-layer ReLU student of an orthonormal teacher on the sphere at population loss, and a deep linear network fitting y = Qx. Spare width lets every minimum be moved, along segments of minima, to one a common centre can reach. No permutation symmetry is used, and nothing is claimed about trained networks. | Active |
| THEORY-779 | In Kunin et al.’s underdamped Langevin model of SGD with momentum on linear regression, with gradient-noise covariance σ²H independent of the parameters, the stationary state is Gaussian with parameter covariance ησ²H(H + λI)⁻¹/(2S(1 − β)) and velocity covariance σ²H/(S(1 − β²)): for small weight decay it spreads equally along every nonzero Hessian direction and not along the null space, each Hessian eigenmode is a damped oscillator with damping ratio (1 − β)/√(2η(1 + β)(ρ + λ)), and the expected squared step is η²σ² tr(H)/(S(1 − β²)) | Kunin et al. (2021), LIT-1011, §§4–6 and App. G, read in NOTE-711; the stationary moments re-derived by the promotion audit. An exact result for a model: Σ ∝ H constant and a quadratic loss are assumptions, and nothing here says a deep network meets them. The probability current is nonzero but reversible: under the model’s assumptions the stationary process is time-reversible once velocities are flipped, so it is “broken detailed balance” only in the paper’s sense, a nonzero current. | Active |
| THEORY-781 | Strong gradient noise in SGD, like weight decay above a threshold, makes the subspace fixed by a mirror symmetry of a network’s loss attracting even where it is a saddle, so that networks trained with SGD at large learning rates or small batches reach sparse, low-rank and identical-neuron solutions, as Ziyin proposes | Ziyin (2024), LIT-1014, read in NOTE-716. Beyond the exact half that THEORY-213 holds, the paper argues by a Lyapunov-exponent analysis that strong SGD noise makes a mirror’s subspace attracting even at a strict saddle, and reads sparsity, low rank and identical neurons in trained networks as its result. That analysis rests on a second-order, diagonal- Hessian approximation, and the network evidence is a ResNet18’s correlation blocks. The test is QUESTION-206. | Proposed |
| THEORY-782 | In networks that learn addition from summed embeddings, the critical training fraction of grokking is the fraction at which the training set’s same-answer pairs come to fix the generalising representation up to translation and scale, at every group size, as Liu, Kitouni, Nolte, Michaud, Tegmark and Williams propose | Liu, Kitouni, Nolte, Michaud, Tegmark and Williams (2022), LIT-858, read in NOTE-662, with Power et al.’s critical fraction (LIT-341). The paper presents the critical training fraction as where the training set’s equations fix the embedding. THEORY-181 holds the theorem for the ideal model and the match at the one size measured, p = 10 with 1D embeddings. Held to addition: for S₃ the networks generalise beyond this mechanism. The multi-size test is QUESTION-199. | Proposed |
| THEORY-788 | Training a deep classifier past zero training error generally collapses its last-layer features onto their class means, drives the centred means toward a simplex equiangular tight frame aligned with the classifier, and makes the decision nearest class mean, as Papyan, Han and Donoho propose from their runs | Papyan, Han and Donoho (2020), LIT-618, title their paper on the “prevalence” of neural collapse and present it as a pervasive outcome of the terminal phase of training. The record holds as shown, in THEORY-083, only what their runs measured: three architectures with cross-entropy on seven balanced datasets without augmentation. This is the general law, filed under ADR-047 v4 when the narrowing dropped it. | Proposed |
| THEORY-790 | In deep networks trained by SGD on real data, minibatch noise drives parameter groups related by a rescaling symmetry toward the point where their gradient noise balances, at a rate set by learning rate over batch size, so that at stationarity they sit neither where gradient flow would leave them nor spread out as under isotropic noise, as Ziyin, Li and Ueda propose | Ziyin, Li & Ueda, LIT-1012, read in NOTE-714, with the Itô term from LIT-1013. The law of balance is proved in the SDE model of SGD, and that part is the Active THEORY-211. The authors present the law as general to networks with rescaling symmetry, but test it only on small synthetic networks. This THEORY holds the general claim, for deep networks on real data, which no measurement in the record yet supports. | Proposed |
| THEORY-794 | In vision models, compositional generalization to unseen combinations of concepts is driven by the diversity of the combinations seen in training rather than by data volume, and arrives with representations that are linearly factored by concept, as Uselis, Dittadi and Oh propose | Uselis, Dittadi & Oh (2025), LIT-667, read in NOTE-525. Their title and abstract present diversity of combinations, not data volume, as what drives compositional generalization in vision models. Their evidence is ResNet-50s trained from scratch on two-concept grids, with checkpoints chosen on the unseen pairs, and that measured part, with the factorization theorem, is the Active THEORY-117. This THEORY holds the general causal claim. | Proposed |
| THEORY-798 | Independently trained neural encoders of the same data learn latent spaces that differ mainly by an angle-preserving map, a rotation or reflection times a rescaling, so their cosine similarities to shared anchor samples agree across seeds, architectures and languages, as Moschella et al. propose | Moschella, Maiorca, Fumero, Norelli, Locatello & Rodolà (2022; ICLR 2023), LIT-849, read in NOTE-652. The hypothesis behind relative representations, filed apart from what their experiments measured (THEORY-175) under ADR-047 v4. The source assumes it and tests it only through its consequences; it never fits a map between two spaces. | Proposed |
| THEORY-825 | Hyperbolic graph embedding models trained on task losses that never mention the hierarchy do not place a tree’s root nearest the origin, and their distance from the origin orders hierarchical levels only partly | Yang, Zhou, Ying, Chen & King (2023), LIT-871, read in NOTE-669, assert for hyperbolic graph models in general what they measured for HGCN on two synthetic trees: the root is not innermost and radius reads level only partly. The general form was the lead clause of THEORY-421 until that account was narrowed to the measurement on 2026-10-20 (ADR-047 v4); it is filed here so that it stays a thesis evidence can confirm or reject. | Proposed |
| THEORY-828 | Fine-tuning from one pretrained model typically lands near the edge of a task-specific low-loss region in weight space that contains the convex hull of the fine-tuned models, and averaging their weights, as model soups and stochastic weight averaging do, works by moving inward | Gueta, Venezian, Raffel, Slonim, Katz and Choshen (2023), LIT-658, read in NOTE-515, conclude that fine-tuning “typically” yields models on the boundary of a region whose centre is better, and read model soups and SWA as picking points inside it (§§8–9). The record holds as shown, in THEORY-105, only the probe-loss comparisons of hull points with members and the BitFit result, for RoBERTa-base on English classification. This is their geometric thesis, filed under ADR-047 v4 when the narrowing dropped it. | Proposed |
| THEORY-1172 | Linear mode connectivity emerges during training, not at initialization: copies trained from a shared state become stable to SGD noise after 1.5–20% of training, and independently trained networks become linearly connected after alignment only gradually | The wide form THEORY-114 held before the tenth status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-114, Active, is its evidence. | Proposed |
| THEORY-1185 | Merging models trained on different tasks from different initialisations fails where their features have no counterpart in the other model, and such features grow more common with depth | The wide form THEORY-113 held before the eleventh status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-113, Active, is its evidence. | Proposed |
| THEORY-1187 | Along a low-loss linear path between two networks, every layer’s features are the interpolation of the endpoints’ features | The wide form THEORY-111 held before the eleventh status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-111, Active, is its evidence. The old title’s averaging clause was the record’s own step and is a Proposed claim of its own; the promote_when here is the layerwise half of the old one. | Proposed |
| THEORY-1188 | Once the architecture’s symmetries are factored out, most of the linear barrier between independently trained solutions disappears; the group that matters is the full one, permutations for MLPs and CNNs but for transformers also an orthogonal map on the residual stream, without which a barrier remains; and the barrier left after alignment falls with width | The wide form THEORY-112 held before the eleventh status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-112, Active, is its evidence. The old title’s last words, that the barrier left after alignment rises with depth, were the record’s composition, not a source’s, and are a Proposed claim of their own. | Proposed |