Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Loss landscapes — the geometry of trained solutions in weight space — mode connectivity, linear and along curves, barriers between minima and the permutation symmetries that hide or remove them, re-basin alignment, star domains and connecting simplexes, local versus global landscape structure, and model merging as its use (ADR-027).
39 of 1352 THEORY documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| THEORY-105 v2 | For RoBERTa-base fine-tuned on English classification data, with loss measured by a linear probe refitted on the frozen encoder, points sampled from the convex hull of models fine-tuned on one dataset, one task family or twelve GLUE and SuperGLUE datasets lost no more than any model fine-tuned elsewhere and beat the group’s own members in 88% to 96.7% of comparisons, and the centroid of the general-level models was a better starting point for BitFit than the pretrained model | Gueta et al. (2023), LIT-658: RoBERTa-base models fine-tuned on the same dataset, the same task family, or any of 12 GLUE and SuperGLUE datasets cluster in weight space. Sampled points in the convex hull of each group beat the group’s own members in 88% to 96.7% of comparisons. Extrapolating past the members raises the loss quickly, so the members sit near the region’s edge and averaging moves inward. The paper calls the region convex, but shows only that the hull lies inside it, by a probe-based loss. Its reading of model soups and SWA as picking points inside the region is argued in its §8, not tested. Moving from the centroid towards the origin stays low-loss far past the members, which the authors cannot explain. Promoted 2026-10-20 on the probe-loss comparisons measured (ADR-047 v3); the geometric thesis is THEORY-828. | Active |
| THEORY-107 | Merging models fine-tuned from one pretrained start loses accuracy through interference in parameter coordinates: redundant small updates dilute the influential ones, and influential updates of different models conflict in sign | Yadav et al. (2023), LIT-675: keeping only the top 20% of each task vector by magnitude loses almost nothing, so most of an update is redundant. Among the entries that remain, signs conflict between models, more so as more models are merged. Merging (IA)³ models with the multitask model’s signs reaches 72.0 against 73.1 for multitask training and 66.4 for TIES with elected signs. That is the strongest evidence that sign is the bottleneck. But the oracle signs come from training on every task’s data, the ablation finds rescaling and the disjoint mean matter more than election, and conflicts are as frequent among checkpoints of one task as across tasks. | Proposed |
| THEORY-108 | Across load and temperature, the connectivity and output similarity of independently trained solutions predict test accuracy where endpoint curvature does not: a flat but poorly connected landscape generalizes worse than a connected one of the same curvature | Yang, Hodgkinson, Theisen, Zou, Gonzalez, Ramchandran & Mahoney (2021), LIT-663: thousands of trained networks on a load by temperature grid, measured by the endpoint Hessian, by mode connectivity on a trained Bezier curve between two independent runs, and by CKA of their outputs. Phases III (flat, poorly connected) and IV-A (flat, connected) have almost the same Hessian and different accuracy; with 10% noisy labels the sharp, unconverged phases beat phase III; label noise barely moves the Hessian while connectivity and CKA fall. The best accuracy is in the flat, connected, high-similarity phase IV-B. The counterexamples are to the anthology’s ANTH-SOTA-012. One architecture family carries most of it, phases are read by eye, and connectivity alone also fails when the load is data amount. | Proposed |
| THEORY-109 v2 | The solutions SGD finds in a deep network lie in one connected low-loss set, joined by simple curves and spanning volumes of measurable dimension, and this is a property of the solutions training finds, not of every minimum | Garipov et al. (2018), LIT-673, and Draxler et al. (2018), LIT-653, found independently that minima of deep networks trained from different initializations are joined by paths of near-constant loss, a one-bend curve or an AutoNEB path, while the straight line between them climbs to near-chance error. Benton et al. (2021), LIT-656, widened curves to simplicial complexes joining up to seven modes and a low-loss region of at least 10 dimensions. Kuditipudi et al. (2019), LIT-671, proved that dropout-stable solutions are always joined, and constructed global minima that are not, so connectivity belongs to found solutions. Proposed: the evidence is pairs and small complexes of CIFAR classifiers. | Proposed |
| THEORY-110 v2 | A set of trained solutions has a centre that is linearly connected, after alignment, to the rest of them: the solution set is a star domain even where it is not convex | Sonthalia et al. (2024), LIT-659, conjecture that below the width at which SGD solutions become convex modulo permutation they are still a star domain, and train a star model whose barriers to held-out solutions, after weight matching, are about 2–5 times lower than those between solutions (ResNet-18 CIFAR-10 0.078 against 0.383; ImageNet 2.794 against 5.948), and not zero. Lin et al. (2024), LIT-682, prove star-shaped linear connectivity for wide two-layer teacher–student and deep linear networks, and find centres for five trained image classifiers with no permutation. Proposed: reduced, not zero, and no held-out test without alignment. | Proposed |
| THEORY-111 v3 | In image classifiers, between two networks linearly connected by spawning from a shared checkpoint or by permutation alignment (MLPs, and VGG-16, ResNet-20 and ResNet-50 on CIFAR-10 and Tiny-ImageNet), nearly every layer’s features at three points on the line were proportional to the interpolation of the endpoints’ features, and pairs that were not linearly connected were far from it | Zhou et al. (2023), LIT-674: whenever two image classifiers are linearly mode connected, by spawning or by permutation, the features of nearly every layer of the interpolated network are proportional to the interpolation of the endpoints’ features. This is layerwise linear feature connectivity, and two conditions, weak additivity of ReLU and a commutativity of weights with features, imply it exactly. Lubana et al.’s Lemma 2, LIT-669, gives the exact one-hidden-layer case: linear connectivity between interpolating minima forces shared activation patterns, and with shared patterns the hidden features are linear along the path. Measured on pairs at three interpolation points, image classification only. The regularity outside image classification, and the step to many-model averages such as soups and SWA, are QUESTION-318. | Active |
| THEORY-112 v2 | Aligning independently trained solutions over the architecture’s symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path’s midpoint, took it to zero or near zero except on BookCorpus (0.42) | Entezari et al. (2022), LIT-652, conjectured that SGD solutions are one basin modulo permutation. Git Re-Basin (2023), LIT-661, aligned MNIST MLPs and 32×-wide ResNet-20s to zero barrier, but not 1×-width models or ImageNet ResNet-50. Ferbach et al. (2024), LIT-680, proved it for wide nets with independent neurons, with a width requirement exponential in the previous layer’s width. Theus et al. (2025), LIT-664, took transformers to zero only with an orthogonal map on the residual stream added; permutations alone left 0.29–1.60. Bo Zhao et al. (2025), LIT-655, place permutations inside the full symmetry group. Lubana et al.’s barriers between models that compute differently are the boundary. Whether most of the barrier goes at standard width on ImageNet or a language corpus under data-free alignment, and whether the barrier left rises with depth, is QUESTION-317. | Active |
| THEORY-113 v2 | Merging ResNets trained from different initialisations on disjoint halves of a label set by permuting one onto the other and averaging fell far short of the ensemble; their cross-model feature correlation fell with depth, merging only the early layers recovered most of the ensemble’s accuracy, and merging features within each model as well as across both helped at 4× width and above; Stoica et al. attribute the failure to features with no counterpart in the other model | Stoica et al. (2023), LIT-665, merge ResNets trained from different initialisations on disjoint label sets. Permute-then-average fails, which they explain by the permuted model lying in a basin for its own task only. Mean cross-model feature correlation falls with depth (0.50, 0.37, 0.27 at three depths). Merging only the early layers and keeping later layers as task heads recovers most of the ensemble’s accuracy. Merging features within one model as well as across both helps once the models are wide enough to hold redundant features. With full merging on ImageNet-1k, ZipIt! is no better than permutation (8.6%). No same-task control separates the effect of task difference from that of depth; that is QUESTION-316. | Active |
| THEORY-114 v2 | In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training | Frankle et al. (2020), LIT-654: two copies trained from step k under different SGD noise are linearly connected only once k is a little way in, at 1.5–3% of training for VGG-16 and ResNet-20 on CIFAR-10 and 16–20% for Inception-v3 and ResNet-50 on ImageNet. Git Re-Basin (2023), LIT-661: between independently trained MLPs, the barrier after permutation alignment is high at initialization and falls gradually over training. Entezari et al.’s theorem proves alignment-LMC at random initialization; Ferbach et al. (2024), LIT-680, show the width that needs grows recursively exponentially with depth, far past practical widths, which is why the two are consistent. | Active |
| THEORY-115 v2 | Linear connectivity from a shared, already-stable start holds only when the objectives the runs optimise share structure, and breaks as that structure is removed | Mirzadeh et al. (2020), LIT-651: from a task-1 solution, a model trained on tasks 1 and 2 together stays on a low task-1 loss line from the start, and one trained on task 2 alone does not. Adding noise to task 2’s inputs, corrupting its labels, or removing classes separates the solutions progressively. The authors explain it by curvature: the multitask direction avoids the earlier task’s top Hessian eigenvectors. Gueta et al. (2023), LIT-658: fine-tuning runs from one pretrained model are linearly connected within a dataset and a task family. Wen et al. (2023), LIT-681: in class-incremental learning the line between adjacent continual minima crosses a ridge in the old tasks’ loss. Limits: much of the measured breakage is forgetting at the endpoint, not a barrier, and FedAvg models from one random start under one global objective are separated by a barrier that grows with heterogeneity. | Proposed |
| THEORY-116 v2 | A linear barrier between two minima that survives the removal of symmetries marks a difference in the input attributes the two models rely on: they compute by different mechanisms | Lubana et al. (2022), LIT-669: models that rely on a synthetic cue and models that ignore it are separated by a linear barrier even after permutation matching. Fine-tuned models that keep a linear path to their parent keep its cue, and those that lose the path lose the cue. Their Conjecture 1 states this claim’s direction. What their Appendix F.3 proves, for a one-hidden-layer ReLU network with interpolating minima, is the converse: linear connectivity forces identical activation patterns, so models relying on attributes of different complexity cannot be linearly connected under any permutation. Zhou et al. (2023), LIT-674, support that converse in deep image classifiers, where linearly connected pairs have features that blend layer by layer. The stated direction rests on experiments with synthetic cues and on a one-layer sufficiency argument. | Proposed |
| THEORY-184 v2 | Under a permutation-equivariant update whose map is K-Lipschitz, coincident neurons stay coincident at every step size, and at step size below 1/K distinct neurons cannot merge in finitely many steps | Yang, Poggio, Chuang and Ziyin (2025), LIT-861, read in NOTE-663. If an update commutes with permuting neurons (GD, SGD or Adam on a permutation-symmetric loss) and its map is K-Lipschitz, one step of size η changes every pairwise neuron distance by a factor in [1 − ηK, 1 + ηK]. This is a proved constraint on trajectories. It does not say that training simplifies above 1/K, that neurons never merge in the limit, or that any scale-dependent shape of the neuron cloud is preserved. | Active |
| THEORY-209 v2 | A converged network trained with SGD is not at a point and does not sample its training loss: under a quadratic approximation it samples a Gibbs distribution on a modified loss, isotropic in the Hessian’s column space, with probability currents that keep it oscillating, so its speed and spread are set by the learning rate, momentum and batch size | Kunin et al. (2021), LIT-1011, read in NOTE-711. Modelled as an underdamped Langevin equation with noise covariance proportional to the Hessian, SGD at the end of training converges to a nonequilibrium stationary state: Gibbs on a modified loss whose curvature undoes the Hessian’s anisotropy, with inverse temperature S(1 − β²), in broken detailed balance. A ResNet-18 on ImageNet explores isotropically in a plane where the loss is anisotropic, oscillates at Hessian-dependent rates, and moves at the predicted speed across a hyperparameter sweep. For the learned-structure line, a trained solution is in part a property of the regime that trained it. Proposed: the deep-learning evidence is one network, under assumptions the paper concedes are strained. | Proposed |
| THEORY-210 v2 | Continuous parameter symmetries stretch a network’s minima into connected manifolds along which the loss is flat, and give gradient flow conserved quantities that fix where on such a manifold training ends; finite steps, gradient noise and weight decay break them, so the position along a symmetry orbit is set by initialization and optimizer, not by the loss | Zhao, Walters & Yu (2025), LIT-970, read in NOTE-712: a survey, so the dynamical results are secondhand. Continuous parameter-space symmetries (GL, scaling, orthogonal, translation) make minima positive-dimensional and connected, which is one source of mode connectivity. Under gradient flow each gives a conserved quantity, such as layer imbalance, that is fixed by initialization and locates the endpoint on the manifold. Under SGD these are not conserved. Read first-hand (Neural Mechanics; Ziyin et al.), weight decay drives the conserved quantities toward zero, and minibatch noise drives a rescaling symmetry’s toward the point where the layers’ gradient noise balances. The survey had reported this as imbalance decaying with large learning rates, and a stationary distribution biased toward balanced solutions that a Langevin model does not predict. This is the null-space half that THEORY-209’s limiting dynamics leave out. Proposed: one survey, with the SGD results unread here. | Proposed |
| THEORY-211 v3 | In the stochastic-differential-equation model of SGD, with a gradient-noise covariance that depends on the parameters, minibatch noise drives the difference of squared norms along a rescaling symmetry, at a rate proportional to learning rate over batch size, to the point on the symmetry orbit where the gradient-noise traces of the two parameter groups are equal, whenever both groups carry noise; so at stationarity SGD neither keeps the difference fixed, as gradient flow does, nor lets it diffuse, as isotropic noise does, and differs from gradient descent by a finite amount however small the noise | Ziyin, Li & Ueda (LIT-1012, read in NOTE-714), with the gradient flow conservation law and the Itô term from Neural Mechanics (LIT-1013). If ℓ(u, w) = ℓ(λu, w/λ) for every sample, then under the SDE model of SGD d(‖u‖² − ‖w‖²)/dt = −T(uᵀC₁u − wᵀC₂w), with T = η/S. The noise has no component along the symmetry, but its dependence on the parameters produces a deterministic drift through Itô’s lemma. The drift points to the noise-balanced point, and weight decay adds to its rate. In a diagonal linear network the stationary distribution follows exactly: it is non-ergodic, has a memory of initialization, and collapses to zero above a critical temperature. The law is re-derived here (2026-10-19) within the SDE model; the tests are small synthetic networks, and whether it holds in deep networks on real data is QUESTION-164. | Active |
| THEORY-213 v3 | Every mirror symmetry of a network’s per-sample loss makes its symmetric subspace invariant under gradient descent and SGD, and when the loss’s Hessian is bounded below by −c, weight decay γ‖θ‖² with γ > c/2 puts every minimum in that subspace; rescaling symmetries that hold at λ = −1, rotation symmetries and permutation symmetries contain such mirrors, whose subspaces are zeroed weights, reduced rank and identical neurons | Ziyin (2024), LIT-1014, read in NOTE-716. If a per-sample loss is invariant under the reflection I − 2OOᵀ, then Oᵀθ = 0 is a stationary condition for GD and SGD. The Hessian is block diagonal there. Above a critical weight decay all minima satisfy the constraint. Under SGD with strong noise the subspace is attractive even at a strict saddle, by a Lyapunov-exponent argument. Rescaling, rotation and permutation symmetries imply mirrors, so they yield sparsity, low rank and merged neurons, with collapse and loss of plasticity as consequences. The SGD half is approximate and the network evidence is limited; the title holds only the exact half. Promoted on 2026-10-20 on the exact half, re-derived. The SGD half is QUESTION-206. | Active |
| THEORY-278 v3 | Under MSE loss, in Han, Papyan and Donoho’s VGG, ResNet and DenseNet runs on five balanced ten-class datasets, the trained classifier stays near the least-squares classifier for its own features from an early epoch; and on that central path, for unconstrained features continually renormalized to identity within-class covariance and without weight decay, gradient flow drives the class-separation singular values to infinity at equal rates, which in that model is neural collapse | Han, Papyan & Donoho (2021), LIT-974, read in NOTE-783. MSE loss splits exactly into a least-squares term and a non-negative deviation term, and the deviation becomes negligible early in training in 15 dataset–network cells. On that central path, with λ = 0, the loss is ½ Σ 1/(ω_j² + C) in the singular values of Σ_W^(−1/2)M̃. Under a flow that re-whitens the features after every step, each ω_j grows as (4t/N)^(1/4), and their ratios tend to 1. That is NC1 and NC2, and NC3–NC4 follow. Not shown: that ordinary training is this flow, or anything for cross-entropy or with weight decay. | Active |
| THEORY-286 v2 | Under class imbalance the last-layer geometry that unregularized cross-entropy prefers is not the simplex ETF but simplex-encoded-label interpolation: the logits equal the simplex-encoded labels, the classifier and embedding Gram matrices come from that matrix’s SVD, any finite weight decay changes the geometry, and training approaches it more slowly as imbalance grows | Thrampoulidis et al. (2022), LIT-994, read in NOTE-764. For the unconstrained-features max-margin problem under STEP imbalance, every solution has logits ŴᵀĤ = Ẑ (one-hot minus 1/k), ĤᵀĤ = UΛUᵀ and ŴᵀŴ = VΛVᵀ from Ẑ’s SVD (Theorem 1). This is the ETF only if the data are balanced or binary. No finite ridge gives it under imbalance (Proposition 1), though the ridge path tends to it (Proposition 2). ResNet-18 and VGG-13 runs sit closer to SELI than to the ETF, and approach it more slowly at larger R. Not shown: that gradient descent reaches it, and anything with more than one seed. | Proposed |
| THEORY-425 | A connecting path trained only on clean data keeps only the behaviour the clean data ask for: between two backdoored or error-injected image classifiers, models near the ends of such a path keep most clean accuracy and lose the attack, and a path flat in ordinary loss can carry a barrier in adversarial loss that tracks the input Hessian’s top eigenvalue | Zhao, Chen, Das, Ramamurthy & Lin (2020), LIT-668, read in NOTE-530. A quadratic Bezier path trained on 50–2,500 clean samples between two backdoored VGG-16s on CIFAR-10 has, at t = 0.1, 88% clean accuracy and 1.1% backdoor success with 2,500 samples, and 63% and 2.5% with 50; injected weight errors fail at every point of such paths. Paths trained with the standard loss between regular and adversarially trained models are flat in standard loss with a PGD-loss barrier correlated with the input Hessian’s top eigenvalue (0.71–0.88). The proposition offered for that correlation rests on an assumption that would remove the barrier. | Active |
| THEORY-509 | Under genericity conditions, two dense or top-k (k ≥ 2) mixture-of-experts layers compute the same function only if they differ by a permutation of the experts with their gates and a common translation of the gate weights, and the translation leaves every interpolation barrier unchanged, so expert order is the only gate symmetry alignment must undo | Tran, Trinh, Bui & Nguyen (2025), LIT-670, read in NOTE-513. Softmax is invariant to a common shift of scores and the sum to the order of terms, so G(n) = ℝ^d × ℝ × S_n preserves the layer; Theorems 4.1–4.2 show that generically it is all of the gate’s symmetry, for dense gating and for top-k with k ≥ 2 (top-1 has an extra scaling). Aligning expert order and then each expert’s hidden units gives barriers of 0.0044–0.0066 between GPT-2 MoE layers fine-tuned from different seeds, against 0.14–0.86 without the order step. Every experiment shares and freezes the backbone around the inserted layer, so it is not evidence about independently trained MoE models. | Active |
| THEORY-521 | In the thermodynamic limit, with examples and capacity growing together at fixed load, generalization error can jump discontinuously with the load when the entropy of hypotheses just above the best error vanishes faster than the training-error energy grows: a spherical perceptron’s error falls smoothly as 1/α, an Ising perceptron’s drops to zero at a critical load | Martin & Mahoney (2017), LIT-672, read in NOTE-542, restating Seung, Sompolinsky & Tishby (1992) and Haussler, Kearns, Seung & Tishby (1996), which the record does not hold. Generalization is set by maximizing s(ε) − α ln(1/(1 − ε)). For weights on a sphere s(ε) ~ ln ε and ε ~ 1/α; for ±1 weights s(ε) ~ −(π²/2)ε² ln ε, the first-order condition has no solution past a critical load, and the error drops to zero. Uniform bounds that see a class only through its size or VC dimension are smooth and blind to this. That deep networks have such a phase diagram is the authors’ conjecture, and their mapping of label noise to load and early stopping to temperature is an analogy. | Proposed |
| THEORY-557 | Low-loss paths between independently trained solutions are found outside supervised image classifiers: graph neural networks trained from different seeds on one graph are joined by quadratic Bezier curves where the straight line has a barrier, and one pair each of diffusion and contrastive models by chains of low-loss checkpoints | Li et al. (2025), LIT-676, read in NOTE-529: GCN node classifiers from different seeds, on 12 real graphs and synthetic CSBM graphs, show linear barriers that a fitted quadratic Bezier curve removes. Yao, Zhang & Tian (2026), LIT-660, read in NOTE-534: one pair of DDPMs and one pair of small CLIP models joined by chains of checkpoints whose training loss stays under a threshold, where AutoNEB’s paths rise steeply on CLIP. The evidence is plots without barrier numbers in the first, one pair per model and retrained anchors in the second. Zhang & Blei’s claim of flat paths between LDA’s ELBO maxima (LIT-677) reports no results. | Proposed |
| THEORY-578 | Between two trained networks that compute nearly the same function, a chain of models minimising the summed Jensen–Shannon divergence between neighbours, a discretised Fisher–Rao geodesic found without labels, can be a low-loss path where the straight line in weights has a barrier | Tan, Long, Zhao & Laine (2023), LIT-662, read in NOTE-521. Two ResNet-20s at 4× width with LayerNorm, CIFAR-10, permuted by Git Re-Basin’s weight matching: the straight line between them still has a large loss barrier, and a chain of 25 models optimised from it to minimise Σ JSD between neighbours on unlabelled training images keeps train and test loss low. One pair, one figure, no numbers in the text. The hypothesis that every geodesic between SGD solutions is mode-connecting is stated, not tested, and a chain whose neighbours agree in output may be low-loss almost by construction. | Proposed |
| THEORY-675 | Without a shared stable start, the linear barrier between solutions of one objective depends on the structure of the training data: between GCNs it falls with graph density and feature separability and is highest at middling homophily, and between federated-averaging models it grows with client heterogeneity and falls with width | Li et al. (2025), LIT-676, read in NOTE-529: on synthetic CSBM graphs the barrier between GCNs from different seeds falls with density and feature separability and peaks at middling homophily; GCN, GraphSAGE and GAT barriers are similar, an MLP’s differ. Zhou, Zhang & Tsang (2023), LIT-666, read in NOTE-517: FedAvg global models from one random start under more heterogeneous clients are separated by a larger line barrier, joined by a one-bend chain, and the barrier falls with width. Both readings named this account and held it back. The GNN proofs are missing and its barriers are plots; the federated bound treats heterogeneity as isotropic noise, and its lemma and theorem conditions conflict as written. | Proposed |
| THEORY-732 | For ReLU networks interpolated linearly between two endpoints, weak additivity and commutativity at every layer make each layer’s features exactly the interpolation of the endpoints’ features, output-layer linearity bounds the path’s error by twice the endpoints’, and for one hidden layer with interpolating minima linear mode connectivity alone gives the exact case | Zhou et al. (2023), LIT-674, read in NOTE-512 with Appendix A’s proofs checked: for an L-layer ReLU MLP on a finite dataset, weak additivity of ReLU over the two networks’ pre-activations and the commutativity W_A H_B + W_B H_A = W_A H_A + W_B H_B at every layer make the interpolated network’s features at every layer exactly the interpolation of the endpoints’ features (Theorem 1). Output-layer linearity, at any positive scale, keeps the path’s error within twice the endpoints’ (Lemma 1). Lubana et al. (2022), LIT-669, Lemma 2: for one hidden layer and interpolating minima, linear mode connectivity forces shared activation patterns, and with shared patterns the hidden features are linear along the path. It says nothing about when the conditions hold. That trained pairs satisfy them is THEORY-111’s empirical claim. | Active |
| THEORY-736 | Two ε-dropout-stable solutions of a multilayer ReLU net, under a loss convex in the output, are joined by a piecewise-linear path whose loss never exceeds the worse endpoint’s by more than ε, and joined pairwise paths compose with no higher barrier than their worst | Kuditipudi et al. (2019), LIT-671, Theorem 1, read in full in NOTE-544, with Draxler et al.’s (2018) concatenation bound from LIT-653. Dropout stability gives every solution a half-width copy with the same loss up to ε. Half the units are then free, so working units can be swapped in and permuted at no cost. That joins any two such solutions. It is a statement about dropout-stable solutions only: whether training produces them is THEORY-109’s open question, and Theorem 4 of the same paper builds disconnected global minima. | Active |
| THEORY-741 | A two-layer ReLU student of width at least kM, fitting an orthonormal teacher of M neurons on the sphere, has for any k global minima a global minimum joined to each by a two-piece linear path of minima; and in a deep linear network of depth L and width above 1 + r(L − 1), any r global minima in general position share a global minimum linearly connected to each | Lin, Li and Wu (2024), LIT-682, Theorems 6, 11 and 18 with Lemmas 7 and 25, read in NOTE-535 and re-derived by the promotion audit. Two toy models whose minima can be written down: a two-layer ReLU student of an orthonormal teacher on the sphere at population loss, and a deep linear network fitting y = Qx. Spare width lets every minimum be moved, along segments of minima, to one a common centre can reach. No permutation symmetry is used, and nothing is claimed about trained networks. | Active |
| THEORY-779 | In Kunin et al.’s underdamped Langevin model of SGD with momentum on linear regression, with gradient-noise covariance σ²H independent of the parameters, the stationary state is Gaussian with parameter covariance ησ²H(H + λI)⁻¹/(2S(1 − β)) and velocity covariance σ²H/(S(1 − β²)): for small weight decay it spreads equally along every nonzero Hessian direction and not along the null space, each Hessian eigenmode is a damped oscillator with damping ratio (1 − β)/√(2η(1 + β)(ρ + λ)), and the expected squared step is η²σ² tr(H)/(S(1 − β²)) | Kunin et al. (2021), LIT-1011, §§4–6 and App. G, read in NOTE-711; the stationary moments re-derived by the promotion audit. An exact result for a model: Σ ∝ H constant and a quadratic loss are assumptions, and nothing here says a deep network meets them. The probability current is nonzero but reversible: under the model’s assumptions the stationary process is time-reversible once velocities are flipped, so it is “broken detailed balance” only in the paper’s sense, a nonzero current. | Active |
| THEORY-781 | Strong gradient noise in SGD, like weight decay above a threshold, makes the subspace fixed by a mirror symmetry of a network’s loss attracting even where it is a saddle, so that networks trained with SGD at large learning rates or small batches reach sparse, low-rank and identical-neuron solutions, as Ziyin proposes | Ziyin (2024), LIT-1014, read in NOTE-716. Beyond the exact half that THEORY-213 holds, the paper argues by a Lyapunov-exponent analysis that strong SGD noise makes a mirror’s subspace attracting even at a strict saddle, and reads sparsity, low rank and identical neurons in trained networks as its result. That analysis rests on a second-order, diagonal- Hessian approximation, and the network evidence is a ResNet18’s correlation blocks. The test is QUESTION-206. | Proposed |
| THEORY-790 | In deep networks trained by SGD on real data, minibatch noise drives parameter groups related by a rescaling symmetry toward the point where their gradient noise balances, at a rate set by learning rate over batch size, so that at stationarity they sit neither where gradient flow would leave them nor spread out as under isotropic noise, as Ziyin, Li and Ueda propose | Ziyin, Li & Ueda, LIT-1012, read in NOTE-714, with the Itô term from LIT-1013. The law of balance is proved in the SDE model of SGD, and that part is the Active THEORY-211. The authors present the law as general to networks with rescaling symmetry, but test it only on small synthetic networks. This THEORY holds the general claim, for deep networks on real data, which no measurement in the record yet supports. | Proposed |
| THEORY-797 | Deep classifiers trained with MSE loss stay near the least-squares classifier for their own features and move along the central path by the dynamics of continually renormalized gradient flow, whose equal-rate growth of the whitened class-separation singular values is how they reach neural collapse, as Han, Papyan and Donoho propose | Han, Papyan & Donoho (2021; ICLR 2022), LIT-974, read in NOTE-783. The paper’s proposed explanation of neural collapse under MSE in real networks, filed apart from its measured central-path proximity and its derivation in a renormalized model (THEORY-278) under ADR-047 v4. Renormalization is imposed, not measured, and every experiment used weight decay, which the derivation excludes. | Proposed |
| THEORY-828 | Fine-tuning from one pretrained model typically lands near the edge of a task-specific low-loss region in weight space that contains the convex hull of the fine-tuned models, and averaging their weights, as model soups and stochastic weight averaging do, works by moving inward | Gueta, Venezian, Raffel, Slonim, Katz and Choshen (2023), LIT-658, read in NOTE-515, conclude that fine-tuning “typically” yields models on the boundary of a region whose centre is better, and read model soups and SWA as picking points inside it (§§8–9). The record holds as shown, in THEORY-105, only the probe-loss comparisons of hull points with members and the BitFit result, for RoBERTa-base on English classification. This is their geometric thesis, filed under ADR-047 v4 when the narrowing dropped it. | Proposed |
| THEORY-975 | In a two-layer attention-only transformer trained on the Pile, the weight-refined local learning coefficients of individual attention heads start alike and diverge over training, and clustering the curves groups the heads by the functional types that ablation assigns them at the end of training, previous-token, current-token, induction and n-gram-memorizing; among layer-1 heads, those running simple algorithms end with the lowest values and those memorizing the most n-grams with the highest | Wang, Hoogland, van Wingerden, Furman and Murfet (2024), LIT-1535, read in NOTE-1188: §4.1, Figs. 1–2 and 11, appendices B, F.5 and G. Per-head curves of how degenerate the loss is in that head’s weights separate the heads by what they end up doing, and the separation is in the curves’ shapes over training, not only their end values. It is one small model, cluster counts chosen by eye, and estimates whose ordering, not value, the authors trust. It does not show that the coefficient predicts a head’s type before the type is visible. | Active |
| THEORY-1172 | Linear mode connectivity emerges during training, not at initialization: copies trained from a shared state become stable to SGD noise after 1.5–20% of training, and independently trained networks become linearly connected after alignment only gradually | The wide form THEORY-114 held before the tenth status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-114, Active, is its evidence. | Proposed |
| THEORY-1185 | Merging models trained on different tasks from different initialisations fails where their features have no counterpart in the other model, and such features grow more common with depth | The wide form THEORY-113 held before the eleventh status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-113, Active, is its evidence. | Proposed |
| THEORY-1187 | Along a low-loss linear path between two networks, every layer’s features are the interpolation of the endpoints’ features | The wide form THEORY-111 held before the eleventh status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-111, Active, is its evidence. The old title’s averaging clause was the record’s own step and is a Proposed claim of its own; the promote_when here is the layerwise half of the old one. | Proposed |
| THEORY-1188 | Once the architecture’s symmetries are factored out, most of the linear barrier between independently trained solutions disappears; the group that matters is the full one, permutations for MLPs and CNNs but for transformers also an orthogonal map on the residual stream, without which a barrier remains; and the barrier left after alignment falls with width | The wide form THEORY-112 held before the eleventh status review narrowed it, kept as a Proposed reading by the ADR-047 kind-1 sweep of 2026-10-30: the source asserts it, and THEORY-112, Active, is its evidence. The old title’s last words, that the barrier left after alignment rises with depth, were the record’s composition, not a source’s, and are a Proposed claim of their own. | Proposed |
| THEORY-1272 | Late in training, SGD weights drift sub-diffusively, with displacement growing as a power of training time set by a spectral dimension and the local learning coefficient | Hennick and De Baerdemacker (2025; ICLR 2026), LIT-1879, read in NOTE-1467. Late-training displacement fits R(t) ∝ t^α with α below ½ on MNIST MLPs, TinyStories models and ImageNet convnets fine-tuned with SGD, read through the Alexander–Orbach relation as α = d_s/(2λ) with d_s ≤ λ. Proposed: d_s is obtained by fitting that same relation, so the fits test the power law and not its link to the LLC; Kunin et al. report super-diffusion in another regime. | Proposed |
| THEORY-1275 | In the toy model of superposition, SGD’s transitions have Bayesian counterparts whose critical sample sizes rise in the order training meets them, yet SGD was not seen to take the posterior’s main transition and ends in a phase the posterior has left at the training sample size | Chen, Lau, Mendel, Wei and Murfet (2023), LIT-1894, read in NOTE-1462. Their Bayesian antecedent hypothesis is one-way: training’s transitions have Bayesian counterparts. From their numbers the antecedents of the chain 4⁺⁺⁻ → 4⁺ → 5 → 5⁺ fall at n ≈ 190, 370 and 1,600–2,500, rising with training. But the posterior’s transition 5 → 6 at n ≈ 600 was not seen in training, and SGD on 1000 samples ends at the 5- or 5⁺-gon while the posterior at n = 1000 favours the 6-gon. Proposed: one model, one SGD setting, estimated learning coefficients. | Proposed |