Topics — What a work is about. Nucleation holds whatever reading the owner files here, whether or not an Anthology of the SOTA topic could also hold it; instruction for ML practice belongs to the anthology (ADR-048). The words are seeded from the clusters the reading-time triage of 2026-09-25 found, and grow by decision.
Loss landscapes — the geometry of trained solutions in weight space — mode connectivity, linear and along curves, barriers between minima and the permutation symmetries that hide or remove them, re-basin alignment, star domains and connecting simplexes, local versus global landscape structure, and model merging as its use (ADR-027).
38 of 2004 LIT documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| LIT-651 | Linear Mode Connectivity in Multitask and Continual Learning | Mirzadeh, Farajtabar, Gorur, Pascanu & Ghasemzadeh (2020). When a multitask solution and a continual solution start from the same point, the multitask solution lies on a low-loss line to every earlier continual solution, while consecutive continual solutions are separated by rising loss. Euclidean distance and CKA do not detect the difference. The multitask direction lies outside the top Hessian eigenvectors of the earlier task. MC-SGD trains towards a point linearly connected to both the previous solution and the current task’s solution, using a small replay buffer, and leads five baselines on three 20-task benchmarks. | Active — read in full 2026-10-03 (NOTE-520); worth reading as the first study of where continual and multitask solutions lie relative to each other. Started from the same point (the previous task’s solution), the multitask solution is linearly connected with low loss to every earlier continual solution, while consecutive continual solutions are not. The authors explain it by curvature: the multitask path moves in directions the earlier task’s Hessian sees as flat, within a region where a second-order expansion holds. MC-SGD turns this into a regulariser and beats EWC, A-GEM, ER-Reservoir and Stable SGD on 20-task Permuted MNIST, Rotated MNIST and Split CIFAR-100. The connectivity breaks with a different start, input noise, label corruption or removed classes. |
| LIT-652 | The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks | Entezari, Sedghi, Saukh & Neyshabur (2022), ICLR 2022. Conjecture: most SGD solutions lie in a set whose members can be permuted so that no barrier remains on the line between any two. Barriers between networks from different initializations first rise then fall with width, rise with depth, and saturate high for VGGs and ResNets. A proof covers a wide one-hidden-layer net at uniform random initialization. The evidence is a model: random permutations of one SGD solution show the same barriers as independent solutions across width, depth and dataset. A simulated annealing search lowers barriers only for shallow networks. | Active — read in full 2026-10-03 (NOTE-537); the source of the conjecture that SGD solutions, once their hidden units are suitably permuted, have no barrier on the straight line between them: one basin modulo permutation. Its proof covers only a one-hidden-layer network at uniform random initialization, and its main evidence is indirect: barriers between independently trained networks behave like barriers between random permutations of a single trained network, across width, depth and dataset, in more than 3,000 networks. Its own permutation search lowers barriers only for shallow nets on easy data and fails for VGGs and ResNets. Second reading under ADR-013; the anthology holds it as ANTH-LIT-251, and that entry misstates the theorem. |
| LIT-653 | Essentially No Barriers in Neural Network Energy Landscape | Draxler, Veschgini, Salmhofer & Hamprecht (2018), ICML 2018. Using AutoNEB, a minimum-energy-path method from molecular statistical mechanics, the paper connects ten minima per architecture for CNNs, ResNets and DenseNets on CIFAR-10 and CIFAR-100. The saddle losses on the best paths are almost those of the minima for deep architectures, and test error rises by at most a few tenths of a percent on CIFAR-10. The conclusion is that minima are points on one connected low-loss manifold, not bottoms of separate valleys, and that the effect grows with width and especially depth. | Active — read in full 2026-10-03 (NOTE-531); the other founding demonstration of mode connectivity, reached independently of Garipov et al. by borrowing the nudged elastic band from chemical physics. Over ten minima per architecture, the highest training loss on the best paths found is almost the minima’s own for deep ResNets and DenseNets on CIFAR-10, with a small residual gap on CIFAR-100, and barriers fall as networks get wider and especially deeper. Its two explanations, resilience and redundancy, are qualitative; the redundancy toy (swapping two hidden units needs a third) is the first statement in this batch of why a permutation costs a barrier without spare capacity. |
| LIT-654 | Linear Mode Connectivity and the Lottery Ticket Hypothesis | Frankle, Dziugaite, Roy & Carbin (2020), ICML 2020. Instability analysis trains two copies of a network from the same state under different SGD noise and measures the error barrier on the straight line between the results. Standard vision networks are unstable at initialization but become stable early in training, after which the outcome is fixed to one linearly connected minimum. Sparse subnetworks found by iterative magnitude pruning reach full accuracy only when they are stable, which explains where lottery tickets were and were not found, and rewinding to an early iterate rather than to initialization finds matching subnetworks at ImageNet scale. | Active — read in full 2026-10-03 (NOTE-514); the source of linear mode connectivity as a measurement and of instability analysis. Two copies of a network trained from the same state under different SGD noise end up linearly connected once that state is a little way into training (3% for ResNet-20, 20% for ResNet-50 on ImageNet), and iterative-magnitude-pruning subnetworks are matching only when they are stable in this sense. The evidence is correlational: stability and matching coincide across the settings tested, and nothing shows that one causes the other. |
| LIT-655 | Understanding Mode Connectivity via Parameter Space Symmetry | Zhao, Dehmamy, Walters & Yu (2025), ICML 2025. Relates the topology of a loss’s symmetry group to that of its minimum. The minimum of a full-rank l-layer linear regression is homeomorphic to GL_h(ℝ)^{l−1}, so it has 2^{l−1} components; a skip connection reduces 4 to 3 in a 1-D case, and permutations connect all components when h ≥ 2. Rescaling symmetry yields minima in one component with arbitrarily large barriers on the straight line. Group actions give explicit constant-loss curves, and a curvature bound κ on such a curve bounds the chord’s distance from the minimum by about κ‖w₂ − w₁‖²/8. | Active — read 2026-10-03 (NOTE-528); worth reading as the account of mode connectivity from continuous parameter symmetries. The minimum of a full-rank l-layer linear network is homeomorphic to GL_h(ℝ)^{l−1} and so has 2^{l−1} connected components, a skip connection merges some of them, and a permutation with negative determinant joins the rest. Rescaling symmetry gives minima in the same component with an arbitrarily large loss barrier on the straight line between them, even after a restricted permutation. A group element g gives an explicit constant-loss curve exp(t log g)·w, and bounded curvature of such curves bounds the barrier on the chord. The theory is for linear networks and homogeneous last layers; the experiments are small synthetic regressions. |
| LIT-656 | Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling | Benton, Maddox, Lotfi & Wilson (2021), ICML 2021. SPRO trains the connecting vertices of simplexes so that loss is low at points sampled uniformly inside them, with a log-volume bonus. It finds simplicial complexes of low loss joining up to seven independently trained networks through shared vertices, and a low-loss region around VGG-16 on CIFAR-10 of at least 10 dimensions. Started from a single trained network, a few vertices cost about a tenth of a training run each, and ensembling inside the simplexes (ESPRO) beats deep ensembles in accuracy, calibration and robustness to corruption. | Active — read in full 2026-10-03 (NOTE-539); mode connectivity raised from curves to volumes. Training the vertices of simplexes so that loss stays low at random points inside them finds simplicial complexes of low loss joining several independently trained networks at once, and a low-loss region around a VGG-16 on CIFAR-10 of at least 10 dimensions. The same trick, run from one trained network, gives cheap ensembles (ESPRO) that beat deep ensembles at equal size and budget. The dimension estimate is a lower bound from one architecture and dataset, and the paper’s picture of one connected volume containing all SGD solutions is drawn from complexes of up to 7 modes. |
| LIT-658 | Knowledge is a Region in Weight Space for Fine-tuned Language Models | Gueta, Venezian, Raffel, Slonim, Katz & Choshen (2023). RoBERTa-base models fine-tuned on similar data end up close in weight space: task vectors cluster by dataset (98% clustering accuracy over 280 models) and by task family (90%), and the grouping follows data type, not data size. Points between the fine-tuned models, along lines and across their convex hull, perform as well or better than the fine-tuned models themselves, and linear extrapolation beyond them quickly fails. The region’s centroid, re-fine-tuned with BitFit, beats starting from the pretrained model on 9 of 12 target datasets. | Active — read in full 2026-10-03 (NOTE-515); worth reading as the paper that turns linear connectivity between fine-tuned models into a claim about regions. RoBERTa-base models fine-tuned on the same dataset, the same task, or any of 12 GLUE/SuperGLUE datasets cluster in weight space by cosine of their task vectors, and the convex hull of each group performs as well as or better than the fine-tuned members: sampled hull points beat the members in 88% of comparisons on MNLI. Linear extrapolation past the members fails quickly, so the fine-tuned models sit near the edge of a small basin. Averaging the region (its centroid) is a better start for further fine-tuning than the pretrained model on 9 of 12 datasets. One pretrained model, classification only, English only. |
| LIT-659 | Do Deep Neural Network Solutions Form a Star Domain? | Sonthalia, Rubinstein, Abbasnejad & Oh (2024), preprint. Conjectures that SGD solution sets, modulo permutation, are star domains, which needs less width than convexity does. Starlight trains a star model by sampling points on the lines to permuted source models. The star model has much lower loss barriers to held-out solutions than those solutions have to each other, across ResNet-18, VGG, DenseNet, CIFAR and ImageNet-1k (e.g. 0.078 against 0.383 on CIFAR-10). It gains with more source models, ranks uncertainty better in BMA than a deep ensemble, and is about 1 point more accurate than a single model. | Active — read 2026-10-03 (NOTE-533); worth reading as the star domain conjecture: where SGD solutions are not convex modulo permutation, they still form a star domain, with a star model linearly connected, after weight matching, to every other solution. Starlight trains a candidate by sampling points on the lines to 50 permuted source models, and the candidate has lower barriers to held-out models than those models have to each other (ResNet-18 CIFAR-10 0.078 against 0.383; ImageNet-1k 2.794 against 5.948). Those barriers are lower, not zero, and the authors say the conjecture is unproved and the barriers are often significantly above zero. Preprint. |
| LIT-660 | Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models | Yao, Zhang & Tian (2026), preprint. Extends layer-wise path construction (LLPF, Tian et al. 2026) to a DDPM on Flowers102 and a NanoCLIP model on Flickr30k. Layers move towards the target mode in a dataflow order (symmetric encoder–decoder pairs for the U-Net, alternating vision and text blocks for CLIP); each layer is rescaled to a fixed variance; the point is retrained with AdamW until its loss is below a threshold. The resulting chains keep training loss below 0.04 (DDPM) and 0.09 (NanoCLIP), where AutoNEB’s maximum NanoCLIP loss is 6.69. Intermediate DDPMs have worse FID than the endpoints at equal training loss. | Proposed — read in full 2026-10-03 (NOTE-534); watching, as the first claim of mode connectivity between independently trained generative and contrastive models: one pair of DDPMs on Flowers102 and one pair of NanoCLIP models on Flickr30k, joined by a chain of checkpoints along which the training loss stays below a threshold, where AutoNEB’s paths rise steeply on NanoCLIP. The path is built by moving layer groups towards the far end, rescaling each layer’s variance, and retraining with AdamW until the loss is under threshold, so every anchor is a retrained model. What would settle it: more than one pair per model, a barrier measure along the whole path rather than at the anchors, and a reason to think a path made of retrained anchors says more about the landscape than that retraining works. It is a preprint of 31 August 2026. |
| LIT-661 | Git Re-Basin: Merging Models modulo Permutation Symmetries | Ainsworth, Hayase & Srinivasa (2023), ICLR 2023. Three ways to permute one network’s hidden units into alignment with another: activation matching (a linear assignment per layer), weight matching (an NP-hard sum of bilinear assignments, solved by coordinate descent over layers), and a straight-through estimator. After alignment the straight line between independently trained networks has zero loss barrier for MLPs on MNIST and wide ResNet-20s on CIFAR-10. Connectivity grows with width and with training time, fails for thin models, and a counterexample shows it is a property of SGD solutions. Merging models trained on disjoint data improves test loss and calibration but not accuracy. | Active — read in full 2026-10-03 (NOTE-532); the strongest evidence for one basin modulo permutation. A weight-matching algorithm that needs no data and runs in seconds aligns independently trained networks so that the straight line between them has no loss barrier, for MLPs on MNIST and for ResNet-20 on CIFAR-10 at large width, the first such result for ResNets. It also finds where the picture stops: thin models are not linearly connected after any permutation it finds, connectivity appears only after some training, ImageNet ResNet-50 keeps a barrier, and a constructed counterexample shows that the property belongs to SGD solutions, not to architectures. Second reading under ADR-013; the anthology holds it as ANTH-LIT-333, and that entry misdescribes the algorithm and overstates the results. |
| LIT-662 | Geodesic Mode Connectivity | Tan, Long, Zhao & Laine (2023), ICLR 2023 Tiny Paper. Treats a network as a distribution p(x, ŷ; θ) on a Fisher–Rao manifold and hypothesises that geodesics between SGD solutions are mode-connecting paths. A chain of 25 models between two ResNet-20s (4× width, LayerNorm), initialised on the straight line after Git Re-Basin weight matching, is optimised to minimise summed Jensen–Shannon divergence between neighbours. The curve is longer in Euclidean terms but shorter in distribution space, and keeps train and test loss low where linear interpolation shows a large barrier. | Active — read in full 2026-10-03 (NOTE-521); worth reading as the short proposal that mode-connecting paths are geodesics of the Fisher–Rao metric on the space of the networks’ output distributions. It approximates a geodesic between two trained ResNet-20s (4× width, LayerNorm), after Git Re-Basin weight matching, by optimising 25 models on a chain to minimise the summed Jensen–Shannon divergence between neighbours, using unlabelled training images only, and the resulting curve has low train and test loss where the straight line does not. One architecture, one pair, one figure; the hypothesis that all geodesics between SGD solutions are mode-connecting is stated, not tested. |
| LIT-663 | Taxonomizing local versus global structure in neural network loss landscapes | Yang, Hodgkinson, Theisen, Zou, Gonzalez, Ramchandran & Mahoney (2021), NeurIPS. Thousands of trained networks are placed on a load-like by temperature-like grid and measured three ways: Hessian top eigenvalue and trace (local sharpness), Bezier-curve mode connectivity between two independently trained models (global connectivity), and CKA similarity of their outputs on mixup points. Two transitions give four phases: I sharp and poorly connected, II sharp and well connected, III flat and poorly connected, IV flat and well connected, with IV split by CKA into IV-A and IV-B. Test accuracy is best in IV-B. Sharpness alone mispredicts accuracy across phases, and in phase III training to zero loss does worse than stopping in the sharp phases. Double descent appears as a dark band at the phase boundary under label noise. | Active — read 2026-10-03 (NOTE-516); worth reading as the empirical taxonomy of loss landscapes by local and global structure: Hessian top eigenvalue and trace separate locally sharp from locally flat, Bezier-curve mode connectivity between independently trained models separates globally poorly-connected from well-connected, and CKA similarity of their outputs splits the well-connected, flat phase in two. Across a load-like by temperature-like grid (width, data amount or label noise against batch size, learning rate or weight decay), the best test accuracy sits in the flat, well-connected, high-similarity phase IV-B; the Hessian alone mispredicts, and where connectivity is poor, training to zero loss can lower test accuracy. Empirical only, mostly ResNet18 on CIFAR-10. |
| LIT-664 | Generalized Linear Mode Connectivity for Transformers | Theus, Cabodi, Anagnostidis, Orvieto, Singh & Boeva (2025), NeurIPS 2025. A hierarchy of function-preserving maps (permutation ⊂ semi-permutation ⊂ orthogonal ⊂ invertible) fits transformer components: permutations for feed-forward units, semi-permutations for attention heads, an orthogonal map on the RMSNorm residual stream. Aligning with them gives, for the first time, low and zero test-loss barriers between independently trained ViTs and GPT-2s (learned matching 0.00 on CIFAR-10/100 and Tiny ImageNet, 0.02 Tiny Shakespeare, 0.42 BookCorpus), for three-way merges and across embedding widths. Permutation-only alignment does not get there. | Active — read 2026-10-03 (NOTE-540); worth reading as the result that independently trained transformers become linearly mode connected once alignment uses their real symmetries: permutations of feed-forward units, semi-permutations of attention heads, and an orthogonal map on the residual stream after RMSNorm reparameterisation. Data-free weight matching cuts test-loss barriers from 1.69–4.34 to 0.34–1.56, and learned matching, which optimises the alignment against task loss at the midpoint, reaches 0.00 on three vision datasets and 0.02 and 0.42 on two language ones. Most of the gap is in the orthogonal map. Small ViTs and GPT-2s only. |
| LIT-665 | ZipIt! Merging Models from Different Tasks without Training | Stoica, Bolya, Bjorner, Ramesh, Hearn & Hoffman (2023), ICLR 2024. Merging differently initialised models trained on disjoint tasks fails under permutation, because the permuted model sits in a basin for its own task and not the other’s. ZipIt! instead merges correlated features within and across models through merge and unmerge matrices, and can stop merging part-way, giving a shared trunk with task heads. The abstract reports a 20–60% improvement over prior work on CIFAR and ImageNet splits; it comes near the ensemble with partial zipping, and proves a tighter LMC barrier bound than Entezari et al. when units are redundant. | Active — read in full 2026-10-03 (NOTE-523), main text and Appendices A–F, the proof in Appendix G followed to its reduction to Entezari et al.; worth reading as the case that permutation is the wrong symmetry for merging models trained on different tasks. ZipIt! matches features by activation correlation within each model as well as across the two, and can stop merging at a chosen layer, leaving a multi-head model. On disjoint halves of CIFAR-10 it reaches 79.1% joint accuracy where Git Re-Basin reaches 46.2%, and partial zipping comes within a few points of the ensemble. Its theorem tightens Entezari et al.’s barrier bound when a model has redundant units. With full merging on ImageNet-1k it is no better than a permutation baseline (8.6%), so the gains there come from leaving later layers unmerged. |
| LIT-666 | Mode Connectivity and Data Heterogeneity of Federated Learning | Zhou, Zhang & Tsang (2023), preprint. Under Dirichlet label skew and PACS domain shift, low-loss overlaps between FedAvg client models and the global model shrink as heterogeneity grows; client models stay close to the global one yet move in different directions within a round. Global models from one initialisation under different heterogeneity have function dissimilarity above 0.2, show an error barrier on the line, and are joined without barrier by a polygonal chain. A mean-field analysis of two-layer networks bounds dropout and connectivity errors by terms proportional to √h(α) and decreasing in width N. | Active — read in full 2026-10-03 (NOTE-517); worth reading as the only study in the batch of connectivity under federated averaging. Client models, though close in weight space, leave the low-loss region of the global model more as client data grow more heterogeneous, and FedAvg global models trained from one initialisation under different heterogeneity are separated on the straight line (accuracy drops of about 10%) but joined by a one-bend polygonal chain. A mean-field bound, extending the dropout-stability route of Kuditipudi et al. and Shevchenko and Mondelli, makes the connectivity error grow with √h(α), a heterogeneity factor, and shrink with width; width and heterogeneity trends in MNIST and CIFAR-10 runs agree. The theory treats heterogeneity as isotropic Gaussian noise on each neuron’s update, an assumption the paper checks only by plotting noise norms. |
| LIT-668 | Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness | Zhao, Chen, Das, Ramamurthy & Lin (2020), ICLR 2020. A Garipov-style quadratic Bezier path between two tampered models, trained on a small clean set, contains models that keep clean accuracy and shed backdoor or injected-error attacks, outperforming fine-tuning, retraining, pruning and weight noise on the clean–attack trade-off in most settings. On a standard-loss path between a regular and an adversarially trained model, standard loss stays flat but robustness loss rises in the middle, and it correlates with the largest input-Hessian eigenvalue (Pearson 0.71–0.88). | Active — read 2026-10-03 (NOTE-530); worth reading as the first use of mode connectivity for adversarial robustness. A quadratic Bezier path trained between two backdoored or error-injected models on as few as 50–2,500 clean samples contains models near its ends that keep most clean accuracy and lose the attack (CIFAR-10 VGG with 2,500 samples: 88% clean, 1.1% backdoor success at t = 0.1). Between a regular and an adversarially trained model, a path trained with the standard loss has no standard-loss barrier but a robustness-loss barrier, which tracks the largest eigenvalue of the input Hessian. The proposition offered for that correlation rests on an assumption that, taken literally, would rule out the barrier it explains. |
| LIT-669 | Mechanistic Mode Connectivity | Lubana, Bigelow, Dick, Krueger & Tanaka (2022), ICML 2023. Two models are mechanistically similar if they are invariant to the same interventions on the data-generating process. Models that use a spurious cue and models that ignore it are mode connected along quadratic paths but not along linear ones, even after permutation matching, and the paper conjectures that a linear barrier implies mechanistic dissimilarity. For a one-hidden-layer ReLU network it proves only the converse: mechanisms of different complexity cannot be linearly connected. Naive fine-tuning on clean data stays linearly connected to the pretrained model and keeps its reliance on the cue; a fine-tune that forces a barrier removes it. | Active — read 2026-10-03 (NOTE-538); worth reading as the paper that ties a loss barrier to a difference in what two models compute. It defines two models as mechanistically similar when they are invariant to the same interventions on the data-generating process, finds that models relying on a spurious cue and models ignoring it are connected by quadratic paths but not by linear ones, even after permutation, and conjectures that a linear barrier always signals mechanistic dissimilarity. The conjecture is proved only for a one-hidden-layer ReLU network on a synthetic process; the evidence is synthetic cues on CIFAR-10, CIFAR-100 and Dominoes. |
| LIT-670 | On Linear Mode Connectivity of Mixture-of-Experts Architectures | Tran, Trinh, Bui & Nguyen (2025), NeurIPS 2025. The group generated by permuting experts with their gates and translating all gating weights by a common vector preserves an MoE’s function. Under genericity conditions it is the whole symmetry of the dense gate and of the top-k gate for k ≥ 2 (Theorems 4.1–4.2). Translation does not change barriers, so only the permutation matters. Matching expert order by centred gate weights or by Gram matrices, then weight-matching inside each expert, brings barriers between MoE layers fine-tuned from different seeds near zero, across dense, top-2 and DeepSeekMoE variants, on ViT and GPT-2 backbones that are shared and frozen. | Active — read 2026-10-03 (NOTE-513); worth reading as the characterisation of the symmetries of a mixture-of-experts gate. Under genericity conditions, two dense or top-k (k ≥ 2) MoEs compute the same function only if they differ by a permutation of experts together with their gates and a common translation of the gating weights (Theorems 4.1–4.2), and the translation leaves every barrier unchanged. Expert-order matching followed by per-expert weight matching gives near-zero barriers between MoE layers fine-tuned from different seeds. But the backbone around the MoE layer is shared and frozen, so what is connected is one inserted layer, not two independently trained networks. |
| LIT-671 | Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets | Kuditipudi, Wang, Lee, Zhang, Li, Hu, Arora & Ge (2019), NeurIPS 2019. Mode connectivity follows from generic robustness properties of trained nets. Two ε-dropout-stable ReLU networks, each able to zero half its units per layer at loss cost ε, are connected by a piecewise-linear path of loss at most max endpoint + ε; ε-noise-stable networks are connected by 10 segments with barrier Õ(ε). Conversely, for any width there is a dataset on which global minima of a two-layer student are disconnected, so connectivity is a property of the solutions training finds. Checked on MNIST convnets and VGG-11 on CIFAR-10. | Active — read in full 2026-10-03 (NOTE-544); the first explanation of mode connectivity that applies to realistic multilayer nets. If two solutions can each lose half their units per layer at a cost of at most ε in loss (dropout stability), they are joined by a piecewise-linear path whose loss never exceeds the worse endpoint by more than ε; a stronger noise-stability condition gives 10 segments. A counterexample shows that connectivity fails for some global minima of arbitrarily overparametrized two-layer nets, so the property belongs to the solutions training finds, not to the landscape. The conditions are checked on small MNIST convnets and one VGG-11 trained with channel-wise dropout. |
| LIT-672 | Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior | Martin & Mahoney (2017), arXiv preprint. Zhang et al.’s observations, that deep networks fit noisy labels and that only early stopping regularizes them, are read through the statistical mechanics of learning. A Very Simple Deep Learning model has two control parameters: a load α = m_eff/N, lowered by randomizing labels, and a temperature τ, raised by stopping early. In the thermodynamic limit (m, N → ∞, α fixed), generalization error can drop discontinuously at a critical αc, and a two-dimensional (α, τ) phase diagram has a spin-glass-like overtraining phase. The mechanism is entropy–energy competition, shown on the continuous and Ising perceptrons: the Ising entropy vanishes near zero error, so the first-order condition loses its interior solution. | Active — read in full 2026-10-03 (NOTE-542); worth reading as the position paper that brought the statistical-mechanics theory of generalization back to deep learning: a network is idealized as a black box governed by a load-like parameter α (effective data per capacity, lowered by label noise) and a temperature-like parameter τ (raised by early stopping), and in the thermodynamic limit m, N → ∞ with α = m/N fixed, generalization has phases with sharp transitions. The mechanism is shown on the continuous versus Ising perceptron: where the entropy of near-optimal hypotheses vanishes (s(ε) ~ −(π²/2)ε² ln ε), the error drops discontinuously at αc, which PAC/VC bounds do not even qualitatively describe. A qualitative argument with no new theorem and no experiment. |
| LIT-673 | Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs | Garipov, Izmailov, Podoprikhin, Vetrov & Wilson (2018), NeurIPS 2018. Optima of deep networks trained from different initializations are joined by simple learned curves (a polygonal chain with one bend, or a quadratic Bezier curve) along which train loss and test error stay nearly constant, although the straight line between them rises to near-chance error. The curve is found by minimizing the loss averaged over a uniform t on the curve. Points along it are functionally diverse, which motivates Fast Geometric Ensembling: short cyclical learning-rate cycles late in training, collecting a checkpoint per cycle, beating Snapshot Ensembles on CIFAR-10, CIFAR-100 and ImageNet at equal budget. | Active — read in full 2026-10-03 (NOTE-527); one of the two founding demonstrations of mode connectivity. Two independently trained networks are joined by a learned one-bend polygonal chain or quadratic Bezier curve along which train loss and test error stay near the endpoints, while the straight segment between them climbs to near chance (VGG-16 on CIFAR-10: 90% test error at the midpoint). Points a short way along a curve already make useful ensemble partners, which motivates Fast Geometric Ensembling. The connectivity evidence is a table of single runs per architecture with no error bars, and the paper does not claim the curves are anything more than found. |
| LIT-674 | Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity | Zhou, Yang, Yang, Yan & Hu (2023), NeurIPS 2023. When two trained networks are linearly mode connected, whether by spawning from a shared early checkpoint or by permutation alignment, the features of nearly every layer of the weight-interpolated network are proportional to the interpolation of the two networks’ features (LLFC). LLFC on the output layer implies LMC. Weak additivity of ReLU and a commutativity condition between layers’ weights and features together imply LLFC, and Git Re-Basin’s two matching objectives are each one factor of the commutativity residual. | Active — read 2026-10-03 (NOTE-512); worth reading as the finding that linear mode connectivity reaches inside the network: whenever two trained networks are linearly connected, by spawning or by permutation, the feature map of almost every layer of the interpolated network points in the direction of the interpolated feature maps (LLFC). Two conditions, weak additivity of ReLU on the two networks’ pre-activations and a commutativity of each layer’s weights with the other network’s features, imply exact LLFC (Theorem 1), and Git Re-Basin’s weight and activation matching each minimise one factor of the commutativity residual. Image classification only. |
| LIT-675 | TIES-Merging: Resolving Interference When Merging Models | Yadav, Tam, Choshen, Raffel & Bansal (2023), NeurIPS 2023. Merging fine-tuned models by averaging or adding task vectors suffers from two kinds of interference: redundant small updates dilute influential ones, and influential updates of different tasks disagree in sign. TIES-Merging trims each task vector to its top-k% magnitudes, elects a sign per parameter by total mass, and averages only values agreeing with it. It beats averaging, task arithmetic, Fisher merging and RegMean across (IA)³, T5 and ViT settings when a validation set tunes it, degrades more slowly as tasks are added, and with oracle signs nearly matches multitask training. | Active — read in full 2026-10-03 (NOTE-526); worth reading as the account of why averaging task vectors loses information: most of a fine-tuning update is small and redundant, and the influential coordinates of different tasks often disagree in sign, so a plain mean shrinks them. TIES-Merging keeps the top 20% of each task vector by magnitude, elects a sign per coordinate by total mass, and averages only the agreeing values. With a validation set it beats task arithmetic, Fisher merging and RegMean on all five settings (by 0.7 to 3.6 points); without one it falls below task arithmetic on T5-base (69.7 against 73.2 in Table 1), which the introduction’s summary of results leaves out. With the multitask model’s signs it reaches 72.0 against 73.1 for multitask training, which is the paper’s strongest evidence that sign is the bottleneck. |
| LIT-676 | Unveiling Mode Connectivity in Graph Neural Networks | Li, Chen, Han, Zeng, Liu & Tang (2025), preprint. GCN solutions for node classification from different seeds are connected by quadratic Bezier curves but often separated on the straight line, more so on co-authorship and heterophilous graphs than on citation graphs. GNN variants (GCN, GraphSAGE, GAT) give similar barriers, MLPs different ones. On CSBM graphs, barriers fall with density and feature separability and are highest at middling homophily. The authors bound the barrier through the spectral gap, bound the generalisation gap by the barrier, and propose a Wasserstein distance between graphs’ interpolation-loss curves that correlates with transfer gaps. | Proposed — read in full 2026-10-03 (NOTE-529); watching, as the first study of mode connectivity in graph neural networks. Two GCNs trained on the same graph with different seeds are usually joined by a low-loss quadratic Bezier curve but often not by a straight line, and the barrier depends on the graph more than on the GNN variant: it falls with density, with feature separability, and with homophily far from one half, in synthetic CSBM graphs and, by the authors’ reading, in 12 real ones. A loss-barrier bound, a generalisation bound in terms of the barrier, and a barrier-based distance between graphs are stated, but the appendix that should hold the proofs does not; the v1 text still carries an editor’s placeholder. What would settle it: the proofs, and numbers for the barriers the figures show. |
| LIT-677 | Unveiling mode-connectivity of the ELBO landscape | Zhang & Blei (2021), NeurIPS 2021 Bayesian Deep Learning workshop. Two SVI runs of LDA on the New York Times corpus give distinct local maxima of the ELBO; the simplified string method, with natural-gradient bead updates, is said to find an essentially flat maximum-energy path between them. Paths are said to flatten as data and the number of topics grow. The authors explain connectivity by over-parameterisation (more topics than the corpus holds) and by continuous statistical degeneracy, offering a landscape account of topic modelling’s “no best answer”. No results are reported in the paper. | Proposed — read in full 2026-10-03 (NOTE-522); watching, as the only work in the record to carry mode connectivity from neural networks to variational inference. It claims that local maxima of the LDA ELBO, from two SVI runs on the New York Times corpus, are joined by essentially flat maximum-energy paths found with the simplified string method, that paths flatten as the number of topics grows, and that statistical degeneracy explains this. The three-page workshop paper reports none of the results it describes: no path, no ELBO values, no synthetic-data experiment; its one figure is a schematic. What would settle it: the experiments themselves, which are not in this version. |
| LIT-680 | Proving Linear Mode Connectivity of Neural Networks via Optimal Transport | Ferbach, Goujaud, Gidel & Dieuleveut (2024), AISTATS 2024. Finding the permutation that aligns two networks’ neurons layer by layer is an optimal-transport problem between empirical weight distributions, so Wasserstein convergence rates bound when the permuted straight line stays low-loss. Wide two-layer nets trained by SGD in the mean-field regime are linearly connected modulo permutation with high probability. Deep nets with i.i.d. neuron weights (as at initialization) are too, once each layer is wider than about (1/ε) to the power of the previous one, a rate shown tight. Low-dimensional weight distributions need less width, and a covariance-weighted weight matching beats plain weight matching. | Active — read in full 2026-10-03 (NOTE-543); the first proof of linear mode connectivity modulo permutation beyond a single hidden layer. Aligning neurons is an optimal-transport problem between the empirical distributions of their weights, so convergence rates of empirical measures in Wasserstein distance bound how wide a network must be for the permuted straight line to stay low-loss. It proves the result for wide two-layer nets trained by SGD in the mean-field regime, and for deep nets whenever neuron weights within a layer are i.i.d., which holds at initialization. The width needed grows like (1/ε) to the power of the previous layer’s width, a rate shown to be tight, and falls if weights concentrate near a low-dimensional subspace. The proofs rest on independence of neurons, which the authors call the central and least realistic assumption. |
| LIT-681 | Optimizing Mode Connectivity for Class Incremental Learning | Wen, Cheng, Qiu, Wang, Pan & Li (2023), ICML 2023. In class-incremental learning the linear path between adjacent continual minima crosses a high-loss ridge for earlier tasks. OPC finds a curved low-loss path, modelled as the line plus a layer-wise Fourier-series perturbation, with the path loss split at a switching point between old-task and new-task memory losses. EOPC averages points in a cylinder around the switching point and is added as post-processing to existing methods, improving PODNet and AANet on CIFAR-100, ImageNet-100 and ImageNet-1K and sharply reducing forgetting. | Active — read in full 2026-10-03 (NOTE-524); worth reading as the class-incremental counterpart of Mirzadeh et al.: between adjacent continual minima in class-incremental learning, the straight line crosses a high-loss ridge for the earlier tasks, and a curved path avoids it. OPC parameterises that path as the line plus a layer-wise Fourier perturbation that vanishes at both ends (2 × 96 × 4 learned numbers for a 96-layer ResNet), splits the path loss at a switching point into old-task and new-task halves, and pushes it to flat regions by sampling a cylinder around it. Averaging points near the switching point (EOPC) adds 0.3 to 5.1 points of average incremental accuracy to PODNet and AANet on CIFAR-100, ImageNet-100 and ImageNet-1K. In Table 4 it beats Bezier and simplex paths only on the longer task sequences. |
| LIT-682 | Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity | Lin, Li & Wu (2024), preprint. For a two-layer ReLU student of an orthonormal teacher, and for deep linear networks, over-parameterisation makes any two global minima two-piece linearly connected and gives any finite set of minima a common centre linearly connected to each (star- shaped connectivity). Under a sparsity bias like SGD’s, the shortest path on the minimum manifold is nearly as short as the straight line, with normalised geodesic distance ≤ 1 + c₂/(r√m). A centre-finding objective locates such centres for independently trained FNNs, VGG16 and ResNets on MNIST and CIFAR-10, with no permutation alignment. | Active — read 2026-10-03 (NOTE-535); worth reading as the theory of star-shaped connectivity without permutations. For a two-layer ReLU student of an orthonormal teacher, and for deep linear networks, sufficiently wide networks have any two minima joined by a two-piece linear path and any finite set of minima linearly joined to a common centre on the minimum manifold. SGD’s bias toward sparse solutions explains why the shortest path on the manifold is nearly as short as the straight line (normalised geodesic distance ≤ 1 + c₂/(r√m)). Experiments find centres for three to five independently trained FNNs, VGG16s and ResNets on MNIST and CIFAR-10 with almost no barrier. Preprint. |
| LIT-850 | Revisiting Model Stitching to Compare Neural Representations | Bansal, Nakkiran & Barak (2021), NeurIPS 34. Revives Lenc and Vedaldi’s model stitching (LIT-363) as a measure of representations: the bottom of one frozen network feeds the top of another through a trained 1×1 convolution, and the change in test error is the “stitching penalty”. Independently seeded networks, and supervised and self-supervised ResNet-50s, stitch at all layers within a few points, where CKA says they differ. Representations trained on more data plug into a weaker network and lower its error; for width and training time this happens only near the top. All empirical, single runs, mostly CIFAR-10. | Active — read in full 2026-10-09 (NOTE-653); worth reading as the paper that made model stitching a standard way of comparing trained networks, and as the source of “stitching connectivity”: two networks of one architecture, trained from different seeds, can be joined at any layer by a learned 1×1 convolution with little loss of test accuracy. Read it knowing that it proves nothing. Its definitions are informal (“low penalty”, “comparable”), stitching connectivity is an informal conjecture, and every result is a single run, mostly ResNet-18 on CIFAR-10, read off plots with no error bars. Its “more is better” ordering is an ordering by one task’s loss through a fixed top network and a restricted map class, not an ordering by information: the paper itself says a pixel shuffle, which loses nothing, can stitch worse, and a negative penalty means the fitted map did not reproduce the layers it replaced. The width and training-time improvements appear only near the top of the network; at mid layers the wider and longer-trained representations stitch slightly worse. |
| LIT-861 | Topological Invariance and Breakdown in Learning | Yang, Poggio, Chuang and Ziyin (2025), ARXIV-2510.02670. For any permutation-equivariant update rule whose map is K-Lipschitz, one step of size η changes the distance between any two neurons by a factor in [1 − ηK, 1 + ηK], and coincident neurons stay coincident. Below η* = 1/K the induced map on the set of neuron vectors is therefore a homeomorphism, measure-preserving for the pushed-forward neuron distribution and a C¹ diffeomorphism under a smoothness condition; above it only a continuous surjection is guaranteed. The paper reads this, with the edge of stability, as two phases of training, the second a topological simplification; the proofs hold, but that reading and the Betti-number experiments go beyond them. | Active — read 2026-10-09 (NOTE-663); worth reading as a short, correct proof that permutation symmetry constrains how neurons can move under training: if the update rule is permutation-equivariant (gradient descent, SGD or Adam on a permutation-symmetric loss), neurons that coincide stay coincident at every step size, and if the update map is K-Lipschitz each step moves every pair of neurons apart or together by at most a factor 1 ± ηK. So below η = 1/K the induced map on the set of neuron vectors is a bi-Lipschitz homeomorphism (a C¹ diffeomorphism under a smoothness condition) and distinct neurons cannot merge in finitely many steps; above it only a continuous surjection is guaranteed. The proofs are a few lines and hold. What they do not carry is the paper’s larger picture: for finitely many neurons a homeomorphism of the neuron set is only a bijection, “topological simplification” above 1/K is permitted, not shown, and the experiments measure Betti numbers of point clouds at a fixed scale, which the theorem does not protect. The large-step runs show the clouds fragmenting (more components), not merging, and in the Adam runs the step size times the measured top Hessian eigenvalue stays below 0.05 where the topology changes. |
| LIT-970 v2 | Symmetry in Neural Network Parameter Spaces | Zhao, Walters & Yu (2025), TMLR. A survey of parameter-space symmetries: transformations of a network’s parameters that leave its function, or its loss, unchanged. It catalogues them for common components (linear layers, homogeneous and other activations, normalization, softmax, attention, LoRA). It surveys when they are complete, so that parameters are identifiable up to them. Continuous symmetries stretch minima into connected manifolds and discrete ones replicate them. Under gradient flow, continuous symmetries give conserved quantities, such as layer imbalance, that label trajectories and locate the endpoint among the minima. Finite steps, gradient noise and weight decay break these conservation laws. | Active — read 2026-10-13 (NOTE-712): §§1–5 and 7–8 in full, §6 in part. A survey, so its claims about the works it cites are secondhand here. It bears on QUESTION-058 and QUESTION-069, and on the null space that THEORY-209 leaves out. The account filed from it is THEORY-210. |
| LIT-1011 | The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion | Kunin, Sagastuy-Brena, Gillespie, Margalit, Tanaka, Ganguli & Yamins (2021). Long after the loss converges, networks trained with SGD keep moving at a constant speed, and their distance from where they were grows as a power law in steps. Modelling SGD with momentum at finite learning rate and batch size as an underdamped Langevin equation, the paper solves linear regression exactly: an Ornstein–Uhlenbeck process whose stationary distribution is a Gibbs distribution on a modified loss, not the training loss, in broken detailed balance, so probability currents drive oscillations in phase space. Under a quadratic approximation, a pre-trained ResNet-18 on ImageNet shows the predicted isotropic exploration, velocity regularization, oscillations and speed. | Active — read 2026-10-13 (NOTE-711): main text, and Appendices A–C and H; the derivations in Appendices D–G were not checked. Held also in the Anthology of the SOTA as ANTH-LIT-454, read there for what the loss curve hides; read here for what a converged run is, under ADR-013. |
| LIT-1012 | Noise Balance and Stationary Distribution of Stochastic Gradient Descent | Ziyin, Li & Ueda (2023; v2 2025). Under the SDE model of SGD with temperature T = η/S, a rescaling symmetry between parameter groups u and w gives d(‖u‖² − ‖w‖²)/dt = −T(uᵀC₁u − wᵀC₂w): the Brownian term vanishes, and minibatch noise drives the quantity to the point where the two groups’ gradient noise balances. Gradient descent conserves it, and gradient descent with isotropic noise diffuses along it. At stationarity SGD therefore differs from gradient descent by a finite amount however small T is, and is not ergodic. For a diagonal linear network of arbitrary depth and width, the paper solves the stationary distribution exactly. It finds a phase transition to the zero solution above a critical T, sign coherence, fluctuation inversion and a tail set by depth alone. | Active — read 2026-10-13 (NOTE-714): §§I–IV in full; the proofs in Appendix A not checked. Cited by LIT-970 as Ziyin et al. (2023a), “Law of balance and stationary distribution of stochastic gradient descent”; retitled in v2. The source of THEORY-211. |
| LIT-1013 | Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics | Kunin, Sagastuy-Brena, Ganguli, Yamins & Tanaka (2021), ICLR. A differentiable symmetry of the loss makes the gradient orthogonal to the symmetry’s generator and constrains the Hessian. Under gradient flow each symmetry then conserves a quantity: translation (softmax) the parameter sum, scale (batchnorm) the norm, and rescale (homogeneous activations) the difference of squared norms. Weight decay and finite learning rates break these laws, and a modified gradient flow gives exact integral expressions for the broken quantities, matched on VGG-16 trained on Tiny ImageNet. Gradient noise is orthogonal to the generators, so it is low-rank along symmetry directions. | Active — read 2026-10-13 (NOTE-713): main text in full, and Appendices F, I and J; Appendices A–E, G and H not checked. The first-hand source for the conservation laws and their breaking that THEORY-210 took from the LIT-970 survey. |
| LIT-1014 | Symmetry Induces Structure and Constraint of Learning | Ziyin (2024), ICML. A loss has an O-mirror symmetry if it is invariant under the reflection I − 2OOᵀ. Rescaling, rotation and permutation symmetries all imply mirror symmetries. Every such symmetry makes Oᵀθ = 0 a stationary condition: once there, (S)GD cannot leave. It also splits the Hessian into blocks. Above a threshold weight decay, all minima satisfy the constraint (Theorem 1). Strong gradient noise makes it attractive even at a saddle (§2.4). As corollaries, rescaling gives sparsity, rotation gives low rank, and permutation gives identical, merged neurons. The paper ties these to collapse phenomena and to loss of plasticity. | Active — read 2026-10-13 (NOTE-716): main text in full; the proofs in Appendix C and the experimental details in Appendix B not checked. Cited by LIT-970 as Ziyin (2024). The source of THEORY-213. |
| LIT-1535 v2 | Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient | Wang, Hoogland, van Wingerden, Furman & Murfet (2024), arXiv:2410.02984, ICLR 2025. Refined variants of the local learning coefficient, a complexity measure from singular learning theory, are applied to individual components of a two-layer attention-only transformer, tracing how attention heads progressively differentiate into distinct functional roles during training, what data they specialize to, and a previously unidentified multigram circuit (abstract). | Active — read 2026-10-26 (NOTE-1188) in full, from the arXiv PDF of v1; worth reading as the record’s first developmental use of the learning coefficient, for per-head trajectories that separate attention heads by function (THEORY-975), and for its observations of heads changing role as circuits form (THEORY-967, Proposed). Read it knowing that it is one two-layer attention-only model with three appendix seeds, that the estimates are trusted only for their ordering, that cluster counts were chosen by eye, and that nothing is predicted before it happens. |
| LIT-1879 v2 | Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent | Hennick & De Baerdemacker (2025; ICLR 2026). Models late-training SGD as sub-diffusion on a porous landscape: a time-fractional Fokker–Planck equation, the local learning coefficient as the medium’s mass dimension, and a spectral dimension d_s fitted from weight displacement, R(t) ∝ t^(d_s/2λ) with d_s ≤ λ. Where an effective scalar diffusion D_ξ is constant on a region, the steady state is exp(−L/D_ξ), the posterior tempered to effective sample size 1/D_ξ. How mass is shared between regions is not derived; in one toy test the cluster masses of SGD solutions match an SGLD posterior after a fitted tempering scale. Displacement fits cover MNIST MLPs, TinyStories models and fine-tuned ImageNet convnets. | Active — read in full 2026-10-31 (NOTE-1467); its “tempered posterior” is a temperature, the posterior at effective sample size 1/D_ξ within a region of constant diffusion, and the “accessibility” that would make SGD differ from Bayes across regions is not derived but fitted (THEORY-1258); its sub-diffusion fits are filed as Proposed (THEORY-1272). |
| LIT-1894 v2 | Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition | Chen, Lau, Mendel, Wei & Murfet (2023). In Elhage et al.’s toy model of superposition with two hidden dimensions, in the high-sparsity limit, the population loss has a closed form and regular k-gons are critical points with derived local learning coefficients. The free energy formula predicts a Bayesian phase transition from the 5-gon to the 6-gon near n = 600, which MCMC confirms. SGD trajectories plateau at the same critical points and move to lower loss and higher estimated complexity, but the 5 → 6 transition was not observed in training. The paper states that there is no necessary relation between the two kinds of transition, and conjectures only that training’s transitions have Bayesian antecedents. | Active — read 2026-10-31, the derivation appendices skimmed (NOTE-1462); it does not argue that SGD and the posterior trace one journey: it states “no necessary relation” between their transitions, proposes only that training’s transitions have Bayesian antecedents, and its main Bayesian transition, 5-gon to 6-gon, was not seen in training (THEORY-1275). |