nucleation

loss-landscapes

Graph View

99 items with this tag.

Oct 31, 2026
CLAIM-524: Whether functionally equivalent realizations are connected depends on which paths are admissible: for f = u₁v₁ + u₂v₂ with nonnegative weights, (1, 1, 0, 0) and (0, 0, 1, 1) are joined by a curve along which f stays 1 though the straight line between them falls to ½, and under any resource budget below √2 times the endpoints' cost no path with f = 1 joins them
- loss-landscapes
- mathematics
Oct 31, 2026
LIT-1879: Almost Bayesian: The Fractal Dynamics of Stochastic Gradient Descent
Oct 31, 2026
LIT-1894: Dynamical versus Bayesian Phase Transitions in a Toy Model of Superposition
Oct 31, 2026
QUESTION-363: When does the order in which stochastic gradient descent comes to occupy a network's organizations over training time follow the order in which they come to dominate the Bayesian posterior as the sample grows, and what fixes the correspondence where it holds: an effective temperature set by learning rate and batch size, the accessibility of regions and the barriers between them, or the time allowed to equilibrate?
Oct 31, 2026
THEORY-1272: Late in training, SGD weights drift sub-diffusively, with displacement growing as a power of training time set by a spectral dimension and the local learning coefficient
- learning-theory
- loss-landscapes
Oct 31, 2026
THEORY-1275: In the toy model of superposition, SGD's transitions have Bayesian counterparts whose critical sample sizes rise in the order training meets them, yet SGD was not seen to take the posterior's main transition and ends in a phase the posterior has left at the training sample size
Oct 30, 2026
CLAIM-498: Averaging weights within a basin averages features: the network whose weights are the average of several networks in one low-loss basin has, in every layer, the average of their features
Oct 30, 2026
CLAIM-499: The linear barrier left after aligning independently trained networks over the architecture's symmetries rises with depth
- loss-landscapes
- anthology-candidate
Oct 30, 2026
THEORY-1172: Linear mode connectivity emerges during training, not at initialization: copies trained from a shared state become stable to SGD noise after 1.5–20% of training, and independently trained networks become linearly connected after alignment only gradually
- loss-landscapes
- anthology-candidate
Oct 30, 2026
THEORY-1185: Merging models trained on different tasks from different initialisations fails where their features have no counterpart in the other model, and such features grow more common with depth
Oct 30, 2026
THEORY-1187: Along a low-loss linear path between two networks, every layer's features are the interpolation of the endpoints' features
Oct 30, 2026
THEORY-1188: Once the architecture's symmetries are factored out, most of the linear barrier between independently trained solutions disappears; the group that matters is the full one, permutations for MLPs and CNNs but for transformers also an orthogonal map on the residual stream, without which a barrier remains; and the barrier left after alignment falls with width
- loss-landscapes
- anthology-candidate
Oct 29, 2026
QUESTION-312: Does linear mode connectivity emerge during training rather than at initialization outside image classification and beyond MLPs: in a language model or transformer, for copies spawned from a shared state and for independent runs aligned under the architecture's symmetry group, and for standard-width convnets or ResNets after alignment?
- loss-landscapes
- learning-theory
Oct 29, 2026
QUESTION-316: Is it the difference in task, and not depth and initialisation alone, that leaves features with no counterpart in the other model and makes merging fail: at matched width, do models trained from different initialisations on disjoint tasks lose cross-model feature correlation faster with depth than models trained on the same task, with merging failing at that depth?
- loss-landscapes
- representation-learning
Oct 29, 2026
QUESTION-317: Does aligning independently trained networks over the architecture's full symmetry group remove most of the linear barrier at standard width, on ImageNet or a language corpus, with an alignment computed without data; is the barrier that remains a difference in what the endpoints compute; and does it rise with depth?
- loss-landscapes
- representation-learning
Oct 29, 2026
QUESTION-318: Does layerwise linear feature connectivity hold outside image classification, in transformers and language models connected by spawning, shared pretraining or alignment, and does it carry from pairs to the many-model averages practice uses, model soups and SWA?
- loss-landscapes
- representation-learning
Oct 27, 2026
QUESTION-286: Under smooth invertible changes of the parameter coordinates, which first-order frustration diagnoses among a learner's demands are invariant: pairwise Euclidean gradient angles, common strict first-order descent, or budgeted feasibility with the update geometry transported?
Oct 26, 2026
LIT-1535: Differentiation and Specialization of Attention Heads via the Refined Local Learning Coefficient
Oct 26, 2026
THEORY-975: In a two-layer attention-only transformer trained on the Pile, the weight-refined local learning coefficients of individual attention heads start alike and diverge over training, and clustering the curves groups the heads by the functional types that ablation assigns them at the end of training, previous-token, current-token, induction and n-gram-memorizing; among layer-1 heads, those running simple algorithms end with the lowest values and those memorizing the most n-grams with the highest
Oct 20, 2026
QUESTION-206: Does stochastic gradient noise drive training into a mirror-symmetric subspace where it is a saddle, beyond the second-order, diagonal-Hessian argument, and does a trained network's distance to that subspace fall to zero past a threshold set by noise and curvature?
- learning-theory
- loss-landscapes
Oct 20, 2026
THEORY-779: In Kunin et al.'s underdamped Langevin model of SGD with momentum on linear regression, with gradient-noise covariance σ²H independent of the parameters, the stationary state is Gaussian with parameter covariance ησ²H(H + λI)⁻¹/(2S(1 − β)) and velocity covariance σ²H/(S(1 − β²)): for small weight decay it spreads equally along every nonzero Hessian direction and not along the null space, each Hessian eigenmode is a damped oscillator with damping ratio (1 − β)/√(2η(1 + β)(ρ + λ)), and the expected squared step is η²σ² tr(H)/(S(1 − β²))
Oct 20, 2026
THEORY-781: Strong gradient noise in SGD, like weight decay above a threshold, makes the subspace fixed by a mirror symmetry of a network's loss attracting even where it is a saddle, so that networks trained with SGD at large learning rates or small batches reach sparse, low-rank and identical-neuron solutions, as Ziyin proposes
Oct 20, 2026
THEORY-790: In deep networks trained by SGD on real data, minibatch noise drives parameter groups related by a rescaling symmetry toward the point where their gradient noise balances, at a rate set by learning rate over batch size, so that at stationarity they sit neither where gradient flow would leave them nor spread out as under isotropic noise, as Ziyin, Li and Ueda propose
Oct 20, 2026
THEORY-797: Deep classifiers trained with MSE loss stay near the least-squares classifier for their own features and move along the central path by the dynamics of continually renormalized gradient flow, whose equal-rate growth of the whitened class-separation singular values is how they reach neural collapse, as Han, Papyan and Donoho propose
Oct 20, 2026
THEORY-828: Fine-tuning from one pretrained model typically lands near the edge of a task-specific low-loss region in weight space that contains the convex hull of the fine-tuned models, and averaging their weights, as model soups and stochastic weight averaging do, works by moving inward
- loss-landscapes
- anthology-candidate
Oct 19, 2026
QUESTION-164: Does the law of balance hold in a deep network trained on real data during its limiting dynamics: for parameter groups related by a rescaling symmetry, does the difference of squared norms move toward the value where the groups' gradient-noise traces are equal, at a rate that scales with learning rate over batch size?
- learning-theory
- loss-landscapes
Oct 19, 2026
THEORY-732: For ReLU networks interpolated linearly between two endpoints, weak additivity and commutativity at every layer make each layer's features exactly the interpolation of the endpoints' features, output-layer linearity bounds the path's error by twice the endpoints', and for one hidden layer with interpolating minima linear mode connectivity alone gives the exact case
- loss-landscapes
- representation-learning
Oct 19, 2026
THEORY-736: Two ε-dropout-stable solutions of a multilayer ReLU net, under a loss convex in the output, are joined by a piecewise-linear path whose loss never exceeds the worse endpoint's by more than ε, and joined pairwise paths compose with no higher barrier than their worst
Oct 19, 2026
THEORY-741: A two-layer ReLU student of width at least kM, fitting an orthonormal teacher of M neurons on the sphere, has for any k global minima a global minimum joined to each by a two-piece linear path of minima; and in a deep linear network of depth L and width above 1 + r(L − 1), any r global minima in general position share a global minimum linearly connected to each
Oct 18, 2026
CASE-115: Two incompatible quadratic demands on one shared parameter, L₁ = (θ − 1)² and L₂ = (θ + 1)², frustrated at θ = 0 for every update budget, and the split of θ into task-routed θ_A and θ_B that makes both satisfiable
Oct 18, 2026
CLAIM-351: A zero aggregate gradient does not show that the demands it aggregates are reconciled: with positive weights it holds both when every demand's own gradient vanishes and when nonzero gradients cancel, and in the second case no update improves them all to first order; even the first is only joint first-order stationarity, which a demand's maximum or saddle also has
Oct 18, 2026
CLAIM-365: On a norm-ball update set the local frustration residual Φ_r = min over ‖d‖ ≤ r of max_i(ε_i + g_iᵀd) equals the maximum over the simplex of Σλ_iε_i − r‖Σλ_ig_i‖_*, so infeasibility of the declared improvements has a certificate, and any admissible actual update exceeds Φ_r by a nonnegative governance regret
Oct 18, 2026
CLAIM-366: Whether several objectives share a strict first-order descent direction, 0 ∉ conv{g_i}, is invariant under smooth invertible reparameterization, while the sign of a Euclidean gradient inner product is not, and a negative inner product does not exclude common descent; a budgeted version is invariant only with the metric pulled back
Oct 18, 2026
QUESTION-124: When do endorsement dynamics turn a transient conflict among an organization's demands into a persistent regime of frustration, and can the condition be stated in an architecture-sensitive update geometry such as modular duality's?
Oct 18, 2026
QUESTION-127: Which diagnoses of frustration among a learner's demands survive quotienting by function-preserving symmetries, which belong only to an update geometry, and which only to a particular training organization with its optimizer?
Oct 16, 2026
THEORY-425: A connecting path trained only on clean data keeps only the behaviour the clean data ask for: between two backdoored or error-injected image classifiers, models near the ends of such a path keep most clean accuracy and lose the attack, and a path flat in ordinary loss can carry a barrier in adversarial loss that tracks the input Hessian's top eigenvalue
- loss-landscapes
- anthology-candidate
Oct 16, 2026
THEORY-509: Under genericity conditions, two dense or top-k (k ≥ 2) mixture-of-experts layers compute the same function only if they differ by a permutation of the experts with their gates and a common translation of the gate weights, and the translation leaves every interpolation barrier unchanged, so expert order is the only gate symmetry alignment must undo
Oct 16, 2026
THEORY-521: In the thermodynamic limit, with examples and capacity growing together at fixed load, generalization error can jump discontinuously with the load when the entropy of hypotheses just above the best error vanishes faster than the training-error energy grows: a spherical perceptron's error falls smoothly as 1/α, an Ising perceptron's drops to zero at a critical load
Oct 16, 2026
THEORY-557: Low-loss paths between independently trained solutions are found outside supervised image classifiers: graph neural networks trained from different seeds on one graph are joined by quadratic Bezier curves where the straight line has a barrier, and one pair each of diffusion and contrastive models by chains of low-loss checkpoints
- loss-landscapes
- anthology-candidate
Oct 16, 2026
THEORY-578: Between two trained networks that compute nearly the same function, a chain of models minimising the summed Jensen–Shannon divergence between neighbours, a discretised Fisher–Rao geodesic found without labels, can be a low-loss path where the straight line in weights has a barrier
Oct 16, 2026
THEORY-675: Without a shared stable start, the linear barrier between solutions of one objective depends on the structure of the training data: between GCNs it falls with graph density and feature separability and is highest at middling homophily, and between federated-averaging models it grows with client heterogeneity and falls with width
Oct 14, 2026
THEORY-278: Under MSE loss, in Han, Papyan and Donoho's VGG, ResNet and DenseNet runs on five balanced ten-class datasets, the trained classifier stays near the least-squares classifier for its own features from an early epoch; and on that central path, for unconstrained features continually renormalized to identity within-class covariance and without weight decay, gradient flow drives the class-separation singular values to infinity at equal rates, which in that model is neural collapse
Oct 14, 2026
THEORY-286: Under class imbalance the last-layer geometry that unregularized cross-entropy prefers is not the simplex ETF but simplex-encoded-label interpolation: the logits equal the simplex-encoded labels, the classifier and embedding Gram matrices come from that matrix's SVD, any finite weight decay changes the geometry, and training approaches it more slowly as imbalance grows
Oct 13, 2026
CLAIM-263: Whether two trained networks compute the same way can be tested by quotienting the architecture's full symmetry group and measuring what remains: a barrier that survives alignment marks a different mechanism, connection after alignment means shared feature structure, and the test cannot run where features have no counterpart
- representation-learning
- loss-landscapes
Oct 13, 2026
LIT-1011: The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion
Oct 13, 2026
LIT-1012: Noise Balance and Stationary Distribution of Stochastic Gradient Descent
Oct 13, 2026
LIT-1013: Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics
Oct 13, 2026
LIT-1014: Symmetry Induces Structure and Constraint of Learning
Oct 13, 2026
QUESTION-080: When a trained network's parameters keep moving after its loss has converged, what about the network stays fixed, and is the last-layer geometry of neural collapse among the things that do?
Oct 13, 2026
QUESTION-081: When the relative scale of a network's layers sets its learning regime under gradient flow, does that regime survive SGD, whose minibatch noise drives the relative scale toward balance, and on what timescale compared with learning?
Oct 13, 2026
THEORY-209: A converged network trained with SGD is not at a point and does not sample its training loss: under a quadratic approximation it samples a Gibbs distribution on a modified loss, isotropic in the Hessian's column space, with probability currents that keep it oscillating, so its speed and spread are set by the learning rate, momentum and batch size
Oct 13, 2026
THEORY-210: Continuous parameter symmetries stretch a network's minima into connected manifolds along which the loss is flat, and give gradient flow conserved quantities that fix where on such a manifold training ends; finite steps, gradient noise and weight decay break them, so the position along a symmetry orbit is set by initialization and optimizer, not by the loss
Oct 13, 2026
THEORY-211: In the stochastic-differential-equation model of SGD, with a gradient-noise covariance that depends on the parameters, minibatch noise drives the difference of squared norms along a rescaling symmetry, at a rate proportional to learning rate over batch size, to the point on the symmetry orbit where the gradient-noise traces of the two parameter groups are equal, whenever both groups carry noise; so at stationarity SGD neither keeps the difference fixed, as gradient flow does, nor lets it diffuse, as isotropic noise does, and differs from gradient descent by a finite amount however small the noise
Oct 13, 2026
THEORY-213: Every mirror symmetry of a network's per-sample loss makes its symmetric subspace invariant under gradient descent and SGD, and when the loss's Hessian is bounded below by −c, weight decay γ‖θ‖² with γ > c/2 puts every minimum in that subspace; rescaling symmetries that hold at λ = −1, rotation symmetries and permutation symmetries contain such mirrors, whose subspaces are zeroed weights, reduced rank and identical neurons
Oct 12, 2026
CLAIM-187: Whether two systems compute the same way can be tested by quotienting the full admissible symmetry group and measuring what remains: a residual barrier marks a different mechanism, and a feature with no counterpart blocks the comparison
- representation-learning
- loss-landscapes
Oct 12, 2026
LIT-970: Symmetry in Neural Network Parameter Spaces
Oct 12, 2026
QUESTION-069: Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?
Oct 09, 2026
LIT-850: Revisiting Model Stitching to Compare Neural Representations
Oct 09, 2026
LIT-861: Topological Invariance and Breakdown in Learning
Oct 09, 2026
THEORY-184: Under a permutation-equivariant update whose map is K-Lipschitz, coincident neurons stay coincident at every step size, and at step size below 1/K distinct neurons cannot merge in finitely many steps
- loss-landscapes
- mathematics
Oct 03, 2026
LIT-651: Linear Mode Connectivity in Multitask and Continual Learning
Oct 03, 2026
LIT-652: The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-653: Essentially No Barriers in Neural Network Energy Landscape
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-654: Linear Mode Connectivity and the Lottery Ticket Hypothesis
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-655: Understanding Mode Connectivity via Parameter Space Symmetry
Oct 03, 2026
LIT-656: Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-658: Knowledge is a Region in Weight Space for Fine-tuned Language Models
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-659: Do Deep Neural Network Solutions Form a Star Domain?
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-660: Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-661: Git Re-Basin: Merging Models modulo Permutation Symmetries
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-662: Geodesic Mode Connectivity
Oct 03, 2026
LIT-663: Taxonomizing local versus global structure in neural network loss landscapes
Oct 03, 2026
LIT-664: Generalized Linear Mode Connectivity for Transformers
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-665: ZipIt! Merging Models from Different Tasks without Training
Oct 03, 2026
LIT-666: Mode Connectivity and Data Heterogeneity of Federated Learning
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-668: Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-669: Mechanistic Mode Connectivity
Oct 03, 2026
LIT-670: On Linear Mode Connectivity of Mixture-of-Experts Architectures
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-671: Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets
Oct 03, 2026
LIT-672: Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior
Oct 03, 2026
LIT-673: Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-674: Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity
Oct 03, 2026
LIT-675: TIES-Merging: Resolving Interference When Merging Models
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-676: Unveiling Mode Connectivity in Graph Neural Networks
Oct 03, 2026
LIT-677: Unveiling mode-connectivity of the ELBO landscape
Oct 03, 2026
LIT-680: Proving Linear Mode Connectivity of Neural Networks via Optimal Transport
Oct 03, 2026
LIT-681: Optimizing Mode Connectivity for Class Incremental Learning
- loss-landscapes
- anthology-candidate
Oct 03, 2026
LIT-682: Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity
- loss-landscapes
- anthology-candidate
Oct 03, 2026
THEORY-105: For RoBERTa-base fine-tuned on English classification data, with loss measured by a linear probe refitted on the frozen encoder, points sampled from the convex hull of models fine-tuned on one dataset, one task family or twelve GLUE and SuperGLUE datasets lost no more than any model fine-tuned elsewhere and beat the group's own members in 88% to 96.7% of comparisons, and the centroid of the general-level models was a better starting point for BitFit than the pretrained model
- loss-landscapes
- anthology-candidate
Oct 03, 2026
THEORY-107: Merging models fine-tuned from one pretrained start loses accuracy through interference in parameter coordinates: redundant small updates dilute the influential ones, and influential updates of different models conflict in sign
- loss-landscapes
- anthology-candidate
Oct 03, 2026
THEORY-108: Across load and temperature, the connectivity and output similarity of independently trained solutions predict test accuracy where endpoint curvature does not: a flat but poorly connected landscape generalizes worse than a connected one of the same curvature
Oct 03, 2026
THEORY-109: The solutions SGD finds in a deep network lie in one connected low-loss set, joined by simple curves and spanning volumes of measurable dimension, and this is a property of the solutions training finds, not of every minimum
- loss-landscapes
- anthology-candidate
Oct 03, 2026
THEORY-110: A set of trained solutions has a centre that is linearly connected, after alignment, to the rest of them: the solution set is a star domain even where it is not convex
- loss-landscapes
- anthology-candidate
Oct 03, 2026
THEORY-111: In image classifiers, between two networks linearly connected by spawning from a shared checkpoint or by permutation alignment (MLPs, and VGG-16, ResNet-20 and ResNet-50 on CIFAR-10 and Tiny-ImageNet), nearly every layer's features at three points on the line were proportional to the interpolation of the endpoints' features, and pairs that were not linearly connected were far from it
Oct 03, 2026
THEORY-112: Aligning independently trained solutions over the architecture's symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path's midpoint, took it to zero or near zero except on BookCorpus (0.42)
- loss-landscapes
- anthology-candidate
Oct 03, 2026
THEORY-113: Merging ResNets trained from different initialisations on disjoint halves of a label set by permuting one onto the other and averaging fell far short of the ensemble; their cross-model feature correlation fell with depth, merging only the early layers recovered most of the ensemble's accuracy, and merging features within each model as well as across both helped at 4× width and above; Stoica et al. attribute the failure to features with no counterpart in the other model
Oct 03, 2026
THEORY-114: In image classification, linear mode connectivity appeared during training and not at initialization: copies of ResNet-20 and VGG-16 on CIFAR-10 and of ResNet-50 and Inception-v3 on ImageNet, trained from a shared state, became stable to SGD noise in test error only after 1.5–20% of training (LeNet on MNIST was stable from the start), and between independently initialized MLPs on MNIST and CIFAR-10 the barrier after weight-matching alignment was high at initialization and fell gradually through training
- loss-landscapes
- anthology-candidate
Oct 03, 2026
THEORY-115: Linear connectivity from a shared, already-stable start holds only when the objectives the runs optimise share structure, and breaks as that structure is removed
Oct 03, 2026
THEORY-116: A linear barrier between two minima that survives the removal of symmetries marks a difference in the input attributes the two models rely on: they compute by different mechanisms

Created with Quartz v5.0.0 © 2026

Repository
Quartz