loss-landscapes.
38 of 1538 NOTE documents. Back to the full index.
| # | Title | Summary | Status |
|---|---|---|---|
| NOTE-512 | Going Beyond Linear Mode Connectivity: The Layerwise Linear Feature Connectivity | LLFC: c·f⁽ℓ⁾(αθ_A + (1 − α)θ_B) = αf⁽ℓ⁾(θ_A) + (1 − α)f⁽ℓ⁾(θ_B) for every layer ℓ and α, some c > 0. Observed whenever LMC holds (spawning or permutation), across MLP, VGG-16, ResNet-20/50. Lemma 1: LLFC ⇒ error ≤ 2ε on the path. Theorem 1: weak additivity of ReLU + commutativity (W_A H_A + W_B H_B = W_A H_B + W_B H_A) ⇒ LLFC with c = 1. Weight and activation matching minimise the two factors of (W_A − PW_BPᵀ)(H_A − PH_B). | Read |
| NOTE-513 | On Linear Mode Connectivity of Mixture-of-Experts Architectures | G(n) = ℝ^d × ℝ × S_n (common gate translation + expert permutation) preserves dense and top-k MoEs and, under genericity, is all of the gate’s symmetry (Thms 4.1, 4.2, k ≥ 2). Translation leaves barriers unchanged. Weight matching = LAP on expert order (centred gates or Gram matrices) + per-expert weight matching, O(n³ + nh³). Barriers near zero (One Billion Word: 0.0044–0.0066 vs 0.14–0.86 without expert ordering), but only the inserted MoE layer is trained; the ViT/GPT-2 backbone is shared and frozen. | Read |
| NOTE-514 | Linear Mode Connectivity and the Lottery Ticket Hypothesis | Instability = max error on the line between two copies trained from the same state under different SGD noise, minus the mean endpoint error; stable means under 2%. Networks are unstable at initialization (except LeNet) and stable after 1.5–20% of training. IMP subnetworks at extreme sparsity are matching when, and only when, they are stable; rewinding to an early iterate makes them so at ImageNet scale. | Read |
| NOTE-515 | Knowledge is a Region in Weight Space for Fine-tuned Language Models | Task vectors of RoBERTa-base fine-tuned on 12 datasets × 20 seeds cluster by dataset with 98% accuracy, and by task family (NLI, sentiment, topic) with 90%. Hull samples of a group’s fine-tuned models beat every outside model on the group’s data and beat the group’s own members in 88% (MNLI) to 96.7% (NLI family) of comparisons; extrapolation past the members fails fast. The centroid of models from other datasets, re-tuned with BitFit, is a better start than the pretrained model on 9 of 12 datasets. | Read |
| NOTE-516 | Taxonomizing local versus global structure in neural network loss landscapes | mc(θ, θ′) = ½(L(θ) + L(θ′)) − L(γ(t*)) on a trained Bezier curve, in 0-1 training error (Eq. 4); mc < 0 is a barrier, mc ≈ 0 well connected, mc > 0 unconverged ends. With Hessian top eigenvalue and trace and output CKA on mixup points, a load by temperature grid splits into phases I–IV, with IV-B (flat, connected, similar) the most accurate. Sharpness alone mispredicts across phases; with 10% label noise, the sharp phases beat flat but poorly connected phase III, and double descent appears along the Hessian transition. | Read |
| NOTE-517 | Mode Connectivity and Data Heterogeneity of Federated Learning | FedAvg, 10 clients, 200 rounds, Dirichlet label skew α ∈ {0.1, 0.2, 0.5} and PACS. Client/global low-loss overlap shrinks with heterogeneity; global models from one start under different α disagree on > 20% of inputs, are separated on the line (about 10% accuracy drop) and joined by a one-bend chain. Theory: ε_C = C e^{C T_max} max(√h(α), √h(α′))((√log N + z)/√N + √ε(√(D + log N) + z)), via mean-field dropout stability. MNIST two-layer net: dropout error < 0.01 at N = 6400, up to 0.06 at N = 200. | Read |
| NOTE-520 | Linear Mode Connectivity in Multitask and Continual Learning | From a shared start ŵ₁, the multitask solution w₂ is linearly connected to ŵ₁ and ŵ₂, but ŵ₁ and ŵ₂ are not; the ŵ₁ → w₂ direction is nearly orthogonal to task 1’s top 50 Hessian eigenvectors, and ŵ₁ → ŵ₂ lies in their span. Different starts (CIFAR), input noise, label corruption and removed classes break the line. MC-SGD averages the loss along both lines with a one-example-per-class buffer: after 20 tasks 85.3 / 82.3 / 63.3% (Permuted MNIST / Rotated MNIST / Split CIFAR-100), multitask 89.5 / 89.8 / 68.8, Stable SGD 80.1 / 70.8 / 59.9. | Read |
| NOTE-521 | Geodesic Mode Connectivity | Length of a path in distribution space = √8 ∫ √dJSD (Fisher–Rao). Minimise the discrete energy Σ JSD(pᵢ‖pᵢ₊₁) over 25 interior models between two weight-matched ResNet-20 (4×, LayerNorm) CIFAR-10 networks, initialised on the line; the result keeps train and test loss low where the line has a large barrier, using unlabelled training images. One pair, no numbers in the text, hypothesis stated for all geodesics. | Read |
| NOTE-522 | Unveiling mode-connectivity of the ELBO landscape | Claims, without reported results, that LDA’s ELBO maxima from two SVI runs (New York Times corpus) are joined by essentially flat maximum-energy paths found by the simplified string method with natural-gradient bead updates; that the paths flatten with data size and number of topics K; and that statistical degeneracy, not only over-parameterisation (K > K*), explains it. The only figure is a schematic. | Read |
| NOTE-523 | ZipIt! Merging Models from Different Tasks without Training | Merging two differently initialised models trained on disjoint tasks: permute-then-average puts the result outside both task basins. Matching correlated features within and across models (Eq. 8) and merging only the early layers lifts CIFAR-10 (5+5) joint accuracy from 46.2 (Git Re-Basin) to 79.1 fully merged and 83.8 at 13/20 layers (ensemble 87.4). Fully merged ImageNet-1k (200+200) stays at 8.6; 10/50 layers reaches 60.9 (ensemble 63.3). Theorem 1 tightens Entezari et al.’s barrier bound by (1 − 2Γ)^{1/(2d+4)} for redundancy fraction Γ. | Read |
| NOTE-524 | Optimizing Mode Connectivity for Class Incremental Learning | In class-incremental learning (PODNet, CIFAR-100), the line between adjacent continual minima loses old-task accuracy in an interval. OPC adds a layer-wise Fourier perturbation δ_θ(λ) with δ(0) = δ(1) = 0 to the line, minimises old-task memory loss before a switching point λ* and new-task loss after it, with cylinder noise for flatness; EOPC averages 10 points near λ*. PODNet + EOPC: CIFAR-100 66.68 / 64.94 / 62.36 (5 / 10 / 25 steps) vs 65.47 / 63.13 / 59.85; forgetting 7.68 / 8.64 / 12.02 vs 19.26 / 25.01 / 28.55. | Read |
| NOTE-526 | TIES-Merging: Resolving Interference When Merging Models | Task vector τ = θ_ft − θ_init. TIES keeps the top 20% of each τ by magnitude, sets γ_m = sgn(Σ τ̂_t), averages only entries agreeing with γ_m, and returns θ_init + λ τ_m. Validated: (IA)³ 66.4 vs 63.9 for task arithmetic; T5-base 73.9 vs 73.2; T5-large 76.9 vs 73.3; ViT-B/32 73.6 vs 71.8 (RegMean); ViT-L/14 86.0 vs 84.5. Unvalidated T5-base 69.7 vs 73.2. Oracle multitask signs give 72.0 against 73.1 for multitask training. | Read |
| NOTE-527 | Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNs | Curve-finding: minimize E_{t∼U(0,1)} L(φ_θ(t)) over the bend θ of a one-bend chain or quadratic Bezier curve between two trained networks. The found curves keep loss and error near the endpoints where the line segment reaches near-chance error (Table 2). Points at t ≥ 0.4 already ensemble as well as the far endpoint. FGE, short cyclical learning rates late in training, beats Snapshot Ensembles at equal budget (Table 1). | Read |
| NOTE-528 | Understanding Mode Connectivity via Parameter Space Symmetry | Minimum of full-rank ‖Y − W_l…W₁X‖² ≅ GL_h(ℝ)^{l−1}: 2^{l−1} components, independent of width. 1-D three-layer: 4 components, 3 with a skip. h ≥ 2: permutations connect all components. Homogeneous last layers: rescaled minima in one component with unbounded midpoint loss, even under last-layer permutations (Props. 5.3–5.4). Symmetry curves γ(t) = exp(t log g)·w have constant loss; curvature ≤ κ bounds the chord’s distance from the level set by (1/κ)(1 − √(1 − (κ‖Δw‖/2)²)) ≈ κ‖Δw‖²/8. | Read |
| NOTE-529 | Unveiling Mode Connectivity in Graph Neural Networks | GCN node-classification solutions from different seeds show linear loss barriers that vary by graph (small on Cora, Citeseer; large on Coauthor-CS, Squirrel) and vanish along fitted quadratic Bezier curves. GCN, GraphSAGE and GAT give similar barriers, MLPs different ones. On CSBM graphs barriers fall with density and feature separability and peak at middling homophily. Stated without proof: a barrier bound through the spectral gap, a generalisation bound proportional to the barrier, and a Wasserstein distance between graphs’ interpolation-loss curves that bounds the domain-adaptation gap. | Read |
| NOTE-530 | Bridging Mode Connectivity in Loss Landscapes and Adversarial Robustness | Bezier path (Garipov) trained on 50–2,500 clean samples between two backdoored or error-injected models: models near the ends keep most clean accuracy and lose the attack (CIFAR-10 VGG, 2,500 samples, t = 0.1: 88% clean, 1.1% backdoor; injection 0% throughout). Standard-loss paths between regular and adversarially trained models are flat in standard loss but have a PGD robustness-loss barrier, correlated with λ_max of the input Hessian (PCC 0.71–0.88). Proposition 1’s proof assumes the loss is constant along the path in a way that, if true for all inputs, would remove the barrier. | Read |
| NOTE-531 | Essentially No Barriers in Neural Network Energy Landscape | AutoNEB finds paths between minima whose highest training loss is almost the minima’s for ResNets and DenseNets on CIFAR-10, with a small gap on CIFAR-100; test error rises at most 0.5% (CIFAR-10) and 2.2% (CIFAR-100) per §4.3, or 0.7% and 2.9% on the ResNets per Appendix B. Concatenating paths bounds every pair through a minimum spanning tree. Barriers shrink with width and depth. | Read |
| NOTE-532 | Git Re-Basin: Merging Models modulo Permutation Symmetries | Weight matching, coordinate descent over per-layer linear assignments for an NP-hard objective, aligns independently trained nets in seconds. After alignment, MNIST MLPs and wide CIFAR-10 ResNet-20s have zero loss barrier; 1×-width models and ImageNet ResNet-50 do not, and LMC appears only during training. A two-unit counterexample shows no permutation can connect some perfect solutions, so LMC is a property of SGD solutions. | Read |
| NOTE-533 | Do Deep Neural Network Solutions Form a Star Domain? | Conjecture 2: for width > αh (h the width at which SGD solutions are convex modulo permutation), the solution set is a star domain modulo permutation. Starlight (sample θ_n ∈ Z, t ∼ U[0,1], step on L((1−t)θ + tπ(θ_n)), re-permute by weight matching each epoch) finds θ⋆ whose train-loss barriers to held-out models are about 2–5× lower than regular–regular ones across 8 settings, falling with |Z| up to 50. Not zero; star loss itself can be high (DenseNet CIFAR-100 0.635 vs 0.006). BMA on the star domain: better AUROC, worse ECE than an ensemble. | Read |
| NOTE-534 | Mode Connectivity Beyond Classifiers: Evidence from Generative and Contrastive Models | One pair of DDPMs (Flowers102) and one pair of NanoCLIP models (Flickr30k) are joined by chains built by staged layer moves, per-layer variance correction and AdamW retraining to a loss threshold. DDPM training loss stays below 0.04 (AutoNEB peaks at 0.17), NanoCLIP below 0.09 (AutoNEB 6.69, accuracy 0.05). Ablations: all-layers, forward-order and SGD variants all raise the average loss. Intermediate DDPMs generate worse (higher FID) at endpoint-level training loss. | Read |
| NOTE-535 | Exploring Neural Network Landscapes: Star-Shaped and Geodesic Connectivity | Two-layer ReLU student of an orthonormal M-neuron teacher: the minimum manifold is explicit (Theorem 6); two random minima are 2-PL connected w.p. ≥ 1 − M((M²−1)/M²)^{m−2M}; any two are 4-PL connected for m ≥ 2M − 1; k minima share a linearly connected centre w.p. ≥ 1 − M((M^k−1)/M^k)^{m−kM}. Normalised geodesic distance is O(√M) for uniform minima but ≤ 1 + c₂/(r√m) for r-sparse ones. Deep linear nets: 2-PL a.s., 3-PL always, star centre a.s. for m > 1 + r(L − 1). Centres found empirically for FNN, VGG16, ResNet on MNIST/CIFAR-10 without permutation. | Read |
| NOTE-537 | The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks | Conjecture 1: wide SGD solutions can be permuted into one linearly connected basin. Theorem 3.1 proves an output-level version only for a one-hidden-layer net at uniform random init, rate Õ(h^{−1/(2d+4)}). The evidence is that independent solutions S and permutations S′ of one solution show the same barriers across width, depth and data; simulated annealing lowers barriers only for shallow nets on easy data. | Read |
| NOTE-538 | Mechanistic Mode Connectivity | Mechanistic similarity = invariance to the same unit interventions on the latents of the data-generating process. Cue-reliant and cue-invariant ResNet-18 and VGG-13 models are quadratic- but not linear-connected, even after activation matching. Conjecture 1 (no LMC up to symmetry ⇒ mechanistic dissimilarity) is not what Appendix F.3 proves. For a one-hidden-layer ReLU net with interpolating minimizers, Lemma 2 (LMC forces identical activation patterns) with Theorem 1 (mechanisms of different complexity force different patterns) proves the converse: dissimilar mechanisms of different complexity ⇒ no LMC under any permutation. The paper presents this as verifying the conjecture. CBFT (cross-entropy + barrier loss + class-mean invariance loss) removes the cue on synthetic CIFAR-10/100 and Dominoes. | Read |
| NOTE-539 | Loss Surface Simplexes for Mode Connecting Volumes and Fast Ensembling | SPRO: train connecting vertices to minimize mean loss at 5 uniform samples in a simplicial complex minus λ log volume. It joins up to 7 modes in one complex and finds a low-loss region of at least 10 dimensions for VGG-16 on CIFAR-10 (volume collapses at the 11th vertex). ESPRO, simplexes grown from single modes at about 10 epochs per vertex, beats deep ensembles in error, calibration and corruption robustness. | Read |
| NOTE-540 | Generalized Linear Mode Connectivity for Transformers | Symmetry hierarchy S1 permutation ⊂ S2 semi-permutation ⊂ S3 orthogonal ⊂ S4 invertible. Transformer alignment = P_FF per layer, semi-permutation P̃_H over heads (matched on QK and OV circuits), one orthogonal O on the residual stream after LayerNorm → RMSNorm. Test-loss barriers (Table 2): vanilla 1.69–4.34, weight matching 0.34–1.56, learned matching 0.00 (ViT ×3), 0.02 (Tiny Shakespeare), 0.42 (BookCorpus); permutation-only learned 0.29–1.60. Learned matching mainly refines O. Small models only. | Read |
| NOTE-542 | Rethinking generalization requires revisiting old ideas: statistical mechanics approaches and complex learning behavior | VSDL model: f = f(x; α, τ), α = m_eff/N lowered by label noise, τ raised by early stopping. Thermodynamic limit m, N → ∞ with α = m/N fixed. Continuous perceptron: s(ε) ~ ln ε, so 1/ε − α = 0 gives ε ~ 1/α (Eqs. 6–8). Ising perceptron: s(ε) ~ −(π²/2)ε² ln ε, and −π²ε ln ε = α has no solution for large α, so ε falls discontinuously to 0 at αc (Eqs. 9–13). Rigorous bound: error ≤ ε* + ε_τ, ε* the rightmost crossing of s(ε) and −α log(1 − ε) (Eqs. 14–18). No new theorem, no experiment. | Read |
| NOTE-543 | Proving Linear Mode Connectivity of Neural Networks via Optimal Transport | Layerwise neuron alignment equals a Wasserstein distance between empirical weight distributions (Birkhoff), so LMC modulo permutation follows when neuron weights are i.i.d. and layers are wide: for mean-field two-layer SGD (Theorem 3.1), and for deep Gaussian or sub-Gaussian nets with m̃_ℓ = Õ((T_ℓ/ε)^{m̃_{ℓ−1}}) (Theorem 5.2), tight by Theorem 5.3. Low-dimensional weights relax this (Theorem 5.4). | Read |
| NOTE-544 | Explaining Landscape Connectivity of Low-cost Solutions for Multilayer Nets | ε-dropout-stable ReLU nets are ε-connected by a piecewise-linear path (Theorem 1); ε-noise-stable nets by 10 segments with barrier Õ(ε) (Theorem 2), or 13 through a narrow low-loss net (Theorem 3). For any width there is a teacher dataset on which a two-layer student’s global minima are disconnected (Theorem 4). MNIST convnets and a VGG-11 on CIFAR-10 satisfy the conditions roughly. | Read |
| NOTE-653 | Revisiting Model Stitching to Compare Neural Representations | An empirical paper with no theorems. It defines the stitching penalty, the change in a top network’s test error when its bottom ℓ layers are replaced by another representation through a trained 1×1 convolution, and uses it to show, in single runs mostly on CIFAR-10, that networks from different seeds, and supervised and self-supervised ImageNet networks, are interchangeable layer by layer where CKA calls them different, and that a bottom trained on more data improves a weaker top. The ordering it gives is by one task’s loss through a fixed head and a restricted map class, which is not an ordering by information. | Read |
| NOTE-663 | Topological Invariance and Breakdown in Learning | Proves that a permutation-equivariant update with a K-Lipschitz update map changes the distance between any two neurons in one step by a factor in [1 − ηK, 1 + ηK], so coincident neurons never split, and for ηK < 1 the induced map on the set of neuron vectors is a bi-Lipschitz homeomorphism (a C¹ diffeomorphism under smoothness), and so no two neurons merge in finitely many steps. The proofs hold. The two-phase “topological simplification” reading and the Betti-number experiments go beyond them. The measured changes are fragmentation, not merging, at ηλ_max far below 1. | Read |
| NOTE-711 | The Limiting Dynamics of SGD | SGD with momentum β, learning rate η, batch size S and weight decay λ is modelled as (η/2)(1 + β)θ̈ + (1 − β)θ̇ + λθ = −g(θ) + √(η/S) √Σ ξ(t). For linear regression with Σ = σ²H this is an OU process in phase space. Its stationary state is Gibbs on a modified loss, with inverse temperature κ = S(1 − β²), in broken detailed balance. Under a quadratic loss at the end of training the same holds for deep networks, and a ResNet-18 on ImageNet shows the predicted isotropy, velocity regularization, oscillations and speed. | Read |
| NOTE-712 | Symmetry in Neural Network Parameter Spaces | A symmetry is a group action on parameters preserving the network function (Def. 2.4) or the loss (Def. 2.6), possibly only on given data or in expectation (Defs. 2.8–2.9). Common components have GL, scaling, sign-flip, orthogonal, translation and permutation symmetries (Table 1). Continuous symmetries make minima connected manifolds, and give gradient flow conserved quantities that label trajectories and endpoints (§5). SGD’s finite steps, noise and weight decay break these, which is an open route to implicit bias (§7.2). | Read |
| NOTE-713 | Neural Mechanics | A symmetry ψ of the loss gives ⟨∇L, ∂_α ψ⟩ = 0 and a Hessian constraint. Under gradient flow, Theorem 1 conserves ⟨θ, ∂_α ψ⟩: for translation ⟨θ_A, 1⟩, for scale |θ_A|², for rescale |θ_A1|² − |θ_A2|². With weight decay λ and the modified loss L + (η/4)|∇L|², these become exponentially decaying memories plus integrals of gradient norms (Eqs. 18–20), matched on VGG-16 with Tiny ImageNet. The noise covariance annihilates the generators (§5, App. F). | Read |
| NOTE-714 | Noise Balance and Stationary Distribution of Stochastic Gradient Descent | SDE: dθ = −∇L dt + √(T C(θ)) dW, with T = η/S. With ℓ(u, w) = ℓ(λu, w/λ), d(‖u‖² − ‖w‖²)/dt = −T(uᵀC₁u − wᵀC₂w) (Theorem 1). The stationary point is Tr C(w) = Tr C(u). With weight decay γ the rate is TC₀ + γ (Corollary 2.1). Diagonal linear network ℓ = (Σᵢ ∏ₖ u_i^{(k)} x − y)²: sign coherence and balance for depth D > 1 (Theorem 3), exact stationary distribution (Theorem 4), collapse above T_c = β₂/α₃ for D = 1, tail ∝ v^{−5+3/(D+1)} independent of data. | Read |
| NOTE-716 | Symmetry Induces Structure and Constraint of Learning | ℓ_γ = ℓ₀(θ, x) + γ‖θ‖². O-mirror symmetry: ℓ₀(θ) = ℓ₀((I − 2OOᵀ)θ). Theorem 1: (1) Oᵀθ = 0 implies Oᵀ∇ℓ_γ = 0; (2) at symmetric points the Hessian splits into ker(Oᵀ) and im(OOᵀ) blocks; (3) above some γ₀ the projection (I − OOᵀ)θ has lower loss; (4) above some γ₁ all minima satisfy Oᵀθ = 0. SGD near the symmetric set: Pθ_{t+1} = Pθ_t − λH(x)Pθ_t, attractive when the Lyapunov exponent E log|1 − λ(ξ + γ)| < 0. Corollaries: sparsity, low rank, homogeneous neurons (Theorems 2–4). | Read |
| NOTE-1188 | Wang, Hoogland, van Wingerden, Furman and Murfet — Refined local learning coefficients of attention heads | Restricts the local learning coefficient, singular learning theory’s measure of how degenerate the loss is near a parameter, to one component’s weights (weight-refined) or to another data distribution (data-refined), and tracks it per attention head over the training of a two-layer attention-only transformer. The heads’ curves start alike and diverge, and cluster by the heads’ functional types; refinement to code separates the two induction heads; refinement to a one-layer model’s outputs locates layer-0 heads giving up simple n-grams as a circuit with layer-1 bracket-matching heads forms. One model, one run in the main text, three more seeds in an appendix. | Read |
| NOTE-1462 | Chen, Lau, Mendel, Wei & Murfet — dynamical versus Bayesian transitions in TMS | In Elhage et al.’s toy model of superposition with two hidden dimensions, regular k-gons are critical points of a closed-form population loss, and their local learning coefficients predict, through the free energy formula, a Bayesian phase transition from the 5-gon to the 6-gon near n = 600 that MCMC confirms. SGD trajectories plateau at the same critical points and move to lower loss and higher estimated complexity. The paper’s Bayesian antecedent hypothesis is one-directional, that training’s transitions have Bayesian counterparts; it says outright that there is “no necessary relation” between the two kinds, and its main Bayesian transition, 5 → 6, was never seen in training. | Skimmed |
| NOTE-1467 | Hennick & De Baerdemacker — Almost Bayesian | Models late-training SGD as sub-diffusion on a “porous” loss landscape, with the local learning coefficient as the medium’s fractal dimension and a spectral dimension d_s fitted from displacement, R(t) ∝ t^(d_s/2λ). Where an effective scalar diffusion coefficient D_ξ is constant on a region, the steady state there is exp(−L/D_ξ), which is the Bayesian posterior tempered to an effective sample size 1/D_ξ. That within-region statement is all the derivation gives: how mass is shared between regions, which is where “accessibility” would act, is not derived, and in the one test it is fitted through a length scale. Displacement exponents are measured on MNIST MLPs, TinyStories models and ImageNet convnets. | Read |