nucleation

CLAIM-263: Whether two trained networks compute the same way can be tested by quotienting the architecture's full symmetry group and measuring what remains: a barrier that survives alignment marks a different mechanism, connection after alignment means shared feature structure, and the test cannot run where features have no counterpart

3 min read

Properties9
StatusProposed
Version3
Rolethesis
Warrantinterpretive
Tagsrepresentation-learning, loss-landscapes
Linelearned-structure
GroundsTHEORY-111 — In image classifiers, between two networks linearly connected by spawning from a shared checkpoint or by permutation alignment (MLPs, and VGG-16, ResNet-20 and ResNet-50 on CIFAR-10 and Tiny-ImageNet), nearly every layer's features at three points on the line were proportional to the interpolation of the endpoints' features, and pairs that were not linearly connected were far from it, THEORY-112 — Aligning independently trained solutions over the architecture's symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path's midpoint, took it to zero or near zero except on BookCorpus (0.42), THEORY-113 — Merging ResNets trained from different initialisations on disjoint halves of a label set by permuting one onto the other and averaging fell far short of the ensemble; their cross-model feature correlation fell with depth, merging only the early layers recovered most of the ensemble's accuracy, and merging features within each model as well as across both helped at 4× width and above; Stoica et al. attribute the failure to features with no counterpart in the other model, THEORY-115 — Linear connectivity from a shared, already-stable start holds only when the objectives the runs optimise share structure, and breaks as that structure is removed, THEORY-116 — A linear barrier between two minima that survives the removal of symmetries marks a difference in the input attributes the two models rely on: they compute by different mechanisms
SupportsCLAIM-187 — Whether two systems compute the same way can be tested by quotienting the full admissible symmetry group and measuring what remains: a residual barrier marks a different mechanism, and a feature with no counterpart blocks the comparison
ComplementsCLAIM-366 — Whether several objectives share a strict first-order descent direction, 0 ∉ conv{g_i}, is invariant under smooth invertible reparameterization, while the sign of a Euclidean gradient inner product is not, and a negative inner product does not exclude common descent; a budgeted version is invariant only with the metric pulled back

Graph View

The claim

The record’s mode-connectivity readings, taken together, give a protocol for asking whether two independently trained networks have found the same solution:

  1. Quotient the full group. Once the architecture’s symmetries are factored out, most of the linear barrier between independently trained solutions disappears. The group must be the full one: for transformers it includes an orthogonal map on the residual stream, since permutations alone leave a barrier (THEORY-112).
  2. Read the residue. A barrier that survives the removal of symmetries “marks a difference in the input attributes the two models rely on: they compute by different mechanisms” (THEORY-116).
  3. Inside a basin, structure interpolates. In the image classifiers measured, along a low-loss linear path nearly every layer’s features were proportional to the interpolation of the endpoints’ (THEORY-111). So connection after alignment is sameness of feature structure, not only of loss.
  4. Know when the test cannot run. Merging fails, Stoica et al. argue, where features have no counterpart in the other model (THEORY-113), and connectivity holds only when the objectives share structure (THEORY-115).

The protocol turns “do these two networks compute the same way?” into a measurement: quotient, then look for a barrier. It also says what a failed comparison means, which is that the test does not apply, not that the answer is no.

Where the record applies it

CLAIM-187 carries the protocol to communicative realizations, where the admissible group, the path and the measure would all have to be defined. It rests on this claim for the protocol and keeps that carry-over, its weak point, as its own. THEORY-108’s phase III is a ready case for the protocol inside this line: whether its poor connectivity survives permutation alignment.

What it does not say

  • It does not say why training reaches connected solutions. That is the line’s open question, and this claim is a way of testing for it.
  • It does not say a symmetry group is always known. For an architecture it is derived; elsewhere it must be chosen.
  • It rests on two Proposed readings, THEORY-115 and THEORY-116. Hence Proposed.

Note of 2026-10-27: universality, the hypothesis its test would probe

Zoom In (LIT-1543, NOTE-1182), read on 2026-10-26 at the owner’s request, states universality: analogous features and circuits form across networks trained on different data and with different architectures (THEORY-986). That is the hypothesis this claim’s test would probe across models, a barrier that survives alignment marking a different mechanism and connection after alignment marking shared feature structure. Zoom In names the test the hypothesis needs and offers evidence it calls “anecdotal”, from vision networks. The nearest measurement the record holds is Towards Monosemanticity’s agreement of features across two seeds of one architecture (NOTE-1191), which compares features by their activations, not by the quotient this claim asks for. The claim is unchanged.

Note of 2026-10-29: what holds it back

Step 2 is THEORY-116’s stated direction, that a barrier surviving alignment marks a different mechanism, which no held source tests on deep networks trained on the same data: the evidence is co-occurrence in one synthetic-cue protocol and a one-layer proof of the converse. Step 4’s “only when the objectives share structure” is THEORY-115’s, whose breaking cases plot raw loss that the endpoint’s forgetting largely explains.

Backlinks