nucleation

CLAIM-499: The linear barrier left after aligning independently trained networks over the architecture's symmetries rises with depth

2 min read

Properties7
StatusProposed
Rolethesis
Warrantinterpretive
Tagsloss-landscapes, anthology-candidate
Linelearned-structure
Bears onQUESTION-317 — Does aligning independently trained networks over the architecture's full symmetry group remove most of the linear barrier at standard width, on ImageNet or a language corpus, with an alignment computed without data; is the barrier that remains a difference in what the endpoints compute; and does it rise with depth?
GroundsTHEORY-112 — Aligning independently trained solutions over the architecture's symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path's midpoint, took it to zero or near zero except on BookCorpus (0.42)

Graph View

The claim

At fixed width, deeper networks keep more of their straight-line barrier after alignment. The record put this together from two sources that THEORY-112 reads. Entezari et al. (LIT-652) find the unaligned barrier rising quickly with depth for MLPs and shallow CNNs, and saturating high for VGG and ResNet, and they blame depth rather than width. Their search found zero-barrier permutations only for MNIST MLPs of depth 1, 2 and 4. Ferbach et al. (LIT-680) bound the width layer ℓ needs for alignment to work by a quantity exponential in the width of the layer before, so the requirement compounds with depth, and their lower bound shows the rate is tight for independent neurons.

What it does not say

Neither source measures the barrier after alignment against depth. Entezari et al.’s curves are before alignment, and Ferbach et al.’s width is sufficient, not necessary, and falls if weights concentrate near a low-dimensional subspace. Nor does it hold for curved paths: Draxler et al. (LIT-653) find barriers on minimum-energy paths falling as networks get deeper.

Note of 2026-10-30: what holds it back

The measurement its defeated_if names has not been made: no reading in the record tracks the barrier after alignment against depth at fixed width under one alignment method (QUESTION-317). It composes Entezari et al.’s depth curves, which are before alignment, with Ferbach et al.’s width bound, which is sufficient and not necessary, and neither measures it.

Backlinks