nucleation

QUESTION-317: Does aligning independently trained networks over the architecture's full symmetry group remove most of the linear barrier at standard width, on ImageNet or a language corpus, with an alignment computed without data; is the barrier that remains a difference in what the endpoints compute; and does it rise with depth?

2 min read

Properties5
StatusOpen
Tagsloss-landscapes, representation-learning
Linelearned-structure
RefinesQUESTION-069 — Is the connectedness of trained solutions a property of the solutions, or of the path family searched, the shared history that produced them and the regime they were trained in?
Cited as “bears on” byCLAIM-499 — The linear barrier left after aligning independently trained networks over the architecture's symmetries rises with depth, THEORY-112 — Aligning independently trained solutions over the architecture's symmetries removed the linear barrier between them for MNIST MLPs and 32×-wide ResNet-20s on CIFAR-10, two-thirds of it for an ImageNet ResNet-50, and not all of it for VGG-16 or ResNet-20 at 1× width, the barrier after weight matching falling with width; for 6- to 8-layer ViTs and GPT-2s, permutations alone removed 63–90% of it, and adding an orthogonal map on the residual stream, learned against the loss at the path's midpoint, took it to zero or near zero except on BookCorpus (0.42), THEORY-1188 — Once the architecture's symmetries are factored out, most of the linear barrier between independently trained solutions disappears; the group that matters is the full one, permutations for MLPs and CNNs but for transformers also an orthogonal map on the residual stream, without which a barrier remains; and the barrier left after alignment falls with width

Graph View

Why it is a question

THEORY-112 holds what alignment removed in each network measured: all of the barrier for MNIST MLPs and 32×-wide ResNet-20s, two-thirds for an ImageNet ResNet-50, not all of it at 1× width, and for small transformers nearly all of it once an orthogonal map on the residual stream, learned against the midpoint loss, was added. Whether most of the barrier is symmetry at the scales practice trains, and what the rest is, is not shown. Nor is the depth clause it once carried: on straight lines after alignment the only measurement is Entezari et al.’s handful of zero-barrier MNIST MLPs (Fig. 7), and Ferbach et al.’s compounding width is a sufficient condition, not a measurement.

What would answer it

THEORY-112’s former promote_when, unchanged:

The residual barrier after full-group alignment accounted for at a scale the sources do not reach. The zero barriers so far are for MNIST MLPs, a 32×-wide ResNet-20 on CIFAR-10, and small ViTs and GPT-2s whose alignment was learned against the loss at the midpoint. ImageNet ResNet-50 keeps a third of its barrier, and BookCorpus keeps 0.42. What would promote: independently trained models at standard width, on ImageNet or a language corpus, brought to near-zero held-out barrier by a data-free alignment over the architecture’s full symmetry group. Or the residual barrier shown to track a difference in what the endpoints compute, as LIT-669’s barriers do, so that “most” is the part that is symmetry and the rest is identified. What would refute it: same-data, same-procedure solutions whose barrier stays near the unaligned value under every alignment the full group allows, learned ones included, when the endpoints are shown to rely on the same input attributes. A third or fourth permutation-finding algorithm on vision models at moderate scale cannot settle it.

For the depth clause: the barrier after alignment tracked against depth at fixed width, with seeds, for convnets or transformers, under one alignment method fixed in advance.

Backlinks