nucleation

CLAIM-498: Averaging weights within a basin averages features: the network whose weights are the average of several networks in one low-loss basin has, in every layer, the average of their features

2 min read

Properties7
StatusProposed
Rolethesis
Warrantinterpretive
Tagsloss-landscapes, representation-learning, anthology-candidate
Linelearned-structure
Bears onQUESTION-318 — Does layerwise linear feature connectivity hold outside image classification, in transformers and language models connected by spawning, shared pretraining or alignment, and does it carry from pairs to the many-model averages practice uses, model soups and SWA?
GroundsTHEORY-111 — In image classifiers, between two networks linearly connected by spawning from a shared checkpoint or by permutation alignment (MLPs, and VGG-16, ResNet-20 and ResNet-50 on CIFAR-10 and Tiny-ImageNet), nearly every layer's features at three points on the line were proportional to the interpolation of the endpoints' features, and pairs that were not linearly connected were far from it

Graph View

The claim

THEORY-111 holds that between two linearly connected image classifiers nearly every layer’s features at points on the line are proportional to the interpolation of the endpoints’ features. The record’s step is to read that as a statement about weight averaging in general: if the interior of a basin is linear in features as well as in loss, then the network whose weights are an average of several members of the basin computes, layer by layer, the average of their features. Weight averaging would then be feature averaging, which is what model soups and stochastic weight averaging do in practice. NOTE-512 proposed it in this form; Zhou et al. do not state it.

What it does not say

It is not measured. Every measurement behind THEORY-111 is an interpolation between two endpoints. A many-model average needs the conditions of Zhou et al.’s Theorem 1 to hold jointly for every member, and nothing in the record tests that. Nor does it say why averaging improves accuracy where it does.

Note of 2026-10-30: what holds it back

The measurement its defeated_if names has not been made: no reading in the record compares an averaged network’s layer features with the average of its members’ features, for soups of many fine-tuned models or for SWA iterates (QUESTION-318). Its ground, THEORY-111, measured pairs at three interpolation points and found features proportional to the interpolation in nearly every layer, not equal to it in every layer, so a reading that answers this should test proportionality layer by layer.

Backlinks