Source
Zhou, Yang, Yang, Yan & Hu (2023), LIT-674, read in NOTE-512; and Lubana, Bigelow, Dick, Krueger & Tanaka (2022), LIT-669, read in NOTE-538, for Lemma 2. What the pairs of image classifiers measured showed is THEORY-111, which this entry extends.
The claim
Zhou et al. name layerwise linear feature connectivity: the feature maps of every layer of two trained networks are linearly connected, so the features of the network on the straight line between them are the interpolation of the endpoints’ features. They state it as a general regularity, that whenever two networks are linearly mode connected, by spawning or by permutation, they satisfy it in nearly all layers (NOTE-512, C1). Their definition allows a scale factor, and their Theorem 1 gives it exactly, with scale 1, under two conditions, weak additivity of ReLU and a commutativity of weights with features. Lubana et al.’s Lemma 2 derives the first of those for a one-hidden-layer network, where the claim is then a theorem.
The measurements are pairs of image classifiers at three points on the line, and the scale fitted in practice is not 1. The claim is that the same holds for any two networks joined by a low-loss line.
What would settle it
The promote_when, which is the layerwise half of QUESTION-318: the same measurement in transformers and language models connected by spawning, shared pretraining or alignment, or an approximate version of Theorem 1. More pairwise image-classifier measurements cannot settle it.