Zhanran Lin, Puheng Li and Lei Wu (2024), arXiv preprint — ARXIV-2404.06391
Key takeaways
- The minimum manifold of a two-layer ReLU teacher–student model is explicit. With an orthonormal teacher of M neurons, inputs uniform on the sphere and m ≥ M student neurons, every global minimum has student neurons that are zero or a positive multiple of one teacher direction, with the weights on each teacher direction summing to one (Theorem 6). Two minima are linearly connected exactly when every student neuron is zero in one of them or points the same way in both (Lemma 7).
- Two pieces suffice, given width. Two minima drawn uniformly from the manifold are 2-piece linearly connected with probability at least 1 − M((M² − 1)/M²)^{m − 2M} (Theorem 8). Any two are 4-piece linearly connected once m ≥ 2M − 1 (Theorem 9).
- Star-shaped connectivity. For k minima, a common centre linearly connected to all of them exists with probability at least 1 − M((M^k − 1)/M^k)^{m − kM} (Theorem 10). With m ≥ kM, a centre 2-piece-connected to all k always exists (Theorem 11). For deep linear networks of depth L and width m > 1 + r(L − 1), r minima almost surely share a linearly connected centre (Theorem 18).
- Geodesic connectivity comes from sparsity. For minima drawn uniformly, the normalised geodesic distance is O(√M) whatever the width (Theorem 12). For minima drawn with a fraction r of neurons switched off, as SGD tends to find, it is at most 1 + c₂/(r√m), which tends to 1 as the width grows (Theorem 14).
- In practice, with no permutations. Minimising the expected loss along the lines from a candidate centre to each of five trained models finds a centre whose fold-line barriers are near zero (loss barrier 3.1e-05 to 1.0e-02 against 1.25 to 16.91 on the direct lines; Table 1). The estimated normalised geodesic distances are 1.001 to 1.051 (Table 2).
Standing in the record
Filed on 2026-10-03 with the owner’s mode-connectivity batch (ADR-027). It is the batch’s theoretical source for star-shaped connectivity, and it reaches the result with no permutation step at all. That sets it apart from Sonthalia et al. (LIT-659), who arrive at a star domain modulo permutation as a relaxation of the convexity conjecture. The two were concurrent. Sonthalia et al. cite this paper as obtaining star models “linearly connected with a finite number of solutions to larger nets like the VGG16”, and differ from it by adding permutations.
It extends Garipov et al. (LIT-673). The paper says its investigation is “inspired by” Garipov’s observation that connecting paths can be piecewise linear with as few as two pieces, and its theorems explain when that two-piece chain exists. It also tightens Kuditipudi et al. (LIT-671), who proved paths of at most ten linear segments under dropout or noise stability. This paper gets two to four segments, but only in its teacher–student and linear settings, which are much narrower than Kuditipudi’s.
Its “geodesic” is the shortest path inside the minimum manifold under the Euclidean metric on parameters. That is a different object from the Fisher–Rao geodesic of Tan et al. (LIT-662).
No instruction for practice. It is flagged for the anthology as part of the landscape line the anthology holds through Entezari et al. and Git Re-Basin.