Ankit Sonthalia, Alexander Rubinstein, Ehsan Abbasnejad and Seong Joon Oh (2024), arXiv preprint — ARXIV-2403.07968
Key takeaways
- The star domain conjecture (Conjecture 2). Let h be the width at which SGD solutions become convex modulo permutation. Then for some 0 < α < 1, networks wider than αh have solution sets that are star domains modulo permutation. There is a star model θ⋆ such that every solution, after weight matching to θ⋆, has near-zero loss barrier with it.
- Starlight. Train θ by sampling a source model θ_n from Z and t from U[0, 1], taking a gradient step on the loss at (1 − t)θ + tπ(θ_n) scaled by (1 − t), and re-permuting all source models to θ by weight matching once per epoch (Algorithm 1).
- Held-out solutions connect better to the star model than to each other. Training-loss barriers between the star model and held-out models, against barriers between two regular models: ResNet-18 CIFAR-10 0.078 vs 0.383, VGG19 0.336 vs 1.281, DenseNet CIFAR-100 3.735 vs 6.920, ResNet-18 ImageNet-1k 2.794 vs 5.948 (Table 1). The star-regular barrier falls as the number of source models grows from 2 to 50 and has not levelled off (Fig. 1). In a width sweep it is about a third of the regular-regular barrier (Fig. 2).
- By-products. Sampling along the lines from the star model gives better AUROC than a deep ensemble but worse ECE (Fig. 4). A star model trained with an added cross-entropy term beats a single model by about 0.1–1.1 points but stays below the ensemble (CIFAR-100, 50 models: 78.4 vs 77.3 single, 81.3 ensemble; Table 2).
Standing in the record
Filed on 2026-10-03 with the owner’s mode-connectivity batch (ADR-027). With Lin, Li and Wu (LIT-682) it is one of the batch’s two star-domain works. Lin et al. prove star-shaped connectivity in toy models without permutation; this paper tests it on real networks with permutation, and adds the held-out check that Lin et al. lack. The papers are concurrent, and this one cites the other as such.
It extends two foundations held here. It takes Entezari et al.’s (LIT-652) conjecture, that SGD solutions are convex modulo permutation once networks are wide enough, as its Conjecture 1. It then states its own conjecture as a relaxation of that one, defined in terms of the width at which Entezari’s conjecture holds. The anthology reads Entezari et al. as ANTH-LIT-251. Its permutations are Git Re-Basin’s (LIT-661) weight matching, run inside Starlight, and its own regular-regular baselines, which reconfirm that thin ResNets are not linearly connected after weight matching, are Git Re-Basin’s failure case (ANTH-LIT-333 in the anthology). Starlight’s Monte Carlo sampling along lines is modelled, the paper says, on Garipov et al.’s (LIT-673) curve fitting, and it places itself beside Benton et al.’s (LIT-656) low-loss simplexes as another way to connect many solutions at once.
On the relation to convexity: the paper does not refute Entezari’s
conjecture. That conjecture is conditioned on width, and this paper’s
evidence is that narrower networks fall short of it. The star domain is
offered as what holds below that width. So it is declared as extends,
not rivals or corrects.
It is flagged for the anthology because star models are offered as a cheaper substitute for an ensemble at inference time, which bears on the anthology’s ensembling practice (ANTH-SOTA-257).