Viet-Hoang Tran, Van-Hoan Trinh, Khanh Vinh Bui and Tan M. Nguyen (2025), 39th Conference on Neural Information Processing Systems (NeurIPS 2025) — ARXIV-2509.11348
Key takeaways
- The gate’s symmetry group. G(n) = ℝ^d × ℝ × S_n acts by permuting experts with their gating rows and adding a common (c_W, c_b) to every gating row. Softmax is translation-invariant and the sum is permutation-invariant, so both dense and top-k MoEs are invariant (Propositions 3.1–3.2, on the dense open set where gate scores are distinct).
- It is the whole of the gate’s symmetry, generically. For dense gating, if experts are pairwise distinct functions and the gate differences W_i − W_j are pairwise distinct, equal functions imply equal expert counts and parameters related by G(n) (Theorem 4.1). For top-k with k ≥ 2, the same holds under stronger conditions: strongly distinct experts and linearly independent consecutive gate differences (Theorem 4.2). Top-1 has an extra positive-scaling symmetry and is set aside.
- Only the permutation matters for barriers. Adding a common translation to the gate leaves the loss barrier unchanged (Eq. 9, Proposition D.1).
- Algorithm 1. Order the experts by a linear assignment on centred gate weights or on Gram-matrix representations of each expert’s weights. Then weight-match the hidden units inside each matched pair, as in Git Re-Basin. Cost O(n³ + nh³), with no data.
- Barriers near zero. On One Billion Word with a 12-layer GPT-2 and 4 or 8 experts, the barrier after full matching is 0.0044–0.0066. Skipping the expert-order step leaves 0.14–0.86 (Table 10). On CIFAR-10 ViTs, matching leaves barriers at about 4–11% of naive interpolation’s (Table 3).
Standing in the record
Filed on 2026-10-03 with the owner’s mode-connectivity batch (ADR-027). It is the batch’s mixture-of-experts case and its most careful statement of which symmetries an architecture component has. The theorems go further than the experiments.
What is actually connected. Every experiment takes a pretrained ViT or GPT-2, replaces one feed-forward layer (first, last, or all) with a randomly initialised MoE, freezes everything else, and fine-tunes only the MoE from different seeds (§6.1, Appendix F). The models compared share all weights outside the MoE. That is LMC of one inserted layer inside a fixed network, a much easier setting than Git Re-Basin’s or Theus et al.’s (LIT-664) independently trained networks. The paper’s conclusion, that permutation symmetry suffices for LMC in MoE models, should be read with that scope.
It extends Git Re-Basin (LIT-661). Step 2 of its algorithm is Git Re-Basin’s weight matching, run per expert, and its expert-ordering step is the same linear-assignment construction applied one level up. The anthology reads Git Re-Basin as ANTH-LIT-333, and its practice ANTH-SOTA-217 (align hidden-unit permutations before averaging) would need the expert order aligned first for MoE layers. The paper takes its barrier definition and the term “winning permutation” from Entezari et al. (LIT-652), and cites Ferbach et al. (LIT-680) for formal guarantees of LMC under optimal-transport alignment and Sonthalia et al. (LIT-659) for star-shaped regions. Its §5.1 rests on that prior work for the step from “the gate’s symmetries are permutations” to “permutations suffice for LMC”.
It is flagged for the anthology: it carries a procedure for aligning MoE layers before averaging.