nucleation

LIT-1411: Theory on Mixture-of-Experts in Continual Learning

1 min read

Properties4
StatusActive — read 2026-10-29 (NOTE-1380), the main paper in full and the experimental appendix, from arXiv v3; worth reading as a solvable model in which identically initialized experts specialize into task sets under a locality loss, and in which a Switch-style balancing loss, left running, erases the gate gaps that encode the specialization, so the gate must be frozen. Read it knowing that the tasks are linear and carry a feature signal that makes routing separable, that task arrival is uniform, that the necessity of termination comes from its own balancing term, and that the proofs (Appendices B–J) were not checked in this reading.
Version2
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-1380 — Li, Lin, Duan, Liang and Shroff — Theory on mixture-of-experts in continual learning

Graph View

Li, H., Lin, S., Duan, L., Liang, Y. and Shroff, N. B. (2025). Theory on mixture-of-experts in continual learning. ICLR 2025. arXiv:2406.16437.

Read on 2026-10-29 (NOTE-1380): the main paper and the experimental appendix in full, the proofs not checked.

Why this is here

The exchange cited it for CASE-117: specialization emerging among identically parameterized operators whose routing and training co-evolve over a changing task distribution is what the paper proves for a gated MoE, and its result that the gate must stop updating for convergence is a condition CASE-117 should test. The exchange gave no description to check. It carries the anthology-candidate flag (ADR-005) because the abstract draws practical design guidance for MoE training.

Backlinks