nucleation

QUESTION-349: On tasks other than Jacobs and colleagues' four-class vowel discrimination, do mixtures of experts trained on the error of a single expert chosen stochastically by the gate, or on the mixture's negative log-likelihood, leave most experts with near-zero mixing proportions while matching the accuracy of a dense network of equal size?

1 min read

Properties4
StatusOpen
Tagslearning-theory, probabilistic-modeling
Linedistributed-agency
Cited as “bears on” byTHEORY-937 — When a mixture of experts is trained on the error of a single expert chosen stochastically by the gate, or on the negative log-likelihood of the mixture, the gradient raises the gate's logit for an expert that fits a case better than the gate-weighted average and lowers it for one that fits worse; on a four-class vowel task, mixtures of 4 or 8 linear experts learned in about half the epochs of backpropagation networks of roughly equal parameter count, at equal accuracy, and divided the task by class pair, leaving all but 2 or 3 experts with mixing proportions effectively zero

Graph View

Why it is a question

THEORY-937 reports one study (NOTE-1147): on a four-class vowel task with two natural subtasks, mixtures of 4 or 8 linear experts left all but 2 or 3 experts with mixing proportions effectively zero, and matched backpropagation networks in accuracy. The authors read the idle experts as economy: “the system tends to use as few experts as it can to solve a problem” (p. 84). A task that needs two or three lines cannot show whether that holds where the subtasks are less obvious. In large sparse mixtures Shazeer et al. (LIT-1507) report imbalance among experts as a failure, so the benign reading of the vowel result may not carry.

What would answer it

A second study, on another task, in which mixtures trained this way leave most experts with near-zero mixing proportions while matching a dense network of equal size. A run in which they do not, under the same training errors, answers no for that task. Whether an idle expert could regain responsibility is QUESTION-132’s question, and a result about it does not answer this one.

Backlinks