nucleation

LIT-1514: Adaptive Mixtures of Local Experts

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1147) from the scan of the published article on Geoffrey Hinton's University of Toronto page; worth reading as the origin of the mixture of experts and as a learning rule in which authority over a case goes to whichever expert does better than the gate-weighted average. Redefining the error as that of a single expert chosen stochastically by the gate makes the experts compete rather than cooperate, and the log-likelihood form adapts the best-fitting expert fastest. On a four-class vowel task, mixtures of 4 or 8 linear experts reached criterion in about half the epochs of backpropagation networks with matched parameters, at the same accuracy, and in every run all but 2 or 3 experts ended with mixing proportions effectively 0. The authors read that as economy; nothing in the paper tests whether the idle experts could recover.
Tagslearning-theory, probabilistic-modeling, anthology-candidate
Cited as “paper” byNOTE-1147 — Jacobs, Jordan, Nowlan and Hinton — Adaptive mixtures of local experts
Cited as “source” byTHEORY-937 — When a mixture of experts is trained on the error of a single expert chosen stochastically by the gate, or on the negative log-likelihood of the mixture, the gradient raises the gate's logit for an expert that fits a case better than the gate-weighted average and lowers it for one that fits worse; on a four-class vowel task, mixtures of 4 or 8 linear experts learned in about half the epochs of backpropagation networks of roughly equal parameter count, at equal accuracy, and divided the task by class pair, leaving all but 2 or 3 experts with mixing proportions effectively zero

Graph View

Jacobs, R. A., Jordan, M. I., Nowlan, S. J. & Hinton, G. E. (1991). Adaptive mixtures of local experts. Neural Computation 3(1): 79–87. doi:10.1162/neco.1991.3.1.79.

Key takeaways

From the reading, NOTE-1147. Page numbers are the journal’s.

  • Competition by the error function. Blending experts’ outputs makes each expert cancel the others’ residual, which “causes them to cooperate nicely, but tends to lead to solutions in which many experts are used for each case” (p. 80). Scoring the expected error of one expert chosen by the gate (Eq. 1.2), or the negative log-likelihood of a mixture (Eq. 1.3), makes each expert produce the whole output, so they compete.
  • Responsibility follows relative success. An expert that does better than the gate-weighted average on a case gains responsibility for it, and one that does worse loses it (pp. 81–82). Under Eq. 1.3 the best-fitting expert is adapted fastest (p. 82; re-derived in the NOTE).
  • Results. On four-class vowel discrimination (75 speakers, first two formants), mixtures of 4 or 8 linear experts matched backpropagation networks of equal parameter count on accuracy (88% train, 90% test) and reached the error criterion in about half the epochs (Table 1, 25 runs each).
  • Symmetry breaking and idle experts. From equal mixing proportions, experts first head for the one-line average solution, then split by class pair. In every run “all but 2 or 3 experts had mixing proportions that were effectively 0 for all cases” (p. 87).

Why this is here

Filed on 2026-10-24 as the origin of the gating dynamics that the lead read as CLAIM-277’s rich-get-richer entrenchment. The reading narrows that (NOTE-1147): here the same loop is the mechanism of specialization, the authors present the idle experts as economy, not as a failure, and no recovery of an idle expert is tested, which is QUESTION-132’s missing measurement. Filed THEORY-937. It carries anthology-candidate as an ML method, and no instruction for practice.

Backlinks