nucleation

LIT-1507: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

2 min read

Properties4
StatusActive — read in full 2026-10-24 (NOTE-1145) from arXiv v1, the only version; worth reading here for one observation and its fix, not for its scale results. The gate of a sparsely-gated mixture of experts "tends to converge to a state where it always produces large weights for the same few experts", which the authors call self-reinforcing because favoured experts are trained more. Two auxiliary losses, the squared coefficient of variation of each expert's summed gate weight and of a smooth estimate of its load, remove the imbalance. In the one ablation (a 256-expert model, 10 epochs) no loss gives CV 3.04 and a peak-to-mean load of 17.8 at test perplexity 39.8, any loss gives 35.6–35.7. The fix equalizes use regardless of merit; the noise in the gate is there mainly to make the load estimate differentiable.
Tagslearning-theory, complex-systems, anthology-candidate
Cited as “paper” byNOTE-1145 — Shazeer et al. — Outrageously large neural networks: the sparsely-gated mixture-of-experts layer
Cited as “source” byTHEORY-1218 — In a sparsely-gated mixture-of-experts layer trained without balancing losses, the gate's concentration on a few experts arises by self-reinforcement: favoured experts are trained more rapidly and so are selected still more, THEORY-927 — In a sparsely-gated mixture-of-experts layer trained without balancing losses, the gate came to give large weights to the same few experts, which the authors attribute to self-reinforcement, favoured experts being trained more and so selected more; in one 256-expert language model, the squared coefficient of variation of expert importance or load as an auxiliary loss reduced the peak-to-mean load from 17.8 to at most 1.47 and test perplexity from 39.8 to 35.6–35.7

Graph View

Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G. & Dean, J. (2017). Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. ICLR 2017. arXiv:1701.06538.

Key takeaways

From the reading, NOTE-1145. Page numbers are arXiv v1’s.

  • The observation. Without constraint the gate converges onto “the same few experts. This imbalance is self-reinforcing, as the favored experts are trained more rapidly and thus are selected even more by the gating network” (p. 5). Eigen et al. saw the same and used a hard constraint early in training. It is stated in one paragraph, with no study of its dynamics.
  • The fix. Two losses: the squared coefficient of variation of each expert’s summed gate weight over a batch (importance), and of a smooth estimate of how many examples it receives (load), the second made differentiable by the gate’s Gaussian noise (Appendix A). Gate weights start at zero, so load starts nearly equal.
  • The ablation (Table 6, 256 experts, 10 epochs): no loss, CV of importance 3.04, max/mean load 17.80, perplexity 39.8; any of the five loss settings, CV 0.03–0.48, max/mean 1.07–1.47, perplexity 35.6–35.7.
  • Scale results. On 1-billion-word language modelling and WMT’14 translation, the MoE models beat the best published results of the time at lower computation, and experts specialize by syntax and semantics (Table 9).

Why this is here

Filed on 2026-10-24 for its gating collapse, which the lead read as CLAIM-277’s and CLAIM-443’s rich-get-richer entrenchment, and its load balancing, read as protected access (NOTE-1145). The reading supports the first: this is the entrenchment loop in a working system, with the mechanism named by the authors. It corrects the second: the balancing losses impose equal use regardless of competence, which is QUESTION-132’s imposed diversity, not CLAIM-362’s protected access to consequential roles. Filed THEORY-927. The record already holds two later load-balancing works, unread, Switch Transformers (LIT-1412) and the auxiliary-loss-free method (LIT-1414). It carries anthology-candidate as an ML method, and no instruction for practice.

Backlinks