nucleation

THEORY-927: In a sparsely-gated mixture-of-experts layer trained without balancing losses, the gate came to give large weights to the same few experts, which the authors attribute to self-reinforcement, favoured experts being trained more and so selected more; in one 256-expert language model, the squared coefficient of variation of expert importance or load as an auxiliary loss reduced the peak-to-mean load from 17.8 to at most 1.47 and test perplexity from 39.8 to 35.6–35.7

1 min read

Properties5
StatusActive
Version2
Tagslearning-theory, complex-systems
SourceLIT-1507 — Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
Cited as “extends” byTHEORY-1218 — In a sparsely-gated mixture-of-experts layer trained without balancing losses, the gate's concentration on a few experts arises by self-reinforcement: favoured experts are trained more rapidly and so are selected still more

Graph View

Source

Shazeer, Mirhoseini, Maziarz, Davis, Le, Hinton & Dean (2017), LIT-1507, read in NOTE-1145 from arXiv:1701.06538v1: section 4 (pp. 5–6), Appendix A and Table 6 (pp. 13–14).

What was actually shown

  • The imbalance. With no balancing loss, CV of importance 3.04, CV of load 3.01, and the most loaded expert at 17.80 times the mean, in MoE-256 after 10 epochs on 1 billion words.
  • The fix. Any of five settings of the importance and load losses gives CVs of 0.02–0.48, peak-to-mean load 1.07–1.47 and test perplexity 35.6–35.7, against 39.8 with neither.
  • The mechanism, stated and not tested: “the favored experts are trained more rapidly and thus are selected even more by the gating network” (p. 5). Eigen et al. (2013) report the same phenomenon.

What this does not say

That the concentration was caused as the authors say: no trajectories, seeds or initializations are compared. That the starved experts lost competence rather than never gaining it. That balanced experts are equally competent; the losses equalize use, not merit, so they keep diversity in place from outside rather than letting a specialist earn its share. Or that the same holds for other architectures or longer training.

Backlinks