nucleation

THEORY-1218: In a sparsely-gated mixture-of-experts layer trained without balancing losses, the gate's concentration on a few experts arises by self-reinforcement: favoured experts are trained more rapidly and so are selected still more

1 min read

Properties4
StatusProposed
Tagslearning-theory, complex-systems
SourceLIT-1507 — Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer
ExtendsTHEORY-927 — In a sparsely-gated mixture-of-experts layer trained without balancing losses, the gate came to give large weights to the same few experts, which the authors attribute to self-reinforcement, favoured experts being trained more and so selected more; in one 256-expert language model, the squared coefficient of variation of expert importance or load as an auxiliary loss reduced the peak-to-mean load from 17.8 to at most 1.47 and test perplexity from 39.8 to 35.6–35.7

Graph View

Source

Shazeer et al. (2017), LIT-1507, section 4 (p. 5); read in full in NOTE-1145 from arXiv v1. What the paper observed and measured is THEORY-927, which this entry extends.

Why it is Proposed

“the favored experts are trained more rapidly and thus are selected even more by the gating network” (p. 5) is the authors’ explanation. No trajectories, seeds or initializations are compared, so early selection is not separated from differences in initial competence.

Backlinks