anthology-of-the-sota

LIT-188: Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer

2 min read

Properties5
StatusActive
Tagsmodel-architecture
Extended byLIT-187 — GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding
Cited as “introduced by” bySOTA-150 — Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound
Cited as “source” bySOTA-150 — Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound

Graph View

Shazeer et al. (2017) — ARXIV-1701.06538

Key takeaways

  • Conditional computation, made to work. The idea — activate only parts of a network per example, so capacity and compute stop rising together — was a decade old and had never delivered. This paper names why: branching breaks GPU batching, large batches are needed to amortise the gating, loss terms are required to keep experts balanced, and network bandwidth becomes the limit.
  • The layer. Up to thousands of feed-forward experts between LSTM layers, with a trainable gating network selecting a sparse combination per example. Capacity up to 137B parameters, more than 1000× a comparable dense model, at minor compute cost.
  • Load balancing was a problem from the first paper. The gate self- reinforces — experts that win early get trained more and keep winning — and the remedy here is an auxiliary loss encouraging equal utilisation. That auxiliary loss is what LIT-171 removes seven years later.
  • Results on language modelling and machine translation at lower compute than the state of the art of the time.

Standing in the anthology

The origin the record was missing. Every MoE document here descends from this layer, and until now the corpus started at LIT-170 — a 2024 paper that opens by diagnosing what its predecessors got wrong. The anthology held the critique and not the thing being criticised.

Filed so that the practice recommending sparse feed-forward layers can name the paper that introduced them, which is what #17 asks of a practice and what it could not do while this was absent.

Two threads run from here into the modern record. The auxiliary loss this paper introduces for balance is the exact mechanism LIT-171 argues should be replaced by a bias, so the record now holds both ends of that argument. And the capacity-without-compute claim is what LIT-170 measures against dense models seven years later.

This layer has not been re-run in a modern Transformer audit. The MoE row in Narang et al.’s audit (LIT-711) is sometimes read as one, but that paper cites Mesh TensorFlow and GShard for it, and it is a Transformer with expert feed-forward layers, GShard’s design (LIT-187), not this LSTM layer. That result is recorded against GShard.

Backlinks