Lepikhin et al. (2020) — ARXIV-2006.16668
Key takeaways
- MoE moves into the transformer. LIT-188’s layer sat between LSTM layers; here every other feed-forward block becomes a mixture of experts, which is the shape every model in this record still uses.
- The engineering is the contribution as much as the architecture. Sharding annotations on tensors, with the compiler deriving the parallel plan — so scaling to 2048 TPU v3 cores did not require rewriting the model. A 600B-parameter translation model over 100 languages in four days.
- Top-2 routing with a capacity factor, plus an auxiliary balancing loss and expert dropout. This is the “conventional top-K-of-N routing” that LIT-170 names four years later and argues does not deliver expert specialisation.
- Quality gains on low-resource languages that dense scaling had not reached.
Standing in the anthology
The middle term the record needed. LIT-170 diagnoses GShard by name and measures against it — DeepSeekMoE 2B matching GShard 2.9B at two thirds the expert parameters and compute — and the corpus held the measurement without the thing measured.
It has also been re-run in someone else’s codebase. Narang et al.’s audit of Transformer modifications (LIT-711) scores a mixture-of-experts Transformer, citing Mesh TensorFlow and this paper for it, against a vanilla T5 baseline under fixed hyperparameters: at 648M parameters it reaches early loss 2.148 ± 0.006 against 2.182 ± 0.005 and final loss 1.785 against 1.838, one of the few modifications in that audit to beat the baseline.
Filed for the lineage rather than for a practice of its own: the top-2 routing and the capacity factor are the design LIT-170 and LIT-171 both change, and reading either of them is easier with this in front of you.