Dai et al., DeepSeek-AI (2024) — ARXIV-2401.06066
Key takeaways
- The diagnosis: conventional top-K-of-N routing, as in GShard, does not actually deliver expert specialisation — experts end up with overlapping rather than focused knowledge, which wastes the parameters sparsity was meant to buy.
- Two strategies, both about the granularity of the choice. Fine segmentation: split into mN smaller experts and activate mK of them, so the router picks from a much larger space of combinations at the same compute. Shared experts: isolate K_s experts that always fire, to hold the common knowledge every routed expert would otherwise learn separately.
- The results are ratios rather than records, which is the right way to read an efficiency claim: DeepSeekMoE 2B matches GShard 2.9B, which has 1.5× the expert parameters and computation; 16B matches LLaMA2 7B at ~40% of the compute; 145B approaches DeepSeek 67B at 28.5%.
- DeepSeekMoE 2B nearly reaches its dense counterpart at equal total parameters — the upper bound an MoE is trying to approach.
Standing in the anthology
The paper every frontier model in this record descends from, and the point where the line stops being about making sparsity work and starts being about making it specialise. Its predecessors are here too — LIT-188, LIT-187, LIT-189 — so the GShard this paper diagnoses by name can now be read rather than inferred. LIT-160 is DeepSeek-V3 adopting it at 671B, LIT-139’s V4 continues it, and Kimi K3’s “Stable LatentMoE” (LIT-131) — 16 of 896 routed experts — is the same fine-grained-plus-shared shape taken further than this paper tried.
That open question — whether the anthology carries MoE routing as practice — is now answered yes, and this note’s own argument against is answered in SOTA-150: being upstream is the reason to state it once rather than leave it as everything’s unexamined premise. This paper sources the design half (SOTA-149); the sparsity claim itself is sourced where it belongs, to LIT-188.