anthology-of-the-sota

SOTA-149: Build the sparse layers from many small experts plus an always-on shared one, not a few large ones

2 min read

Properties7
StatusActive
Consensusemerging
Tagsmodel-architecture
SourceLIT-170 — DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Introduced byLIT-170 — DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Explained byTHEORY-005 — Dense feed-forward layers are already mixtures of experts, and pre-training settles the partition before the neurons
ExtendsSOTA-150 — Make the feed-forward layers a sparse mixture of experts once the model is large enough to be compute-bound

Graph View

SOTA-150 says make the layer sparse. This says how to cut it up, and the diagnosis behind it is the interesting part.

Conventional top-K-of-N routing does not actually deliver expert specialisation. Experts end up holding overlapping knowledge rather than focused knowledge, which wastes the parameters sparsity was supposed to buy — you pay for capacity and get redundancy.

LIT-170 attributes that to the granularity of the choice and changes it in two ways:

  • Fine segmentation. Split into mN smaller experts and activate mK of them. Same compute, but the router now picks from a combinatorially larger space, so a token can be served by a combination rather than by whichever single large expert is least wrong.
  • Shared experts. Isolate K_s experts that always fire. Knowledge every token needs stops being replicated across routed experts, which is what those experts were spending capacity on.

The measurement that matters most is the last one in the paper: DeepSeekMoE 2B nearly reaches its dense counterpart at equal total parameters. That is the ceiling a mixture of experts is trying to approach and normally does not, and approaching it is the specialisation claim being cashed out.

Where the rest of the family sits

Three practices’ worth of MoE material, which is what LIT-171 predicted:

  • SOTA-150 — make the layer sparse at all.
  • This one — how to cut it up.
  • SOTA-148 — how to keep the experts evenly loaded, with a bias rather than an auxiliary loss.

They are separable claims: you can accept sparsity and reject fine segmentation, or adopt fine segmentation and still balance with an auxiliary loss. The record files them apart so that a reader can.

Backlinks