A dense transformer runs every parameter for every token, so capacity and cost rise together. A mixture of experts breaks that coupling: hold many feed-forward experts, route each token to a few, and total parameters stop determining per-token compute.
The ratios are the right way to read the claim, because an efficiency result stated as a benchmark score hides what it cost (LIT-170):
| DeepSeekMoE 2B | matches GShard 2.9B, which has 1.5× the expert parameters and compute |
| DeepSeekMoE 16B | matches LLaMA2 7B at ~40% of the compute |
| DeepSeekMoE 145B | approaches the same team’s dense 67B at 28.5% |
“Once the model is large enough to be compute-bound” is doing real work in the title.
SOTA-125 is the record recommending the opposite at the other end: at a tiny parameter budget, spend on depth and state width, not on experts nobody can afford to hold in memory. Nothing here argues for sparsity at small scale.
A re-run at 223M, in the codebase it came from
Narang et al. (LIT-711) put about fifty transformer modifications through one T5 setup with fixed hyperparameters, and most of them lost to the dense baseline. Sparse experts did not. Mixture of experts (648M total parameters) reached early loss 2.148 and Switch (1.1B) 2.135, against 2.182 ± 0.005 dense, at about 11.7T against 11.1T operations. Switch had the best final loss in the whole table, 1.758 against 1.838.
The paper files both under the codebase caveat itself: they “were originally invented in the Mesh TensorFlow codebase that we use”. So this confirms the measurement at small scale and says nothing about transfer. It also measures a cost the FLOP match hides. Steps per second were 3.20 and 3.18 against 3.50, about 9% slower at equal FLOPs. A 223M model is at the small end of the scale range the title’s condition excludes, so this is a cost measured where the practice does not apply, not a counterexample to it.
Why this was not filed until now
The record has held this material for a while and deliberately left it unfiled. LIT-170 put the question plainly: whether the anthology carries MoE routing as practice, “and it is not obvious it should, since the choice is upstream of almost everything it does recommend.”
That is the argument this practice answers rather than ignores. Being upstream is a reason to say it once, clearly, rather than to leave it as the unexamined premise of a dozen downstream recommendations — which is what it had become.
SOTA-146 already carries a caveat about RL being unstable on mixture-of-experts specifically, and until now that caveat qualified an architecture the record had never recommended.
Where the width goes, once you have said yes
This practice answers whether to make the feed-forward layers sparse. It does not answer where the parameters should sit, and LIT-196 argues that the second question has a different answer than the field’s defaults assume.
Its premise is a roofline result: at latency-critical batch sizes the per-expert token count is small, arithmetic intensity is low, and expert computation is bandwidth-bound rather than compute-bound. Optimising accuracy per FLOP is then optimising against the wrong constraint, and accuracy per parameter — memory footprint, weight-loading bandwidth, routing traffic, sharding — is what binds. LatentMoE follows that by down-projecting tokens into a narrow latent space before the routed experts and keeping those experts’ weights there, while routing and the shared experts stay at full width; dispatch volume and weight-loading bandwidth both fall by the width ratio, and the saving is spent on more experts rather than on a smaller model.
Worth carrying here because it is the base Kimi K3’s expert layer is built on (LIT-131), and because the premise is the sort of claim that silently decides an architecture. Not filed as its own practice: one originating lab with one outside adopter, and no independent comparison against a conventional MoE at matched serving conditions.
The line this came from
The record now holds it end to end, which it did not when this practice was first drafted:
| LIT-188 | 2017 | the sparsely-gated layer itself, between LSTM layers, and the auxiliary balancing loss that came with it |
| LIT-187 | 2020 | into the transformer, sharded across 2048 TPUs; top-2 routing with a capacity factor |
| LIT-189 | 2021 | top-1 routing, a float32 router for stability, a trillion parameters |
| LIT-170 | 2024 | many small experts plus a shared one, and the dense comparisons above |
| LIT-160 | 2024 | the recipe at frontier scale — 671B total, 37B active, trained end to end and reported in full, which is what made the argument load-bearing rather than a scaling study |
Reading the granularity argument across them is the reason the whole line is worth holding: GShard says route to two experts, Switch says one, DeepSeekMoE says many small ones and take more of them. Three answers to how finely the router should choose, and the record recommends the third (SOTA-149).
The same span covers load balancing. LIT-188 introduces an auxiliary loss to keep experts evenly used, because the gate self-reinforces; seven years later SOTA-148 recommends taking that loss back out and using a bias instead. The record holds both ends of that argument.