A mixture-of-experts needs its experts used evenly: an unbalanced load either collapses routing onto a few experts or wastes the capacity of the rest. The standard remedy is an auxiliary loss that penalises imbalance, added to the training objective with a coefficient.
That coefficient is a trade, and it is the problem. Large enough to balance the load, the auxiliary term injects interference gradients — updates that serve balance rather than the objective you actually care about. Balance and quality get traded against each other through a number nobody can set correctly.
Loss-Free Balancing takes the balancing out of the gradient entirely. Before the top-K decision, add a per-expert bias to the routing scores, and update each bias from that expert’s recent load: overloaded experts get their bias lowered, idle ones raised. Routing changes. The loss does not.
Because no interference gradient exists, this is not a better point on the balance/quality trade — it removes the trade. LIT-171 reports better performance and better balance than an auxiliary-loss control at 3B parameters and 200B tokens, and LIT-160 is DeepSeek-V3 running it at 671B.
What this presumes, and does not argue
It presumes you are training a mixture-of-experts. It is not a recommendation to train one. When this was filed the record had no practice saying “use a mixture of experts” (the question was open as #17). That practice has since been filed, and this one extends SOTA-150, which recommends making the feed-forward layers a sparse mixture of experts once the model is compute-bound. This is the rule that only exists once that one is followed. The auxiliary balancing loss it replaces is the one that came with the sparsely-gated layer at the start of that practice’s line.
That separation is what makes this fileable on its own. LIT-171’s own standing section worried the opposite way — that a load-balancing practice “would arrive without the context that makes it meaningful” because the record carries no MoE practices at all. The resolution is that architecture choice and technique are different kinds of claim. Whether to build a sparse model is an editorial judgement about where the field is going; how to keep its experts evenly loaded once you have is a technique with a mechanism and an ablation, and it is answerable without the other question being settled.
Variations, and one qualification from the originating lab
Kimi K3 (LIT-131) keeps the recommendation and replaces the update rule. At 896 routed experts per layer it reports that balancing “exceeds the regime in which existing auxiliary-loss-free bias updates remain well behaved”: the fixed-step rule’s step size trades slow adaptation against load oscillation, and neither end of that trade is acceptable at ~900 experts. Quantile Balancing sets each expert’s bias directly from the router-score quantile matching its target load, read from a single forward pass — Top-(k+1) selection makes the (k+1)-th entry the cutoff an expert must beat — and estimated at scale from a per-expert histogram of margins, one all-reduce of bin counts. The parts this practice turns on are kept: the bias is excluded from the mixture weights, so no gradient flows through it, and the update lands only on the next step, so no batch is routed with a bias derived from itself.
So the bias on the routing score is what generalised; the fixed step is what did not, and the report says at roughly what size it stopped.
DeepSeek-V4 (LIT-139) runs a small auxiliary loss alongside the bias. Bias update speed 0.001, “augmented by a slight sequence-wise balance loss that prevents extreme imbalance within individual sequences” at weight 0.0001. That is the originating lab qualifying its own result three models on, and it is worth being exact about what it concedes. The bias regulates load across a batch; nothing in the mechanism reaches imbalance inside one sequence, which is the failure the 0.0001 loss is aimed at. The claim this practice makes — that you do not need an auxiliary loss to balance experts — survives. The stronger reading, that an auxiliary loss has no remaining job at all, does not.
Why Proposed, and why emerging now
The head-to-head evidence is still one lab’s: LIT-171’s authors are DeepSeek-AI and LIT-160 is DeepSeek’s own model. What changed is that “nobody outside has published a result either way” stopped being true. K3 is an outside group that adopted the method at four times the scale, hit a limit in it, and published the fix — stronger evidence than a silent adoption, weaker than a controlled comparison.
It still does not meet the promotion condition, and deliberately so: that condition asks for balance and quality against an auxiliary-loss control, and K3 runs no such control. The condition anticipated something like this — “a model that merely ships loss-free balancing without running that comparison is not the missing evidence” — and K3 does more than ship while still not being what was asked for.
The record’s unreplicated practices are all Proposed
(SOTA-122, SOTA-124,
SOTA-125, SOTA-144). This one has moved off
that shelf without becoming settled.
The MoE material now in the record
Three practices’ worth, and this is the first of them filed:
- LIT-170 — fine-grained plus shared experts, the architecture itself. Not filed; that is the open question.
- LIT-171 — this.
- LIT-180 — the RL objective is unstable on mixture-of-experts specifically.
That last one is already in the registry: SOTA-146 carries the instability as a caveat on an architecture the record has never recommended.