nucleation

LIT-1414: Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

1 min read

Properties4
StatusActive — read 2026-10-29 (NOTE-1388) in full, from arXiv v1; worth reading as the method that takes load balancing out of the gradient: a per-expert bias, used only to choose the top-K experts and moved by the sign of each expert's load deviation on the previous batch. Read it knowing that the control responds to how often an expert is used, not to what it contributes, that the evidence is one run per configuration at 1B and 3B with perplexity gains of about 0.05, and that the conclusion's 300B tokens for the 3B model should be 200B.
Version2
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-1388 — Wang, Gao, Zhao, Sun and Dai — Auxiliary-loss-free load balancing for mixture-of-experts

Graph View

Wang, L., Gao, H., Zhao, C., Sun, X. and Dai, D. (2024). Auxiliary-loss-free load balancing strategy for mixture-of-experts. arXiv:2408.15664.

Read in full on 2026-10-29 (NOTE-1388).

Why this is here

The exchange cited it for CASE-117: balancing expert load by adjusting each expert’s access from its recent load, without a term in the objective, is a contribution-based control of influence, one of the regimes CASE-117 compares. The exchange gave no description to check. It is ML practice and carries the anthology-candidate flag (ADR-005).

Backlinks