nucleation

LIT-1412: Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity

1 min read

Properties4
StatusActive — read 2026-10-29 (NOTE-1363) in full, from arXiv v3 (the JMLR version); worth reading as the paper that made top-1 expert routing with a fixed capacity and the auxiliary balancing loss α·N·Σf_iP_i the standard for sparse Transformers, with the stability fixes that let it scale to 1.6 trillion parameters at T5's FLOPs per token. Read it knowing that nearly every comparison is a single run, that it never measures what experts specialize in, that its balancing loss is not minimized only by uniform routing as §2.2 says, and that one quoted margin over T5-XXL (Switch-C's) is 0.051, not "over 0.061".
Version2
Tagslearning-theory, anthology-candidate
Cited as “paper” byNOTE-1363 — Fedus, Zoph and Shazeer — Switch Transformers

Graph View

Fedus, W., Zoph, B. and Shazeer, N. (2022). Switch Transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(120), 1–39. arXiv:2101.03961.

Read in full on 2026-10-29 (NOTE-1363).

Why this is here

The exchange cited it for CASE-117 as the standard case of routing among experts with an explicit load-balancing term, the imposed alternative to diversity that emerges. The exchange gave no description to check. The auxiliary loss is standardly attributed to the paper; the abstract does not name it. It is ML practice and carries the anthology-candidate flag (ADR-005).

Backlinks