Source
Kim et al. (2025), LIT-441 — ARXIV-2509.14786.
The claim, stated so the wrong version cannot be read into it
Ensembles beating single models is old and uninteresting when the comparison
is against one member. The claim here is against the same total parameter
count: K models of size N against one model of size K·N, under a
fixed corpus, with both recipes properly regularized first.
The recommendation originates in LIT-441. On a fixed 200M-token corpus it
trained K models identical except for seed, averaged their logits, fitted a
power law in K at four member sizes, and set the resulting asymptote against
that of its own weight-decay-tuned parameter-scaling recipe; distilling the
ensemble back into one model is its proposal for paying the inference cost.
Two 300M models beat one 600M. In the limit, the ensembling asymptote (≈3.34) is below the parameter-scaling asymptote (≈3.43), and even a 4-member ensemble beats what parameter scaling reaches at infinity. The two compose: taking members to infinity and then member size to infinity gives ≈3.13.
Members differ only in random seed — data order and initialization. Either source of randomness alone gets most of the benefit.
Members should be tuned for the ensemble, not as models
This is the part most likely to be skipped and it is worth ~0.17 loss. The
hyperparameters that are best for a single model are not the ones that give
the best many-member asymptote: the ensemble wants more epochs and less
weight decay per member — each member individually more overfit. Tuning
against the limit rather than against K = 1 moved the asymptote from 3.34
to 3.17.
The offered explanation is Allen-Zhu and Li’s multi-view account: the data admits several sufficient feature sets, one model is biased toward learning one of them, and independently seeded members happen to pick up different ones. A more overfit member commits harder to its view.
Distillation is what makes it deployable
An ensemble is K times the inference cost, which would normally end the
discussion. Distilling an 8-member ensemble into a single 300M student
retains about 83% of the gain and still beats the parameter-scaling
asymptote outright. Self-distillation — a 300M teacher into a 300M student of
identical architecture — also improves on its teacher, which removes the
large model from training as well as from serving.
The distillation leg has a second, older measurement from a different method. Hinton et al. (LIT-680) distil a 10-member speech ensemble into one model the size of a member, using temperature-softened soft targets, and keep 86% of the frame-accuracy gain. Kim et al. use sequence-level distillation. Two methods, a decade apart, land at about the same fraction. That strengthens this half and says nothing about the other.
So the practice is the pair, not the ensemble alone: ensemble to extract the data efficiency, distil to pay for it once.
What it is competing with
Every scaling decision in this record assumes the unit being scaled is one model. This says that at sufficient parameter count under a fixed corpus, that default is the wrong move — which is a claim the record has no other document to set against, because nothing in it trains an ensemble.
The aggregate effect, combined with SOTA-255’s regularization, is reported as ~5.2x the data efficiency of the standard recipe at 200M seed tokens, and the fitted data-scaling laws say the multiple is roughly constant across token counts.
Conditions, and why this is Proposed
The headline comparison is between two extrapolations. Both asymptotes
are fitted limits, and the composed figure is an asymptote of asymptotes. The
sensitivity analysis covers seed variance in the fits, not error in the
double limit. The K = 4 result stands without extrapolation and is the part
to lean on.
One group, 200M seed tokens, members up to 1.4B. Whether ensemble scaling still wins once members are individually under-parameterized for the corpus is not measured, and that crossing is exactly where frontier training sits.
The comparison is at matched total parameters, not matched wall-clock or
matched engineering effort. K independent runs plus a distillation pass is
a different operational proposition from one run, and none of that cost
appears in the accounting.
Known implementations
- None in the record.