Source
Pearce and Song (2024), LIT-688, explaining the gap between LIT-028 and LIT-068.
What to do
- Define N so that
6NDmatches the FLOPs training actually spends. That includes the output head, and the input embedding too if you follow this source. Porian et al. count the head and exclude the input embedding. State which you did. - Fit loss against compute as
L = (C/C₀)^{−γ} + E, with the offsetE, not as a pure power law. - Do not set your exponent against another study’s until both count N the same way.
Why
The optimal non-embedding size against non-embedding compute “is not a power
law”. Its local slope moves from β/(α/3+β) to β/(α+β) as embeddings stop
being a large fraction of the model. A study fit at small scale on the
non-embedding count reads off the small-scale slope and calls it the
exponent. Relabelling the same five runs changes the exponent from 0.49 to
0.74.
Conditions
- Matters most below a few hundred million parameters. At frontier scale the head is a small share and the definitions converge.
- The source’s evidence is small. It is five runs at 0.8–4.6M with context 16, plus a simulation from Chinchilla’s own fit. Porian et al. (LIT-690) is the large test. Counting the head’s FLOPs takes 0.129 off the exponent, and warmup and per-size tuning each take about as much (SOTA-421). This practice is the counting part only.
- Counting is not the whole Kaplan–Chinchilla gap. Do not read this as “Kaplan was wrong because of embeddings”.
Known implementations
- None recorded.