Anil et al. (2023) — ARXIV-2305.10403
Key takeaways
- Independently re-derives compute-optimal scaling at larger compute than Chinchilla and reaches the same answer: data and parameters should grow “roughly 1:1”, against the previous trend of scaling the model 3× faster than the dataset. Independent replication at greater scale is the strongest evidence a scaling claim can get
- States the inference-economics corollary most papers leave implicit: “it is more beneficial to invest more compute in training a smaller model compared to modifying a model’s architecture to be more inference-efficient… generally more efficient to train a smaller model with more tokens, for a fixed inference and training budget”
- Also checks the scaling fits against downstream metrics, not only loss — the half nobody replicates, and the half that matters for capabilities with thresholds
- Data mixture is ranked above architecture: translation pairs were a minor part of the mixture and were enough to put the model on par with production translation services
- Memorization measured with designed canaries — an interleave canary (two pre-training documents interleaved in batches of 50 tokens, preserving some linguistic structure) and a shuffle canary — deliberately between random outliers and unmodified text
Standing in the anthology
Read — NOTE-039. Retagged from model-architecture, which is the one
thing this report deliberately does not disclose.
Nothing is sourced to this paper and the reading files no practice, because the record already sources 1:1 allocation to Chinchilla, which is correct — this is the confirmation, not the origin.
The reading completes a chain. The record held the corrected answer without
holding either the superseded one it replaced (LIT-040) or the independent
confirmation that settled it. A reader asking how confident should I be in
1:1? now has all three, and can see the answer changed once and then held
under replication at greater compute.
The loose end is the inference-economics claim: it is live, it bears directly
on SOTA-160, and neither the report nor the record shows the comparison
behind it. Sizes, token counts and mixture proportions are all withheld —
nothing here is reproducible.