Hoffmann et al. (2022) — ARXIV-2203.15556
Key takeaways
- Compute-optimal scaling laws
- Data/model size relationships
- Training efficiency analysis
- Resource allocation strategies
- chinchilla scaling laws, relating model size to data/compute budget
- don’t sleep on data quality
Comparisons
The paper’s test of its own rule is a comparison with Gopher (LIT-617). Its fits predicted that for Gopher’s compute budget the optimal model is four times smaller and trained on four times more tokens, so it trained Chinchilla at 70B parameters on 1.4T tokens, with Gopher’s architecture and setup bar listed differences, against Gopher’s 280B on 300B. Chinchilla reaches 67.5% on MMLU, “greater than a 7% improvement over Gopher”, and 65.1% against 54.4% on the same BIG-bench tasks, worse on only 4 of 62.
Chen et al. (LIT-685) fit this paper’s loss form against their own “sub-scaling” law on the Pile at 20M to 7B parameters. Their law is Chinchilla’s form with each coefficient allowed to grow by up to a third as tokens per parameter rise, and their own Table 4 has the Chinchilla form with the lowest prediction error at 4B.