Grattafiori et al., Meta (2024) — ARXIV-2407.21783
Key takeaways
- A dense Transformer at 405B with a 128K context window — notable in a record whose recent half is entirely mixture-of-experts, and a useful control: this is what the same era looks like without sparsity.
- Native multilinguality, coding, reasoning and tool use, delivering quality comparable to the leading closed models of the time on a wide evaluation.
- Released with pre-trained and post-trained weights at 405B, plus Llama Guard 3 for input and output safety.
- Image, video and speech are integrated compositionally rather than natively — a contrast with the record’s 2026 entries (LIT-131, LIT-135), which build vision into the backbone.
- Its length is the point: the report is one of the few complete accounts of a frontier-scale dense recipe, which is why nearly every later paper in the corpus benchmarks against it or trains on top of it.
Standing in the anthology
Filed as the dense baseline the record’s other entries are implicitly measured against, and as a dependency several already have: Tulu 3 (LIT-172) post-trains Llama 3.1 bases, and Nemotron-CC (LIT-178) defines its long-horizon target by beating Llama 3.1 8B at 15T tokens.
No practice is drawn from it. That is deliberate rather than an omission: the report is a recipe in the sense of a complete account, not in the sense of a transferable recommendation with an ablation behind it, and this anthology sources practices to the paper that isolated a change.
The architecture, rather than the report, has since been measured. The long-context ablations of LIT-801 pretrain a Llama 3 architecture on Olmo 3 data for 140B tokens and extend it with the same recipe as 25 other 7–8B variants. It scores 52.4 on HELMET at 32K, among the best in the pool, against 47.5 for the Olmo 3 architecture. That holds although it carries GQA at eight KV heads, one of the four features the paper finds harmful. Adding QK norm and post-norm to it costs about 4 points. The authors read this as evidence that Llama 3’s ease of extension is architectural rather than a matter of its undisclosed data. It is also a reason why recipes tuned on Llama models may not transfer.