Hui et al. (2024) — ARXIV-2409.12186
Key takeaways
- Continued pretraining of the general Qwen2.5 backbones on more than 5.5T code-heavy tokens, at 0.5B, 1.5B, 3B, 7B, 14B and 32B.
- Three data levers: cleaning, scalable synthetic generation, and a balanced mix of code, general text and math so that code gains do not cost general and mathematical ability.
- Fill-in-the-middle is part of pretraining, first at file level and then at repository level with dedicated special tokens; the FIM data also serves next-token prediction, which is the reading LIT-119 confirms in SOTA-128.
- State of the art across more than ten benchmarks covering generation, completion, reasoning and repair at the time of release.
Standing in the anthology
The FIM format and the reference small coder (0.5B) that LIT-119’s 90M coder is measured against. Filed for the recipe rather than the scores.
No practice of its own, and the reason is ADR-017’s. This is a model report: it describes a recipe in full and isolates none of it, so what it supplies is adoption rather than evidence about any particular claim. The FIM format it documents is sourced to LIT-124, which ran the ablations; this report is where the record can see the format actually shipped, and where LIT-119’s 90M coder gets its 0.5B reference point.