Bavarian et al. (2022) — ARXIV-2207.14255
Key takeaways
- A data transformation, not an architecture: cut a document into prefix, middle and suffix, move the middle to the end with sentinel tokens, and the model learns to infill from the surrounding context.
- FIM-for-free: transforming a large fraction of the training data does not harm left-to-right perplexity or sampling quality across a wide range of scales, so infilling is an added capability rather than a trade.
- Ablations over the knobs that matter: the transformation rate, PSM (prefix-suffix-middle) against SPM ordering, and how the span is chosen — character-level random spans rather than token or line boundaries — plus applying the transform at the context level rather than the document level.
- The recommendation is to train autoregressive models with FIM by default.
Standing in the anthology
The origin of every FIM recipe the corpus touches. SOTA-128 is a footnote to it: whether to mask the loss on the prefix and suffix is one knob this paper leaves unstated, and LIT-119 measures it.