Source
Bavarian et al. (2022), LIT-124 — ARXIV-2207.14255.
A data transformation, not an architecture: cut a document into prefix, middle and suffix, move the middle to the end with sentinel tokens, and the model learns to infill from the surrounding context. Nothing about the model changes.
FIM-for-free is the claim that makes it a default. Transforming a large fraction of the training data does not harm left-to-right perplexity or sampling quality, across a wide range of scales. Infilling is therefore an added capability rather than a trade, and the paper’s own recommendation is to train autoregressive models with FIM by default.
Both the measurement and the default are LIT-124’s: Bavarian et al. ran the transformation across a range of model scales, found no loss in left-to-right perplexity or sampling quality, and proposed FIM-by-default on that basis.
The knobs it settles, and the one it does not
Ablated here:
- The transformation rate — what fraction of documents to transform.
- PSM against SPM ordering — prefix-suffix-middle or suffix-prefix-middle.
- How the span is chosen — character-level random spans rather than token or line boundaries.
- Where the transform applies — at the context level rather than the document level.
Left unstated: whether to mask the loss on the prefix and suffix, as one would mask a prompt in SFT. That is SOTA-128, which extends this and answers it with two 90M runs — compute the loss on every token.
Filed late, and the inversion is the point
The record has carried SOTA-128 — a footnote to this practice, about one
knob this paper leaves unstated — since before it carried the practice
itself. That is the same shape as the hyper-connections trunk and the
matrix-preconditioner class: the thing somebody published a refinement of got
filed, and the thing being refined did not.
Known implementations
- Qwen2.5-Coder, Falcon-H1-Tiny, and the reference recipes the FIM literature describes.