Source
Falcon-LLM Team (2026), LIT-119 — the Falcon-H1-Tiny technical blogpost.
The comparison
This is a knob on a practice already taken: it only arises once fill-in-the-middle is part of the training mix by default SOTA-174, and it settles the one question that practice’s paper leaves open. Whether to mask the loss on the prefix and suffix of a FIM sample, as one would mask a prompt in SFT, is left unstated in ARXIV-2207.14255 (LIT-124) and only implicit in ARXIV-2409.12186 (LIT-125). Two 90M runs on the same mix (80% FIM, 10% code, 10% web and math), 80 GT with 20 GT of decay, one masking the non-middle tokens and one not: the unmasked run was clearly better across the coding benchmarks. The authors note the unmasked run simply trains on more tokens for the same budget, and conclude that for a fixed token and compute budget the unmasked data is the more efficient use of it — FIM samples serve both the infilling objective and next-token prediction.
That pair of runs is LIT-119’s, from the Falcon-H1-Tiny-Coder work, and it is where the recommendation is first stated outright; the earlier papers leave it to be inferred. The comparison is reported as a plot and a verdict (“clearly better”), not as per-benchmark numbers, and it is one run per arm.
Two details that travel with it
From the same work:
use PSM format with dedicated <|prefix|>, <|suffix|>, <|middle|>
tokens; and construct samples in which the model must produce the
indentation after the prefix itself, since HumanEval-FIM prompts do not
include it and a model that never had to predict indentation starts a new
function instead.
Known implementations
- Falcon-H1-Tiny-Coder-90M