anthology-of-the-sota

LIT-203: Fewer Truncations Improve Language Modeling

2 min read

Properties4
StatusActive
Tagsdata-pipeline
Cited as “introduced by” bySOTA-152 — Pack training documents by best fit instead of concatenating and splitting
Cited as “source” bySOTA-152 — Pack training documents by best fit instead of concatenating and splitting

Graph View

Ding et al. (2024) — ARXIV-2404.10830

Key takeaways

  • The default nobody questions. Concatenate documents, split into equal lengths, insert a boundary token. It is universal because it wastes no tokens on padding. The cost is that documents which would have fit are fragmented anyway, and the fragments are trained on as if they were whole — the model learns to continue text whose beginning it never saw.
  • The claim is not just “packing is untidy”. The paper argues analytically, through a simplified model, that truncation harms learning, and then measures it. Truncation removes the grounding a fact depends on, which is why the effect shows up most strongly as hallucination.
  • The method is bin packing, honestly named. Segment documents longer than the sequence length into chunks, then partition the chunks into the fewest sequences possible. That is the NP-hard bin-packing problem, so the paper uses a Best-Fit-Decreasing approximation — hence Best-fit Packing. Because it minimises the number of sequences, it adds no padding: the same training efficiency as concatenation, with the truncations removed.
  • The evidence is broad. Models from 7B to 13B, sequence lengths 2k to 8k, on both natural-language and programming-language corpora, evaluated on 22 tasks across reading comprehension, NLI, context following, summarization, world knowledge and program synthesis. Relative gains of +4.7%, +16.8% and +9.2% on the three headline categories, and closed-domain hallucination down by up to 58.3%.

Standing in the anthology

The source of SOTA-152, and one of the multiply-cited absences the reference pass in #40 found — cited independently by Olmo 3 (LIT-130) and DeepSeek-V4 (LIT-139).

Worth having because of the shape of the finding rather than its size. This is a data-plumbing decision, made once in a dataloader, that nobody revisits because it looks like an implementation detail with an obviously optimal answer. The record’s other data practices are about what to put in the corpus; this is about what the corpus becomes on the way to the model.

Backlinks