Source
Sennrich et al. (2015), LIT-003 — ARXIV-1508.07909.
Known implementations
- llama2
The problem it solved
A word-level vocabulary has to be closed, so anything outside it becomes an unknown token and the model can neither read nor produce it — names, numbers, morphology, code, any language with productive compounding. A character-level vocabulary is open and pays for it in sequence length.
The recommendation originates in LIT-003, written for neural machine translation with RNN encoder-decoders on WMT tasks. There the standard fix for out-of-vocabulary words was to back off to a dictionary outside the model. Sennrich et al. instead encoded rare and unknown words as sequences of subword units, adapting byte pair encoding from compression into segmentation. That made the model itself open-vocabulary, and it improved translation of rare words specifically.
BPE takes the middle: start from characters, repeatedly merge the most frequent adjacent pair, stop at a chosen vocabulary size. Frequent words end up as single tokens, rare ones decompose into pieces, and nothing is ever out of vocabulary because the character level is always available underneath. That property — open vocabulary at a bounded size — is why almost every model in this record tokenises this way. The exception is H-Net (LIT-679), a byte-level model that learns its own chunk boundaries. At matched data and FLOPs it ties a BPE Transformer with one stage of chunking and beats it with two, up to 1.3B-equivalent FLOPs and at twice the wall-clock (SOTA-418). That is one group at small scale. It is a controlled challenge to this default, not a replacement for it.
What the choice costs, which the title does not say
The vocabulary size is a real trade and it is set once, before training. Larger means shorter sequences and so cheaper attention, and a larger embedding and output layer whose cost is paid on every token. Smaller means the reverse, plus worse handling of anything the merges did not reach.
The merges are also fit to a corpus, so a tokeniser trained on one distribution is systematically inefficient on another — a language, a programming language or a domain under-represented at fitting time spends more tokens per unit of meaning, permanently, and no amount of later training recovers it.
That last point is the reason this is worth a body rather than a line. The tokeniser is one of the few decisions that cannot be revised without retraining, and its failure mode is a quiet tax rather than an error.