Sennrich et al. (2015) — ARXIV-1508.07909
Key takeaways
- Translation is an open-vocabulary problem and NMT models were not, backing off to a dictionary for out-of-vocabulary words — handling them outside the model rather than in it
- The fix is to encode rare and unknown words as sequences of subword units, which makes the model itself open-vocabulary
- Byte pair encoding, adapted from compression to segmentation: start from characters, repeatedly merge the most frequent adjacent pair, stop at a chosen number of merges
- Frequent words survive as single units, rare ones fragment, and the character level is always reachable — so nothing is unrepresentable
- The merge table is fit to a corpus, which is where the permanent cost of an unrepresentative fitting distribution comes from
Standing in the anthology
The source of SOTA-007.
Read in full for #114; the reading is NOTE-010. The practice is confirmed.
One framing the reading adds: the contribution is open-vocabulary modelling, and BPE is the mechanism reached for. Stated as a recommendation for BPE, the practice makes the algorithm the point — and the field has since replaced the algorithm repeatedly while keeping the property.
The first tokenizer-free model in the record was measured against BPE directly. Hwang, Wang and Gu’s H-Net (LIT-679) uses a GPT-2 BPE Transformer as its tokenized baseline, matched in bytes per batch and FLOPs per byte at up to 1.3B-equivalent FLOPs, one run each. One stage of learned chunking only ties it (0.728 against 0.730 bits/byte at XL); two stages beat it (0.715, and 58.2 against 55.5 on seven zero-shot tasks). The largest gains are on Chinese, where a merge table fit elsewhere compresses least: 3.62 bytes per token for Llama-3’s BPE against 5.81 for learned chunking. The paper says its own bits/byte for the BPE baseline is overestimated, which favours H-Net.