anthology-of-the-sota

SOTA-418: To drop the tokenizer without losing to BPE, use a two-stage hierarchy with learned chunk boundaries and SSM encoder and decoder layers, not an isotropic byte model

2 min read

Properties5
StatusProposed
Consensusunreplicated
Tagsrepresentation-and-encoding, model-architecture
SourceLIT-679 — Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
Introduced byLIT-679 — Dynamic Chunking for End-to-End Hierarchical Sequence Modeling

Graph View

Source

Hwang, Wang and Gu (2025), LIT-679.

What to do

If you want a tokenizer-free model:

  1. Do not use an isotropic byte-level stack. At matched FLOPs, LlamaByte and MambaByte trail a BPE Transformer by about 9 points of downstream average.
  2. Go hierarchical. Put a byte encoder and decoder around a main network that runs on chunks. Make the encoder and decoder SSM (Mamba-2) layers. The source finds this better than attention in a way that increases with the Mamba share, even on BPE inputs.
  3. Learn the boundaries, with smoothing, and use two stages. One learned stage performs like whitespace chunking, which performs like BPE. The second stage is where the gain appears: 0.715 against 0.730 bits/byte at 1.3B FLOPs.

The recommendation originates with LIT-679, which introduced H-Net and its two-stage learned chunking. Its evidence is a chain of baselines that each change one ingredient, all matched in bytes per batch and FLOPs per byte on a 100B-token FineWeb-Edu subset. At the XL (1.3B) budget a GPT-2 BPE Transformer scores 0.730 bits/byte and 55.5 downstream average, whitespace chunking 0.726, one learned stage 0.728, and two stages 0.715 and 58.2. At Large, the isotropic LlamaByte scores 0.859 and 44.1 against the Transformer’s 0.756 and 53.3, which is the 9-point gap behind the first instruction.

Why

A fixed tokenizer commits to one segmentation before training, and it compresses unevenly across languages and domains. Learned chunking lets the model choose its units: whitespace-like at the first stage, multi-word units at the second. The gain is largest where BPE’s segmentation fits worst, which is Chinese in the source.

Conditions

  • Up to 1.3B-equivalent FLOPs, one seed per configuration. The source says it “remains to validate H-Net at larger model sizes”.
  • About 2× slower training in wall-clock at matched FLOPs, and more parameters at matched FLOPs.
  • The English baseline is GPT-2’s tokenizer. A modern BPE vocabulary might close some of the gap. On code the learned chunks only tie whitespace chunking.
  • Stability measures are part of the recipe: norms at sub-network ends and per-stage learning-rate multipliers. Without the smoothing module the compression ratio “fluctuate[s] severely”.

Known implementations

  • H-Net, the source’s code and checkpoints.

Backlinks