Borgeaud et al. (2021) — ARXIV-2112.04426
Key takeaways
- Comparable to GPT-3 and Jurassic-1 on the Pile with 25× fewer parameters, by cross-attending to chunks retrieved from a 2-trillion-token database
- The retriever is a frozen, pretrained BERT — no retriever training, no re-indexing as the model changes. The paper’s claim is that this works at scale, removing the moving part retrieval systems usually break on. Chunked cross-attention keeps cost linear in the amount retrieved
- The gain is constant from 150M to 7B — retrieval is not a small-model crutch that scale absorbs
- Capability increases after training ends: the model “can be improved at evaluation time by increasing the database size and the number of retrieved neighbours”. No gradient step
- Proposes an evaluation aware of test/train document proximity, noting it is “relevant for all language models, and especially for retrieval-enhanced models since they have direct access to the training dataset during evaluation”
Standing in the anthology
Read — NOTE-060. Nothing is sourced here; retrieval-augmented pretraining is not a line the anthology tracks.
The finding worth recording is an allocation one. Every allocation practice in
the record — the Chinchilla line, SOTA-160’s joint token-and-quantization
decision — trades parameters against tokens seen during training, and assumes
capability is fixed when training ends. This trades parameters against tokens
available at inference, and its evaluation-time improvement is a direct
counterexample to the assumption. The record has no place to put that.
Two of the takeaways replaced were wrong. “Efficient similarity search” is not a contribution — the retriever is an off-the-shelf frozen BERT, and the point is that none had to be built. “Memory-efficient architecture” is backwards: the 25× parameter reduction is bought with a two-trillion-token database, whose cost the paper does not price.