Mikolov, Chen, Corrado and Dean, Google (2013) — ARXIV-1301.3781
Key takeaways
- The move (§3): most of a neural language model’s cost is the non-linear hidden layer. Remove it and train only a projection and a (hierarchical-softmax) output. CBOW averages the context vectors to predict the middle word. Skip-gram uses the middle word to predict each context word in a window
- The benchmark (§4.1): analogies like big : biggest :: small : ?, answered by the nearest word to vec(biggest) − vec(big) + vec(small). 8,869 semantic and 10,675 syntactic questions. Exact match only
- Scaling (Table 2): CBOW accuracy rises with both dimension and data, with diminishing returns in either alone
- Architectures on the same 320M words at 640 dimensions (Table 3): RNNLM 9/36 (semantic/syntactic %), NNLM 23/53, CBOW 24/64, skip-gram 55/59. Skip-gram is far ahead on semantic questions
- Against published vectors (Table 4): skip-gram 300d on 783M words scores 53.3% total, while earlier public embeddings score 2–25%, trained on 37M–990M words at 50–640 dimensions
- One epoch on twice the data matches or beats three epochs (Table 5). With DistBelief, 1000-dimensional skip-gram on 6B words reaches 65.6% in 2.5 days × 125 cores (Table 6)
Standing in the anthology
Filed on request, with its follow-up LIT-603 (negative sampling, subsampling, phrases) and GloVe (LIT-602), which compares against word2vec. The two word2vec papers are the prediction-based side of the count-versus-predict question GloVe addresses.
No practice. Its instruction, a shallow model on much more data, is a 2013 trade-off that later embedding and language-model work has moved past. The durable contributions are the architectures and the analogy test, which later papers, GloVe included, report on.
The headline comparison is loose. “Large improvements in accuracy at much lower computational cost” is largely Table 4, which compares vectors trained on different corpora, sizes and dimensions. The controlled comparison is Table 3.
Read — NOTE-325.