LIT-610: Efficient Estimation of Word Representations in Vector Space

Duplicate, retired. This paper was already filed as LIT-604, and the record requires one entry per arXiv identifier. This entry’s body is kept as written; LIT-604 is the one to cite.

Mikolov et al. (2013) — ARXIV-1301.3781

Key takeaways

• The contribution is computational, and the paper says so first. Both architectures exist to remove cost: “large improvements in accuracy at much lower computational cost, i.e. it takes less than a day to learn high quality word vectors from a 1.6 billion words data set”. The deletion that buys it is the non-linear hidden layer that previous neural language models carried.
• Two architectures, mirror images. CBOW predicts the current word from its context; Skip-gram predicts the surrounding words from the current one. Neither is presented as obviously better and the paper measures both.
• Hierarchical softmax is the other half of the saving. With the vocabulary as a Huffman binary tree, the output layer costs about log₂(V) evaluations rather than V. Frequency-ordered, so common words are shallow — the structure of the signal is used to cut the compute.
• The linear-regularity result, and the part that made it durable. vector("King") − vector("Man") + vector("Woman") lands nearest “Queen”, which the paper calls “somewhat surprising”. What matters for this record is what came next: rather than leave it as an anecdote they designed a Semantic-Syntactic Word Relationship test set to measure it across many relations. Turning a striking example into a countable quantity is DP-009 before the fact.

Standing in the anthology

Unit G of #304, ranked last by the audit and filed with the qualification intact: weak substrate. Only three record documents mention word2vec and all three use it as a benchmark task inside the µP / Tensor Programs cluster, not as a technique the record has a view on. It is filed for lineage, not because anything here leans on it.

The record has filed pre-arXiv and pre-modern substrate deliberately before — LIT-421 (LeCun et al. 1998), LIT-422 (ImageNet), LIT-424 (CIFAR), LIT-395 (Dropout) — so the precedent is established. This is the same kind of entry.

Its successor LIT-609 is the one that matters more here: it carries negative sampling, which #304 found the record could not name.