Blei, Ng and Jordan, Berkeley and Stanford (JMLR 3, 2003) — https://jmlr.org/papers/v3/blei03a.html
Key takeaways
- The argument starts from an assumption about text (§1, §3.1). LSI, pLSI and tf-idf all treat a document as a bag of words, which in probability terms is exchangeability of its words. By de Finetti’s theorem, an exchangeable sequence is a mixture, conditionally i.i.d. given a latent parameter. So a principled bag-of-words model needs a document-level mixing distribution. The authors call exchangeability “a major simplifying assumption”, justified mainly because it makes methods efficient
- The model (§3): for each document, draw topic proportions θ ~ Dir(α). For each word, draw a topic from θ, then the word from that topic’s distribution. Each document mixes several topics, unlike the mixture of unigrams (one topic per document). Unlike pLSI, it defines a probability for a document never seen in training, because θ is a random variable and not a per-training-document parameter
- Inference (§5): exact posteriors are intractable. A mean-field variational bound and EM fit α and the topics
- Perplexity (§7.1, Figure 9): 5,225 C. elegans abstracts and 16,333 AP newswire articles, 10% held out. LDA has the lowest perplexity at every number of topics. Uncorrected, the baselines overfit badly (Table 1, AP): the mixture of unigrams reaches 4.19 × 10¹⁰⁶ at 50 topics, and pLSI with marginalization reaches 5.04 × 10⁶. The plotted baselines are corrected, pLSI by “folding in”, which the authors say gives it “an unfair advantage” by refitting k − 1 parameters on the test document
- Classification (§7.2, Figure 10): Reuters-21578, 8,000 documents, two binary tasks (EARN, GRAIN). An SVM on 50 LDA topic proportions (a 99.6% feature reduction) matches or beats an SVM on all word features across training-set sizes
- Collaborative filtering (§7.3, Figure 11): EachMovie, users as documents and movies as words. LDA has the best predictive perplexity
Standing in the anthology
Filed on request, as a follow-up to the phrase-as-lemma work (LIT-410),
and tagged signal-structure first. The paper is a method, but it is built
on a stated claim about what text is like: its words are exchangeable, and
a document is a mixture of topics. That claim sources THEORY-086. It
also carries representation-and-encoding (topic proportions as a
low-dimensional document representation) and generative-modeling (the
classic generative model of a corpus).
No practice. Nothing in it is what a practitioner would do today in place of something else. Its lasting contribution is the model and the framing, and those are what the note and the theory hold.
Measured by people, not perplexity. This paper’s case for LDA is held-out perplexity. Chang et al. (LIT-597) fit LDA, pLSI and the correlated topic model at 50, 100 and 150 topics on NYT and Wikipedia and ask Mechanical Turk workers to spot an intruding word in a topic or an intruding topic in a document. LDA usually does best on word intrusion, and LDA and pLSI beat CTM on topic intrusion, while CTM generally has the best held-out likelihood. The ranking this paper’s metric produces is not the ranking people produce.
Read — NOTE-322.