Source
Blei, Ng and Jordan (2003), LIT-592. Read as NOTE-322.
The account
A document is not about one thing. A newswire article about a foundation’s grants to an opera and a school draws its words from arts, money and education at once (the paper’s Figure 8). Modelled as a single topic, it must be forced into one cluster, and any word that cluster rarely saw makes its probability collapse. Modelled as a mixture, each word can come from whichever topic suits it, and the document’s topic proportions are a compact description of what it is about.
LIT-592 derives the mixture rather than assuming it: words exchangeable within a document are, by de Finetti, conditionally i.i.d. given a latent parameter, which LDA takes to be the document’s topic proportions. Its test is held-out perplexity, lowest for LDA at every number of topics on C. elegans abstracts and AP newswire (Figure 9), where the uncorrected mixture of unigrams reaches 4.19 × 10¹⁰⁶ at 50 topics on AP (Table 1).
What it explains
- Why the mixture of unigrams overfits as topics are added (Table 1). Finer single-topic clusters leave more words unseen in each, and held-out perplexity explodes
- Why topic proportions work as a low-dimensional document representation. They matched full word features for classification with 0.4% of the dimensions
Where it is weak
- Confounded with the prior. LDA differs from the mixture of unigrams in two ways, several topics per document and a Dirichlet prior. The paper does smooth the baseline, but it does not isolate the first
- Perplexity only, on two small corpora, and for bag-of-words models. Human evaluation came later (LIT-597). LDA’s topics were the most interpretable of three models, but people and model agreed least on documents that span disparate subjects (“Book”), and most on documents about one concept (“Lindy Hop”). Those multi-subject documents are the case this account is meant to handle best
- A modelling claim about text, not a measurement of it. The exchangeability it starts from is, in the authors’ words, a simplifying assumption chosen for efficiency