Chang, Boyd-Graber, Gerrish, Wang and Blei, Princeton, Maryland and Facebook (NIPS 2009) — https://papers.nips.cc/paper_files/paper/2009/hash/f92586a25bb3145facd64ab20fd554ff-Abstract.html
Key takeaways
- The gap it names (§1–2): topic models are presented and used as if their topics were meaningful, for browsing, corpus exploration and model checking. But they are evaluated by held-out likelihood or external tasks, and nobody had measured the meaning
- Word intrusion (§3.1): show the five most probable words of a topic plus an intruder, a word improbable in this topic but probable in another. Model precision is the fraction of subjects who pick the planted intruder
- Topic intrusion (§3.2): show a document’s title and first sentences, its three most probable topics, and one improbable topic. Topic log odds compares the model’s probability for the planted intruder with its probability for the one subjects picked
- Setup (§4.1–4.3): 8,447 NYT articles and 10,000 Wikipedia articles, with proper nouns removed so subjects did not need encyclopedic knowledge. pLSI, LDA and CTM at K = 50, 100 and 150, with 20% held out. Each task was done by 8 Mechanical Turk workers
- Likelihood (Table 1): CTM is generally best, then LDA, then pLSI. The gaps are small, for example NYT at 50 topics: LDA −7.3214, CTM −7.3335, pLSI −7.3384
- Humans (Figures 3 and 6): on word intrusion LDA usually does best and CTM worst, and pLSI degrades as K grows. On topic intrusion, LDA and pLSI beat CTM. Topic log odds are nearly flat across K for LDA and pLSI
- Where people and model disagree (Figure 4): agreement is highest on documents about one unambiguous concept (“Lindy Hop”) and lowest on documents that span disparate areas (“Book”)
Standing in the anthology
Filed on request, after Latent Dirichlet Allocation (LIT-592). It is
compared_against that paper, whose model it evaluates, and it sources
SOTA-368. It also answers the gap NOTE-322 recorded: whether low
perplexity means topics a person would recognize.
The claim that travels is stronger than the evidence. The discussion says likelihood-based metrics “are, indeed, negatively correlated” with the human measures. What the figures show is 18 fitted models with downward regression lines, driven mostly by one model: CTM has the best likelihood and the worst interpretability. Within LDA and pLSI, topic log odds barely move across K, and no correlation coefficient is reported for the likelihood relationship. What is well supported is the weaker claim: likelihood does not track interpretability, and CTM is a clean counterexample.
Read — NOTE-323.