Source
Newman et al. (2010), LIT-600. Read as NOTE-324.
The practice
- Use an external corpus, not the training corpus. The point is to ask whether people use these words together, so count co-occurrence somewhere broad and independent of the model’s data. Wikipedia worked on both news and books topics
- Score pairs in a sliding window. PMI(wᵢ, wⱼ) = log p(wᵢ, wⱼ) / p(wᵢ)p(wⱼ), with co-occurrence in a 10-word window, averaged over the 45 pairs of the top ten words
- Prefer it to knowledge-base and search-engine scores. WordNet measures were erratic, and Google collapsed on the books corpus
- Treat it as a proxy for the human task (SOTA-368), not a replacement. Validate it against people on your own models before relying on it to choose between them
Evidence
The measure and its validation both come from LIT-600. Newman et al. had nine annotators rate 237 LDA topics from Gigaword news and Internet Archive books, then scored the same topics with WordNet, Wikipedia and Google measures. Mean PMI over Wikipedia in 10-word windows reached Spearman ρ = 0.77 against the human scores on both corpora, while the WordNet measures ranged from −0.31 to 0.66 and Google title matches fell from 0.80 on news to 0.51 on books.
Conditions
- Validated on LDA topics only
- The agreement ceiling in the source is not like-for-like, so “as good as a human annotator” overstates it. What is shown is a strong correlation with the group mean