anthology-of-the-sota

LIT-600: Automatic Evaluation of Topic Coherence

2 min read

Properties6
StatusActive
Tagsanalysis-and-evaluation, representation-and-encoding
ExtendsLIT-597 — Reading Tea Leaves: How Humans Interpret Topic Models
Cited as “introduced by” bySOTA-371 — Score topic coherence automatically as the average pointwise mutual information of a topic's top-word pairs, estimated on a large external reference corpus
Cited as “paper” byNOTE-324 — Automatic Evaluation of Topic Coherence
Cited as “source” bySOTA-371 — Score topic coherence automatically as the average pointwise mutual information of a topic's top-word pairs, estimated on a large external reference corpus

Graph View

Newman, Lau, Grieser and Baldwin, NICTA, UC Irvine and Melbourne (NAACL-HLT 2010) — https://aclanthology.org/N10-1012/

Key takeaways

  • The task (§1, §5): judge a topic, shown as its top ten words, for coherence. The question given to annotators was whether you could imagine using it as a subject heading in a search interface. Topics came from LDA on 55,000 Gigaword news articles (200 topics) and 12,000 Internet Archive books (400 topics). 237 were sampled, and 9 annotators rated each on a 3-point scale
  • The candidates (§4): nine WordNet similarity measures, four Wikipedia measures (link-based MIW, category-overlap RACO, document cosine DOCSIM, and term co-occurrence PMI), and two Google measures (title matches and log hit counts). Pairwise scores over the 45 word pairs are combined by mean or median
  • PMI (§4.2): log p(wᵢ, wⱼ) / p(wᵢ)p(wⱼ), with co-occurrence counted in a sliding 10-word window over about 1 billion words of English Wikipedia
  • Results (Tables 2 and 3, Spearman ρ against the human scores):
    • News: PMI 0.78 median and 0.77 mean. Google TITLES 0.80. IAA 0.79 and 0.73
    • Books: PMI 0.74 median and 0.77 mean. Google TITLES 0.51. IAA 0.82 and 0.78
    • WordNet: from −0.31 to 0.66, with the best measure different on each corpus
  • The authors’ reading: word-pair co-occurrence in a large external corpus models topic coherence well, and Wikipedia is the most consistent resource

Standing in the anthology

Filed on request. It extends Reading Tea Leaves (LIT-597), which showed that topic quality has to be measured with people, and it automates that measurement. It sources SOTA-371.

The comparison to human agreement flatters the method. The “upper bound” is the average correlation between one annotator and the mean of the other eight. The methods are correlated with the mean over annotators, a less noisy target than any single person. So approaching or passing that number does not mean matching a human judge. The collections were also chosen to give topics of “more variable quality”, which widens the spread and inflates rank correlations. And only LDA topics were scored.

Read — NOTE-324.

Backlinks