anthology-of-the-sota

SOTA-368: When a model's latent space is meant to be read by people, measure its interpretability with a human task such as word or topic intrusion, not by held-out likelihood

2 min read

Properties5
StatusActive
Consensusunreplicated
Tagsanalysis-and-evaluation, representation-and-encoding
SourceLIT-597 — Reading Tea Leaves: How Humans Interpret Topic Models
Introduced byLIT-597 — Reading Tea Leaves: How Humans Interpret Topic Models

Graph View

Source

Chang et al. (2009), LIT-597. Read as NOTE-323.

The practice

  • Decide what the model is for. If the product is a prediction, likelihood measures it. If the product is a latent space people will browse, label or explore, likelihood does not measure that, and choosing models by it can select the less interpretable one
  • Word intrusion for components: show a component’s top words plus one word that is improbable in it and probable elsewhere, and score the rate at which people find the planted word
  • Topic intrusion for assignments: show an item with its top components and one improbable component, and score how often people reject the planted one
  • Report both beside the likelihood, per model and per number of components

Both tasks come from LIT-597, which introduced them and ran them with Mechanical Turk subjects on pLSI, LDA and CTM at 50, 100 and 150 topics over NYT and Wikipedia. CTM had the best held-out likelihood and the worst word- and topic-intrusion scores, while LDA usually did best with people.

Conditions

  • Shown for topic models of text. The argument, that a predictive score ignores the representation, applies to any latent space meant for people. The evidence is for topics
  • The strong form is not established. “Better likelihood means worse topics” rests on one model, CTM. “Likelihood does not tell you” is what the data supports

Backlinks