Arora 2012 - Experiments and Evaluation Metrics
Summary
Compared methods: Recover (original), RecoverKL, RecoverL2 and a fast Gibbs sampler (MALLET). On semi-synthetic NY Times and NIPS corpora the new algorithms beat Recover and approach Gibbs with more data; on real corpora held-out probability is similar, coherence is better for Recover-based topics and unique words per topic are fewer.
Main Content
Data. NY Times: 295k documents, vocabulary 15k, mean length 298. NIPS abstracts: 1,100 documents, vocabulary 2,500, mean length 68. Pruned by document frequency. Semi-synthetic corpora: train model by MCMC, generate documents (length 300 / 70, document-topic Dirichlet with symmetric hyperparameter 0.03). Corpus sizes: NYT 50,000 to 2,000,000 documents; NIPS 2,000 to 100,000; plus “infinite data” (true ). Gibbs: average of 10 saved states, 100 iterations apart, after 1,000 burn-in.
Metrics (5.1)
reconstruction error: match learned to true topics by LP (Hungarian-type assignment) then per topic. Held-out probability per token (approximated, Wallach et al. 2009). Coherence (Mimno et al. 2011): , document counts, . Unique words: for each topic take the most probable words and count those absent from every other topic’s top- set (redundancy measure).
| Section | Finding |
|---|---|
| 5.2 Efficiency (Fig. 1) | Python Recover algorithms are faster than heavily optimised Java Gibbs (Yao et al. 2009); Gibbs linear in corpus size; RecoverL2 time varies only 33 to 50 s; estimating takes 7 s for the largest corpus; FastAnchorWords under 6 s |
| 5.3 Semi-synthetic (Figs 2-3) | Recover is poor except with infinite noiseless data; Gibbs has lower for small corpora; RecoverKL/L2 improve and have lower variance with more data (comparable or better after 40,000 NIPS documents); on NIPS with topic matching fails for Recover |
| 5.4 Separability (Fig. 4) | adding a synthetic anchor to each topic drives to 0 for Recover and near 0 for KL/L2 - residual error is a separability violation (and incomplete optimisation) |
| 5.5 Correlation (Fig. 5) | logistic-normal topics with block covariance (10 groups; and ): Recover degrades most; others, especially Gibbs, are robust; error rises with correlation and falls with corpus size; infinite-data error equals the uncorrelated case |
| 5.6 Real documents (Fig. 6, Table 1) | Held-out sets: 230 NIPS documents (average over 5 splits), 59k NYT. Recover fails on NIPS (matrix inversion); on NYT Recover has noticeably worse held-out log-probability. Gibbs best on average held-out probability (, paired t-test) but within between-document variability; Gibbs lower coherence, more unique words |
Table 1 (closest/median/farthest topic pairs, NYT, RecoverL2 vs Gibbs): e.g. baseball topic “run inning game hit season anaheim angel” vs “run inning hit game ball pitch”; Elian Gonzalez topic; internet topic “file sport read internet email” vs “web site com www mail”. Recover-based topics tend to have more specific words (Anaheim Angels vs pitch). Appendix C reproduces 100 NYT topics (Tables 2-4) and the full set of semi-synthetic plots (Figs 9-13; for NIPS with Recover yields log probabilities for some held-out documents).
Conclusions (Sec. 6). Runtime is effectively independent of corpus size; sample complexity is somewhat greater than MCMC but RecoverL2 gives comparable results in a fraction of the time; a hybrid (initialising MCMC from the output) had not yet been found to help.
Examples
Coherence by hand
Two words co-occur in 40 of 100 documents; the second appears in 50 documents: contribution . A pair that never co-occurs but with the second word in 50 documents: . The sum penalises topics mixing unrelated words.
Connections
Yardsticks and stability checks reused for newer topic models: Concept - Topic Model Evaluation and Stability. The comparison point (LDA with Gibbs): Blei 2003 - Smoothed LDA, Examples and Experiments.