Blei 2003 - Related Models and the Topic Simplex
Summary
Unigram models use one point on the word simplex; mixture of unigrams selects one of points per document; pLSI gives each training document its own mixture but is not a generative model of new documents and has parameters; LDA uses a smooth Dirichlet distribution over the topic simplex with parameters.
Main Content
| Model | Doc probability | Parameters | Limitation |
|---|---|---|---|
| Unigram (4.1) | single distribution for the whole corpus | ||
| Mixture of unigrams (4.2; Nigam et al. 2000) | mixing + | exactly one topic per document | |
| pLSI (4.3; Hofmann 1999) | (linear in ) | is a dummy index; no generative model for unseen documents; overfits | |
| LDA | Eq. 3 | fixed |
Mixture of unigrams has free mixing parameters, LDA has Dirichlet parameters, i.e. only one more.
Geometric interpretation (4.4)
All four models live on the -simplex of word distributions. The unigram model is one point. Latent-variable models choose points spanning the topic simplex. Mixture of unigrams: each document is generated from one vertex. pLSI: documents lie at an empirical set of points inside the topic simplex (one per training document, a discrete empirical distribution on the topic simplex). LDA: a smooth distribution (Dirichlet) over the whole topic simplex.
Why pLSI overfits. Assigning probability to a new document by marginalising over restricts it to topic proportions seen in training; folding-in refits parameters on the test document and is therefore an unfair advantage. In the mixture of unigrams overfitting arises from peaked posteriors (hard clustering, as in naive Bayes) producing very small word probabilities for unseen words.
Examples
Parameter counts
, , : LDA has parameters regardless of ; pLSI has .
Connections
This is the pLSI-to-LDA step that Arora 2012 - Practical Provable Topic Modeling - Overview revisits through admixtures, and the motivation for “after LDA” comparisons (E73).