Teh 2010 - DP Mixture Models and Applications

Summary

The DP mixture $x_i\mid {\theta }_i\sim F({\theta }_i)$, ${\theta }_i\mid G\sim G$, $G\sim \text{DP}(\alpha ,H)$ is an infinite mixture model: the number of used components is inferred, grows like $\alpha \log n$, and the DP also serves for model validation (a nonparametric relaxation around a parametric base) and density estimation (kernel-smoothed DP draws).

Overview

Teh gives three applications: (1) Bayesian model validation, (2) density estimation, (3) clustering via DP mixtures. The mixture representation of Eq. 15 and its stick-breaking form (Eq. 16) are the core of what practitioners fit.

Main Content

Finite model whose limit is the DP mixture (Teh Eq. 1)

$$\pi \mid \alpha \sim \text{Dir}(\frac{\alpha }{K},\dots ,\frac{\alpha }{K}),{\theta }_k^{\ast }\mid H\sim H,z_i\mid \pi \sim \text{Mult}(\pi ),x_i\mid z_i\sim F({\theta }_{z_i}^{\ast }).$$

For large $K$ the number of components used by $n$ items is approximately $O(\alpha \log n)$ independent of $K$, so the model is well defined as $K\to \infty$ (an “infinite mixture”).

DP mixture model (Teh Eqs. 15-16)

$$x_i\mid {\theta }_i\sim F({\theta }_i),{\theta }_i\mid G\sim G,G\mid \alpha ,H\sim \text{DP}(\alpha ,H);$$

equivalently $\pi \sim \text{GEM}(\alpha )$, ${\theta }_k^{\ast }\sim H$, $z_i\mid \pi \sim \text{Mult}(\pi )$, $x_i\mid z_i,{\theta }^{\ast }\sim F({\theta }_{z_i}^{\ast })$, with $G={\sum }_k{\pi }_k{\delta }_{{\theta }_k^{\ast }}$ and ${\theta }_i={\theta }_{z_i}^{\ast }$.

Applications.

• Model validation: use the parametric model as the DP base measure $H$; if the parametric model fits as well as the DP-relaxed one, there is evidence for validity. The DP supplies a broad comparison class.
• Density estimation: $p(x)=\int f(x\mid \theta )G(\theta )d\theta$ (Eq. 14) smooths the discrete $G$; this is exactly the DP mixture when $F$ has density $f$.
• Clustering: the posterior over $z$ (or the CRP partition) gives clusters; the number of clusters is a posterior quantity. This replaces model averaging over $K$ in a finite mixture.

Inference. Teh points to Neal (2000) for MCMC (collapsed Gibbs over the CRP, Algorithms 1-8), and mentions split-merge and auxiliary-variable samplers, sequential Monte Carlo, expectation propagation and variational methods. In PyMC practice, collapsed samplers are not available and NUTS operates on a truncated stick-breaking model (see below).

Examples

Practical checklist for a truncated DP mixture (the E26 / E75 workflow)

1. Choose truncation $K$; check posterior leftover mass ${\prod }_k(1-{\beta }_k)$ and whether the last components are used (a binding truncation means increase $K$).
2. Prior on $\alpha$ (Gamma): the implied prior on the number of clusters is $\alpha (\psi (\alpha +n)-\psi (\alpha ))$; simulate it before fitting.
3. Do not interpret component labels; they are exchangeable under the prior. Use co-clustering matrices $C_{ij}=P(z_i=z_j\mid \text{data})$ and a point partition (Binder or VI loss) - see Concept - Label Switching and Label-Invariant Summaries.
4. Sensitivity: refit with $\alpha$ priors shifted by an order of magnitude and compare posterior $K$ and density estimates. The number of clusters is the least robust output.

Relevance to marketing measurement

Customer heterogeneity (basket values, response elasticities) is often multimodal; DP mixture random effects are a flexible alternative to a Gaussian prior in a hierarchical model (Hierarchical Models). They are not a substitute for identification: clusters are descriptive and the inferred $K$ should not be read as a causal number of segments.

Connections

Vault: Monsters and Mixtures (finite-mixture pathologies), Nonparametric Models Overview, MCMC Basics, Posterior Predictive Checking (PPC for density fit). Generalised priors on $K$ (flatter than the DP): De Blasi 2015 - Mixture Models and the Prior on the Number of Clusters.

See Also

Teh 2010 - Stick-Breaking Construction, Teh 2010 - Extensions and Reference Map