Griffiths 2011 - Latent Feature Models and the Finite Beta-Bernoulli Prior

Summary

A latent feature model represents each object by $F=Z\odot V$, a binary pattern $Z$ times feature values $V$. The IBP is a prior on $Z$ derived from a finite model ${\pi }_k\sim \text{Beta}(\alpha /K,1)$, $z_{ik}\sim \text{Bernoulli}({\pi }_k)$ whose expected number of ones is bounded by $N\alpha$ independent of $K$.

Main Content

Latent feature models (Sec. 3). Object $i$ has latent vector $f_i$; properties $x_i$ are generated from $p(X\mid F)$. With $F=Z\otimes V$ (Hadamard product), $Z$ is a binary $N\times K$ matrix with $z_{ik}=1$ iff object $i$ has feature $k$, and $V$ holds feature values (continuous as in sparse PCA, or discrete). Dense models (PCA, cooperative vector quantisation) set all of $Z$ to 1. The prior is $p(F)=P(Z)p(V)$; focus on $P(Z)$ because it fixes the effective dimensionality. Desiderata: objects exchangeable; tractable inference; method: finite model, then $K\to \infty$.

Finite beta-Bernoulli feature model (Eq. 9)

$${\pi }_k\mid \alpha \sim \text{Beta}(\frac{\alpha }{K},1),z_{ik}\mid {\pi }_k\sim \text{Bernoulli}({\pi }_k),i\le N,k\le K,$$

independently across $i$ and $k$. The choice $r=\alpha /K,s=1$ gives $B(\alpha /K,1)=K/\alpha$ and is what makes the $K\to \infty$ limit work (relaxed to $s=\beta$ in the two-parameter IBP).

Marginal of $Z$ (Eq. 10)

Integrating out $\pi$ by beta-binomial conjugacy,

$$P(Z)=\prod _{k=1}^K\frac{\frac{\alpha }{K}\Gamma (m_k+\frac{\alpha }{K})\Gamma (N-m_k+1)}{\Gamma (N+1+\frac{\alpha }{K})},m_k=\sum _i{z}_{ik}.$$

It is exchangeable in rows (depends only on the column counts $m_k$).

Sparsity bound (Eq. 11)

$E[1^T{z}_k]=N\frac{\alpha /K}{1+\alpha /K}$, hence $E[1^{T}Z1]=\frac{N\alpha }{1+\alpha /K}\le N\alpha$ for all $K$, and the limit $K\to \infty$ is $N\alpha$.

Proof: $E[{\pi }_k]=r/(r+s)=(\alpha /K)/(1+\alpha /K)$; sum over $N$ rows and $K$ columns.

Examples

Expected ones for $N=100,\alpha =5$

$E[1^{T}Z1]\to 500$, i.e. five features per object on average, regardless of the (arbitrarily large) $K$ - the model is sparse by construction.

Connections

Compare to continuous latent factors: Factor Analysis and PPCA. The CRP-analogue limit argument: Griffiths 2011 - Infinite Mixtures and the CRP Review.

See Also

Griffiths 2011 - The Indian Buffet Process - Overview