Blei 2003 - LDA Beyond Text Baskets and Sessions

Summary

Replace words by items: shopping baskets, web sessions, headline tests or colour bins. The paper’s own EachMovie experiment (Sec. 7.3) is the non-text precedent. In a probabilistic program, topics per token are summed out so that only ${\theta }_d$ and ${\varphi }_k$ are sampled; label switching and sparsity then drive the diagnostics (Hungarian alignment, label-free summaries).

Main Content

Mapping (the paper’s remark in Sec. 2, “not tied to text”). Document = unit with a bag of discrete observations: a customer (items purchased), a web session (pages visited), a headline test, a poster (colour bins). Topic = a mixture over items/pages/colours. For marketing: customers as documents and products as words.

Collapsed likelihood. From the continuous-mixture view: for document $d$ with item counts $n_{dv}$, $p(n_d\mid {\theta }_d,\varphi )={\prod }_v{p}_{dv}^{n_{dv}}$ with $p_d={\theta }_d^T\varphi$ (multinomial). This removes the $N_d$ discrete variables $z_{dn}$ and leaves a continuous model that gradient-based samplers can handle.

PyMC-style sketch (untested; PyMC 5)

import pymc as pm, numpy as np
# counts: (D, V) array of item counts; K topics
with pm.Model() as lda:
    theta = pm.Dirichlet("theta", a=np.full(K, 0.1), shape=(D, K))
    phi   = pm.Dirichlet("phi",   a=np.full(V, 0.1), shape=(K, V))
    p = pm.math.dot(theta, phi)                 # (D, V), z summed out
    pm.Multinomial("n", n=counts.sum(1), p=p, observed=counts)

Caveats: multimodal posterior, label switching across chains, funnel-like geometry for small Dirichlet concentrations; diagnose with label-free summaries and expect NUTS to struggle on sparse, high-dimensional simplices.

Label switching and alignment. Topic labels are exchangeable under the prior, so a posterior sample’s topic index $k$ is meaningless across draws/chains. To compare estimated topics $\hat{\varphi }$ to truth (known-truth check) or across restarts, match topics by minimum cost assignment (Hungarian algorithm) on L1 distance and then compare. Label-free summaries: ${\theta }_d^T\varphi$ (predicted item distribution), co-assignment of items/documents, topic coherence.

import numpy as np
from scipy.optimize import linear_sum_assignment
rng = np.random.default_rng(0)
K, V = 3, 6
true = rng.dirichlet(np.ones(V) * .3, K)
est = np.abs(true[[2, 0, 1]] + rng.normal(0, .001, (K, V))); est /= est.sum(1, keepdims=True)
cost = np.abs(true[:, None, :] - est[None, :, :]).sum(2)     # L1 distances
r, c = linear_sum_assignment(cost)
print(c, cost[r, c].round(3))        # [1 2 0] matches the permutation; L1 ~ 0.005

Examples

Baskets (UCI Online Retail, E72 setting)

Customers $\times$ products matrix with $\approx 3,661\times 2,087$ entries (counts): a topic is a product bundle; ${\theta }_d$ is the customer’s mix of bundles. Held-out evaluation: hold out purchases within customers and compare ${\theta }_d^T\varphi$ log-predictive; this is the collaborative-filtering protocol of Sec. 7.3 (hold out one item, predict it).

Relevance to marketing measurement

Topic proportions can be used as customer descriptors in downstream models (Customer Lifetime Value - Overview, BG-NBD Model covariates). They are noisy estimates: propagate posterior uncertainty rather than plugging in point values, and avoid reading topics as causal segments.

Connections

Concept - Topic Model Evaluation and Stability, Concept - Short-Text Topic Models GSDMM and BTM (when documents have only a few tokens), Monsters and Mixtures.

See Also

Blei 2003 - Latent Dirichlet Allocation - Overview