Concept - Neural and Embedding Topic Models

Summary

Concept note (no raw source). Neural topic models amortise variational inference for LDA-like models with an encoder network (NVDM, ProdLDA); embedding-based models (ETM) tie topics to word embeddings; clustering-based pipelines (Top2Vec, BERTopic) cluster document embeddings and extract keywords. They trade posterior uncertainty and interpretability guarantees for flexibility and short-text performance.

Main Content

Model	Idea	Reference
NVDM	VAE for bag-of-words: Gaussian latent document code, softmax decoder	Miao, Yu, Blunsom, ICML 2016
ProdLDA	Replace LDA’s mixture of topic-word distributions by a product of experts; amortised VI with a Laplace approximation to the Dirichlet in the softmax basis	Srivastava and Sutton, ICLR 2017
ETM	Topic-word distribution ${\beta }_k=\text{softmax}({\rho }^T{\alpha }_k)$ from word embeddings $\rho$ and topic embeddings ${\alpha }_k$	Dieng, Ruiz, Blei, TACL 2020
Top2Vec	Joint document/word embeddings, density-based clusters in reduced space	Angelov, arXiv 2020
BERTopic	Sentence-transformer embeddings, UMAP, HDBSCAN, class-based TF-IDF for topic words	Grootendorst, arXiv 2022

Amortised inference for a topic model

Replace per-document variational parameters $({\gamma }_d,{\varphi }_d)$ by an encoder $q_{\psi }({\theta }_d\mid {\mathbf{w}}_d)$ with parameters $\psi$ shared across documents; optimise the ELBO $E_q[\log p({\mathbf{w}}_d\mid {\theta }_d)]-D_{KL}(q_{\psi }\Vert p(\theta ))$ with the reparameterisation trick (Reparameterization Trick and Variational Autoencoders, Stochastic and Black-Box Variational Inference). Compared with the per-document coordinate updates of Blei 2003 - Variational Inference and Parameter Estimation, inference is a single forward pass.

Practical contrasts. (i) Non-conjugate and deep architectures are easy to add (metadata, embeddings); (ii) results vary across random seeds and training runs, so stability checks are needed (Concept - Topic Model Evaluation and Stability); (iii) a point-estimated decoder has no posterior over topics unless extended; (iv) clustering pipelines assign each document to one cluster (plus an outlier class for HDBSCAN), more like GSDMM than LDA (Concept - Short-Text Topic Models GSDMM and BTM).

Examples

Where each fits

Short noisy headlines: BTM/GSDMM or embedding clustering. Large corpora with metadata and a need for speed: amortised neural models. Small baskets, need for uncertainty and a clean generative story: LDA or hierarchical models with full Bayesian inference (Blei 2003 - LDA Beyond Text Baskets and Sessions).

Relevance to marketing measurement

Embedding-based topics can use pretrained semantic information for sparse text (creative copy, queries). Because they are estimated, not measured, treat topic labels as noisy covariates in downstream causal or response models and propagate or bootstrap that noise.

Connections

Variational Inference - Overview, Teh 2010 - Extensions and Reference Map (HDP for nonparametric choice of the number of topics).

See Also

Concept - Topic Model Evaluation and Stability