Nonparametric Models Overview

Summary

Part V of BDA3 (Chapters 19-23) covers Bayesian approaches to flexible, nonparametric modeling — from splines and Gaussian processes to mixture models and Dirichlet processes. These allow the data to determine the functional form rather than imposing rigid parametric assumptions.

Parametric Nonlinear Models (Ch 19)

• Nonlinear regression models (e.g., pharmacokinetics, dose-response)
• Hierarchical structure for population-level inference about nonlinear parameters

Basis Function Models (Ch 20)

• Splines: piecewise polynomial fits with knots — smooth and flexible
• Basis selection and shrinkage: regularization over basis coefficients prevents overfitting
• Bayesian approach: prior on coefficients provides automatic smoothing

Gaussian Process Models (Ch 21)

A Gaussian process defines a distribution over functions:

$$f\sim \mathcal{G}\mathcal{P}(m(x),k(x,x^{\prime }))$$

• Kernel function $k(x,x^{\prime })$ encodes assumptions about smoothness and correlation
• Naturally provides uncertainty bands over the entire function
• Computationally $O(n^3)$ — challenging for large datasets

Finite Mixture Models (Ch 22)

$$p(y\mid \theta )=\sum _{k=1}^K{\lambda }_{k}f(y\mid {\varphi }_k)$$

• Label switching: posterior is invariant to permutation of component labels — requires care in interpretation
• Applications: clustering, density estimation, robust regression
• Hierarchical priors on $K$ or component parameters

Dirichlet Process Models (Ch 23)

The Dirichlet process extends finite mixtures to an infinite number of components:

$$G\sim \text{DP}(\alpha ,G_0)$$

• Concentration parameter $\alpha$ controls the number of clusters
• DP mixtures: nonparametric density estimation with automatic complexity selection
• Hierarchical DP: share clusters across groups

See Also

• Bayesian Linear Regression — the parametric starting point
• Model Comparison — comparing parametric vs. nonparametric fits
• Efficient MCMC — computation for these complex models
• Hierarchical Linear Models — hierarchical regression as the parametric precursor before moving to GP/nonparametric models