Synthetic Likelihood Construction

Summary

The core estimator. Given summary statistics $\mathbf{s}$ of the data, assume $\mathbf{s}\sim N({\mathbf{\mu }}_{\theta },{\mathbf{\Sigma }}_{\theta })$. For any $\mathbf{\theta }$, simulate $N_r$ replicate data sets, convert each to a statistics vector, and estimate ${\hat{\mathbf{\mu }}}_{\theta }$ and ${\hat{\mathbf{\Sigma }}}_{\theta }$. The log synthetic likelihood $l_s(\mathbf{\theta })$ is then the MVN log-density of the observed $\mathbf{s}$ under those estimates. $l_s$ is much smoother in $\mathbf{\theta }$ than the true density, invariant to reparameterization, robust to uninformative statistics, and behaves like a genuine likelihood as $N_r\to \infty$ — so it is explored by Metropolis–Hastings MCMC, with MLE recovered by quadratic regression and model comparison via AIC/GLRT.

Overview

This note states the synthetic-likelihood algorithm and its statistical properties — the machinery that turns the phase-insensitive statistics into well-founded inference. It is the computational heart of Wood (2010).

Main Content

Multivariate-normal approximation (Wood 2010, Eq. 2)

The chosen summary statistics are taken to be approximately multivariate normal:

$$\mathbf{s}\sim N({\mathbf{\mu }}_{\theta },{\mathbf{\Sigma }}_{\theta }).$$

The mean ${\mathbf{\mu }}_{\theta }$ and covariance ${\mathbf{\Sigma }}_{\theta }$ are generally intractable functions of the model parameters $\mathbf{\theta }$, but for any $\mathbf{\theta }$ they can be estimated by simulation. Using regression coefficients as statistics promotes the normality that supports this approximation.

Evaluating the synthetic likelihood (Wood 2010, Fig. 2 & Eq.)

For a given parameter vector $\mathbf{\theta }$:

1. Use the model to simulate $N_r$ replicate data sets ${\mathbf{y}}_1^{\ast },{\mathbf{y}}_2^{\ast },\dots$ and convert each to a statistics vector ${\mathbf{s}}_1^{\ast },{\mathbf{s}}_2^{\ast },\dots$ — exactly as $\mathbf{y}$ was converted to $\mathbf{s}$.
2. Estimate the mean: ${\hat{\mathbf{\mu }}}_{\theta }={\sum }_i{\mathbf{s}}_i^{\ast }/N_r$.
3. Form $\mathbf{S}=({\mathbf{s}}_1^{\ast }-{\hat{\mathbf{\mu }}}_{\theta },{\mathbf{s}}_2^{\ast }-{\hat{\mathbf{\mu }}}_{\theta },\dots )$ and estimate the covariance: ${\hat{\mathbf{\Sigma }}}_{\theta }=\mathbf{S}{\mathbf{S}}^{\top }/(N_r-1)$ (a robust covariance estimator can be advantageous here).
4. Drop irrelevant constants; the log synthetic likelihood is

$$l_s(\mathbf{\theta })=-\frac{1}{2}(\mathbf{s}-{\hat{\mathbf{\mu }}}_{\theta }{)}^{\top }{\hat{\mathbf{\Sigma }}}_{\theta }^{-1}(\mathbf{s}-{\hat{\mathbf{\mu }}}_{\theta })-\frac{1}{2}\log |{\hat{\mathbf{\Sigma }}}_{\theta }|.$$

Properties

• Measures fit, but smoothly. Like any likelihood, $l_s(\mathbf{\theta })$ measures the consistency of $\mathbf{\theta }$ with the data — but it is a much smoother function of $\mathbf{\theta }$ than the true density $f_{\theta }$, making it optimizable and samplable.
• Generality. Handles hidden state variables, complicated observation processes, missing data, and multiple data series.
• Invariance & robustness. $l_s$ is invariant to reparameterization and robust to the inclusion of uninformative statistics, so very careful statistic selection is unnecessary; statistics may be freely transformed to improve the normality approximation (Eq. 2).
• Asymptotic in $N_r$. $l_s$ behaves like a conventional likelihood in the $N_r\to \infty$ limit, giving access to likelihood-based inference machinery.

Exploring $l_s$ by MCMC

Metropolis–Hastings exploration (Wood 2010, Methods summary)

$l_s$ usually displays residual small-scale roughness, so smooth-function optimizers fail; instead use Metropolis–Hastings MCMC. From a parameter guess ${\mathbf{\theta }}^{[0]}$, iterate for $k=1,2,\dots$:

1. Propose ${\mathbf{\theta }}^{\ast }={\mathbf{\theta }}^{[k-1]}+{\mathbf{\delta }}^{[k]}$, with ${\mathbf{\delta }}^{[k]}$ from a convenient symmetric distribution.
2. Set ${\mathbf{\theta }}^{[k]}={\mathbf{\theta }}^{\ast }$ with probability $\min [1,\exp \{l_s({\mathbf{\theta }}^{\ast })-l_s({\mathbf{\theta }}^{[k-1]})\}]$; otherwise ${\mathbf{\theta }}^{[k]}={\mathbf{\theta }}^{[k-1]}$.

The chain both locates and quantifies the range of parameter values consistent with the data. (A flat prior makes the acceptance ratio depend only on $l_s$; informative priors enter multiplicatively as usual.)

Point estimation, model comparison, and checking

• MLE via quadratic regression. Near the maximum-likelihood estimate $\hat{\mathbf{\theta }}$, the $N_r\to \infty$ limit of $l_s$ is estimated by quadratic regression of the sampled $l_s({\mathbf{\theta }}^{[k]})$ values on the ${\mathbf{\theta }}^{[k]}$ from the converged chain — recovering $\hat{\mathbf{\theta }}$ and the standard likelihood theory for inference.
• Model comparison. Alternative models compared by AIC or generalized likelihood-ratio testing.
• Model-checking diagnostic. If the model fits,

$$(\mathbf{s}-{\hat{\mathbf{\mu }}}_{\theta }{)}^{\top }{\hat{\mathbf{\Sigma }}}_{\theta }^{-1}(\mathbf{s}-{\hat{\mathbf{\mu }}}_{\theta })\sim {\chi }_{\dim (\mathbf{s})}^2.$$

Connections

• Built on the phase-insensitive statistics that make $\mathbf{s}$ approximately normal.
• The simulate-statistics-then-score loop parallels MSM / Indirect Inference / the SME (match simulated vs. observed summaries) but yields an explicit parametric likelihood rather than a quadratic moment criterion.
• Closely related to ABC: both are likelihood-free and summary-statistic-based, but synthetic likelihood replaces ABC’s acceptance threshold with an MVN density, enabling standard MCMC and likelihood theory.

See Also

• Synthetic Likelihood - Overview — method summary and contribution
• Chaos and Phase-Insensitive Statistics — the statistics fed into $l_s$
• Nicholson’s Blowfly Application — $l_s$, MCMC, and the ${\chi }^2$/AIC diagnostics in action
• Model Comparison — AIC and likelihood-ratio model selection