Meager 2019 - Hierarchical Model Specifications

Summary

Four nested model specifications: the Rubin (1981) model on reported estimates (eq. 1), a full-data model with control means and a correlated bivariate upper level (eq. 2), a fully interacted subgroup model (eq. 3) and a model with site-level covariates (eq. 4). Classical meta-analysis (${\sigma }_{\tau }=0$), no pooling (${\sigma }_{\tau }=\infty$) and full-pooling OLS are special cases; partial pooling shrinks site estimates in inverse proportion to ${\sigma }_{\tau }$.

Overview

The statistical heart of the paper. This note defines each model with every symbol, shows which familiar estimators it nests, and states the exchangeability requirement.

Main Content

Equation (1) - Rubin (1981) model

Studies $k=1,\dots ,K$ report estimates ${\hat{\tau }}_k$ with standard errors ${\widehat{se}}_k$:

$${\hat{\tau }}_k\sim N({\tau }_k,{\widehat{se}}_k^2),{\tau }_k\sim N(\tau ,{\sigma }_{\tau }^2)\forall k.$$

$\tau =E[{\tau }_k]$ is the general (average) effect; ${\sigma }_{\tau }^2=\text{var}({\tau }_k)$ is genuine heterogeneity; and ${\sigma }_{\tau }^2$ also “captures a notion of external validity” as the extent to which any ${\tau }_k$ predicts any other ${\tau }_{k^{\prime }}$. The first line models sampling variation, the second true heterogeneity.

Why Gaussian. The link is motivated by RCT estimators being unbiased and asymptotically normal (so it often imposes no more structure than the original inference); it offers lower mean squared error in many cases (Efron and Morris 1977); and Gaussian hierarchical models give reliable inference on the mean and variance even when the true distribution is not Gaussian (McCulloch and Neuhaus 2011).

Nesting of familiar estimators

Model (1) nests: classical meta-analysis (set ${\sigma }_{\tau }^2=0$: all ${\tau }_k=\tau$; $\hat{\tau }$ is the inverse-variance-weighted average of the ${\hat{\tau }}_k$); the original individual analyses / no pooling (${\sigma }_{\tau }^2\to \infty$ returns the original estimates); and intermediate partial pooling for finite ${\sigma }_{\tau }^2$. Proof sketch: with ${\sigma }_{\tau }^2=0$ the marginal likelihood is ${\hat{\tau }}_k\sim N(\tau ,{\widehat{se}}_k^2)$, whose MLE is the inverse-variance average; with ${\sigma }_{\tau }^2\to \infty$ the prior on ${\tau }_k$ becomes flat so the posterior mean of ${\tau }_k$ is ${\hat{\tau }}_k$.

Equation (2) - Full-data joint model (main specification)

Let $y_{ik}$ be the outcome (e.g. profit) of household $i$ in site $k$ and $T_{ik}\in \{0,1\}$ the treatment indicator:

$$y_{ik}\sim N({\mu }_k+{\tau }_k{T}_{ik},{\sigma }_{y_k}^2)\forall i,k,\left(\begin{array}{c}{\mu }_k\\ {\tau }_k\end{array}\right)\sim N\left(\left(\begin{array}{c}\mu \\ \tau \end{array}\right),V\right),V=\left(\begin{array}{cc}{\sigma }_{\mu }^2& {\sigma }_{\tau \mu }\\ {\sigma }_{\tau \mu }& {\sigma }_{\tau }^2\end{array}\right)\forall k.$$

Symbols: ${\mu }_k$ site control mean; ${\sigma }_{y_k}^2$ site-specific outcome variance; $\mu ,\tau$ upper-level means; $V$ upper-level covariance including the covariance ${\sigma }_{\tau \mu }$ between control means and treatment effects. The Gaussian likelihood is the analogue of the OLS in the original RCTs (the Gaussian kernel is the least-squares objective; MLE point estimates equal OLS, while robust/clustered SEs have no simple likelihood counterpart). The assumption on the outcome data is acknowledged as unrealistic; alternatives are in Meager (2016).

Rationale: information in the control means $\{{\mu }_k\}$ may help if correlated with treatment effects.

Equation (3) - Fully interacted subgroup model

With $L$ binary household-level covariates $X_{ik}$, the power set of subgroups creates $2^L$ intercepts and $2^L$ slopes. Index subgroups by $\pi (p)$: $\{1,\dots ,2^L\}\to \{0,1{\}}^L$ and write $X_{ik}^{\pi (p)}={\prod }_l[X_{lik}{]}^{1\{\pi (p{)}_l=1\}}$ (the subgroup indicator). Then

$$y_{ik}\sim N\left(\sum _{p=1}^{2^L}\right[{\mu }_k^p+{\tau }_k^p{T}_{ik}]X_{ik}^{\pi (p)},{\sigma }_{y_k}^2),\left(\begin{array}{c}{\mu }_k^p\\ {\tau }_k^p\end{array}\right)\sim N\left(\left(\begin{array}{c}{\mu }^p\\ {\tau }^p\end{array}\right),V_p\right),V_p=\left(\begin{array}{cc}{\sigma }_{{\mu }^p}^2& {\sigma }_{{\tau }^p{\mu }^p}\\ {\sigma }_{{\tau }^p{\mu }^p}& {\sigma }_{{\tau }^p}^2\end{array}\right)\forall p,k.$$

Tractability is maintained by enforcing independence across the $2^L$ subgroup blocks. It extends one study’s subgroup analysis to all studies, testing how general a subgroup effect is, even for covariates the original authors did not report.

Equation (4) - Site-level covariates

With $M$ site-level covariates $W_k$ (loan size, interest rate, saturation, randomisation unit, …):

$$y_{ik}\sim N({\mu }_k+{\tau }_k{T}_{ik},{\sigma }_{y_k}^2),\left(\begin{array}{c}{\mu }_k\\ {\tau }_k\end{array}\right)\sim N\left(\left(\begin{array}{c}\mu +W_k{\beta }_{\mu }\\ \tau +W_k{\beta }_{\tau }\end{array}\right),V\right).$$

${\beta }_{\tau }$ ($M$-vector) captures how $W_k$ predicts ${\tau }_k$ and ${\beta }_{\mu }$ the analogue for ${\mu }_k$. With $M\ge K$ (as here: at least seven contextual variables for seven studies) the model overfits and may be analytically intractable; the paper uses a ridge approach instead (Meager 2019 - Covariates - Prior Business Experience and Site-Level Ridge). If the covariates correlated with effects are known, only conditional exchangeability is required.

Special cases of model (2)

Pooling all the data in a single OLS with site fixed effects and a common slope $\tau$ (as in Banerjee, Duflo, Goldberg, Karlan, Osei, Parienté, Shapiro et al. 2015) is the special case ${\sigma }_{\tau }^2=0$, ${\sigma }_{\mu \tau }=0$ and no hierarchy on $\mu$. Neither it nor classical inverse-variance meta-analysis estimates heterogeneity or uses it to inform uncertainty on $\tau$ or ${\tau }_{K+1}$: “These aggregation methods offer no channel through which the data can signal that the studied effects are heterogeneous”.

Pooling and shrinkage

Partial pooling

Estimation of ${\sigma }_{\tau }^2$ is possible even for small $K$. If ${\hat{\sigma }}_{\tau }^2\approx 0$ there is perfect external validity: the hierarchical model pools all data into a single homogeneous effect and shrinks $\{{\hat{\tau }}_k\}$ toward each other; if ${\sigma }_{\tau }^2$ is large, there is no pooling and no shrinkage ($\tau$ becomes uninformative with wide intervals); intermediate values give partial pooling with shrinkage “inversely proportional to the size of ${\sigma }_{\tau }^2$” (Rubin 1981). In the conjugate case the posterior mean of ${\tau }_k$ given $\tau ,{\sigma }_{\tau }$ is ${\omega }_k\tau +(1-{\omega }_k){\hat{\tau }}_k$ with ${\omega }_k={\widehat{se}}_k^2/({\sigma }_{\tau }^2+{\widehat{se}}_k^2)$ (Meager 2019 - Pooling Factors and Measuring External Validity).

Exchangeability

Exchangeability requirement

The treatment effects must be exchangeable: their joint distribution is invariant to permutation of the $K$ indices (Diaconis 1977), e.g. researchers have no knowledge of the relative ordering of the effects before seeing the data. Hence external validity can be assessed only for sites that are exchangeable with those studied; ${\tau }_{K+1}$ applies only to exchangeable sites; and since only RCT data are aggregated, results may not generalise to contexts that could not plausibly have been studied by RCTs. Policymakers must judge whether their context is comparable.

Examples

Nesting check. Take two sites with $\hat{\tau }=(10,0)$ and $\widehat{se}=(5,5)$. Classical meta-analysis gives $\hat{\tau }=5$ for both sites; no pooling returns $(10,0)$; with ${\sigma }_{\tau }=5$ the weight on the pooled mean is $\omega =25/(25+25)=0.5$, giving posterior-mean site effects $0.5\cdot 5+0.5\cdot 10=7.5$ and $2.5$ (taking $\tau =5$). Partial pooling lies between the two extremes.

Connections

• General treatment of hierarchical Bayes in the vault: Hierarchical Models, Hierarchical Linear Models, Partial Pooling as Multiple Comparisons Correction.
• Fixed vs random effects and why random-effects is not always conservative: Sterne 2011 - Magnesium Example, Model Choice and Adjustment.
• Nonparametric identification of the upper-level distribution: Andrews Kasy 2019 - Identification Using Meta-Studies.
• Prior regularisation for MMM coefficients: Bayesian Estimation and Priors for MMM.

Relevance to marketing measurement

Eq. (4) is the structure of a lift-calibrated MMM: channel effect in market/experiment $k$ drawn from a population with mean depending on covariates (spend level, vertical). Eq. (1) is the minimal “combine geo experiment results” model when only point estimates and SEs are available.

See Also

• Meager 2019 - Bayesian Estimation, Priors and Computation
• Meager 2019 - Bayesian Hierarchical Analysis of Microcredit - Overview
• Overfitting and Information Criteria
• Roever 2020 - Bayesian Random-Effects Meta-Analysis with bayesmeta - Overview
• IntHout 2014 - HKSJ vs DerSimonian-Laird Random-Effects Meta-Analysis - Overview