Andrews Kasy 2019 - Identification Using Meta-Studies

Summary

Meta-studies report estimates with varying standard errors. If the true effect is independent of the standard error across latent studies and publication depends only on the z-statistic, then the published distribution at a larger standard error is not a noised-up copy of that at a smaller one; the deviation identifies $p(\cdot )$ and $\mu$ up to scale (Theorem 3).

Overview

This note covers Section 3.2: the meta-study DGP, the conditional density of the z-statistic given $\sigma$, Theorem 3, the sign-normalised corollary, and dependence of publication on $\sigma$.

Main Content

Definition 3 - Meta-study data generating process

$({\sigma }_i^{\ast },{\Theta }_i^{\ast },X_i^{\ast },D_i)$ are jointly i.i.d. with

$${\sigma }_i^{\ast }\sim {\mu }_{\sigma },{\Theta }_i^{\ast }|{\sigma }_i^{\ast }\sim {\mu }_{\Theta },X_i^{\ast }|{\Theta }_i^{\ast },{\sigma }_i^{\ast }\sim N({\Theta }_i^{\ast },{\sigma }_i^{\ast 2}),D_i|X_i^{\ast },{\Theta }_i^{\ast },{\sigma }_i^{\ast }\sim \text{Bernoulli}(p(X_i^{\ast }/{\sigma }_i^{\ast })).$$

Observed: i.i.d. draws $(X_j,{\sigma }_j)=(X_{I_j}^{\ast },{\sigma }_{I_j}^{\ast })$. Write $Z_i^{\ast }=X_i^{\ast }/{\sigma }_i^{\ast }$ and $Z_j=X_j/{\sigma }_j$. Key identifying assumption: ${\Theta }^{\ast }$ is statistically independent of ${\sigma }^{\ast }$ (studies with larger samples do not have systematically different estimands); publication depends on the z-statistic alone.

Lemma 3 - Meta-study density

In Definition 3 the conditional density of $Z$ given $\sigma$ is

$$f_{Z|\sigma }(z|\sigma )=\frac{p(z)}{E[p(Z^{\ast })\mid \sigma ]}\int \phi (z-\frac{\theta }{\sigma })d{\mu }_{\Theta }(\theta ).$$

Consequently for ${\sigma }_2>{\sigma }_1$

$$\frac{f_{Z|\sigma }(z|{\sigma }_2)}{f_{Z|\sigma }(z|{\sigma }_1)}=\frac{E[p(Z^{\ast })|\sigma ={\sigma }_1]}{E[p(Z^{\ast })|\sigma ={\sigma }_2]}\cdot \frac{\int \phi (z-\theta /{\sigma }_2)d{\mu }_{\Theta }}{\int \phi (z-\theta /{\sigma }_1)d{\mu }_{\Theta }},$$

where the first factor does not depend on $z$.

The ratio on the left is identified; the idea is to invert the right-hand side to recover ${\mu }_{\Theta }$, which then identifies $p$.

Theorem 3 - Nonparametric identification using meta-studies

In Definition 3, suppose the support of $\sigma$ contains an open interval. Then $p(\cdot )$ is identified up to scale and ${\mu }_{\Theta }$ is identified as well.

Proof idea (supplement): consider ${\partial }_{\sigma }\log f_{Z|\sigma }(z|\sigma )$; the $z$-free normalising factor drops out, leaving a function of ${\mu }_{\Theta }$ alone, which can be inverted. I have not reproduced the proof (it is in the supplement, not the PDF).

Corollary 3 - Sign-normalised estimates

In Definition 3, assume additionally that $p$ is symmetric, $p(x/\sigma )=p(-x/\sigma )$. If we observe i.i.d. $(|X|,\sigma )$, then $p$ is nonparametrically identified on $\mathbb{R}$ up to scale and the distribution of $|{\Theta }^{\ast }|$ is identified.

Dependence on the standard error

If $D_i|\cdot \sim \text{Bernoulli}(p(X_i^{\ast }/{\sigma }_i^{\ast })q({\sigma }_i^{\ast }))$, Theorem 3 still identifies $p$. The generalised model is observationally equivalent to Definition 3 with the distribution of ${\sigma }^{\ast }$ reweighted by $q(\sigma )/E[q({\sigma }^{\ast })]$.

Worked picture (Fig 3)

Illustrative example continued - selection on significance with varying $\sigma$

Setup. ${\Theta }^{\ast }\sim N(1,1)$ independent of ${\sigma }^{\ast }$; $X^{\ast }|{\Theta }^{\ast },{\sigma }^{\ast }\sim N({\Theta }^{\ast },{\sigma }^{\ast 2})$; $p(X^{\ast }/{\sigma }^{\ast })=1$ if $|X^{\ast }/{\sigma }^{\ast }|>1.96$ and $0.1$ otherwise. Compare two standard deviations, $A$ (small) and $B$ (large) in Fig 3. Solution. Latent: the distribution of $X^{\ast }$ at larger $\sigma$ is the small-$\sigma$ distribution convolved with normal noise. Published: statistically insignificant observations are missing at larger $\sigma$ (the insignificant band $|X|<1.96\sigma$ widens with $\sigma$), so published $X$ tend to be larger on average at larger $\sigma$. Interpretation. The failure of the convolution relationship in the published data is attributed to selection; how the published conditional distribution of $X$ varies with $\sigma$ identifies $p$ up to scale.

Because the meta-study route relies on the independence assumption, the paper cross-checks it against the replication route in the Camerer and OSC data: the estimates are similar, though meta-study estimates suggest somewhat less selection (Andrews Kasy 2019 - Estimation and Replication Applications in Economics and Psychology).

Examples

Fig 3 example above. Multiple estimates per study (minimum-wage data): the model’s conditional independence only needs $D_i\perp ({\Theta }_i^{\ast },{\sigma }_i^{\ast })|X_i^{\ast }/{\sigma }_i^{\ast }$; the paper clusters standard errors by study (Andrews Kasy 2019 - Meta-Study Applications - Minimum Wage and Deworming).

Connections

• Egger-type regressions rely on a related intuition (small-sample estimates are larger when selected): Andrews Kasy 2019 - Relation to Existing Approaches and Meta-Regressions, Sterne 2011 - Tests for Asymmetry and the Egger Regression.
• Replication-based alternative: Andrews Kasy 2019 - Identification Using Replication Studies.
• Hierarchical models for ${\mu }_{\Theta }$ (the upper level): Meager 2019 - Hierarchical Model Specifications.

Relevance to marketing measurement

The repository of lift tests is a meta-study: each test has an estimate and a standard error. The independence assumption (true lift not related to test size) is the one to defend - budgets and test durations are often correlated with the channel and thus with true effect sizes, which breaks the assumption exactly as Sterne et al. warn about size-effect correlations.

See Also

• Andrews Kasy 2019 - Setting and the Truncated Sampling Model
• Andrews Kasy 2019 - Identification of and Correction for Publication Bias - Overview
• Sterne 2011 - Funnel Plots and the Causes of Asymmetry