Andrews Kasy 2019 - Identification Using Replication Studies

Summary

If each published original estimate $X$ has an independent replication $X^r$ of the same parameter, and selection acts only on $X$, then the joint density of $(X,X^r)$ is asymmetric with $f(b,a)/f(a,b)=p(b)/p(a)$. Hence $p(\cdot )$ is nonparametrically identified up to scale (Theorem 1), and with normal estimates and unequal variances identification persists (Corollary 1), including under sign normalisation (Corollary 2).

Overview

This note covers Section 3.1: the replication DGP, the symmetry argument, Theorem 1 and its testable restrictions, extensions to unequal variances, normalised signs, and selection that depends on the true effect (Theorem 2), with the paper’s worked picture (Fig 2).

Main Content

Definition 2 - Replication data generating process

$({\Theta }_i^{\ast },X_i^{\ast },D_i,X_i^{\ast r})$ are jointly i.i.d. with

$${\Theta }_i^{\ast }\sim \mu ,X_i^{\ast }|{\Theta }_i^{\ast }\sim f_{X^{\ast }|{\Theta }^{\ast }},D_i|X_i^{\ast },{\Theta }_i^{\ast }\sim \text{Bernoulli}(p(X_i^{\ast })),X_i^{\ast r}\mid D_i,X_i^{\ast },{\Theta }_i^{\ast }\sim f_{X^{\ast }|{\Theta }^{\ast }}(\cdot |{\Theta }_i^{\ast }).$$

We observe i.i.d. $(X_j,X_j^r)=(X_{I_j}^{\ast },X_{I_j}^{\ast r})$. Key features: $(X^{\ast },X^{\ast r})$ is symmetric in distribution, and selectivity acts only on $X^{\ast }$, not on the replication. Plausible for systematic replication projects (OSC 2015; Camerer et al. 2016) but may fail for non-systematic replications published only when they “debunk” earlier results.

Lemma 2 - Replication density

Conditional on $\Theta$, $f_{X,X^r|\Theta }(x,x^r|\theta )=\frac{p(x)}{E[p(X_i^{\ast })|{\Theta }_i^{\ast }=\theta ]}{f}_{X^{\ast }|{\Theta }^{\ast }}(x|\theta )f_{X^{\ast }|{\Theta }^{\ast }}(x^r|\theta )$. The marginal density is

$$f_{X,X^r}(x,x^r)=\frac{p(x)}{E[p(X_i^{\ast })]}\int f_{X^{\ast }|{\Theta }^{\ast }}(x|\theta )f_{X^{\ast }|{\Theta }^{\ast }}(x^r|\theta )d\mu (\theta ).$$

Hence for any $a,b$ with non-zero denominators

$$\frac{f_{X,X^r}(b,a)}{f_{X,X^r}(a,b)}=\frac{p(b)}{p(a)}.$$

Proof sketch: $D\perp X^{\ast r}|(X^{\ast },{\Theta }^{\ast })$ gives the product; integrating out $\Theta$ gives the marginal; the integral is symmetric in $(x,x^r)$ so cancels in the ratio.

Theorem 1 - Nonparametric identification using replication experiments

In the setup of Definition 2, assume the support of $f_{X^{\ast },X^{\ast r}}$ is of the form $A\times A$ for some measurable set $A$. Then $p(\cdot )$ is nonparametrically identified on $A$ up to scale.

Argument. By Lemma 2, $p(b)/p(a)$ is an observable density ratio for every pair in $A$; fixing $p(a_0)=1$ at one point fixes the scale. Scale is not identifiable because multiplying $p$ by a constant leaves the published distribution unchanged.

Testable restrictions

Define $h(a,b)=\log f_{X,X^r}(b,a)-\log f_{X,X^r}(a,b)$. By Lemma 2, $h(a,b)=\log p(b)-\log p(a)$, hence for any three values $a,b,c$

$$h(a,b)+h(b,c)+h(c,a)=0.$$

One could build a nonparametric specification test from this and an estimate of $f_{X,X^r}$; the paper instead tests restrictions of a larger identified model (Thm 2 specification, below).

Worked picture (Fig 2)

Illustrative example continued - selection on significance with replications

Setup. ${\Theta }^{\ast }\sim N(1,1)$; $Z^{\ast }|{\Theta }^{\ast }\sim N({\Theta }^{\ast },1)$; replication $Z^{\ast r}|{\Theta }^{\ast }\sim N({\Theta }^{\ast },1)$; $p(z)=1$ for $|z|>1.96$ and $0.1$ otherwise. Region $A$: original significant, replication not ($|Z|>1.96,|Z^r|\le 1.96$). Region $B$: original not significant, replication significant. Solution. By symmetry of the latent process the masses of $A$ and $B$ are equal (simulation: $0.1437$ and $0.1442$). After selection, $B$ retains only 10% of its draws. The published mass ratio is $A:B=0.1437:0.01442\approx 9.97\approx 10=p(\text{sig})/p(\text{insig})$ - exactly the ratio Lemma 2 predicts. Interpretation. The excess of points in $A$ over $B$ (asymmetry about the diagonal $Z=Z^r$) is the signature of selection, and its size identifies $p$ up to scale. Fig 2 draws 100 latent pairs (left) and the published subset (right, 90% of insignificant originals deleted). The simulation numbers are my replication of this setup.

Generalisations (Sec. 3.1.2)

Different variances. Replication sample sizes differ from originals, and may depend on $Z^{\ast }$ (power calculations in OSC and Camerer et al. make ${\sigma }^{\ast }$ increase with $|Z^{\ast }|$ but unrelated to ${\Theta }^{\ast }$ conditionally). Normalising the original’s variance to one:

$${\Theta }_i^{\ast }\sim \mu ,Z_i^{\ast }|{\Theta }_i^{\ast }\sim N({\Theta }_i^{\ast },1),D_i|Z_i^{\ast },{\Theta }_i^{\ast }\sim \text{Bernoulli}(p(Z_i^{\ast })),{\sigma }_i^{\ast }|Z_i^{\ast },D_i,{\Theta }_i^{\ast }\sim f_{\sigma |Z^{\ast }},Z_i^{\ast r}|{\sigma }_i^{\ast },Z_i^{\ast },D_i,{\Theta }_i^{\ast }\sim N({\Theta }_i^{\ast },{\sigma }_i^{\ast 2}).\text{\hspace{1em}(2)}$$

Corollary 1

In the setup (2), given i.i.d. draws of $(Z,Z^r)$ (and $\sigma$), $p(\cdot )$ is nonparametrically identified on $\mathbb{R}$ up to scale, and $\mu$ is identified as well. Proof idea: recover the symmetric setting by (de)convolving $Z^r$ with normal noise given $Z$ and $\sigma$, then apply Theorem 1; normality allows recovery of $\mu$.

Normalised sign. In the paper’s datasets the sign of the initial estimate is set positive and the sign of $Z^r$ adjusted (the studies examine different outcomes, so relative signs are arbitrary).

Corollary 2

In the setup (2), assume $p$ is symmetric, $p(z)=p(-z)$, and $f_{\sigma |Z^{\ast }}(\sigma |z)=f_{\sigma |Z^{\ast }}(\sigma |-z)$. If we observe $(W,W^r)=\text{sign}(Z)\cdot (Z,Z^r)$, then $p(\cdot )$ is nonparametrically identified on $\mathbb{R}$ up to scale and the distribution of $|{\Theta }^{\ast }|$ is identified.

Selection depending on the true effect (Sec. 3.1.3)

Publication may depend on other findings in the manuscript or on unreported results, so $D\perp {\Theta }^{\ast }|X^{\ast }$ may fail. Allow $D_i|Z_i^{\ast },{\Theta }_i^{\ast }\sim \text{Bernoulli}(p(Z_i^{\ast },{\Theta }_i^{\ast }))$ in a model otherwise as (2) (display (3)); this accommodates alternative specifications and robustness checks not in the replication data.

Theorem 2

In the setup (3), $f_{Z|\Theta }$ is nonparametrically identified. The joint density of $(Z,Z^r,\sigma ,\Theta )$ is

$$f_{Z,Z^r,\sigma ,\Theta }(z,z^r,\sigma ,\theta )=\frac{p(z,\theta )}{E[p(Z^{\ast },{\Theta }^{\ast })]}\phi (z-\theta )\frac{1}{\sigma }\phi \left(\frac{z^r-\theta }{\sigma }\right)f_{\sigma |Z^{\ast }}(\sigma |z)\frac{d\mu }{d\nu }(\theta ).$$

Without restrictions $p(z,\theta )$ is not identified (divide $p$ by $g(\theta )$ and multiply $d\mu /d\nu$ by $g(\theta )$ for an observationally equivalent model), but $p(z,\theta )$ is identified up to a $\theta$-dependent normalisation, since $f_{Z|\Theta }(z|\theta )/f_{Z^{\ast }|{\Theta }^{\ast }}(z|\theta )=p(z,\theta )/E[p(Z^{\ast },{\Theta }^{\ast })|{\Theta }^{\ast }=\theta ]$; e.g. impose ${\sup }_{z}p(z,\theta )=1$.

This is used for specification checks: the applications fit a parametric version and test $p(z,\theta )=p(z)$ (score tests, p-values 0.53 and 0.42; see Andrews Kasy 2019 - Estimation and Replication Applications in Economics and Psychology).

Examples

Fig 2 example above (regions $A$ and $B$). A second use: if only 5%-significant originals are ever published ($p\approx 0$ otherwise), region $B$ is empty and the ratio $f(b,a)/f(a,b)$ diverges - the extreme case behind the McCrary et al. conservative tests (Andrews Kasy 2019 - Median-Unbiased Estimation and Corrected Confidence Sets).

Connections

• Meta-analytic alternative that needs no replications: Andrews Kasy 2019 - Identification Using Meta-Studies.
• How often significant results “replicate” even without selection: Andrews Kasy 2019 - Relation to Existing Approaches and Meta-Regressions.
• Estimates from the Camerer and OSC data: Andrews Kasy 2019 - Estimation and Replication Applications in Economics and Psychology.

Relevance to marketing measurement

A re-run (hold-out) of a geo test in a comparable market is a replication $X^r$ of the same $\Theta$. If your team only re-runs tests that looked strong, selection acts on $X$ only and the assumptions of Theorem 1 hold; if re-runs happen only for tests that look suspicious, they do not.

See Also

• Andrews Kasy 2019 - Setting and the Truncated Sampling Model
• Andrews Kasy 2019 - Identification of and Correction for Publication Bias - Overview
• Geo-Experiment Methodology - Overview