Andrews Kasy 2019 - Median-Unbiased Estimation and Corrected Confidence Sets

Summary

For known $p(\cdot )$ and a normal estimate, inverting the truncated CDF $F_{X|\Theta }(x|\theta )$ in $\theta$ gives an $\alpha$-quantile-unbiased estimator ${\hat{\theta }}_{\alpha }(X)$. Its median version and the equal-tailed interval built from two quantiles exactly correct bias and coverage for the published study’s own parameter. With $p$ estimated, a Bonferroni construction keeps coverage at least $1-\alpha$.

Overview

Section 4 turns identification into inference. This note states Theorem 4 and Lemma 5, the corrected interval (including a caveat about endpoint ordering), the continued illustrative example (Fig 4) with recomputed numbers and code, the McCrary et al. relation, the paper’s stance on publication standards, and the Bonferroni interval for estimated $p$. Inference is frequentist, conditional on $\Theta$ (standard notions of bias and size control), whereas identification used an empirical-Bayes view with ${\Theta }^{\ast }\sim \mu$. Supplement extensions: optimal quantile-unbiased estimators with nuisance parameters, Bayesian inference.

Main Content

Truncated CDF and the quantile-unbiased estimator

With $f_{X|\Theta }$ from Lemma 1, define

$$F_{X|\Theta }(x|\theta )={\int }_{-\infty }^x{f}_{X|\Theta }(\tilde{x}|\theta )d\tilde{x}=\frac{1}{E[p(X^{\ast })|{\Theta }^{\ast }=\theta ]}{\int }_{-\infty }^{x}p(\tilde{x})f_{X^{\ast }|{\Theta }^{\ast }}(\tilde{x}|\theta )d\tilde{x}.$$

For $\alpha \in (0,1)$ let ${\hat{\theta }}_{\alpha }(x)$ solve

$$F_{X|\Theta }\left(x\right|{\hat{\theta }}_{\alpha }(x))=\alpha ,\text{\hspace{1em}(4)}$$

so $x$ sits at the $\alpha$-quantile of the distribution implied by ${\hat{\theta }}_{\alpha }(x)$. (An approach used by D. Andrews 1993 and Stock and Watson 1998.)

Theorem 4

If for all $x$, $F_{X|\Theta }(x|\theta )$ is continuous and strictly decreasing in $\theta$, tends to one as $\theta \to -\infty$ and to zero as $\theta \to \infty$, then ${\hat{\theta }}_{\alpha }(x)$ defined in (4) exists, is unique, and is continuous and strictly increasing in $x$. If further $F_{X|\Theta }(x|\theta )$ is continuous in $x$ for all $\theta$, then ${\hat{\theta }}_{\alpha }(X)$ is $\alpha$-quantile unbiased under Definition 1:

$$P({\hat{\theta }}_{\alpha }(X)\le \theta \mid \Theta =\theta )=\alpha \text{for all }\theta .$$

Proof sketch: ${\hat{\theta }}_{\alpha }(X)\le \theta ⟺F(X|{\hat{\theta }}_{\alpha }(X))\ge F(X|\theta )⟺\alpha \ge F(X|\theta )$, and $F(X|\theta )$ is uniform on $(0,1)$ under the truncated model (probability integral transform).

Lemma 5

If $X^{\ast }|({\Theta }^{\ast },{\sigma }^{\ast })\sim N({\Theta }^{\ast },{\sigma }^{\ast 2})$, $p(x)>0$ for all $x$ and $p$ is almost everywhere continuous, then the assumptions of Theorem 4 hold.

Estimator and interval. The median-unbiased estimator is ${\hat{\theta }}_{1/2}(X)$. The paper states the equal-tailed level $1-\alpha$ set as $[{\hat{\theta }}_{\alpha /2}(X),{\hat{\theta }}_{1-\alpha /2}(X)]$. Other selection-corrected intervals exist: inverting UMP unbiased tests in exponential families (Fithian, Sun and Taylor 2014), weighted-average-risk-minimising unbiased estimators (Mueller and Wang 2015), or the MLE based on $f_{X|\Theta }$.

Endpoint ordering under convention (4)

With $F(x|\theta )$ strictly decreasing in $\theta$, solving (4) makes ${\hat{\theta }}_{\alpha }(x)$ decrease in $\alpha$. In my numerical check (below) ${\hat{\theta }}_{0.025}(2.0)=3.075>{\hat{\theta }}_{0.975}(2.0)=-0.605$. The equal-tailed interval therefore has lower endpoint ${\hat{\theta }}_{1-\alpha /2}(X)$ and upper endpoint ${\hat{\theta }}_{\alpha /2}(X)$, i.e. $[{\hat{\theta }}_{1-\alpha /2},{\hat{\theta }}_{\alpha /2}]$, which is what the code uses. The printed order in the working paper appears to use the opposite labelling of quantiles; the coverage claim is unaffected.

Illustrative example continued (Fig 4)

Corrected estimator and interval in the ten-fold-selectivity model

Setup. $Z\sim N(\theta ,1)$ latent; $p(z)=1$ for $|z|>1.96$, $0.1$ otherwise. For observed published $z$ compute the median-unbiased estimate and the 95% interval; compare with $z$ and $[z-1.96,z+1.96]$. Solution (recomputed with the code below).

published $z$	median-unbiased ${\hat{\theta }}_{1/2}$	corrected 95% CI	naive 95% CI
2.0	0.717	[-0.605, 3.075]	[0.04, 3.96]
2.5	1.832	[-0.178, 4.345]	[0.54, 4.46]
3.0	2.758	[0.342, 4.939]	[1.04, 4.96]
4.0	3.975	[1.735, 5.959]	[2.04, 5.96]
5.0	4.999	[2.977, 6.960]	[3.04, 6.96]

Interpretation. The corrected estimate lies below $z$ for small positive $z$ and converges to $z$ (the paper: the difference “is eventually decreasing”); the corrected interval has exactly correct coverage, is narrower than the usual one for small $z$ (width 3.68 vs 3.92 at $z=2$), wider for moderate $z$ (4.52 at $z=2.5$), and essentially identical from $z\approx 5$. This matches the description of Fig 4. Under selection, a $z\approx 2$ is not evidence against zero: the corrected interval at $z=2.0$ contains 0.

import numpy as np
from scipy.stats import norm
from scipy.optimize import brentq
 
C, P_INS = 1.96, 0.1          # p(z) = 0.1 on [-C, C], 1 outside
 
def mass(a, b, th): return norm.cdf(b - th) - norm.cdf(a - th)
 
def trunc_cdf(x, th):
    xi = min(max(x, -C), C)
    num = P_INS * mass(-C, xi, th) + mass(-np.inf, min(x, -C), th) + (mass(C, x, th) if x > C else 0.0)
    den = P_INS * mass(-C, C, th) + mass(-np.inf, -C, th) + mass(C, np.inf, th)
    return num / den
 
def theta_hat(x, a):          # solve F(x | theta) = a   (equation 4)
    return brentq(lambda t: trunc_cdf(x, t) - a, -15, 15)
 
for z in [2.0, 2.5, 3.0, 4.0, 5.0]:
    lo, hi = theta_hat(z, .975), theta_hat(z, .025)       # F decreases in theta
    print(f"z={z}: median-unbiased={theta_hat(z, .5):.3f}, 95% CI=[{lo:.3f}, {hi:.3f}]")

Relation to McCrary et al. (2016)

McCrary et al. propose conservative tests under an extreme publication bias in which insignificant results are never published. Take $p(z)$ as in that model ($p=0$ on insignificant results). Then for testing $\theta =0$ the equal-tailed interval contains zero if and only if the McCrary test fails to reject. So the present method generalises it to partial selection.

What this does not recommend

Do not change publication standards to match corrected cut-offs

The paper does not advocate adjusting publication standards to reflect the corrected critical values: if more stringent cut-offs were used in publication, the corrections would have to be adjusted (an “arms race” of selectivity). The purpose is for readers of published research to draw valid inferences taking the publication rule as given. The rule itself can be chosen on other grounds (welfare, incentives) - see Andrews Kasy 2019 - Conclusion, Optimal Publication Rules and Limitations.

Estimation error in $p(\cdot )$

If $p_{\eta }(\cdot )$ is indexed by a finite-dimensional $\eta$ with a consistent asymptotically normal $\hat{\eta }$ and ${\hat{\theta }}_{\alpha }(x;\eta )$ is continuously differentiable in $\eta$ (true for the paper’s specifications), median-unbiased estimation is hard but valid confidence sets are straightforward.

Bonferroni interval for estimated $p$

For $0<\gamma <\alpha$ consider

$$[{\hat{\theta }}_{1-\frac{\alpha -\gamma }{2}}(X;\hat{\eta })-c_{1-\frac{\gamma }{2}}{\hat{\sigma }}_L(X),{\hat{\theta }}_{\frac{\alpha -\gamma }{2}}(X;\hat{\eta })+c_{1-\frac{\gamma }{2}}{\hat{\sigma }}_U(X)],$$

where $c_{1-\gamma /2}$ is the $1-\gamma /2$ quantile of $N(0,1)$ and ${\hat{\sigma }}_L,{\hat{\sigma }}_U$ are delta-method standard errors of the two endpoint estimators (endpoint labelling follows the convention in the warning above). If the model for $p$ is correctly specified, Bonferroni’s inequality gives coverage at least $1-\alpha$ in large samples. For a fully nonparametric $p$ (no asymptotically normal estimator), take an initial level-$(1-\gamma )$ confidence set $CS$ for $\eta$ and use $[{\inf }_{\eta \in CS}{\hat{\theta }}_{\text{lower}}(X;\eta ),{\sup }_{\eta \in CS}{\hat{\theta }}_{\text{upper}}(X;\eta )]$ at level $\alpha -\gamma$.

I reconstructed the exact endpoint indices from the surrounding text (subscripts were not recoverable from the PDF text layer); the structure (level $\alpha -\gamma$ on the quantile, normal critical value at $\gamma$) follows the stated Bonferroni logic: total miss probability $(\alpha -\gamma )+\gamma =\alpha$.

Examples

Fig 4 example above; empirical use in the economics lab data: the paper’s Fig 6 compares original, adjusted (using $\bar{p}=0.029$) and replication estimates for 18 studies (Andrews Kasy 2019 - Estimation and Replication Applications in Economics and Psychology).

Connections

• Where $p$ comes from: Andrews Kasy 2019 - Identification Using Replication Studies, Andrews Kasy 2019 - Identification Using Meta-Studies.
• Selection-adjusted inference is the frequentist cousin of shrinkage in hierarchical models: Hierarchical Models, Partial Pooling as Multiple Comparisons Correction.
• The winner’s-curse flavour of the problem: Ioannidis 2005 - Why Most Published Findings Are False - Overview.

Relevance to marketing measurement

For a reported lift with $z\approx 2$ from a library with a known preference for significant winners, the corrected estimate is roughly a third of the headline in the paper’s stylised model ($0.72$ vs $2.0$ in z units). Using such a corrected value, with its wider interval, as an informative prior in a Bayesian MMM is more defensible than the raw headline number. Choosing $p$ is the hard part; the paper’s identification results are how to estimate it.

See Also

• Andrews Kasy 2019 - Setting and the Truncated Sampling Model
• Andrews Kasy 2019 - Identification of and Correction for Publication Bias - Overview
• Bayesian Estimation and Priors for MMM