Cinelli Hazlett 2020 - Robustness Value and Extreme Scenarios

Summary

The robustness value $R{V}_q$ is the single partial-$R^2$ a confounder must have with both treatment and outcome to cut the estimate by $100q\%$; $R{V}_{q,\alpha }$ adds the requirement that significance be lost. The extreme-scenario statistic $R_{Y\sim D|X}^2$ is the minimal treatment partial $R^2$ of a confounder that explains all residual outcome variance and still zeroes the effect. Both come from $t$ and $df$ alone.

Overview

Both measures are consequences of the relative-bias identity $\text{rel. bias}=BF/|f_{Y\sim D|X}|$ (relative bias theorem). Throughout, $f_{Y\sim D|X}=t_{{\hat{\tau }}_{res}}/\sqrt{df}$.

Main Content

Robustness value (Eq. 18)

Let $f_q:=q|f_{Y\sim D|X}|$. The robustness value for a proportional reduction $q$ of the point estimate is

$$R{V}_q=\frac{1}{2}\left(\sqrt{f_q^4+4f_q^2}-f_q^2\right).$$

A confounder with $R_{Y\sim Z|D,X}^2=R_{D\sim Z|X}^2=R{V}_q$ reduces ${\hat{\tau }}_{res}$ by exactly $100q\%$. Any confounder with both partial $R^2$ below $R{V}_q$ cannot. $RV\equiv R{V}_{q=1}$ (bring the estimate to zero).

Derivation (Supp. A.2). Set $R_{Y\sim Z|D,X}^2=R_{D\sim Z|X}^2=RV$. Then $BF=\sqrt{RV}\cdot \sqrt{RV/(1-RV)}$ must equal $f_q$. Squaring, $R{V}^2=f_q^2(1-RV)$, i.e. $R{V}^2+f_q^{2}RV-f_q^2=0$ (Eq. 34); the positive root is Eq. 35. Because $BF$ is increasing in both arguments, anything below the diagonal point is also insufficient.

Robustness value with statistical significance (Eqs. 19-20)

With $t_{\alpha ,df-1}^{\ast }$ the critical $t$ and $f_{\alpha ,df-1}:=|t_{\alpha ,df-1}^{\ast }|/\sqrt{df-1}$, define $f_{q,\alpha }:=f_q-f_{\alpha ,df-1}$. If $f_{q,\alpha }<0$ then $R{V}_{q,\alpha }=0$; otherwise

$$R{V}_{q,\alpha }=\frac{1}{2}\left(\sqrt{f_{q,\alpha }^4+4f_{q,\alpha }^2}-f_{q,\alpha }^2\right).$$

A confounder with both partial $R^2$ equal to $R{V}_{q,\alpha }$ makes the adjusted test fail to reject $H_0:\tau =(1-q)|{\hat{\tau }}_{res}|$ at level $\alpha$ (equivalently the $1-\alpha$ CI covers $(1-q)|{\hat{\tau }}_{res}|$).

Caveats (Supp. A.4): because a confounder lowers residual variance of $Y$ (VRF), a larger $R_{Y\sim Z|D,X}^2$ can help significance. The “nothing weaker on both axes can overturn” guarantee holds when $R{V}_{q,\alpha }>1-1/f_q^2$ (Eq. 46), which usually holds, e.g. for $q=1$ whenever $R_{Y\sim D|X}^2\le 50\%$. Otherwise set $R{V}_{q,\alpha }=(f_q^2-f_{\alpha ,df-1}^{\ast 2})/(1+f_q^2)$ (Eq. 48). As $df\to \infty$, $R{V}_{q,\alpha }\to R{V}_q$; with $t^{\ast }=0$ it equals $R{V}_q$.

Extreme-scenario interpretation of $R_{Y\sim D|X}^2$

If a confounder explains all residual outcome variance ($R_{Y\sim Z|D,X}^2=1$), it brings the estimate to zero iff $R_{D\sim Z|X}^2\ge R_{Y\sim D|X}^2$. So $R_{Y\sim D|X}^2=t^2/(t^2+df)$ is the minimal treatment association such a worst-case confounder must have.

Proof (Supp. A.2). Set $q=1$, $R_{Y\sim Z|D,X}^2=1$ in $BF=|f_{Y\sim D|X}|$: $f_{D\sim Z|X}=|f_{Y\sim D|X}|$, i.e. $R_{D\sim Z|X}^2/(1-R_{D\sim Z|X}^2)=R_{Y\sim D|X}^2/(1-R_{Y\sim D|X}^2)$, hence the partial $R^2$ values coincide.

For intermediate cases the paper plots curves of the adjusted estimate against $R_{D\sim Z|X}^2$ for $R_{Y\sim Z|D,X}^2\in \{100\%,75\%,50\%\}$ (Fig. 3).

Bias factor table (Supp. D, Table 2)

Reading: compute $|f|=t/\sqrt{df}$. To zero the estimate, $BF>|f|$; for a change of $100q\%$ use $q|f|$. For Darfur $|f|\approx 0.15$: $R_{D\sim Z|X}^2=5\%,R_{Y\sim Z|D,X}^2=40\%$ gives $BF=0.145<0.15$ (robust) but $R_{D\sim Z|X}^2=40\%,R_{Y\sim Z|D,X}^2=5\%$ gives $0.183>0.15$ (not robust). Excerpt (rows: $R_{D\sim Z|X}^2$, columns $R_{Y\sim Z|D,X}^2$):

$R_D^2\backslash R_Y^2$	5%	10%	20%	40%	60%	80%
5%	0.051	0.073	0.103	0.145	0.178	0.205
10%	0.075	0.105	0.149	0.211	0.258	0.298
20%	0.112	0.158	0.224	0.316	0.387	0.447
40%	0.183	0.258	0.365	0.516	0.632	0.730

Frank (2000) impact thresholds for non-zero nulls (Supp. A.5)

Counterexample: no $X$, standardised variables, $n=1000$, ${\hat{\tau }}_{res}=0.5$, $\text{se}=0.0274$, target $\hat{\tau }=-0.5$ (relative bias 2). Solving with $R_{Y\sim Z}=R_{D\sim Z}$ gives impact $2/3$, but a confounder with impact as low as $0.51$ also works, so impact thresholds do not generalise beyond the zero null; the RV does.

Examples

Darfur robustness value

$t=4.18$, $df=783$: $f=4.18/\sqrt{783}=0.1494$, $RV=\frac{1}{2}(\sqrt{0.1494^4+4\cdot 0.1494^2}-0.1494^2)=0.1386\approx 13.9\%$. With $\alpha =0.05$: $t_{0.05,782}^{\ast }=1.963$, $f_{\alpha }=1.963/\sqrt{782}=0.0702$, $f_{1,\alpha }=0.0792$, giving $R{V}_{\alpha }=7.7\%$ (paper 7.6%). $R_{Y\sim D|X}^2=4.18^2/(4.18^2+783)=2.2\%$.

import numpy as np
from scipy import stats
def rv(t, df, q=1.0, alpha=None):
    f = q*abs(t)/np.sqrt(df)
    if alpha is not None:
        f -= stats.t.ppf(1-alpha/2, df-1)/np.sqrt(df-1)
        if f <= 0: return 0.0
    return 0.5*(np.sqrt(f**4+4*f**2)-f**2)
print(rv(4.18,783), rv(4.18,783,alpha=.05), 4.18**2/(4.18**2+783))

Connections

• E-value comparison: Cinelli Hazlett 2020 - Informal Benchmarking Pitfalls and Related Methods.
• Use of the RV with expert bounds: Cinelli Hazlett 2020 - Bounding Confounder Strength with Observed Covariates.
• Marketing use (e.g. is a brand-lift or MMM coefficient robust?): Cinelli Hazlett 2020 - Planted Confounders and Bayesian Sketch.

See Also

Sensitivity Analysis in Observational Studies, Cinelli Hazlett 2020 - Making Sense of Sensitivity - Overview