Cinelli Hazlett 2020 - Planted Confounders and Bayesian Sketch

Summary

The OVB formula is an exact algebraic identity, so it can be verified by planting a synthetic confounder of known partial $R^2$ values and comparing the formula’s predicted adjusted estimate with the regression that includes it. Putting priors on the two partial $R^2$ turns the identity into a Bayesian sensitivity analysis (a posterior over $\tau$ given a prior over confounder strength). This note records the check, a PyMC sketch (untested), and how the idea maps to the job-training and ad-targeting examples.

Overview

This is a synthesis note (concept note) built on the paper’s formulas; the simulation and code are this vault’s own and the PyMC model is untested.

Main Content

Planting a confounder of exact strength (E107 pattern)

Procedure for a benchmark with a known answer (for instance an experiment or a hidden measured covariate):

1. Run the restricted regression to get ${\hat{\tau }}_{res}$, $\text{se}$, $df$ and the residuals $D^{\perp X}$, $Y^{\perp X,D}$.
2. Construct a synthetic $Z$ whose residualised version has target correlations ${\rho }_{DZ}$ with $D^{\perp X}$ and ${\rho }_{YZ}$ with $Y^{\perp X,D}$ (e.g. $Z=a{D}^{\perp X}+b{Y}^{\perp X,D}+c\epsilon$, then solve for coefficients).
3. Refit with $Z$ included; compare ${\hat{\tau }}_{res}-\hat{\tau }$ with the formula $\sqrt{R_{Y\sim Z|D,X}^2{R}_{D\sim Z|X}^2/(1-R_{D\sim Z|X}^2)}\cdot \text{sd}(Y^{\perp X,D})/\text{sd}(D^{\perp X})$.

Simulation check

Data: $X,Z\sim N(0,1)$, $D=0.8Z+0.5X+\epsilon$, $Y=0.6Z+0.4X+0\cdot D+\epsilon$, $n=5000$, true effect 0. Restricted ${\hat{\tau }}_{res}=0.2927$. Realised $R_{D\sim Z|X}^2=0.4130$, $R_{Y\sim Z|D,X}^2=0.1904$. Formula bias $=0.310343509$; actual ${\hat{\tau }}_{res}-\hat{\tau }=0.310343509$ (agree to 15 digits). The identity holds because it is algebra, not a model.

Robustness value as a pre-registered benchmark

For a job-training or lift regression, compute $RV$ and $R_{Y\sim D|X}^2$ first, then ask which observed covariate has comparable partial $R^2$ values. A recurring check in observational benchmarks against experiments (LaLonde NSW vs CPS; see also Experimental Benchmarks for Observational Ad Measurement): hide a measured covariate, compute the bound with $k=1$, then verify the true shift stays inside it. By the bounding theorem the shift of the hidden-covariate estimate is bounded by the formal $(R_{D\sim Z|X}^2,R_{Y\sim Z|D,X}^2)$ evaluated at $k_D=k_Y=1$.

Bayesian version: a prior over confounder strength

Treat $(R_{Y\sim Z|D,X}^2,R_{D\sim Z|X}^2)=(a,b)$ as unknown sensitivity parameters with priors, and $\tau$ as

$$\tau ={\tau }_{res}-\text{sign}\cdot \sqrt{\frac{ab}{1-b}}\sqrt{df}{\text{se}}_{res}.$$

Given a posterior ${\tau }_{res}\sim N({\hat{\tau }}_{res},{\text{se}}_{res}^2)$ (or a full Bayesian regression’s posterior), the induced posterior for $\tau$ is a prior-dependent sensitivity analysis. Because $(a,b)$ are not identified by the data, their posterior equals their prior; only the resulting distribution of $\tau$ changes. This is the logic (and the honesty requirement) of the “unidentified parameters” discussion in Franks 2019 - Flexible Sensitivity Analysis Without Observable Implications - Overview.

# PyMC 5 sketch (untested): prior over confounder strength -> posterior over tau.
import numpy as np, pymc as pm
tau_hat, se, df = 0.097, 0.023, 783
with pm.Model() as m:
    tau_res = pm.Normal("tau_res", tau_hat, se)             # stand-in for regression posterior
    # priors encode "no stronger than k x Female": Beta mass below the bound, here (0.12, 0.01)
    r2_y = pm.Beta("r2_y", 2, 14)                           # mean ~ 0.125
    r2_d = pm.Beta("r2_d", 1.5, 148)                        # mean ~ 0.01
    direction = pm.Bernoulli("harmful", 0.5)                # does Z hurt or help the estimate?
    bias = pm.math.sqrt(r2_y*r2_d/(1-r2_d))*np.sqrt(df)*se
    tau = pm.Deterministic("tau", tau_res - (2*direction-1)*bias)
    pm.Deterministic("prob_tau_gt0", tau > 0)
    idata = pm.sample(2000, tune=1000, chains=4, random_seed=0)

Marketing reading

• Ad lift / propensity estimates (E101-E103): the omitted “hidden targeting” variable is a confounder of exposure and conversion; partial $R^2$ values answer “how much of the outcome and of delivery would targeting have to explain”.
• MMM: omitted drivers (seasonality shifts, distribution, promotions) of both spend and sales; the multi-confounder bound summarises them in one number. Experiments (lift tests) provide the benchmark for how large $R_{Y\sim Z|D,X}^2$ can plausibly be, similar in spirit to Gordon et al 2019 - Sensitivity to Unobservables and Bias-Reduction Ratio.

Examples

See the simulation above and the Darfur reproduction in Cinelli Hazlett 2020 - Darfur Application and Reporting.

Connections

• Propensity Score in Bayesian CI: Bayesian propensity analyses where $U$ can be given a prior.
• Sensitivity Analysis in Observational Studies: E-value, Rosenbaum bounds, copula approaches.
• Bayesian Estimation and Priors for MMM: priors that act as sensitivity assumptions.

See Also

Cinelli Hazlett 2020 - Making Sense of Sensitivity - Overview, Cinelli Hazlett 2020 - Informal Benchmarking Pitfalls and Related Methods