Uncovering Causal Estimates from Non-Experimental Data

Matthew Reda Causal inference

Randomize when you can — but most of the questions a measurement team actually faces (did the campaign move sales, did the price change cause the churn, what did the policy do) arrive after the fact, with no control group in sight. A century of statistics and econometrics has produced a toolkit for exactly this situation, and every tool in it purchases its causal claim with a different assumption. This post is a working tour: what each strategy assumes, when it breaks, and how to choose among them.

The strategies below look very different on the surface — reweighting individuals, differencing panels, exploiting eligibility cutoffs, constructing synthetic states — but they all attack one underlying problem, and naming it first makes the menu coherent.

The Selection Problem

In the potential-outcomes framework (Rubin, 1974), each unit \( i \) carries two outcomes — \( Y_{1i} \) if treated, \( Y_{0i} \) if not — and we observe only one of them: \( Y_i = Y_{0i} + (Y_{1i} - Y_{0i})D_i \). The naive move is to compare treated and untreated group means. That comparison decomposes as

$$ \underbrace{E[Y_i \mid D_i = 1] - E[Y_i \mid D_i = 0]}_{\text{naive comparison}} \;=\; \underbrace{E[Y_{1i} - Y_{0i} \mid D_i = 1]}_{\text{ATT}} \;+\; \underbrace{E[Y_{0i} \mid D_i = 1] - E[Y_{0i} \mid D_i = 0]}_{\text{selection bias}}. $$

The first term is the average treatment effect on the treated — the quantity we want. The second is selection bias: the systematic difference in baseline outcomes between the groups. People who take the treatment are rarely like people who don't, and the naive comparison mixes that pre-existing difference into the estimate (Angrist & Pischke, 2009). Randomization solves the problem by force, making \( D_i \) independent of the potential outcomes so the second term is exactly zero. Every strategy below is a different way of zeroing it without randomizing. The potential-outcomes machinery itself, and its structural-causal-model counterpart, are developed at more length in the causal inference guide.

Naive comparison vs. back-door adjustment

The true effect is fixed at τ = 1.0. A confounder Z pushes units into treatment and lifts their baseline outcome, so the naive treated-minus-control gap carries the selection-bias term. Drag the confounder’s strength: the naive estimate marches away from the truth, while adjusting for Z holds the estimate on target — as long as Z is observed and the model is right.

0.70
Naive estimate—
Adjusted estimate—
True effect τ1.00
Naive bias—

Naive gap = τ + selection bias (proportional to confounder strength); the back-door adjustment Σz P(Y|X,z)P(z) subtracts it. The adjusted line stays flat on τ only because Z is measured here — an unmeasured confounder leaves the naive curve with nothing to close it.

Selection on Observables

The most direct route asserts that treatment looks random once you condition on the right observed covariates.

Definition: Conditional Independence Assumption (CIA)

Treatment is independent of the potential outcomes given covariates \( X_i \):

$$ (Y_{1i}, Y_{0i}) \;\perp\!\!\!\perp\; D_i \mid X_i, $$

together with overlap: \( 0 \lt P(D_i = 1 \mid X_i) \lt 1 \) for every covariate profile that occurs. Also called unconfoundedness or ignorability. It says that within a stratum of \( X \), who got treated is as good as a coin flip.

Under the CIA, several estimators recover the effect. Regression adjustment models the outcome in each arm and averages the difference in fitted values, \( \hat\tau = N^{-1}\sum_i [\hat\mu_1(X_i) - \hat\mu_0(X_i)] \). Propensity-score methods compress the covariates into a single score \( e(X_i) = P(D_i = 1 \mid X_i) \) and match or stratify on it (Rosenbaum & Rubin, 1983a). Inverse probability weighting (IPW) reweights units by \( 1/e(X_i) \) or \( 1/(1 - e(X_i)) \) so the weighted groups have balanced covariates. Doubly-robust estimators combine an outcome model with a propensity model and remain consistent if either one is correctly specified — a genuinely useful insurance policy, sketched in the deep dive below.

Deep diveWhy the doubly-robust estimator forgives one wrong model

The augmented IPW estimator starts from the regression prediction and corrects it with a weighted residual:

$$ \hat\tau_{\mathrm{DR}} = \frac{1}{N}\sum_{i=1}^{N}\left[ \hat\mu_1(X_i) - \hat\mu_0(X_i) + \frac{D_i\,(Y_i - \hat\mu_1(X_i))}{\hat e(X_i)} - \frac{(1 - D_i)\,(Y_i - \hat\mu_0(X_i))}{1 - \hat e(X_i)} \right]. $$

Its bias, worked out term by term, is an expectation of a product of the two model errors — schematically \( E\!\left[ \big(e(X) - \hat e(X)\big)\big(\mu(X) - \hat\mu(X)\big) \big/ \hat e(X) \right] \). If the outcome model is right, the residual corrections have mean zero within each stratum and the weighting error doesn't matter; if the propensity model is right, the weighted residuals correct whatever the outcome model got wrong. Either factor hitting zero kills the whole product. The price: when both models are misspecified, the errors multiply, and extreme propensity estimates near 0 or 1 blow up the weights — overlap is not a formality.

⚠️ Bad controls

Condition only on pre-treatment variables. A covariate that is itself affected by treatment either blocks part of the causal path you are trying to measure or opens a collider path and manufactures bias out of nothing. "Throw every column into the regression" is not a conservative default — it is a common way to turn an unbiased estimate into a biased one.

DAGs and the Backdoor Criterion

Which variables belong in the adjustment set? The CIA as stated gives no guidance. Pearl's directed acyclic graphs replace the vague instruction "control for confounders" with a checkable graphical rule (Pearl, 2009). Three junction types govern everything:

The fork X ← Z → Y. The observed confounder Z (amber) opens a back-door path that biases the naive X → Y comparison; conditioning on Z (amber dashed edges) blocks it, and the adjustment formula recovers the causal effect.

A set \( Z \) satisfying the backdoor criterion — it blocks every path from treatment into the outcome that enters the treatment through an arrow pointing at it, without touching mediators or colliders — licenses the adjustment formula

$$ P(Y \mid \mathrm{do}(X = x)) \;=\; \sum_z P(Y \mid X = x, Z = z)\, P(Z = z). $$

Just as valuable is the negative verdict: when no valid adjustment set exists among the variables you observe, the graph says so explicitly, and the honest conclusion is that selection-on-observables cannot work with your data. That is the cue to reach for one of the designs below, each of which imports exogenous variation from somewhere else.

Instrumental Variables

When a confounder is unobserved, no amount of conditioning helps. Instrumental variables replace "we measured every confounder" with a different bargain: find a variable \( z_i \) that moves treatment but has no other route to the outcome. Two requirements do all the work: relevance — the instrument actually shifts treatment, \( \mathrm{Cov}(D_i, z_i) \neq 0 \) — and the exclusion restriction — the instrument affects the outcome only through treatment. Relevance is testable; exclusion is an argument you must win on substantive grounds. With a binary instrument, the estimand is the Wald ratio:

$$ \hat\rho_{\mathrm{IV}} \;=\; \frac{E[Y_i \mid z_i = 1] - E[Y_i \mid z_i = 0]}{E[D_i \mid z_i = 1] - E[D_i \mid z_i = 0]} \;=\; \frac{\text{reduced form}}{\text{first stage}}, $$

and two-stage least squares generalizes it: regress treatment on the instrument, then regress the outcome on the fitted treatment. The classics illustrate what a credible instrument looks like — the Vietnam draft lottery for military service (Angrist, 1990), quarter of birth for years of schooling (Angrist & Krueger, 1991), Maimonides' rule for class size (Angrist & Lavy, 1999). Each is a rule or lottery that nobody chose in anticipation of the outcome.

💡 IV estimates LATE, not ATE

When treatment effects vary across people, IV does not recover the population average effect. It identifies the local average treatment effect — the effect for compliers, the units whose treatment status the instrument actually changed (Imbens & Angrist, 1994). Before running 2SLS, ask whether the complier subpopulation is the one your decision is about. A clean instrument answering the wrong population's question is still the wrong answer.

Difference-in-Differences

Panel data — units observed before and after some of them are treated — buys a different assumption. Difference the treated group's before/after change against the control group's, and anything constant within a group over time cancels out:

$$ \hat\beta_{DD} \;=\; \big(\bar Y_{\mathrm{treat,\,after}} - \bar Y_{\mathrm{treat,\,before}}\big) - \big(\bar Y_{\mathrm{control,\,after}} - \bar Y_{\mathrm{control,\,before}}\big), $$

equivalently a regression \( Y_{ist} = \gamma_s + \lambda_t + \beta D_{st} + \varepsilon_{ist} \) with group and time fixed effects. The identifying assumption is common trends: \( E[Y_{0ist} \mid s, t] = \gamma_s + \lambda_t \) — the groups may sit at different levels, but absent treatment they would have moved along the same trend. That assumption is untestable in the treatment period but has a testable shadow: plot the pre-treatment paths, and include leads of treatment in an event-study specification. Diverging pre-trends are disqualifying. The canonical application is Card and Krueger (1994), who compared fast-food employment in New Jersey and Pennsylvania around New Jersey's minimum-wage increase and found employment did not fall. A Bayesian formulation encodes parallel trends as a model constraint and returns a full posterior over the effect, which makes the uncertainty — and the sensitivity to the trend assumption — explicit rather than buried in a standard error.

Regression Discontinuity

Sometimes a bureaucracy hands you an experiment: treatment is assigned by a rule — a test-score cutoff, an eligibility threshold, a vote-share margin — applied to a continuous running variable. Units just below and just above the cutoff are, in every respect except treatment, nearly identical, so the jump in outcomes at the threshold is a local causal effect. In a sharp design, treatment switches deterministically at the cutoff \( x_0 \), and the limit of the outcome difference across it identifies \( E[Y_{1i} - Y_{0i} \mid x_i = x_0] \). In a fuzzy design, only the probability of treatment jumps, and the estimator is exactly IV with the threshold indicator \( \mathbf{1}(x_i \ge x_0) \) as the instrument — yielding, again, a LATE at the cutoff. Two validity checks are standard: pre-treatment covariates should show no jump at the threshold, and the density of the running variable should be smooth there — a spike just above the cutoff (the McCrary test) is the signature of units manipulating their way in. Lee (2008) used bare-majority U.S. House elections this way to estimate incumbency advantage. The honest caveat mirrors IV's: you learn the effect at the cutoff, and extrapolating far from the threshold is an assumption, not a result.

Synthetic Control

DiD needs enough units for group averages to mean something. When the treated unit is a single state or country, synthetic control builds its counterfactual explicitly: a weighted combination of untreated donor units, \( \hat Y^{N}_{1t} = \sum_j w_j Y_{jt} \) with \( w_j \ge 0 \) and \( \sum_j w_j = 1 \), where the weights are chosen to reproduce the treated unit's pre-treatment trajectory as closely as possible. The effect estimate is the post-treatment gap \( \hat\tau_{1t} = Y_{1t} - \hat Y^{N}_{1t} \). Because there is one treated unit, inference runs by permutation: apply the same procedure to every donor as a placebo and ask whether the real unit's gap is unusually large. In the canonical study, Abadie, Diamond, and Hainmueller (2010) estimated that California's Proposition 99 tobacco program reduced cigarette sales by roughly 25 packs per capita by 2000, with a placebo-based p-value near 0.029. The design's assumption is that a convex combination matching the pre-period continues to be the right counterfactual after it — which fails if the pre-fit is poor or the donors are hit by their own shocks. The generalized synthetic control method (Xu, 2017) unifies DiD and synthetic control under an interactive fixed-effects model, accommodating multiple treated units and time-varying confounding factors.

Metalearners for Heterogeneous Effects

Everything so far targets an average. When the decision needs to know for whom the treatment works — which segments, which markets — the target becomes the conditional average treatment effect, \( \tau(x) = E[Y_{1i} - Y_{0i} \mid X_i = x] \), and machine learning supplies the flexible function approximation. The metalearner taxonomy (Künzel et al., 2019) organizes the options: the S-learner fits a single model with treatment as one more feature (fine when heterogeneity is weak, prone to shrinking the effect away); the T-learner fits separate outcome models per arm and differences them (fine when both arms are large); the X-learner cross-imputes individual effects — use the control-arm model to impute what treated units would have done, and vice versa — then regresses the imputed effects on covariates. The X-learner's advantage is sharpest in unbalanced designs: with a large control group and a small treated group, it estimates the CATE at the fast rate the big arm supports whenever the effect surface is smoother than the outcome surfaces. The crucial caveat: all of this lives inside the CIA. Metalearners buy flexibility in modeling heterogeneity, and nothing against unobserved confounding — no amount of ML sophistication compensates for a confounder you didn't measure.

Time-Series Counterfactuals

When the intervention hits a single series — a national campaign launch, a platform change — and even donors are unavailable, the counterfactual can be extrapolated from the series' own history plus contemporaneous covariates. Bayesian structural time-series (the CausalImpact approach of Brodersen et al., 2015) fits a state-space model to the pre-intervention period — local linear trend, seasonality, and a spike-and-slab regression that selects among candidate control series — then projects the counterfactual path forward. The effect is the observed-minus-predicted gap, delivered as a posterior over pointwise, cumulative, and running-average impacts. The load-bearing assumptions are that the control series are not themselves touched by the intervention and that their pre-period relationship to the outcome persists — a national media launch that also lifts your "control" search category violates the first quietly.

Sensitivity Analysis

Every strategy above rests on at least one assumption the data cannot verify: unconfoundedness, exclusion, common trends, continuity, pre-period stability. Sensitivity analysis converts "assume it holds" into "here is how wrong it could be before the conclusion changes."

Definition: E-value

The E-value is the minimum strength of association, on the risk-ratio scale, that an unmeasured confounder would need with both treatment and outcome to fully explain away the observed effect (VanderWeele & Ding, 2017). A large E-value means only an implausibly strong hidden confounder could account for the finding; an E-value barely above the observed association means the causal reading is fragile.

The idea predates the label: Rosenbaum and Rubin (1983b) modeled a hidden binary confounder and traced how the estimate moves as its assumed strength grows, and copula-based approaches (Franks, D'Amour & Feller, 2020) parameterize the non-identified dependence directly, separating what the data constrain from what only assumptions do. Whatever the flavor, the practice belongs in the report, not the appendix. This framework applies the same discipline to its own outputs — an MMM is, at heart, a selection-on-observables design, and the identification contract states each assumption it rests on and whether it is testable.

Choosing a Strategy

The decision guide is shorter than the menu suggests. Draw the causal graph first — it tells you whether your observables plausibly close every backdoor path. If they do, adjustment, weighting, or a doubly-robust estimator does the job, and metalearners extend it to heterogeneous effects. If an important confounder is unobserved, look for imported exogeneity: an instrument if a lottery or rule moves treatment; a threshold if assignment follows a cutoff; panel differencing if the groups share trends; a synthetic or extrapolated counterfactual if the treated unit is one aggregate series. Then match the estimand to the decision — IV and RD deliver local effects for compliers and near-threshold units, DiD and synthetic control deliver effects on the treated — and pressure-test whichever untestable assumption you bought. The same logic drives how this framework treats media measurement: observational MMM as the always-on adjustment estimate, experiments layered on top where the causal reading is weakest.

Strategy Selection bias handled by Key assumption Estimand Typical failure
Regression / IPW / doubly-robust Conditioning on covariates All confounders observed, overlap ATE / ATT Unmeasured confounder; bad controls
DAG backdoor adjustment Graph-derived adjustment set The causal graph is correct ATE Wrong or incomplete graph
Instrumental variables / 2SLS Exogenous instrument Exclusion restriction LATE (compliers) Weak instrument; side channels to outcome
Difference-in-differences Panel differencing Common trends ATT Diverging pre-trends; anticipation
Regression discontinuity Threshold comparison Continuity at the cutoff LATE at the cutoff Manipulation of the running variable
Synthetic control Constructed donor counterfactual Pre-period match extrapolates ATT (single aggregate unit) Poor pre-fit; donor shocks
Metalearners (S/T/X) ML under the CIA Unconfoundedness + overlap CATE Same as CIA — confounding is untouched
BSTS / CausalImpact Time-series extrapolation Stable, unaffected control series ATT (over time) Spillover into controls; structural break

Takeaways

  • Every quasi-experimental method targets the same selection-bias term; each replaces randomization with a different untestable assumption. Choose by which assumption you can defend, not by which method is fashionable.
  • Draw the DAG before choosing an estimator — it decides whether adjustment can work at all and what a valid adjustment set is.
  • Know the estimand you are buying: IV and RD deliver local effects for specific subpopulations; DiD and synthetic control deliver effects on the treated.
  • Machine learning helps with effect heterogeneity, never with unmeasured confounding.
  • Sensitivity analysis is part of the estimate. Report E-values or bounds; a causal claim that collapses under a modest hidden confounder should not be presented as causal.

References