Causal Inference 05: Measuring One Experiment

Reading a lift test honestly is a skill, not a formality. Notebooks 00–04 pushed observational modeling as far as it goes and hit the same wall every time — a residual the data cannot settle, culminating in the latent-confounder wall of notebook 04. The cure is an experiment: randomization severs every back-door by construction. But an experiment only buys truth if it's read honestly — and the field is littered with lift tests that weren't. This page mirrors nbs/causal/causal_05_measuring_one_experiment.ipynb: Veranda Home, a national home & garden retailer, has ten test markets and wants to measure Search — the channel every observational rung kept over-crediting. Before a single dollar moves, three pieces of homework, all on pre-period data, all pure pandas — no MCMC, no MMM.

The three failure modes this page is built to prevent: treatment markets picked by convenience (the big ones), not by matching; significance declared with a t-test whose assumptions the data violates; and estimators chosen after seeing the data, because one of them "found" lift. Every claim below is graded — the synthetic world's answer key stays sealed until the notebook's end.

1 — Ten markets, one question

The brand at DMA grain (the geo_heterogeneous world from src/mmm_framework/synth/): 10 markets × 130 weeks of weekly sales, each market with its own baseline, budget share, and — because this world is honest about geography — its own true regional effectiveness. The question: what is Search's incremental ROAS, really?

2 — Design: matched pairs, then a coin flip

geo_lift_design builds the treatment geometry a referee would accept:

• Matched pairs — markets paired by pre-period KPI similarity (residualized against spend), so each treated market carries its own counterfactual twin;
• Randomization inside each pair — a coin flip decides which twin gets treated, severing any lingering association between assignment and demand;
• Placebo-calibrated power — the design's minimum detectable effect comes from re-running the analysis on 123 pre-period placebo windows, not from a textbook variance formula.

The measured design: 5 matched randomized pairs (Boston–Atlanta, Dallas–Minneapolis, Denver–Chicago, Seattle–Portland, Miami–Phoenix), an 8-week holdout, design MDE ≈ 0.27 ROAS at a readout SE of 0.095.

from mmm_framework.planning.design import design_options, geo_lift_design
from mmm_framework.planning.simulation import (
    build_sim_panel, build_geo_assignment, run_aa_simulation,
    run_ab_simulation, fixed_lift_injector, methodology_leaderboard,
    pooled_did_estimator, per_pair_did_estimator, regadj_geo_estimator,
)

# 1. Design: matched randomized pairs + placebo-calibrated MDE
opts = design_options("veranda_geo.csv", "Sales", "Search")
design = geo_lift_design("veranda_geo.csv", "Sales", "Search",
                         design="holdout", duration=8, seed=42)

# 2. A/A: each estimator's REAL false-positive rate, measured on null windows
panel = build_sim_panel("veranda_geo.csv", "Sales", "Search")
assignment = build_geo_assignment(panel, seed=42)
aa = run_aa_simulation(panel, regadj_geo_estimator, assignment,
                       duration=8, seed=42, name="regression-adjusted")

# 3. A/B: inject known lifts into real history, measure empirical power
injector = fixed_lift_injector(panel, assignment, duration=8,
                               pct_of_baseline=0.10)
ab = run_ab_simulation(panel, pooled_did_estimator, assignment, injector,
                       duration=8, aa_result=aa, seed=42, name="pooled DiD")

# 4. One honest recommendation: validity first, then power, then cost
lb = methodology_leaderboard("veranda_geo.csv", "Sales", "Search",
                             duration=8, seed=42)

3 — The A/A discipline: test your test on nothing

Before trusting any estimator with a real experiment, run it on null data: 115 rolling windows in which no experiment happened, asking how often it cries "significant!" anyway. Weekly KPI series are autocorrelated — adjacent weeks share trend, season, and demand shocks — so the effective sample size is far below the nominal one, and naive standard errors are too small. Three candidate estimators, same frozen assignment. The DiD estimand for a treated–control pair:

\[ \hat{\tau} \;=\; \bigl(\bar{Y}_{T,\text{test}} - \bar{Y}_{T,\text{pre}}\bigr) \;-\; \bigl(\bar{Y}_{C,\text{test}} - \bar{Y}_{C,\text{pre}}\bigr) \]

Measured — nbs/artifacts/causal_05_methodology.json, 115 null windows per estimator. Naive t-test false-positive rates: pooled DiD 0.9%, per-pair DiD 8.7%, regression-adjusted 26.1% — the most "sophisticated" of the three is the one that lies, at more than five times the nominal 5%. The design-calibrated critical value (the empirical 95th percentile of |placebo estimates|) restores 5.2% for every estimator, by construction.

4 — Power against injected truth

The A/A test protects against false positives; the A/B simulation measures power against false negatives. Take real history, inject a known lift into the treated markets across a grid of effect sizes, and count how often each (calibrated) estimator detects it. Because the injected lift is known, this is the estimator graded against a sealed key — the same epistemics as the rest of the series, at simulation speed.

Measured — injected-truth power curves for the two calibrated DiD estimators (their curves coincide on this panel). At a quarter of the expected effect, power is 69%; from half the expected effect upward, both detect the planted lift every time. At the expected effect size — a 10%-of-baseline lift — power = 100% for both, comfortably past the 80% bar.

5 — The leaderboard: one honest recommendation

methodology_leaderboard runs the full A/A + A/B battery for every estimator the data supports, on one frozen assignment, and ranks by validity first (honest false-positive rate), then power, then cost. The data supports three designs (geo_lift, matched_market_did, national_flighting; recommended: geo_lift), and the estimator ranking lands:

Two caveats the leaderboard itself reports: sliding placebo windows overlap, so the rates are indicative rather than independent Bernoulli trials; and the false-positive rate is measured against the analytic decision rule — the calibrated critical value is what restores nominal size.

6 — What a pre-registration now says

The homework above compresses to five lines, written before the test:

Every choice was made on pre-period data — nothing was picked because it "found lift". That discipline is what makes the number an experiment produces worth folding into a model, which is exactly what notebook 06 does next.