Causal Inference 07: Many Experiments

Evidence composes — and a portfolio needs bookkeeping. Notebook 06 repaired one channel with one experiment. A real measurement program accumulates a portfolio: readouts of different channels, windows, designs and vintages — plus the occasional inherited "study" of dubious provenance. This page mirrors nbs/causal/causal_07_many_experiments.ipynb: three very different pieces of evidence folded into Veranda Home's realistic seven-channel model, each one graded against the sealed key. Three lessons: experiments compose; a test on one twin of a collinear pair resolves the split but not the sum; and the portfolio's most dangerous member is the confident wrong one.

1 — The baseline and its honest ignorance

The realistic seven-channel world's causal fit, reloaded from cache. The ridge: Radio and Print flighted in lockstep (spend correlation ≈ 0.996), so the data pins their combined effect while each channel's own interval stays enormous — 90% interval widths of 3,871 (Radio) and 4,034 (Print) KPI units, against true contributions of 3,301 and 2,097. Points: Radio −36%, Print −16%, both covered by their wide intervals. And a detail that matters later: the pair-sum itself is under-read — combined estimate 3,853 vs true 5,398, −29%. The observational data got the total wrong, not just the split.

2 — Experiment 1: the Radio go-dark

Ten dark weeks for Radio (window 2022-09-26 → 2022-11-28), read with notebook 05's discipline: window ROAS 0.94 ± 0.10 (the world's true window ROAS: 0.94 — a well-run test). Folded in via the likelihood route and re-graded:

Measured — nbs/artifacts/causal_07_portfolio.json. The tested channel is repaired: Radio lands at −10% (from −36%) and its interval collapses by 74%. The twin moves too — Print's interval shrinks 43%, because resolving Radio resolves the split of the pinned pair-sum. But the honest part most case studies omit: Print's point estimate got worse, −16% → −56%. The data had pinned the pair-sum ~30% low; the experiment fixed Radio's share of it, so the whole sum deficit now lands on Print — with more confidence.

3 — Experiment 2: off-panel Display

The Display holdout ran after the modeling window closed — spring 2024, while the panel ends in late 2023. Classic grounds for "sorry, the test doesn't overlap the data, we can't use it." With off-panel calibration you can: declare the test's spend level (eval_spend, per period), duration (eval_periods), and adstock regime (cold_start — the channel ramps from zero in the test markets), and the likelihood attaches to the channel's global response curve evaluated at that spend. No training rows involved, so window overlap is unnecessary.

from mmm_framework.calibration import ExperimentEstimand, ExperimentMeasurement

radio_dark = ExperimentMeasurement(
    channel="Radio", test_period=("2022-09-26", "2022-11-28"),
    value=0.94, se=0.10,
    estimand=ExperimentEstimand.ROAS, distribution="normal",
    name="radio_dark_2022",
)
display_offpanel = ExperimentMeasurement(
    channel="Display", test_period=("2024-03-04", "2024-04-22"),
    value=0.59, se=0.08,
    estimand=ExperimentEstimand.ROAS, distribution="normal",
    eval_spend=45.0, eval_periods=8, eval_units=1,   # $45k/week, 8 weeks, 1 cell
    adstock_state="cold_start",                      # ramps from zero in test markets
    name="display_offpanel_2024",
)
model.add_experiment_calibration([radio_dark, display_offpanel])
results = model.fit(draws=1000, tune=1000)

Measured: at $45k/week (cold start) the world's true off-panel ROAS is 0.56; the simulated readout comes in at 0.59 ± 0.08. After the merge, Display's interval width drops 4,113 → 1,898 (−54%) and its point estimate doesn't degrade (+19% → +16%). The assumption you buy it with, stated out loud: structural stationarity — the response curve (saturation, carryover, effectiveness) is the same in spring 2024 as in the fitted window. Time-varying worlds break this; pricing evidence decay is notebook 09's job.

4 — Experiment 3: the confident wrong certificate

An inherited deck claims Search ROAS 0.95 ± 0.10 ("agency lift study, 2021"). Before merging anything, run the tension check — compare the claimed value against the current posterior for the same estimand, combining both uncertainties:

\[ z \;=\; \frac{v_{\text{claim}} - \mu_{\text{model}}} {\sqrt{\text{SE}_{\text{claim}}^2 + \sigma_{\text{model}}^2}} \;=\; \frac{0.95 - 0.55}{\sqrt{0.10^2 + 0.11^2}} \;\approx\; 2.8 \]

A >2σ gap before merging. Policy: investigate provenance — was the estimand matched? the estimator calibrated? the market representative? Park it in the ledger as disputed. The notebook merges it anyway, as narrator, to price the mistake:

Measured — the bad certificate drags a healthy estimate away from truth: Search relative error +8% → +26%, truth pushed outside the interval — and the interval tightens while it happens (width 2,752 → 1,741). Confidently wrong, by injection. A likelihood commits — including to lies.

5 — The portfolio ledger

Every readout, its vintage, estimand, SE, and what it did to the posterior — the bookkeeping a planning engine consumes. The legacy study is excluded from the final model; it stays in the ledger as disputed.

Measured — what each honest piece of evidence bought, as 90% interval width reduction: Radio −74% (its own go-dark), Print −43% (spillover — resolving the twin resolves the split), Display −54% (the off-panel holdout). Each merged readout repaired its target and tightened its neighborhood.

6 — Portfolio discipline