Causal Inference 08: Designing the Next Experiment

The model stops being the patient and becomes the surgeon's assistant. Rung 5 designed a test from the raw geo panel alone — deliberately, because an experiment's validity must never depend on the model it will later repair. But its efficiency should absolutely use the model. This page mirrors nbs/causal/causal_08_designing_the_next_experiment.ipynb: Veranda Home's fitted posterior (the cached rung-2 fit — designing costs posterior passes, not refits) prices two candidate tests from notebook 07's to-do list — TV, the stubborn residual under-read, and Print, the twin that inherited the pair-sum bias — and hands them opposite verdicts.

1 — Candidate one: a TV flighting test

Veranda's data is national, so the recommended design family is budget-neutral flighting: alternate TV spend between a high and a low level in randomized blocks — same annual budget, honest identification from the contrast. The design engine's default for this dataset: national_flighting, 12 weeks, spend levels [0.5, 1.5] of plan, budget-neutral — with a minimum detectable effect of 1.46 ROAS (the framework's power rule, \( \mathrm{MDE} = 2.8\,\hat\sigma_{\text{design}} \), at 80% power and 5% two-sided size).

from mmm_framework.planning.design import design_experiment
from mmm_framework.planning.design_anchor import model_anchored_effect
from mmm_framework.planning.opportunity_cost import compute_opportunity_cost

design = design_experiment("veranda_weekly.csv", "Sales", "TV")
# -> national_flighting | 12 weeks | levels [0.5, 1.5] | MDE 1.46 ROAS

anchor = model_anchored_effect(mmm, design, max_draws=60, random_seed=42)
# -> the readout the test would produce: median 0.39, 90% HDI [0.25, 0.49]

cost = compute_opportunity_cost(mmm, design, max_draws=60, random_seed=42,
                                kpi_kind="revenue")
# -> spend_at_risk 0.0, P(KPI loss over the window) 0%

2 — What would we actually see? The model-anchored readout

The design's MDE says what is detectable. The posterior says what is expected: push the candidate schedule through the fitted response curves — a spend perturbation on the test window only, not the global curve — and read the distribution of incremental ROAS the test would measure. For the default TV design: median 0.39, 90% HDI [0.25, 0.49]. The expected readout sits far below the 1.46 MDE — as specified, this test would probably come back "inconclusive". Better to know that before spending a quarter on it.

Measured — nbs/artifacts/causal_08_design.json. What the default 12-week TV flighting would read (median 0.39, 90% HDI [0.25, 0.49]) against what it could detect (MDE 1.46). The optimizer's job is to pull the detection bar down to the readout — its recommended design (section 4) lands at 0.63.

3 — What does learning cost? (Almost nothing, if you design it that way)

The posterior counterfactual of running the schedule vs business-as-usual prices the test's short-term risk: dollars at risk $0, P(net KPI loss over the window) 0% — the posterior actually expects a small positive KPI delta (+92 units) over the window, since the flighting spends the same dollars on a curve that is not linear. One honest footnote: randomized blocks against a varying baseline round to approximately neutral — the notebook's window nets to +2.2% of annual TV spend, not exactly zero. Budget-neutral flighting is still the free lunch of measurement: the same media plan, rearranged into an instrument.

4 — The Pareto choice

Candidate designs trade four objectives — MDE ↓, power shortfall ↓, short-term cost ↓, duration ↓ — and no single design wins all four. suggest_experiment sweeps the design space (intensity × duration), keeps the non-dominated frontier, and recommends the knee among powered designs. The sweep over ±30–100% intensity and 6–16 weeks:

Measured — the TV candidate grid from the notebook's seeded run. The recommended design: ±21% intensity × 16 weeks → MDE 0.63 ROAS, power shortfall 0.02. Note what the optimizer traded: a gentler contrast than the default's ±50%, held for longer — more weeks of a smaller, cheaper wiggle beat a short violent one.

from mmm_framework.planning.experiment_optimizer import (
    suggest_experiment, cooldown_weeks)
from mmm_framework.agents.model_ops import identify_structural_parameters

sug = suggest_experiment(mmm, "veranda_weekly.csv", "Sales", "TV",
                         duration_min=6, duration_max=16,
                         intensity_min=30.0, intensity_max=100.0,
                         max_draws=40, random_seed=42)
rec = sug["recommended"]        # ±21% x 16 weeks -> MDE 0.63, shortfall 0.02

gap = cooldown_weeks(mmm, "TV") # {'alpha': 0.46, ..., 'cooldown_weeks': 4}

ident = identify_structural_parameters(mmm, dataset_path="veranda_weekly.csv",
                                       kpi="Sales", channel="TV", duration=24)

Measured — the design frontier for both candidates, on one axis. TV's candidates descend toward the expected readout as duration grows; the big marker is the recommended, powered pick. Print's candidates (section 5) never get anywhere near the line — the best national Print design's MDE is 3.9 ROAS, an order of magnitude above any plausible readout.

5 — The engine's most valuable answer: "no"

Now Print — the channel notebook 07 flagged as the top re-test priority. Print's national spend is a rounding error of TV's, so a national budget-neutral flighting contrast moves almost no dollars, and no candidate in the sweep comes close to powered: MDEs run 6.4 → 4.7 → 3.9 ROAS across 6/11/16 weeks, with power shortfalls stuck at 0.70–0.73 (best available: MDE 4.7, shortfall 0.71). The honest recommendation is not "run the least-bad one" — it is change the mode: a matched-pair geo holdout (rung 5's machinery) concentrates the contrast in a subset of markets, or measures the Radio+Print pair jointly. The design engine just saved a quarter's Print budget from buying an "inconclusive". A test you shouldn't run is a result too.

6 — Cool-downs: carryover contaminates the calendar

Two tests on the same channel back-to-back are not independent: the first test's adstock is still paying out during the second's window. The fitted carryover implies a washout period per channel — the minimum gap before the next test on that channel, and the minimum block length inside a flighting schedule:

Measured — cooldown_weeks reads each channel's fitted adstock and returns the smallest gap after which the residual carryover falls below 5%. These numbers become scheduling constraints in notebook 09's calendar.

7 — Identifying the curve, not just the effect

Everything above measures an average effect. Budget allocation needs the curve — the saturation shape and the carryover — and rung 2 showed those parameters barely move under observational data. A multi-level flighting schedule (≥3 spend levels) is the design that can, in principle, trace the curve. identify_structural_parameters computes, per structural parameter, the detection power and the expected prior→posterior contraction the schedule would buy — Fisher-information math against the fitted posterior, with a fail-closed self-check that the forward op mirrors the graph.

Measured — expected prior→posterior contraction per structural parameter (log scale), 12-week vs 24-week 3-level TV schedules. Detection power barely moves (β: 0.830 → 0.833; saturation λ: 0.678 → 0.682; adstock α: ≈1.00 at both), but contraction is roughly 10× larger at 24 weeks — and only the 24-week commitment crosses the engine's identification bar: identifies_anything = False at 12w, True at 24w. Curvature knowledge is expensive; buy it deliberately or don't claim it.

8 — The design checklist this rung leaves behind