Causal Inference 10: The Closed Loop

The capstone. Every rung of the ladder, welded into one operating loop and run three full cycles on the world where this series began — unobserved_confounding, the history whose dashboard fooled the naive read in notebook 00 and whose back-door no control variable could fully close in notebook 01. Per cycle: fit on everything known so far → prioritize with the EIG × EVOI grid → design the test window → measure (the readout is generated from the sealed truth plus sampling noise at the design's SE — an honest experiment, never an oracle) → calibrate via the likelihood route, and go around again. The narrator grades every cycle against the key; the analyst inside the loop never sees it. This page mirrors nbs/causal/causal_10_the_closed_loop.ipynb.

1 — The sealed benchmarks

Before the loop starts, the narrator computes three numbers the analyst never sees: the true media value of the historical allocation (14,885 KPI units), the true value of the best possible allocation, grid-searched on the true response curves (15,122), and the gap between them — 238 KPI units, the most any reallocation of the same total budget could win. That headroom is the yardstick every "optimal" plan below is graded against.

2 — The loop

Three cycles. Each round's test is a 10-week go-dark on the grid's top untested channel, in the next free window — staggered, clear of the previous test's cool-down (rungs 5 & 8 discipline, compressed). Readout SE: 0.09 — rung-5-grade precision.

Measured — the notebook's loop log. Note round 2: Display tops the ranking again (its readout is a year old in loop time and the posterior still leans on it), but the one-test-per-channel-per-year rule sends the budget to the top untested channel, Search. Prioritization is a policy, not a reflex.

from mmm_framework.planning.priority import compute_experiment_priorities
from mmm_framework.planning.budget import optimize_budget

# every round, on the current calibrated fit:
grid, portfolio = compute_experiment_priorities(mmm, max_draws=120, random_seed=42)
plan = optimize_budget(mmm, total_budget=annual_media_budget, random_seed=42)
# portfolio["evpi"] is the loop's fuel gauge; the narrator grades `plan`
# against the sealed true response curves the analyst never sees

3 — Unsealing: does the posterior converge on the truth?

The analyst saw posteriors tighten. The narrator can say something stronger: they tightened toward the right answers. Mean |relative attribution error| against the sealed key falls every round — 31.5% → 25.5% → 20.9% → 17.6% — as each go-dark readout enters the model as an in-graph likelihood term (rung 6's route) and the posterior re-negotiates the channels it could never separate from the confounder observationally.

Measured — nbs/artifacts/causal_10_closed_loop.json. Three experiments walk the posterior onto the key: error falls monotonically and lands at a bit over half its starting level. Not zero — three go-darks measured three window responses, and the residual lives in the parts of the curves no test has touched yet.

4 — Unsealing: do the decisions get better?

Parameters are means, not ends. Each round the analyst also asked the model for its optimal reallocation of the same total budget. The narrator evaluates every one of those plans on the true response curves — including the plan the round-0 model would have shipped.

Measured — the round-0 trap: reallocating on the confounded model would have shipped a plan worse than doing nothing — true value 13,962 vs the historical plan's 14,885, i.e. −923 KPI units, because the "optimal" plan moves budget toward the over-credited demand chasers. Three experiments later, round 3's plan is worth 14,750: +788 recovered, within 0.9% of the status quo and 68% of the way from the round-0 plan to the true optimum (15,122). Note the shape of the repair: the first readout (Display) barely moves the plan (−923 → −928); the TV and Search readouts do the heavy lifting. The loop's first job wasn't to squeeze out the 238-unit headroom — it was to stop a confident 900-unit mistake.

5 — Unsealing: is the loop self-limiting?

A healthy measurement program spends less on learning as it learns. EVPI — the total remaining value of perfect information, rung 9's ceiling — is the loop's own fuel gauge: 350 → 308 → 222 → 237 KPI units across the three cycles. When EVPI dips below the cost of the next test, the rational program pauses — until decay (rung 9) re-inflates it.

Measured — the value of what's left to learn. The wobble between rounds 2 and 3 (222 → 237) is Monte-Carlo noise on a simulated quantity plus a decision problem that reshapes with each refit; the notebook's asserts gate the trend, not strict monotonicity.

6 — The ladder, climbed

Eleven notebooks ago, Veranda Home's dashboard said everything works and the naive read doubled Social's value. Here is what each rung contributed to the loop that just ran:

7 — Three habits are the whole method

Where the loop re-enters: evidence decays, the world drifts, and next quarter's EVPI creeps back up. The loop doesn't end — that's the point. For the platform's automated version of this cycle — the T₀–T₅ measurement loop with experiment registries, calibration tracking and report generation — see the platform overview and the Experimental Measurement Lifecycle notebook series in the repository.