Causal Inference 09: The Measurement Program

A year of learning, budgeted like the investment it is. Notebook 08 chose one experiment well. A measurement program is a different problem: seven channels, finite testing capacity, cool-downs, seasons — and evidence that quietly expires while you're busy elsewhere. This page mirrors nbs/causal/causal_09_planning_the_measurement_series.ipynb: the framework prices every candidate test in two currencies — EIG (expected information gain, nats: what a feasible test would teach) and EVOI (expected value of information, KPI units: what knowing is worth to the budget decision) — under one ceiling, EVPI, the value of perfect information about everything. A channel can be wildly uncertain and still worth nothing to test — if no plausible answer would change the allocation.

1 — The priority grid

Inputs: the current posterior (the series' rung-2 fit on Veranda's seven-channel history) and the evidence ledger from notebook 07 — Radio's go-dark readout window ended 2022-11-28, Display's off-panel holdout ended 2024-04-22. Planning date: 2024-07-01. Per channel, the engine simulates a feasible national-pulse test, computes EIG against the channel's current ROI posterior (decayed where evidence exists), and EVOI by re-optimizing the budget across simulated readouts:

Measured — nbs/artifacts/causal_09_program.json (grid order = priority order). Print — the channel whose split the Radio test resolved onto shaky ground in notebook 07 — is the program's top priority, and the engine found it independently of rung 7's narrative. Note Video: one of the widest ROI posteriors on the board (2.12 ± 0.39), a respectable EIG of 0.25 nats — and an EVOI of 0.6. Knowing more about Video changes nothing the budget would do.

from mmm_framework.planning.priority import compute_experiment_priorities
from mmm_framework.planning.eig import channel_half_life, decayed_sigma, reexperiment_due

grid, portfolio = compute_experiment_priorities(
    mmm,
    evidence={"Radio": {"end_date": "2022-11-28"},
              "Display": {"end_date": "2024-04-22"}},
    as_of="2024-07-01", max_draws=150, random_seed=42,
)
# portfolio["evpi"] -> 1,624 KPI units (4.4% of the optimal-budget value)

due, fresh_eig = reexperiment_due(0.10, 84, channel_half_life("Radio"),
                                  sigma_exp=0.10)
# -> a fresh Radio test would buy 0.70 nats: retest due

Measured — the measurement quadrant. Bubble size = spend share, color = the engine's verdict. The diagnostic corners: Print and Radio (high both — test now), Video (informative but worthless — deprioritize), TV (neither — the tight posterior of the biggest spend line). EIG says what a test would teach; EVOI says whether anyone should pay for the lesson.

2 — An honest asterisk the grid needs

3 — EVPI: the ceiling on the learning budget

Summing what perfect information about everything would be worth gives the program its budget ceiling: EVPI = 1,624 KPI units, against an optimal-budget value under current knowledge of 36,638 — the program's rational scale is 4.4% of the value it protects. Any measurement plan costing more than EVPI is, by construction, destroying value — no matter how scientific it feels. And any plan costing far less is probably leaving correctable bias in the allocation.

4 — Evidence decays

A readout is a photograph, not a covenant. Markets drift — creative wears out, auctions reprice, competitors move — so the information an experiment bought erodes on a half-life. The framework re-inflates the posterior sd as \( \sigma^2(t) = \sigma_0^2 \, e^{\lambda t} \) with \( \lambda = \ln 2 / h \), and re-flags a channel for testing when a new feasible test would again clear the EIG bar. Radio (half-life 52 weeks, tested 84 weeks before the planning date — its variance has roughly tripled since) has decayed enough that a fresh test would buy 0.70 nats — over the bar, retest due. Display (half-life 26 weeks, tested 10 weeks ago) is still fresh.

The framework's decay law, drawn exactly — decayed_sigma from planning/eig.py with each channel's default half-life (channel_half_life: Radio 52w, Display 26w) and the notebook's post-experiment sds (Radio 0.10, Display 0.15). The dotted verticals mark where each channel actually stands at the 2024-07-01 planning date. "We tested that in 2022" is a date, not an excuse.

5 — The calendar

Scores in hand, the calendar is a constrained packing problem: a greedy EVOI-per-week schedule under the program's real constraints — one national test at a time, per-channel cool-downs (rung 8), a Q4 freeze (no experiments in the holiday quarter, program weeks 13–26), re-tests when decay triggers. The TV validation test is scheduled by policy, not EVOI — it's pinned to the last slot, using rung 8's 16-week design.

Measured — nbs/artifacts/causal_09_program.json (solid = test, faint = cool-down, shaded band = Q4 freeze). The Print geo holdout starts immediately; the freeze pushes the Radio re-test to week 26; Display and Social follow. The EVOI-track tests finish by week 56; the policy-pinned TV validation runs weeks 57–73, spilling into the next cycle — cool-downs and a frozen quarter stretch five tests across substantially more calendar than the naive sum of their durations. Calendar is the scarcest thing the program has.

6 — The program's memory

Every fitted model in the loop persists a per-run metrics snapshot — ROI posteriors, response curves, the EIG/EVOI portfolio, allocation gaps — so next quarter's planning starts from recorded state, not from whoever remembers the last deck. compute_run_metrics is the same snapshot the platform stores per fit:

from mmm_framework.planning.history import compute_run_metrics

snapshot = compute_run_metrics(mmm, max_draws=100, random_seed=42)
# keys: approximate, channels, fit_method, n_draws, portfolio,
#       response_curves, schema_version   (7 channels tracked; schema v2;
#       fit method nuts, approximate False)

7 — The program's operating rules