Causal Inference 06: Calibrating the Model

One honest experiment meets a confidently confounded posterior. The Search lift test designed in notebook 05 has run: an 8-week go-dark, read with the pre-registered calibrated estimator. The readout lands on the desk of the analyst who owns Veranda Home's national MMM — the model that over-credits Search by +43% and cannot know it. This page mirrors nbs/causal/causal_06_calibrating_the_model.ipynb: two mechanically different routes for folding the experiment into the Bayesian MMM — and they are not equivalent. A prior suggests; a likelihood commits. Then the quietest failure in applied calibration: the same number filed under the wrong estimand, and a six-figure experiment evaporating without a single warning.

1 — The patient: a confidently confounded model

The baseline is the series' rung-1 fit on the unobserved_confounding history (156 weeks, national grain): a latent demand wave drives both Search spend and sales, so the model routes demand's credit to the channel that chases it. Graded against the sealed key: Search +43% over-credited, truth outside the 90% interval — with every sampler diagnostic green. Social, the other demand-chaser, sits at +38%. TV (−18%) and Display (−27%) carry the compensating under-credit, but their intervals at least cover.

2 — The readout, and what an experiment actually measures

The go-dark window: weeks 100–111, Search off in the treated markets. The readout — generated in the notebook from the world's sealed truth plus sampling noise at the design's SE, a well-run experiment rather than an oracle — is window incremental ROAS 0.49 ± 0.08 (the world's true window ROAS: 0.56, within one SE). Note the carryover tail: the lost effect extends past the window's end, because the window's spend was still working the next month. A competent readout — and the model's in-graph estimand — accounts for it.

3 — Route A: the experiment as a prior

ExperimentCalibrator reduces the lift to the coefficient scale using the fitted model's own response shape (the design factor), derives a tight prior, and refits. The derivation does its job perfectly: from the readout it computes a coefficient target of β = 1.15 ± 0.19, against a confounded fitted mean of β ≈ 2.20 — "cut Search roughly in half."

from mmm_framework.calibration import ExperimentCalibrator
from mmm_framework.validation.results import LiftTestResult

lift = LiftTestResult(
    channel="Search", test_period=(dark_start, dark_end),
    measured_lift=readout_roas * window_spend,   # KPI units
    lift_se=readout_se * window_spend,
)
cal = ExperimentCalibrator(model)          # the fitted, confounded MMM
report = cal.derive_priors([lift])         # -> beta target 1.15 +/- 0.19
out = cal.calibrate([lift], refit=True)    # refit with the derived prior

And then the data does what data does. After the refit, Search sits at +42% — the posterior barely moved. A prior worth one experiment's information faces 156 weeks of confounded likelihood, and the likelihood wins. A prior is a suggestion.

4 — Route B: the experiment as data

add_experiment_calibration attaches the readout to the graph as an observation of the model-implied window ROAS:

\[ \text{ROAS}_{\text{readout}} \;\sim\; \mathcal{N}\!\bigl(\text{ROAS}_{\text{window}}(\theta),\; \text{SE}^2\bigr) \]

Now agreement with the experiment is part of what "fitting the data" means — the sampler must find parameters that explain the history and the test, jointly re-negotiating the coefficient, the saturation curve, and the carryover.

from mmm_framework.calibration import ExperimentEstimand, ExperimentMeasurement

readout = ExperimentMeasurement(
    channel="Search", test_period=(dark_start, dark_end),
    value=0.49, se=0.08,                 # the calibrated readout from nb05
    estimand=ExperimentEstimand.ROAS,    # a go-dark measures AVERAGE window ROAS
    distribution="normal", name="search_dark",
)
model.add_experiment_calibration([readout])   # the readout becomes an observation
results = model.fit(draws=1000, tune=1000)    # jointly re-negotiates the posterior

Measured — nbs/artifacts/causal_06_calibration.json. Search relative error vs the sealed truth: baseline +43%, Route A (prior) +42%, Route B (likelihood) +5% — with the 90% interval covering truth again. The wrong-estimand filing (section 5) sits at +38%. One experiment, two routes: only the likelihood commits.

Measured — calibration is surgical. Search (tested) goes +43% → +5%; Social — the other demand-chaser, untested — keeps its confounding bias almost untouched, +38% → +40%. An experiment repairs what it measured, not what you wish it had measured. That is why notebook 07 builds a portfolio.

5 — The quietest failure: the wrong estimand

Rung 2 drew the line between average ROAS (what a go-dark measures) and marginal ROAS (what the next dollar buys). Here is what happens when the analyst files this readout under the wrong one — same number 0.49, same SE 0.08, declared as marginal at +10% spend (estimand=ExperimentEstimand.MROAS, spend_lift_pct=10.0): Search lands at +38%, essentially as biased as the never-calibrated baseline.

Why so quiet? Saturation puts the marginal below the average — the confounded model's marginal ROAS near current spend already sits close to the misfiled number, so the likelihood term is satisfied without moving anything. The constraint lands non-binding. No error, no warning, no correction: the test is simply spent. Estimand bookkeeping isn't pedantry — it's the difference between an experiment that repairs the model and one that vanishes into it.

6 — Calibration discipline, in five lines