Identifiability & Experiment Calibration

An observational MMM cannot pin down per-channel ROI on its own: carryover decay, saturation strength, and the coefficient \(\beta_c\) all fight to explain the same wiggle in the data, so very different parameter triples fit almost identically. This page derives the exact algebra mmm_framework uses to break that tie with a randomized experiment — the prior route (design factor \(K_c\) + Gamma moment-matching) and the likelihood route (an in-graph estimand term) — and shows the measured result: calibration pulls Aurora’s Search ROAS from an overstated 1.04 down to 0.64, against a known truth of 0.66. It mirrors nbs/math/math_05_calibration.ipynb, where every identity below is assert-encoded against the framework’s real calibration code.

1 · The equifinality trap

A channel’s modeled contribution is the composition of three learned pieces:

A long-carryover + weak-saturation setting can be made to look almost exactly like a short-carryover + strong-saturation setting, with \(\beta_c\) absorbing whatever scale difference remains. With only observational data the likelihood is nearly flat along that \((\text{decay},\,\text{saturation},\,\beta)\) ridge — this is equifinality, and it is the identifiability problem stated plainly:

The notebook demonstrates it with the framework’s own transforms. Regime A uses geometric decay \(\alpha = 0.70\) with weak saturation \(\lambda = 4\); regime B uses \(\alpha = 0.50\) with strong saturation \(\lambda = 8\). Run on a bursty spend plan, the two peak-normalized response shapes correlate at 0.951 despite a 0.20 gap in decay rate — no realistic noise level separates them.

Computed live in your browser — the same transform math as the notebook (normalized geometric FIR adstock, \(l_{\max}=24\), then \(\mathrm{sat}(u) = 1 - e^{-\lambda u}\)), on a seeded recreation of the bursty spend plan. The two lines are genuinely different (decay, saturation) regimes; the live correlation is printed on the chart. The baked notebook measures 0.951 for its seeded plan and asserts it exceeds 0.9.

Indistinguishable in-sample, the two regimes are not interchangeable for decisions. Scale the same spend plan by a factor \(k\) and the weak-saturation regime keeps growing while the strong-saturation regime plateaus — same fit, different attribution of headroom, opposite budget advice. Drag the slider:

Computed live in your browser — total window response \(R(k) = \sum_t \mathrm{sat}\big(k\cdot \mathrm{adstock}(x)_t\big)\) for each regime, normalized so both equal 1 at the observed plan (\(k=1\)), where the fits agree. Away from \(k=1\) they diverge — exactly the question (“what happens if we spend more?”) the model exists to answer. Adstock is linear in spend, so \(\mathrm{adstock}(kx) = k\,\mathrm{adstock}(x)\) and the whole curve is one saturation evaluation.

2 · The model being calibrated

Calibration is only meaningful relative to a specific generative model (derived in full in math 00 and fitted in math 04). The KPI is standardized, \(\tilde y_t = (y_t - \bar y)/s_y\), and its mean is a sum of additive blocks:

Saturation is the exponential form \(\mathrm{sat}(u) = 1 - e^{-\lambda u}\) (math 02); default adstock (since June 2026) is a parametric in-graph geometric kernel with learned decay \(\alpha_c \sim \mathrm{Beta}(1,3)\) — the legacy path blends two fixed-\(\alpha\) kernels with a \(\mathrm{Beta}(2,2)\) mix weight (math 01); the coefficient prior is \(\beta_c \sim \mathrm{Gamma}(\mu{=}1.5,\ \sigma{=}1.0)\) — positive by construction. Because the model is additive and \(\mathrm{sat}(0) = 0\), the decomposition \(y \approx \text{baseline} + \sum_c \text{contribution}_c\) is exact in the mean, and a channel’s contribution is exactly what zeroing its spend would remove. That one fact powers everything below.

3 · Route 1 — the prior route (ExperimentCalibrator)

The two-stage recipe: fit with default priors → derive an experiment-anchored prior on the tested channel’s \(\beta_c\) → refit. The derivation rests on one clean piece of linear algebra.

The key linearity, and the design factor \(K_c\)

Over an experiment window \(P\), the channel’s counterfactual contribution is

The bracket has no \(\beta_c\) in it, so contribution is linear in \(\beta_c\): \(\text{contribution}_c = \beta_c K_c\). A full-holdout lift test measures that contribution directly as \(\widehat L \pm \mathrm{se}_L\), and the inversion is one division:

The right-hand identity is exact algebra, not MCMC — the notebook asserts \(\beta^{\text{target}}\cdot K_c = \widehat L\) to atol=1e-12. With the measured demo numbers: a lift of \(1.95 \pm 0.30\) and \(K_c = 1.3000\) give \(\beta^{\text{target}} = 1.5000\) and \(\beta^{\text{se}} = 0.2308\), and \(1.5000 \times 1.3000 = 1.95\) exactly.

Why \(K_c\) is a per-draw average, not a ratio of means

\(K_c\) depends on the saturation/adstock shape, which is itself uncertain and — by equifinality — negatively correlated with \(\beta_c\) across posterior draws (a draw with a larger coefficient tends to carry stronger saturation, hence a smaller per-unit-\(\beta\) contribution). The framework’s design_factor(...) therefore computes the ratio per draw and averages:

In the notebook’s synthetic posterior with \(\mathrm{corr}(\beta, K) = -0.916\), the per-draw estimator recovers the marginal mean exactly (\(K_c = 1.3000\)) while the ratio-of-means shortcut drifts to 0.9959 — a 0.304 gap, 23% of the answer, leaked in entirely by the \(\beta\)–shape covariance. That gap is the whole reason design_factor exists.

Pooling several tests: inverse-variance meta-analysis

A channel tested more than once yields several \((\beta^{\text{target}}_i,\ \beta^{\text{se}}_i)\) pairs. combine_inverse_variance pools them with fixed-effect weights \(w_i = 1/\mathrm{se}_i^2\):

Measured: targets \([1.50,\ 1.62,\ 1.40]\) with ses \([0.30,\ 0.12,\ 0.50]\) pool to \(\bar\beta = 1.5938,\ \mathrm{sd} = 0.1087\). The combined precision \(84.56\) equals the sum of the individual precisions, the pooled sd beats the tightest single test (0.1087 < 0.12), and the estimate sits closest to the most precise test — all assert-encoded.

Rendering the anchor as a Gamma prior (moment matching)

The coefficient prior must stay a positive \(\mathrm{Gamma}\), so mean_sd_to_gamma(m, s) converts the pooled \((m, s)\) by moment matching. For \(\mathrm{Gamma}(\alpha,\ \text{rate})\) the mean is \(\alpha/\text{rate}\) and the variance \(\alpha/\text{rate}^2\); solving:

For the pooled target this gives \(\mathrm{Gamma}(\alpha = 214.79,\ \text{rate} = 134.77)\), which round-trips to \((1.5938,\ 0.1087)\) to nine decimals. The full pipeline, derive_channel_prior, also drops non-positive lifts before pooling: fed two usable tests plus one negative-lift test, it returns \(\beta^{\text{target}} = 1.5848,\ \beta^{\text{sigma}} = 0.1187\) (a \(\mathrm{Gamma}(178.18,\ 112.43)\) ROI prior) and notes the excluded test.

Computed live in your browser — exact Gamma densities at the measured parameters. The default prior \(\mathrm{Gamma}(\mu{=}1.5, \sigma{=}1.0)\) (i.e. \(\alpha = 2.25\), rate \(=1.5\)) lets \(\beta\) roam; one test (\(m = 1.5000,\ s = 0.2308\)) relocates and tightens it; pooling three tests (\(\alpha = 214.79\), rate \(= 134.77\)) pins it. The experiment doesn’t just move the prior — it adds the information the likelihood ridge couldn’t supply.

4 · Route 2 — the likelihood route (ExperimentMeasurement)

Route 1 inverts the experiment with the saturation/adstock shape held fixed at the first-stage posterior — it can only encode a contribution, and its derived prior is marginally tighter than a fully joint treatment would justify. Route 2 instead folds the experiment in as an extra likelihood term inside the PyMC graph:

where \(\theta\) are the channel’s live in-graph parameters \((\beta_c,\ \lambda_c,\ \alpha_c)\). Because the estimand is a function of the live graph, this single fit updates \(\beta_c\), the s-curve, and the adstock kernel jointly — the posterior is forced to find parameter combinations that explain the time series and reproduce the experiment, which collapses the equifinality ridge instead of just re-centering one marginal. Three estimands are supported, with \(\text{contribution} = s_y \sum_{t\in P} \beta\,\mathrm{sat}(\mathrm{adstock}(x_t))\):

That multi-estimand menu is the second advantage over the prior route: a ROAS or marginal-ROAS experiment has no expression as a \(\beta\) prior, but it is one line as a likelihood term. Worked example from the notebook (measured): with \(s_y = 12\), window contribution \(0.18\), spend \(3.0\), a \(+10\%\) perturbation to \(0.192\) gives \(\text{CONTRIBUTION} = 2.16\), \(\text{ROAS} = 0.72\), \(\text{MROAS} = 0.48\).

For strictly positive estimands like ROAS, a lognormal measurement error is available; its log-scale sd is moment-matched from the natural-scale coefficient of variation (lognormal_sigma_from_moments):

5 · Measured: calibration pulls Search onto the truth

Aurora’s Search channel is the textbook confounded channel — its spend chases latent demand, so even a demand-controlled observational model overstates its ROI. Ground truth (the data-generating process is known): Search’s true ROAS is 0.664. The notebook fits an uncalibrated demand-controlled model, then refits with one in-graph ROAS experiment (value \(= 0.664\), \(\mathrm{se} = 0.07\), a 38-week window) and compares. Both fits converge (max \(\widehat R = 1.00\)); the uncalibrated fit tracks the data at correlation 0.961. The diagnostics cannot tell them apart — the attribution can:

Measured — baked outputs of math_05_calibration.ipynb (seeded, 400 draws, 2 chains). Uncalibrated Search ROAS 1.040 with a 94% contribution-HDI band of [0.01, 2.76] in ROAS units (contribution HDI divided by window spend); experiment-calibrated point estimate 0.639 (the calibrated run’s interval is not printed in the baked outputs, so no band is drawn). Dashed line: true ROAS 0.664. Distance to truth: 0.376 → 0.024. The model-implied ROAS estimand node (experiment_Search_roas_0_model_estimand) concentrates at posterior mean 0.661 against the measured 0.664 — the likelihood term doing its job.

Under the hood, the decomposition shows where the correction lands. Search’s mean per-week contribution drops from 33.14 to 20.49; the credit it was stealing spreads modestly across TV (19.45 → 22.92), Social (21.12 → 22.38), and Display (18.42 → 19.99):

Measured — mean per-week channel contributions from compute_component_decomposition(), uncalibrated vs calibrated (baked outputs). The experiment is an anchor on Search; it shrinks Search directly, and the other channels move only because the freed-up variance has to go somewhere. Nothing about their own confounding has been tested — see the doctrine below.

6 · Interactive: how strong must the test be?

The mechanics of the pull are precision-weighting. In a toy conjugate Normal–Normal model — observational evidence \(\mathcal N(\mu_0, \tau^2)\), experiment \(\widehat L \sim \mathcal N(\theta, \mathrm{se}^2)\) — the posterior is

Set \(\mu_0 = 1.04\) (the overstated observational estimate) and \(\widehat L = 0.664\) (what a clean lift test measures), and drag the experiment’s standard error from sloppy to sharp:

Computed live in your browser — illustrative toy model. The real MMM is not conjugate (the estimand is a nonlinear function of \(\beta\), \(\lambda\), and the adstock weight), but the precision-weighting logic is identical: the posterior lands at the precision-weighted average of what the time series says and what the experiment says. At \(\mathrm{se} = 0.07\) — the demo’s value — the experiment carries ~99% of the weight and the posterior sits essentially on the test. A sloppy test (\(\mathrm{se} \gtrsim 1\)) barely moves the needle: an under-powered experiment is not calibration, it’s decoration.

7 · Doctrine: experiments calibrate what structure cannot identify

Three things to carry out of this page:

1. Equifinality is structural, not a data-size problem. The \((\text{decay}, \text{saturation}, \beta)\) ridge is nearly flat in the observational likelihood; controls help but cannot flatten a channel that chases demand. Sampling longer sharpens the wrong posterior.
2. A randomized experiment breaks the tie because contribution is linear in \(\beta_c\). \(\text{contribution}_c = \beta_c K_c\), so a measured lift inverts cleanly to a coefficient anchor — and the per-draw design factor \(K_c\) is the bridge that keeps the \(\beta\)–shape covariance from corrupting it.
3. Calibration is surgical, not contagious. The experiment anchors the tested channel only. In the Aurora demo the untested channels just absorb the re-allocated variance; in stress 05 the same move pulls Search from 5.59 to 0.65 (truth 0.66) while the untested channels stay wrong. One lift test fixes one channel’s identification — it is not absolution for the model.

Where this lives in practice: the measurement & calibration guide covers the operating loop (which channels to test, geo vs holdout designs, how often to refresh); stress 03 shows the failure calibration exists to fix; math 04 derives the Bayesian machinery these likelihood terms plug into; and math 06 extends the same logic to mediation and cross-product effects — where the identification gaps are even wider and priors do even more of the work.

Two further companions close the loop: stress 05 · the gauntlet measures this calibration math across the full sequence of adversarial worlds, and the measurement & calibration guide shows how experiment selection (EIG/EVOI priorities) and the experiment lifecycle operate around it — deciding which test to run before this page's machinery decides what the result is worth.