Pressure Testing the MMM

A marketing mix model that converges cleanly and fits the data beautifully can still be badly wrong about attribution. We fit the framework to synthetic worlds engineered around real-world difficulties — where the true causal contribution of every channel is known — and grade every estimate against that truth. This page is the scorecard; the seven interactive guides walk through each failure mode and the recovery moves, with the same story and charts as the nbs/stress_0X notebooks.

16
synthetic worlds, each breaking one named assumption
8
silent failures — wrong attribution, green diagnostics
≤ 1.02
r-hat on every silent failure (0 divergences)
7%
median error when no assumption is broken (positive control)

⚠️ The motivating worry

The checks analysts trust — R-hat, divergences, posterior-predictive checks, in-sample fit — validate the computation, not the causal claim. In the recorded matrix below, not one attribution failure was caught by convergence diagnostics. Every silent failure sampled cleanly.

How the tests work

Every world comes from the scenario harness in tests/synth/. Each one starts from a clean core world drawn from the model's exact structural family, then injects one real-world violation (the capstone injects many at once). Because ground truth is the model's own counterfactual zero-out estimand, the "true contribution" and the model's compute_counterfactual_contributions are directly comparable — there is no apples-to-oranges gap in the grading.

Definition: Silent failure

A representable scenario where recovery is wrong — median per-channel error > 25%, worst channel > 50%, or 90%-interval coverage < 75% — yet the checks an analyst acts on are green: MCMC convergence (r-hat < 1.05, no divergences) and the unobserved-confounding robustness value. The model looks healthy. The numbers are wrong.

Estimates are graded on per-channel total contribution: median and worst-case relative error, sign and size of the total-media error, and the fraction of channels whose true contribution lands inside the 90% credible interval ("coverage"). The full recorded scorecard lives at tests/synth/results/stress_matrix.md and is re-recorded whenever framework defaults change.

The scorecard: 16 worlds

Each bar is one synthetic world. recovers means the model got attribution right (or a non-convergence check flagged the problem); silent failure means attribution is wrong while every check an analyst acts on stayed green; expected marks worlds the model structurally cannot represent (negative effects under a positivity prior; the kitchen-sink capstone fit demand-blind). Hover any bar for the broken assumption and the per-world numbers.

Measured — recorded scorecard from tests/synth/results/stress_matrix.md (re-recorded 2026-06-10 on current defaults; seeded NumPyro runs). The dashed line marks the 25% median-error gate. Note where the red bars sit: confounding, collinearity, saturation form, spend outliers, trend breaks, evolving seasonality, dense controls — none of them exotic, all of them quiet.

Deep diveThe recorded scorecard as a plain table, world by world

The same scorecard as a plain table — one row per world, in the order the matrix records them. "Silent failure" means attribution is wrong while every check an analyst acts on stayed green.

World What it violates Median |error| Max |error| 90% CI coverage Silent failure?
clean— (positive control)7%7%100%no — recovers
unobserved_confoundingno unobserved confounding (exogeneity of spend)23%69%75%yes — silent failure
reverse_causalityno unobserved confounding (exogeneity of spend)13%19%100%no — recovers
multicollinearityidentifiability (independent spend variation)39%60%100%yes — silent failure
adstock_misspecadstock functional form / carryover window80%96%50%no — flagged (fragile robustness value)
saturation_misspecsaturation functional form (concavity / no threshold)43%65%25%yes — silent failure
time_varying_betatime-invariant coefficients (stationarity)13%20%100%no — recovers
heavy_tailed_noiseGaussian, homoscedastic likelihood4%37%75%no — recovers
synergyadditive separability of channels4%6%100%no — recovers
spend_outliersrobustness of per-channel max-normalization39%70%0%yes — silent failure
negative_effectpositivity of media effects (Gamma prior, β ≥ 0)19%101%50%expected (unrepresentable)
trend_breaksmooth global trend (no structural breaks)27%41%100%yes — silent failure
seasonality_misspecstatic low-order Fourier seasonality37%104%75%yes — silent failure
dense_controlsa parsimonious, correctly-chosen control set16%52%75%yes — silent failure
confounding_controlledno unobserved confounding (noisy-proxy control)10%50%100%yes — silent failure
aurora_kitchen_sinkmultiple at once (confounding + mediation + non-1-exp saturation)92%223%25%expected (unrepresentable)

Scorecard provenance: tests/synth/results/stress_matrix.md, repo snapshot 2026-06-12 (matrix re-recorded 2026-06-10 on current defaults; seeded NumPyro runs). One honest caveat: the clean world's ~7% median error is an upper bound on real-world performance — the control DGP is drawn from the model's own structural family, and real data never is.

The checks were green everywhere

If convergence diagnostics could see attribution error, the points below would slope upward: worse attribution, worse r-hat. They don't. Every world — including the one where the worst channel is off by 104% — sampled with r-hat ≤ 1.02 and zero divergences. (The only divergences in the whole matrix belong to the deliberately unrepresentable kitchen-sink world.) Convergence tells you the sampler explored the posterior; it cannot tell you the posterior is centered on the wrong answer.

Measured — each point is one world: worst-channel attribution error (x) against r-hat (y). The shaded band is the conventional r-hat < 1.05 "all clear" zone. Red silent failures live happily inside it.

💡 What this means in practice

Run convergence checks — they are necessary. But treat them as a statement about the sampler, never as evidence the attribution is right. The checks that actually move the needle are structural: spend-vs-demand correlation before fitting, split-window refits, posterior contribution correlations, lift-test calibration. Each guide below demonstrates the relevant ones.

📜 The identification contract

Each world in this matrix breaks one named assumption. The full list — all seven, stated formally, labeled testable or untestable, with these scorecard numbers as the price of breach — lives in one place: the identification contract.

Known strengths & weaknesses

The honest summary an analyst should carry around. Everything here is measured in the notebooks, not asserted from theory.

✅ Where the framework holds up

  • Clean recovery: on the positive control the parametric pipeline recovers every channel at ~7% median error with 100% coverage.
  • Reverse causality (last week's sales driving this week's spend): 13% median error, full coverage — the adstocked design absorbs it.
  • Channel synergy left out of the model: ~4% error — additive misspecification is forgiving here.
  • Heavy-tailed noise: 4% median error; attribution is robust even where the Gaussian likelihood is wrong.
  • Slow effectiveness drift: time-varying betas average out to ~13% error on totals, and split-window refits expose the drift when you look.
  • Decoy controls don't steal credit: media-tracking decoys cost precision, not attribution — the adstock/saturation pipeline separates them.
  • Honest collinear intervals: under near-perfect collinearity the posterior widens and the contribution "seesaw" is visible in posterior correlations — the uncertainty is real, and reported.
  • Calibration is surgical: one ROAS lift test pulls the confounded channel's bias from +110% to +14% without disturbing the others (a measured before/after pair from the stress-03 notebook bake; the recorded scorecard's worst-channel bias for this world is +69%).

❌ Where it fails silently

  • Demand confounding: spend that chases demand inflates the chasing channel +69% (worst channel; 23% median) in the recorded scorecard, with every diagnostic green — an earlier notebook bake measured +110%. Only design (roles, controls, experiments) fixes it.
  • "We controlled for it" isn't a fix: a noisy proxy of the confounder still leaves a +50% worst-channel bias in the recorded scorecard (an earlier bake measured ~+35% with truth outside the interval; under current defaults the wide interval covers — wrong and quiet either way).
  • Saturation form: Hill-type thresholds fit with a concave default read +38% on total media — green across the board.
  • Spend outliers: a handful of 15× spikes crush max-normalization — coverage drops to 0%. (A 30-second max/median screen catches this before fitting.)
  • Trend breaks & evolving seasonality: structural time misfit flows into whichever channel's spend co-moves with it — direction unpredictable, totals often still right.
  • Mediation: the base model structurally loses mediated channels (TV ROAS 0.22 vs true 2.14); extensions recover them but add prior-determined splits the data cannot identify.
  • Resolved (June 2026) — legacy adstock default: the old default two-point blend recovers the clean world at ~28% median error vs 7% parametric. That finding drove a change: use_parametric_adstock=True is now the shipped default; the blend remains an explicit opt-out for reproducing older fits.
  • Resolved (June 2026) — holdout validation: the out-of-time forecaster (PosteriorForecaster._saturate, validation/backtest.py) now reconstructs each channel's actual saturation form (Hill, Michaelis–Menten, tanh, logistic, or none), so Hill-fit models get correct holdout predictions — the earlier logistic-fallback bug is gone.

The fix ladder

The series' central recovery doctrine: pivots are cheap to try and easy to misuse. Functional-form pivots help only when form is the actual problem; identification problems are solved by design, not by sampling longer. Work the ladder top-down.

  1. Estimand first. Pin down what "contribution" means (counterfactual zero-out) before reading any posterior. If you can't say what number would be wrong, no diagnostic can save you.
  2. EDA pre-flight. Spend-vs-structure correlation, the max/median outlier screen, spend-pattern collinearity — each takes seconds and catches failures sampling never will. Misfit goes to whichever channel's spend co-moves with it.
  3. Structural pivots. Causal roles for confounders, piecewise trend for breaks, event dummies for holidays (median error → 9% on the holiday world), Weibull adstock and Hill saturation where the data demand them — each measured before/after in the guides.
  4. Experiments calibrate what structure can't. On the kitchen-sink world, one lift test pulls Search ROAS from 5.59 to 0.65 against a truth of 0.66 — and untested channels stay wrong. Calibration is surgical, not contagious.

The interactive guides

Seven guides, one per stress notebook. Read 00 first — it sets the doctrine and the diagnostic toolkit. 01–04 and 06 stand alone; 05 is the capstone that throws everything at the model at once. Every measured number on these pages comes from the seeded notebook runs.

Each stress guide is the empirical counterpart to a theory page — the math says what the model assumes, the stress test measures what happens when the world disagrees:

STRESS 00

The Rosy Picture

The doctrine. A positive control recovers at 7% error — and an equally-green confounded fit is +111% wrong on Search. What each diagnostic can and cannot see.

Same green dashboard, opposite verdicts.
STRESS 01

Carryover & Shape

Delayed carryover vs a geometric kernel (median error 88%); Hill thresholds vs a concave default (+40% total, all green); 15× spend spikes (coverage 0%) — and the pivots that fix each.

The kernel that can't bend, bends the answer.
STRESS 02

Time Structure

A trend break plus a reactive media ramp sends TV to −63%; evolving seasonality scrambles the channel split while totals stay right; drift hides inside green totals.

Misfit flows to whoever co-moves with it.
STRESS 03

Confounding & Selection

Demand-chasing spend (+110% on Search, all green), the noisy-proxy trap, the collinear seesaw, 25 candidate controls — and the one lift test that closes the gap.

"We controlled for it" is not a fix.
STRESS 04

Extension Traps

The base model loses a mediated channel; NestedMMM recovers it but over-credits its sibling 2.6×; a cross-effect flips sign with the prior; a wrong mediator fits perfectly.

Extensions are cure and disease.
STRESS 05

The Gauntlet

Everything at once: ranking fully inverted, Search 8× overstated. The full recovery workflow — EDA, triage, structural pivots, experiment calibration — each step measured.

Search ROAS 5.59 → 0.65; truth 0.66.
STRESS 06

Geography & Hierarchy

Panel worlds: a cross-geo adstock-bleed bug found and fixed by this series; national numbers green while per-geo errors hit +330% and the regional ROI ranking fully inverts.

Right nationally, backwards regionally (ρ = −1).
Deep diveReconciling guide-card figures with the canonical scorecard

A note on the card numbers — guide-card figures mirror each guide's baked notebook run. Where a card disagrees with the recorded scorecard (e.g. confounded Search +110/+111% vs the matrix's +69% worst-channel; delayed-carryover median 88% vs 80%; Hill-threshold total +40% vs +38%), the scorecard at tests/synth/results/stress_matrix.md is canonical — the larger figures are earlier bakes recorded under older defaults.

💡 Run it yourself

Every guide mirrors a runnable notebook in nbs/ (stress_00_the_rosy_picture.ipynb … stress_06_geography_and_hierarchy.ipynb), backed by the scenario harness in tests/synth/. Every computational cell ends in a seeded assert encoding the claim it demonstrates — if the notebook executes clean, the story on these pages is still true. Source: github.com/redam94/mmm-framework.

References