Stress Test 00: The Rosy Picture
Your MMM converged. R-hat is 1.00 across the board, zero divergences, the posterior-mean fit traces the KPI beautifully. Should you believe the ROAS table? Not on that evidence. This opener fits the same model to two synthetic worlds — one clean, one where spend secretly chases demand — and shows that every check an analyst acts on is green on both, while one of them over-credits the demand-chasing channels by large multiples. It sets the doctrine the whole stress series runs on: green diagnostics validate the computation, never the causal claim.
🧭 Before you start
Prerequisites: Workshop 03–05 or equivalent MMM experience; the Aurora series helpful. · Time: ~30–40 min.
You will learn:
- Why green sampler diagnostics — r-hat 1.00, zero divergences, a passing PPC — can coexist with attribution that is wrong by multiples: they validate the computation, never the causal claim.
- How a counterfactual zero-out estimand and a clean positive control (7% median error, 100% coverage) make model error measurable instead of rhetorical.
- What each check in the diagnostic toolkit can and cannot see — sampler checks test sampling, PPC tests fit, and none of them tests attribution.
Run it:
nbs/stress/stress_00_the_rosy_picture.ipynb on GitHub.
1 · The estimand discipline: what number would be wrong?
A stress test is only as honest as its definition of "truth", and an MMM is only as
honest as its definition of "contribution". Before reading any posterior — before
fitting — you must be able to answer: if this model is wrong, which
number is wrong, and against what? The series' answer, used in every world in
src/mmm_framework/synth/dgp.py: contribution is the counterfactual
zero-out, evaluated on the noiseless structural mean,
\[ C_c \;=\; \sum_{t=1}^{T}\Big(\, m_t(\mathbf{x}) \;-\; m_t\big(\mathbf{x}^{(c\to 0)}\big) \,\Big) \]
where \( m_t(\cdot) \) is the world's true mean response and
\( \mathbf{x}^{(c\to 0)} \) is the observed spend history with channel \(c\) set to zero
everywhere (carryover handles itself — zero spend adstocks to zero). The model's estimate,
via compute_counterfactual_contributions, is computed the same way:
zero the channel's column, difference the posterior predictions. Truth and estimate are
the same estimand on the same KPI scale, so the gap between them
is the attribution error — not an artifact of mismatched definitions.
- For confounded worlds, the truth is causal. Zeroing spend does not change the demand-driven baseline — so when the model's estimate departs from it, that gap is exactly the causal bias the diagnostics ought to catch (and, below, don't).
- Nothing here is unfair to the model. The estimand is the model's own; it is graded on the question it claims to answer.
- Representable vs not. A world is representable when the truth lies inside the model's hypothesis space (e.g. positive effects under the positive-only prior). Failures on representable worlds are the model's problem; failures on unrepresentable ones are expected by construction and labeled as such.
If you cannot state what number would be wrong, no diagnostic can save you — there is nothing for it to be wrong about. That discipline is rung one of the fix ladder.
2 · Two worlds, one dashboard
The positive control. dgp.build("clean") draws data from
the model's exact generative family — geometric adstock,
\( 1-e^{-\lambda x} \) saturation, additive channels, linear trend, Fourier-representable
seasonality, Gaussian noise, and exogenous spend. If the model can't
recover this world, nothing downstream means anything. It can: every channel lands within
8% of truth, every 90% interval covers it — 7% median error, 100%
coverage.
The punchline world. In dgp.build("unobserved_confounding")
a latent demand signal (seasonal + growth + AR(1), hidden from the model) drives
both the baseline and the media budgets: Search and Social chase demand
the way real bidding and pacing systems do. Everything else is the clean world. The only
change is that spend is no longer exogenous — there is an open back-door from demand to
sales that the model cannot block, because the confounder is not in the data. The model
fits it happily: demand-driven spend is an excellent predictor of demand-driven
sales, and the likelihood loves it.
Measured in the notebook (seeded run, current defaults) — true causal
contribution (dark bars) vs the model's estimate with 90% HDI whiskers, from
nbs/stress/stress_00_the_rosy_picture.ipynb. Clean: median |error| 7%, coverage
100%. Confounded: Search +44% and Social +38% with the
truth outside the interval — not just wrong, confidently wrong — median |error|
32%, coverage 50%. The recorded harness run of the same world
(tests/synth/results/stress_matrix.md, higher fidelity) logs Search at
+69%, truth outside the CI, with the same green diagnostics.
⚠️ What the diagnostics said
Clean fit: r-hat max 1.009, 0 divergences, min bulk ESS 355. Confounded fit: r-hat max 1.012, 0 divergences, min bulk ESS 402. PPC: pass on both. In-sample R²: 0.90 vs 0.88. The dashboard every analyst checks cannot tell these worlds apart. The mechanism is mundane: Search spend rises exactly when hidden demand lifts the baseline, so the likelihood's cheapest explanation hands demand's effect to Search — and the sampler then converges correctly on that wrong posterior. R-hat measures whether the chains agree; they agree on the biased answer.
💡 Framework fix that reshaped this page's numbers
When this notebook was first built, the confounded world read Search at ~+111% at notebook fidelity. The stress work traced part of that to a framework defect: the linear-trend growth prior was too tight (sigma 0.1 — the fitted trend reached only ~43% of a true trend's amplitude), so the starved trend pushed even more of the demand signal into media. The prior was loosened to 0.5 on 2026-06-10 (fitted trend now reaches ~95% of true amplitude), which halved the confounded bias to ~+44% — better, still far outside any acceptable error, and Search/Social are still mis-ranked. The same starved prior also made PPC false-alarm on the clean control in older recordings; it now passes. Both eras teach the same lesson from opposite directions: these verdicts track the fit configuration, not the causal claim.
3 · The silent-failure gate
The two fits above are one row each of a systematic sweep: the harness in
tests/synth/harness.py fits the same model on 16 worlds and grades every
estimate against the zero-out truth. The series' central definition:
Definition: Silent failure
A representable scenario where recovery is wrong — median per-channel |error| > 25%, worst channel > 50%, or 90%-interval coverage < 75% — yet the checks an analyst acts on are green: MCMC convergence (r-hat < 1.05, zero divergences) and the unobserved-confounding robustness value. PPC and the causal refutation suite are reported but excluded from the gate: their verdicts have proven config-sensitive across recordings (both false-alarmed on the clean control under the old trend prior; both pass on clean and on several wrong-attribution worlds under current defaults). A check whose verdict tracks the sampler configuration rather than the causal claim cannot serve as an all-clear.
The recorded matrix (re-recorded 2026-06-10): 16 worlds, 8 silent failures — unobserved confounding, the noisy-proxy "we controlled for it" trap, multicollinearity, saturation misspecification, spend outliers, trend breaks, evolving seasonality, dense controls — every one of them with r-hat ≤ 1.02 and zero divergences. The full scorecard chart lives on the series hub; the per-failure stories live in guide 01, 02 and 03.
4 · The diagnostic toolkit: what each check can and cannot see
If the routine checks can't catch silent failures, what is each one for? The notebook runs the framework's full validation suite on both fits — the one we know is right and the one we know is wrong — and records which checks discriminate. None do. The measured columns below are that experiment.
| Check | Actually tests | Can see | Cannot see | Clean fit (right) | Confounded fit (wrong) |
|---|---|---|---|---|---|
| R-hat / ESS / divergences | The sampler explored the posterior of the model as written | Broken sampling; non-identification sometimes (via divergences) | Any causal error — converges happily on biased posteriors | 1.009 · 0 div | 1.012 · 0 div |
| In-sample fit / R² | Predictive adequacy | Gross misspecification | Confounding — a confounder improves fit | R² 0.90 | R² 0.88 |
| Posterior predictive checks | Replicated data resembles observed data | Wrong noise model; missing dynamics | Mis-attribution at the same fit quality (and it has false-alarmed on good models) | pass | pass |
| Confounding robustness value (RV) | How strong a hidden confounder must be to nullify each channel | Nothing by itself — it is a bound, computed from the possibly-biased posterior | Whether such a confounder exists | 0.17–0.29 · none fragile | 0.12–0.22 · none fragile |
| Causal refutation suite | Effect behaves sanely under placebo / perturbation refits | Implementation bugs; fragile effects | Calm, stable bias; poor specificity (false alarms on fine worlds) | pass | pass |
| Parameter learning (prior→posterior) | Whether the posterior differs from the prior, per parameter | Prior-dominated reporting; prior–data conflict | Bias in well-identified parameters — a confounded β learns strongly, toward the wrong value | verdicts vary even on the correct fit (see below) | |
Measured in the notebook (seeded run) — validation suite
(ModelValidator, PPC + confounding RV + 4-test refutation at the harness's
fidelity) on both fits. Per-channel RVs: clean TV 0.25 / Search 0.29 / Social 0.23 /
Display 0.17; confounded TV 0.19 / Search 0.22 / Social 0.19 / Display 0.12 — the
confounding RV reads essentially the same on the clean and the truly confounded
fit, and flags nothing on either. At the recorded harness fidelity the
refutation suite does fail placebo_treatment on the confounded world
— but it fails the same test on worlds whose attribution is fine
(reverse_causality, heavy-tailed noise), and has historically false-alarmed
on the clean control. A detector with that specificity is a red-flag generator for
development, never a green light.
The honest non-catcher, demonstrated on the clean fit.
compute_parameter_learning compares each parameter's prior to its posterior —
contraction \( 1 - \sigma^2_{\text{post}} / \sigma^2_{\text{prior}} \), overlap, shift —
and is the most misread tool in the kit. On the clean fit (recovery is good,
every truth inside its interval) the per-channel β verdicts still span the range:
Social 0.08 (weak), TV 0.46 (moderate), Display 0.59 (strong), Search 0.90 (strong).
TV carries the largest true contribution (5,668) but not the most-learned β —
its β trades off against the saturation and adstock parameters while the
contribution stays tightly identified. Contraction measures
informativeness, not importance. Use it as an audit of which reported numbers
are data versus priors restated — never as a channel ranking.
Guide 04 leans on this doctrine hard.
What's missing from every row of the table: information the model doesn't have — a real confounder measured well, or randomized evidence (a lift test). That is why the series' recovery arc runs through design and calibration (guide 03, guide 05), not through more diagnostics.
5 · Three uncertainty-first lenses
Scalar diagnostics compress the posterior to a verdict; fit plots keep the uncertainty visible. The notebook's §3.1 builds three views from the posterior you already have, each answering a different question. Because the data is synthetic, the true weekly contribution path can be overlaid — the view a real analyst never gets, and exactly where the two worlds finally separate.
Lens 1 — the posterior-predictive band: a fit check, with a fit check's blind spot
Illustrative simulation in your browser (seeded), styled after the notebook's figure. The measured punchline: both fits reproduce the KPI superbly — R² 0.90 (clean) vs 0.88 (confounded), with the 90% predictive band hugging the observed weeks in both worlds. If your defense of an MMM is "look how well it tracks the data," you have defended the confounded model too.
Lens 2 — contribution-over-time bands vs the true weekly path
Illustrative simulation in your browser, styled after the notebook's figure; the dashed line is the true weekly Search contribution. The measured punchline (seeded notebook run): on the clean fit the truth lives inside the 90% band 88% of weeks for Search (100% for TV); on the confounded fit the Search band excludes the truth ~90% of weeks — the truth sits below the band week after week (inside only 10%), while TV in the same wrong fit stays honest at 100%. The bias is a level shift, not noise: the inherited demand credit. The catch: the dashed line is unobservable in real life. What this view gives a real analyst is the model's full week-by-week claim with honest widths — the artifact to put in front of domain experts and test against lift windows, instead of one ROAS number stripped of its uncertainty.
Lens 3 — posterior correlation of channel totals: the separability report
Illustrative simulation in your browser (correlations of simulated posterior draws), matching the measured result: max |pairwise correlation| of channel totals is 0.08 on the clean fit and 0.07 on the confounded fit — both posteriors separate the channels cleanly, including the wrong one. The confounded posterior is not confused about the split; it is confidently wrong about the level. This lens diagnoses identification between channels, not causality — and it gets loud exactly where it should: on the shared-calendar collinear world of guide 03, every pairwise correlation goes negative (min ≈ −0.5) — the joint posterior's own statement that the split is a seesaw.
💡 Reading the three lenses together
Predictive bands validate fit; contribution bands expose the model's actual week-by-week claim; the totals' correlation matrix reports which numbers are jointly pinned down versus traded off. None of them detects confounding — that lesson stands — but they turn "trust the table" into reviewable, testable claims. Plot the uncertainty, not just the verdicts.
6 · Build your own confounder
The mechanism is simple enough to run in your browser. Below, a 156-week world is simulated client-side: sales depend on Search, TV, trend, seasonality, and a latent demand signal (AR(1)) the regression never sees. The slider sets how strongly the Search budget chases that hidden demand. A naive least-squares "MMM" (intercept + trend + annual seasonality + Search + TV) is refit on every move. Watch two numbers: the Search attribution error inflates with confounding strength — while in-sample R² stays high and even improves, because the chasing channel is soaking up the demand signal the model otherwise couldn't explain.
Illustrative simulation in your browser — a seeded toy world with closed-form least squares, not the framework. At strength 0 the toy recovers its own positive control (Search estimate ≈ truth, interval covers it). As strength rises the point estimate runs away and the 90% interval leaves the truth behind — the same level-shift signature as the measured confounded fit above — while the non-chasing TV channel stays covered throughout. No goodness-of-fit number on this chart ever gets worse.
7 · Resolved: the legacy adstock default
✅ Finding closed (June 2026) — and it changed the framework
Every fit on this page uses use_parametric_adstock=True, so the model
estimates the same kernel family the world was generated with. With the legacy
two-point blend adstock — use_parametric_adstock=False, the
default before June 2026 — the clean positive control recovers at
~28% median error vs 7% parametric: the old default underperformed on the
model's own generative family. This finding drove a change:
parametric adstock is now the shipped default; the blend remains an
explicit opt-out for reproducing older fits.
Guide 01 covers kernel choice in depth.
8 · What to take away
What to take away
- Green diagnostics validate the computation, not the causal claim. Both fits here converged identically (r-hat ≤ 1.012, 0 divergences, PPC pass); one was wrong by multiples on the channels that matter.
- Insist on a comparable estimand before fitting. Truth here is the counterfactual zero-out on the structural mean — the same question the model answers — so error is measurable, not rhetorical. Ask "what number would be wrong?" first.
- Run a positive control. The clean world's 7% / 100% is what earns every later accusation: same model, same settings, correct answer.
- Know each check's scope. Sampler checks test sampling; PPC and R² test fit; the RV bounds sensitivity; parameter learning audits informativeness (contraction ≠ importance). None tests attribution.
- The fixes are upstream of the fit: causal structure, data hygiene, and experiments. The rest of the series — 01 through 06 — is about exactly that.
💡 Run it yourself
This page mirrors nbs/stress/stress_00_the_rosy_picture.ipynb (authored by
nbs/builders/build_stress_00_rosy_picture.py), backed by the scenario DGPs in
mmm_framework.synth, the scoring harness in tests/synth/, and the recorded scorecard in
tests/synth/results/stress_matrix.md. Every computational cell ends in a
seeded assert encoding the claim it demonstrates — if the notebook executes clean, the
story on this page is still true. Next:
Stress 01 — Carryover & Shape.
Source:
github.com/redam94/mmm-framework.