Stress Test 01: Carryover & Shape

Every channel in an MMM gets a shape: a carryover kernel that says when the effect arrives, and a saturation curve that says how it bends with spend. No sampler diagnostic ever inspects either. This guide mirrors nbs/stress/stress_01_carryover_and_shape.ipynb: three worlds from src/mmm_framework/synth/dgp.py where exactly one shape assumption is wrong, three naive fits that converge beautifully onto badly wrong attribution, and three measured pivots. Stress 00 set the doctrine — green diagnostics ≠ correct attribution — this is where the doctrine starts costing money.

In every act the ground truth is the model's own estimand — counterfactual zero-out on the noiseless structural mean — so the gap between truth and compute_counterfactual_contributions is pure model error, never an apples-to-oranges artifact. Recorded scorecard: tests/synth/results/stress_matrix.md (see the hub scorecard).

Act 1 — Carryover: the kernel the model cannot draw

The world. adstock_misspec keeps everything from the clean control — same flighting, same concave saturation, same Gaussian noise — and changes only the carryover. The true kernel is a delayed Weibull (shape 1.8–2.6, scale 6–9, support out to 26 weeks): a flight's effect builds for several weeks before it peaks (peak lags 3–7 across channels) and carries 15–42% of its mass past week 8. The model's default is a geometric kernel with l_max=8:

\[ w^{\text{geo}}_k = \frac{\alpha^{k}}{\sum_{j=0}^{L-1}\alpha^{j}}, \qquad k = 0,\dots,L-1 \qquad\text{vs.}\qquad w^{\text{wei}}_k \;\propto\; \frac{s}{\lambda}\left(\frac{k+1}{\lambda}\right)^{s-1} e^{-\left((k+1)/\lambda\right)^{s}} \]

The geometric weights are monotone decreasing for every \( \alpha \in (0,1) \): the peak is forced to lag 0 and nothing exists after week 8. The misspecification is structural — no parameter value can produce a hump. Try it:

Illustrative simulation in your browser. Defaults are TV's true kernel from the DGP (shape 2.6, scale 9 — the dark dashed hump) against the model's default geometric window. Drag α anywhere you like: the geometric family always peaks at lag 0 and always decays monotonically. Raising α toward 1 flattens the kernel — it still cannot put its mass away from lag 0, and everything in the shaded region beyond l_max simply does not exist for the model.

The naive fit, graded

The notebook fits the stress-matrix spec honestly — parametric adstock, default per-channel AdstockConfig.geometric() at l_max=8, NumPyro, 500 draws × 2 chains — and grades against truth.

What happened. With the true effect arriving 3–9 weeks after the spend, a lag-0-peaked kernel can't line media up with its response. The regression does the only thing it can: it routes the orphaned media signal into the baseline — trend and seasonality soak up the slow carryover hum — and shrinks the channels toward zero. The one channel whose true kernel is closest to geometric (Search: shape 1.8, an early modest hump) survives; the long-memory channels are nearly erased. The fitted geometric kernel piles its mass on lags 0–1 while the true kernel hasn't even peaked yet.

The pivot — a kernel that can represent the truth

Per-channel AdstockConfig.weibull(l_max=26), consumed in-graph by the core model. Two lessons here, and the first is a trap the framework itself had to fix:

Attempt 1 — flexibility with an off-target prior fails. The framework's old default Weibull scale prior was a fixed Gamma(2, 1): mean 2 weeks, almost no mass past week 5, regardless of the lag window. Asking it to find a scale-6–9 kernel over a 26-lag window produces a sampler fight — the notebook reproduces it by passing the old prior explicitly, and divergences spike and/or r-hat degrades past 1.05. A diagnostic finally fires, but it is reporting a prior–likelihood fight, not which kernel is true.

Attempt 2 — the current default. AdstockConfig.weibull(l_max=26) now gives scale ~ Gamma(2, 2/m) with mean \( m = \max(2, (L-9)/2) \) — mean 8.5 weeks at \( L=26 \), the decade the window itself implies. Same data, same likelihood, same flexibility, no hand-tuned priors — and it samples cleanly (r-hat < 1.05, zero divergence storm).

Uncertainty in the kernel — not just its mean

A posterior-mean kernel hides the most decision-relevant fact about carryover: how wide the plausible kernel family still is after 156 weeks of data. The notebook rebuilds the full posterior kernel band per channel from the raw draws and lays the truth on top. The figure below recreates that exhibit for TV:

Illustrative recreation in your browser (seeded simulation of the notebook's figure geometry). The measured result it depicts: the Weibull pivot's 90% posterior kernel band covers the true kernel at 92–100% of lags per channel, while the naive geometric band is narrow around a shape that cannot bend — it excludes the truth at the truth's own peak lag for every channel. Wide-and-honest beats narrow-and-wrong: the flexible kernel's uncertainty is the honest statement of what 156 weekly observations can know about carryover.

Choosing a kernel when you don't know the truth

In real life there is no true_contribution. Both fits store pointwise log-likelihood (idata_kwargs={"log_likelihood": True}), so az.compare(..., ic="loo") ranks the kernels on predictive grounds — and it prefers the Weibull kernel decisively (the elpd difference exceeds twice its standard error). Functional-form choice can be data-driven, but only if you fit the candidates and compare; the default never volunteers that it is wrong.

Measured improvement, naive → pivot (notebook seeded run): the pivot roughly halves the 88% median error and doubles coverage — with zero divergences and no hand-tuned priors — but does not reach zero: a 26-week kernel is still weakly identified from three years of weekly data, and the build's asserts gate exactly that honesty (error drops by >20 points yet stays above 20%). The remaining error lives in wide, truthful posteriors instead of confident wrong ones. That trade is the win.

Act 2 — Saturation: the S-curve the model flattens into a ramp

The world. saturation_misspec swaps one ingredient: the true response is an S-shaped Hill curve — Hill coefficient ≈ 3, half-saturation at 0.40–0.45 of max spend. Below the threshold, spend does almost nothing; above it, response climbs steeply, then plateaus. The model's default is the strictly concave logistic curve, steepest at zero — exactly backwards in the threshold region:

\[ f_{\text{hill}}(x) = \beta\,\frac{x^{s}}{x^{s} + K^{s}} \qquad\text{vs.}\qquad f_{\text{1-exp}}(x) = \beta\,\bigl(1 - e^{-\lambda x}\bigr) \]

And the spend sits mostly below the threshold: roughly three-quarters of all weeks live in the region where the truth says spend did nothing and every concave curve is at its steepest. Try to match the S-curve with the concave family:

Illustrative simulation in your browser. The dark dashed curve is TV's true Hill response (K = 0.45, s = 3); the tick marks along the bottom are a seeded draw of the normalized spend distribution (~3/4 of weeks below the threshold band, shaded). With logistic selected, no λ reproduces the flat toe: a concave curve is steepest exactly where the truth is flattest, so low-spend weeks get credit they never earned. Switch the family to Hill and two parameters express the threshold immediately.

The naive fit, graded

Measured — recorded matrix row saturation_misspec (per-channel detail, tests/synth/results/stress_matrix.md). Truth vs estimate: TV +65%, Search +11%, Social +53%, Display +33% — total media +38%, coverage 25% (truth outside the 90% interval for three of four channels). Contribution error here is not noise: it is the curve mismatch integrated over the spend distribution, concentrated where the spend rug meets the threshold.

The pivot — flip the channel's saturation config to Hill

One config line per channel, same model everywhere else, same grader. Measured (notebook seeded run, gated by the build's asserts): median error drops from ~43–46% to under 15%, the aggregate bias at least halves, and the half-saturation points are genuinely recovered — median \( |\hat{K} - K| < 0.12 \). The honest residual is TV: its heavy carryover smears adstocked spend across the threshold, blurring the S-bend, so TV stays over-credited by > 30% even after the pivot. Three channels land near truth; the fourth needs spend variation it never got — or an experiment (see stress 03 for the lift-test move).

Uncertainty in the response shape

The better question than "did the posterior mean land near the truth?" is "does the model's full posterior response band contain the truth?" Pushing every posterior draw (β jointly with its curve parameters) through the response function gives a 90% curve band per channel:

Illustrative recreation in your browser (seeded simulation of the notebook's band geometry for TV; the naive curve is genuinely least-squares-fit to Hill-truth responses over the simulated spend, so its miss pattern is structural, not drawn by hand). The measured result it depicts: mean truth-coverage of the posterior response band goes ~58% (1−exp) → ~86% (Hill) when the family flips. The naive band is the quiet scandal: narrow and truth-excluding through the threshold region — precise about a curve geometry its family cannot produce.

The identifiability bill: Hill costs one extra free parameter per channel (K and s vs a single λ). compute_parameter_learning confirms the data paid for them — most shape parameters contract strongly against their priors. That check matters every time you add flexibility: contraction tells you the data spoke; the posterior mean tells you what it said.

Act 3 — Spend outliers: one bad cell in a spreadsheet beats your sampler

The world. spend_outliers is the most mundane scenario in the series and the most vicious row in the matrix (recorded coverage: 0%). Ordinary sales from ordinary spend — then the observed spend is corrupted with a single ~15× data-entry spike per channel: a mis-keyed invoice, a units error. The model normalizes each channel by its training max, so one phantom week rescales the entire series for every other week.

Illustrative simulation in your browser — a seeded spend series with the one ~15× phantom week marked. In the DGP, truth is computed on the un-spiked spend: the spike never happened, but the model's normalizer believes it did.

Illustrative simulation in your browser, mirroring the notebook's histogram. Under the spiked max, every real week collapses into the bottom sliver of the saturation input — the largest real week sits below ~0.07 of max (measured in the notebook: < 0.1) — so the curve is only ever evaluated in its near-linear toe, the channel cannot exhibit curvature, and the compressed dynamic range attenuates everything. Toggle to the winsorized max and spend occupies the full curve again.

The naive fit, graded

Measured — recorded matrix row spend_outliers (per-channel detail): TV −49%, Search −29%, Social −23%, Display −70%; median error 39%, total media −41%, and not one channel's truth lands inside its 90% interval — coverage 0%, with r-hat 1.01 and zero divergences.

The pivot — a 30-second pre-flight screen, then winsorize

The screen: max/median per channel — in this world the spiked channels read > 50× where clean channels run ~5–20× — then the gap between the max and the second-largest week. A genuine heavy week sits in a continuum; a data-entry error sits alone, an order of magnitude above its neighbor. In the notebook the screen flagged exactly the four injected spikes with zero false positives (every flagged week's max-vs-second-max gap > 5×). The fix: cap the flagged week at the second-largest observed value (or verify it against the invoice and drop it), then refit.

Measured improvement, naive → pivot (notebook seeded run, gated by the build's asserts): median error 39% → under 30%, total-media error −41% → better than −30%, coverage 0% → at least 50%. One capped cell per channel undoes the bulk of the collapse; TV keeps a modest residual under-estimate at this fit budget — winsorizing repairs the scale, not every last percent. Run the screen before every fit: the framework normalizes by the training max and will not warn you.

Three functional-form failures, side by side

All three pivots improved both error and coverage — and none reached zero error. That is the honest shape of functional-form repair: you buy back most of the damage, and the model starts telling the truth about what remains.

Naive columns: recorded matrix (tests/synth/results/stress_matrix.md, re-recorded 2026-06-10) and notebook seeded runs. Pivot columns: notebook seeded runs — stated at the bounds the build script's asserts gate, so a clean re-bake guarantees them.