Stress Test 04: Extension Traps

The series' recurring prescription is when the estimand is wrong, change the model. This guide stress-tests the prescription itself. The framework's extension models — NestedMMM (mediation), MultivariateMMM (multiple outcomes), CombinedMMM (both at once) — are real cures for real structural biases, and every one of them ships with new latent paths the data may not identify. When that happens the extension doesn't fail loudly: it returns a confident posterior whose shape came from the prior, dressed in the language of structure. Every claim below is measured against the Aurora synthetic world (nbs/builders/aurora.py), whose true per-channel ROAS and mediated shares are known exactly.

Act 1 · The mediation problem: the channel the base model cannot see

Aurora's TV and Display sell almost nothing directly. They build brand awareness, and awareness drives sales — their true mediated share is ≈ 1, and their true total-effect ROAS is ≈ 2 (TV 2.14, Display 2.11). For a budget decision the right estimand is the total effect, direct plus indirect:

The structure of Aurora's brand channels. Almost all of TV's and Display's effect flows through awareness; the dashed direct path is nearly zero in the generative truth.

Fit the base BayesianMMM — with the demand confounder properly controlled, so nothing from stress 03 is leaking in — and TV comes back at ROAS 0.33 against a true 2.14, with Display likewise recovered at under half its truth. Where did the effect go? Not "into the missing mediator variable". The mechanism is nastier: TV is an always-on channel with heavy carryover, so its saturated, adstocked exposure — the regressor the coefficient actually multiplies — barely moves week to week (measured CV ≤ 0.13 across every plausible carryover value). A near-constant regressor is exchangeable with the intercept and trend: the data cannot credit it, so the baseline absorbs the brand effect. No amount of better priors fixes a constant regressor — this is an information problem, and that decides what an extension must bring to actually help.

The naive "fix" — "awareness predicts sales, add it as a control" — is the textbook bad-control trap (awareness is post-treatment; conditioning on it blocks the indirect path). Measured here, though, it changes almost nothing: TV moves by about −0.04 ROAS and the awareness control's coefficient is ≈ 0.12. The trap doesn't visibly bite because there was nothing left to steal — the baseline had already taken TV's effect. Two failures masking each other: an A/B "with vs without the control" robustness check returns "no difference, the control is safe", and tells you nothing about the rung you should actually be standing on.

NestedMMM: the cure that over-credits the sibling

NestedMMM changes two things at once. It changes the estimand — TV and Display route through a latent awareness series \(m_t\), so each channel's total coefficient becomes the product-of-coefficients indirect path plus the direct one, \(\beta_{c\to m}\,\gamma_m + \delta_c\). And it changes the information: the monthly awareness survey enters the likelihood and anchors \(m_t\) — new data the base model never used, which is exactly the currency the rung-(a) failure demanded.

Toggle the three configurations below — they are the notebook's actual fits, on the same data, with the measured numbers.

Measured in the notebook (PyMC 5 seeded run) — true total-effect ROAS (dark) vs each fit's estimate, for the two mediated channels. Base model: TV 0.33 vs true 2.14; Display 0.67 vs true 2.11 (under a third of its truth) — unchanged under PyMC 6. Correct mediator (PyMC 5): TV 2.29 [1.97, 2.62]; Display 5.47 [4.76, 6.15]. Under PyMC 6 the same fit only partially recovers: TV total-effect ROAS ~0.23 with a 90% interval spanning zero, Display ~0.48 (proportion mediated ~0.69 / ~0.40) — the wide intervals are the honest signal, not a rescue. Wrong mediator (PyMC 5): TV 2.36 and Display 5.06 — each as plausible as the correct fit's. 94% HDIs where shown.

Read what the nested fit does and does not buy you. The survey anchors the awareness path and the estimand becomes the total effect, so the model finally sees the mediation the base fit missed — proportion mediated ~0.69 (TV) and ~0.40 (Display), where the base model credited essentially nothing. But under PyMC 6 the ROAS is only partially recovered: TV's total-effect ROAS lands near 0.23 with a 90% interval that still spans zero, and Display near 0.48 — both far below their true ≈ 2.1, both weakly identified. The honest signal is the width of the intervals, not a point rescue. (On PyMC 5 the same notebook read as a near-rescue — TV 2.29 covering the truth, Display over-credited ≈ 2.6× — a confident split that PyMC 6's shifted posterior no longer supports.)

Why? The two channels share one mediator, and the data pins only their joint path. Compare the structural parameters to truth:

Measured in the notebook (seeded run) — the structural parameters are not recovered; only (some of) the products \(\beta\gamma\) faced the data. Never quote \(\beta\), \(\gamma\) point estimates as mechanism.

TV's edge into awareness is unidentified for the same reason the base model failed: its exposure is near-constant, so \(\beta_{\mathrm{TV}}\) is exchangeable with the mediator's intercept. On PyMC 5 the prior split the brand credit roughly evenly between the two channels feeding the shared mediator, and Display's smaller spend then inflated its ROAS into a confident over-credit. Under PyMC 6 the same non-identification surfaces as width instead: the posterior no longer settles on a confident split but stays wide, with TV's total-effect interval spanning zero. Either way, the allocation between two channels feeding the same mediator is not data-identified. The unidentifiability didn't vanish — it moved into the mediator equation, one level down, where nobody was looking. Treat both the per-channel split and the per-mediator totals as weakly identified — read them with their full intervals, and lean on a per-channel experiment or genuinely decorrelated exposures before trusting either.

The counting doctrine: data moments vs structural parameters

Both traps on this page reduce to arithmetic you can do before fitting anything: count the numbers the data offers, and count the knobs the structure asks for. The cleanest case is Act 2's cross-effect. For the residual co-movement of two outcomes, the likelihood supplies exactly three moments:

\[ \mathrm{Var}(y_1), \qquad \mathrm{Var}(y_2), \qquad \mathrm{Cov}(y_1, y_2). \]

The structure asks for four parameters: the residual scales \(\sigma_1, \sigma_2\), the residual correlation \(\rho\), and the directional cross-effect \(\psi\) on the observed sibling outcome:

\[ \mu_{1,t} = a + \textstyle\sum_c \beta_c u_{c,t} + \psi\, y_{2,t}, \qquad \mathbf{y}_t \sim \mathrm{MvNormal}(\boldsymbol{\mu}_t, \Sigma). \]

With \(\psi\) multiplying the observed \(y_2\), \(\psi\) and \(\rho\) produce observationally equivalent co-movement: the data pins their total, and the split is whatever the prior says. Four asks of three. The shared-mediator split in Act 1 is the same disease with different anatomy — two \(\beta_{c \to m}\) edges constrained mostly through one product with \(\gamma_m\). And the knob bill compounds as the models do:

Measured in the notebook (seeded run) — free scalar parameters per extension model against the same 104 weekly observations. The combined model's count is after the 2026-06-10 fix removed three phantom ψ knobs (31 → 28).

Contraction ≠ importance: how to read compute_parameter_learning

The framework's anti-vacuity audit, compute_parameter_learning (diagnostics/learning.py), compares each parameter's prior to its posterior: contraction \(= 1 - \mathrm{var}_{\text{post}}/\mathrm{var}_{\text{prior}}\) asks did the data speak? The doctrine has two clauses that this notebook makes concrete:

• Contraction tells you whether the data spoke; the posterior mean tells you what it said. In the multivariate fit below, under PyMC 6 \(\psi\) does not pin to ≈ 0 — its posterior relocates to a sizeable negative (mean ≈ −1.13), barely overlapping the prior (overlap ≈ 0.03, a shift of ≈ 5 prior-sds). The data spoke through location, and what it said is prior-dependent, as the flip below shows.
• Contraction is computed within a parameterization. It certifies the data informed that parameterization — not that the parameterization asks an identifiable question. A prior-dominated split shows weak contraction: in the multivariate fits, 13–16 of 19 free parameters sat at contraction < 0.1 — posteriors that are the prior wearing a lab coat.

Illustrative simulation in your browser (smooth densities recreated in JS), quoting the measured labels from the notebook's seeded run: left, a parameter the data informed — under PyMC 6 ψ's posterior barely overlaps its prior (overlap ≈ 0.03), relocating ≈ 5 prior-sds to a sizeable negative rather than contracting; right, the typical prior-dominated knob — 13–16 of the multivariate model's 19 free parameters sit at contraction < 0.1, posterior ≈ prior.

Act 2 · MultivariateMMM: the ψ that flips sign with the prior

Aurora sells two products, and in truth summer Cold Brew pressure eats Original's sales while both ride one demand wave. MultivariateMMM offers the two couplings counted above: the directional cross-effect \(\psi\) (Cold Brew → Original) and the residual correlation \(\rho\). The moment count predicts something testable and damning: refit the same model on the same data with a different prior on \(\psi\), and the story should flip. It does:

Illustrative recreation in your browser of the notebook's measured flip (densities redrawn in JS; each curve scaled to its peak for readability). Measured punchline, seeded run: under the cannibalization prior \(\psi = -\mathrm{HalfNormal}\), the posterior collapses onto the boundary — ψ ≈ −0.0003 — and the co-movement lands in ρ = +0.41. Under \(\psi \sim \mathrm{Normal}(0, 0.5)\) on the same data, ψ = +1.80 — mechanically "Cold Brew lifts Original", worth ≈ 78% of Original's weekly sales — while ρ flips to −0.14. ψ's contraction is ≈ 1.0 under both priors.

Two fits, one likelihood, two contradictory mechanisms; the only thing that changed is the prior. And the trap inside the trap: under the cannibalization prior, \(\mathbb{P}(\psi < 0) \approx 1\) — which sounds like overwhelming evidence and is in fact a property of the prior (every HalfNormal draw is already negative). A slide saying "we are >99.9% sure Cold Brew cannibalizes Original" would survive most reviews and contain no data-derived content about the sign at all. Also worth saying plainly: the DGP does contain true substitution, and neither fit found it — a cross-effect on the observed sibling outcome can't, no matter the prior. To measure causal cannibalization, move something exogenous: cross-price, a promo experiment, a share model.

The wrong-mediator trap: a wrong mediator fits perfectly

Mediation analysis has a dirty secret: the model cannot tell a mediator from anything else that co-moves with sales. Suppose the modeler, impressed by the nested recovery, nominates the category demand index as the funnel metric — "it tracks our sales and our spend, it must be mid-funnel". In Aurora's truth that variable is a noisy proxy of the demand confounder: it causes spend (the team chases it) and sales; media does not cause it. Same NestedMMM, same channel mapping, backwards arrow. If the extension has any self-awareness, something should fire. Measured, nothing does:

Measured in the notebook (PyMC 5 seeded run) — the right-vs-wrong scorecard. The one diagnostic that reacts at all (negative contraction on the β edges) reacts identically in the correct fit: it flags scale tension, and is not a wrong-mediator detector. Under PyMC 6 the correct-mediator fit recovers only partially (proportion mediated ~0.69 / ~0.40, TV total-effect ROAS ~0.23 with a 90% interval spanning zero); the wrong-mediator fit was not re-run, but the durable point is unchanged — right and wrong mediators share the same clean-sampling signature, so fit alone cannot tell them apart.

The wrong fit's story is causally upside down — it claims TV builds category demand, when in truth demand drives TV's budget — and any budget simulation leaning on that path ("cut TV, demand falls") inherits the inverted arrow. Yet a modeler comparing it to the base fit would see "TV finally makes sense!" and ship it. Mediator validity is an external causal assumption, full stop — it is not testable from fit. It must be earned outside the model: time order, an experiment that moves the mediator, a DAG argument that survives review (stress 03's EDA tell is media predicts the candidate, not the candidate predicts sales). The extension will faithfully decompose whatever arrow you hand it, including a backwards one.

The extension decision rule

Everything this notebook measured compresses into one workflow: reach for an extension when the DAG says the base estimand is wrong, not when you want a richer story; price every new latent path before fitting (identified by countable moments, pinned by external evidence, or honestly labeled a prior assumption — there is no fourth category); audit the new parameters with compute_parameter_learning; and fit the signless prior as a sensitivity — if the story flips, the direction was never yours to report.