Stress Test 05: The Gauntlet

The capstone. Guides 00–04 broke the model's assumptions one at a time; this one breaks them all at once and walks the entire analyst workflow — first data pull → EDA → naive fit → triage → structural pivots → an experiment → a budget answer you could defend. The naive fit's channel ranking comes out fully inverted, with Search roughly 8× overstated. The recorded recovery arc ends at Search ROAS 5.59 → 0.65, against a truth of 0.66 — and the untested channels stay wrong, because calibration is surgical, not contagious.

The world: everything broken at once

aurora_kitchen_sink (from src/mmm_framework/synth/dgp.py) is Aurora Coffee Co.: two years of weekly data, four channels — and four simultaneous violations, fit demand-blind (the demand proxy is the column nobody exported):

1. Latent-demand confounding — Search and Social budgets chase demand, which also drives sales (stress 03's killer);
2. Mediation — TV and Display work almost entirely indirectly, by building awareness (~97–99% of their true effect routes through it; stress 04);
3. Cannibalization — Cold Brew steals from Original each summer (invisible in a single-KPI view);
4. Hill saturation — the true response curve is not the default logistic (stress 01's form trap).

The naive fit: a ranking turned upside down

First, what the deadline ships: the default model on the data as pulled — KPI, four spend columns, a price control. No demand proxy, default adstock, default logistic saturation. It converges. It fits beautifully (R² ≈ 0.92 — hold that thought for the fit-quality trap). And its attribution is the truth, almost exactly reversed:

Measured — recorded matrix row for aurora_kitchen_sink (tests/synth/results/stress_matrix.md, PyMC 500×500×2, seed 0, re-recorded 2026-06-10). The naive fit hands TV's contribution to Search: TV 13,053 → 1,057 (−92%), Search 2,193 → 7,091 (+223%), Display 6,037 → 1,718 (−72%), Social 1,508 → 2,899 (+92%). Coverage: 1 channel in 4. The estimated ROAS ranking equals the naive corr(spend, sales) ranking — [Search, Social, Display, TV] — the dashboard mirage with credible intervals on it.

Reading the alarms — what "loud" actually looks like

This is the one world in the matrix with any divergences — and even here the alarm volume is unreliable. An earlier recording of this same row (code state 2026-06-08) was genuinely loud: r-hat 1.11, 108 divergences. Re-recorded on later code, same config and seed: r-hat 1.02, 11 divergences. The notebook's fast NumPyro fit is quieter still. Same model, same data, same ~90% wrongness every time; the siren did not survive a code revision. Two lessons the capstone insists on:

• When alarms fire, they are a gift — don't bribe them. Bumping target_accept to 0.99 until the divergences stop is treating the smoke detector, not the fire. Diagnose before re-tuning.
• When alarms don't fire, you've learned almost nothing. Naive variants of this same fit were measured at zero divergences — still ~4–8× wrong on Search. Every silent failure in the 16-world matrix sampled with r-hat ≤ 1.02 and zero divergences. Convergence checks the sampler, never the attribution.

Decision point: whichever volume you got — a converged-looking fit that mirrors a correlation table you already know is confounded, or a noisy fit with warnings — you do not ship. You work the recovery ladder. Each step below is measured.

Step 1 · The EDA pre-flight — half the failures are visible before sampling

The pre-flight takes minutes and no MCMC. On this world it flags, before any model exists:

• Outlier screen — clean. Every channel's spend max/median is well under the ~8× danger line, so stress 01's normalization-crushing spike failure is ruled out for free.
• Identifiability (CoV) — two red flags. TV and Display are nearly always-on (spend CoV ≈ 0.11): almost no variation to learn from. Expect prior-dominated, wide posteriors no matter what you do observationally. Flag them for experiments now, before any fit.
• The demand-chasing screen — the killer. Search's spend correlates ≈ 0.85 with sales (a dashboard would crown it) and ≈ 0.93 with the category-demand index, which itself correlates 0.86 with sales. The second number poisons the first: when spend chases demand, corr(spend, sales) measures the bidding rule, not the ad. TV pairs the lowest naive correlation (≈ 0.29) with — the grader knows — the highest true ROAS.
• Causal roles. The demand proxy is a confounder (must include, don't shrink); price is a precision control; the awareness tracker is a mediator — exclude it as a control or you destroy TV/Display's effect (stress 04). The mediator call comes from institutional knowledge — awareness is what TV is bought to do — not from a correlation screen; TV's spend is too flat for the screen to see it.

Play with the mechanism that generates the mirage — the true ROAS below never changes; only how aggressively the budget chases demand does:

Illustrative simulation in your browser (seeded PRNG) — not notebook output. A channel with a fixed true ROAS of 0.66 whose budget chases demand: as chasing strength rises, corr(spend, sales) climbs toward 0.9 while the ad's true effect stays unchanged. A naive correlation far above any plausible ROAS is a confounding tell, not a performance read. The measured world's numbers: corr(Search spend, sales) ≈ 0.85, corr(Search spend, proxy) ≈ 0.93.

Step 2 · Triage — which symptoms point where

Don't re-tune; diagnose, with the toolkit from stress 00. On the naive fit, each instrument reads (all analyst-visible, measured in the seeded notebook run):

Triage verdict: two independent observable fingers — the prior-data conflict on the demand-chasing channel, and demand structure in the residuals — point at the same omitted variable. The first pivot is structural, and the EDA already named it: close the back-door.

Step 3 · Structural pivots, each one graded

Pivot (a): add the demand control — it helps, and it isn't enough

Refit with the category-demand index as a control. Measured (seeded NumPyro run, 400×400×2): Search's ROAS collapses 2.6 → 1.0 (−62% from one defensible spec change), and the stage's median contribution error falls ~95% → ~65%. But read what came with the new midpoint:

• The interval is the result. Search's 90% interval is now roughly [0.1, 2.4] — it contains "kill the channel" and "double it". Not a number you take to a budget meeting.
• Controlled ≠ identified. Search's spend correlates ≈ 0.93 with the control just added: after conditioning, almost no independent Search variation remains. The model splits a shared signal between the beta and the control, and priors plus noise decide the split. This is stress 03's noisy-proxy lesson at full scale.
• The mediated channels barely move (TV ≈ 0.21 → ≈ 0.4 on a tiny base), still with intervals several times their estimates — the EDA's flat-spend flag stands.

Pivot (b): the functional-form pivot — it reshuffles, it doesn't rescue

The stress-01 discipline: refit under a deliberately different, defensible response family — in-graph parametric adstock plus per-channel Hill saturation, \( \beta\,\dfrac{x^{s}}{x^{s}+K^{s}} \) in place of the default \( \beta\,(1-e^{-\lambda x}) \) — and read it as a spec-sensitivity experiment. Measured: the form change moves every channel's estimate by ≤ 10%, while the demand control had moved Search by 62%. Nothing observable (fit, intervals, diagnostics) prefers either form. Conclusion the analyst can defend without truth access: the response form is not the binding constraint — identification is choosing these numbers.

Pivot (c): acknowledge the mediation — and its limits

TV and Display still print sub-1 ROAS midpoints inside intervals several times that size. The structural case needs no truth: the DAG routes their effect through awareness — a pathway this single-equation model can only smear into the baseline — and their spend is nearly flat (CoV ≈ 0.11). The right structural model is a nested/mediation MMM with the awareness tracker as mediator outcome — that is stress 04's subject, including the trap that an extension adds unidentifiable knobs when mediator data is weak. And even a perfect mediation model cannot conjure identification out of a CoV-0.11 spend series.

Decision point: the observational program has hit its identification ceiling — the demand-chasers are controlled-but-collinear, the brand channels are mediated-and-flat. More observational modeling from here is knob-turning. The next dollar of effort buys more as a randomized experiment than as a fifth structural variant.

Step 4 · The experiment — surgical, not contagious

Aurora runs geo-holdout lift tests on the two cheapest-to-test channels — Search and Social (always-on auction media; a clean two-month holdout reads out fast). Each readout (Search ROAS 0.66 ± 0.07, Social 0.49 ± 0.07 over the test window) is folded in as an in-graph likelihood term on the model-implied ROAS estimand (experiments= / add_experiment_calibration) and the model refit. Measured:

• The calibrated channels land. Search ≈ 0.62, Social ≈ 0.48 in the seeded notebook run — truth (0.66 / 0.49) inside tight intervals. In the recorded full-fidelity arc, the same move is Search ROAS 5.59 → 0.65 against a truth of 0.66.
• The stage's median error falls to ~31% (from ~95% naive).
• The untested channels stay wrong. TV remains ~−75% under-credited; its interval never tightens. Calibrating Search and Social did nothing for TV and Display — anchoring one channel is not evidence about another.

Measured in the notebook (seeded run) — relative interval width (90% HDI width ÷ estimate) after calibration: Search 0.40, Social 0.68 — versus TV 2.10 and Display 2.32, which never drop below ~2.1 at any stage. Search's width was 1.64 one stage earlier: the experiment, not the spec work, is what tightened it. Above the dashed line, the interval spans "kill the channel" to "double it". Interval width is the one analyst-visible feature that tracks the truth-gap — believe the width, not the midpoint.

The ROAS ladder — the whole drama in one chart

Per-channel ROAS estimates at each workflow stage, with the grader's truth as a dashed line. Step through the stages, or see the whole trajectory at once:

Measured anchors from the seeded notebook run (NumPyro 400×400×2, seed 0; build-script probe 2026-06-10): Search 2.6 → 1.0 → ~1.0 → 0.62 (true 0.66); TV 0.21 → ~0.4 → ~0.4 → 0.55 (true 2.14); Social ends at 0.48 (true 0.49); stage median errors 95% → 65% → 71% → 31%. Cells without a recorded point value (Social and Display at intermediate stages, the form-pivot column) are placed inside the probe's recorded constraints — the form change moves every channel ≤ 10%, the mediated channels barely move at any stage — and should be read qualitatively. The story to read: Search is the skyscraper two structural pivots only half-demolish and one experiment levels; TV and Display never reach their dashed lines, because no observational pivot — and no calibration of other channels — identifies a flat-spend, ~98%-mediated channel.

The fit-quality trap — four excellent fits, four different stories

If interval width is the analyst-visible signal that works, here is the popular one that doesn't: how well the model traces the sales curve. All four stages — from the demand-blind disaster to the calibrated finale — produce fits a slide deck would celebrate:

Measured in the notebook (seeded run) — R² by stage: 0.918, 0.930, 0.929, 0.927 (total spread 0.011); 90% predictive-band coverage 0.91–0.92 at every stage — while the truth-gap runs 95% → 65% → 71% → 31%. The line you watch in a readout meeting is flat; the number that matters falls by two-thirds. Fit quality and attribution identifiability are different axes: a predictive check cannot choose between attribution stories that all predict the same curve.

Where did the credit actually move? The decomposition (additive by construction) tells it: the naive model's media wedge is 16% of sales with Search alone at 9.8%; the final model's is 11% with Search at 2.4% — the weekly demand swings the naive model credited to Search's beta now flow through the demand control (std of the controls component 14.6 → 23.2; std of Search's media component 18.7 → 10.5). And the residual-vs-proxy correlation falls +0.246 → +0.007. The grader adds the sting: the true media share is 26% — even the calibrated final model under-credits total media, because TV/Display's mediated effect is parked in the baseline, invisible to this equation.

The decision table — the series in one artifact

Symptom → likely cause → cheapest check → pivot → where it's demonstrated. This is the table to copy out of the whole series. Filter by symptom category:

Compiled from the notebook's closing artifact. Bold 05 entries are demonstrated live on this page's world; linked guides carry the full treatment of each row.

The fix ladder — doctrine recap

Identification problems are solved by design, not by sampling longer. Every recovery in this series, ordered by where the leverage actually was:

1. Estimand first. Pin down what "contribution" means (counterfactual zero-out) before reading any posterior. If you can't say what number would be wrong, no diagnostic can save you.
2. EDA pre-flight. The mirage table, the CoV screen, the max/median outlier screen, the causal-role DAG — minutes of work that flagged every failure this world contains before the first sample was drawn.
3. Structural pivots — chosen from symptoms, never from disliked answers. The demand control moved the headline channel 62% (it helps; it doesn't identify). The functional-form pivot moved everything ≤ 10% and made the truth-gap slightly worse — forms matter only when form is the problem.
4. Experiments calibrate what structure can't. Search 5.59 → 0.65 against a truth of 0.66 — surgical. And not contagious: the untested channels stayed wrong, which is precisely why the deliverable ends with a costed test plan, not a full table of midpoints.

The pre-specification discipline that survives this notebook: lock the spec (causal roles, forms, priors, which channels get calibrated) before looking at ROAS; pivot only on measured diagnostics; and always escalate to an experiment for flat-spend channels (CoV ≲ 0.3), demand-tracking channels, collinear pairs that matter, and any channel about to receive a large budget move.