Stress Test 03: Confounding & Selection

Where stress 01 and stress 02 attacked the model's functional form, this notebook attacks its causal structure — the assumptions no amount of curve flexibility can rescue. Budgets that chase latent demand inflate the chasing channel by roughly +110% while every diagnostic stays green. "We controlled for it" with a noisy proxy still leaves ~+35% residual bias. Near-perfectly collinear spend turns the channel split into a posterior seesaw. And the only thing that fully repairs a confounded channel is a number bought from outside the dataset: one lift test pulls Search from +110% to +14%. This page mirrors nbs/stress/stress_03_confounding_and_selection.ipynb.

Every world below comes from src/mmm_framework/synth/dgp.py: a clean core world drawn from the model's exact structural family plus exactly one causal violation, with the true counterfactual contribution of every channel recorded. The arc per act is the series template — world → naive fit → truth gap → diagnostics → pivot → measured improvement. The doctrine was set in stress 00: green diagnostics validate the computation, never the causal claim.

Act 1 — Demand-chasing: the #1 silent failure

The world. unobserved_confounding is the clean control world with one change: a latent demand signal (seasonal + growth + AR(1)) pushes both the baseline and the budgets of the demand-chasing channels (Search and Social). Marketers do this on purpose — bid harder when the category is hot. The model never sees the demand series; it only sees spend and sales moving together, and it pays demand's credit to the channels that chased it.

The spend-vs-demand correlations look modest in EDA — nothing a screen would scream about. The bias is large anyway, because confounding strength is the product of both paths: \( \text{bias} \propto \mathrm{corr}(\text{spend}, D) \times \gamma_{D \to \text{sales}} \) — the same demand also drives the baseline hard, so the spend-side correlation alone understates the damage.

Measured — per-channel recording from tests/synth/results/stress_matrix.md (re-recorded 2026-06-10, seeded NumPyro at matrix fidelity): Search reads +69% with the truth outside its 90% interval; the non-chasers are fine. The magnitude is fidelity- and defaults-dependent — the notebook's original seeded run logged ~+110% on Search (the series punchline), earlier matrix recordings reached +153%, and the 2026-06-10 trend-prior loosening roughly halved the bias at notebook fidelity (~+40%). What never changes is the shape: the bias lands surgically on the demand-chasers, same model, same data quality, same priors.

Act 2 — The noisy proxy: "we controlled for it" is not a fix

In real life you never observe demand itself; you observe a proxy — a category index, a search-trends panel, distribution data. The confounding_controlled twin of act 1's world adds exactly that: the same latent demand, observed as CategoryDemand = 100·demand + noise, included as a control and marked CONFOUNDER so it keeps its wide prior.

It helps a lot — and not enough. In the notebook's seeded run the worst chaser goes from ~+110% naive to ~+35% controlled, with the truth still outside its interval; the 2026-06-10 matrix recording has the same pair at +69% → +50%. The reason is mechanical: a proxy measures the confounder with error, and measurement error only partially closes the back-door. With proxy reliability \( \lambda = \sigma_D^2 / (\sigma_D^2 + \sigma_\epsilon^2) \), the residual confounding bias scales roughly with \( (1 - \lambda) \) of the naive bias. Drag the slider to see how proxy quality sets the floor:

Illustrative simulation in your browser — a seeded 156-week omitted-variable simulation (40 replicates per noise level) of regressing the KPI on spend plus a noisy proxy of the confounder; the dashed curve is the closed-form attenuation rule, residual bias ≈ (1 − reliability) × naive bias. The anchors are measured: the naive ceiling is the notebook's +110% Search overstatement, and the marked operating point is the notebook's own proxy world, which leaves ~+35% residual bias with truth outside the HDI (matrix recording: +69% → +50%, still flagged a silent failure). Nothing in the data tells you where your proxy sits on this curve.

Act 3 — Collinearity: the model can't split what the buy didn't split

The world. multicollinearity keeps the clean world's deep flighting pulses — total media is just as identifiable — but drives every channel from one shared burst calendar (channels bought together, one trafficking plan; mean pairwise spend correlation > 0.85). Each channel keeps its own true effect; only the separability is gone.

The recorded result (matrix fidelity, 2026-06-10): the point estimates shuffle real credit between channels — median error 39%, worst channel +60% — while the combined-media total lands within −5% of truth and every channel's truth stays inside its (very wide) 90% interval: 100% coverage. The intervals widen honestly — in the notebook every channel's interval is wider than its own estimate.

Measured — multicollinearity per-channel table, tests/synth/results/stress_matrix.md (2026-06-10). r-hat 1.01, 0 divergences. silent failure on the split — but listen to the intervals.

The posterior seesaw: the joint posterior's own confession

Wide per-channel intervals are only half of what the posterior is telling you. The other half lives in the joint distribution: sum each channel's contribution over time per posterior draw and correlate the totals across channels. On the clean control (stress 00 §3.1, distinct flighting calendars) those correlations sit near zero. Here, every channel was bought on one calendar, and the posterior says so in its own language: every pairwise correlation is negative (worst pair ≈ −0.5) — draws where one channel takes more credit are draws where its shelf-mates take less. The split is a seesaw: conserved in total, arbitrary in allocation.

Illustrative recreation of the notebook's posterior-correlation heatmap (channel contribution totals, per draw). The measured punchlines it is built to match (seeded asserts in the notebook): on the collinear world every off-diagonal is negative — no pair above +0.05, the worst pair below −0.35 (≈ −0.5), at least three pairs below −0.20; on the clean control all pairs sit near 0. This check needs no ground truth — it is available in every real engagement.

Illustrative recreation of the seesaw drawn as draws: the worst pair's per-draw contribution totals, a cloud with correlation ≈ −0.5. The cloud's centers are the measured point estimates (Social 7,252; Display 5,580) and the diamond is the measured truth (5,147; 3,485) — inside the joint cloud, as the 100% coverage says. Conditioning on one coordinate pins the other: that is why one lift test on either member collapses the seesaw.

Act 4 — 25 candidate controls: selection without re-opening the back-door

Acts 1–3 had one or two controls. Real briefs arrive with a spreadsheet: macro indices, weather, pricing, category trackers, competitor pulls — some causal, most not, several suspiciously co-moving with your own media. Including everything costs precision (25 coefficients on 156 weeks); dropping the wrong one re-opens act 1. Which controls you keep is a causal decision; how you regularize the keepers is a statistical one. The dense_controls world tests the machinery for splitting that difference. Its 25 candidates, each group a designed trap:

• demand_proxy — a genuine confounder: latent demand drives sales and the TV/Search budgets chase it (act 1's mechanism, buried in the pile);
• price, weather — genuine precision controls (real, modest effects);
• noise_1..18 — pure-noise random walks, true effect zero — but random walks will spuriously correlate with a trending KPI on 156 weeks;
• decoy_tv/search/social/display — media-tracking decoys: noisy copies of each channel's spend (corr ≈ 0.9) with no causal effect — designed bait for any selector that picks controls by correlation.

The contest. Four fits, identical except for how the controls enter:

The roles are asserted spec, not learned (ControlVariableConfig.causal_role). The selection machinery's contract (model/base.py:_build_control_betas): confounder-role controls keep the wide un-shrunk Normal no matter what — act 2 showed what even a noisy proxy costs; a shrunk one is a half-open back-door.

Who survives selection? The shrinkage profile

Illustrative recreation of the notebook's beta_controls shrinkage profile (|posterior mean| on a log scale, grouped by true role). The measured mechanics it encodes (seeded asserts): the horseshoe shrinks the noise pile and the decoys at least 2× harder than the plain Normal (in the run, several-fold), the exempt confounder keeps its coefficient essentially untouched (|β| > 0.3, not shrunk vs the no-selection fit), and demand_proxy is the lone survivor of the 90%-HDI-excludes-zero test. The known cost, also measured: the horseshoe over-shrinks the genuine precision controls (price/weather land well below their no-selection estimates).

Honest findings — including the one the world wasn't designed to show:

• The decoys did not steal media's credit — under any prior. Built as the trap (corr ≈ 0.9 with spend, zero effect), they get near-zero coefficients even with no selection. Why: the decoys track raw spend, but the regressor the likelihood sees is the adstocked, saturated transform of spend — a linear shadow of a channel is not collinear with the channel's transformed effect. A genuine robustness property, with a boundary: a decoy that mirrored your flighting and carryover would not be separated this way.
• The real cost of the kitchen sink is precision, not theft. Twenty-two junk coefficients don't bias a role-correct model much — they blur it: the weakest channel's interval roughly doubles versus the oracle.
• Selection bought the oracle's accuracy without the oracle's knowledge. Horseshoe (and spike-slab, nearly identically) matches or beats the oracle's media recovery while restoring most of the lost interval width. expected_nonzero≈4 encodes a prior belief, not a discovery.
• One channel stays under-credited in every variant, oracle included (matrix recording: Display −52%, outside its CI — the row that flags dense_controls as a silent failure, med 16% / worst 52% / coverage 75%). Its bias is a property of this world's low SNR for that channel, not of the control strategy. Selection fixes the control problem; it does not manufacture identification that isn't there.

Act 5 — The real fix: experiment calibration

Act 2 ended with an uncomfortable residue: even with the proxy controlled, the worst channel keeps a double-digit bias, and nothing in the data can locate the floor proxy noise sets. Act 3 ended with an honest shrug. Both point the same way: the information that identifies a confounded or collinear channel is not in the observational data — you have to buy it.

A geo lift test measures a channel's incremental effect by randomization, immune to demand-chasing by design. The framework folds it into the model as an in-graph likelihood term (add_experiment_calibration, called before fit): the measured ROAS becomes a data point whose model expectation is the channel's own estimand, expressed through the same beta, saturation, and adstock parameters the time series informs. Experiment and time series negotiate in one posterior. The notebook simulates a well-powered test the way a real one works: it measures the truth with a 10% standard error.

Measured in the notebook (seeded run) — the series punchline: one ROAS lift test on Search pulls its overstatement from +110% to +14%, with the truth back inside a sharply tighter interval (the seeded assert pins the contraction at < 0.7× the naive width). The dashed line is zero error — the truth.

What to take away