Stress Test 06: Geography & Hierarchy

Notebooks 00–05 stress-tested the model on national time series. Real engagements increasingly arrive as panels — KPI by geography, often by geography × product — and panels change the stress surface three ways at once. This notebook found (and fixed) a structural bug in how adstock treated stacked panel data; then it built worlds with per-geography ground truth and discovered the headline failure of the series' panel chapter: a model that is green and accurate nationally while its per-geo errors reach +330%, per-geo coverage collapses to 31%, and TV's regional ROI ranking comes out fully inverted (Spearman \(\rho = -1\)). Same posterior, same clean diagnostics — it depends entirely on the level at which you grade it.

What a panel adds, and what can go wrong with each addition:

Act 0 — The bug the panel surfaced: carryover across the geo boundary

A panel model receives its observations as one stacked vector. The MFF loader orders them period-major: (week 1, North), (week 1, South), …, (week 2, North), … Adstock is a convolution over time:

\[ a_t \;=\; \sum_{\ell=0}^{L-1} w_\ell \, x_{t-\ell} \]

where \(t\) must index weeks within one geography. Before this notebook's build, both adstock paths — the parametric in-graph convolution and the legacy two-point blend — convolved straight down the stacked observation vector. On a geo panel, an observation's "lag 1" was therefore a different geography at the same week: TV aired in the North adstocked into the South's Tuesday. A geography with zero spend in every week still received media contributions.

Nothing crashed. Sampling converged. The model simply estimated kernels and contributions for a physically meaningless quantity — the series' definition of a silent failure, this time structural rather than statistical. The notebook demonstrates it with the simplest possible probe: one impulse of TV spend in the North at week 5, zero everywhere else.

Faithful pure-JS recreation of the notebook's pure-NumPy demonstration (geometric kernel \(w_\ell \propto 0.6^\ell\), \(L=8\), 16 weeks, two geos, impulse at week 5 in the North) — same arithmetic, computed live in your browser. Pre-fix, the stacked convolution hands the South 38% of the total carryover weight — a geography that never spent a dollar — and chops the North's tail in half. Post-fix the South is exactly 0. The notebook then runs the same impulse through the shipped model: across 50 prior draws, the dark geo's largest absolute contribution is 0.000000 (measured in the notebook, seeded run).

The panel worlds

Worlds come from src/mmm_framework/synth/dgp_geo.py — balanced panels with the same estimand discipline as the rest of the series (see the hub page), except the counterfactual ground truth is recorded per geography / per cell (true_contribution_by_geo), so the model can be graded at the level the budget decisions actually happen. Helpers national_scenario() (the aggregated view) and geo_scenario(g) (per-geo slices) power Acts 2 and the split-fit pivot.

Act 1 — The panel positive control: four geographies, one truth each

geo_clean is the model's exact hypothesis space, panel edition: four geographies share every response parameter (betas, adstock, saturation, trend, seasonality) and differ in baseline offset and market size. Each geo flights on its own calendar, so the panel carries genuine cross-sectional variation. If the panel machinery is sound, recovery should be clean at the national level and inside every geography — and it is:

The notebook also slices the stress 00 uncertainty lenses by geography: per-geo posterior-predictive fit bands (the national R² of a panel fit can hide a geo it systematically misses) and per-geo contribution-path bands graded against each geo's true weekly path — the truth sits inside the 90% band in every spot-checked geo×channel pair. Inside a representable world, the per-geo bands are honest.

Act 2 — What aggregation throws away

Most teams have geo-level data and model nationally anyway. The notebook sums the geo_clean world across its four geographies — same weeks, same total spend, same causal truth — and refits. Two structural things are lost: replication (520 observations become 130; four independent flighting calendars blur into one smoother national curve) and the curve itself — each geo responds at its own saturation operating point, and a sum of four saturation curves evaluated at four different points is not a saturation curve of the summed spend (Jensen's inequality). The aggregated world is, strictly, outside the model's family even though every geo individually was inside it.

Measured in the notebook (seeded run) — 90% interval width ratio, national fit ÷ panel fit, per channel: TV 1.06×, Search 2.53×, Social 2.33×, Display 1.97× — about 2.1× wider on the median channel. The aggregate fit stays green (median error 8%, full coverage), but its worst-channel point error is 21% vs the panel's 9%.

Aggregation here doesn't break the model — coverage survives because the intervals widen to admit what was lost — but you pay twice: materially wider intervals on most channels, and the largest point errors drifting to the channels whose geo-level operating points differed most. If geo data exists, the panel fit is not a luxury; it is the cheaper way to buy precision you would otherwise need more years of history for. And note the quiet doctrine point: both fits are green; only the known truth reveals which numbers drifted.

Act 3 — The regional ROI trap: one beta, four markets

geo_heterogeneous keeps Act 1's skeleton and changes what the model cannot see: channel effectiveness now differs by geography — multipliers from 0.3× to 1.8× on the shared response. TV is a powerhouse in the North (1.8×) and nearly dead in the West (0.3×); Search inverts (0.6× North, 1.7× West). And budgets chase performance the way real regional allocation does: geos where a channel works get a larger share of its budget, so spend and response heterogeneity are coupled by construction.

In the default pooled fit the hierarchy is an intercept per geo and one global beta per channel, so per-geo effectiveness falls outside that fit's hypothesis space — the pooled beta lands near a spend-weighted average, and every geo's readout inherits it. It is not structurally impossible: the opt-in HierarchicalConfig.vary_media_by_geo=True partial-pools each channel's beta across geos (_build_channel_betas_geo), the in-model version of the per-geo-betas-with-shrinkage fix this act motivates.

Read that strip again: nationally this model is excellent — small errors, full coverage, clean sampler. An analyst grading it nationally would ship it. Now grade the same posterior at the level the budget meeting actually uses. The centerpiece: regional ROI ranking, the one-number summary regional teams are judged on.

Measured in the notebook (seeded run) — TV's per-geo ROI ranking, true vs estimated under the pooled fit: Spearman \(\rho = -1.0\), a perfect inversion. True TV ROI (from the DGP): North 0.25, East 0.16, South 0.15, West 0.05 per unit spend. The model ranks them exactly backwards. Across channels (measured): TV \(\rho = -1.0\), Search \(+0.8\), Social \(-0.8\), Display \(0.0\) — the channel with the widest true heterogeneity gets the most confidently wrong regional ranking.

Illustrative reconstruction in your browser of the pooled-beta mechanism: each geo's error is the spend-weighted pooled effectiveness divided by its own true multiplier, minus one — computed live from the DGP's published multipliers and budget rule. The notebook's measured anchors match this pattern: per-geo median |error| 25%, per-geo coverage 31%, worst cell West/TV at +330%, and corr(true multiplier, per-geo error) of −0.87 to −0.97 across all four channels. National totals (black diamonds) stay essentially exact throughout.

Why the sign is the worst possible sign. The pooled beta is a compromise: above the truth in weak geos, below it in strong ones. So the per-geo table over-credits exactly the markets where the channel works least — a budget optimizer reading it would move money toward the geos the channel is failing in and away from where it prints. The readout isn't merely noisy; it argues for the value-destroying reallocation. The notebook's contribution-path band makes it visceral: in the West (true TV multiplier 0.3×), the pooled 90% band floats entirely above the true path — truth below the band 100% of weeks (measured).

The pivot: grade at the decision level, then split per geography

Two moves, both measured. First, grade at the decision level: the national table is an average over the readouts people actually use. If regional tables drive money, the per-geo grade is the grade. Second, the structural pivot available today: by default the framework pools intercepts only (beta_<ch> is shared per channel), but enabling vary_media_by_geo (off by default) gives each channel a partial-pooled per-geo coefficient with shrinkage in a single fit. The other option — the one this notebook demonstrates — is to refit per geography: four small models, each free to find its own beta, at the price of 4× fewer observations per fit.

Measured in the notebook (seeded run) — pooled panel fit vs four split per-geo refits, graded against the same per-geo truth: median |error| 25% → 13%, 90% coverage 31% → 88%, worst cell +330% → 44%. TV's regional ROI ranking goes from \(\rho = -1.0\) to \(\rho = +0.8\). Four small honest models beat one large confidently-wrong one — for the regional question.

Heterogeneity playground: how much spread breaks the regional readout?

The mechanism is general, so you can drive it yourself. The simulation below pools one global beta across four geos whose true effectiveness spreads around 1× by the slider amount (at full spread it reproduces the TV pattern of the notebook's world: 1.8× / 0.9× / 1.0× / 0.3×). Toggle whether budgets chase performance. Watch the per-geo readout errors grow with spread — while the national total stays accurate at every setting, because the pooled beta is precisely the spend-weighted compromise that gets the national sum right.

Illustrative simulation in your browser — pooled-beta arithmetic on four geos (market sizes 1.30 / 0.70 / 1.00 / 0.55, the DGP's), no sampling. Two things to notice: the weakest geo's over-credit explodes as spread grows (at full spread with chasing budgets it reaches roughly +340%, the same order as the notebook's measured +330% for West/TV); and performance-chasing budgets make it worse — they pull the pooled beta toward the strong geos, pushing the weak-geo readout further from its truth. The national bar never moves.

Act 4 — Geography × product: six cells, end to end

geo_product is the positive control at the full cross-classification: 3 geographies × 2 products, two years weekly (624 observations). Baseline level is geo offset plus product offset — exactly the model's additive geo + product hierarchy — response parameters are shared, and each product line flights independently with its own channel tilt (Core is TV-heavy, Premium is Search/Social-heavy). Truth is recorded per cell.

Measured in the notebook (seeded run) — posterior correlation of channel totals on the geo×product fit (the stress 00 separability lens, panel edition): every pairwise correlation sits within ±0.08. Six independent cell calendars keep the channels cleanly separable — contrast the collinear seesaw in stress 03, where every pairwise correlation is negative (min ≈ −0.5).

The closing scorecard line, asserted in the notebook: the panel machinery is sound; the regional claim is only as good as the pooling assumption behind it. Acts 1 and 4 recover per cell because their worlds match the intercept-only hierarchy; Act 3 fails regionally because its world doesn't — and nothing in the data volunteers which world you are in.

What to take away