The MMM Walkthrough

This is what the full analyst workflow looks like when the world is kind — realistic, messy, low signal-to-noise, but not engineered to break the model. We run the whole arc on a synthetic world with known causal ground truth: EDA before any fitting, a pre-specified model, a naive first fit that fails (with green diagnostics), a DAG-driven revision that recovers, lift-test calibration for what the data cannot identify, and a final recommendation with honest confidence labels. Every number is measured. For the same arc on a world built to be hostile, see stress 05 · The Gauntlet; for the doctrine and scorecard behind both, see Pressure Testing.

The world is mmm_framework.synth.dgp.make_realistic(seed=42): 156 weekly observations, 7 media channels (TV, Search, Social, Display, Video, Radio, Print) and 13 candidate controls. Because the data is synthetic, every channel's true counterfactual contribution is recorded — so each modeling iteration gets graded, not guessed at. New to MMM entirely? Start with Workshop 03 · Your First MMM and Getting Started; this page assumes you know what a fit looks like and shows you the workflow around it.

Part 1 — EDA before you model

For MMM, EDA is not mainly about the KPI. It is about two questions that decide whether the model can possibly succeed: is each channel identifiable (does its spend vary enough, independently enough, to estimate an effect?) and are the controls causally valid (which are confounders you must keep, which are post-treatment "bad controls" you must drop?). Six checks answer those.

1.1 — Look at the KPI

Orient first: scale, trend, seasonality, anomalies. These decide your trend/seasonality spec and warn of structural breaks a constant-coefficient MMM won't handle (a sharp break needs a dummy or piecewise trend, or it leaks into media — measured in stress 02). Here: mild upward trend, yearly seasonality, no breaks. Measured stats: mean 814, sd 137, range 544–1145.

Illustrative recreation (seeded) — a weekly KPI series with the same structure the notebook plots (mild trend + annual cycle + noise), anchored to the notebook's measured stats: mean 814, sd 137, min 544, max 1145 over 156 weeks. Look for: trend (yes, mild), seasonality (yes, annual), level shifts (none).

1.2 — Spend variation = identifiability

A coefficient can only be estimated if its driver varies. An always-on channel with near-constant spend is confounded with the intercept — no amount of data separates its effect from the baseline. The screen is the coefficient of variation (CoV = sd/mean), with ~0.3 as the rule-of-thumb floor. Measured here: every channel passes comfortably (Radio and Print lowest at 0.93; Search highest at 1.34) — flighting is bursty, which is good news for identification.

1.3 — Collinearity: which channels can't be separated

If two channels always move together, the data identifies their combined effect but not the split. The measured screen finds exactly one problem — and it's severe:

Radio and Print are bought together — a perfectly collinear pair. The data can estimate their sum, never their split. Options: model them as one grouped channel, or anchor one with an experiment (we do the latter in v3). Reporting separate point ROIs for a collinear pair is a trap — stress 03 shows the posterior "seesaw" this produces.

1.4 — Outliers and the normalization trap

The model normalizes each channel by its training max; one mega-spike compresses every normal week into a sliver of the response curve. The 30-second screen is max ÷ median. Measured here: TV 12.9×, Search 16.7×, Social 10.3×, Display 9.6×, Video 10.7×. These exceed the ~8–10× alarm line — but inspection shows they are genuine bursty flighting (the same heavy-tailed pattern that gave us healthy CoV in §1.2), not one-week data-entry spikes. The screen's job is to make you check. Contrast stress 01, where genuine 15× one-week spikes crush interval coverage to 0% until winsorized.

1.5 — Exogeneity: is spend driven by demand?

The deadliest MMM bias: a hidden demand signal drives both spend (teams bid harder when demand is high) and sales — then corr(spend, sales) is mostly demand, not media effect. Screen every channel against any demand proxy you have.

Measured — each channel's correlation with the latent demand index (solid) next to its naive correlation with sales (hatched). TV (0.53) and Search (0.48) chase demand: their sales correlations (0.57, 0.46) are mostly demand, not effect. Never read corr(spend, sales) as ROI — the second bar is the trap.

1.6 — Classify the controls: build the DAG

The single highest-leverage EDA step. For each candidate control, ask how it relates to spend and to the KPI:

Measured — all 13 candidate controls placed by their measured correlations: max |corr with any channel's spend| (x) vs |corr with KPI| (y), colored by the world's answer-key role. category_demand and distribution sit high on both axes — confounders, keep with a wide prior. brand_awareness is strongly predicted by TV/Video (corr 0.56) — the tell of a mediator (media → awareness → sales): including it blocks the path and under-counts those channels. The noise_* cluster hugs the origin.

This classification is the model spec — the DAG in five lines:

category_demand / distribution ──► spend AND sales   (confounders: keep, wide prior)
TV, Search ──────────────────────► sales             (but chase demand)
TV, Video ──► brand_awareness ────► sales            (MEDIATOR: exclude — bad control)
price, competitor, weather, holiday ► sales          (precision: keep, regularized)
noise_1..6 ──╴(no edge)╶──────────  sales            (irrelevant: prior shrinks them)
Radio ≈ Print (collinear)                            (needs an experiment to split)

1.7 — How hard is this world? (it is not "simple synthetic data")

It's easy to make synthetic data the model nails — draw it from the model's own assumptions and recovery is near-perfect (the positive control on the scorecard recovers at ~7% median error, 100% coverage). This world is deliberately the opposite. Measured difficulty: 20 candidate factors to sort out (6 of them pure noise); media is only ~26% of total sales; the worst spend–demand confounding correlation is 0.53 (exogeneity violated); the worst channel collinearity is 1.00 (a pair the data cannot split). The structure the model must fight is in the data, not the sampler — which is why a simple recovery test tells you only that the sampler works.

Part 2 — The pre-specified model and the prior gate

Question: each channel's incremental contribution, with honest uncertainty, to defend a budget reallocation. Story: the DAG from §1.6. Model (pre-specified, before any fit): additive, geometric adstock (estimated per channel), saturating response, Fourier seasonality, linear trend, positive coefficients, and per-control prior widths assigned by causal role — confounders wide (never shrunk), the rest regularized:

\[ y_t \;=\; \alpha \;+\; \mathrm{trend}_t \;+\; \mathrm{season}_t \;+\; \sum_{c} \beta_c \left(1 - e^{-\lambda_c \tilde{x}_{c,t}}\right) \;+\; \gamma^{\top} z_t \;+\; \varepsilon_t \]

where \( \tilde{x}_{c,t} \) is adstocked, max-normalized spend and \( z_t \) the retained controls. Fits use NumPyro NUTS, 1000 draws × 4 chains (as executed in the baked notebook).

The prior predictive gate: do the priors generate plausible sales before seeing the likelihood? Measured: the prior-predictive 90% KPI range is [685, 1750] against an observed range of [544, 1145] — the observed mean (814) sits comfortably inside the prior band, which is neither absurdly wide nor too tight to contain the data. Priors pass; proceed.

Iteration 1 — v1, the naive fit (failure)

A first pass without the DAG: include the obvious controls and the tempting brand_awareness, but omit the demand confounders — their back-door is invisible until you draw the story. The diagnostics gate passes: r-hat = 1.021, zero post-tuning divergences, 11.4 s wall-clock. And the answer is badly wrong.

Measured — estimated total contribution per channel (bars) against the recorded ground truth (dark ticks), for each of the three iterations. Hover for the full grade row: relative error, whether truth landed in the 90% interval, the relative interval width, and the prior→posterior learning verdict. v1: Search +101% and TV −45% with truth outside their intervals. v2: every confounded channel snaps back (Search +7%, TV −18% — though TV's truth stays just outside its interval, and its learning verdict reads relocated: the posterior moved more than a prior-sd without narrowing). v3: lift tests pin Radio/Print/TV/Search coefficients — and TV's residual bias survives (−19%, truth still outside the now-narrowed interval).

Iteration 2 — v2, apply the causal structure (recovery)

Revise using the DAG: add the confounders (category_demand, distribution) with a wide, un-shrunk prior via CausalControlRole.CONFOUNDER; drop the mediator; keep the precision controls. Measured result (r-hat 1.020, zero divergences, 10.2 s): the large confounded/mediated errors collapse — TV −45% → −18%, Search +101% → +7%, Video −33% → +9% — and every channel's truth but TV's is inside its 90% interval (TV's sits just outside; the residual-bias thread below picks that up). Toggle the chart above to v2 and compare.

Two confirmations that the structure is doing its job, both measured from the posterior: the demand confounder is now the largest-credited control (coefficient 0.308, distribution 0.191), and the six noise_* controls are shrunk to ≈ 0 (between −0.10 and +0.11) by the regularizing prior — automatic soft variable selection, no extra step.

Variable selection with a horseshoe prior

With many candidate controls you can replace the soft Normal regularizer with an explicit horseshoe selection prior (ControlSelectionConfig(method="horseshoe")) — which, by construction here, never shrinks the confounders (shrinking a confounder re-opens the back-door). Refit and compare:

Measured — posterior mean |coefficient| for every control under the Normal prior vs the horseshoe, grouped by true role. The horseshoe drives the irrelevant noise controls markedly closer to zero (e.g. noise_3: 0.113 → 0.022, noise_2: 0.023 → 0.003) while the confounders keep — or grow — their full coefficients (category_demand 0.308 → 0.306, distribution 0.191 → 0.217). Honest caveat: in this low-SNR world it also shrinks the genuinely-weak precision controls, which is why the framework keeps selection off by default. The horseshoe fit sampled clean (0 divergences) but slower (11.6 s vs 10.2 s) — it adds a per-control tau/lambda/c2/z hierarchy. More on this trade in Variable Selection and stress 03.

But v2 isn't done — and the model tells you about one remaining problem and hides the other:

• Radio and Print are unidentified, and the model says so: their contribution intervals are ~1.9× and ~2.2× wider than the estimates themselves — the honest signal of the collinear pair EDA flagged in §1.3. A wide interval is the model routing you to an experiment.
• TV is still ~18% low, and nothing flags it: residual confounding from a noisy demand proxy that the control only partly removes. In this run the grader's coverage column does show the miss (truth just outside the 90% interval) — but that column needs the answer key a real analyst never has; no observational diagnostic catches a noisy-proxy bias (stress 03 measures this trap in isolation: ~+35% residual bias with green checks).

Both need randomized evidence.

Iteration 3 — v3, bring in experiments

You can't fix the above with more observational modeling. The notebook calibrates with geo-lift experiments on the channels the model flagged as unidentified (Radio, Print) plus — as standard practice — the biggest spenders (TV, Search), since a confident-but-wrong estimate is invisible to every observational check. Each lift test is well-powered (SE = 12% of the true lift) and fed to ExperimentCalibrator(...).calibrate(lifts, refit=True).

Measured — posterior SD of each channel's coefficient before (v2) and after (v3) calibration. The four anchored channels contract sharply: TV 1.05 → 0.39, Search 0.44 → 0.10, Radio 0.68 → 0.10, Print 0.70 → 0.07. The three un-anchored channels move far less (Social 0.68 → 0.57, Display 0.71 → 0.72, Video 0.67 → 0.71) — calibration is surgical, not contagious.

Step 9 — Decision outputs: communicate with honest uncertainty

The deliverable is never a bare point estimate. It states what we know, how well, and on what evidence — measured final table:

Budget guidance (the notebook's closing line): shift toward well-identified winners (Search, Video); treat the collinear Radio/Print pair as one line item; HOLD provisional channels (TV) pending a cleaner experiment. Note Radio's −46% point error sits inside an interval ~2× the estimate — the pair's individual numbers are honestly wide, which is exactly why they're reported as one line item.

How much compute did this take?

Measured wall-clock from the baked run (156 obs, 7 channels, 1000 draws × 4 chains, NumPyro): v1 naive 11.4 s, v2 causal 10.2 s, v2 + horseshoe 11.6 s, v3 calibration refit 9.8 s — 43 s total across 4 fits. Cost drivers from the robustness study: time scales with observations and draws, super-linearly in channels (the adstock kernel sits inside every gradient step); NumPyro runs ~2× faster than the PyMC sampler at equal draws; the horseshoe's per-control hierarchy costs extra; and the refutation suite (skipped in the notebook for speed) refits the whole model ~4×.

Part 3 — Failure-mode quick reference

Meta-lessons. (1) Modeling is iterative — the DAG drove v2, the model's own uncertainty drove v3. (2) Diagnostics check the sampler, not the truth. (3) Causal structure is the spec. (4) Let the model tell you what it can't do — wide intervals route you to the right experiment. (5) Experiments identify, but must be designed for the channel you're trying to rescue.

When the world is not kind — confounding plus mediation plus the wrong saturation form, all at once — the same arc still works, but every step gets harder and the naive fit gets much worse (ranking fully inverted, Search 8× overstated). That is stress 05 · The Gauntlet.