Causal Inference 02: The MMM as a Causal Model

Rung 1 taught adjustment on a four-channel toy. Veranda Home's real modeling problem looks nothing like a toy: seven channels (two of them weak and bought on the same calendar), thirteen candidate controls (two genuine confounders, four precision variables, six irrelevant series someone exported from the warehouse, and one variable that will quietly sabotage the model if you include it), demand-chasing budgets, and realistically low media signal-to-noise. This page mirrors nbs/causal/causal_02_mmm_as_causal_model.ipynb, which treats the fitted MMM the way a careful practitioner should — as a causal model making explicit claims, every one of them graded against the sealed answer key.

The world on the analyst's desk

The data dictionary below is the analyst's causal homework, done: each candidate control tagged by why it belongs (or doesn't) in the model. Two things deserve special note — brand_awareness is a mediator (TV and Video build it, it drives sales: conditioning on it would block the very effect we're estimating), and Radio/Print are bought on one flighting calendar (Print's spend is nearly a scaled copy of Radio's; correlation 0.996).

Spend shares: TV 28%, Search 20%, Social 15%, Display 13%, Video 11%, Radio 8%, Print 5%.

The causal fit, graded

The model gets the defensible control set — both confounders, the four precision controls, the six irrelevant series, and not the mediator — and its counterfactual contributions are graded against the sealed key:

Counterfactual contribution (KPI units) vs the sealed key — nbs/artifacts/causal_02_estimands.json.

Mostly honest: five of seven channels land within ~±20% and covered. The residual wrinkle is TV — the hardest demand-chaser and a mediated channel — still under-read by 18% even with the right controls, with the truth outside its interval. The demand index is a noisy proxy; rung 1's lesson compounds at scale. Structure (notebook 03) and experiments (notebooks 05+) exist precisely for what adjustment leaves behind. Radio's −36% is a different story — the ridge, below, where the model at least says it doesn't know.

Three estimands, three different questions

"What's the ROAS?" is not one question. The three below have different answers, different uses, and different failure modes — a report that doesn't say which one it's quoting is not yet a claim:

Because these worlds ship an executable truth, all three get graded — including the marginal, which almost never gets graded in real life. The true marginal comes from nudging each channel's whole spend path by +10% inside the data-generating process:

\[ \mathrm{mROAS}_c \;=\; \frac{\sum_t \bigl( f_c(1.1\, s_{c,t}) - f_c(s_{c,t}) \bigr)} {0.1 \sum_t s_{c,t}} \]

marg = model.compute_marginal_contributions(spend_increase_pct=10.0)
lrn = model.compute_parameter_learning(prior_samples=2000)

Measured — estimated average vs marginal ROAS per channel. Saturation makes the marginal less than the average for every channel — in the estimate and in the truth (e.g. Video: average 2.09 vs marginal 1.57 estimated; 1.93 vs 1.44 true). Ranking channels by average ROAS and allocating budget by that ranking is a category error this chart makes visible: defense numbers and allocation numbers are different estimands.

The bad-control trap, sprung on purpose

brand_awareness correlates beautifully with sales. A naive modeler adds it ("more signal!"). But it is a mediator: TV and Video work through it. In a total-effect model, conditioning on a mediator amputates exactly the effect you're paid to measure. The framework refuses a control explicitly marked MEDIATOR at build time — so the notebook does what the naive modeler does, and adds it unmarked:

Measured — the moment the mediator enters the control set, TV's estimated contribution drops 7,302 → 5,996 (−18%) and Video's 8,956 → 6,863 (−23%) — both walking away from a truth they were already below or near. A variable can be predictive, well-measured, statistically significant — and still make the model causally worse. Roles first, R² later.

Earning trust by attack: the refutation battery

A causal claim you haven't tried to break isn't a claim, it's a hope. The framework's refutation suite attacks the fitted model four ways, each test refitting on perturbed data:

from mmm_framework.validation import ModelValidator, ValidationConfigBuilder

summary = ModelValidator(model).validate(
    ValidationConfigBuilder().quick().with_causal_refutation().build()
)

Did the data actually teach the model anything?

The second trust check: compare each parameter's prior to its posterior. A parameter whose posterior is just its prior echoed back (prior-dominated) is a number the analyst chose, not one the data produced — and it deserves an asterisk in any deck.

The diagnostic's headline is beta_TV: by far the lowest contraction of the seven media coefficients (0.13) combined with the largest prior→posterior move (1.53 → 2.73) — the relocated signature, a posterior that moved far from the prior without narrowing. That is the model waving a small flag over exactly the channel the sealed key says carries residual bias: the data is pulling hard against the prior, somewhere off to the side. It's an oblique warning, not a diagnosis — but it's the only internal check on this page that points at TV at all.

The dignity of "I don't know": the Radio/Print ridge

Radio and Print carry real, material effects in truth — but their spend moves in lockstep (correlation 0.996). Observational data can identify their combined effect; the split between them is simply not in the data. The honest output is exactly what the model produces: enormous intervals on each.

Measured — 90% interval width ÷ |estimate| per channel. The two collinear channels carry, by far, the widest relative intervals: Print 2.30 and Radio 1.85, against 0.40–1.12 for everything else. This is not a failure — it is the model honestly saying "I know Radio+Print matter; I cannot tell you the split." A model that returned tight intervals here would be lying. The split isn't a modeling problem; it's a missing-information problem — and a single Radio lift test (notebooks 05–07) will snap the ridge shut.

What this rung bought

A fitted MMM, treated properly, is a structured set of causal claims: named estimands, a defended control set, checks that attack the model from inside, and wide intervals where the data is silent. On the realistic world that discipline gets five of seven channels right and — crucially — labels the places it can't settle: TV's residual bias (invisible internally, flagged only obliquely by the relocation diagnostic) and the Radio/Print split (loudly visible as variance). Both open questions have the same answer, and it isn't a better prior: for channels that work through a funnel, model the funnel (03 · Structural Mediation); for what no structure can settle, buy information with an experiment (notebooks 05–07).