Aurora 01 · Causality — can we trust the number?

The dashboard says fund Search. Before Aurora Coffee Co. moves a dollar, the data-science team asks the causal question: what would have to be true for "Search ROAS" to mean what we think it means? This chapter draws the causal map, computes what to adjust for, shows why adjustment alone isn't enough — and anchors the model to a randomized geo-lift experiment that finally lands on the truth. Because the Aurora world is synthetic with a known ground truth, every number below is graded against the real answer. This page mirrors nbs/aurora/01_causality.ipynb; all ROAS figures are measured from its baked outputs.

Step 1 — Draw the causal map (a DAG)

Causal claims need assumptions, and assumptions should be explicit. The framework lets you state them as a directed acyclic graph (DAGSpec in mmm_framework.dag_model_builder). Here is Aurora's world, exactly as the notebook declares it: a latent Demand drives both paid-search bidding and sales (a confounder); Awareness sits on the path from TV to Sales (a mediator); Price is a clean predictor of sales. Sales itself is the sum of two coupled product lines — Original and Cold Brew cannibalize each other, which is chapter 03's problem. (For legibility the map shows two representative channels — TV the brand-builder and Search the demand-chaser; the fitted models below use all four.)

Aurora's causal map — the same graph the notebook builds with DAGSpec. The red dashed path (when a control set leaves it open) is the back-door Search ← Demand → Sales; everything Search "earns" along it belongs to demand. Diagram of the notebook's declared structure; use the buttons below to condition on different control sets.

Choose what to condition on

This is the chapter in one interactive exhibit. Pick an adjustment set; the map shows which paths stay open, and the readout shows the measured Search ROAS the notebook actually got under that specification (truth: 0.66).

ROAS readouts are measured — hardcoded from the baked outputs of nbs/aurora/01_causality.ipynb (demand-blind 2.51, demand-controlled 1.04, experiment-calibrated 0.66). The {Demand, Awareness} option has no number because the framework refuses to fit it — see Step 3.

Step 2 — Identify the effect: what must we adjust for?

Given the graph, identification_report applies the back-door criterion: find a set \(Z\) that blocks every non-causal path from treatment to outcome without touching the causal ones, so that

\[ P(\text{sales} \mid do(\text{search})) \;=\; \sum_{z} P(\text{sales} \mid \text{search}, z)\, P(z). \]

The notebook's run, verbatim:

rep = identification_report(dag, treatment_id="search", outcome_id="sales")

Adjustment set (condition on these): ['demand']
Descendants of Search (must NOT adjust): ['sales']
Effect identifiable? True

Back-door paths found:
   search ← demand → sales  — blocked by: ['demand']

So the model must condition on Demand to close the Search ← Demand → Sales back-door — and must never condition on Awareness when the goal is TV's total effect, because Awareness sits on the causal path itself. Adjusting for it would throw away exactly the effect we want to measure: the classic bad control.

Step 3 — Bad-control detection: the framework enforces it

Role typing isn't documentation — it's enforcement. dag_to_mff_config translates the graph into a model configuration and tags every control with its causal role:

cfg = dag_to_mff_config(dag)

  Demand     -> CausalControlRole.CONFOUNDER
  Price      -> CausalControlRole.PRECISION_CONTROL
  Awareness  -> CausalControlRole.MEDIATOR

When you then try to build a model that conditions on a mediator (or a collider), the builder raises — the bad control never makes it into the likelihood. A non-expert who drags "Awareness" into the control list gets a hard error explaining why, not a silently biased TV coefficient. That is why the {Demand, Awareness} button above has no ROAS to report: that fit does not exist.

Step 4 — Why adjustment alone isn't enough

Here's the uncomfortable truth most MMM decks skip. The notebook fits Aurora's model two ways — blind to demand vs. controlling for demand — and grades the recovered ROAS against the known truth (all four channels, 400 draws × 2 chains, seeded):

Measured — hardcoded from the baked outputs of nbs/aurora/01_causality.ipynb (Step 5 table). Controlling for demand shrinks every bar of bias, but Search's stays stubbornly positive: 2.51 → 1.04 against a truth of 0.66 (+0.38 of leftover bias). TV and Display are understated either way — their value hides one step upstream, behind Awareness.

Why doesn't the demand control finish the job? Because Aurora bids Search in proportion to demand, the demand proxy and demand-chasing spend are highly collinear — the regression cannot fully separate them, and Search keeps absorbing demand's credit. This is the central limitation of any observational MMM, and the framework says so out loud. The fix isn't more controls. It's randomized evidence. (The same residual-confounding failure, measured across 16 synthetic worlds, is the headline of Pressure Testing and the capstone of Stress 05 · The Gauntlet.)

Step 5 — Anchor to an experiment

Last quarter Aurora ran a geo-lift test on Search and Social and measured their incremental ROAS. The framework folds that straight into the likelihood via add_experiment_calibration / ExperimentMeasurement: the experiment becomes a data point the model must agree with, updating the coefficient, the saturation curve, and the adstock jointly:

from mmm_framework.calibration import ExperimentMeasurement, ExperimentEstimand

experiments = [
    ExperimentMeasurement("Search", window, value=0.66, se=0.07,
                          estimand=ExperimentEstimand.ROAS),
    ExperimentMeasurement("Social", window, value=0.49, se=0.07,
                          estimand=ExperimentEstimand.ROAS),
]
m_cal = BayesianMMM(panel, base_config(), TrendConfig(type=TrendType.LINEAR),
                    experiments=experiments)

In likelihood terms: alongside \(y_t \sim \mathcal{N}(\mu_t, \sigma)\), the model also has to explain \(\widehat{\text{ROAS}}_{\text{exp}} \sim \mathcal{N}(\text{ROAS}_{\text{model}}, \text{se}^2)\). The notebook's measured result:

Search: observational 1.04  →  calibrated 0.66   (true 0.66)
Social: observational 0.70  →  calibrated 0.48   (true 0.49)

Measured — the notebook's nb01_punchline chart, rebuilt from its printed values. Each calibrated channel's ROAS slides off its inflated observational value onto the truth (dashed line) once the experiment enters the likelihood. The demand mirage is gone. Calibration mechanics — lift tests, estimands, standard errors — are covered in Measurement & Calibration.

The whole chapter in one chart: the ROAS ladder

Three specifications, one truth. Naively, Search was the dashboard's hero (spend–sales correlation ≈ 0.85 in chapter 00). A demand-blind MMM gives the mirage a number: ROAS 2.51, nearly four times the truth. Each rung of the ladder removes one layer of self-deception:

Measured — every point is from the baked Step-5/Step-6 tables of nbs/aurora/01_causality.ipynb; dashed lines are the known true ROAS. Search: 2.51 → 1.04 → 0.66 (truth 0.66). Social: 1.14 → 0.70 → 0.48 (truth 0.49). Switch to "All four" and note that TV (truth 2.14) and Display (truth 2.11) stay understated at every rung — calibrating Search/Social frees up a little credit (TV 0.33 → 0.42, Display 0.67 → 0.78) but can't reveal effects that flow through an unmodeled mediator. That's chapter 03.

Takeaways