Aurora 02: The Base MMM

Chapter 2 of the Aurora Coffee Co. story. The causal groundwork is laid (Chapter 1 drew the DAG and identified demand as the back-door to close), so now we fit mmm_framework's core Bayesian MMM and read it the way an analyst would: channel contributions and ROAS with honest uncertainty, where the next dollar goes, and a what-if. Because Aurora is a synthetic world with a known ground truth, every claim gets graded — including the ones the base model gets wrong. It will get two channels badly wrong, and that failure is the cliffhanger.

One model, demand-controlled

The configuration is deliberately boring: the MFF-style Aurora panel with the demand control included (the back-door from Chapter 1), parametric geometric adstock with a learned decay rate \(\alpha_c \sim \mathrm{Beta}(1,3)\) and saturation per channel (the framework's defaults — both learned; the legacy fixed-decay blend is an opt-out), yearly seasonality, and a linear trend. Bayesian sampling returns a full posterior, not a point estimate. The builder makes the whole specification one readable expression:

from mmm_framework import BayesianMMM, ModelConfigBuilder, SeasonalityConfigBuilder, TrendConfig, TrendType
from mmm_framework.analysis import MMMAnalyzer

panel = aurora.base_panel(control_demand=True)          # the back-door, closed
model_config = (ModelConfigBuilder().bayesian_pymc()
                .with_chains(2).with_draws(600).with_tune(600).with_target_accept(0.9)
                .with_seasonality_builder(SeasonalityConfigBuilder().with_yearly(order=2))
                .build())

mmm = BayesianMMM(panel, model_config, TrendConfig(type=TrendType.LINEAR))
results = mmm.fit(draws=600, tune=600, chains=2, cores=1, random_seed=0)
# fitted: 4 channels · 104 weeks

The diagnostics gate — honest uncertainty starts here

Before trusting any number, check that the computation worked: \(\hat{R} \approx 1\) and healthy effective sample size mean the chains agree; divergences flag geometry problems. The demo run reports (measured):

The notebook is explicit that these 2-chain, 600-draw fits trade rigor for speed; real analyses use ≥4 chains and ≥1000 draws/tune. The deeper doctrine — established in Stress 00 — is that this gate validates the sampler, never the causal claim. We will limp through the gate and still end this page with two channels undervalued three- to six-fold.

Every ROAS number below is a summary of these posterior coefficient distributions. Wide intervals are honest uncertainty, not indecision:

Measured — posterior mean and 94% HDI for each channel coefficient \(\beta_c\), from the baked notebook's results.summary(). Every coefficient is wide and none is pinned clearly away from zero — even Search, the largest mean (1.74), has an HDI reaching down to 0.12 — the model genuinely isn't sure any single channel does much. Hold that thought.

How much came from the data vs the priors?

A posterior interval tells you what to believe after the fit — not how much of that belief is the data talking versus the prior. The framework's compute_parameter_learning reports, per parameter, the contraction \( c = 1 - \mathrm{Var}_{\text{post}}/\mathrm{Var}_{\text{prior}} \) (\(c \to 1\): the data pinned it; \(c \approx 0\): prior-dominated) plus prior↔posterior overlap and a standardized shift.

Measured — contraction per parameter from the baked notebook's learning table. Read the bars honestly: all four adstock decay rates \(\alpha_c\) show negative contraction (adstock_alpha_TV −0.91 down to adstock_alpha_Social −0.57) with high prior overlap (0.48–0.75) — the posterior comes back wider than the \(\mathrm{Beta}(1,3)\) prior, a weak-identification / prior↔data-tension flag: observational data barely pins carryover (the carryover/saturation equifinality; see Stress 01). Three of the four carry verdict weak; adstock_alpha_Search (purple bar) is verdict relocated — shift z +1.60, the posterior moved 1.6 prior-sd without narrowing, so the evidence dominated the location and the width reflects likelihood flatness, not "the data taught nothing". But the pieces that drive ROI — saturation rates, intercept, controls, seasonality, \(\sigma\) — are strongly learned (\(c\) up to 0.99). And beta_Search shows negative contraction too (−0.11, with 0.88 prior overlap — verdict prior-dominated): the one coefficient the data cannot pin is exactly the demand-confounded channel Chapter 1 warned about.

Does the model track revenue?

The last gate before reading contributions: posterior-predictive fit. The notebook overlays observed weekly revenue on the prediction with its 90% credible band; the fit hugs the data.

Illustrative recreation (seeded) — the shape of the notebook's actual-vs-fitted chart (trend + yearly seasonality + media bumps, 104 weeks), not the real Aurora panel values. The point survives the recreation: a model can track revenue this well and still misattribute it — fit quality is necessary, nowhere near sufficient.

Contributions & ROAS — with credible intervals

The estimand is counterfactual: a channel's contribution is the revenue that disappears if you turn it off. compute_counterfactual_contributions(compute_uncertainty=True) attaches a posterior interval to every number, and MMMAnalyzer turns contributions into ROAS — revenue per $1 of spend:

The same table as the centerpiece picture — estimated ROAS with its credible interval, and the true ROAS (known, because Aurora is synthetic) as a diamond:

Measured bar heights and truth diamonds (baked ROI table vs aurora.true_roas); whisker widths are approximated by scaling each channel's measured 94% β-posterior HDI onto its ROAS (contribution is near-proportional to β) — the notebook computes them exactly from contribution draws. What to see: the intervals are doing honest work (Search's wide band straddles its truth at the low end; Social's covers 0.49), but the truth diamonds for TV and Display sit far above even the top whisker. That is not sampling noise. It is structure the model cannot see — section below.

Where does the next dollar go? Marginal vs average ROAS

Total ROAS prices the average dollar already spent. Budgeting needs the marginal dollar — what a +10% bump returns — because saturation means channels don't scale forever. compute_marginal_contributions(spend_increase_pct=10) measures it:

Measured — average vs marginal ROAS from the baked tables. Every channel's next dollar earns less than its average dollar (Search: 1.05 → 0.80): all four are on the concave part of their saturation curves. The gap between the pair of bars is the budgeting signal — and it is why "Search has the best ROAS, give it everything" is wrong even when you believe the model.

The geometry behind that gap: average ROAS is the slope of the chord from zero to the current operating point on the channel's response curve; marginal ROAS is the slope of the tangent there. On a saturating curve the tangent is always flatter than the chord.

Illustrative — a saturating response curve \(R(s) = A\,(1 - e^{-k s})\) computed live in your browser, with \(A\) and \(k\) calibrated per channel so that the chord and tangent at current spend equal the measured average and marginal ROAS above. The marked point is the channel's actual two-year spend. The notebook derives the same picture from the fitted saturation posteriors.

A what-if: move 20% of Search into TV

The scenario API answers the planner's question directly: mmm.what_if_scenario({"Search": 0.8, "TV": 1.2}) re-runs the fitted model under the new spend path. Measured result:

Try the move yourself. The slider shifts a share of budget between Search and TV (the notebook's multiplier convention: Search × (1 − x), TV × (1 + x)) and re-prices the scenario on the response curves calibrated above — i.e. this is the base model's worldview, mirage included:

Illustrative — computed live from the calibrated response-curve approximation, anchored to measured numbers (baseline revenue $87,302k; per-channel average and marginal ROAS). At the notebook's setting (+20%) the approximation gives ≈ −0.31% against the measured −0.30% — close enough to trust the shape. Every point on this curve inherits the base model's blind spot: it prices TV at 0.35, not 2.14.

The honest residual — what the base model gets wrong here

Grade the model against the oracle. The measured gap, channel by channel:

Measured — estimated minus true ROAS per channel (baked chk table): TV −1.79, Search +0.39, Social +0.16, Display −1.44. Two failure modes, opposite signs, one fit.

Two distinct things are happening, and neither is noise:

• Search is still overstated (1.05 vs 0.66). The demand control closes most of the back-door, but Search spend chases demand so tightly that residual confounding survives — the learning diagnostic flagged it (beta_Search: negative contraction, 0.88 prior overlap — prior-dominated). Observational control alone cannot finish the job; the fix is a geo-lift experiment folded into the likelihood, which Chapter 1 demonstrates pulling Search from 2.51 (demand-blind) and 1.04 (demand-controlled) onto the truth of 0.66.
• TV and Display are undervalued (0.35 vs 2.14; 0.67 vs 2.11). They work almost entirely through brand awareness — a mediated pathway (TV → awareness → sales) the base model has no node for, so it can only credit the tiny direct slice. To value a brand channel you must model the mediation: that is Chapter 3's NestedMMM (and Stress 04 covers where extensions themselves can mislead).

Takeaways