The Experiment Is the Prior

Matthew Reda MMM

A media mix model fit to observational spend answers a correlational question and is routinely mistaken for a causal one. The channel coefficient it reports is not the incremental effect of advertising — it is that effect tangled up with everything that made spend rise and fall in the first place. A randomized geo experiment cuts the tangle for one channel, at one moment. Calibration is the machinery that carries that hard-won causal fact back into the model as a prior, so the whole system is anchored to something an experiment actually verified. This is the single most important thing you can do to make an MMM trustworthy — and it is worth understanding why, and what it cannot do.

Why the Coefficient Isn't Causal

Start with the uncomfortable premise. When you regress sales on media spend, the fitted coefficient inherits four distinct problems, catalogued in Chan and Perry's (2017) survey of the field's identification troubles.

Spend is endogenous. Marketing budgets are not set by a coin flip; they respond to demand, seasonality, product launches, and the business cycle. Spend rises in the quarters when sales were going to rise anyway, so the coefficient absorbs the demand shock that drove both. This is textbook confounding — an open back-door path from spend to sales through latent demand — and it is why paid search is the canonical offender: search budget chases users already expressing intent, so a naive regression credits search with conversions it merely intercepted (Chen et al., 2018).

Spend is collinear. Channels ramp and cut together, so their spend series move in near-lockstep. The design matrix is close to singular, individual channel effects trade off against one another, and the model cannot cleanly attribute credit — it can only find a ridge of combinations that fit about equally well.

There is barely any data. A typical national MMM has two or three years of weekly observations and little independent variation in spend. With so little signal, the likelihood is nearly flat in the directions that matter, and the prior does most of the work — which is a liability when the prior is wrong and a lever when it is right.

The response is nonlinear. Advertising carries over across weeks and saturates at high spend. Get the adstock and saturation transforms wrong and the coefficient is biased before confounding even enters. Together these mean the fitted number is an artifact of the joint history of spend and demand, not a causal quantity you can act on.

How Wrong Is Observational?

It would be comforting to think that with enough covariates, the confounding washes out. The largest study on the question says otherwise, and it is worth quoting because it reframes the whole exercise. Gordon, Zettelmeyer, Bhargava, and Chapsky (2019) ran fifteen randomized advertising experiments at Facebook — roughly 500 million user-experiment observations and 1.6 billion impressions — and then asked whether standard observational methods could reproduce the experimental answer.

⚠️ Observational ad measurement was off by a factor of three

Across exposed-versus-unexposed comparisons, propensity-score matching, regression adjustment, and inverse-probability weighting, the observational estimates predominantly overstated ad effectiveness — and for roughly half of the studies with a purchase outcome, the estimate was off by about a factor of three, in some cases in either direction (Gordon et al., 2019). No single method consistently won. Upper-funnel outcomes like registrations aligned better, precisely because unexposed users rarely reach those pages on their own, which quietly makes the naive comparison closer to the experiment.

The follow-up closed the last escape hatch. In "Close Enough?", Gordon, Moakler, and Zettelmeyer (2023) applied double/debiased machine learning and stratified propensity-score matching — with more than five thousand user features and deep-learning nuisance models — across 663 experiments. The sophisticated methods beat the crude ones, but neither reliably recovered the experimental effect. The lesson is not "use a better estimator." It is that the confounding in advertising data is not a covariate-richness problem you can regress away. You need the experiment. And an MMM, which has vastly less user-level detail than Facebook's internal data, is in no better position — which is exactly why its coefficients need an external causal anchor.

What the Experiment Identifies

A geo lift test does something the observational likelihood structurally cannot: it manufactures independent variation in spend. Randomly assign markets to a treatment that raises (or cuts, or holds out) a channel's spend, and the difference in outcomes between arms estimates an incremental effect — incremental sales per incremental dollar, an incremental ROAS — with a causal design behind it. The methodology and its assumptions are the subject of a companion post; here the point is narrower. Because the treatment was randomized, the estimate identifies the parameter that the confounded, collinear, data-starved regression left unidentified. It is a shaft of light down one of the flat directions of the likelihood.

The intuition the whole field runs on: randomized experiments give unbiased but imprecise effect estimates; observational models give precise but biased ones. Calibration marries the two — it uses the experiment's unbiasedness to correct the model's bias, and the model's structure to extend the experiment's one-shot, one-channel measurement across the full response curve, the other channels, and time.

Two Ways to Inject It

There are two clean mechanisms for carrying the experiment into the model, and the major open-source tools split neatly across them.

The prior route vs. the likelihood route

Prior route. Put an informative prior directly on the channel's ROI (or coefficient), centered on the experiment's point estimate with a spread set by its standard error. The model's likelihood is then pulled toward the experimentally identified value. This is Google Meridian's approach, and the contribution of Zhang and colleagues (2023) is to reparameterize the model so that ROAS is itself a parameter — which turns "set a prior from the experiment" into a one-line, interpretable operation rather than an opaque prior on a standardized coefficient.

Likelihood route. Add a term that scores how far the model's own implied incremental estimand — its decomposed channel contribution or ROAS over the experiment window — sits from the measured lift. This is essentially Meta's Robyn mechanism: it adds a lift-error objective (MAPE.LIFT) alongside fit error (NRMSE) and a business-logic penalty (DECOMP.RSSD), and lets a multi-objective optimizer trade them off.

Robyn's calibration input is a small table — for each experiment: the channel, the test window, the absolute measured lift, the spend during the window, and a confidence flag that it checks exceeds 80%. It also asks whether the experiment measured the channel's immediate effect or its total effect including carryover, recommending the immediate scope for Facebook-lift-style tests. Meridian, on the prior side, ships helpers to build a LogNormal ROI prior either from a point estimate and standard error or from a confidence interval, and a setting to point the prior at the experiment's calibration window. The two routes are not rivals so much as duals — this framework implements both, an informative ROI prior and an in-graph experiment likelihood on the model's contribution, ROAS, or marginal-ROAS estimand, so you can calibrate on whichever quantity the experiment actually measured.

The Precision Argument

Why does a prior from one experiment move a model fit to years of data? Because the years of data are far less informative than they look. For approximately Gaussian quantities, Bayesian updating is additive in precision — the inverse of variance:

$$ \underbrace{\tau_{\text{post}}}_{\text{posterior precision}} \;=\; \underbrace{\tau_{\text{prior}}}_{\text{experiment}} \;+\; \underbrace{\tau_{\text{data}}}_{\text{observational likelihood}}, \qquad \mu_{\text{post}} \;=\; \frac{\tau_{\text{prior}}\,\mu_{\text{prior}} + \tau_{\text{data}}\,\mu_{\text{data}}}{\tau_{\text{prior}} + \tau_{\text{data}}}. $$

The posterior mean is a precision-weighted average of the experiment and the data. When spend is collinear and barely varies, the observational likelihood carries little precision in the channel's direction — \( \tau_{\text{data}} \) is small — so a reasonably tight experiment prior dominates. Meridian frames these priors as guardrails for exactly this reason: they pull a noisy, weakly-identified estimate toward a value that was actually measured. Jin and colleagues (2017) put the flip side plainly in their own simulation — with a small sample, "prior distributions have a big impact on the posteriors and may lead to biased estimates." That is a warning about bad priors and, read the other way, a mandate for good ones.

The experiment is the prior: precision-weighting a channel's ROAS

The observational model, fit to years of collinear spend, believes this channel returns a ROAS near 2.6 — but weakly, with a wide, biased posterior (the grey curve is fixed). A geo experiment supplies a second, unbiased Gaussian belief (blue). Their precision-weighted product is the calibrated posterior (red). Drag the experiment's estimate and its standard error: as the test gets sharper, the posterior slides toward it and narrows.

1.6
0.50
Experiment prior (mean ± sd)—
Posterior mean—
Posterior sd—
Uncertainty reduction—

Precision adds: \(\tau_{\text{post}} = \tau_{\text{data}} + \tau_{\text{exp}}\). Because the collinear observational likelihood carries little precision in this channel's direction, even a moderately tight experiment moves the posterior a long way — and shrinks its width, which is where the decision-relevant payoff lives.

💡 Calibration buys precision, not just accuracy

The payoff shows up in the width of the intervals, not only their center. Zhang and colleagues (2023) report that an uncalibrated geo-hierarchical MMM produced posterior ROAS estimates whose standard deviation was roughly five times that of the same model calibrated with an experiment prior. Sharper intervals change decisions: a budget reallocation you would not make on a factor-of-five-wide ROAS becomes defensible once the experiment has tightened it. This is the same logic by which a geo-level model already beats a national-only one (Sun et al., 2017) — more identifying variation, tighter credible intervals, safer decisions.

Local Effect, Global Curve

Here is the subtlety that separates calibration done carefully from calibration done wrong. An experiment measures a local quantity: the marginal or average effect at the specific spend level, time, geography, creative, and audience it ran under. The MMM parameter it is meant to inform is a global one: a coefficient governing the entire response curve. These are not the same number, and mapping one onto the other has to respect the transforms in between.

The model's marginal effect at spend \( s \) is not the raw coefficient \( \beta \) but \( \beta \) times the local slope of the saturation-of-adstock curve evaluated at \( s \). So an experiment run at a particular spend constrains \( \beta \) only through that curve at that point. Put a prior naively on the raw coefficient and you have implied an arbitrary, spend-dependent ROI that will not match the experiment at any other operating point. This is the deeper reason the prior belongs on ROI rather than on the coefficient: ROI lives on the decision scale, comparable across channels regardless of spend magnitude or units, and it is the scale on which you have intuition and prior beliefs in the first place. Meridian defaults its treatment prior to ROI for precisely this reason, reserving a coefficient prior as a rarely-recommended fallback because it is "difficult to translate into business terms."

⚠️ The counterfactuals may not even match

An MMM's ROI is usually defined against a zero-spend counterfactual — what sales would be if the channel went dark. An experiment's counterfactual is whatever the control arm did: often a reduced-spend holdout, not full darkness. As Meridian's own documentation warns, "the ROI measured by an experiment rarely aligns perfectly with the ROI measured by MMM." Before treating a lift number as a prior mean, confirm you are calibrating like-for-like — the same estimand, over a comparable spend contrast — or you will anchor the model to an answer to a different question.

What Calibration Doesn't Fix

Calibration is powerful enough that it is tempting to treat a calibrated model as settled truth. Five cautions keep that instinct honest.

External validity is bounded. A lift test is specific to its time, geo, creative, audience, and competitive context. A holiday test on a hero creative in high-intent markets does not license the same ROI in a January always-on flight. Treat the experiment as evidence about the world it ran in, and discount its transportability deliberately.

Priors go stale. Media effectiveness drifts as algorithms, audiences, and competitors move, so a once-valid calibration slowly misleads. The principled response is to re-test on a cadence and to discount older evidence rather than trust it forever — the power-prior idea (Ibrahim and Chen, 2000) of raising external evidence to a fractional weight is the formal knob for this.

Bias can flow sideways. Because channel spends are collinear, pinning one channel's ROI with a tight prior does not only fix that channel — it can push residual confounding onto the correlated, un-calibrated channels. "Calibrating one channel helps the whole model" is true, but double-edged: the same coupling that spreads the correction can displace bias rather than remove it. Calibrating the highest-spend, most-collinear channels, or several jointly, is the mitigation.

A noisy test deserves a wide prior. The standard error of the experiment is not decoration — it sets how hard the prior pulls. Set it too tight around a noisy point estimate and you have let one under-powered test overrule everything else. Widen the prior to reflect both the experiment's own uncertainty and your doubt about its applicability; Robyn's confidence gate exists to keep low-quality tests out entirely.

It does not certify the model. A calibrated channel can still sit in a mis-specified model. Calibration corrects a targeted causal parameter; it does not validate the transforms, the controls, or the computation — those are the jobs of prior predictive checks, simulation-based calibration, and posterior predictive checks. Anchoring the wrong model to a right number still gives you the wrong model.

Which Channel, and When

Experiments cost money and attention, so the natural question is where to spend them. The answer is an information-value calculation, not a hunch: test the channel where an experiment most reduces decision uncertainty per dollar — typically high-spend channels with wide posterior uncertainty and high expected information gain. Calibrating a channel the model already pins tightly, or one too small to move the budget, is wasted budget. And because effects decay, calibration is not a one-time chore but a loop: the value of a stale prior erodes, which itself triggers the next test.

This is where a single MMM becomes a standing measurement program. Fit the model, read which channel's ROI is both consequential and uncertain, run the experiment that resolves it, fold the result back in as a prior, re-allocate, and let the decay clock schedule the next test. The model tells you which experiment is worth running; the experiment tells you what the model got wrong. Neither is sufficient alone — the observational model is precise but biased, the experiment unbiased but narrow — and the calibration loop is what makes them add up to something you can put a budget behind.

Takeaways

  • An observational MMM coefficient is not causal: spend is endogenous, collinear, scarce, and nonlinearly transformed. At Facebook scale, observational ad measurement was off by roughly a factor of three even with thousands of covariates.
  • A randomized geo experiment identifies the incremental effect the likelihood cannot. Calibration carries that causal fact into the model as a prior.
  • Two routes: an informative prior on ROI (Meridian) or an in-graph likelihood on the model's implied lift (Robyn). Put the prior on ROI, not the raw coefficient — ROI is the decision scale.
  • The math is precision-weighting: a weakly-identified observational likelihood cedes to a tighter experiment prior. Calibration cuts posterior ROAS uncertainty — one study reports roughly five-fold.
  • Respect the local-vs-global mismatch: an experiment measures a marginal effect at one spend; map it through the saturation and adstock curve, and confirm the counterfactuals match.
  • Calibration does not fix external validity, drift, sideways bias flow, or model misspecification. Re-test on a cadence, discount stale evidence, and validate the model separately.

References