The Experiment Is the Prior

Matthew Reda MMM

A media mix model fit to observational spend answers a correlational question and is routinely mistaken for a causal one. The channel coefficient it reports mixes the incremental effect of advertising with everything that made spend rise and fall in the first place. A randomized geo experiment cuts the tangle for one channel, at one moment. Calibration is the machinery that carries that hard-won causal fact back into the model as a prior, so the whole system is anchored to something an experiment actually verified. It is the single most important thing you can do to make an MMM trustworthy, which is reason enough to know exactly how it works and where it stops working.

Why the Coefficient Isn't Causal

Start with the uncomfortable premise. When you regress sales on media spend, the fitted coefficient inherits four distinct problems, catalogued in Chan and Perry's (2017) survey of the field's identification troubles.

Spend is endogenous. Marketing budgets are not set by a coin flip. They respond to demand, seasonality, product launches, and the business cycle. Spend rises in the quarters when sales were going to rise anyway, so the coefficient absorbs the demand shock that drove both. That is textbook confounding, an open back-door path from spend to sales through latent demand. It is also why paid search is the canonical offender: search budget chases users already expressing intent, so a naive regression credits search with conversions it merely intercepted (Chen et al., 2018).

Spend is collinear. Channels ramp and cut together, so their spend series move in near-lockstep. The design matrix is close to singular, individual channel effects trade off against one another, and clean attribution goes out of reach. What the model finds instead is a ridge of combinations that fit about equally well.

There is barely any data. A typical national MMM has two or three years of weekly observations and little independent variation in spend. With so little signal, the likelihood is nearly flat in the directions that matter. The prior does most of the work. That is a liability when the prior is wrong and a lever when it is right.

The response is nonlinear. Advertising carries over across weeks and saturates at high spend. Get the adstock and saturation transforms wrong and the coefficient is biased before confounding even enters. Take the four together and the fitted number is an artifact of the joint history of spend and demand rather than a causal quantity you can act on.

How Wrong Is Observational?

It would be comforting to think that with enough covariates, the confounding washes out. The largest study on the question says otherwise, and its numbers reframe the whole exercise. Gordon, Zettelmeyer, Bhargava, and Chapsky (2019) ran fifteen randomized advertising experiments at Facebook (roughly 500 million user-experiment observations and 1.6 billion impressions), then asked whether standard observational methods could reproduce the experimental answer.

⚠️ Observational ad measurement was off by a factor of three

Across exposed-versus-unexposed comparisons, propensity-score matching, regression adjustment, and inverse-probability weighting, the observational estimates predominantly overstated ad effectiveness. For roughly half of the studies with a purchase outcome, the estimate was off by about a factor of three, in some cases in either direction (Gordon et al., 2019). No single method consistently won. Upper-funnel outcomes like registrations aligned better, and they did so precisely because unexposed users rarely reach those pages on their own, which quietly moves the naive comparison closer to the experiment.

The follow-up closed the last escape hatch. In "Close Enough?", Gordon, Moakler, and Zettelmeyer (2023) applied double/debiased machine learning and stratified propensity-score matching, with more than five thousand user features and deep-learning nuisance models, across 663 experiments. The sophisticated methods beat the crude ones, but neither reliably recovered the experimental effect. Confounding in advertising data is not a covariate-richness problem you can regress away, and no estimator on the shelf will turn it into one. You need the experiment. An MMM works with far coarser data than Facebook's internal user-level records, so it is in no better position. That is exactly why its coefficients need an external causal anchor.

What the Experiment Identifies

A geo lift test does something the observational likelihood structurally cannot: it manufactures independent variation in spend. Randomly assign markets to a treatment that raises (or cuts, or holds out) a channel's spend, and the difference in outcomes between arms estimates an incremental effect (incremental sales per incremental dollar, an incremental ROAS) with a causal design behind it. The methodology and its assumptions are the subject of a companion post. Here the point is narrower. Because the treatment was randomized, the estimate identifies the parameter that the confounded, collinear, data-starved regression left unidentified. It is a shaft of light down one of the flat directions of the likelihood.

The intuition the whole field runs on is a trade: randomized experiments give unbiased but imprecise effect estimates, and observational models give precise but biased ones. Calibration marries the two. It uses the experiment's unbiasedness to correct the model's bias, and the model's structure to extend the experiment's one-shot, one-channel measurement across the full response curve, the other channels, and time.

Two Ways to Inject It

There are two clean mechanisms for carrying the experiment into the model, and the major open-source tools split neatly across them.

The prior route vs. the likelihood route

Prior route. Put an informative prior directly on the channel's ROI (or coefficient), centered on the experiment's point estimate with a spread set by its standard error. The model's likelihood is then pulled toward the experimentally identified value. This is Google Meridian's approach, and the contribution of Zhang and colleagues (2023) is to reparameterize the model so that ROAS is itself a parameter. That turns "set a prior from the experiment" into a one-line, interpretable operation rather than an opaque prior on a standardized coefficient.

Likelihood route. Add a term that scores how far the model's own implied incremental estimand (its decomposed channel contribution or ROAS over the experiment window) sits from the measured lift. This is essentially Meta's Robyn mechanism: it adds a lift-error objective (MAPE.LIFT) alongside fit error (NRMSE) and a business-logic penalty (DECOMP.RSSD), and lets a multi-objective optimizer trade them off.

Robyn's calibration input is a small table. Each experiment contributes one row: the channel, the test window, the absolute measured lift, the spend during the window, and a confidence flag that it checks exceeds 80%. It also asks whether the experiment measured the channel's immediate effect or its total effect including carryover. For Facebook-lift-style tests it recommends the immediate scope. Meridian, on the prior side, ships helpers to build a LogNormal ROI prior either from a point estimate and standard error or from a confidence interval, plus a setting to point the prior at the experiment's calibration window. The two routes are duals of each other. This framework implements both, an informative ROI prior and an in-graph experiment likelihood on the model's contribution, ROAS, or marginal-ROAS estimand, so you can calibrate on whichever quantity the experiment actually measured.

The Precision Argument

A prior from one experiment moves a model fit to years of data. Those years look far more informative than they are, because collinear spend carries little independent signal about any one channel. For approximately Gaussian quantities, Bayesian updating is additive in precision, the inverse of variance:

$$ \underbrace{\tau_{\text{post}}}_{\text{posterior precision}} \;=\; \underbrace{\tau_{\text{prior}}}_{\text{experiment}} \;+\; \underbrace{\tau_{\text{data}}}_{\text{observational likelihood}}, \qquad \mu_{\text{post}} \;=\; \frac{\tau_{\text{prior}}\,\mu_{\text{prior}} + \tau_{\text{data}}\,\mu_{\text{data}}}{\tau_{\text{prior}} + \tau_{\text{data}}}. $$

The posterior mean is a precision-weighted average of the experiment and the data. When spend is collinear and barely varies, the observational likelihood carries little precision in the channel's direction (\( \tau_{\text{data}} \) is small), so a reasonably tight experiment prior dominates. Meridian frames these priors as guardrails for exactly this reason. They pull a noisy, weakly-identified estimate toward a value that was actually measured. Jin and colleagues (2017) put the flip side plainly in their own simulation, where with a small sample "prior distributions have a big impact on the posteriors and may lead to biased estimates." That is a warning about bad priors and, read the other way, a mandate for good ones.

The experiment is the prior: precision-weighting a channel's ROAS

The observational model, fit to years of collinear spend, believes this channel returns a ROAS near 2.6 but holds that belief weakly, with a wide, biased posterior (the grey curve is fixed). A geo experiment supplies a second, unbiased Gaussian belief (blue). Their precision-weighted product is the calibrated posterior (red). Drag the experiment's estimate and its standard error: as the test gets sharper, the posterior slides toward it and narrows.

1.6
0.50
Experiment prior (mean ± sd)…
Posterior mean…
Posterior sd…
Uncertainty reduction…

Precision adds: \(\tau_{\text{post}} = \tau_{\text{data}} + \tau_{\text{exp}}\). Because the collinear observational likelihood carries little precision in this channel's direction, even a moderately tight experiment moves the posterior a long way. It also shrinks the posterior's width, which is where the decision-relevant payoff lives.

💡 Calibration buys precision as well as accuracy

The payoff shows up in the width of the intervals as much as in their center. Zhang and colleagues (2023) report that an uncalibrated geo-hierarchical MMM produced posterior ROAS estimates whose standard deviation was roughly five times that of the same model calibrated with an experiment prior. Sharper intervals change decisions: a budget reallocation you would not make on a factor-of-five-wide ROAS becomes defensible once the experiment has tightened it. This is the same logic by which a geo-level model already beats a national-only one (Sun et al., 2017). More identifying variation buys tighter credible intervals, and tighter credible intervals make for safer decisions.

Local Effect, Global Curve

One subtlety separates calibration done carefully from calibration done wrong. An experiment measures a local quantity: the marginal or average effect at the specific spend level, time, geography, creative, and audience it ran under. The MMM parameter it is meant to inform is a global one: a coefficient governing the entire response curve. These are not the same number, and mapping one onto the other has to respect the transforms in between.

The model's marginal effect at spend \( s \) is not the raw coefficient \( \beta \) but \( \beta \) times the local slope of the saturation-of-adstock curve evaluated at \( s \). So an experiment run at a particular spend constrains \( \beta \) only through that curve at that point. Put a prior naively on the raw coefficient and you have implied an arbitrary, spend-dependent ROI that will not match the experiment at any other operating point. This is the deeper reason the prior belongs on ROI rather than on the coefficient: ROI lives on the decision scale, comparable across channels regardless of spend magnitude or units, and it is the scale on which you have intuition and prior beliefs in the first place. Meridian defaults its treatment prior to ROI for precisely this reason, and it reserves a coefficient prior as a rarely-recommended fallback because a coefficient is "difficult to translate into business terms."

⚠️ The counterfactuals may not even match

An MMM's ROI is usually defined against a zero-spend counterfactual, meaning what sales would be if the channel went dark. The experiment's counterfactual is whatever the control arm did, and that is often a reduced-spend holdout rather than full darkness. As Meridian's own documentation warns, "the ROI measured by an experiment rarely aligns perfectly with the ROI measured by MMM." Before treating a lift number as a prior mean, confirm you are calibrating like-for-like, on the same estimand and over a comparable spend contrast. Otherwise you anchor the model to the answer to a different question.

What Calibration Doesn't Fix

Calibration is powerful enough that it is tempting to treat a calibrated model as settled truth. Five cautions keep that instinct honest.

External validity is bounded. A lift test is specific to its time, geo, creative, audience, and competitive context. Run it over the holidays on a hero creative in high-intent markets, and it does not license the same ROI in a January always-on flight. Treat the experiment as evidence about the world it ran in, and discount its transportability deliberately.

Priors go stale. Media effectiveness drifts as algorithms, audiences, and competitors move, so a once-valid calibration slowly misleads. The principled response is to re-test on a cadence and to discount older evidence rather than trust it forever. The power-prior idea (Ibrahim and Chen, 2000) of raising external evidence to a fractional weight is the formal knob for this.

Bias can flow sideways. Because channel spends are collinear, pinning one channel's ROI with a tight prior can push residual confounding onto the correlated, un-calibrated channels. "Calibrating one channel helps the whole model" is true and double-edged. The same coupling that spreads the correction can displace bias instead of removing it, which is why the mitigation is to calibrate the highest-spend, most-collinear channels, or several of them jointly.

A noisy test deserves a wide prior. The standard error of the experiment sets how hard the prior pulls. Set it too tight around a noisy point estimate and you have let one under-powered test overrule everything else. Widen the prior to reflect both the experiment's own uncertainty and your doubt about its applicability. Robyn's confidence gate exists to keep low-quality tests out of the fit entirely.

It does not certify the model. A calibrated channel can still sit in a mis-specified model. Calibration corrects a targeted causal parameter. It says nothing about the transforms, the controls, or the computation, which are the jobs of prior predictive checks, simulation-based calibration, and posterior predictive checks. Anchoring the wrong model to a right number still gives you the wrong model.

Which Channel, and When

Experiments cost money and attention, so the natural question is where to spend them. Answer it with an information-value calculation. Test the channel where an experiment most reduces decision uncertainty per dollar, which typically means a high-spend channel with wide posterior uncertainty and high expected information gain. Calibrating a channel the model already pins tightly, or one too small to move the budget, is wasted budget. And because effects decay, calibration is a loop. The value of a stale prior erodes, and that erosion is what schedules the next test.

This is where a single MMM becomes a standing measurement program. Fit the model, read which channel's ROI is both consequential and uncertain, run the experiment that resolves it, fold the result back in as a prior, re-allocate, and let the decay clock schedule the next test. The model tells you which experiment is worth running, and the experiment tells you what the model got wrong. Neither is sufficient alone. The observational model is precise but biased, the experiment unbiased but narrow, and the loop that joins them is what turns a media mix model into something you can put a budget behind.

Takeaways

  • An observational MMM coefficient is not causal: spend is endogenous, collinear, scarce, and nonlinearly transformed. At Facebook scale, observational ad measurement was off by roughly a factor of three even with thousands of covariates.
  • A randomized geo experiment identifies the incremental effect the likelihood cannot. Calibration carries that causal fact into the model as a prior.
  • Two routes: an informative prior on ROI (Meridian) or an in-graph likelihood on the model's implied lift (Robyn). Put the prior on ROI rather than the raw coefficient, because ROI is the decision scale.
  • The math is precision-weighting: a weakly-identified observational likelihood cedes to a tighter experiment prior. Calibration cuts posterior ROAS uncertainty, by roughly five-fold in one study.
  • Respect the local-vs-global mismatch: an experiment measures a marginal effect at one spend, so map it through the saturation and adstock curve, and confirm the counterfactuals match.
  • Calibration does not fix external validity, drift, sideways bias flow, or model misspecification. Re-test on a cadence, discount stale evidence, and validate the model separately.

References