The Prior Was Doing More Than You Thought

Matthew Reda Causal inference

This framework recently grew a second way to estimate the same model. It is a fast frequentist path that fixes the carryover and saturation curves by search and solves the rest in closed form, instead of estimating everything at once under priors. The point was speed and hard constraints. What it produced on the way was a useful piece of self-knowledge. Graded against a synthetic world with an open back-door path (latent demand driving both spend and sales), the new path over-credited media by +41.6% where the existing Bayesian point estimate over-credited by +5.9%. Same model, same data, same target. A sevenfold difference in bias. The explanation is simple, and once you have seen it you start seeing it everywhere: the media priors had been quietly pulling the estimate down, and a penalty whose strength gets chosen by out-of-sample prediction cannot do that. Under confounding, the coefficient that predicts best is the confounded one.

None of that means the new path is broken or the old one is secretly correct. It means a regularizer is an assumption about the answer rather than a neutral numerical convenience, and that until you have two of them you have no easy way to notice how much work yours was doing. The rest of this post is about that mechanism: why it happens, and why a cancellation that helps is still not a fix. The last section is what I think it suggests building.

The Result That Prompted This

The comparison was run on five simulated worlds where the true per-channel contribution is known by construction, so "how wrong is it" is a subtraction rather than a judgement call. Three estimators saw identical data: the new ridge path, the existing fast point estimate (fit(method="map")), and full MCMC. On the world built to look like a well-behaved MMM, all three agreed and the new path was marginally the most accurate. Speed is where it pulls ahead: on the other four worlds it was comfortably the fastest (1.7–2.9 s against MAP's 2.6–5.6 and full MCMC's 8.9–19.6), while here it and MAP are near-tied. On the world with an unobserved confounder, they came apart.

Signed bias in total media contribution, five worlds

Positive means over-crediting media. The middle row is the one this post is about.

WorldRidgeMAPNUTS
Well-behaved (control)−5.2%−4.8%−5.2%
Unobserved confounding+41.6%+5.9%+7.9%
Wrong saturation shape+68.8%+39.9%+38.9%

Full table, including the two worlds omitted here and the absolute errors, in tests/frequentist/test_recovery_comparison.py.

Read the control row first, because it rules out the boring explanations. All three land within half a percentage point of each other, and all three are slightly under. That is ordinary shrinkage on a world where nothing is wrong, and it says the three estimators are solving the same problem competently. Whatever happens in the confounding row is not a solver bug or a scaling mistake, and it is not one estimator quietly fitting a different model.

Now the confounding row. Every estimator over-credits, which is exactly right: an open back-door path inflates the apparent effect of spend, and no fitting procedure closes a back-door path. That part is a property of the data. The size is not. Nobody should read +41.6% against +5.9% as the same answer with noise on it.

Who Picks the Shrinkage

Both paths shrink. Ridge adds a penalty that pulls coefficients toward zero, and a Bayesian prior pulls them toward wherever the prior's mass sits. What separates the two is who decides how hard to shrink.

On the frequentist path, the penalty strength is chosen the way a careful modeller would choose it: by rolling-origin out-of-sample error, refitting on a prefix of history and scoring on the periods it has not seen. That is the right way to select a penalty for a forecasting problem, and it is the same criterion the transform search uses. On the Bayesian path, nothing chooses the penalty strength at all. The prior is the penalty strength, fixed in advance by a statement about what media coefficients plausibly look like.

That distinction is invisible when the model is correctly specified, because then the coefficient that predicts best and the coefficient that is true are the same coefficient. Under confounding they are not, and the gap between them is precisely the bias.

Why prediction prefers the confounded coefficient

Take one channel. Latent demand \( u \) drives both spend and sales. Here \( x \) is spend, \( b \) the true causal effect, \( t \) demand's direct effect on sales, and \( \kappa \) how strongly demand moves spend:

$$ x = \kappa u + \sqrt{1-\kappa^2}\,v, \qquad y = b\,x + t\,u + \varepsilon. $$

Regressing \( y \) on \( x \) alone does not converge to \( b \). It converges to \( b + t\kappa \), the causal effect plus the back-door path. Now ask what a forecaster would want instead: the coefficient that minimises out-of-sample squared error on fresh data from the same world. Differentiating \( E[(y - \beta x)^2] \) gives

$$ \beta^{\star} \;=\; b + t\kappa, $$

the confounded value exactly. Prediction actively prefers the confounded coefficient over the causal one, because the confounder keeps on predicting in the test data too. So a penalty tuned by cross-validation is being asked to find the number this argument says it should find, and it duly finds it. It selects almost no shrinkage, and the full back-door bias survives.

A prior is under no such obligation. If it says media coefficients are typically modest, it pulls a confounding-inflated estimate down regardless of how well the inflated value predicts. In this framework's default configuration that prior is a Gamma with mean 1.5 on the standardized coefficient (or a LogNormal centred on break-even ROI, on the agent's path). That is informative enough to matter, and it points downward relative to an estimate that confounding has pushed up.

Two Objectives That Disagree

The figure makes the disagreement concrete. One channel, one confounder, and a slider for how strongly demand drives spend. The horizontal axis is how hard the estimate is shrunk, running from no shrinkage on the right to heavy shrinkage on the left. Two curves: how far the estimate lands from the truth, and how well it predicts on held-out data. Two markers: where cross-validation puts the penalty, and where a fixed, moderately informative prior puts it.

Where the two criteria part company

These are analytic large-sample values for a single confounded regressor. Nothing here is sampling noise. Distance from the truth is the absolute bias in the estimated media effect. Out-of-sample error is prediction error on held-out data, plotted against its own scale on the right. The dashed vertical marks where cross-validation lands, which is the minimum of the prediction-error curve. A fixed prior's shrinkage is the dotted vertical, and it stays where it is no matter what the data does. Drag the slider to open and close the back-door path.

0.55
Bias at CV's penalty…
Bias at the prior's…
Best achievable bias…

Slide the confounding to zero and cross-validation is exactly right (no shrinkage, no bias), while the prior is clearly worse and pulls a perfectly good estimate down by 28%. That is the ordinary, well-understood cost of an informative prior, and it is the right thing to see before the rest of the argument. The prior is fixed. Being fixed is a different thing from being smart.

Now open the back-door path. Cross-validation's marker does not move. It stays at no shrinkage, because that is still what predicts best, so its bias climbs with the confounding: 20%, 44%, 68% as the slider advances. The prior's marker does not move either, but the reason is the opposite one, since nothing in the data is talking to it. Between them, the shrinkage that would actually minimise the bias slides steadily leftward, crossing the prior's fixed position at moderate confounding and continuing past it. Neither criterion is tracking that point, and neither is trying to.

That crossing is the whole story. At the slider's default the prior lands at +3.7% bias against cross-validation's +44%, close to the +5.9% and +41.6% the two real estimators produced on the simulated world. The toy model's parameters were chosen to be plausible rather than fitted to that result, so treat the correspondence as a check rather than a forecast, one that tests whether the mechanism being described is big enough to explain what was measured. The Bayesian path has no idea the confounder exists. What it did was cruder: a fixed downward pull happened to point the opposite way from an upward bias, at roughly the right magnitude.

A Second Bias Is Not a Fix

This is the part worth being careful about. The comfortable reading of the table, "the Bayesian path handles confounding better," is wrong in a way that would cost someone money.

The prior detected nothing and corrected nothing. It applied the same downward pull it always applies, and on this world that pull happened to point against the bias and happened to be roughly the right size. Three things had to line up for that: the prior's centre had to sit below the confounded estimate, its strength had to be neither too weak to matter nor strong enough to overshoot, and the confounding had to be upward. Flip any one of them and the same prior makes the same fit worse rather than better. Picture a high-performing channel whose true effect sits above the prior's centre, or a confounder that suppresses the apparent effect instead of inflating it.

You also cannot tell from the fit which case you are in. Both look like a converged model with reasonable intervals. The cancellation leaves no residue in the diagnostics, because there is nothing wrong with the sampler. Your model answered the question it was asked, and that question omitted a variable.

The honest summary

Two biases partially cancelling is not the same as either bias being absent. The frequentist path shows more of the confounding because it is doing less of the offsetting, which arguably makes it the more candid of the two on this world even though its number is further from the truth. The route to actually closing a back-door path is a randomised experiment (mmm_framework.calibration). No choice of penalty or prior substitutes for one.

Why One Path Could Hide This

This needed a second estimator to notice, and the reason generalises past this framework.

With a single estimation path, the prior's contribution to the answer is not separately visible. You can measure how much the posterior narrowed relative to the prior, and this framework does, with a flag on any coefficient whose posterior is mostly its prior. What that tells you is how much the data moved your uncertainty. It does not tell you how much the prior moved your point estimate relative to a world without it, because there is no such world to compare against. Prior-predictive checks have the same limitation: they ask whether the prior implies sensible data before fitting, not whether it is silently offsetting a bias afterwards.

A second estimator that shrinks differently supplies the missing comparison. The two paths differ in exactly one interesting respect (who chooses the shrinkage), so their disagreement is a readout on that choice. Four of the five worlds produced broad agreement, which is the boring and reassuring case. On the confounded one they did not, and the size of the gap was the size of the prior's contribution.

That is a decent argument for keeping a mechanically different second opinion around even when it is the less accurate of the two. Its disagreement carries information that neither path's own diagnostics report.

Running the comparison

The two paths are the same model with a different estimator, so the comparison is a config change rather than a rewrite:

from mmm_framework import BayesianMMM, ModelConfigBuilder, TrendConfig, TrendType

# The Bayesian point estimate: priors choose the shrinkage.
bayes_config = ModelConfigBuilder().bayesian_numpyro().build()
model = BayesianMMM(panel, bayes_config, TrendConfig(type=TrendType.LINEAR))
bayes_results = model.fit(method="map", random_seed=42)

# The frequentist path: out-of-sample error chooses the shrinkage.
ridge_config = ModelConfigBuilder().frequentist_ridge().build()
ridge_model = BayesianMMM(panel, ridge_config, TrendConfig(type=TrendType.LINEAR))
ridge_results = ridge_model.fit(random_seed=42)

The frequentist fit reports converged as None rather than True, because there is no chain to assess. Every interval it produces is labelled a confidence interval rather than a credible one, since a bootstrap sampling distribution does not license the probability statement a posterior does.

What This Doesn't Claim

Four honest limits, since the result is easy to over-read in either direction.

It is one confounded world, not a survey. The +41.6% versus +5.9% gap is a measurement on a specific simulated world with a specific confounding strength and a specific prior. The mechanism generalises. That particular ratio does not, and a different prior centre or a weaker back-door path moves it substantially.

Neither number is usable. Both estimators are badly wrong on this world. One is 42% off and the other 6%, and a 6% error on a number driving budget is no success story either. The comparison is about the mechanism behind the difference, and it carries no recommendation to accept the smaller error.

Cross-validation is not being criticised. Selecting a penalty by out-of-sample error is the correct thing to do when the goal is prediction, and it is what the frequentist path is for. The point is narrower: predictive skill and causal accuracy are different targets, and a procedure optimising one has no reason to land on the other. That objection is not new. It is the same reason this framework's model-averaging work does not select specifications by predictive fit, and the same reason the transform search carries a caveat about it.

Informative priors are not being recommended as a confounding remedy. They are not one. A prior tight enough to offset a back-door path is also tight enough to override real evidence about a channel that works, and you cannot tune it for the first job without accepting the second.

Where This Goes Next

The useful thing about a finding like this is that it suggests specific instruments rather than general caution.

Report the two estimates side by side, and treat the gap as a diagnostic. The pieces are all in place. Both paths run on the same model object, and the frequentist one takes a couple of seconds. What is missing is a surface that runs both and shows the disagreement per channel, so a large gap prompts the right question ("what is my prior doing here?") instead of going unnoticed. This is close to what the framework's triangulation panel already does for MMM against experiments and platform data, applied inward to two estimators of the same model.

Quantify the prior's contribution to the point estimate directly. The existing prior-to-posterior contraction check measures the prior's effect on uncertainty. Its natural sibling measures the effect on location: refit with the prior progressively widened and watch the coefficient move. A coefficient that walks a long way as its prior loosens is one whose value is substantially an assumption. That is worth knowing whether or not a confounder is involved.

Point the experiment planner at the disagreement. A channel where the two paths disagree sharply is a channel where an assumption is doing unusual amounts of work, which makes it a good candidate for the next lift test. The framework already ranks channels by expected information gain and expected value of information. Estimator disagreement is a cheap signal that could feed into that ranking.

The habit underneath all three is the same. When a model hands you an answer, ask how much of that answer came from the data and how much came from the way you asked. A second estimator that shrinks differently is one of the cheaper ways to find out.

References