Can't Calibrate and Validate on the Same Experiment

Matthew Reda· July 26, 2026· Experiments

A team runs three geo-lift tests over a year. The first, on Search, gets folded straight into the model (one calibration call, then a refit). Months later, a client deck opens the triangulation panel and reads that the experiment and the model "agree within uncertainty: convergent evidence from two independent methods, the most trustworthy read available." Nobody wrote a bug here. The model's own likelihood pulled the experiment's readout into Search's posterior at fit time, and the report is now citing that same readout, unchanged, as proof of the number it just moved. Calling that "two independent methods" agreeing is a bit like a thermostat confirming the temperature it just set itself. The rest of this post works out why that happens in real numbers, and ends on a habit worth building into how experiments get planned.

This is really a point about what a model is for in the first place. A model is a simplification built to be useful for a decision, never a finished and perfectly true account of the market, and it earns that usefulness by being checked in sharper and sharper ways over time. Noticing that a held-out experiment needs to actually stay held out is exactly that kind of sharpening. It is a useful next question to ask of a model, and it is the sort of question that only surfaces once the easier ones have been answered. Measurement practice matures the way models do, by picking up one more honest habit at a time.

A Sentence Already on the Record

Stop Validating Your MMM With Holdout Error already named this, in passing, as check #5 on a longer list. Agreement with an experiment that was held out of the fit is the single most informative validation check there is, because it brings information the model's own data never had. It only works if the experiment really was held out. Fold it into the likelihood first, that post noted, and checking agreement afterward is circular: "calibration makes the experiment the prior, and a prior cannot then validate the posterior."

That sentence is exactly right, and it deserves credit before anything else here gets said. What one item on someone else's list can't do is slow down and walk through the rest of it: the code that makes this mistake easy to fall into anyway, the general principle it's an instance of, what the failure looks like in numbers, and a real next step for fixing it. So take this post as the fuller version of an idea that post already got right, worked out in enough detail to act on.

The Mechanism Has a Name

Start with the plain version. If a result helps build an estimate, and then that same result gets used again to check whether the estimate is right, that is one thing counted twice and called confirmation. Statisticians call it "double dipping." The same piece of evidence votes once when it shapes the number, then votes again when it is asked to grade the number.

Neuroscience ran into a version of this before marketing measurement did. Kriegeskorte, Simmons, Bellgowan and Baker (2009) described a common practice in brain-imaging studies: pick the most active regions using one statistical test, then report the effect size of that same test within those regions, as if the second read were independent confirmation of the first. It wasn't. The region had been chosen precisely because that test looked strong there, so the reported effect sizes came out inflated. Some of them described patterns that weren't really there at all.

The MMM version has the same shape, with a channel standing in for a brain region and an experiment standing in for the statistical test. Calibrate Search's posterior with a lift test, then re-check that same lift test against Search's now-calibrated posterior, and you've made the identical move. Be precise about what did and didn't happen here. The posterior for Search really did move toward the truth, which is calibration working as intended, and the case for doing it is a good one: an MMM's own coefficient isn't a causal estimate by itself, and a randomized experiment tells the model something the spend data alone couldn't. What didn't happen is a second, independent confirmation. The posterior sits close to the experiment because the experiment is exactly what pulled it there. Calling that closeness "agreement between two independent methods" gets the numbers right and the meaning wrong.

One honest caveat belongs here rather than in a footnote later. No paper applies this specifically to marketing mix models yet. Kriegeskorte and coauthors were writing about brain imaging, not advertising, and a look through the public documentation of other calibration-forward MMM tools (including Google's Meridian, whose own docs are refreshingly candid about what calibration can and can't buy you) turns up plenty on estimand mismatch and holdout-window mechanics, but nothing on this specific reuse. So what follows is a well-established statistical idea, confirmed in a different field two decades ago, showing up here for what looks like the first time: applied to marketing measurement generally, and to this framework's own reporting code specifically.

Three Code Paths That Never Check Each Other

Three separate pieces of code touch this, and none of them talks to the other two. One tool decides which experiments get folded into a model's likelihood. A second piece of reporting logic decides which channels get labeled "experiment-validated." A third builds the triangulation panel that actually prints the word "Convergent." Each one does its own job correctly. The opportunity is that none of the three currently asks what the other two already know.

apply_experiment_calibration is the tool that folds an experiment into the next fit. Called with no arguments, it stages every completed experiment in the project by default, including ones already folded into an earlier fit, so a routine refit quietly keeps every calibration a model has ever had. That default is convenient. It is also the reason a team can drift, refit by refit, toward having calibrated away the very evidence they meant to keep in reserve. Passing explicit ids doesn't change that: they merge with whatever's already staged, and nothing in the tool distinguishes "fold this one in" from "leave that one alone."

The evidence-tier logic that labels a channel "experiment-validated" checks one thing: does this channel's name appear in the model's own list of experiments it was calibrated against? That's a reasonable question, and the answer is useful information. It just isn't the same question as "has this channel been checked against evidence the model hasn't already seen," and today nothing distinguishes the two answers from each other.

The clearest case is the triangulation report itself, since it's the piece that prints the actual word "Convergent." Left to its default, it pulls its "experiment" comparison point from that same calibration list:

// illustrative: triangulation.py, condensed
exps = experiments if experiments is not None else getattr(model, "experiments", None)

Called the common way, with no explicit list of experiments supplied, the report ends up comparing the model to the very evidence that shaped it, then narrating the result as agreement between "two independent methods." That's the sentence worth teaching the report to say more precisely. The model and the experiment aren't independent here, and a reader can't tell that from the badge alone.

None of these three pieces of code is doing anything wrong in isolation. The good next thing to build is in the seam between them: give the tool that stages a calibration input a way to tell the reporting code it just fed a channel its prior, and give the reporting code that prints "independent confirmation" a way to ask whether its input already passed through the likelihood. That is a small amount of shared state between three modules that each already do their own job well. No rewrite of any of them required.

The Registry Has No Role Field

You'd expect the experiment lifecycle to already track this somewhere: a status, a flag, anything that says "this one is for checking, not calibrating." It doesn't, yet. The registry's experiments (platform/sessions.py) move through a clean lifecycle, draft → planned → running → completed → calibrated, and each record keeps a channel, a design, a value with its uncertainty, and which model run consumed it. What it doesn't keep is a field for the job an experiment was planned to do. An experiment is either folded into a fit or it isn't, and there's no place today to say which one it was always meant to be. That's a small thing for a registry to be missing, and an easy one to add, which is part of why it's worth naming: the fix here doesn't require rethinking how experiments are run, only recording a decision a good team is already making informally.

The figure below makes the two disciplines concrete against a single, realistic plan: three geo-lift tests, one channel each, scheduled over a year. Move the slider to change how many of the three get folded into calibration, then watch what happens to the one thing a held-out check is supposed to supply. That thing is evidence the model has not already seen.

What the Badge Actually Shows

The registry gap explains how this can happen. It doesn't, on its own, show how misleading the resulting badge can look. For that, take a worked example with a planted truth, the usual house convention on this blog. What follows is a simulated illustration of the mechanism, not a real customer fit.

Take one channel, Display, whose true incremental ROI in this planted world is 1.20 exactly. Every dollar of Display spend really returns $1.20. Display's spend happens to track a demand signal the model can't fully see, so its uncalibrated estimate runs high: mean 2.05, a 71% overstatement of the truth. A geo-lift test on Display comes back close to the truth: 1.15 ± 0.18. Fold that one experiment in as a precision-weighted update, the same mechanics apply_experiment_calibration runs, and the posterior moves to 1.24 ± 0.17, just a 3% overstatement. That's calibration doing exactly what it's for: a badly biased estimate pulled to within a few percent of the truth by one honest experiment.

Now open the triangulation panel. It reports, correctly, that Display's model estimate (1.24) and Display's experiment (1.15) agree, well inside the reconciler's own 30% tolerance for calling two sources convergent. What the panel doesn't show, because nothing yet asks the question, is a second channel sitting in the same report. Paid Social was never touched by any experiment, and it carries the same kind of demand-chasing bias Display had before its own test: true ROI 0.85, uncalibrated estimate 1.62, a 90% overstatement with nothing to flag it. Display's badge says "Convergent." Social's number, right next to it, is wrong by almost the same margin Display was wrong by before its one lift test, and the green badge on Display gives no hint of that.

This Is Not an Argument for More Experiments

It's tempting to read the two figures above as an argument for running a fourth lift test, or a fifth, until there's one to spare. Fixing this takes no extra experiments at all, which is the good news hiding in this post. A team with exactly one geo-lift test on a channel can already do this right. Just don't fold it into calibration. Leave that channel's estimate resting on the model and its priors, and treat the one experiment on hand as the one honest, held-out check on the model's structure. That's a smaller, cheaper measurement program than adding a test, and every bit as rigorous. The habit worth building is deciding, on purpose, which job each already-planned experiment is going to do.

"We need more lift tests to do this properly" undersells what's available for free, right now: a decision about which planned experiment calibrates and which one stays reserved. More experiments do help with other things (more channels covered, tighter intervals, faster re-testing as effects drift), but they don't fix double dipping on their own. The shortage here was never data. It was bookkeeping, and bookkeeping is a good thing to get in the habit of doing early: decide, before anything runs, which use of a given piece of evidence comes first. The teams that build this into their experiment-planning workflow from the start won't experience it as a constraint. They'll just be the ones whose "Convergent" badge means what it says.

What This Does Not Establish

Passing a genuinely held-out check is real progress, but it isn't a certificate on the whole model. Everything the calibration post already says calibration can't fix applies here too. One held-out lift test is specific to its channel, its time window, its geography, its creative, and whatever else was happening competitively while it ran. Passing that check confirms the model's response for that one channel, at that time and place. It doesn't confirm the trend, the adstock choice, the controls on every other channel, or how the model behaves outside the window the experiment covered. Those still need the other checks that belong alongside a held-out experiment rather than in place of one: recovery against a known answer, refutation and placebo tests, and calibration diagnostics such as SBC.

None of this means the triangulation and evidence-tier reporting is broken or not worth having. It already shows, correctly, what a model was calibrated against, which is useful on its own. The finding here is narrower: that same reporting doesn't yet distinguish "this experiment confirms the model" from "this experiment built the model," and a reader can't tell which one they're looking at from the badge alone. That's a concrete thing to sharpen in an otherwise-working system. Sharpening it is no strike against the system, and none of this is a reason to stop trusting the rest of what the report tells you.

Building the Habit: Pre-Registering Roles, Not Just Designs

This framework already has the right instinct for half of this, and it just hasn't reached the other half yet. preregister_experiment locks a lift test's design (channel, window, target precision) before it runs, precisely so nobody can quietly re-argue the analysis after seeing the results. The pre-fit Model Design Readout does the same thing for the model itself: assumptions, priors, and prior-predictive checks written down before the fit ever sees data. Both are pre-registration of a design. The natural next step is pre-registering a role.

In practice that means one decision, made early: before any lift test in a program runs, for every channel with more than one planned experiment, settle up front which one calibrates the model and which one stays reserved to check it. Concretely, that's a role field on the experiment record, sitting next to calibrated_run_id (which already tracks which fit consumed an experiment), marking each one "calibration" or "holdout" at pre-registration time, locked the same way a design already is. It's a calibration tool that double-checks before staging anything marked holdout. And it's a triangulation report that can finally say which kind of "Convergent" it means: confirmed by a reserved experiment is a stronger claim than agrees with the experiment that built it, and the report should be able to tell a reader which one it's making.

None of that is built yet. Saying so plainly matters, because this is a roadmap and not a shipped feature. Until it lands, the same result comes from a simple habit. For any channel where a real held-out validation claim matters down the road, don't pass that experiment's id into a calibration call, for as long as the model lives, including refits where the tool's default would otherwise restage it. Write the decision down somewhere the next person doing a refit will actually see it. That one habit (reserving at least one experiment per channel, on purpose, before you need it) is available today, for free, with the experiments most teams are already planning to run.

A role field on an experiment record is a small thing to build. Nothing about it is a patch on something broken. It's the kind of small, concrete thing that makes a model steadily more trustworthy for the decisions riding on it, one more increment in a model that, like every model, is never really finished and just keeps getting more useful.

References

• Kriegeskorte, N., Simmons, W. K., Bellgowan, P. S. F., & Baker, C. I. (2009). Circular Analysis in Systems Neuroscience: The Dangers of Double Dipping. Nature Neuroscience, 12(5), 535–540.
• Leamer, E. E. (1978). Specification Searches: Ad Hoc Inference with Nonexperimental Data. Wiley.