Guide · doing the work
Folding an experiment in
You have an observational model of the whole system and one clean experimental measurement of a small corner of it. The experiment is far more trustworthy and far less complete. Neither on its own answers the decision. Calibration is how the trustworthy thing disciplines the complete one.
Why not just use the experiment
The experiment answers a smaller question than the decision asks
On the calibrate page, an experiment measures a first-period lift of 5.322 with a standard error of 0.053. That is a precise, randomized, defensible number.
The decision needs something else: the steady-state effect of a dose the experiment never tested, which is 32. The experiment measured one point on a curve; the decision asks about a different point on the same curve, after the dynamics have settled.
So the experiment cannot answer the question by itself, and the observational model — fitted on data where treatment and confounder correlate at 0.96 — cannot be trusted on its own either. Calibration is the operation that uses each for what it is good at.
This is a transfer, and it needs a licence
Moving from “first-period lift at the tested
dose” to “steady-state effect at the decision dose” crosses the
window and intervention facets. Every word of
the transportability guide applies, and the
corrections are recorded in a ledger rather than absorbed into a fitted
parameter.
The measurement knows what it measured
A Measurement in axiom carries the
estimand it is a measurement of — here
first_period_lift@rct-2026Q1, with target hash
4760098b3929. That is what lets the
transfer be checked rather than assumed.
The three routes
Where the experiment enters the model changes what it does
There is no single “incorporate the experiment” operation. There are three, they make different claims, and they give different answers on the same data.
As a prior
Turn the measurement into a prior on the relevant parameter and fit as usual. Simple and familiar — but it says the experiment constrains that parameter, when what it actually constrains is a derived quantity the parameter only partly controls.
Says: “my belief about this parameter is now centred here.”
As a likelihood term
Add the measurement to the likelihood as an observation of
the derived quantity, computed through the same forward() the model
uses. The experiment then constrains whatever combination of parameters actually
produces it.
Says: “whatever the parameters are, they must reproduce this measurement.”
As a hard constraint
Require the model to match the measurement exactly. Rarely right — it treats a measurement with a standard error as though it had none, and it makes the experiment unfalsifiable by the rest of the data.
Says: “this is true, full stop.”
The middle row is the instructive one. By the obvious diagnostic — is the parameter near its true value — the prior route looks best of the three. By the only diagnostic that matters — does the model agree with the experiment on the quantity the experiment measured — it fails. A parameter is not a quantity, and calibrating the first does not calibrate the second.
The check
Agreement, stated as a number
After calibrating, ask the model to predict the thing the experiment measured, and compare. That comparison is a z statistic against the measurement's own standard error, and it is the check that distinguishes the three routes above: -4.29, -2.82 and -0.64 respectively.
A model that cannot reproduce the experiment it was calibrated against has not been calibrated. It has been nudged, and the nudge went somewhere else in the parameter space.
When the model and the experiment genuinely disagree after an honest fold-in, that is information — usually about the transfer, not the experiment. The window was wrong, the population differed, or the dose the experiment tested is on a part of the curve the model has the wrong shape for.
Calibration is a loop, not a step
Fit, find what the decision is most sensitive to, measure that, fold it in, re-decide, and look again at what is now most uncertain. Each turn tells you what the next experiment should be. The depot tutorial runs one full turn, and the experiment reverses the decision the observational fit would have made.
Keep the ledger across the loop
Every fold-in writes lines — the pooling of measurements, the design factor, the transfer corrections. After three turns that ledger is the only account of why the current model believes what it believes. See provenance.