Skip to content
Universal Model ProtocolDocs

Calibration (closes gap 5)

§10NormativeDraft 2.3Latest draft

A test result updates a model through one of six routes, after a fixed sequence of checks, and every update produces a new model version whose calibration record states the route, the excision window, the diagnostics and any override.

1. Checks, in order, before any route
  1. 0 Double countingExcise the flight or model it jointly
  2. 1 EstimandSeven fields and scope match
  3. 2 AdmissibilityClass A, B or C only; others set aside, G-EXP-05
  4. 3 GranularityTest grain mapped to parameters
  5. 4 ConflictFlagged results run provisionally
  6. 5 Route diagnosticsA failure falls back to the next route
  7. 6 Post-checkWorse held-out accuracy keeps it provisional
  • R1 PriorNeeds Tier A. The test's result becomes the channel's starting belief (its prior) at refit. Certified.
  • R2 LikelihoodNeeds Tier A. The test is added to the fit as extra evidence (a likelihood term), including interval lift. Certified for MROAS and CONTRIBUTION; interval lift at pilot until study 3.
  • R3 PoolingNeeds Tier A, hierarchical. The test updates a category-level setting (a hyperparameter) that informs sibling channels. Certified.
  • R4 ReweightNeeds Tier A or B with joint draws. Existing draws are importance-reweighted; no refit. Used while PSIS passes; otherwise an approximate summary update.
  • R5 Curve adjustNeeds Tier B. Curve parameters are rescaled near the tested spend. Pilot until study 3.
  • R6 Attach onlyAny tier. The result is stored beside the model, unchanged. Default for Tier C, D, X.

A test result never edits a model in place. It passes the checks, then takes one of six routes. Every update makes a new model version, with a calibration record that names the route, the excision window, the diagnostics and any override.

How deep a route goes depends on how much of the model we can see. Tiers C, D and X keep the result beside the model by default (R6).

Figure 10.1 Six routes from a test result into a model. Every result passes the same checks first, admissibility among them. Pick a model tier to see which routes it opens: tiers C, D and X keep results beside the model, unchanged.

Routes

RouteNeedsIdeaStatus in v1
R1 PriorTier AExperimental posterior becomes the channel prior at refitSupported
R2 LikelihoodTier AExperiment added as a likelihood term, including interval liftSupported for MROAS and CONTRIBUTION; interval lift at pilot until study 3
R3 PoolingTier A, hierarchicalTest updates a category hyperparameter that informs siblingsSupported
R4 ReweightTier A or B with joint drawsImportance-reweight existing draws; no refitUsed while PSIS passes; otherwise an approximate summary update
R5 Curve adjustTier BRescale parameters near the tested spendPilot until study 3
R6 Attach onlyAnyStore the result beside the model, unchangedDefault for Tier C, D, X

Checks, in order (G-CAL-*)

  1. Double counting. If the flight sits in the training window, the excision or joint-model choice is recorded; if nobody chose, the default excision is applied and labelled (G-CAL-02).

  2. Estimand. Seven fields and scope match, or the alignment ladder in section 5 is applied and recorded (G-EST-02).

  3. Admissibility. Class A, B or C evidence is used for sales-return calibration, and class E for brand-family calibration only; anything else is set aside and the step continues without it (G-EXP-05).

    2a. Measurement mapping. For Tier C, D and X models, any requested route is replaced by R6 (attach only). For other tiers, the test’s estimand MUST map onto a model parameter or curve. Otherwise the result is attached only (G-CAL-04).

  4. Granularity. The mapping from the test’s grain to the model’s parameters is recorded, or the default split rule is applied and labelled (G-CAL-03). With no registered split rule, the result is attached only.

  5. Conflict. Multiplicity-controlled check; a flagged calibration runs provisionally under a robust mixture prior and routes to the measurement lead with the diagnostic checklist in the calibration record (estimand match, spillover, duration versus carryover floor, prior sensitivity) (G-CAL-01).

  6. Route diagnostics. PSIS for R4, convergence and held-out accuracy for R1 to R3; a failure falls back to the next route rather than stopping (G-CAL-05).

  7. Post-check. If held-out predictive accuracy degrades, the new version is kept as provisional and the previous version stays on the production endpoint until review.

Interval lift, credited and extended. PyMC-Marketing’s add_lift_test_measurements already scores a lift test as the difference in a channel’s saturation curve between x and x + Δx for each posterior draw, so the core idea is not new and UMP credits it. UMP’s extension adds what that version leaves out:

\Delta y^{(s)} = \beta^{(s)} \sum_{t \in W} \Big[ S\big(A^{(s)}(x_0 + \Delta x)_t\big) - S\big(A^{(s)}(x_0)_t\big) \Big]

Here s indexes posterior draws, A is the adstock operator applied to the spend path, S the saturation curve and W the measurement window including cool-down. The extension has four parts:

  • adstock dynamics inside and after the flight, not an instantaneous step;
  • both arms of an asymmetric design (for example −30% and +30%) scored jointly against one control;
  • a signed likelihood (Student-t on the signed effect) instead of a Gamma on absolute values, so cut arms keep their sign;
  • the same construction for Tier B curves, and for Tier C tables only after simulation validates it.

Interval lift stays at pilot status until the calibration bake-off (study 3) shows coverage within 5 points of nominal on partial-cut designs (proposed).

Reweighting degeneracy. When the experiment is far more precise than the prior (7.7 times in one illustrative case), importance weights collapse onto a few draws. R4 MUST compute Pareto-smoothed importance sampling diagnostics (Vehtari et al., 2024). With S draws, R4 proceeds only if k̂ < min(1 − 1/log10 S, 0.7) and the PSIS effective sample size, 1 / (sum of squared Pareto-smoothed normalised weights), multiplied by the draws’ relative MCMC efficiency r_eff, with S the draw count on the bundle-wide thinning index, is at least 400 (proposed). Otherwise R4 falls back to a summary-level update on the channel’s marginal, labelled approximate. The update is a precision-weighted normal update: with prior mean mu_0 and SD sigma_0 for the channel parameter, and test estimate y with standard error sigma_e (both on the estimand of section 5), mu_post = (mu_0/sigma_0^2 + y/sigma_e^2) / (1/sigma_0^2 + 1/sigma_e^2) and sigma_post^2 = 1 / (1/sigma_0^2 + 1/sigma_e^2). The draws are then moment-matched to (mu_post, sigma_post), and a refit through R1 or R2 is queued. The response carries PSIS_UNRELIABLE as a fallback note, not an error. The moment-matched fallback leaves the channel’s draws no longer jointly consistent with the rest, so the calibrated output carries dependence per_channel for that channel until the queued refit lands.

Excision, with adstock tails. The excised window runs from flight start to flight end plus L weeks, where L is the 95th percentile across draws of the lag at which the channel’s cumulative adstock weight reaches 95%. Only the tested channel’s rows in the tested geos are excised or zero-weighted. The alternative is joint modelling: keep the rows and encode treatment assignment in the MMM’s likelihood, so the experiment’s data enter once. The record states which was done.

Granularity. Geo-set tests map to those geos’ coefficients through the hierarchy, with recorded weights, not to national β. A national mapping requires a representativeness check of the test geos. Placement tests (Reels) map to model media (Meta) through the recorded split assumption from section 4, and the calibration routes to review.

Conflict thresholds with multiplicity. Each test’s conflict statistic is z = (y - E[y_rep]) / sqrt(Var[y_rep] + sigma_e^2), where y_rep is the model’s pre-calibration posterior-predictive draw of the test’s estimand at the tested spend, scope and window. A calibration cycle is the set of tests calibrated in one calibration artifact. When the cycle holds one test, the test is flagged when |z| > 2 (proposed). When it holds two or more, a test is flagged exactly when the Benjamini-Hochberg step-up procedure at q = 0.10 (proposed) rejects it on the two-sided p-values of the z statistics. The |z| > 2 rule is not applied in addition. BH assumes positive regression dependence, which tests sharing a posterior satisfy only approximately. When tests share geos or flight weeks, the Benjamini-Yekutieli procedure MAY be declared instead. The calibration record names the procedure used and m. A flagged calibration still runs, provisionally: the test enters through a robust mixture prior (weight 0.8 on the experiment likelihood and 0.2 on a vague component centred on the model’s prior with SD 1.0 on the log-effect scale, proposed, recorded on the calibration) whose vague component limits how far one surprising result can pull the model, and the reviewer confirms, revises or reverts it. Every override is logged with reviewer and reason.

Vendor curves. Tier C tables are attach-only for calibration. A test result may lower their trust or trigger a request for a better export, but UMP does not rescale a black-box table until a method passes the bake-off.