Closed-Loop Measurement & Calibration

A comprehensive guide to integrating measurement into Marketing Mix Modeling. The Bayesian MMM and geo-lift experiments are not competing paradigms - they are complementary nodes in one inference graph. This page lays out how to wire them into a single self-correcting cycle that compounds learning over time.

The Two-Paradigm Problem

The MMM is the system of record for "what is each channel worth." It produces a clean ROI per channel, and the planning team allocates budget against those numbers. Separately, the team runs geo-lift experiments - paid search holdouts, CTV market tests, audio matched-market trials - each producing its own causal estimate. Sometimes the two agree; sometimes they don't. When they disagree, someone picks the source they trust more for that channel and moves on.

The structural problem is not that either system is bad - both are well-built. The problem is that neither is wired into the other. The MMM doesn't know which channels need experimental backup; the experiments don't feed back into the next MMM fit. So the team keeps paying for both and keeps accepting whichever number the analyst trusted that week.

Two Complementary Objectives

Once you accept that experiments are the calibration tool, you still face the question of which experiments. Two distinct objectives emerge - they are correlated but not identical, and a good framework integrates both.

A boutique channel may have huge posterior variance (high epistemic priority) but receive 1% of spend (low decision stakes). A workhorse channel may have moderate variance but receive 30% of spend, making any uncertainty extraordinarily costly. EIG and EVOI together produce a priority map that respects both.

Bayesian Foundations: Prior, Likelihood, Posterior

Bayesian inference is the discipline of updating beliefs in the face of new evidence. Three objects appear repeatedly in this guide:

• Prior $p(\theta)$ - what you believe about an unknown parameter $\theta$ before seeing experimental data. For us, this is what the MMM tells us about a channel's ROI.
• Likelihood $p(\hat{y} \mid \theta)$ - the probability of observing the experimental outcome $\hat{y}$ if the truth were $\theta$.
• Posterior $p(\theta \mid \hat{y})$ - the updated belief about $\theta$ after seeing $\hat{y}$.

For the rest of this guide, $\theta_k$ denotes a channel-level causal parameter (typically ROI or elasticity) for channel $k$, and $\hat{y}_k$ denotes the noisy estimate produced by an experiment. The MMM provides $p(\theta_k)$. The experiment provides $p(\hat{y}_k \mid \theta_k)$. The product is what we want.

The Gaussian Conjugate Update

When the prior is Gaussian and the experiment likelihood is Gaussian (the standard assumption for geo-lift difference-in-differences), the math simplifies dramatically. With prior $\theta \sim \mathcal{N}(\mu_0, \sigma_0^2)$ and likelihood $\hat{y} \mid \theta \sim \mathcal{N}(\theta, \sigma_e^2)$, the posterior is also Gaussian:

Equation (2) is the workhorse for almost everything that follows. EIG, calibration, and the bridge from frequentist estimates all reduce to applications of this single update.

Decision Structure

Let $\theta_k$ denote the true causal ROI for channel $k \in \{1, \ldots, K\}$. The MMM produces a joint posterior $p(\boldsymbol{\theta}) = p(\theta_1, \ldots, \theta_K \mid y, X, \mathcal{M})$. The planner selects an experimental portfolio $A \subseteq \{1, \ldots, K\}$ together with a design $d_k$ for each chosen channel:

Each experiment produces a noisy estimate of the true causal effect. For a geo-lift difference-in-differences design, this is approximately Gaussian:

where $s^2_{\text{geo},k}$ is the residual geo-week variance from the pre-period, $n$ is geos per arm, and $T$ is the test duration in weeks. More geos, longer tests, and bigger spend deltas all shrink experimental noise.

Expected Information Gain (EIG)

For a given channel $k$ and design $d$, the Expected Information Gain is the expected KL divergence from the prior to the updated posterior, integrated over what experimental outcome we might see. Equivalently, it is the mutual information between the unknown parameter and the experimental outcome:

Under the Gaussian-Gaussian setup, EIG has a remarkably clean closed form. Because the posterior variance does not depend on the realized value of $\hat{y}$, the expected entropy reduction simplifies to:

Practical inputs required

• Extract MMM marginals - for each channel $k$, compute $\sigma_k$ from the posterior samples of its ROI coefficient. Use the marginal standard deviation, not the standard error of the posterior mean.
• Estimate experimental noise per design - estimate $\sigma_{\text{exp},k}^2(d)$ from a power analysis fit on the pre-period geo-week panel. This links design choices to achievable precision.
• Compute the EIG grid - sweep across channels x designs, apply Eq. (6), and produce a ranked grid of $(k, d)$ pairs.
• Constrain & select - apply geo non-overlap, budget caps, minimum duration, and non-interference. Solve the constrained selection problem; greedy is usually sufficient (see Submodularity).

For non-Gaussian posteriors - hierarchical models, non-conjugate priors, skewed contributions - a nested Monte Carlo estimator works in $O(N^2)$ samples. For prioritization purposes, the Gaussian approximation in Eq. (6) is almost always sufficient. Monte Carlo refinement is reserved for borderline channels.

Expected Value of Information (EVOI)

EIG treats every bit of uncertainty reduction as equally valuable. But uncertainty about a channel receiving 1% of spend is far less costly than equivalent uncertainty about a channel receiving 40% of spend. EVOI prices uncertainty in dollars by accounting for the asymmetric cost of being wrong.

Define the downstream action as a budget allocation $\mathbf{b} = (b_1, \ldots, b_K)$ with $\sum_k b_k = B$. The organization picks $\mathbf{b}^*$ based on its current beliefs and earns utility $U(\mathbf{b}, \boldsymbol{\theta}) = \sum_k \theta_k \cdot f(b_k)$, where $f(\cdot)$ is a (usually concave) channel response function.

Closed-form EVOI is generally hard, but a useful linearized approximation drops out when reallocations only happen if the experiment changes the ordering of channels - i.e., if the posterior mean crosses a decision threshold:

Structural drivers

• High prior uncertainty ($\sigma_k$ large) - wide credible intervals mean large potential for belief revision, increasing EVOI.
• High spend share ($s_k$ large) - channels receiving big budget fractions have higher stakes; errors cost more.
• Decision threshold proximity - when the posterior mean is near a budget reallocation threshold (e.g., borderline ROI = 1.0), even small uncertainty reductions have high value.
• Convex response (saturation) - when the response function is concave, correcting misallocation has compounding returns.

The 2x2 Priority Map

Combine EIG and EVOI into a single priority score and a stakeholder-friendly visualization:

where $\widetilde{\cdot}$ denotes min-max normalization to $[0, 1]$, $\lambda$ trades off epistemic vs. instrumental value, $\gamma$ penalizes expensive experiments, and $\rho$ penalizes operational risk.

Submodularity & Greedy Selection

A non-obvious but enormously useful fact: the EIG of a set of experiments is a submodular function of the set. Submodularity means diminishing returns - adding the same experiment to a small portfolio yields more information than adding it to a large one. Formally, for sets $A \subseteq A'$:

Submodularity has a major practical payoff: greedy selection is provably near-optimal. The Nemhauser-Wolsey-Fisher (1978) result says greedy attains at least $(1 - 1/e) \approx 63\%$ of the optimum under cardinality constraints. The pseudocode is trivial:

# Greedy portfolio construction
A = []
while len(A) < C_ops:
    feasible = [k for k in channels if is_feasible(k, A)]
    if not feasible: break
    k_star = argmax(feasible, key=lambda k: score(k, A))
    A.append(k_star)
# By submodularity: EIG(A_greedy) ≥ (1 - 1/e) · EIG(A_optimal)

Practical implication: after about three experiments per cycle, additional ones add diminishing returns. The cycle naturally caps the test program at a sensible level - so any one quarter's experimental budget should be spent on the top 2-3 priorities, not spread thin.

Experimental Design: From Channel Selection to Experimental Spec

Selecting which channel to test is a prioritization problem (EIG, EVOI, §3–4). Specifying how to run the test — which geos to treat, how much to spend, how long to run — is a distinct design problem. A geo-level Bayesian MMM provides the inputs needed to answer every design question rigorously, replacing heuristic defaults with posterior-derived quantities.

Quantities the MMM Provides

Step 1 · Treated-Geo Selection by Information Yield

Not all geos contribute equally to resolving channel uncertainty. The information yield $I_{k,g}$ scores each geo by how much a test there would compress the posterior on $\beta_{k,g}$:

The numerator is the product of prior uncertainty (wide posterior = more to learn) and saturation slope squared (steep curve = spend increment translates to large signal). The denominator is residual noise. A geo saturated for channel $k$ has a small $S_k'(x_{k,g})$ and therefore low yield regardless of how uncertain the posterior is. Rank geos descending by $I_{k,g}$ and select the top $G$.

Step 2 · Matched-Pair Construction via Posterior Mahalanobis Distance

For each selected treatment geo $g$, choose a control geo $g'$ that minimises posterior Mahalanobis distance over the geo random effects $\mathbf{u}_g$:

where $\boldsymbol{\mu}_{u,g}$ is the posterior mean of geo $g$'s random-effect vector and $\boldsymbol{\Sigma}_u$ is the posterior covariance matrix of those effects. Matching on Mahalanobis distance over the full posterior — rather than on raw KPI correlation — accounts for parameter uncertainty and produces 20–40% lower residual variance in synthetic studies. Exclude any control candidate with cross-geo posterior correlation $|\mathrm{Cor}(\beta_{k,g},\beta_{k,g'})| > 0.5$ (spillover screen; see Step 6).

Step 3 · Treatment Intensity from the Saturation Curve

The optimal spend increment for geo $g$ equates signal-to-noise across the panel:

where $\eta$ is target power, $G$ is the number of treatment geos, and $T$ is planned duration. The factor $S_k'(x_{k,g})^{-1}$ is the key innovation: a geo near saturation (small slope) requires a proportionally larger spend increment to produce the same signal as a sub-saturated geo. In practice $\Delta\mathrm{spend}_g^*$ varies 3–5× across geos for the same channel; applying a uniform percentage uplift systematically under-powers some geos and wastes budget in others.

Step 4 · Duration from Serial Correlation

Weekly KPI within a geo is serially correlated with AR(1) coefficient $\rho_g$, estimated from the MMM residuals. The corrected standard error for the difference-in-differences estimator is:

When $\rho_g = 0$ this reduces to the familiar $\sigma/\sqrt{T}$ form. The minimum duration $T^*$ for target power $1-\beta$ solves:

The implicit equation is solved by one-dimensional search. For empirically common values $\rho_g \in [0.2, 0.5]$, EIG flattens after 6–10 weeks — running longer accumulates cost with minimal additional information.

Step 5 · Pre-Experiment Simulation

Before committing budget, validate the proposed design by using the fitted MMM as a forward simulator:

1. Sample $\tilde\theta$ from the posterior $p(\theta \mid \mathcal{D})$.
2. Generate simulated KPI for each geo under the proposed spend schedule, adding noise $\mathcal{N}(0,\sigma_g^2)$ scaled by the serial-correlation factor $\sqrt{1+(T-1)\rho_g}$.
3. Fit the planned estimator (DiD, SCM, etc.) to the simulated data and record whether it reaches significance at $\alpha$.
4. Repeat 500–2,000 times; the fraction of significant replicates is the simulated power.

Simulated power catches issues — skewed KPI distributions, heterogeneous geo variance, adstock contamination — that closed-form approximations miss. If simulated power falls short, adjust $\Delta\mathrm{spend}_g^*$, add geos, or extend duration and re-simulate.

Step 6 · Spillover and Contamination Diagnostics

Cross-geo spillover (national TV halo, retargeting bleed across DMAs) deflates the estimated lift by partially treating the control. Diagnose contamination with the cross-geo posterior correlation:

Flag any control candidate with $|r_{g,g'}| > 0.5$ and exclude it from the donor pool. Adstock carryover introduces time-domain contamination in the first 1–2 weeks; include a wash-in buffer or model carryover explicitly in the geo-level analysis. Both diagnostics are computed from the fitted MMM posterior — no additional data collection is required.

Worked Example — CTV Campaign

Five candidate geos were scored for a Connected-TV channel. The table summarises the design output of Steps 1–6.

Atlanta's lower $\Delta\mathrm{spend}^*$ (18% vs 31%) reflects its steeper saturation slope: the same ROI uncertainty is resolved more cheaply where the response curve is still responsive. St. Louis was excluded despite ranking 3rd by $I_{k,g}$ because its cross-geo posterior correlation with Atlanta ($r = 0.71$) exceeded the spillover threshold.

Calibration: Using Experiments to Anchor the Next MMM

Calibration is the process of using experimental causal estimates to anchor the MMM's channel parameters so that model-implied ROIs are consistent with ground truth. This can range from soft regularization (Bayesian priors informed by experiments) to hard constraints (fixing parameters at experimental point estimates - which we generally avoid). Three mechanisms cover the main use cases.

Mechanism 1 - Soft calibration via Bayesian priors

The experiment on channel $k$ produces $p(\tau_k \mid \hat{y}_k)$. Pass that distribution as the prior on the corresponding MMM parameter:

The scale transform matters. A geo-lift $\hat{\tau}$ is in outcome units per spend-dollar in the test window, while an MMM coefficient may be in standardized units. Convert carefully and document the conversion rule in the calibration spec.

Mechanism 2 - Likelihood augmentation

where $g(\theta_k)$ maps MMM channel parameters to the implied geo-lift effect size - accounting for the spend delta in the test, the saturation curve evaluated at the test spend level, and any adstock effects in the experiment window. Constructing $g(\cdot)$ correctly is the hardest part of this mechanism; miscalibration introduces systematic bias.

Mechanism 3 - Hierarchical pooling

where $\mathrm{cat}[k]$ is the media category of channel $k$ (e.g., video, search, social). An experiment on CTV updates $\mu_{\text{video}}$, which partially informs the OLV estimate. Pooling strength is controlled by $\sigma_{\text{cat}}$ via a hyperprior - tighter $\sigma_{\text{cat}}$ pools more strongly.

Calibration validity checks

• Scale consistency - verify that the experimental $\hat{\tau}_k$ and the MMM-implied effect use the same outcome variable and time aggregation.
• Coverage check - confirm that the MMM prior 95% CI for channel $k$ covers the experimental point estimate. If not, diagnose model misspecification before calibrating.
• Quality gate - only use experiments that pass pre-specified quality checks (balance test, pre-period MAPE, sufficient MDE, no contamination evidence) for calibration inputs.
• Attribution reconciliation - after refitting, confirm total attributed conversions still match observed outcomes - calibration should not break aggregate coherence.
• Predictive validation - hold out a test period and compare pre- vs. post-calibration MAPE. Calibration should improve predictive accuracy or at minimum not degrade it.

The Virtuous Cycle: Six-Step Orbit

The full adaptive loop treats MMM calibration as a sequential Bayesian inference problem across measurement cycles. Each cycle has six steps - three modeling steps that happen against historical data, and three field steps that happen in the market.

Information decay & re-experimentation triggers

Experimental calibration has a shelf life. An experiment from 18 months ago, run under different competitive conditions, spend levels, or creative strategies, may no longer accurately represent current channel effectiveness. Model an exponential decay of experimental information over time:

where $\lambda_{\text{decay}}$ is calibrated from observed year-over-year MMM coefficient stability. As $\sigma_{k,\text{eff}}^2$ grows, EIG and EVOI for re-experimentation rise again - creating a principled re-experimentation schedule rather than relying on calendar-based intuitions. Decay rates differ by channel: fast-moving digital decays in 6-12 months; stable broadcast can hold 18-24.

Adaptive stopping criteria

A channel exits the active experimentation pool when all three conditions hold:

$T_{\text{refresh}}$ depends on media market dynamics, creative/targeting shifts, and competitive changes. Typical values: 12-24 months for stable channels, 6-12 months for fast-moving digital surfaces.

Why It Compounds

The point of running this loop is not any single experiment - it is the compounding of belief updates over successive cycles. A useful way to see the value is to plot the trajectories of four quantities that each cycle is trying to move:

1. MMM posterior precision - the 95% credible interval width on each channel's ROI. As experiments calibrate channels, widths contract.
2. Budget allocation - the share of total spend going to each channel. As ROIs become better-identified, allocations migrate toward the (unknown but increasingly well-estimated) optimum.
3. Misallocation cost - the gap between current spend and the spend that would maximize portfolio outcome under the current posterior. The running tab of wasted dollars per week.
4. Decision efficiency - the fraction of the theoretically optimal portfolio return that is actually captured, averaged over the current posterior. Rises as sigma shrinks and allocation converges to the optimum.

For a representative five-channel portfolio over five quarterly cycles, the loop produces the kind of trajectory below. The exact magnitudes depend on starting MMM uncertainty and spend asymmetry, but the shape - sharp early gains followed by diminishing returns - is structural.

Frequentist Tools, Bayesian Framing

Most geo-lift estimators in active production use are frequentist by construction: difference-in-differences with two-way fixed effects, synthetic control, augmented synthetic control, and time-based regression. Stakeholders speak that language. Pre-registration documents demand it. None of this is in tension with the Bayesian framework above.

The Bridge Equation

A frequentist point estimate $\hat{\beta}_k$ paired with a standard error $\mathrm{SE}_k$ is - under mild assumptions - a Gaussian likelihood. That likelihood plugs straight into the conjugate update alongside the MMM prior:

The translation table is short: estimator output $\hat{\beta}_k$ becomes the likelihood mean; $\mathrm{SE}_k^2$ becomes the likelihood variance $\sigma_e^2$; the MDE relates to the achievable $\sigma_{\text{exp}}$ through the design's power calculation. Every frequentist estimator the measurement team already runs - DiD, SCM, ASC, time-based regression, BSTS / CausalImpact - feeds the same loop without anyone switching teams.

When the bridge isn't enough - small samples, multiple competing estimators, sequential monitoring, or borrowing strength across markets - go fully Bayesian on the experiment side too. Tools include BEST (robust Bayesian estimation), HDI + ROPE for decision-theoretic readouts, hierarchical / multilevel models, Bayesian synthetic control, and sequential Bayesian inference (no alpha-spending required). These all output posteriors that plug into Eq. (11) directly.

Implementation: A 90-Day Rollout

A staged rollout with explicit decision gates. Phase 1 is a 90-day pilot that delivers decision-grade evidence about whether to continue. The steady-state quarterly cadence is what comes after.

Prerequisites

• Geo-week panel data - 3+ years of the outcome metric (revenue, conversions, sessions) at the geo grain. Daily is fine; the model will aggregate.
• Per-channel media spend at the same geo grain or with a documented allocation rule from national to geo.
• Pre-period MAPE benchmarks from the existing MMM, by channel and aggregate. Needed to credibly compare the new pipeline.
• Past geo-lift readouts with point estimates and standard errors. These are the calibration inputs for the first cycle. Verdicts ("CTV worked") are not enough - we need the underlying numbers.
• Tooling: PyMC ≥ 5 or NumPyro for the Bayesian MMM and hierarchical experiment models; ArviZ for diagnostics; pyfixest or fixest (R) for the frequentist DiD baseline; JAX optional but recommended for speed.

Success Metrics

Anti-Patterns & Failure Modes

Six failure modes seen in similar programs. Calling them out up front so we recognize them when they happen.

• "Run all the experiments we can think of." Submodularity says marginal information per experiment falls fast. Running 8 experiments to learn what 3 would have taught you is wasted budget. The priority engine is supposed to cap the program; treating it as an unconstrained idea-generator defeats the point.
• "Calibrate every channel, every cycle." Calibration has a shelf life and a cost. Re-experimenting on a channel whose last test is 4 months old and stable is just lighting test budget on fire. Use the re-experimentation trigger; it exists for this reason.
• "Use a flat prior so we're not biasing the answer." A flat prior reproduces the OLS pathology. Priors are how you regularize; refusing to use informative priors is a refusal to do the regularization work. Defensible > flat.
• "The Bayesian MMM is wider than OLS, so it's worse." The Bayesian MMM is wider because the OLS interval was lying about its precision. Width is information, not weakness. Reporting templates need to make this distinction explicit.
• "Let's just plug the experimental point estimate in as a fixed parameter." Hard calibration by point-substitution discards the experimental SE and produces over-confidence in the calibrated channel. Use the soft prior (M1) or augmented likelihood (M2). Never the point.
• "The model said it; I just report what comes out." The Bayesian MMM "says" what your priors and likelihood structure say. The analyst owns those choices; reporting templates should attribute structural decisions to the analyst, not to "the model."

Closing Principle