A Geo-Holdout Is a Bayesian Experimental Design: Computing Its EIG

Matthew Reda Experimental design

Most teams choose geo tests by intuition: hold out a familiar set of markets, run for six weeks, hope the readout is clean. But once you have a fitted media mix model, the value of any proposed test is a computable quantity — the expected information gain. This post walks through the encoding that makes that possible, and the machinery that makes it tractable.

A geo-holdout looks like an operational decision — which markets go dark, for how long — but it is exactly the object that Bayesian experimental design (BED) was built to optimize. The trick is to notice that a fitted MMM already provides two of the three things BED needs, and that the holdout itself is just the third: a point in a design space. Once the pieces are named, "which test should we run?" stops being a debate and becomes an optimization problem.

The Three Ingredients

Bayesian experimental design needs a prior over unknowns, a space of possible experiments, and a likelihood connecting the two. For a geo-holdout run on top of a media mix model, all three are already on the table:

💡 The MMM is the simulator

This is the load-bearing observation. Estimating the value of an experiment requires simulating its outcomes, and a Bayesian MMM does that for free: draw parameters from the posterior, feed them the perturbed spend plan, and sample sales. No auxiliary model, no separate power-analysis tool — the same object you use for budget decisions scores your experiments.

The Holdout as a Design Vector

A geo experiment partitions markets into treatment and control and imposes a spend change on the treated set. Written out as a vector:

$$ \xi = \big(\, a_g \in \{0,1\},\; m,\; \Delta_g,\; [t_0, t_1] \,\big) $$

with one component per decision the team actually makes:

The observation \( y \) is then the vector of realized geo-week sales over the test and post-test periods. Classical geo methodology — the geo-based regression designs of Vaver and Koehler (2011) and the time-based regression estimator of Kerman, Wang, and Vaver (2017) — works with exactly this design object; the Bayesian treatment changes how the design is chosen, not what it is.

What Is the Experiment Worth?

The currency of Bayesian experimental design is entropy reduction. For a particular hypothetical outcome \( y \), the information gained is the drop in Shannon entropy from prior to posterior. Since \( y \) is unknown at design time, we average over the outcomes the model itself predicts:

Definition: Expected Information Gain

With the marginal predictive \( p(y \mid \xi) = \mathbb{E}_{p(\theta)}[\, p(y \mid \theta, \xi) \,] \), the expected information gain of design \( \xi \) is

$$ \mathrm{EIG}(\xi) \;=\; \mathbb{E}_{p(\theta)\,p(y \mid \theta, \xi)}\!\left[ \log p(y \mid \theta, \xi) - \log p(y \mid \xi) \right], $$

the mutual information between parameters and outcome under design \( \xi \). The Bayesian optimal design is \( \xi^* = \arg\max_\xi \mathrm{EIG}(\xi) \) (Lindley, 1956; Rainforth et al., 2024).

The EIG has several equivalent forms — as a posterior-to-prior log ratio, as mutual information, as a likelihood-to-marginal log ratio — and each suggests a different estimator. The likelihood form above is the convenient one for MMMs, where the parameter vector is large but the outcome (a panel of geo-week sales) is comparatively low-dimensional.

How big a holdout?

Hold out a fraction \( h \) of \( G=50 \) geos and cut their spend. The effect estimate's precision scales with \( h(1-h) \) — both a treated and a control group are needed — so the information a Gaussian effect earns, \( \mathrm{EIG}=\tfrac12\log_2\!\big(1+\kappa\,s_\beta^2\,h(1-h)\big) \), peaks at a half split. Net the forgone value of the dark geos against it and the recommended holdout pulls back below one-half. Slide the split to move the marker; widen the effect-size prior to see a bigger test become worth taking.

0.30
1.2
EIG at h—
Net value at h—
Recommended holdout—
Verdict—

Too small a holdout starves the signal; too large sacrifices revenue faster than it buys bits. The information-optimal split is a half, but once the value at risk is priced in, the net-value peak sits nearer a third — and moves toward a half as the effect you are chasing grows more uncertain.

Aiming at What You Care About

The plain EIG rewards information about all of \( \theta \) — including main effects the model has already learned well from observational variation. A test chosen that way spends budget re-confirming what you know. The fix is to name the decision-relevant subset \( \psi \subset \theta \) — say, the TV×search interaction and the marginal ROAS it implies — and target the marginal EIG:

$$ \mathrm{EIG}_{\psi}(\xi) \;=\; \mathbb{E}_{p(\psi, y \mid \xi)}\!\left[ \log \frac{p(\psi \mid y, \xi)}{p(\psi)} \right], $$

with nuisance parameters integrated out. Under this objective the optimizer routes the experiment's information budget toward the quantities the business decision actually turns on. In practice this is the difference between a test that tightens a headline ROAS you already trust and one that resolves the interaction term blocking a reallocation decision.

Computing the EIG

The EIG is doubly intractable: the outer expectation is an integral, and the inner marginal \( p(y \mid \xi) \) is itself an intractable integral that changes with every simulated outcome. The baseline tool is the nested Monte Carlo (NMC) estimator:

# illustrative
# Nested Monte Carlo estimate of EIG for one candidate design xi
# theta ~ MMM posterior draws; simulate() pushes perturbed spend
# through adstock -> saturation -> regression + noise
eig = 0.0
for n in range(N):                      # outer: joint draws
    theta_n = posterior.draw()
    y_n = simulate(theta_n, xi)
    log_lik = log_p(y_n, theta_n, xi)
    # inner: fresh prior draws to estimate the marginal p(y | xi)
    marginal = mean(p(y_n, posterior.draw(), xi) for m in range(M))
    eig += (log_lik - log(marginal)) / N

NMC works, but slowly. It is biased for any finite inner sample size \( M \), its cost is \( C = NM \), and its error shrinks at only \( \mathcal{O}(C^{-1/3}) \) — markedly worse than the \( \mathcal{O}(C^{-1/2}) \) of ordinary Monte Carlo (Rainforth et al., 2018).

Deep diveWhy NMC is biased, and the optimal inner/outer split

For a fixed simulated outcome \( y_n \), the inner average \( \frac{1}{M}\sum_m p(y_n \mid \theta_m, \xi) \) is an unbiased estimate of the marginal \( p(y_n \mid \xi) \). But the estimator uses its logarithm, and by Jensen's inequality

$$ \mathbb{E}\big[ \log \hat{p} \big] \;\le\; \log \mathbb{E}\big[ \hat{p} \big] \;=\; \log p(y_n \mid \xi), $$

so the inner term is negatively biased in log space and the overall EIG estimate is biased upward — the estimator flatters every design, and flatters designs with noisier inner estimates the most. The bias is \( \mathcal{O}(1/M) \). Balancing the outer variance \( \mathcal{O}(1/N) \) against the squared bias \( \mathcal{O}(1/M^2) \) at fixed budget \( C = NM \) gives the optimal allocation \( M \propto \sqrt{N} \), and with it the overall \( \mathcal{O}(C^{-1/3}) \) convergence rate. Two escape routes exist: randomized multi-level Monte Carlo debiasing, which restores \( \mathcal{O}(C^{-1/2}) \) at higher per-sample cost (Goda et al., 2022), and the variational estimators discussed next.

The modern alternative is to make the inner problem learned rather than re-estimated: fit an amortized approximation to the posterior or the marginal once, and reuse it across simulated outcomes. The Barber–Agakov posterior bound, the variational marginal bound, and the contrastive bounds (ACE, PCE) all follow this pattern; they turn the EIG into a differentiable objective with \( \mathcal{O}(T^{-1/2}) \) convergence (Foster et al., 2019). For a measurement team the practical consequence is simple: scoring hundreds of candidate geo designs is feasible on the same hardware that fit the MMM.

Optimizing over Designs

Scoring designs is half the problem; searching the design space is the other half.

⚠️ Don't grid-search geo assignments

With \( G \) geos the assignment space has \( 2^G \) elements — at fifty markets, a quadrillion candidate holdouts. Enumerating a handful of hand-picked designs and choosing the best-scoring one silently searches a vanishing fraction of the space.

Because the variational EIG bounds are differentiable in \( \xi \), the search can run by stochastic-gradient ascent on the design itself. Continuous knobs — the spend multiplier \( \Delta_g \), the window endpoints — take gradients directly. The binary assignment \( a_g \) needs a relaxation: optimize a per-geo inclusion probability, or use a Gumbel-softmax relaxation and anneal it. Gradient-based BED of this kind has been demonstrated on design spaces of several hundred dimensions (Foster et al., 2020), comfortably covering realistic geo counts. In a standing measurement program the design problem repeats each cycle, and the same machinery can be amortized into a design policy that maps the current posterior to the next experiment (Foster et al., 2021).

Closing the Loop

Running \( \xi^* \) produces data, and the readout is a counterfactual problem: what would the treated geos have sold without the change? Structural time-series methods — the CausalImpact machinery of Brodersen et al. (2015) — estimate exactly that posterior counterfactual. Feeding the measured lift back into the MMM as a calibration observation contracts the targeted parameters, and the next cycle's design is chosen against the updated posterior. That loop — fit, design, test, calibrate, refit — is the subject of the continuous learning methodology page, and the marginal-EIG targeting above is what keeps each cycle pointed at the decision that matters.

Takeaways

  • A geo-holdout is a point in a design space: (which geos, which channels, how much, how long). Perturb two channels jointly if an interaction is the target.
  • A fitted MMM supplies both the prior and the outcome simulator — computing an experiment's expected information gain requires nothing you don't already have.
  • Use the targeted (marginal) EIG so the test resolves decision-relevant parameters instead of re-confirming known main effects.
  • Estimate with variational or contrastive bounds and optimize designs by stochastic gradients — nested Monte Carlo plus grid search does not scale to realistic geo counts.

References