Continuous Learning in Media Measurement (with Interaction Effects)

Matthew Reda MMM

A measurement study answers one question about one quarter's plan, and its answer starts aging the day it lands. This post — the capstone of the series — argues for the alternative: a standing program in which a living response model and a stream of deliberately designed experiments keep each other honest, including on the channel interactions no testing calendar could ever enumerate.

Everything earlier in the series is a component of this machine. The quasi-experimental posts built instruments for reading a single test cleanly; the experimental-design posts built the targeting system that decides which test to run. What remains is to assemble them — and the assembly is forced by the hardest object in media measurement: the interaction effect.

Why One-Shot Measurement Falls Short

The default posture of media measurement is static: fit a media mix model to two or three years of weekly observational history, read off channel effects, hand the deck to planning. Two problems undermine it, and they are different problems.

The first is information starvation. A typical MMM sample is small relative to the flexibility of the model — a hundred-odd weekly observations against saturation shapes, carryover rates, and a coefficient per channel. In this regime the posterior for many parameters is dominated by the prior, and the data cannot correct a bias the prior brings in. The consequence surfaces exactly where it hurts: Jin et al. (2017) found that the optimal media mix implied by a fitted MMM carries large variance — in one of their scenarios the posterior over the optimum was multimodal, with three distinct candidate allocations. The decision layer does not average this uncertainty away; it amplifies it.

The second is decay. Even a well-identified estimate describes the mix you actually ran, in the market that existed while you ran it. Spend levels move, creative rotates, competitors react, channels change their auction dynamics. A one-off geo experiment patches the first problem locally — it injects clean causal information about one channel at one point in time — but it is just as much a snapshot as the model it calibrates. Twelve months later, both are historical documents.

Continuous learning replaces the one-shot fit with a loop: model the response surface as a Bayesian posterior, decide the allocation against that posterior, experiment where the posterior is weakest and the decision most sensitive, and update. Each pass through the loop is cheap; the program, not any single study, is the unit of measurement.

The Cell Explosion

Interactions — TV priming branded search, two prospecting channels cannibalizing the same audience — are the terms that make the loop necessary rather than merely nice. They are hard for a structural reason: they are precisely the parameters with the least observational identification. Channels are planned together; seasonal flighting moves budgets up and down in unison; the cross-terms the planner most wants are confounded by the planner's own past behavior.

Could you simply test your way to them? Count the cells. A full factorial over \( K \) channels — every on/off combination — needs \( 2^K \) cells: 256 experimental conditions for eight channels, over a thousand for ten. Add spend levels rather than on/off and it becomes \( L^K \). Restricting ambition to pairwise synergies still leaves \( \binom{K}{2} = K(K-1)/2 \) joint perturbations — 45 two-channel experiments for ten channels, each demanding its own geos, its own weeks, its own budget at risk. No testing calendar covers this, and a calendar that tried would collide with decay: by the time the last cell reads out, the first has expired.

The combinatorial problem is real, but it is a problem with enumeration, not with learning. The rest of this post is about the two moves that dissolve it: model interactions continuously instead of cell by cell, and let the prior carry the interactions the data have not yet earned.

Model the Surface, Not the Grid

The estimand is not a table of cell means. It is a response surface: expected outcome as a function of the whole spend allocation. Two model families make that surface learnable without a factorial.

Parametric interaction terms. The standard Bayesian MMM already handles the two nonlinearities that break linear regression — carryover (adstock) and saturation. Interactions enter the same mean function as cross-terms on the transformed spends:

$$ \mathbb{E}[y_t] \;=\; \mu \;+\; \sum_{m} \beta_m\, f_m(x_{m,t}) \;+\; \sum_{m \lt m'} \gamma_{mm'}\, f_m(x_{m,t})\, f_{m'}(x_{m',t}), $$

where \( f_m \) is channel \( m \)'s adstock-then-saturation transform and \( \gamma_{mm'} \) is a synergy (positive) or cannibalization (negative) coefficient. The market-response literature catalogues multiplicative and other non-additive functional forms of exactly this kind (Hanssens, Parsons, & Schultz, 2001) — interaction as a modeled quantity, not an experimental cell.

How one interaction term moves the budget

The heatmap is the joint response of two channels over their (saturating) spend. Channel 1 is the stronger performer, so with no interaction the money tilts its way. Drag the interaction coefficient γ: synergy (γ > 0) rewards feeding both channels and pulls the optimal split toward balance; cannibalization (γ < 0) punishes overlap and concentrates spend. The ★ is the true optimum along the budget line; the ○ is the allocation you’d pick if you ignored interactions and trusted the γ = 0 model.

+0.50
Interaction—
Optimal split (ch1/ch2)—
Naïve split (γ=0)—
Lost response—

Ignoring a real interaction is not a rounding error: the naïve split (○) can sit well off the true ridge (★), and the gap between them is response left on the table. A positive γ warps the surface toward the diagonal; a negative γ carves a valley along it, pushing the optimum into a corner.

Nonparametric surrogates. Alternatively, place a Gaussian process over the response surface. A GP with a non-additive kernel represents interactions of every order implicitly, through the covariance function; there is nothing to enumerate, only a smooth surface to infer from wherever data happen to sit (Rasmussen & Williams, 2006). Reduced-rank constructions such as Hilbert-space GPs keep this tractable at MMM data sizes (Solin & Särkkä, 2020).

💡 Cells become queries

This is the pivotal reframing. Both model families replace enumeration with a function: you learn a continuous response surface and query it at any allocation. A "cell" stops being a slot in a grid you must fill and becomes a point you may — or may not — choose to sample. The question shifts from "how do we cover the design?" to "which single point teaches us the most?" — which is a question with a computable answer.

Let Interactions Exist — Cheaply

A parametric surface with all pairwise terms still carries \( K(K-1)/2 \) extra coefficients into a likelihood that was information-starved before they arrived. The answer is not more data per coefficient but stronger structure over the coefficients: priors under which many candidate interactions can exist in the model while only the genuinely supported ones survive with any magnitude.

Definition: Global–local shrinkage prior

Each interaction coefficient gets its own scale, tied together by a shared global scale:

$$ \gamma_j \mid \lambda_j, \tau \;\sim\; \mathcal{N}\!\big(0,\; \tau^2 \lambda_j^2\big), \qquad \lambda_j \sim \mathrm{C}^{+}(0, 1). $$

The global scale \( \tau \) pulls every coefficient toward zero; the heavy-tailed half-Cauchy local scale \( \lambda_j \) lets an individual coefficient escape entirely when the data insist. The result — the horseshoe prior — shrinks noise aggressively while leaving strong signals nearly untouched (Carvalho, Polson, & Scott, 2010).

The regularized horseshoe refines this for exactly the regime MMM lives in. It lets the analyst encode an expected number of nonzero coefficients — "I believe perhaps three of these 45 interactions are real" — through the prior's effective-nonzeros parameter, and it keeps even the escaped coefficients regularized when the likelihood is weak, so a barely-identified interaction cannot balloon (Piironen & Vehtari, 2017).

Deep diveEncoding sparsity: the regularized horseshoe's effective nonzeros

The regularized horseshoe replaces the raw local scale with a slab-limited version,

$$ \gamma_j \mid \lambda_j, \tau, c \;\sim\; \mathcal{N}\!\big(0,\; \tau^2 \tilde{\lambda}_j^2\big), \qquad \tilde{\lambda}_j^2 \;=\; \frac{c^2 \lambda_j^2}{c^2 + \tau^2 \lambda_j^2}, $$

so that small coefficients feel the ordinary horseshoe while large ones are softly capped by the slab width \( c \) — the cap that matters when the likelihood alone cannot restrain them. Each coefficient's shrinkage can be summarized by a factor \( \kappa_j \in [0,1] \) (1 = shrunk to zero, 0 = untouched), and the effective number of nonzero coefficients is \( m_{\mathrm{eff}} = \sum_j (1 - \kappa_j) \). Working backward from a prior guess for \( m_{\mathrm{eff}} \) gives a principled scale for the global parameter,

$$ \tau_0 \;=\; \frac{m_{\mathrm{eff}}}{D - m_{\mathrm{eff}}} \cdot \frac{\sigma}{\sqrt{n}}, $$

with \( D \) candidate coefficients, \( n \) observations, and noise scale \( \sigma \). For an MMM this is a directly meaningful elicitation: the planner's belief about how many channel pairs genuinely interact becomes the prior's anchor, instead of an arbitrary regularization constant.

The same logic extends hierarchically. Where interaction structure repeats across units — geo × channel effects, say — partial pooling treats the unit-level effects as draws from a population distribution, pulling noisy cells toward the group mean roughly in proportion to their noise while leaving well-estimated cells alone. That data-adaptive shrinkage is also a multiple-comparisons correction: with dozens of candidate effects examined at once, the pooled estimates are the ones you can rank and select from without a false-discovery reckoning (Gelman, Hill, & Yajima, 2012).

The net effect of shrinkage is the statistical half of the answer to the cell explosion: the model can nominally contain far more interaction terms than you could ever test, because the prior spends degrees of freedom only where the data demand it. Unsupported interactions sit dormant near zero at essentially no cost.

ApproachHow it handles interactionsWhat it costs
Parametric cross-termsOne interpretable \( \gamma_{mm'} \) per channel pair on transformed spendsQuadratic growth in terms; the functional form must be roughly right
GP surrogateAll-order interactions implicit in a non-additive kernelLess interpretable; needs reduced-rank approximations at scale
Global–local shrinkageCarries many candidate terms; only supported ones keep magnitudePrior elicitation matters; shrinkage controls variance, not identification

⚠️ Shrinkage is not identification

A horseshoe prior makes many interaction terms cheap to carry; it does not make them cheap to learn. A coefficient the data pattern cannot distinguish from zero simply stays at its prior, however elegant that prior is. Learning \( \gamma_{mm'} \) requires variation in which the two channels move jointly and independently of everything else — which observational flighting almost never provides. That variation must be designed. Shrinkage buys you the right to wait; experiments pay the bill.

Each designed experiment pays down the uncertainty

Start from the shrinkage prior on one interaction γ — centered at zero, wide, straddling both signs. Every round of joint-perturbation experiment delivers a noisy read of γ; Bayes updates fold them in, and the credible band contracts toward the truth. Drag the number of rounds and watch the interval narrow until it clears zero — the point at which you have actually identified the synergy, not merely assumed it away.

4
Posterior mean γ—
Posterior sd—
95% interval—
Sign identified?—

The band is the 95% credible interval for the true interaction (dotted line, γ = +0.45). Zero rounds is the prior; a handful of targeted joint flights — the ones EIG-maximization proposes — is enough to pull the interval off zero. That is shrinkage handing off to identification.

The Closed Loop

Which experiments, in which order? This is where Bayesian experimental design takes over, and the loop closes:

  1. Fit. Maintain the posterior over the full surface — main effects, transforms, and the shrunken interaction terms.
  2. Decide. Allocate budget by minimizing expected loss under the current posterior. This step happens every cycle regardless; the model is always the best available basis for action.
  3. Design. Score candidate perturbations — geo-holdouts, spend scale-ups, joint two-channel flights — by their expected information gain about the parameters that are both wide and decision-relevant, and run the best one.
  4. Read out and calibrate. Analyze the experiment with the quasi-experimental machinery built earlier in the series, and feed the measured lift back into the model as a calibration observation.
  5. Repeat from the updated posterior.

The design step is the engine. For a candidate design \( \xi \), the expected information gain is the expected reduction in posterior entropy about the latent parameters \( \theta \):

$$ \mathrm{EIG}(\xi) \;=\; \mathbb{E}_{p(y \mid \xi)}\big[ \mathrm{H}[p(\theta)] - \mathrm{H}[p(\theta \mid y, \xi)] \big], \qquad \xi^{*} = \arg\max_{\xi} \mathrm{EIG}(\xi), $$

a criterion that goes back to Lindley (1956). The previous post worked through the full encoding — the holdout as a design vector, the fitted MMM as both prior and outcome simulator, the estimators that make the doubly intractable expectation computable — so here the point is what the criterion does for interactions. Aim \( \theta \) at the interaction coefficients that are still uncertain and that move the budget decision, and EIG-maximization proposes, on its own, the test that best resolves them: typically a design that varies two channels jointly, because that is what the interaction likelihood responds to. Instead of testing all cells, you test the handful that pay for themselves in information.

Three properties make this loop a practical program rather than a research exercise. It is adaptive: sequential Bayesian design chooses each experiment against the posterior conditioned on everything observed so far, so every cycle starts from the current state of knowledge. It is amortizable: when cycles are frequent, re-optimizing the EIG each time can be replaced by a policy network trained offline — Deep Adaptive Design — which maps the measurement history to the next design in a single, non-myopic forward pass (Foster et al., 2021). And it scales: gradient-based Bayesian experimental design has delivered roughly twice the EIG of standard baselines on a 400-dimensional design problem and outperformed human experts on a 100-dimensional real-world design task (Foster et al., 2020; Rainforth et al., 2024) — regimes where enumerating cells is hopeless but following gradients is routine.

One more property matters for a standing program: the loop has no natural end. Because effects drift, the value of past information decays, and a parameter pinned down last year quietly re-widens; the loop re-tests it when its uncertainty again crosses the threshold of decision relevance. The continuous learning methodology page covers the operational side of this cadence — how a program decides that a previously measured effect is due for re-measurement rather than treating any readout as permanent.

One Logic: Information and Value

One unification hides in the loop. Choosing an experiment to maximize expected information gain is the same expected-value-of-information calculation that Bayesian optimization performs when it chooses its next evaluation through an acquisition function (Garnett, 2023). The two differ only in objective. If the goal is to characterize the surface — learn the interactions — the objective is EIG about parameters. If the goal is to find the best allocation, swap in expected improvement, GP-UCB, or the knowledge gradient; the entropy-search family makes the connection literal, since it is EIG about the location of the optimum. If the goal is to earn while learning, the same posterior drives a bandit. All three run on the same Bayesian surrogate, and a mature program moves among them as the decision changes.

Two extensions from the optimization literature map directly onto media practice: batch design, for proposing several geo tests to run in parallel, and multi-fidelity modeling, which treats cheap-but-confounded observational reads and expensive-but-clean experiments as two fidelities of evidence about one surface — in effect, what experiment-calibrated MMM already does. The derivations behind the EIG estimators and acquisition functions used here are collected on the mathematical foundations page.

Where the Series Meets

This is the point of the series. The quasi-experimental methods of the earlier posts — time-based regression for geo splits, synthetic control, staggered difference-in-differences, structural time-series counterfactuals — are the instruments: how a single designed perturbation becomes a credible causal number. The experimental-design machinery — EIG, adaptive design, amortized policies, the bandit and optimization variants — is the targeting system: it points those instruments where the posterior is weakest and the decision most exposed. The response model is the memory that binds them, carrying every readout forward and handing the targeting system an honest map of what is still unknown.

Interactions are the acid test of the arrangement precisely because no single study can identify a surface. A one-off holdout measures a main effect at a point; only a program — surrogate, shrinkage, and an information objective spending experiments where they pay — can learn how channels behave together. That is what continuous learning in media measurement means: not measuring more often, but building the loop in which each measurement chooses the next one.

Takeaways

  • One-shot MMM is information-starved (the prior dominates) and its answers decay; a standing loop — fit, decide, design, run, calibrate, refit — is the alternative.
  • The interaction cell explosion (\( 2^K \) factorial cells, \( K(K-1)/2 \) pairwise tests) is a problem with enumeration, not learning: model a continuous response surface and query it instead.
  • Global–local shrinkage (horseshoe, regularized horseshoe) and partial pooling let the model carry many candidate interactions while only evidence-backed ones keep magnitude.
  • Shrinkage is not identification: learning an interaction requires a designed joint perturbation, and targeted EIG picks the few perturbations that pay for themselves.
  • EIG-based design, Bayesian-optimization acquisitions, and bandits are one value-of-information logic with different objectives — characterize, optimize, or earn.

References