Continuous Learning in Media Measurement (with Interaction Effects)
A measurement study answers one question about one quarter's plan, and its answer starts aging the day it lands. This post is the capstone of the series, and it argues for the alternative: a standing program in which a living response model and a stream of deliberately designed experiments keep each other honest, including on the channel interactions no testing calendar could ever enumerate.
Everything earlier in the series is a component of this machine. The quasi-experimental posts built instruments for reading a single test cleanly, and the experimental-design posts built the targeting system that decides which test to run. What remains is the assembly, and the shape of the assembly is forced by the hardest object in media measurement: the interaction effect.
Why One-Shot Measurement Falls Short
The default posture of media measurement is static: fit a media mix model to two or three years of weekly observational history, read off channel effects, hand the deck to planning. Two problems undermine it, and they are different problems.
The first is information starvation. A typical MMM sample is small relative to the flexibility of the model: a hundred-odd weekly observations against saturation shapes, carryover rates, and a coefficient per channel. In this regime the posterior for many parameters is dominated by the prior, and the data cannot correct a bias the prior brings in. The consequence surfaces exactly where it hurts. Jin et al. (2017) found that the optimal media mix implied by a fitted MMM carries large variance, and in one of their scenarios the posterior over the optimum was multimodal, with three distinct candidate allocations. The decision layer does not average that uncertainty away. It amplifies it.
The second is decay. Even a well-identified estimate describes the mix you actually ran, in the market that existed while you ran it. Spend levels move, creative rotates, competitors react, channels change their auction dynamics. A one-off geo experiment patches the first problem locally (it injects clean causal information about one channel at one point in time), but it is just as much a snapshot as the model it calibrates. Twelve months later, both are historical documents.
Continuous learning replaces the one-shot fit with a loop: model the response surface as a Bayesian posterior, decide the allocation against that posterior, experiment where the posterior is weakest and the decision most sensitive, and update. Each pass through the loop is cheap. The program, not any single study, is the unit of measurement.
The Cell Explosion
TV may prime branded search. Two prospecting channels may be cannibalizing the same audience. Interactions like these are the terms that make the loop necessary rather than merely nice, and they are hard for a structural reason: they are precisely the parameters with the least observational identification. Channels are planned together. Seasonal flighting moves budgets up and down in unison, and the cross-terms the planner most wants are confounded by the planner's own past behavior.
Testing your way to them is the obvious move. Count the cells first. A full factorial over \( K \) channels, every on/off combination, needs \( 2^K \) cells: 256 experimental conditions for eight channels, over a thousand for ten. Add spend levels rather than on/off and it becomes \( L^K \). Restricting ambition to pairwise synergies still leaves \( \binom{K}{2} = K(K-1)/2 \) joint perturbations. That is 45 two-channel experiments for ten channels, each demanding its own geos, its own weeks, its own budget at risk. No testing calendar covers this, and a calendar that tried would collide with decay: by the time the last cell reads out, the first has expired.
The combinatorial problem is real, but it is a problem with enumeration, not with learning. The rest of this post is about the two moves that dissolve it: model interactions continuously instead of cell by cell, and let the prior carry the interactions the data have not yet earned.
Model the Surface, Not the Grid
The estimand is a response surface rather than a table of cell means: expected outcome as a function of the whole spend allocation. Two model families make that surface learnable without a factorial.
Parametric interaction terms. The standard Bayesian MMM already handles the two nonlinearities that break linear regression, carryover (adstock) and saturation. Interactions enter the same mean function as cross-terms on the transformed spends:
$$ \mathbb{E}[y_t] \;=\; \mu \;+\; \sum_{m} \beta_m\, f_m(x_{m,t}) \;+\; \sum_{m \lt m'} \gamma_{mm'}\, f_m(x_{m,t})\, f_{m'}(x_{m',t}), $$where \( f_m \) is channel \( m \)'s adstock-then-saturation transform and \( \gamma_{mm'} \) is a synergy (positive) or cannibalization (negative) coefficient. The market-response literature catalogues multiplicative and other non-additive functional forms of exactly this kind (Hanssens, Parsons, & Schultz, 2001). Interaction there is a modeled quantity, not an experimental cell.
How one interaction term moves the budget
The heatmap is the joint response of two channels over their (saturating) spend. Channel 1 is the stronger performer, so with no interaction the money tilts its way. Drag the interaction coefficient γ. Synergy (γ > 0) rewards feeding both channels and pulls the optimal split toward balance. Cannibalization (γ < 0) punishes overlap and concentrates spend. The ★ is the true optimum along the budget line, and the ○ is the allocation you’d pick if you ignored interactions and trusted the γ = 0 model.
Ignoring a real interaction is not a rounding error: the naïve split (○) can sit well off the true ridge (★), and the gap between them is response left on the table. A positive γ warps the surface toward the diagonal. A negative γ carves a valley along it and drives the optimum into a corner.
Nonparametric surrogates. Alternatively, place a Gaussian process over the response surface. A GP with a non-additive kernel represents interactions of every order implicitly, through the covariance function. There is nothing to enumerate, only a smooth surface to infer from wherever data happen to sit (Rasmussen & Williams, 2006). Reduced-rank constructions such as Hilbert-space GPs keep this tractable at MMM data sizes (Solin & Särkkä, 2020).
💡 Cells become queries
This is the reframing that matters. Both model families replace enumeration with a function: you learn a continuous response surface and query it at any allocation. A "cell" stops being a slot in a grid you must fill and becomes a point you may choose to sample, or may leave alone. The question shifts from "how do we cover the design?" to "which single point teaches us the most?", and that second question has a computable answer.
Let Interactions Exist, Cheaply
A parametric surface with all pairwise terms still carries \( K(K-1)/2 \) extra coefficients into a likelihood that was information-starved before they arrived. Observational history has no more data per coefficient to give. What you can add is structure over them: priors under which many candidate interactions live in the model while only the genuinely supported ones survive with any magnitude.
Definition: Global–local shrinkage prior
Each interaction coefficient gets its own scale, tied together by a shared global scale:
$$ \gamma_j \mid \lambda_j, \tau \;\sim\; \mathcal{N}\!\big(0,\; \tau^2 \lambda_j^2\big), \qquad \lambda_j \sim \mathrm{C}^{+}(0, 1). $$The global scale \( \tau \) pulls every coefficient toward zero. The heavy-tailed half-Cauchy local scale \( \lambda_j \) lets an individual coefficient escape entirely when the data insist. That combination is the horseshoe prior, which shrinks noise aggressively while leaving strong signals nearly untouched (Carvalho, Polson, & Scott, 2010).
The regularized horseshoe refines this for exactly the regime MMM lives in. Through the prior's effective-nonzeros parameter, the analyst encodes an expected count of live coefficients, a belief of the form "perhaps three of these 45 interactions are real." It also keeps the escaped coefficients regularized when the likelihood is weak, so a barely-identified interaction cannot balloon (Piironen & Vehtari, 2017).
Deep diveEncoding sparsity: the regularized horseshoe's effective nonzeros
The regularized horseshoe replaces the raw local scale with a slab-limited version,
$$ \gamma_j \mid \lambda_j, \tau, c \;\sim\; \mathcal{N}\!\big(0,\; \tau^2 \tilde{\lambda}_j^2\big), \qquad \tilde{\lambda}_j^2 \;=\; \frac{c^2 \lambda_j^2}{c^2 + \tau^2 \lambda_j^2}, $$so that small coefficients feel the ordinary horseshoe while large ones are softly capped by the slab width \( c \). That cap earns its keep when the likelihood alone cannot restrain them. Each coefficient's shrinkage can be summarized by a factor \( \kappa_j \in [0,1] \) (1 = shrunk to zero, 0 = untouched), and the effective number of nonzero coefficients is \( m_{\mathrm{eff}} = \sum_j (1 - \kappa_j) \). Working backward from a prior guess for \( m_{\mathrm{eff}} \) gives a principled scale for the global parameter,
$$ \tau_0 \;=\; \frac{m_{\mathrm{eff}}}{D - m_{\mathrm{eff}}} \cdot \frac{\sigma}{\sqrt{n}}, $$with \( D \) candidate coefficients, \( n \) observations, and noise scale \( \sigma \). For an MMM this is a directly meaningful elicitation: the planner's belief about how many channel pairs really interact becomes the prior's anchor, instead of an arbitrary regularization constant.
The same logic extends hierarchically. Where interaction structure repeats across units (geo × channel effects, say), partial pooling treats the unit-level effects as draws from a population distribution. It pulls noisy cells toward the group mean roughly in proportion to their noise and leaves well-estimated cells alone. That data-adaptive shrinkage doubles as a multiple-comparisons correction: with dozens of candidate effects examined at once, the pooled estimates are the ones you can rank and select from without a false-discovery reckoning (Gelman, Hill, & Yajima, 2012).
Shrinkage is the statistical half of the answer to the cell explosion. The model can nominally contain far more interaction terms than you could ever test, because the prior spends degrees of freedom only where the data demand it. Unsupported interactions sit dormant near zero at essentially no cost.
| Approach | How it handles interactions | What it costs |
|---|---|---|
| Parametric cross-terms | One interpretable \( \gamma_{mm'} \) per channel pair on transformed spends | Quadratic growth in terms. The functional form must be roughly right |
| GP surrogate | All-order interactions implicit in a non-additive kernel | Harder to interpret. Needs reduced-rank approximations at scale |
| Global–local shrinkage | Carries many candidate terms, and only supported ones keep magnitude | Prior elicitation matters. Shrinkage controls variance, not identification |
⚠️ Shrinkage is not identification
A horseshoe prior makes many interaction terms cheap to carry. It does not make them cheap to learn. A coefficient the data pattern cannot distinguish from zero simply stays at its prior, however elegant that prior is. Learning \( \gamma_{mm'} \) requires variation in which the two channels move jointly and independently of everything else, and observational flighting almost never supplies it. That variation must be designed. Shrinkage buys you the right to wait. Experiments pay the bill.
Each designed experiment pays down the uncertainty
Start from the shrinkage prior on one interaction γ, centered at zero, wide, straddling both signs. Every round of joint-perturbation experiment delivers a noisy read of γ. Bayes updates fold them in, and the credible band contracts toward the truth. Drag the number of rounds and watch the interval narrow until it clears zero. That is the moment you have identified the synergy rather than assumed it away.
The band is the 95% credible interval for the true interaction (dotted line, γ = +0.45). Zero rounds is the prior. A handful of targeted joint flights, the ones EIG-maximization proposes, is enough to pull the interval off zero. That is shrinkage handing off to identification.
The Closed Loop
Bayesian experimental design settles what is left (which experiments to run, and in what order). The loop closes here:
- Fit. Maintain the posterior over the full surface: main effects, transforms, and the shrunken interaction terms.
- Decide. Allocate budget by minimizing expected loss under the current posterior. This step happens every cycle regardless, because the model is always the best available basis for action.
- Design. Score candidate perturbations (geo-holdouts, spend scale-ups, joint two-channel flights) by their expected information gain about the parameters that are both wide and decision-relevant, and run the best one.
- Read out and calibrate. Analyze the experiment with the quasi-experimental machinery built earlier in the series, and feed the measured lift back into the model as a calibration observation.
- Repeat from the updated posterior.
The design step is the engine. For a candidate design \( \xi \), the expected information gain is the expected reduction in posterior entropy about the latent parameters \( \theta \):
$$ \mathrm{EIG}(\xi) \;=\; \mathbb{E}_{p(y \mid \xi)}\big[ \mathrm{H}[p(\theta)] - \mathrm{H}[p(\theta \mid y, \xi)] \big], \qquad \xi^{*} = \arg\max_{\xi} \mathrm{EIG}(\xi), $$a criterion that goes back to Lindley (1956). The previous post worked through the full encoding: the holdout as a design vector, the fitted MMM as both prior and outcome simulator, the estimators that make the doubly intractable expectation computable. Here the point is what the criterion does for interactions. Aim \( \theta \) at the interaction coefficients that are still uncertain and that move the budget decision, and EIG-maximization proposes, on its own, the test that best resolves them: typically a design that varies two channels jointly, because that is what the interaction likelihood responds to. Instead of testing all cells, you test the handful that pay for themselves in information.
Three properties make this loop a practical program rather than a research exercise. It is adaptive: sequential Bayesian design chooses each experiment against the posterior conditioned on everything observed so far, so every cycle starts from the current state of knowledge. It is amortizable: when cycles are frequent, re-optimizing the EIG each time can be replaced by a policy network trained offline, Deep Adaptive Design, which maps the measurement history to the next design in a single, non-myopic forward pass (Foster et al., 2021). And it scales: gradient-based Bayesian experimental design has delivered roughly twice the EIG of standard baselines on a 400-dimensional design problem and outperformed human experts on a 100-dimensional real-world design task (Foster et al., 2020; Rainforth et al., 2024). Those are regimes where enumerating cells is hopeless but following gradients is routine.
One more property matters for a standing program: the loop has no natural end. Because effects drift, the value of past information decays, and a parameter pinned down last year quietly re-widens. The loop re-tests it when its uncertainty again crosses the threshold of decision relevance. The continuous learning methodology page covers the operational side of that cadence, including how a program decides a previously measured effect is due for re-measurement rather than treating any readout as permanent.
One Logic: Information and Value
One unification hides in the loop. Choosing an experiment to maximize expected information gain is the same expected-value-of-information calculation that Bayesian optimization performs when it chooses its next evaluation through an acquisition function (Garnett, 2023). The two differ only in objective. If the goal is to characterize the surface (to learn the interactions), the objective is EIG about parameters. If the goal is to find the best allocation, swap in expected improvement, GP-UCB, or the knowledge gradient. The entropy-search family makes the connection literal, since it is EIG about the location of the optimum. If the goal is to earn while learning, the same posterior drives a bandit. All three run on the same Bayesian surrogate, and a mature program moves among them as the decision changes.
Two extensions from the optimization literature map directly onto media practice: batch design, for proposing several geo tests to run in parallel, and multi-fidelity modeling, which treats cheap-but-confounded observational reads and expensive-but-clean experiments as two fidelities of evidence about one surface. That is, in effect, what experiment-calibrated MMM already does. The derivations behind the EIG estimators and acquisition functions used here are collected on the mathematical foundations page.
Where the Series Meets
This is the point of the series. The quasi-experimental methods of the earlier posts (time-based regression for geo splits, synthetic control, staggered difference-in-differences, structural time-series counterfactuals) are the instruments: how a single designed perturbation becomes a credible causal number. The experimental-design machinery of EIG, adaptive design, amortized policies, and the bandit and optimization variants is the targeting system. It points those instruments where the posterior is weakest and the decision most exposed. The response model is the memory that binds the two. It carries every readout forward and hands the targeting system an honest map of what is still unknown.
Interactions are the acid test of the arrangement precisely because no single study can identify a surface. A one-off holdout measures a main effect at a point. Only a program can learn how channels behave together: a surrogate, shrinkage, and an information objective that spends experiments where they pay. Continuous learning in media measurement means building the loop in which each measurement chooses the next one. Measuring more often, on its own, only buys you more snapshots.
Takeaways
- One-shot MMM is information-starved (the prior dominates) and its answers decay. The alternative is a standing loop: fit, decide, design, run, calibrate, refit.
- The interaction cell explosion (\( 2^K \) factorial cells, \( K(K-1)/2 \) pairwise tests) is a problem with enumeration, not learning: model a continuous response surface and query it instead.
- Global–local shrinkage (horseshoe, regularized horseshoe) and partial pooling let the model carry many candidate interactions while only evidence-backed ones keep magnitude.
- Shrinkage is not identification: learning an interaction requires a designed joint perturbation, and targeted EIG picks the few perturbations that pay for themselves.
- EIG-based design, Bayesian-optimization acquisitions, and bandits are one value-of-information logic with different objectives: characterize, optimize, or earn.
References
- Jin, Y., Wang, Y., Sun, Y., Chan, D., & Koehler, J. (2017). Bayesian Methods for Media Mix Modeling with Carryover and Shape Effects. Google Research.
- Chan, D., & Perry, M. (2017). Challenges and Opportunities in Media Mix Modeling. Google Research.
- Hanssens, D. M., Parsons, L. J., & Schultz, R. L. (2001). Market Response Models: Econometric and Time Series Analysis. Kluwer Academic Publishers.
- Carvalho, C. M., Polson, N. G., & Scott, J. G. (2010). The Horseshoe Estimator for Sparse Signals. Biometrika, 97(2).
- Piironen, J., & Vehtari, A. (2017). Sparsity Information and Regularization in the Horseshoe and Other Shrinkage Priors. Electronic Journal of Statistics, 11(2).
- Gelman, A., Hill, J., & Yajima, M. (2012). Why We (Usually) Don't Have to Worry About Multiple Comparisons. Journal of Research on Educational Effectiveness, 5(2).
- Lindley, D. V. (1956). On a Measure of the Information Provided by an Experiment. Annals of Mathematical Statistics, 27(4).
- Rainforth, T., Foster, A., Ivanova, D. R., & Bickford Smith, F. (2024). Modern Bayesian Experimental Design. Statistical Science, 39(1).
- Foster, A., Jankowiak, M., O'Meara, M., Teh, Y. W., & Rainforth, T. (2020). A Unified Stochastic Gradient Approach to Designing Bayesian-Optimal Experiments. AISTATS.
- Foster, A., Ivanova, D. R., Malik, I., & Rainforth, T. (2021). Deep Adaptive Design: Amortizing Sequential Bayesian Experimental Design. ICML.
- Rasmussen, C. E., & Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press.
- Solin, A., & Särkkä, S. (2020). Hilbert Space Methods for Reduced-Rank Gaussian Process Regression. Statistics and Computing, 30.
- Garnett, R. (2023). Bayesian Optimization. Cambridge University Press.