Adstock Dynamics and the Timing of Sequential Experiments

Matthew Reda Experiments

A media experiment ends when the spend change ends — but its effect does not. Carryover keeps moving sales for weeks after a flight stops, and that quietly breaks two things analysts rely on: the measurement window of a single test, and the spacing between consecutive tests. This post turns the retention rate you already estimate into concrete rules for a testing calendar.

Carryover is usually discussed as a modeling problem: pick an adstock form, put a prior on the retention rate, let the sampler sort it out. But the same dynamics that complicate the model contaminate the experiments meant to validate it. A holdout read too early understates the channel it was supposed to measure; two tests run back to back superpose, and each readout silently includes a piece of the other's answer. Neither failure announces itself. The fix is not better experiment analytics — it is timing, and the timing is computable from a single parameter.

The Effect Outlives the Exposure

The adstock transform exists because advertising works through memory, goodwill, and word-of-mouth — mechanisms that persist after the exposure. Instead of regressing sales on this week's spend, the model regresses on a weighted history of spend, so the regression sees the cumulative media pressure rather than the instantaneous flight. (The adstock mathematics page develops the functional forms in full; here we only need their timing implications.)

Definition: Geometric adstock and the retention rate

With spend \( x_t \) and a maximum carryover duration of \( L \) periods, the normalized geometric adstock of Jin et al. (2017) is

$$ \mathrm{adstock}(x_t;\,\alpha, L) \;=\; \frac{\sum_{l=0}^{L-1} \alpha^{\,l}\, x_{t-l}}{\sum_{l=0}^{L-1} \alpha^{\,l}}, \qquad 0 \lt \alpha \lt 1, $$

where \( \alpha \) is the retention rate: the fraction of this period's effect that survives into the next. Dropping the normalization, the same dynamics are a one-line recursion, \( a_t = x_t + \alpha\, a_{t-1} \) — each period the accumulated stock decays by \( \alpha \) and new spend tops it up. This is the marketing analog of the Koyck geometric distributed lag (Koyck, 1954; Hanssens, Parsons, & Schultz, 2001), whose carryover coefficient \( \lambda \) plays exactly the role of \( \alpha \).

Two consequences of the geometry are worth internalizing. First, the effect of a burst of spend is spread over roughly \( \lambda/(1-\lambda) \) periods on average — the Koyck mean lag. At a retention rate of 0.8 that is four weeks of average delay, on top of the flight itself. Second, geometric decay is the optimistic case: it assumes the effect peaks in the exposure week. Jin et al.'s delayed adstock, \( w(l) = \alpha^{(l-\theta)^2} \), lets the peak arrive \( \theta \) periods after exposure — the empirically common pattern for upper-funnel media like brand TV. A channel with a delayed peak has not even reached its maximum effect when a short test window closes.

Sizing the Measurement Window

Suppose an experiment changes a channel's spend over weeks \( [t_0, t_1] \). Because of carryover, sales keep responding to that change well past \( t_1 \). The counterfactual attribution in Jin et al. (2017) is explicit about this: ROAS is computed by comparing predicted sales under historical versus perturbed spend, summed over

$$ t_0 \;\le\; t \;\le\; t_1 + L - 1, $$

that is, over the change period and the post-change period out to the maximum carryover duration. The change window is framed by pre- and post-periods each of length \( L \): the pre-period so that adstock flowing in from before the test is accounted for, the post-period so that the tail of the test's own effect is captured.

⚠️ Truncation bias runs one way

Reading a test at its end date does not add noise — it adds bias, and always in the same direction. Every week of uncounted tail is incremental effect the readout attributes to zero, so a truncated window systematically understates lift and ROAS. A channel with slow carryover, measured with a short window, will look weaker than a fast-decaying channel of identical true effectiveness. If that readout then calibrates the MMM, the bias propagates into the model as an artificially low coefficient.

The practical rule: the readout window is the flight plus the effective carryover duration. For geometric adstock, the residual fraction of effect still live \( k \) weeks after a flight ends is \( \alpha^k \), so the number of weeks needed for the residual to fall below 5% is

$$ L_{\text{eff}} \;\approx\; \frac{\ln(0.05)}{\ln(\alpha)}. $$

For a channel with a delayed peak, start this clock at the peak — roughly \( \theta \) weeks after exposure — not at the flight's last day. Jin et al. use \( L = 13 \) weeks in both their simulations and their real application; for their parameter choices the weights beyond that horizon are negligible, which gives a sense of the scale involved: a quarter, not a fortnight.

Washouts Between Sequential Tests

The same arithmetic that sizes the readout window sizes the gap between tests. If the next experiment on a channel (or in a geography) should start from a clean baseline, the previous test's adstock must first decay to a negligible level. The washout length is driven entirely by the retention rate:

Retention rate \( \alpha \) Half-life (weeks) Weeks to \( \lt 5\% \) residual
0.30.63
0.51.05
0.71.99
0.83.114
0.96.629

Adstock washout, and the cost of testing too soon

A four-week flight (shaded bars) accumulates adstock \( a_t = x_t + \alpha\,a_{t-1} \); after it ends the stock decays as \( \alpha^k \) (curve, as % of peak). The green marker is the earliest clean start — when the residual drops below 5%. Slide your next test earlier than that and the leftover carryover (red) contaminates its baseline. Drag the retention rate and the start week.

0.70
6
Half-life—
Washout to <5%—
Residual at chosen start—
Next test—

The contaminating residual scales with the previous test's effect, not the next one's — so a small expected lift following a large test needs a tighter tolerance than 5%. Start before the green marker and the second readout silently absorbs a slice of the first test's answer.

The table assumes weekly data and geometric decay; the half-life is \( \ln(0.5)/\ln(\alpha) \) and the washout column is \( \ln(0.05)/\ln(\alpha) \), rounded up. The nonlinearity is the point. Moving the retention rate from 0.5 to 0.8 does not roughly double the required spacing — it triples it. A search channel with fast decay supports a test nearly every month; a brand channel at \( \alpha = 0.9 \) supports perhaps two clean sequential tests a year. Testing calendars that allocate the same cadence to every channel are implicitly assuming every channel has the same retention rate.

Weeks to a 5% residual, \( \ln(0.05)/\ln(\alpha) \), and the half-life \( \ln(0.5)/\ln(\alpha) \), across the retention rate. The washout curve steepens sharply as \( \alpha \to 1 \): the marked points are the table's rows, and the gap between them widens far faster than \( \alpha \) does. Moving from \( \alpha = 0.5 \) to \( 0.8 \) roughly triples the required spacing rather than doubling it — which is why one testing cadence cannot fit every channel.

Deep diveFrom retention rate to washout length

Write the unnormalized adstock recursion \( a_t = x_t + \alpha a_{t-1} \) and consider a test that perturbs spend during \( [t_0, t_1] \), returning to baseline afterward. The perturbation to the adstock state at the flight's end is some \( \Delta a_{t_1} \); with no further perturbed spend, it evolves as \( \Delta a_{t_1+k} = \alpha^k \Delta a_{t_1} \). Since the sales response is a function of the adstock state, the test's lingering influence \( k \) weeks after the flight scales with the state residual \( \alpha^k \) (exactly proportional in the linear-response regime; approximately so under mild saturation around the operating point). Solving \( \alpha^k \le \varepsilon \) gives \( k \ge \ln \varepsilon / \ln \alpha \): the washout for tolerance \( \varepsilon \). The normalized form used in fitting rescales the weights but not their ratio, so the same \( k \) applies. For delayed adstock the kernel \( \alpha^{(l-\theta)^2} \) rises until lag \( \theta \) and then falls faster than geometrically, so a conservative washout is \( \theta \) plus a few half-widths of the kernel — the delay shifts the clock more than the decay lengthens it. The tolerance is a judgment call: 5% is a common working choice, but if the next test's expected lift is small relative to the previous test's, tighten \( \varepsilon \) — the contaminating residual is scaled by the previous effect size, not the next one.

💡 Estimate the retention rate at the cadence you test at

The washout table is only as good as the \( \alpha \) that feeds it, and retention rates are notoriously sensitive to temporal aggregation: Clarke (1976) showed that carryover estimated from annual data implies durations 20–50 times longer than estimates from monthly data on the same process, because aggregation sums the within-period carryover into the apparent retention. Use the shortest data interval available — for most MMM stacks, weekly — and take the spacing rule from the model's posterior on \( \alpha \) rather than a category benchmark. If you want conservative spacing, size the washout from the upper end of that posterior, not the point estimate.

When Tests Read Each Other's Answers

Now consider the failure mode the washout exists to prevent. Experiment \( k \) ends; experiment \( k{+}1 \) begins in the same market a week later. The sales series during the second test superposes two effects: the new test's response and the decaying tail of the old one. A difference-in-differences or holdout readout of test \( k{+}1 \) cannot tell them apart — the design is confounded, and each estimate absorbs a share of the other's effect. The error's direction depends on the signs of the interventions: a spend increase following a holdout looks weaker than it is (the baseline is still depressed by the holdout's tail), while two same-signed tests inflate each other.

There are two defensible responses, and they trade information for simplicity:

The same logic applies when the tests are scored prospectively. In a sequential design loop, the value of a candidate next experiment is its incremental expected information gain — an EIG evaluated against the current posterior (Rainforth et al., 2024). If that posterior does not condition on the in-flight carryover from the previous test, the candidate design gets credit for "resolving" variation that the lingering adstock already explains, and the loop overestimates how much new information the next test will buy.

Carryover as a Latent State

Both remedies above treat carryover as an inconvenience. The cleaner framing treats it as what it literally is: a hidden quantity that evolves over time and drives the observations. That is a state-space model. The adstock accumulator becomes (part of) the latent state \( \mathbf{x}_k \): it decays by \( \alpha \) each period and is topped up by spend (the transition equation), and, after saturation, it contributes to observed sales (the observation equation):

$$ \mathbf{x}_k = \mathbf{A}\,\mathbf{x}_{k-1} + \mathbf{q}_{k-1}, \qquad \mathbf{y}_k = \mathbf{H}\,\mathbf{x}_k + \mathbf{r}_k, $$

with the retention rate sitting on the diagonal of the transition matrix \( \mathbf{A} \) (Särkkä, 2013). Under this encoding, a delayed outcome is nothing special — just an observation of a slowly decaying state. The Kalman filter runs forward through the data attributing each week's sales to the current state estimate; the smoother runs backward and reattributes them using everything observed later. Both passes are linear in the number of time steps — the machinery was built for exactly this "act now, observe later" structure.

This is also, not coincidentally, the machinery behind the standard experiment readout. The Bayesian structural time-series approach of Brodersen et al. (2015) — the CausalImpact construction — is a state-space decomposition into trend, seasonality, and regression components whose posterior predictive supplies the counterfactual over the post-intervention period. For a geo test whose effect trails off gradually rather than switching cleanly off, that is the right readout instrument: it does not force a hard end date onto an effect that does not have one, and its credible intervals widen honestly as the tail decays into the noise.

Scheduling with Delayed Information

Sequential experimentation formalizes the testing calendar as an adaptive design loop: after history \( h_{t-1} \) of designs and outcomes, choose the next design by maximizing the incremental EIG,

$$ \mathrm{EIG}(\xi_t \mid h_{t-1}) \;=\; \mathbb{E}\!\left[ \log \frac{p(y_t \mid \theta, \xi_t, h_{t-1})}{p(y_t \mid \xi_t, h_{t-1})} \right], $$

an ordinary EIG with the prior replaced by the current posterior (Foster et al., 2019; Rainforth et al., 2024). The standard loop assumes each outcome is fully observed before the next design is chosen. Carryover breaks that assumption: when it is time to pick \( \xi_t \), the informative sales from \( \xi_{t-1} \) are still accruing. A greedy loop that pretends otherwise mis-times its choices twice over — it reads the previous test too early (truncation bias flowing into the posterior) and then scores the next design against that corrupted posterior.

Two remedies parallel the single-test case. The first is delay-aware scheduling: condition the design step on the partially observed outcome — the filtered state from the section above — rather than waiting for, or worse fabricating, a final readout. The filter is exactly the object that says how much of test \( t-1 \)'s answer has arrived and how much uncertainty remains in the tail. The second is to go non-myopic: a policy in the Deep Adaptive Design family (Foster et al., 2021) is trained offline to maximize the total EIG over the whole horizon, and in doing so internalizes the delay structure — it learns to space and sequence tests so that late-arriving information is still information the plan counts on. This timing discipline is one of the constraints baked into the continuous learning methodology: a fit → design → test → calibrate loop only converges if each cycle's readout is clean when it is consumed.

A Calendar That Respects Carryover

For an analyst planning the next two quarters of tests, the pieces above reduce to a short procedure:

None of this requires new estimation machinery. It requires taking a parameter the model already produces and letting it set the tempo of the measurement program, rather than letting the fiscal calendar do it.

Takeaways

  • Carryover means a test's effect outlives its flight: the readout window must extend past the spend change by the effective carryover duration, or ROAS is biased low — always low.
  • The washout between sequential tests is computable from the retention rate: \( \ln(0.05)/\ln(\alpha) \) weeks to reach a 5% residual — about 5 weeks at \( \alpha=0.5 \), 14 at \( \alpha=0.8 \), 29 at \( \alpha=0.9 \).
  • Back-to-back tests superpose through lingering adstock; either wait out the washout or model the residual carryover explicitly as a known covariate — and score the next design against a posterior that conditions on in-flight effects.
  • The cleanest treatment makes carryover a latent state: Kalman filtering/smoothing attributes delayed sales to past exposures in linear time, and BSTS counterfactuals read out post-period effects honestly.
  • Different channels earn different testing cadences. A fast-decay channel supports monthly tests; a high-retention brand channel supports two clean tests a year.

References