Geo Experiments: TBR, Power, and the Stationarity You're Assuming

Matthew Reda July 3, 2026 Experiments

Geo experiments are the workhorse of incrementality measurement: partition markets into treatment and control, perturb spend, read off the lift. Two Google papers define the classical methodology — Vaver and Koehler (2011) for design and power, Kerman, Wang, and Vaver (2017) for the time-based regression estimator that works down to a single matched-market pair. Both rest on an assumption that is easy to state and easy to violate: the pre-period relationship between treatment and control must hold through the test.

This post walks the classical pipeline end to end: the geo-based regression (GBR) design and its closed-form power analysis; the time-based regression (TBR) estimator — pre-period regression of the treatment aggregate on the control aggregate, extrapolated into a counterfactual and cumulated into a lift curve; then the failure mode and the design choices that protect against it.

Why Geo Experiments

Advertisers need the incremental impact of spend — what would have happened without it — not correlational metrics like cost-per-click. Observational methods are cheap but lean on strong modeling assumptions; user-level randomized experiments are rigorous but blind to behavior downstream of the initial exposure. Geo experiments split the difference: randomized control-group rigor at an aggregated, privacy-friendly geographic level.

The units are geos — non-overlapping regions small enough to target advertising individually, large enough that ad delivery and response tracking are geographically accurate. Postal codes are too small; the 210 U.S. Designated Market Areas (DMAs) are a standard choice. Every geo experiment shares one temporal structure:

Pretest — no campaign differences; treatment and control run at the same baseline.
Intervention — the treatment geos' campaigns are modified, creating a nonzero ad-spend differential relative to control.
Cooldown — campaigns reset, but lagged effects (offline sales following research and consideration) may still accrue; the window extends a delay $ \nu $ past the spend change to capture them.

The design vector is the same one that appears in the Bayesian experimental-design companion post: which geos, how large a spend change, over what window. GBR and TBR are the classical answer — designs sized to a target confidence-interval half-width rather than maximal expected information gain, with a linear regression in place of a full media mix model.

GBR: The Founding Design

Vaver and Koehler (2011) analyze the experiment with a single cross-sectional regression, one observation per geo: each geo's series is collapsed to a pretest total and a test total, and replication across geos supplies the power.

Definition: Geo-based regression model

For geos $ i = 1, \dots, N $,

$$ y_{i,1} \;=\; \beta_0 + \beta_1\, y_{i,0} + \beta_2\, \delta_i + \epsilon_i, $$

where $ y_{i,1} $, $ y_{i,0} $ are geo $ i $'s aggregate response in the test and pretest periods and $ \delta_i $ is the spend differential — actual test-period spend minus the spend that would have occurred without the experiment. Fit is by weighted least squares with $ w_i = 1/y_{i,0} $, absorbing the heteroscedasticity of differently sized geos. $ \beta_0, \beta_1 $ soak up seasonal shifts common to the pretest→test transition; $ \beta_2 $ is the return on ad spend (ROAS).

Constructing the spend differential

With no pretest spend, $ \delta_i $ is simply the treatment geo's test-period spend. When baseline spend is scaled up or down, the counterfactual spend is estimated from an auxiliary regression fit on control geos only, whose campaigns were never touched:

$$ s_{i,1} \;=\; \gamma_0 + \gamma_1\, s_{i,0} + \mu_i, \qquad i \in C, $$

weighted by $ 1/s_{i,0} $. Trained where no treatment occurred, this captures the seasonal drift of spend absent any intervention; the differential is $ \delta_i = s_{i,1} - (\gamma_0 + \gamma_1 s_{i,0}) $ for treated geos, zero for controls. A planned aggregate budget change $ \Delta $ can instead be prorated across treatment geos by pretest volume.

Assignment is randomized — the guard against unmeasured geo differences masquerading as treatment effects — but constraining the randomization helps: rank geos by pretest volume, partition into groups of size $ M $, pick one per group for treatment. This size-stratified assignment cut the ROAS interval width by about 10% in the paper's data.

The flagship example makes the case for the whole enterprise. A multi-week search-ads experiment across the 210 DMAs converged to $ \beta_2 \approx 1/3 $ incremental click per dollar — a cost per incremental click of $3, against a naively reported cost-per-click of $2.40: a 20% understatement of true incremental cost. Read on offline sales, the same experiment showed cumulative incremental revenue still climbing after spend returned to baseline — the lagged response that motivates the cooldown, roughly absent for clicks.

Power Analysis Before You Spend

The design half of the methodology is a variance formula. With mean pretest response $ \bar y_0 $ and mean differential $ \bar\delta $, linear theory gives

$$ \mathrm{var}(\beta_2) \;=\; \frac{\sigma_\epsilon^2}{\big(1-\rho_{y\delta}^2\big)\left[\sum_{i=1}^N w_i(\delta_i-\bar\delta)^2\right]}, \qquad \rho_{y\delta}^2 \;=\; \frac{\left[\sum_i w_i(y_{i,0}-\bar y_0)(\delta_i - \bar\delta)\right]^2}{\sum_i w_i(y_{i,0}-\bar y_0)^2 \, \sum_i w_i (\delta_i-\bar\delta)^2}, $$

with $ \sigma_\epsilon^2 $ the residual variance and $ \rho_{y\delta} $ the weighted correlation between pretest volume and spend differential. The 95% confidence-interval half-width on ROAS is $ 2\sqrt{\mathrm{var}(\beta_2)} $. Everything here except $ \sigma_\epsilon $ is a design choice, and $ \sigma_\epsilon $ comes from history — so the formula becomes a pre-experiment power calculator:

Carve historical geo data into pseudo pretest and test windows matching the proposed lengths.
Randomly assign geos (with the stratified procedure).
Hypothesize the differentials $ \delta_i $.
Estimate $ \sigma_\epsilon $ from the reduced model $ y_{i,1} = \hat\beta_0 + \hat\beta_1 y_{i,0} + \hat\epsilon $ — the regression with the $ \delta_i $ term dropped.
Plug into the variance formula; repeat over many randomizations and circularly shifted pseudo-windows, and average.

The result is a predicted interval for an experiment that has not yet spent a dollar. One caution: circular shifting reuses data, and once the hypothesized experiment outlasts the available history the prediction turns overly optimistic — the paper shows it tracking the realized interval closely up to exactly that point, then diverging.

Time-Based Regression: Power from Time, Not Geos

GBR's power scales with the number of geos, so it fails when only a handful exist — smaller countries, subregions, or the deliberately cheap matched market test of one test region against one control. TBR fills the gap by drawing power from pretest time points instead, degrading gracefully down to two geos; it underlies Google's open-source Matched Markets tooling. Where GBR collapses each geo to two totals, TBR collapses across geos at every time point: one treatment series $ y_t $, one control series $ x_t $.

Definition: TBR pretest model and counterfactual

During the pretest,

$$ y_t \;=\; \alpha + \beta x_t + \epsilon_t, \qquad \epsilon_t \sim \mathcal N(0, \sigma^2)\ \text{i.i.d.}, $$

with the control series assumed a sufficient, stable predictor of the treatment series across all three periods absent the intervention. For each test-period time point the potential outcome is the extrapolation

$$ y_t^* \;=\; \alpha + \beta x_t + \epsilon_t^*, $$

whose posterior predictive folds in both the estimation uncertainty in $ (\alpha, \beta, \sigma) $ and a fresh prediction error $ \epsilon_t^* $.

Fit uses a noninformative prior on $ (\alpha, \beta, \log\sigma) $: conditional on $ \sigma $, the posterior of $ (\alpha,\beta) $ is Normal with the classical OLS mean and covariance $ \sigma^2 (X'X)^{-1} $; integrating out $ \sigma $ gives a shifted, scaled $ t $-distribution with $ n-2 $ degrees of freedom, $ n $ the number of pretest points. Everything downstream stays closed-form.

From counterfactual to cumulative lift and iROAS

The pointwise causal effect is the gap between what happened and what the pretest relationship predicts:

$$ \phi_t \;=\; y_t - y_t^*, \qquad \Delta(t) \;=\; \sum_{t'=1}^{t} \phi_{t'}, $$

cumulated from the first day of the intervention. Since $ y_t $ is observed and $ y_t^* $ has a posterior, $ \phi_t $ and $ \Delta(t) $ inherit posteriors — again $ t $-distributions, no simulation required. Incremental ROAS is the ratio of two such analyses, one on the response metric and one on cost:

$$ \mathrm{iROAS}(t) \;=\; \frac{\Delta_{\text{resp}}(t)}{\Delta_{\text{cost}}(t)}, $$

its posterior obtained by drawing from the two $ t $-distributions and dividing, with posterior medians as point estimates. One clean special case: with zero pretest spend the counterfactual cost is zero with certainty, so $ \Delta_{\text{cost}}(t) $ is the observed cumulative spend and the iROAS posterior is exactly a scaled, shifted $ t $-distribution.

TBR is explicitly a simplified cousin of CausalImpact (Brodersen et al., 2015): the same train-on-pretest, extrapolate, difference logic, but a single static regression on one control aggregate in place of a state-space model with local level, trend, seasonality, and covariate selection. The trade is right exactly when there is one clean control series and nothing to select among.

Reading a lift, and pricing the design that measures it

The control aggregate (blue) predicts the treatment aggregate in the pretest; that fit is extrapolated into the shaded test window as a counterfactual (dashed, with a 95% predictive band) and the post-period gap is the lift. More treatment geos shrink the residual noise and tighten every interval; a bigger per-period lift is easier to detect. The readout turns the closed-form posterior scale into a minimum detectable effect and the power against the effect you dialed in.

True per-period lift 3.00

Number of treatment geos 20

Estimated cumulative lift—

MDE (80% power, per period)—

Power vs this lift—

Verdict—

Power against per-period lift for the current design; the dot marks your dialed-in effect and the dotted line the 80%-power MDE. All of it is conditional on stationarity — the pretest treatment–control relationship holding through the test.

Deep diveThe closed-form posterior scale of the cumulative effect

The appendix of Kerman, Wang, and Vaver (2017) derives the variance of the cumulative effect at the end of a test period of length $ T $, conditional on $ \sigma $:

$$ \mathrm{Var}\big(\Delta(T)\mid\sigma\big) \;=\; T^2\,\mathrm{Var}(\alpha) \;+\; \mathrm{Var}(\beta)\Big(\sum_t x_t\Big)^{\!2} \;+\; 2T\,\mathrm{Cov}(\alpha,\beta)\Big(\sum_t x_t\Big) \;+\; T\sigma^2. $$

The first three terms are pretest estimation uncertainty propagated through the extrapolation; the last is fresh prediction noise accumulating linearly in $ T $. Integrating over $ \sigma $ yields the $ t $-scale

$$ T s \left( v_\alpha + 2\bar x_T\, v_{\alpha\beta} + v_\beta\, \bar x_T^2 + \tfrac{1}{T} \right)^{1/2}, $$

with $ s $ the residual-SD estimate, $ \bar x_T $ the test-period mean of the control series, and $ v_\alpha, v_\beta, v_{\alpha\beta} $ entries of $ (X'X)^{-1} $. Dividing by cumulative cost $ \bar c\, T $ gives the iROAS posterior directly, and every sensitivity result in the next section falls out by inspection.

What Moves the Interval

TBR's design procedure is the time-series analogue of GBR's: build pseudo-geo-experiment datasets from history — each starting at a different date, running the full experiment length, recycling dates from the start of the series when the window runs off the end — analyze each with the standard TBR pipeline, and take the median half-width as the prediction. The closed-form scale then says how each lever moves it:

Design lever	Effect on iROAS CI half-width
Spend intensity $ \bar c $	The dominant lever: multiplying intensity by $ f $ divides the half-width by $ f $ — limited only by ad inventory and diminishing returns.
Pretest length $ n $	Shrinks the width at rate $ 1/\sqrt n $, but bounded below: as $ n \to \infty $ it approaches $ \sigma_0 / (\bar c \sqrt T) $. More history cannot substitute indefinitely for spend or test length.
Test length $ T $	Improves precision with diminishing returns at fixed intensity.
Cooldown length	At fixed total spend, longer cooldown lowers average intensity, so the width grows — roughly as $ \sqrt T $ for large $ T $. Just long enough for $ \Delta(t) $ to flatten, no longer.
Control-group volume	No effect: rescaling $ x_t $ leaves the posterior scale unchanged once refit.
Treatment-group volume	Rescaling $ y_t $ by $ \kappa_y $ rescales the residual SD and the width by the same $ \kappa_y $ — fewer treatment geos directly widens the interval (and proportionally lowers cost).
Geo count	Never enters the estimator directly, but very few geos means less protection from randomization, and $ \sigma $ tends to suffer; high treatment–control correlation offsets this.

The paper validates the loop on the 210-DMA revenue experiment. The pseudo-experiment procedure priced the target iROAS precision at about $22,000 of incremental spend; the budget was $20,000, so the target was relaxed to $ 0.5 \times 22{,}000 / 20{,}000 = 0.55 $ by pure rescaling — no re-simulation, since width scales as $ 1/f $ in spend. The experiment spent $18,273 and achieved precision 0.62; in cost-per-unit-half-width terms, prediction and realization differed by about 3%.

The Stationarity You're Assuming

Everything above — counterfactual, closed-form posteriors, power analysis — is conditional on one substantive assumption.

Definition: TBR stationarity (stability) condition

TBR is unbiased provided $ y_t = \alpha + \beta x_t + \epsilon_t $ holds with the same $ (\alpha, \beta) $ and stationary error structure in the pretest, where it is fit, and the test period, where it is extrapolated — in the counterfactual absence of the intervention. Whatever process translates the control aggregate's movements into the treatment aggregate's must not itself shift: no differential trend, level shift, or seasonal pattern in the treatment–control relationship that is not caused by the ad change being measured.

When the assumption holds, TBR is remarkably well behaved. The paper simulates weekly geo sales as a mix of a common seasonal component and geo-specific noise, sweeping cross-geo correlation $ \rho \in \{0, 0.5, 0.8\} $ and coefficient of variation $ c \in \{0.15, 0.25, 0.5\} $, 2,000 datasets per combination. Across all nine scenarios and pretest lengths from 4 to 40 points, empirical coverage of the 90% and 50% posterior intervals matched nominal, and the posterior-median iROAS was practically unbiased — squared bias averaged 0.04% of mean squared error. Essentially all of TBR's error is variance, not bias, when the pretest relationship is genuinely stable.

Then the paper breaks it on purpose. Injecting sustained exponential baseline growth of about half a percent per week — a proxy for a strong trend, such as a holiday season, shifting the relationship mid-experiment — turns the picture: point estimates become biased, approaching the true iROAS of 2.0 only slowly as treatment–control correlation rises, and 90% coverage falls well below nominal at low correlation. The static regression is extrapolating a relationship that no longer exists.

The proposed fix is TBR-OR: orthogonal (Deming) regression, an errors-in-variables model that treats the control series itself as noisy, fixing the otherwise-unidentifiable variance ratio at the empirical pretest ratio $ \mathrm{Var}(y)/\mathrm{Var}(x) $. Under the same sustained-growth simulation its point estimates are noticeably less biased.

⚠️ TBR-OR is not a free upgrade

The treatment–control correlation sits in the denominator of the orthogonal-regression slope. When it is low (roughly $ \rho \lt 0.5 $), especially with little seasonality in the data, the slope becomes unstable and predictions unreliable — the paper's verdict is that TBR-OR "should not be used at all" in that regime, and its bootstrap intervals under-cover more severely than TBR's. Plain TBR remains the default; reserve TBR-OR for experiments known to span strong, sustained trend or seasonal change — precisely where plain stationarity is most stressed.

Protecting the Assumption by Design

Stationarity cannot be verified during the test — the counterfactual is unobservable by construction. What the methodology offers instead is design choices and diagnostics that make violations less likely and more visible:

Choose a control aggregate that tracks treatment tightly. High pretest correlation shrinks the residual variance (precision) and is the regime where even misspecified estimators stay closest to truth (validity). Size-stratified or paired assignment is the cheap way to get it.
Schedule around known non-stationarity. A window straddling a holiday season or major promotion is where the relationship is most likely to drift. If the calendar cannot move, plan the TBR-OR analysis in advance — or step up to a state-space counterfactual (CausalImpact), built to absorb exactly this drift.
Keep the cooldown honest. If $ \Delta(t) $ has not visibly flattened by the end of a reasonably chosen cooldown, something is wrong — a long-lived lagged effect, an interfering event, or a shifting relationship the static regression cannot represent. The flattening curve is TBR's closest thing to an internal validity check.
Run the power analysis first. An underpowered test invites over-reading noise; the pseudo-experiment procedure prices the test before any incremental dollar is committed, and the 3% prediction accuracy above is evidence the pricing is real.

Where does the readout go? A geo experiment's iROAS is the ground truth that calibrates a media mix model, contracting the channel parameters observational data cannot pin down. And because relationships drift, one experiment is never the end: a standing design–test–calibrate–retest program is the subject of the continuous learning page. The classical machinery here sizes each test; the Bayesian companion post shows how a fitted model chooses which test to run next.

💡 GBR or TBR?

Mostly a function of geo count. Tens of geos, single ROAS number wanted: GBR. A handful of geos — or a matched-market pair — and a full time-series readout wanted: TBR. Many donor geos and doubt that one control aggregate will track treatment: a synthetic-control counterfactual is the natural third branch. All three share the same design space; they differ only in how the counterfactual is built.

Takeaways

GBR estimates ROAS from one cross-sectional regression over geos; its variance formula plus pseudo-experiment simulation prices the test before any spend is committed.
TBR regresses the treatment aggregate on the control aggregate in the pretest, extrapolates the counterfactual, and cumulates the gap — closed-form $ t $-distribution posteriors for lift and iROAS, workable down to two geos.
Spend intensity is the dominant precision lever (half-width scales as $ 1/f $); pretest length helps only up to $ \sigma_0/(\bar c \sqrt T) $; extending the cooldown at fixed budget makes precision worse.
Validity rests on stationarity of the pretest treatment–control relationship. Under a sustained trend, TBR biases and under-covers; TBR-OR helps there but fails outright at low correlation.
Protect the assumption by design — tightly tracking controls, scheduling around known drift, a cooldown just long enough for the lift curve to flatten — then feed the readout back into the model that plans the next test.

🧰 This method ships in the framework

Time-based regression is a first-class named method in the experiment toolkit (tbr in the planning.methods registry), with a two-tier engine: a fast conjugate Gaussian counterfactual inside the A/A·A/B power-simulation loop, and a full Bayesian structural time-series (CausalImpact-style local level + control regression) for the headline read-out with pointwise and cumulative credible bands. Selectable in the Design Studio's method picker next to synthetic control, GBR, and matched-market DiD.

References

Vaver, J., & Koehler, J. (2011). Measuring Ad Effectiveness Using Geo Experiments. Google Research.
Kerman, J., Wang, P., & Vaver, J. (2017). Estimating Ad Effectiveness Using Geo Experiments in a Time-Based Regression Framework. Google Research.
Brodersen, K. H., Gallusser, F., Koehler, J., Remy, N., & Scott, S. L. (2015). Inferring Causal Impact Using Bayesian Structural Time-Series Models. Annals of Applied Statistics, 9(1).
Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). CRC Press.

Design lever	Effect on iROAS CI half-width
Spend intensity \( \bar c \)	The dominant lever: multiplying intensity by \( f \) divides the half-width by \( f \) — limited only by ad inventory and diminishing returns.
Pretest length \( n \)	Shrinks the width at rate \( 1/\sqrt n \), but bounded below: as \( n \to \infty \) it approaches \( \sigma_0 / (\bar c \sqrt T) \). More history cannot substitute indefinitely for spend or test length.
Test length \( T \)	Improves precision with diminishing returns at fixed intensity.
Cooldown length	At fixed total spend, longer cooldown lowers average intensity, so the width grows — roughly as \( \sqrt T \) for large \( T \). Just long enough for \( \Delta(t) \) to flatten, no longer.
Control-group volume	No effect: rescaling \( x_t \) leaves the posterior scale unchanged once refit.
Treatment-group volume	Rescaling \( y_t \) by \( \kappa_y \) rescales the residual SD and the width by the same \( \kappa_y \) — fewer treatment geos directly widens the interval (and proportionally lowers cost).
Geo count	Never enters the estimator directly, but very few geos means less protection from randomization, and \( \sigma \) tends to suffer; high treatment–control correlation offsets this.