Switchback Experiments

Matthew Reda Experiments

A geo experiment randomizes where; a user A/B test randomizes who. A switchback randomizes when — it flips the entire system between treatment and control over time and reads the effect off the difference. That sounds like a strange thing to do until you hit a setting where splitting users is quietly biased: a marketplace, an auction, any place where treated and control units compete for the same finite pool. The catch is that time-randomization trades one interference problem for another — carryover — and everything interesting about the design is about managing that trade.

Why Not Just Split Users

The workhorse online experiment splits users into treatment and control and compares them. Its validity rests on the stable-unit-treatment-value assumption (SUTVA): one user's outcome does not depend on another user's assignment. In a two- or three-sided marketplace, that assumption fails outright. A ride-hailing pricing change tested on half the riders still draws from the same pool of drivers; a new ranking rule shown to half the sessions still competes in the same auction for the same inventory. Treated and control users interfere through the shared resource, so the A/B contrast measures a distorted, congestion-mediated thing rather than the effect of deploying the policy to everyone.

This is not a hypothetical. The design emerged from exactly this pain at ride-sharing and delivery platforms — Lyft's engineering team described "time-split" testing to escape marketplace interference (Chamandy, 2016), and DoorDash formalized region-by-time "switchback" tests because a shared Dasher fleet makes simultaneous treatment and control orders statistically dependent (Kastelman and Ramesh, 2018). The idea itself is far older: agricultural statisticians ran "switchback" or change-over trials on dairy cows in the 1930s and 40s, alternating each animal between diets to cancel between-cow variation, and already writing a regression term for the lagged carryover effect (Cochran, Autrey, and Cannon, 1941). The modern marketplace rediscovered a very old design under a very new kind of interference.

What a Switchback Is

Formally, a switchback is a cluster-randomized experiment where the cluster is time. Take a single aggregate unit — a whole city, market, or platform — and a binary treatment \( W_t \in \{0, 1\} \). Divide the horizon \( t = 1, \dots, T \) into blocks of length \( l \), hold the treatment constant within each block, and re-randomize it at the block boundaries. Because the entire system is on one arm at any instant, there are never two co-existing treatment regimes to interfere with each other. You have converted a spatial interference problem into a temporal one.

Definition: The global average treatment effect

The estimand is the difference between running the system always treated versus always in control, averaged over the experiment window. With \( \tau_t \) the stable per-period effect at time \( t \),

$$ \tau_{\mathrm{GATE}} \;=\; \frac{1}{T}\sum_{t=1}^{T} \tau_t. $$

This is the "if we shipped it to everyone, forever" quantity a decision-maker actually wants (Hu and Wager, 2022). A related target, the filtered ATE, averages only over a chosen subset of periods — the ones that are not still settling after a switch.

Note what the estimand is not: it is not the effect on a single user, and it is defined conditional on the environment during the experiment. If the two weeks you ran the switchback were unusually hot, promotional, or seasonal, the GATE you measured is the GATE in that environment. External validity — "was this window representative?" — is a first-order concern, not a footnote, precisely because the design leans so hard on the time dimension.

Carryover Is the Whole Problem

Randomizing over time defeats spatial interference but creates a new adversary: the treatment at time \( t \) affects not only \( Y_t \) but future outcomes, because the system carries state. Right after a switch from control to treatment, the outcome still reflects the lingering control regime — surge prices that haven't reset, a matching queue that hasn't cleared, demand that hasn't responded. Naively differencing treated against control periods folds that contamination straight into the estimate. There are two ways to model it, and they lead to different designs.

The first assumes carryover has a finite, known order \( m \): an action at \( t \) has fully dissipated by \( t + m \), so \( Y_t \) depends only on the treatments in the window \( W_{t-m:t} \). Surge-pricing effects that "dissipate after one or two hours" fit this picture, and \( m \) can be set from domain knowledge or estimated from data (Bojinov, Simchi-Levi, and Zhao, 2023). The second, more realistic, stance is geometric mixing: the system is a non-stationary Markov process with mixing time \( t_{\mathrm{mix}} \), and the influence of an action decays like \( e^{-(t' - t)/t_{\mathrm{mix}}} \) but never reaches exactly zero (Hu and Wager, 2022). Under geometric mixing the remedy is a burn-in — discard the first \( b \) observations after every switch, while the system is still relaxing toward its new steady state, and analyze only the settled periods.

⚠️ Carryover contamination is the failure mode, and it is asymmetric with the horizon

The lag-\( p \) "always-on vs always-off" estimand is not even well-defined once the carryover length approaches the horizon \( T \) — there is no clean stretch of pure-treatment or pure-control state to observe. Everything below about choosing the switching interval is, at bottom, a negotiation with carryover: the longer the effect persists, the longer your blocks and burn-ins must be, and the less data you have left to estimate anything.

The Interval Tradeoff

How long should a block be? This is the single design knob, and it sits on a clean bias–variance tension. Short blocks give many independent randomizations — low variance — but leave little time for the system to settle, so carryover bias is large. Long blocks give clean, settled measurement — low bias — but few randomizations, so variance is large. Burn-in adds a second knob that spends data to buy bias reduction.

Hu and Wager (2022) make the tension quantitative, and the punchline is sobering. For the naive block difference-in-means estimator targeting the global effect under geometric mixing, the root-mean-square error cannot beat order \( T^{-1/3} \) — markedly slower than the \( T^{-1/2} \) you would get with no carryover at all. The rate-optimal choice there is a single long block scaling like \( T^{1/3} \). The good news is that a smarter analysis recovers most of the loss: with a burn-in of order \( t_{\mathrm{mix}} \log T \) and either a filtered estimand or a bias-corrected inverse-propensity estimator, the RMSE improves to order \( \sqrt{\log T / T} \) — within a \( \sqrt{\log T} \) factor of the carryover-free ideal. The practical shape of that solution is counterintuitive: most periods should be burn-in. You switch relatively often, throw away the majority of each block as it settles, and keep only the tail.

Their ride-sharing simulation shows what this buys in coverage, the number that actually matters for a decision. Adding burn-in to the difference-in-means estimator cut the bias from about 2.64 to 0.09 and lifted the confidence-interval coverage from 78% toward a healthy 92% — turning an over-confident, mis-centered interval into a usable one.

The block-length dial

A single system over \( T=96 \) periods, re-randomized on/off at each block boundary (seeded). Treatment carries over for \( m \) periods after every switch, so the shaded periods right after a flip are contaminated and bias the naive difference. Short blocks switch often (low variance) but spend most of their length settling (high bias); long blocks measure cleanly but give few randomizations (high variance). The lower panel traces that bias–variance trade — the total-error minimum sits near block length \( m+1 \).

4
3
Effective switches—
Carryover contamination—
Bias (rel.)—
Variance (rel.)—

Contamination falls as blocks grow — but variance rises with them, because there are fewer independent switches. The sweet spot aligns the block length to the carryover order, \( \ell \approx m+1 \), exactly the minimax rule from the design table below.

Inference Without a Model

A quietly radical feature of the modern switchback literature is that it is design-based. Potential outcomes are treated as fixed but unknown; the only randomness is the assignment path the coin flips generated. This lineage — Fisher, Neyman, Kempthorne, Rubin — buys you inference that does not depend on getting an outcome model right.

💡 The estimator is a weighted average, and it is unbiased by construction

The workhorse is the Horvitz–Thompson estimator: weight each period's outcome by the inverse probability of the treatment path it actually received. Under a properly randomized switchback this is unbiased for the temporal effect, with no likelihood, no stationarity assumption, and no functional form for how spend maps to sales (Bojinov and Shephard, 2019; Bojinov, Simchi-Levi, and Zhao, 2023). You can go further and compute an exact randomization p-value for the sharp null of no temporal effect by re-running the randomization on the observed data — a test that is valid in finite samples, whatever the outcome process looks like.

Two assumptions are load-bearing. Probabilistic assignment: every period must have a strictly-between-zero-and-one chance of each arm, or the inverse-propensity weights blow up. And no anticipation: the assignment at time \( t \) must not depend on future potential outcomes — formally, the future outcomes do not Granger-cause the treatment (Bojinov and Shephard, 2019). A randomized switchback satisfies this by construction, which is exactly why an observational "we turned the feature on when it seemed slow" on/off history does not qualify: there, the toggling responds to the very outcomes you are trying to measure. Bojinov and Shephard's original application was not marketing at all but a hedge fund comparing human versus algorithmic trade execution across equity-index futures — a reminder that "randomize a single system over time" is a general causal tool. They also note a variance-reduction trick with a familiar flavor: subtracting a one-step-ahead proxy prediction of the outcome before analyzing is the design-based cousin of CUPED.

Designing It Well

If block length trades bias against variance, is there an optimal schedule? Bojinov, Simchi-Levi, and Zhao (2023) pose it as a minimax problem — choose the randomization points and probabilities to minimize the worst-case variance over an adversarial choice of potential outcomes — and solve it. The answer is reassuringly simple and gives a rule you can carry in your head.

Design choiceConsequence (finite carryover order \( m \))
Switch every periodWastes power: about 28–32% higher variance than optimal
Block length \( \approx m + 1 \)Within 1–2% of the optimal minimax design
One giant blockToo few randomizations; variance dominated
Over-specify \( p \gt m \)Stays unbiased at small variance cost
Under-specify \( p \lt m \)Estimates a different, weighted estimand

The headline is the middle row: align the block length to the carryover order, roughly \( m + 1 \), and you capture almost all of the achievable efficiency. The optimal schedule is nearly regularly spaced at that interval, with slightly longer boundary blocks and switching probabilities kept to multiples of one-half to avoid unstable weights. And when you are unsure of \( m \), the asymmetry is your friend: over-specifying the carryover order costs a little variance but preserves unbiasedness, while under-specifying quietly changes the quantity you are estimating. When in doubt, assume the effect lingers longer than you think.

The Autocorrelation Coupling

There is a second time-series subtlety that interacts with carryover in a way that surprises people: the serial correlation of the outcome's noise. Standard two-sample tests assume independent observations; switchback outcomes are anything but, and using a naive t-test on them gives anti-conservative variance and false positives. But autocorrelation is not merely a nuisance to correct — its sign changes the optimal design (Wen, Shi, Yang, Tang, and Zhu, 2025).

The result is a small decision table. When carryover is weak and the outcome errors are positively autocorrelated, switching more frequently improves precision — rapid flips average out the correlated noise. When errors are negatively correlated, infrequent switching (an alternating-day design) is most efficient. When carryover is large, infrequent switching wins regardless, because you need long settled stretches. So the same instinct — "switch fast" — that helps in one regime hurts in another, and the diagnosis hinges on quantities you have to estimate from history.

None of this makes switchbacks free. Pankratev (2026), working from DoorDash's cluster-by-time designs, derives a closed-form variance formula whose bottom line is blunt: a switchback is strictly less powerful than a naive A/B test on the same data. The extra variance term is the price of validity under interference. Worse, in a dense but imbalanced marketplace the penalty compounds — a size-imbalance factor \( (1 + cv^2) \) attaches permanently to system-wide shocks and does not shrink as you pile more observations into existing cells. In one worked regime the switchback needed on the order of fourteen times more data than a naive A/B test for equal power, though moving to a finer cell-level estimator roughly doubled the power back. The lesson is not to avoid switchbacks — it is to reserve them for the settings that genuinely need them, and to budget for the power tax.

Switchbacks in Media

Where does this land for advertising measurement? Switchbacks are the right tool for system-level, fast-washing levers that cannot be split cleanly by user: real-time bidding algorithms, budget pacing, auction and ranking logic, ad-load and slot allocation, marketplace and retail-media pricing. These share the auction- or inventory-interference that biases user-split tests, and their effects dissipate in minutes to hours, so temporal blocks stay short and power stays affordable. Delivery and ride-hailing platforms have reported using switchbacks to measure the incrementality of marketplace search ads under exactly these conditions.

⚠️ Long adstock is the killer — do not switchback brand media

The optimal block length scales with the carryover length, and burn-in must cover the settling time. If a channel's effect decays over weeks — the regime of brand and upper-funnel advertising, where adstock lingers — a temporal switchback would demand multi-week blocks plus multi-week burn-ins, discarding most of the horizon and collapsing power. That is precisely the "carryover comparable to the horizon" regime where the rate floor bites. For durable media the appropriate tools are the spatial ones: geo experiments, staggered rollouts, or a long-run MMM calibrated by lift tests. Switch time for fast levers; switch geography for slow ones.

It is worth seeing the switchback next to its spatial sibling. A geo experiment randomizes regions and holds time, spending cross-sectional units to buy immunity from temporal carryover — which is why it suits durable effects that need weeks to accrue. A switchback randomizes time and holds geography, spending temporal units to buy immunity from cross-user interference — which is why it suits fast, congestion-mediated effects. Neither dominates; they fail in opposite directions, and a clustered design that randomizes spatial clusters and time blocks together (Jia, Kallus, and Yu, 2023) exists precisely for the problems that have both kinds of spillover at once. The framework's own continuous-learning loop leans on the geo side because media carryover is long; a switchback is the instrument you reach for when the lever you are testing resets faster than a decision cycle.

Takeaways

  • A switchback randomizes a single system on and off over time, defeating the marketplace interference that biases user-level A/B tests when treated and control units share a resource pool.
  • The estimand is the global "always-on vs always-off" effect, defined conditional on the experiment's environment — so external validity is a first-order concern.
  • Carryover sets everything. Block length should track the carryover length (about \( m + 1 \) for finite order \( m \)); with geometric mixing, most of each block becomes burn-in and the naive estimator's rate floor is a slow \( T^{-1/3} \).
  • Inference is design-based: a Horvitz–Thompson estimator and exact randomization tests give validity with no outcome model, resting on probabilistic assignment and no anticipation.
  • Switch faster under weak carryover and positive noise autocorrelation; switch slower under strong carryover or negative autocorrelation — the optimal cadence flips with the regime.
  • Use switchbacks for fast levers (bidding, pacing, pricing, matching); use geo experiments for durable brand media, where long adstock makes temporal switching collapse.

🧰 This method ships in the framework

Switchback is a first-class named method in the experiment toolkit (switchback in the planning.methods registry): a block-randomized on/off schedule whose block length defaults to the channel's fitted adstock washout, an analysis burn-in at block boundaries, and power that is honest to both carryover and autocorrelation (the AR(1) design effect on detrended levels, plus A/A calibration on your own history). A user-level companion — the ghost-ads power calculator — covers the individually-randomized case.

References