Switchback Experiments
A geo experiment randomizes where, and a user A/B test randomizes who. A switchback randomizes when: it flips the entire system between treatment and control over time and reads the effect off the difference. That sounds like a strange thing to do until you hit a setting where splitting users is quietly biased, a marketplace or an auction or any other place where treated and control units compete for the same finite pool. Time-randomization trades one interference problem for another, and the new one is carryover. Everything interesting about the design is a way of managing that trade.
Why Not Just Split Users
The workhorse online experiment splits users into treatment and control and compares them. Its validity rests on the stable-unit-treatment-value assumption (SUTVA): one user's outcome does not depend on another user's assignment. In a two- or three-sided marketplace, that assumption fails outright. A ride-hailing pricing change tested on half the riders still draws from the same pool of drivers. The new ranking rule you show to half the sessions still competes in the same auction for the same inventory. Treated and control users interfere through the shared resource. The A/B contrast then measures a distorted, congestion-mediated thing rather than the effect of deploying the policy to everyone.
This is not a hypothetical. The design emerged from exactly this pain at ride-sharing and delivery platforms. Lyft's engineering team described "time-split" testing to escape marketplace interference (Chamandy, 2016), and DoorDash formalized region-by-time "switchback" tests because a shared Dasher fleet makes simultaneous treatment and control orders statistically dependent (Kastelman and Ramesh, 2018). The idea itself is far older: agricultural statisticians ran "switchback" or change-over trials on dairy cows in the 1930s and 40s, alternating each animal between diets to cancel between-cow variation, and already writing a regression term for the lagged carryover effect (Cochran, Autrey, and Cannon, 1941). The modern marketplace rediscovered a very old design under a very new kind of interference.
What a Switchback Is
Formally, a switchback is a cluster-randomized experiment where the cluster is time. Take a single aggregate unit (a whole city, market, or platform) and a binary treatment \( W_t \in \{0, 1\} \). Divide the horizon \( t = 1, \dots, T \) into blocks of length \( l \), hold the treatment constant within each block, and re-randomize it at every block boundary. Because the entire system is on one arm at any instant, there are never two co-existing treatment regimes to interfere with each other. You have converted a spatial interference problem into a temporal one.
Definition: The global average treatment effect
The estimand is the difference between running the system always treated versus always in control, averaged over the experiment window. With \( \tau_t \) the stable per-period effect at time \( t \),
$$ \tau_{\mathrm{GATE}} \;=\; \frac{1}{T}\sum_{t=1}^{T} \tau_t. $$This is the "if we shipped it to everyone, forever" quantity a decision-maker actually wants (Hu and Wager, 2022). A related target, the filtered ATE, averages only over a chosen subset of periods: the ones that have stopped settling after a switch.
The estimand is a whole-system quantity, and it is defined conditional on the environment that prevailed during the experiment. It is not an individual-level effect. If the two weeks you ran the switchback were unusually hot, promotional, or seasonal, the GATE you measured is the GATE in that environment. External validity ("was this window representative?") is a first-order concern here, and it earns that status precisely because the design leans so hard on the time dimension.
Carryover Is the Whole Problem
Randomizing over time defeats spatial interference but creates a new adversary: the treatment at time \( t \) moves \( Y_t \) and future outcomes alike, because the system carries state. Right after a switch from control to treatment, the outcome still reflects the lingering control regime. Surge prices have not reset, the matching queue has not cleared, demand has not caught up. Naively differencing treated against control periods folds that contamination straight into the estimate. There are two ways to model it, and they lead to different designs.
The first assumes carryover has a finite, known order \( m \): an action at \( t \) has fully dissipated by \( t + m \), so \( Y_t \) depends only on the treatments in the window \( W_{t-m:t} \). Surge-pricing effects that "dissipate after one or two hours" fit this picture, and \( m \) can be set from domain knowledge or estimated from data (Bojinov, Simchi-Levi, and Zhao, 2023). The second, more realistic, stance is geometric mixing: the system is a non-stationary Markov process with mixing time \( t_{\mathrm{mix}} \), and the influence of an action decays like \( e^{-(t' - t)/t_{\mathrm{mix}}} \) but never reaches exactly zero (Hu and Wager, 2022). Under geometric mixing the remedy is a burn-in. Discard the first \( b \) observations after every switch, while the system is still relaxing toward its new steady state, and analyze only the settled periods.
⚠️ Carryover contamination is the failure mode, and it is asymmetric with the horizon
The lag-\( p \) "always-on vs always-off" estimand stops being well-defined once the carryover length approaches the horizon \( T \). No clean stretch of pure-treatment or pure-control state is left to observe. Everything below about choosing the switching interval is, at bottom, a negotiation with carryover: the longer the effect persists, the longer your blocks and burn-ins must be, and the less data you have left to estimate anything.
The Interval Tradeoff
Block length is the single design knob, and it sits on a clean bias–variance tension. Short blocks give many independent randomizations (low variance) but leave the system little time to settle, so carryover bias is large. Long blocks give clean, settled measurement (low bias) but few randomizations, so variance is large. Burn-in adds a second knob that spends data to buy bias reduction.
Hu and Wager (2022) make the tension quantitative, and the punchline is sobering. For the naive block difference-in-means estimator targeting the global effect under geometric mixing, the root-mean-square error cannot beat order \( T^{-1/3} \). That is markedly slower than the \( T^{-1/2} \) you would get with no carryover at all, and the rate-optimal choice under that estimator is a single long block scaling like \( T^{1/3} \). A smarter analysis recovers most of the loss. With a burn-in of order \( t_{\mathrm{mix}} \log T \) and either a filtered estimand or a bias-corrected inverse-propensity estimator, the RMSE improves to order \( \sqrt{\log T / T} \), within a \( \sqrt{\log T} \) factor of the carryover-free ideal.
The practical shape of that solution surprises people. Most periods should be burn-in. You switch relatively often, throw away the majority of each block as it settles, and keep only the tail.
Their ride-sharing simulation shows what this buys in coverage, the number that actually matters for a decision. Adding burn-in to the difference-in-means estimator cut the bias from about 2.64 to 0.09 and lifted the confidence-interval coverage from 78% toward a healthy 92%. An over-confident, mis-centered interval became a usable one.
The block-length dial
A single system over \( T=96 \) periods, re-randomized on/off at each block boundary (seeded). Treatment carries over for \( m \) periods after every switch, so the shaded periods right after a flip are contaminated and bias the naive difference. Short blocks switch often (low variance) but spend most of their length settling (high bias). Long blocks measure cleanly but give few randomizations (high variance). The lower panel traces that bias–variance trade, and the total-error minimum sits near block length \( m+1 \).
Contamination falls as blocks grow, but variance rises with them, because there are fewer independent switches. The sweet spot aligns the block length to the carryover order, \( \ell \approx m+1 \), exactly the minimax rule from the design table below.
Inference Without a Model
A quietly radical feature of the modern switchback literature is that it is design-based. Potential outcomes are treated as fixed but unknown. The only randomness in the whole setup is the assignment path the coin flips generated. That lineage (Fisher, Neyman, Kempthorne, Rubin) buys you inference that does not depend on getting an outcome model right.
💡 The estimator is a weighted average, and it is unbiased by construction
The workhorse is the Horvitz–Thompson estimator: weight each period's outcome by the inverse probability of the treatment path it actually received. Under a properly randomized switchback this is unbiased for the temporal effect, with no likelihood, no stationarity assumption, and no functional form for how spend maps to sales (Bojinov and Shephard, 2019; Bojinov, Simchi-Levi, and Zhao, 2023). You can go further and compute an exact randomization p-value for the sharp null of no temporal effect by re-running the randomization on the observed data. That test is valid in finite samples, whatever the outcome process looks like.
Two assumptions are load-bearing. Probabilistic assignment: every period must have a strictly-between-zero-and-one chance of each arm, or the inverse-propensity weights blow up. And no anticipation: the assignment at time \( t \) must not depend on future potential outcomes. Formally, the future outcomes do not Granger-cause the treatment (Bojinov and Shephard, 2019). A randomized switchback satisfies this by construction, which is exactly why an observational "we turned the feature on when it seemed slow" on/off history does not qualify: there, the toggling responds to the very outcomes you are trying to measure. Bojinov and Shephard's own application came from finance, where a hedge fund compared human versus algorithmic trade execution across equity-index futures. Randomizing a single system over time is a general causal tool. They also note a variance-reduction trick with a familiar flavor: subtracting a one-step-ahead proxy prediction of the outcome before analyzing is the design-based cousin of CUPED.
Designing It Well
An optimal schedule exists, and it has been worked out. Bojinov, Simchi-Levi, and Zhao (2023) pose the design as a minimax problem (choose the randomization points and probabilities to minimize the worst-case variance over an adversarial choice of potential outcomes) and then solve it. The answer is reassuringly simple, and it gives a rule you can carry in your head.
| Design choice | Consequence (finite carryover order \( m \)) |
|---|---|
| Switch every period | Wastes power: about 28–32% higher variance than optimal |
| Block length \( \approx m + 1 \) | Within 1–2% of the optimal minimax design |
| One giant block | Too few randomizations, so variance dominates |
| Over-specify \( p \gt m \) | Stays unbiased at small variance cost |
| Under-specify \( p \lt m \) | Estimates a different, weighted estimand |
The middle row is the headline. Align the block length to the carryover order, roughly \( m + 1 \), and you capture almost all of the achievable efficiency. The optimal schedule is nearly regularly spaced at that interval, with slightly longer boundary blocks and switching probabilities kept to multiples of one-half to avoid unstable weights. And when you are unsure of \( m \), the asymmetry is your friend: over-specifying the carryover order costs a little variance but preserves unbiasedness, while under-specifying quietly changes the quantity you are estimating. When in doubt, assume the effect lingers longer than you think.
The Autocorrelation Coupling
A second time-series subtlety interacts with carryover in a way that surprises people: the serial correlation of the outcome's noise. Standard two-sample tests assume independent observations. Switchback outcomes are anything but, and a naive t-test on them gives anti-conservative variance and false positives. The sign of that autocorrelation also moves the optimal design, which makes it a good deal more than a nuisance to correct after the fact (Wen, Shi, Yang, Tang, and Zhu, 2025).
The result is a small decision table. When carryover is weak and the outcome errors are positively autocorrelated, switching more frequently improves precision, because rapid flips average out the correlated noise. Negatively correlated errors reverse the advice, and infrequent switching (an alternating-day design) becomes the efficient choice. Under large carryover, infrequent switching wins regardless, because you need long settled stretches. So the same instinct ("switch fast") helps in one regime and hurts in another, and the diagnosis hinges on quantities you have to estimate from history.
None of this makes switchbacks free. Pankratev (2026), working from DoorDash's cluster-by-time designs, derives a closed-form variance formula whose bottom line is blunt: a switchback is strictly less powerful than a naive A/B test on the same data. The extra variance term is the price of validity under interference. Worse, in a dense but imbalanced marketplace the penalty compounds: a size-imbalance factor \( (1 + cv^2) \) attaches permanently to system-wide shocks and does not shrink as you pile more observations into existing cells. In one worked regime the switchback needed on the order of fourteen times more data than a naive A/B test for equal power, though moving to a finer cell-level estimator roughly doubled the power back. That arithmetic argues for choosing switchbacks carefully, never for abandoning them. Reserve them for the settings that genuinely need them, and budget for the power tax when you do.
Switchbacks in Media
In advertising measurement, switchbacks fit system-level, fast-washing levers that cannot be split cleanly by user: real-time bidding algorithms, budget pacing, auction and ranking logic, ad-load and slot allocation, marketplace and retail-media pricing. These all share the auction- or inventory-interference that biases user-split tests. Their effects also dissipate in minutes to hours, so temporal blocks stay short and power stays affordable. Delivery and ride-hailing platforms have reported using switchbacks to measure the incrementality of marketplace search ads under exactly these conditions.
⚠️ Long adstock is the killer: do not switchback brand media
The optimal block length scales with the carryover length, and burn-in must cover the settling time. If a channel's effect decays over weeks (the regime of brand and upper-funnel advertising, where adstock lingers), a temporal switchback would demand multi-week blocks plus multi-week burn-ins. You would discard most of the horizon and collapse the power you started with. That is precisely the "carryover comparable to the horizon" regime where the rate floor bites. For durable media the appropriate tools are the spatial ones: geo experiments, staggered rollouts, or a long-run MMM calibrated by lift tests. Switch time for fast levers. Switch geography for slow ones.
Set the switchback next to its spatial sibling. A geo experiment randomizes regions and holds time, spending cross-sectional units to buy immunity from temporal carryover, which is why it suits durable effects that need weeks to accrue. A switchback randomizes time and holds geography, spending temporal units to buy immunity from cross-user interference, and that is why it suits fast, congestion-mediated effects. Neither dominates. They fail in opposite directions, and a clustered design that randomizes spatial clusters and time blocks together (Jia, Kallus, and Yu, 2023) exists precisely for the problems that have both kinds of spillover at once. The framework's own continuous-learning loop leans on the geo side because media carryover is long. A switchback is the instrument you reach for when the lever you are testing resets faster than a decision cycle.
Takeaways
- A switchback randomizes a single system on and off over time, defeating the marketplace interference that biases user-level A/B tests when treated and control units share a resource pool.
- The estimand is the global "always-on vs always-off" effect, defined conditional on the experiment's environment, which makes external validity a first-order concern.
- Carryover sets everything. Block length should track the carryover length (about \( m + 1 \) for finite order \( m \)). With geometric mixing, most of each block becomes burn-in, and the naive estimator's rate floor is a slow \( T^{-1/3} \).
- Inference is design-based: a Horvitz–Thompson estimator and exact randomization tests give validity with no outcome model, resting on probabilistic assignment and no anticipation.
- Switch faster under weak carryover and positive noise autocorrelation, and slower under strong carryover or negative autocorrelation. The optimal cadence flips with the regime.
- Use switchbacks for fast levers (bidding, pacing, pricing, matching), and geo experiments for durable brand media, where long adstock makes temporal switching collapse.
🧰 This method ships in the framework
Switchback is a first-class named method in the experiment toolkit (switchback in the planning.methods registry): a block-randomized on/off schedule whose block length defaults to the channel's fitted adstock washout, an analysis burn-in at block boundaries, and power that is honest to both carryover and autocorrelation (the AR(1) design effect on detrended levels, plus A/A calibration on your own history). The ghost-ads power calculator is the user-level companion, and it covers the individually-randomized case.
References
- Bojinov, I., Simchi-Levi, D., & Zhao, J. (2023). Design and Analysis of Switchback Experiments. Management Science, 69(7).
- Bojinov, I., & Shephard, N. (2019). Time Series Experiments and Causal Estimands: Exact Randomization Tests and Trading. Journal of the American Statistical Association, 114(528).
- Hu, Y., & Wager, S. (2022). Switchback Experiments under Geometric Mixing. arXiv:2209.00197.
- Wen, Q., Shi, C., Yang, Y., Tang, N., & Zhu, H. (2025). Unraveling the Interplay between Carryover Effects and Reward Autocorrelations in Switchback Experiments. ICML (PMLR 267). arXiv:2403.17285.
- Pankratev, S. (2026). Powerful Switchback Experiments, Or Not? arXiv:2606.03012.
- Jia, S., Kallus, N., & Yu, C. L. (2023). Clustered Switchback Designs for Experimentation Under Spatio-temporal Interference. arXiv:2312.15574.
- Glynn, P. W., Johari, R., & Rasouli, M. (2020). Adaptive Experimental Design with Temporal Interference: A Maximum Likelihood Approach. NeurIPS, 33.
- Xiong, R., Chin, A., & Taylor, S. J. (2024). Data-Driven Switchback Experiments: Theoretical Tradeoffs and Empirical Bayes Designs. arXiv:2406.06768.
- Cochran, W. G., Autrey, K. M., & Cannon, C. Y. (1941). A Double Change-Over Design for Dairy Cattle Feeding Experiments. Journal of Dairy Science, 24(11).
- Chamandy, N. (2016). Experimentation in a Ridesharing Marketplace. Lyft Engineering.
- Kastelman, D., & Ramesh, R. (2018). Switchback Tests and Randomized Experimentation Under Network Effects at DoorDash. DoorDash Engineering.