Modern Staggered Difference-in-Differences
Campaigns and product features rarely launch everywhere at once. They roll out market by market over months. The regression most teams reach for to measure such a rollout, two-way fixed effects, can get the answer badly wrong when effects differ across markets or build over time. Here is why it fails, and what the modern rebuild of difference-in-differences puts in its place.
Picture a brand launching a new always-on campaign DMA by DMA: ten markets in January, fifteen more in April, another dozen in September, and a set of markets with no launch planned. Every market that starts, stays on. Six months in, the CMO asks the obvious question: what did the launch do to sales? The design is a staggered adoption panel, one of the most common structures in applied measurement. So it pays to know exactly when the standard tools work, and when they quietly invert the answer.
A staggered-adoption calendar: each launch wave turns on in a different month and stays on, alongside a never-launched holdout. This timing structure is exactly what breaks a single two-way-fixed-effects coefficient once effects differ by wave or build over time.
The Classic 2×2 and Parallel Trends
Difference-in-differences begins with two groups and two periods. One group receives treatment between the periods. The other never does. Each group's before–after change removes anything fixed about that group, and differencing the two changes removes the shared time trend:
$$ \hat{\beta}_{DD} = \big(\bar{Y}_{\text{treat,after}} - \bar{Y}_{\text{treat,before}}\big) - \big(\bar{Y}_{\text{control,after}} - \bar{Y}_{\text{control,before}}\big). $$The identifying assumption is parallel trends: absent treatment, the treated group's outcome would have moved in parallel with the control group's. Levels may differ, since a launch market can be structurally bigger than a holdout market. The trends are what must match. This is the same style of untestable structural assumption that underpins any observational causal inference: the data alone cannot certify it, so the design has to make it plausible. Card and Krueger's (1994) minimum-wage study is the canonical application, where New Jersey raised its minimum wage and Pennsylvania did not. The estimator generalizes naturally to a regression with group and time effects, which makes adding covariates and extra periods easy (Angrist & Pischke, 2009). A Bayesian formulation of the same 2×2 model returns a full posterior over the effect rather than a point estimate, with the parallel-trends assumption embodied as a single shared slope (Vincent, 2022).
Where Two-Way Fixed Effects Breaks
With many periods and staggered launch dates there is no single "before" and "after," so the standard move is the two-way fixed effects (TWFE) regression:
$$ Y_{i,t} = \alpha_i + \lambda_t + \beta\, D_{i,t} + \varepsilon_{i,t}, $$with unit effects \( \alpha_i \), time effects \( \lambda_t \), and a treatment dummy \( D_{i,t} \) that switches on when market \( i \) launches. The temptation is to read \( \beta \) as "the" average treatment effect. It is not. Under staggered adoption, OLS mechanically constructs \( \beta \) as a weighted average of many underlying cohort-period effects, with weights chosen by the estimator's variance algebra rather than by any question you asked. Goodman-Bacon (2021) shows the estimate decomposes into all possible 2×2 comparisons in the panel, including ones that use already-treated markets as controls for later launchers. The trouble arrives when treatment effects are dynamic, and a campaign effect that builds with adstock and awareness is exactly that. In those "control" markets the still-growing effect gets differenced away as if it were a baseline trend. Some effects then enter \( \beta \) with negative weights, and the estimate can come out negative even when the launch helped every single market.
Nor does the familiar event-study regression with leads and lags rescue things. Sun and Abraham (2021) show that under heterogeneous effects each estimated lag coefficient is contaminated by other cohorts' effects at other horizons. The plotted "dynamics" can be an artifact of the weighting rather than a property of any cohort. And marketing rollouts are heterogeneous almost by construction. Early-launch markets are usually picked because someone expects them to respond well, effects ramp over weeks, and creative or budgets differ by wave.
💡 The failure is heterogeneity, not staggering alone
If every market had exactly the same effect at every point in time, TWFE would be fine, and in that special case all the estimands discussed below, along with TWFE's \( \beta \), coincide. What breaks it is the interaction of staggered timing with effects that vary by cohort or evolve with exposure. Unless you are prepared to assert a constant, instantaneous, permanent effect, you need the machinery below.
When a single coefficient inverts the answer
A population panel with an early wave and a late wave (plus a never-treated group). The green markers are the clean, cohort-averaged event study: leads at zero, lags rising as the effect builds. The dashed lines compare the heterogeneity-robust true ATT (green) against the TWFE single coefficient (red). Slide up the dynamics-and-heterogeneity strength and watch what happens. TWFE uses already-treated markets as controls, differences away their still-growing effect, and drifts far below the true ATT, understating by half or more an effect that is unambiguously positive (the same mechanism that, in harsher panels, drives the coefficient outright negative).
At zero the effect is constant and homogeneous, so TWFE equals the true ATT. Every step away from zero adds dynamics or heterogeneity, the exact conditions under which the negative-weight pathology bites.
The Building Block: ATT(g,t)
Callaway and Sant'Anna (2021) rebuild staggered DiD by separating three steps that TWFE fuses together: identify disaggregated effects, aggregate them with weights you choose, and infer. The disaggregated object is indexed by cohort and calendar time. Define a unit's group \( g \) as the period it is first treated (with \( g = \infty \) for the never-treated), and require staggered adoption: once treated, always treated.
Definition: Group-time average treatment effect
For a cohort first treated in period \( g \), observed at calendar time \( t \),
$$ \mathrm{ATT}(g,t) \;=\; \mathbb{E}\big[\, Y_t(g) - Y_t(0) \mid G_g = 1 \,\big], $$the average difference between the outcome under first-treatment-at-\( g \) and the never-treated potential outcome, among group-\( g \) units. It imposes no restriction on how effects differ across cohorts, when treatment started, or how effects evolve over time. In the two-period, two-group case it collapses to the canonical ATT.
In the rollout example, \( \mathrm{ATT}(\text{Apr}, \text{Jun}) \) is the average lift in the April-launch wave two months in. Fix the cohort and sweep \( t \), and you trace that wave's ramp curve. Compare across cohorts and you learn whether early waves genuinely respond differently. Each number carries a clean causal interpretation on its own, which is precisely what a single TWFE coefficient cannot claim. The paper's empirical application (county minimum-wage increases and teen employment) reads the family directly: the cohort treated in 2004 shows effects growing from roughly \(-3.4\%\) on impact to \(-13.6\%\) three years later, a dynamic pattern a single pooled coefficient would flatten into mush.
What You Have to Assume
Point-identifying the whole family \( \{\mathrm{ATT}(g,t)\} \) requires four conditions beyond random sampling. They are the staggered-panel analogue of the identification assumptions any causal model should state up front.
- Limited anticipation. There is a known horizon \( \delta \ge 0 \) such that units do not respond before \( g - \delta \). With \( \delta = 0 \) this is the standard no-anticipation condition. Allowing \( \delta \gt 0 \) handles, say, markets that see teaser activity a period before launch. The trade-off is that the larger the anticipation window you allow, the stronger the parallel-trends condition you need.
- Conditional parallel trends (never-treated comparison). Conditional on covariates \( X \), each cohort's untreated outcome path would have moved in parallel with the never-treated group's. Natural when a sizable, comparable set of no-launch markets exists.
- Conditional parallel trends (not-yet-treated comparison). Alternatively, use as controls the units not yet treated by the relevant date. This exploits more comparison units (better precision) and is essential when nearly everyone eventually launches. Unlike the never-treated version, though, it does restrict pre-treatment trends, and it can fail if early periods experienced different shocks than later ones.
- Overlap. Each cohort has positive mass, and the generalized propensity score (the probability of belonging to cohort \( g \) given covariates) is bounded away from one. No cohort may be perfectly predictable from observables.
The word conditional is doing real work. Launch order is rarely random: bigger, faster-growing markets go first. If market size and growth also drive baseline sales trends, unconditional parallel trends is not credible. Parallel trends within strata of market characteristics still might be. Callaway and Sant'Anna's framework is the first to incorporate covariate-specific trends into staggered designs, and only pre-treatment covariates qualify, since post-launch covariates can themselves be moved by the campaign.
⚠️ Don't pre-test your way to an assumption
It is tempting to run statistical tests on pre-treatment trends and let the result choose your comparison group or bless the design. Roth (2022) shows this conditioning distorts subsequent inference. Estimates that survive a pre-test are a selected sample, and their confidence intervals no longer mean what they say. Choose the comparison group from context, meaning the markets that genuinely resemble the launch markets absent the launch. Use pre-trend estimates as a visual credibility check rather than a gate.
Doubly Robust Estimation
Under those assumptions, each \( \mathrm{ATT}(g,t) \) is nonparametrically identified three equivalent ways (Callaway & Sant'Anna, 2021, Theorem 1): by outcome regression (model the comparison group's outcome evolution given covariates), by inverse probability weighting (model cohort membership, following Abadie, 2005), or by a doubly robust combination of both (extending Sant'Anna & Zhao, 2020, and the matching-evaluation tradition of Heckman, Ichimura & Todd, 1997). All three compare "long differences": the outcome change from a reference period \( g - \delta - 1 \), the last period before anticipation could contaminate the baseline, out to period \( t \). With no anticipation and no covariates, everything collapses to the intuitive 2×2-style contrast of average changes.
The doubly robust estimand, for the never-treated comparison with \( \delta = 0 \), reweights the long difference net of the outcome-regression prediction:
$$ \mathrm{ATT}_{dr}(g,t) = \mathbb{E}\!\left[\left(\frac{G_g}{\mathbb{E}[G_g]} - \frac{\tfrac{p_g(X)\,C}{1-p_g(X)}}{\mathbb{E}\!\left[\tfrac{p_g(X)\,C}{1-p_g(X)}\right]}\right)\Big(Y_t - Y_{g-1} - m_{g,t}(X)\Big)\right], $$where \( p_g(X) \) is the generalized propensity score, \( C \) flags never-treated units, and \( m_{g,t}(X) \) is the comparison group's expected outcome change given covariates. In practice both nuisance functions are fit with ordinary parametric workhorses (a logit for \( p_g \), least squares for \( m_{g,t} \)), and the estimator is a two-step plug-in. That extra structure earns its keep. The DR estimator remains consistent if either the propensity model or the outcome model is correctly specified, not necessarily both. Neither model is ever exactly right, so buying two chances at consistency for the price of one specification is the sensible default.
Deep diveWhy double robustness works, in one paragraph
Algebraically, the DR estimand is the IPW estimand minus a correction term involving \( m_{g,t}(X) \). If the propensity score is correct, the treated and reweighted-comparison averages of any function of \( X \) coincide, so the correction has expectation zero by the law of iterated expectations, and the DR estimand inherits IPW's consistency whatever \( m \) is. Symmetrically, the DR estimand equals the outcome-regression estimand plus a term that vanishes whenever \( m \) is the true comparison-group regression, whatever \( p_g \) is. Each nuisance model covers for the other's failure, and the estimand only breaks when both are wrong. The same theorem carries a caveat. The three estimands are identical as identification targets but generally differ as finite-sample estimators once nuisances are fitted. And a TWFE regression with covariates entered linearly does not recover \( \mathrm{ATT}(g,t) \) unless effects are homogeneous in \( X \) and covariate-specific trends are ruled out. Regression adjustment is no substitute for the estimand.
Aggregations That Answer Questions
A rollout with a dozen cohorts and two years of data produces a large family of \( \mathrm{ATT}(g,t) \)'s, far too many to brief a CMO on. The second step aggregates them with explicit, non-negative, researcher-chosen weights: \( \theta = \sum_g \sum_t w(g,t)\, \mathrm{ATT}(g,t) \). Different weighting schemes answer different questions:
| Aggregation | Question it answers | Form |
|---|---|---|
| Event study \( \theta_{es}(e) \) | What is the average effect \( e \) periods after launch? | Average of \( \mathrm{ATT}(g, g+e) \) over cohorts observed at exposure \( e \) |
| Group \( \theta_{sel}(\tilde g) \) | How did the \( \tilde g \) launch wave do overall? | Mean of \( \mathrm{ATT}(\tilde g, t) \) over its post periods |
| Calendar \( \theta_c(\tilde t) \) | What was the effect in period \( \tilde t \) among live markets? | Cohort-share-weighted \( \mathrm{ATT}(g, \tilde t) \) for \( g \le \tilde t \) |
| Overall \( \theta_{sel}^{O} \) | What was the average effect of ever launching? | Group effects \( \theta_{sel}(g) \), averaged over cohorts by size |
The event-study aggregation is the proper target for the ramp-curve plot every marketer wants, without the contaminated-lag pathology of dynamic TWFE. It carries one caveat: as \( e \) grows, later cohorts drop out of the average, so the curve mixes true dynamics with compositional change. The balanced variant fixes the set of cohorts observed for at least \( e' \) periods, trading fewer groups for cleaner dynamics. For a single headline number, the recommended overall parameter averages each cohort's post-launch effect first and then averages across cohorts by size,
$$ \theta_{sel}^{O} \;=\; \sum_{g \in \mathcal{G}} \theta_{sel}(g)\, P\big(G = g \mid G \le \mathcal{T}\big), $$the natural multi-period analogue of the classic ATT: the average effect experienced by markets that ever launched, without over-weighting the waves that happen to have been live longest. Because every scheme uses non-negative weights, the positive-effects-everywhere-but-negative-estimate pathology is ruled out by construction. These summaries genuinely differ from one another whenever effects are heterogeneous. They coincide with each other, and with TWFE, only in the constant-effect special case.
Simultaneous Inference
The estimators are \( \sqrt{n} \)-asymptotically normal with an influence function that accounts for estimating the nuisance models. Rather than plug-in standard errors, Callaway and Sant'Anna recommend a multiplier bootstrap: perturb each unit's estimated influence-function contribution by an i.i.d. mean-zero, unit-variance weight (Mammen, 1993) and recompute. No propensity model is re-fit per draw. Every cohort is represented in every draw, which an empirical bootstrap cannot promise, since it can resample a small cohort away entirely. Clustering amounts to drawing weights at the cluster level. And the whole procedure is fast: the paper's application, with a thousand bootstrap iterations, runs in about three seconds.
The reason to care is simultaneity. An event-study plot displays many estimates at once, and pointwise 95% intervals around each understate the joint uncertainty of the path. Some interval will exclude its truth by chance alone. The multiplier bootstrap yields uniform confidence bands: a widened band, calibrated from the bootstrap distribution of the studentized maximum deviation, that covers the entire path of effects with the stated probability. Pre-treatment placebo effects (which should be zero under the assumptions) can be estimated alongside and plotted inside the same band. That gives you a read on parallel trends that is honest about multiple testing.
The stakes are not hypothetical. Staggered state adoption, dynamic effects, cohort heterogeneity: the paper's minimum-wage application carries all the complications a marketing rollout shares. There the TWFE post-treatment coefficient is a statistically insignificant \(-0.008\), while the heterogeneity-robust overall effect under conditional parallel trends is about \(-3.1\%\), with event-study effects growing from roughly \(-2.4\%\) on impact to \(-7.1\%\) at three years of exposure. The method choice decides the qualitative conclusion here, well beyond the decimal places.
Reading a Rollout in Practice
For a measurement team the recipe is concrete. Encode each market's launch date as its group. Decide from context whether never-launched markets are credible controls or whether not-yet-launched markets must serve. Pick the pre-launch covariates that make within-stratum parallel trends defensible, estimate the \( \mathrm{ATT}(g,t) \) family doubly robustly, and report the event-study band plus the overall \( \theta_{sel}^{O} \) with uniform intervals (the reference implementation is the R package did). The output slots naturally into a broader measurement program. A staggered-rollout readout is exactly the kind of quasi-experimental evidence that can calibrate an MMM's channel priors, and its assumptions deserve the same explicit pre-registration we argue for in stating identification assumptions generally.
Takeaways
- A market-by-market rollout is a staggered adoption design. TWFE's single coefficient averages cohort-period effects with estimator-chosen, possibly negative weights. It can report a negative number when every market benefited.
- The fix is to disaggregate first: \( \mathrm{ATT}(g,t) \), the effect for the cohort launched at \( g \) measured at \( t \), is causally interpretable under any pattern of heterogeneity.
- Identification needs limited anticipation, conditional parallel trends against a deliberately chosen comparison group (never-treated or not-yet-treated), and overlap. Choose the comparison group by context rather than by pre-tests.
- Estimate doubly robustly: consistency survives one misspecified nuisance model. Aggregate with transparent non-negative weights matched to the question, whether that is a ramp curve, a wave report card, or one headline ATT.
- Use multiplier-bootstrap uniform bands so the event-study plot and its pre-trend placebos are honest about multiple comparisons.
🧰 This method ships in the framework
Matched-market difference-in-differences is a first-class named method in the experiment toolkit (did_mmt in the planning.methods registry), with residual-matched pairs, cluster-robust per-pair estimation, placebo-calibrated standard errors, and empirical power from the A/A·A/B simulation harness. It sits alongside synthetic control, TBR, and GBR in the Design Studio's method picker and the methodology leaderboard.
References
- Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-Differences with Multiple Time Periods. Journal of Econometrics, 225(2).
- Goodman-Bacon, A. (2021). Difference-in-Differences with Variation in Treatment Timing. Journal of Econometrics, 225(2).
- Sun, L., & Abraham, S. (2021). Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects. Journal of Econometrics, 225(2).
- de Chaisemartin, C., & D'Haultfœuille, X. (2020). Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects. American Economic Review, 110(9).
- Sant'Anna, P. H. C., & Zhao, J. (2020). Doubly Robust Difference-in-Differences Estimators. Journal of Econometrics, 219(1).
- Abadie, A. (2005). Semiparametric Difference-in-Differences Estimators. Review of Economic Studies, 72(1).
- Heckman, J. J., Ichimura, H., & Todd, P. E. (1997). Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme. Review of Economic Studies, 64(4).
- Card, D., & Krueger, A. B. (1994). Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania. American Economic Review, 84(4).
- Roth, J. (2022). Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends. American Economic Review: Insights, 4(3).
- Mammen, E. (1993). Bootstrap and Wild Bootstrap for High Dimensional Linear Models. Annals of Statistics, 21(1).
- Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press.
- Vincent, B. T. (2022). Difference in Differences. PyMC Examples.