Modern Staggered Difference-in-Differences

Matthew Reda June 17, 2026 Causal inference

Campaigns and product features rarely launch everywhere at once — they roll out market by market over months. The regression most teams reach for to measure such a rollout, two-way fixed effects, can get the answer badly wrong when effects differ across markets or build over time. This post walks through why it fails, and through the modern rebuild of difference-in-differences that replaces it.

Picture a brand launching a new always-on campaign DMA by DMA: ten markets in January, fifteen more in April, another dozen in September, and a set of markets with no launch planned. Every market that starts, stays on. Six months in, the CMO asks the obvious question: what did the launch do to sales? The design is a staggered adoption panel, and it is one of the most common structures in applied measurement — which makes it worth understanding exactly when the standard tools work and when they quietly invert the answer.

A staggered-adoption calendar: each launch wave turns on in a different month and stays on, alongside a never-launched holdout. This timing structure is exactly what breaks a single two-way-fixed-effects coefficient once effects differ by wave or build over time.

The Classic 2×2 and Parallel Trends

Difference-in-differences begins with two groups and two periods. One group receives treatment between the periods; the other never does. Each group's before–after change removes anything fixed about that group; differencing the two changes removes the shared time trend:

$$ \hat{\beta}_{DD} = \big(\bar{Y}_{\text{treat,after}} - \bar{Y}_{\text{treat,before}}\big) - \big(\bar{Y}_{\text{control,after}} - \bar{Y}_{\text{control,before}}\big). $$

The identifying assumption is parallel trends: absent treatment, the treated group's outcome would have moved in parallel with the control group's. Levels may differ — a launch market can be structurally bigger than a holdout market — but the trends must match. This is the same style of untestable structural assumption that underpins any observational causal inference: the data alone cannot certify it, so the design has to make it plausible. The canonical application is Card and Krueger's (1994) minimum-wage study — New Jersey raised its minimum wage, Pennsylvania did not — and the estimator generalizes naturally to a regression with group and time effects, which makes adding covariates and extra periods easy (Angrist & Pischke, 2009). A Bayesian formulation of the same 2×2 model returns a full posterior over the effect rather than a point estimate, with the parallel-trends assumption embodied as a single shared slope (Vincent, 2022).

Where Two-Way Fixed Effects Breaks

With many periods and staggered launch dates there is no single "before" and "after," so the standard move is the two-way fixed effects (TWFE) regression:

$$ Y_{i,t} = \alpha_i + \lambda_t + \beta\, D_{i,t} + \varepsilon_{i,t}, $$

with unit effects $ \alpha_i $, time effects $ \lambda_t $, and a treatment dummy $ D_{i,t} $ that switches on when market $ i $ launches. The temptation is to read $ \beta $ as "the" average treatment effect. It is not. Under staggered adoption, OLS mechanically constructs $ \beta $ as a weighted average of many underlying cohort-period effects, with weights chosen by the estimator's variance algebra rather than by any question you asked. Goodman-Bacon (2021) shows the estimate decomposes into all possible 2×2 comparisons in the panel — including ones that use already-treated markets as controls for later launchers. When treatment effects are dynamic — and a campaign effect that builds with adstock and awareness is exactly that — the still-growing effect in those "control" markets gets differenced away as if it were a baseline trend. The result is that some effects enter $ \beta $ with negative weights: the estimate can come out negative even when the launch helped every single market.

Nor does the familiar event-study regression with leads and lags rescue things. Sun and Abraham (2021) show that under heterogeneous effects each estimated lag coefficient is contaminated by other cohorts' effects at other horizons — the plotted "dynamics" can be an artifact of the weighting, not a property of any cohort. And in marketing rollouts heterogeneity is the norm, not the exception: early-launch markets are usually chosen because they are expected to respond well, effects ramp over weeks, and creative or budgets differ by wave.

💡 The failure is heterogeneity, not staggering alone

If every market had exactly the same effect at every point in time, TWFE would be fine — in that special case all the estimands discussed below, and TWFE's $ \beta $, coincide. The problem is the interaction of staggered timing with effects that vary by cohort or evolve with exposure. If you would not be willing to assert a constant, instantaneous, permanent effect, you need the machinery below.

When a single coefficient inverts the answer

A population panel with an early wave and a late wave (plus a never-treated group). The green markers are the clean, cohort-averaged event study — leads at zero, lags rising as the effect builds. The dashed lines compare the heterogeneity-robust true ATT (green) against the TWFE single coefficient (red). Slide up the dynamics-and-heterogeneity strength: TWFE uses already-treated markets as controls, differences away their still-growing effect, and drifts far below the true ATT — understating by half or more an effect that is unambiguously positive (the same mechanism that, in less forgiving panels, drives the coefficient outright negative).

Effect dynamics & cohort heterogeneity 0.80

True ATT (heterogeneity-robust)—

TWFE coefficient—

TWFE error—

Sign of TWFE—

At zero the effect is constant and homogeneous, so TWFE equals the true ATT. Every step away is dynamics or heterogeneity — the exact conditions under which the negative-weight pathology bites.

The Building Block: ATT(g,t)

Callaway and Sant'Anna (2021) rebuild staggered DiD by separating three steps that TWFE fuses together: identify disaggregated effects, aggregate them with weights you choose, and infer. The disaggregated object is indexed by cohort and calendar time. Define a unit's group $ g $ as the period it is first treated (with $ g = \infty $ for the never-treated), and require staggered adoption: once treated, always treated.

Definition: Group-time average treatment effect

For a cohort first treated in period $ g $, observed at calendar time $ t $,

$$ \mathrm{ATT}(g,t) \;=\; \mathbb{E}\big[\, Y_t(g) - Y_t(0) \mid G_g = 1 \,\big], $$

the average difference between the outcome under first-treatment-at-$ g $ and the never-treated potential outcome, among group-$ g $ units. It imposes no restriction on how effects differ across cohorts, when treatment started, or how effects evolve over time; in the two-period, two-group case it collapses to the canonical ATT.

In the rollout example, $ \mathrm{ATT}(\text{Apr}, \text{Jun}) $ is the average lift in the April-launch wave two months in. Fixing the cohort and sweeping $ t $ traces that wave's ramp curve; comparing across cohorts reveals whether early waves genuinely respond differently. Each number carries a clean causal interpretation on its own — which is precisely what a single TWFE coefficient cannot claim. The paper's empirical application (county minimum-wage increases and teen employment) reads the family directly: the cohort treated in 2004 shows effects growing from roughly $-3.4\%$ on impact to $-13.6\%$ three years later, a dynamic pattern a single pooled coefficient would flatten into mush.

What You Have to Assume

Point-identifying the whole family $ \{\mathrm{ATT}(g,t)\} $ requires four conditions beyond random sampling — the staggered-panel analogue of the identification assumptions any causal model should state up front.

Limited anticipation. There is a known horizon $ \delta \ge 0 $ such that units do not respond before $ g - \delta $. With $ \delta = 0 $ this is the standard no-anticipation condition; allowing $ \delta \gt 0 $ handles, say, markets that see teaser activity a period before launch. A subtle trade-off: the larger the anticipation window you allow, the stronger the parallel-trends condition you need.
Conditional parallel trends — never-treated comparison. Conditional on covariates $ X $, each cohort's untreated outcome path would have moved in parallel with the never-treated group's. Natural when a sizable, comparable set of no-launch markets exists.
Conditional parallel trends — not-yet-treated comparison. Alternatively, use as controls the units not yet treated by the relevant date. This exploits more comparison units (better precision) and is essential when nearly everyone eventually launches — but unlike the never-treated version it does restrict pre-treatment trends, and it can fail if early periods experienced different shocks than later ones.
Overlap. Each cohort has positive mass, and the generalized propensity score — the probability of belonging to cohort $ g $ given covariates — is bounded away from one. No cohort may be perfectly predictable from observables.

The word conditional is doing real work. Launch order is rarely random: bigger, faster-growing markets go first. If market size and growth also drive baseline sales trends, unconditional parallel trends is not credible — but parallel trends within strata of market characteristics may be. Callaway and Sant'Anna's framework is the first to incorporate covariate-specific trends into staggered designs, and only pre-treatment covariates qualify, since post-launch covariates can themselves be moved by the campaign.

⚠️ Don't pre-test your way to an assumption

It is tempting to run statistical tests on pre-treatment trends and let the result choose your comparison group or bless the design. Roth (2022) shows this conditioning distorts subsequent inference: estimates that survive a pre-test are a selected sample, and their confidence intervals no longer mean what they say. Choose the comparison group from context — which markets genuinely resemble the launch markets absent the launch — and use pre-trend estimates as a visual credibility check, not a gate.

Doubly Robust Estimation

Under those assumptions, each $ \mathrm{ATT}(g,t) $ is nonparametrically identified three equivalent ways (Callaway & Sant'Anna, 2021, Theorem 1): by outcome regression (model the comparison group's outcome evolution given covariates), by inverse probability weighting (model cohort membership, following Abadie, 2005), or by a doubly robust combination of both (extending Sant'Anna & Zhao, 2020, and the matching-evaluation tradition of Heckman, Ichimura & Todd, 1997). All three compare "long differences" — the outcome change from a reference period $ g - \delta - 1 $, the last period before anticipation could contaminate the baseline, out to period $ t $. With no anticipation and no covariates, everything collapses to the intuitive 2×2-style contrast of average changes.

The doubly robust estimand, for the never-treated comparison with $ \delta = 0 $, reweights the long difference net of the outcome-regression prediction:

$$ \mathrm{ATT}_{dr}(g,t) = \mathbb{E}\!\left[\left(\frac{G_g}{\mathbb{E}[G_g]} - \frac{\tfrac{p_g(X)\,C}{1-p_g(X)}}{\mathbb{E}\!\left[\tfrac{p_g(X)\,C}{1-p_g(X)}\right]}\right)\Big(Y_t - Y_{g-1} - m_{g,t}(X)\Big)\right], $$

where $ p_g(X) $ is the generalized propensity score, $ C $ flags never-treated units, and $ m_{g,t}(X) $ is the comparison group's expected outcome change given covariates. In practice both nuisance functions are fit with ordinary parametric workhorses — a logit for $ p_g $, least squares for $ m_{g,t} $ — and the estimator is a two-step plug-in. The payoff for the extra structure: the DR estimator remains consistent if either the propensity model or the outcome model is correctly specified, not necessarily both. Since neither model is ever exactly right, buying two chances at consistency for the price of one specification is the sensible default.

Deep diveWhy double robustness works, in one paragraph

Algebraically, the DR estimand is the IPW estimand minus a correction term involving $ m_{g,t}(X) $. If the propensity score is correct, the treated and reweighted-comparison averages of any function of $ X $ coincide, so the correction has expectation zero by the law of iterated expectations — the DR estimand inherits IPW's consistency, whatever $ m $ is. Symmetrically, the DR estimand equals the outcome-regression estimand plus a term that vanishes whenever $ m $ is the true comparison-group regression, whatever $ p_g $ is. Each nuisance model covers for the other's failure; the estimand only breaks when both are wrong. A caveat from the same theorem: the three estimands are identical as identification targets but generally differ as finite-sample estimators once nuisances are fitted — and a TWFE regression with covariates entered linearly does not recover $ \mathrm{ATT}(g,t) $ unless effects are homogeneous in $ X $ and covariate-specific trends are ruled out. Regression adjustment is not a substitute for the estimand.

Aggregations That Answer Questions

A rollout with a dozen cohorts and two years of data produces a large family of $ \mathrm{ATT}(g,t) $'s — too many to brief a CMO on. The second step aggregates them with explicit, non-negative, researcher-chosen weights: $ \theta = \sum_g \sum_t w(g,t)\, \mathrm{ATT}(g,t) $. Different weighting schemes answer different questions:

Aggregation	Question it answers	Form
Event study $ \theta_{es}(e) $	What is the average effect $ e $ periods after launch?	Average of $ \mathrm{ATT}(g, g+e) $ over cohorts observed at exposure $ e $
Group $ \theta_{sel}(\tilde g) $	How did the $ \tilde g $ launch wave do overall?	Mean of $ \mathrm{ATT}(\tilde g, t) $ over its post periods
Calendar $ \theta_c(\tilde t) $	What was the effect in period $ \tilde t $ among live markets?	Cohort-share-weighted $ \mathrm{ATT}(g, \tilde t) $ for $ g \le \tilde t $
Overall $ \theta_{sel}^{O} $	What was the average effect of ever launching?	Group effects $ \theta_{sel}(g) $, averaged over cohorts by size

The event-study aggregation is the proper target for the ramp-curve plot every marketer wants, without the contaminated-lag pathology of dynamic TWFE. It carries one caveat: as $ e $ grows, later cohorts drop out of the average, so the curve mixes true dynamics with compositional change. The balanced variant fixes the set of cohorts observed for at least $ e' $ periods — fewer groups, cleaner dynamics. For a single headline number, the recommended overall parameter averages each cohort's post-launch effect first and then averages across cohorts by size,

$$ \theta_{sel}^{O} \;=\; \sum_{g \in \mathcal{G}} \theta_{sel}(g)\, P\big(G = g \mid G \le \mathcal{T}\big), $$

the natural multi-period analogue of the classic ATT: the average effect experienced by markets that ever launched, without over-weighting the waves that happen to have been live longest. Because every scheme uses non-negative weights, the positive-effects-everywhere-but-negative-estimate pathology is ruled out by construction. These summaries genuinely differ from one another whenever effects are heterogeneous — they only coincide (with each other, and with TWFE) in the constant-effect special case.

Simultaneous Inference

The estimators are $ \sqrt{n} $-asymptotically normal with an influence function that accounts for estimating the nuisance models. Rather than plug-in standard errors, Callaway and Sant'Anna recommend a multiplier bootstrap: perturb each unit's estimated influence-function contribution by an i.i.d. mean-zero, unit-variance weight (Mammen, 1993) and recompute. No propensity model is re-fit per draw, every cohort is represented in every draw (an empirical bootstrap can resample a small cohort away entirely), clustering amounts to drawing weights at the cluster level, and the whole procedure is fast — the paper's application, with a thousand bootstrap iterations, runs in about three seconds.

The reason to care is simultaneity. An event-study plot displays many estimates at once, and pointwise 95% intervals around each understate the joint uncertainty of the path — some interval will exclude its truth by chance alone. The multiplier bootstrap yields uniform confidence bands: a widened band, calibrated from the bootstrap distribution of the studentized maximum deviation, that covers the entire path of effects with the stated probability. Pre-treatment placebo effects — which should be zero under the assumptions — can be estimated alongside and plotted inside the same band, giving a multiple-testing-honest read on parallel trends.

The stakes are not hypothetical. In the paper's minimum-wage application — staggered state adoption, dynamic effects, cohort heterogeneity, all the complications a marketing rollout shares — the TWFE post-treatment coefficient is a statistically insignificant $-0.008$, while the heterogeneity-robust overall effect under conditional parallel trends is about $-3.1\%$, with event-study effects growing from roughly $-2.4\%$ on impact to $-7.1\%$ at three years of exposure. The method choice changes the qualitative conclusion, not just the decimal places.

Reading a Rollout in Practice

For a measurement team the recipe is concrete. Encode each market's launch date as its group; decide from context whether never-launched markets are credible controls or whether not-yet-launched markets must serve; pick the pre-launch covariates that make within-stratum parallel trends defensible; estimate the $ \mathrm{ATT}(g,t) $ family doubly robustly; and report the event-study band plus the overall $ \theta_{sel}^{O} $ with uniform intervals (the reference implementation is the R package did). The output slots naturally into a broader measurement program: a staggered-rollout readout is exactly the kind of quasi-experimental evidence that can calibrate an MMM's channel priors, and its assumptions deserve the same explicit pre-registration we argue for in stating identification assumptions generally.

Takeaways

A market-by-market rollout is a staggered adoption design. TWFE's single coefficient averages cohort-period effects with estimator-chosen, possibly negative weights — it can report a negative number when every market benefited.
The fix is to disaggregate first: $ \mathrm{ATT}(g,t) $, the effect for the cohort launched at $ g $ measured at $ t $, is causally interpretable under any pattern of heterogeneity.
Identification needs limited anticipation, conditional parallel trends against a deliberately chosen comparison group (never-treated or not-yet-treated), and overlap — choose by context, not pre-tests.
Estimate doubly robustly: consistency survives one misspecified nuisance model. Aggregate with transparent non-negative weights matched to the question — ramp curve, wave report card, or one headline ATT.
Use multiplier-bootstrap uniform bands so the event-study plot and its pre-trend placebos are honest about multiple comparisons.

🧰 This method ships in the framework

Matched-market difference-in-differences is a first-class named method in the experiment toolkit (did_mmt in the planning.methods registry), with residual-matched pairs, cluster-robust per-pair estimation, placebo-calibrated standard errors, and empirical power from the A/A·A/B simulation harness. It sits alongside synthetic control, TBR, and GBR in the Design Studio's method picker and the methodology leaderboard.

References

Callaway, B., & Sant'Anna, P. H. C. (2021). Difference-in-Differences with Multiple Time Periods. Journal of Econometrics, 225(2).
Goodman-Bacon, A. (2021). Difference-in-Differences with Variation in Treatment Timing. Journal of Econometrics, 225(2).
Sun, L., & Abraham, S. (2021). Estimating Dynamic Treatment Effects in Event Studies with Heterogeneous Treatment Effects. Journal of Econometrics, 225(2).
de Chaisemartin, C., & D'Haultfœuille, X. (2020). Two-Way Fixed Effects Estimators with Heterogeneous Treatment Effects. American Economic Review, 110(9).
Sant'Anna, P. H. C., & Zhao, J. (2020). Doubly Robust Difference-in-Differences Estimators. Journal of Econometrics, 219(1).
Abadie, A. (2005). Semiparametric Difference-in-Differences Estimators. Review of Economic Studies, 72(1).
Heckman, J. J., Ichimura, H., & Todd, P. E. (1997). Matching as an Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme. Review of Economic Studies, 64(4).
Card, D., & Krueger, A. B. (1994). Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania. American Economic Review, 84(4).
Roth, J. (2022). Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends. American Economic Review: Insights, 4(3).
Mammen, E. (1993). Bootstrap and Wild Bootstrap for High Dimensional Linear Models. Annals of Statistics, 21(1).
Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press.
Vincent, B. T. (2022). Difference in Differences. PyMC Examples.

Aggregation	Question it answers	Form
Event study \( \theta_{es}(e) \)	What is the average effect \( e \) periods after launch?	Average of \( \mathrm{ATT}(g, g+e) \) over cohorts observed at exposure \( e \)
Group \( \theta_{sel}(\tilde g) \)	How did the \( \tilde g \) launch wave do overall?	Mean of \( \mathrm{ATT}(\tilde g, t) \) over its post periods
Calendar \( \theta_c(\tilde t) \)	What was the effect in period \( \tilde t \) among live markets?	Cohort-share-weighted \( \mathrm{ATT}(g, \tilde t) \) for \( g \le \tilde t \)
Overall \( \theta_{sel}^{O} \)	What was the average effect of ever launching?	Group effects \( \theta_{sel}(g) \), averaged over cohorts by size