Stress Test 02: Time Structure

Three worlds whose time structure lies outside the model's hypothesis space — a structural break the brand reacted to, seasonality no low-order Fourier basis can say, and effectiveness that drifts under a constant coefficient. In every act the un-representable structure has to land somewhere, and it lands on media exactly when spend correlates with it. The punchline: under the default linear trend the trend-break world reads TV at −63% of its true contribution; the holiday world scrambles the channel split (TV +33%, Display −70% when first measured) while the total stays roughly right — and convergence diagnostics were green in every single fit.

This guide mirrors nbs/stress/stress_02_time_structure.ipynb. The series doctrine is set in stress 00: green diagnostics do not mean correct attribution. This is the time-structure version of that failure. The model's default spec is a linear trend, order-2 yearly Fourier seasonality, and constant positive betas; each act breaks exactly one of those, and we grade every estimate against known causal ground truth from the scenario harness in tests/synth/.

Act 1 — A structural break the brand reacted to

Mid-series the category takes a level shock — think COVID, a distribution loss, a PR crisis: the baseline drops ~140 KPI units overnight, then claws most of it back along a new recovery slope (+1.45/week). And the brand does what real brands do in a crisis: it ramps TV and Display roughly 60% from the break onward to "win back" demand. So the spend step and the baseline shock are the same event on the calendar.

The model's default trend is a single straight line. A line cannot drop 140 units in one week, and whatever part of the shock-and-recovery the line cannot carry is available to whichever regressor moves at the break — which is exactly the ramped media.

Illustrative simulation in your browser (seeded) recreating the trend_break world's shape: sales with the true baseline's level shock and recovery slope (top), and TV/Display spend stepping up ~60% at exactly the break week (bottom). Measured in the notebook (seeded run): under the naive linear trend TV comes out around −63% vs its true contribution with the truth outside its 90% interval; Display is similarly crushed; the two non-ramped channels are recovered fine (Search within ~15%). The fitted straight line absorbed little of the shock — even the best possible straight line through the true trend-plus-shock captures barely a quarter of its swing.

Note the direction surprised the scenario's own designer. It was designed expecting the post-break recovery to be mis-credited to the ramped channels. The measured failure is the mirror image: the level shock dominates the post-break window, so the model sees 60% more TV/Display buying visibly fewer sales — and concludes the ramped channels barely work. Same disease either way: spend moved in lockstep with unmodeled baseline structure, and the structure was billed to media. Had the brand cut spend in the crisis, the bias would flip to over-credit.

Act 1 pivots — bend the trend, or name the event

Two families of fix. The first is a trend that can bend: TrendType.PIECEWISE, the Prophet-style slope-change trend. A continuous piecewise-linear function can never jump, but it can fake a cliff as two violent slope changes between adjacent changepoints — if the prior lets it dive. When this notebook was first authored, the default changepoint prior did not.

Illustrative simulation in your browser — stylized fitted-trend shapes against the true trend-plus-shock. Measured in the notebook (seeded run): at the old default changepoint prior (Laplace scale 0.05) the piecewise trend's correlation with the true baseline path is 0.33 — no better than the line, because a Laplace(0, 0.05) prior on slope changes refuses to pay for a 140-unit cliff. At the current default (0.5) the correlation is 0.86: the trend visibly tracks the shock-and-recovery, median channel error drops below 20%, and TV claws back a large part of its credit (its error improves by 20+ points).

The second family is more surgical: if you can date the break (you usually can — it was a crisis), hand the model the event as control columns rather than buying a more flexible trend. A step dummy \( \mathbf{1}[t \ge t_0] \) represents the level shift; adding a ramp \( (t - t_0)_+ \) also represents the new recovery slope — everything you know about the shock's shape, in two columns.

Measured pivot summary (seeded notebook run):

One honest residual: TV stays under-credited by roughly a fifth to a third in every pivot. The flexible trends widen its interval enough to cover the truth; the tighter event controls can leave it just outside. That is the identification limit, not a spec bug — the brand changed TV at exactly the week the world changed, so the data cannot fully split "ramped TV" from "recovering baseline". Only an experiment fully separates them (see stress 03 for how lift-test calibration closes exactly this kind of gap).

Act 2 — Seasonality the Fourier basis cannot say

Three departures from the model's order-2 yearly Fourier \( \sum_{k=1}^{2} a_k \sin\tfrac{2\pi k t}{52} + b_k \cos\tfrac{2\pi k t}{52} \), all common in retail: the seasonal amplitude grows ~60% across the three years (the brand scales); there are sharp holiday spikes — weeks 47–48 (Black Friday/Cyber) and 50–51 (Christmas), +160 KPI each, far too sharp for any low-order Fourier; and Social's flighting is seasonal — its budget runs 3× in Q4, sitting directly on the spikes.

Illustrative simulation in your browser — the world's true seasonal component (growing amplitude + spikes) against the best possible order-2 Fourier approximation, computed by least squares right here in the page. The shaded gap is baseline KPI with nowhere to live: a 1-week spike needs harmonics near order ~26, and the amplitude growth is not periodic at all. In the notebook that unrepresentable gap sums to a meaningful fraction of all media contribution — and it has to be billed to someone.

Measured — and another design surprise. The world was built so Social would harvest the holiday lift. The split got scrambled instead of the targeted channel inflated: when first measured (under the framework's old, stiffer trend priors) the same world read TV +33% and Display −70%; re-run at the current defaults the victim shifted — Display loses roughly two-thirds of its contribution, Social comes out modestly under, TV lands close. In both runs total media stays within ~15% of truth and convergence is green. The lesson is sharper than the designed one: you cannot predict which channel pays for misspecified seasonality — the misfit finds the accounting-cheapest victim, and the totals stay plausible throughout.

The pivot: events are controls, not seasonality

A holiday spike is not "seasonality" in the Fourier sense — it is a dated, recurring event, and it belongs in the design matrix as a control column, exactly like Act 1's step dummy. The tempting alternative — crank the Fourier order to 6 and stay inside the seasonality block — is measured too.

Illustrative simulation in your browser — residuals (actual − fitted) by calendar week for the static-Fourier fit vs the holiday-dummy pivot; holiday weeks marked. Measured in the notebook (seeded run): the holiday control is the clear winner — median channel error drops to ~9% and Social lands almost exactly on its truth. Raising the Fourier order to 6 barely dents the damage: it can sketch a Q4 hump but not 1-week spikes, the amplitude growth stays unrepresentable, and Display remains more than 35% under truth. (The feared "flexible season eats media" backfire did not materialize on totals here — order-6 simply fails to fix the problem.) One blemish even in the winning pivot: with the spikes controlled, Display still ends about a quarter under truth — the amplitude growth is still unmodeled and still has to land somewhere.

Act 3 — Effectiveness that drifts under a constant beta

In the time_varying_beta world, TV's effectiveness fatigues from 1.4× to 0.6× over three years (creative wear-out); Search more than doubles at a mid-series break (0.6× → 1.5×, an algorithm/creative refresh); Social wobbles seasonally; Display is constant. The model fits one constant beta per channel.

The recorded matrix grades this world recovers — median error ~13%, full coverage — at the harness's fitting budget. The notebook's faster budget tells a less comfortable story: TV's 3-year total comes out ~40% under-credited and outside its 90% interval. A verdict that flips between "green" and "badly wrong" with the fitting budget is itself a finding — totals-level green under drifting effectiveness is fragile, not robust. And the deeper point survives in Search, which is green on totals in both configurations — and still gives you the wrong forward-looking answer.

The diagnostic that is also the pivot: split-window refits

You will never have the true effectiveness curves. What you can always do: fit the halves separately and compare. If the per-window betas agree with the full fit, constancy was fine; if they diverge, the full fit is an average over different regimes — and your next-quarter decision should not be based on it. Try it: the slider sets how hard effectiveness drifts, and the page refits all three windows by least squares on each move.

Illustrative simulation in your browser — true effectiveness slides from (1+d)× down to (1−d)× while the page refits a constant-coefficient model on the full window and on each half. The full-window estimate sits between the two half-window estimates — it is a time-average over different regimes — yet the full fit's R² barely moves as you push the drift up. Measured in the notebook (seeded run): Search's split-window betas disagree by more than 1.7× (true effectiveness jump: 2.5×) while constant Display's halves agree, correctly clearing it; the full fit's late-window Search credit is 25%+ below truth (and its early-window TV slice 35%+ below) while the dedicated half fits land close. That disagreement is computable on any real dataset — no truth needed — and it is precisely the forward-looking error: a budget set from the full fit prices Search at a fraction of its current effectiveness and TV above its current, fatigued one.

On the scorecard

Both new worlds this notebook introduced to the harness are recorded in tests/synth/results/stress_matrix.md as SILENT FAILURE: trend_break at median error 27%, worst channel 41% (with 100% coverage — the wide intervals are doing honest work), and seasonality_misspec at median 37%, worst 104% (Social, with truth outside its interval). Both sampled with r-hat 1.02 and zero divergences. (time_varying_beta grades green at the harness budget — see Act 3 for why that verdict is fragile.)

Measured — per-channel relative error on total contribution from the recorded matrix (re-recorded 2026-06-10, harness fidelity: PyMC 500×500, parametric adstock, seeded). Trend break: TV −37%, Search −17%, Social +6%, Display −41%. Evolving seasonality: TV −17%, Search −24%, Social +104%, Display +50%. Notebook-fidelity runs (NumPyro 300×2) push the trend-break TV error to −63% — the magnitude moves with the fitting budget; the failure mode doesn't.

What to take away

Run it yourself