Causal Inference 00: The Ladder of Evidence

Veranda Home sells patio furniture, planters, and garden tools across the country. Like every brand, it has a dashboard: weekly spend on four channels, weekly sales, three years of history. And like every brand, it keeps asking the dashboard a question the dashboard cannot answer: "what did each channel actually cause?" The dashboard shows what happened. Causation is a claim about what would have happened otherwise — a counterfactual — and no amount of staring at observed history settles a counterfactual by itself. This page mirrors nbs/causal/causal_00_the_ladder.ipynb, the opening notebook of an eleven-part series in which every world is synthetic, every world ships a sealed answer key, and every claim is graded against ground truth.

The series' conceit keeps everyone honest: the analyst never sees the answer key while modeling. The key — each channel's true causal contribution, computed from the data-generating process itself — is unsealed only to grade. The grading machinery is itself machine-verified on every run, and every headline number on this page comes from the notebook's recorded artifact, not from anyone's memory of it.

The dashboard says everything works

Here is the dashboard-level "evidence" from Veranda's three years: every channel's weekly spend is positively correlated with sales.

A dashboard reader — or a last-click report, or a correlation matrix — would call this marketing that works, with Search and Social the stars. Hold that ranking; the rest of the page dismantles it.

The naive read, graded against the sealed key

The simplest quantitative version of the dashboard's story: regress sales on spend (ordinary least squares, all four channels at once) and call each slope "the ROAS". This is what a spreadsheet, a BI-tool trendline, or a junior analyst's first model does. Then unseal the key. The world's true causal ROAS is the exact counterfactual quantity the CMO is asking about — zero out one channel's spend inside the data-generating process, sum the difference in response (carryover included), divide by dollars spent:

\[ \mathrm{ROAS}^{\text{true}}_c \;=\; \frac{\sum_t \bigl( y_t(\mathbf{s}) - y_t(\mathbf{s} \mid s_c = 0) \bigr)} {\textstyle\sum_t s_{c,t}} \]

Measured — nbs/artifacts/causal_00_naive_vs_truth.json. The two demand-chasing channels read high: Social's naive read is 1.8× its true ROAS and Search's is 1.26×. TV — which actually carries the largest true return per dollar of the four (0.68) — reads lowest (0.27, just 0.39× of truth). A budget reallocation taken from this table would move money in exactly the wrong direction.

Why: the hidden third cause

The world this history was drawn from (unobserved_confounding) has a latent demand signal — seasonal appetite for patio furniture, housing turnover, weather, consumer mood — that does two things at once: it lifts sales directly, and it lifts spend, because Veranda's budgeting chases demand. Search and Social budgets are turned up when the category heats up.

That double arrow is a back-door path: spend ← demand → sales. Correlation flows along it even when advertising does nothing. The channels that chase demand hardest (Search, Social) inherit the most borrowed correlation — which is exactly the pattern the naive read produced.

"Just build a better model" doesn't fix it

Maybe the naive read fails because OLS ignores adstock and saturation? The notebook uses the real thing: a Bayesian MMM with geometric carryover, saturating response curves, a trend, and full uncertainty — the same model family this whole framework is built on:

from mmm_framework.synth import build
from mmm_framework.model import BayesianMMM, TrendConfig, TrendType
from mmm_framework.config import InferenceMethod, ModelConfig

sc = build("unobserved_confounding", seed=1)   # the world, answer key sealed
panel = sc.panel()                             # what the analyst gets to see

model = BayesianMMM(
    panel,
    ModelConfig(inference_method=InferenceMethod.BAYESIAN_NUMPYRO,
                n_draws=500, n_tune=500, n_chains=2,
                use_parametric_adstock=True),
    TrendConfig(type=TrendType.LINEAR),
)
results = model.fit(random_seed=7)
contrib = model.compute_counterfactual_contributions(
    compute_uncertainty=True, hdi_prob=0.9)

Measured — the MMM's counterfactual contributions graded against the sealed key. The MMM is a far better model of the data — and still over-credits the demand-chasers: Search +43% and Social +38%, with the truth outside both channels' 90% intervals. The bias is in the world, not in the estimator.

Same dashboard, different world — the opposite failure

Here's the uncomfortable part. Draw a different hidden world — clean, where spend is set by a media calendar that ignores demand entirely, so there is no back-door at all — and put its dashboard next to Veranda's. The correlation tables are the same kind of table. Nothing on either dashboard announces which world generated it. Yet in the clean world the naive OLS read is biased in the opposite direction: it under-reads every channel, because straight lines can't represent carryover and saturation.

Measured — naive read ÷ true ROAS in both worlds. Confounded world: Social 1.80×, Search 1.26× (over-read), TV 0.39× (crushed). Clean world: every channel between 0.43× and 0.80× (uniformly under-read). Same method, same-looking data, opposite failure. The size and direction of your error is a property of the world — and the world is exactly what observational data won't show you.

The ladder of evidence

So: dashboards can't answer causal questions, and better curve-fitting doesn't rescue them. What does? Stronger evidence, rung by rung. Each rung of this series buys a specific, nameable thing — and has a specific, nameable failure mode that the next rung exists to fix.

Every notebook in the series keeps three habits. Sealed keys: the truth exists, is machine-checked, and is only unsealed to grade. Directional asserts: every headline claim executes as an assert — if the world or the framework drifts, the notebook fails to bake rather than silently telling a stale story. One brand, many worlds: Veranda Home's dashboard looks the same in every notebook; what changes is the hidden history — which is the whole point.