Real-Data Onboarding

Every other workflow on this site starts from a synthetic world with an answer key. Real engagements start from a CSV export with three date formats, a duplicated week, holes in the spend columns, a TOTAL row someone left in, and a negative number nobody can explain. This page follows that file through cleaning → MFF assembly → the pre-fit EDA gate → fitting → validation → report — and spends most of its time on the question the rest of the site postpones: which checks substitute for ground truth when none exists?

Measured (from the baked notebook nbs/demos/real_data_onboarding.ipynb) — every number on this page comes from that notebook's executed run, recorded in nbs/artifacts/real_data_onboarding.json. Nothing here is illustrative.

1 — The mess (what the client actually sent)

158 rows × 16 columns, and almost every classic defect at once. The first six rows, exactly as pandas reads them:

Measured (from the baked notebook nbs/demos/real_data_onboarding.ipynb) — the printed head of artifacts/client_export_messy.csv. Red cells are defects pandas has already swallowed silently: dates in three formats, TV as comma-separated text (and in dollars while every other channel is in $000s), a blank spend cell.

The full defect inventory, all of it visible in the file's dtypes and row count:

• Mixed date formats, rotating row by row: ISO (53 rows), US 01/11/2021 (52), month-name 2021-Jan-18 (52);
• a fully duplicated week (copy-paste artifact);
• ~3% missing spend cells in Search and Display, and Social logging dark weeks inconsistently as 0 or blank;
• TV as text with thousands separators, in dollars (all 156 cells);
• a trailing TOTAL summary row and an unnamed junk column (Unnamed: 15);
• inconsistent headers: 'TV ' (trailing space), 'search_spend', 'Social Spend', 'display', 'VIDEO'…;
• one negative spend value — a refund booked as negative exposure.

One decision happens before cleaning: the client also offered a brand-tracker awareness series, and we decline it. In this world brand_awareness is a post-treatment mediator (media → awareness → sales); conditioning on it blocks part of TV and Video's effect — the classic bad control. The walkthrough measures the damage of getting this wrong; here we just get it right at the door. What stays: two confounders (category_demand, distribution) and four precision controls (price, competitor_promo, weather, holiday).

2 — Triage & cleaning: every rule counted, every policy stated

Cleaning is modeling — each rule changes the data the model sees. So the notebook prints each fix with the number of cells it touched, and the run records the ledger:

Measured (from the baked notebook nbs/demos/real_data_onboarding.ipynb) — the cleaning ledger as recorded in real_data_onboarding.json. Exit asserts: 156 unique, consecutive weekly rows; no NaNs; no negative spend.

And because this world has a (disclosed) answer key, the notebook can do something a real engagement can't: price the cleaning itself. Comparing cleaned spend to the file's secret generating spend, the policies perturbed Search by 3.1% of its total, Display by 1.9%, Social by 0.9% — and every other channel by ~0%. Cleaning policies are small interventions with quantifiable cost; on real data you can't measure that cost, which is exactly why each policy gets logged rather than applied silently.

3 — Assemble the Master Flat File

The framework's loaders consume the MFF long format — one row per variable × period (× geography × product when they exist). mff_from_wide_format does the reshape; the head of the result:

      Period Geography Product Campaign Outlet Creative VariableName  VariableValue
0 2021-01-04                                                      TV      34.281720
1 2021-01-04                                                  Search      26.019952
2 2021-01-04                                                  Social       9.935614
3 2021-01-04                                                 Display       0.000000
4 2021-01-04                                                   Video      57.246752

The MFFConfig declares what each variable is — including each control's causal role, written down before the first fit:

mff_config = MFFConfig(
    kpi=KPIConfig(name="Sales", dimensions=[DimensionType.PERIOD]),
    media_channels=[MediaChannelConfig(name=c, dimensions=[DimensionType.PERIOD])
                    for c in CHANNELS],
    controls=[ControlVariableConfig(
        name=c, dimensions=[DimensionType.PERIOD],
        causal_role=ROLES.get(c, CausalControlRole.PRECISION_CONTROL))  # confounders: wide, un-shrunk priors
        for c in CONTROLS],
)
panel = load_mff(mff_df, mff_config)   # -> PanelDataset(156 obs, 7 channels, 6 controls)

Measured (from the baked notebook nbs/demos/real_data_onboarding.ipynb) — the reshape is lossless: 14 variables × 156 weeks = 2,184 MFF rows; the panel sees 7 channels + 6 controls + the KPI (mean 814, sd 137). category_demand and distribution carry CausalControlRole.CONFOUNDER, which routes them to wide, un-shrunk coefficient priors — shrinking a confounder re-introduces the bias it exists to remove (why).

4 — The EDA gate: what it flagged, and what acting on it means

mmm_framework.eda is the pre-fit quality gate — validators, missingness, collinearity, and robust outlier detection, all run before BayesianMMM ever sees the data. On the cleaned panel it returned 0 errors and exactly the flags this data deserves:

Measured (from the baked notebook nbs/demos/real_data_onboarding.ipynb) — validate_dataset / missingness_matrix / collinearity_analysis / detect_outliers on the assembled MFF. A gate you don't act on is decoration; each row's third column is the acting.

5 — The fit

The pre-specified configuration used across the series: parametric geometric adstock, logistic saturation, yearly seasonality, linear trend, NumPyro NUTS — 4 chains × 500 draws after 500 tuning steps. Measured: r-hat max 1.006, 0 divergences, 20 s wall-clock. Convergence is the entry ticket for everything below, not evidence of correct attribution — the rosy-picture demo exists because green sampler diagnostics say nothing about whether the answer is right.

6 — What replaces the truth column

On synthetic worlds we grade against a known answer key. Client data has no answer key, so this is the battery that substitutes for one. None of these checks can prove the attribution is right — that's what experiments are for — but each catches a specific way of being wrong:

6a — Did the data teach us anything — and where does non-identification actually show?

Measured (from the baked notebook nbs/demos/real_data_onboarding.ipynb) — compute_parameter_learning (contraction/verdict) next to each channel's relative contribution-interval width.

Two verdicts, one of them counter-intuitive. First, the gate's short-history warning did not become prior-domination: 7/7 channel betas landed strong/moderate. Second — the instructive part — you might expect the collinear Radio/Print pair to show up here as prior-dominated betas. It doesn't. Their marginal posteriors contract respectably, because the likelihood does constrain each coefficient through the pair's shared calendar. Marginal learning cannot see a joint pathology. Where the non-identification does surface — with no truth column needed — is interval width: Radio and Print's 90% contribution intervals run 2.0–2.2× their point estimates, against 0.4–0.7× for the well-identified channels. The pair's split is unpinned, and the posterior says so in width. Reporting their point ROIs without those intervals would be the lie.

6b — Can it predict data it has never seen?

Rolling-origin backtest (protocol in depth): first training window 104 weeks, four refits at 13-week steps, 52 graded out-of-time weekly forecasts. This check needs no ground truth — it runs on any engagement.

Measured (from the baked notebook nbs/demos/real_data_onboarding.ipynb) — all four refits converged (r-hat ≤ 1.008, 0 divergences). MASE < 1 means the model beats "copy last year" where that baseline is computable; the 80% interval covering 78.8% of 52 held-out weeks is calibration checked where it counts, out of time. Caveat from the doctrine: forecast skill validates the predictive model, not the causal one — the confounded worlds in stress 03 forecast fine while attributing wrongly.

6c — Does the story survive fitting on less data?

Refit the identical specification on the first 104 weeks only and compare channel shares of total media contribution (graded over the same window) against the full-156-week fit. Measured: rank correlation ρ = 0.89; the top-3 set kept 2 of 3 members (Video and TV held #1–#2 in both fits; Social and Search traded the #3 spot); Radio and Print stayed #6–#7 in both. If the ranking had reshuffled when a third of the data left, the "story" would have been a property of the sample, not the market. Note what honest expectations look like here: the identified channels hold steady, while the Radio/Print region — which §6a showed the data never pinned down — is where wobble is allowed and expected.

6d — What only an experiment can add

Everything above checks internal consistency and predictive validity. None of it can detect a confounder nobody measured: a model can learn sharply, forecast at 3% MAPE, and replicate across windows while attributing demand to the channel that merely chased it. The fix is randomized evidence, not more diagnostics: a geo-lift test on Radio vs Print to resolve the split observational data cannot; a holdout on TV or Search (the demand-chasing channels in this world) to bound confounding no within-sample check sees. Measured lifts feed back as coefficient-scale priors via mmm_framework.calibration — the calibration loop. On a real engagement this subsection is not optional reading; it is the plan.

7 — The reveal: the cell your real data will not have

Because the file is (disclosed) synthetic, the notebook ends §6 by opening the answer key and grading the entire pipeline — mess, cleaning policies, MFF, model — against each channel's true causal contribution:

Measured (from the baked notebook nbs/demos/real_data_onboarding.ipynb) — total media contribution 31,595 vs true 33,648 (−6.1%); the 90% intervals covered truth on 7/7 channels. The biggest point-estimate misses (Radio −38%, TV −17%) are exactly where the battery had already withdrawn confidence: Radio sits inside a 2.0× interval the model refused to narrow, and TV was the one channel §6a graded only moderate. Part of the residual error is the cleaning itself — §2 measured up to 3.1% of a channel's spend perturbed by the policies, and truth is computed on the undamaged spend.

On your real data you will not get this table. Sections 6a–6d are what you get instead — and on this run, their verdicts pointed at the same channels the answer key did.

8 — The deliverable

The run closes by rendering the client-facing artifact: MMMReportGenerator(model=mmm, panel=panel, results=results, config=ReportConfig(...)) .to_html(...) produced a self-contained 2.0 MB HTML report (artifacts/onboarding_report.html) — decomposition, ROI, response curves, diagnostics. It ships alongside the §6 battery, never instead of it: the report states the answers; the battery delimits which answers the report is allowed to state confidently. See the example report for what the rendered output looks like.

What to take away

• Cleaning is modeling. Missing-spend→0, zeroing the refund, the unit fix — each changed the data the model saw, so each was a stated, counted policy. The honest pipeline logs its interventions and (here, uniquely) measures their cost: ≤3.1% of any channel's spend.
• The EDA gate earns its keep before the first fit — and its flags map to actions: the obs/param warning became §6a's verification, the collinearity warning became §6a's width finding plus §6d's lift test, and the outlier detector audited the cleaning policy itself.
• No truth column? Run the battery. Learning, backtest, stability — none proves causality; each catches a specific way of being wrong; together they delimit what the report may claim.
• Know where non-identification shows. Not in the collinear pair's beta marginals (they contracted fine) — in contribution intervals 2× the estimate. Wide intervals are the correct answer; the lift test is the fix.
• Experiments are the plan, not an appendix. Only randomized evidence catches unmeasured confounding; the calibration loop turns measured lifts into priors for the next fit.
• The reveal is a luxury of synthetic data — disclosed, not hidden. Graded against the answer key: totals −6%, truth covered 7/7, and the misses concentrated exactly where the battery had already said "don't trust the point estimate."