Acceptance bar and vendor validation (closes gaps 3 and 4)
A model is certified only after it shows three things on pre-registered tests: that it computes correctly, that it recovers known causes when spend responds to demand, and that its recommendations beat simple alternatives. Vendor curves are checked against the adopting organisation’s own experiments; until they pass, they are used with a lower trust label and a wider range.¶
The Media Tracking Model is why the bar changed. It ran without errors, fitted well and produced tidy rankings. On simulated data with planted drivers, its top metric matched the truth 31 to 32% of the time, the same as drawing names from a hat. Simulation-based calibration (Talts et al., 2018) would not have caught that, because SBC asks whether the sampler recovers the prior, and MTM’s failure was that twelve segment models shared one aggregate weekly KPI with no segment-level variation to identify anything. That is an identification failure, so the bar now tests identification directly.¶
- 1 Computation Does the code do what it claims? Contract kit, convergence (R-hat, ESS), simulation-based calibration (SBC) with decision-relevant test quantities. Pass mark: all pass; R-hat < 1.01, bulk ESS ≥ 400 (proposed).
- 2 Identification Does it recover causes when spend chases demand (spend is endogenous)? A known-truth simulator with demand-driven spend, correlated channels, varied saturation and adstock. Pass mark: bias on mROAS within ±20% of truth (median across sims); 80% intervals cover within ±10 points of nominal (proposed).
- 3 Decisions Do its recommendations help? Regret of the recommended allocation versus the oracle; comparison with naive and random allocations. Pass mark: regret ≤ 25% of the oracle's gain over the current plan; beats naive with an interval excluding zero (proposed).
Running cleanly is not enough. The MTM pipeline ran without errors, fitted well and still matched the planted truth no better than chance. A model is certified only after it clears all three layers on pre-registered tests.
All numeric pass marks are proposed, calibrated in study 2 before they bind.
- Negative control: MTM pipelineMust fail.
- Positive control: correctly specified MMMMust pass.
A bar that fails the positive control or passes MTM is itself broken.
- Not yet testedUsable, labelled unvalidated, with reduced trust: a weaker best guess.
- Tested and failedExcluded from decisions. The step falls back to the next-best capability or library benchmarks (G-ACC-01).
Three layers of evidence¶
| Layer | Question | Tests | Pass mark |
|---|---|---|---|
| 1 Computation¶ | Does the code do what it claims? | Contract kit, convergence (R-hat, ESS), SBC with decision-relevant test quantities | All pass. On every parameter and derived quantity that enters a decision estimand: rank-normalised split R-hat (maximum of bulk and folded) < 1.01; bulk ESS >= 400 and tail ESS >= 400, from >= 4 chains; zero post-warmup divergences, or a recorded reparameterisation; E-BFMI >= 0.3. SBC: a pre-registered number of simulations (>= 200), ranks on the thinned draws, and an ECDF-difference uniformity test at alpha 0.01 per test quantity with Holm adjustment across quantities. Engines without MCMC replace R-hat and ESS with a bootstrap-stability check pre-registered in the battery, or record Layer 1 as not applicable, labelled. |
| 2 Identification¶ | Does it recover causes when spend is endogenous? | Known-truth simulator with demand-driven spend, correlated channels, varied saturation and adstock | Per channel with spend share >= 5%: the median over simulations of (estimate - truth) / max(|truth|, delta) is within +/-0.20, with delta registered (proposed 0.1 mROAS). 80% interval coverage is within +/-10 points of nominal, with N simulations chosen so the coverage MCSE is <= 2.5 points (N >= 256), and the MCSE is reported. |
| 3 Decisions¶ | Do its recommendations help? | Regret of the recommended allocation versus the oracle; comparison with naive and random allocations | Regret <= 25% of max(oracle gain over the current plan, the materiality threshold of G-MAT-01). The mean regret difference versus naive allocation is below zero, with a 95% paired bootstrap interval over simulations that excludes zero. |
N, delta, the spend-share cut and the bootstrap are pre-registered with the battery. All numeric marks in the table (the R-hat, ESS, chain, E-BFMI, SBC, bias, coverage, MCSE and regret values, the 5% spend-share cut and the 95% bootstrap level) are proposed. Bias is measured on the mROAS estimand the simulator plants as truth; regret is measured on the KPI estimand of the simulated decision.¶
All numeric marks are proposed and are calibrated in study 2 before they bind. Until then, the proposed values apply as published in the threshold set, a versioned machine-readable registry (section 21) that gate policies and the suite cite by version. Every certification, acceptance result and conformance claim MUST record the threshold-set version it was tested against. When a mark binds or changes, existing certificates keep their status until their next scheduled refresh or for 90 days (proposed), whichever is sooner. They are then re-run against the new set. A capability that fails the re-run moves to unvalidated, not to failed, unless the new evidence shows it failing the bar.¶
Null battery on the client’s own data. Shuffled target, noise regressors, placebo time shift, and a block-shuffled KPI whose block length is at least the channel’s adstock memory L (section 10). The real fit must stand above the 95th percentile of the null distribution for its top-k driver ranking.¶
Simulator requirements. Spend is set by a rule that reacts to forecast demand, seasonality and lagged KPI, following the 2026 endogenous-spend benchmark design; AMSS-style scenarios are included. The suite MUST include the MTM pipeline as a negative control that fails, and a correctly specified MMM as a positive control that passes. A bar that fails the positive control or passes MTM is itself broken.¶
Untested is not the same as failed. A capability that has not yet been through the bar is still usable, labelled unvalidated, with reduced trust; that is a weaker best guess. A capability that has been through the bar and failed it is excluded from decisions, because its output is known to be no better than chance, and the step falls back to the next-best capability or to library benchmarks. MTM sat in the first group until it was tested, then moved to the second. A claim of acceptance MUST link its acceptance report. A claimed acceptance status with no linked acceptance_report counts as untested under G-ACC-01 and carries ACCEPTANCE_UNSUBSTANTIATED.¶
Forking paths and multiple testing. The battery, thresholds and simulator seeds are pre-registered per family version before any candidate is tested. The bar is run on the whole pipeline including its model-selection rule, so a Robyn-style search over thousands of candidates is tested as one procedure, not as its winner. Null-test p-values across the battery are combined with Holm’s procedure at family-wise α = 0.05.¶
Vendor validation (outcomes analysis). Model risk guidance (SR 11-7, carried into SR 26-2) holds that vendor models need validation as much as in-house ones. A Tier C or X artifact reaches validated for decisions only when:¶
- the vendor has answered a conceptual-soundness questionnaire (method, estimand, data window, adstock treatment, how uncertainty was derived);¶
- at least two admissible tests (class A or B) on channels covering most of the account’s spend fall inside the table’s band, or within |z| ≤ 2 (proposed) of its prediction where no band exists, at the tested spend and estimand (proposed counts);¶
- realised versus predicted lift is monitored as each new test lands, and two consecutive misses (proposed) return the table to unvalidated.¶
An unvalidated table stays usable at every materiality tier, with trust unvalidated and its label shown. For a decision at the Full materiality tier the approver acknowledges that a key curve is unvalidated, and the band is widened from the test library when the vendor supplied none (G-VEN-01).¶