Rollout
This section is informative.¶
Nine studies from the research review, plus a benchmark pilot added in this draft, now sit inside the phase plan, and three of them must report before any client-facing use: the estimand round trip, the acceptance-bar benchmark and the agent evaluation harness.¶
Pre-rollout studies¶
| # | Study | Pass criterion | Blocks |
|---|---|---|---|
| 1¶ | Estimand round trip: Meridian, Robyn, PyMC-Marketing and in-house frameworks on shared simulated truth, exchanged through UMP | Declared-equivalent estimands agree within Monte Carlo error; every mismatch is caught, aligned and labelled, and each widened interval covers the truth whenever one of its registered alternatives holds | Exit of P0 |
| 2¶ | Acceptance-bar benchmark: endogenous-spend simulator plus AMSS-style scenarios, MTM as negative control | Positive control passes, MTM fails; pass marks calibrated and pre-registered | Certification (P3), and pilot status for any new family |
| 3¶ | Calibration bake-off: R1, R2 with interval lift, R4 reweighting, on known truth, including partial cuts and adstock tails | Coverage within 5 points of nominal; PSIS rule catches every collapse | Interval lift and R5 leaving pilot (P2) |
| 4¶ | Interval-lift write-up crediting PyMC-Marketing | Internal paper with signed, adstock-aware, two-arm results | Publication of the extension |
| 5¶ | Agent evaluation harness: golden set, citation scoring, injection red team | Section 17 pass marks | Newton read-only pilot (P1) and every Newton release |
| 6¶ | Historical placebo study: TBR, Trimmed Match and augmented SC on our own null windows | Measured false-positive rates and design effects replace defaults | Default design parameters (P2) |
| 7¶ | Decision-record back-test of past reallocations | Optimiser’s-curse shrinkage estimated by channel | Curse adjustment leaving provisional (P3) |
| 8¶ | Audit-log hardening: Merkle log with RFC 3161 and external anchoring, rotation and erasure drills | Tamper and forgery fixtures detected; erasure leaves proofs valid | Sealing decisions in production (P1) |
| 9¶ | Governance pilot metrics for one quarter | Cycle time, bypass rate and override frequency baselined | Scaling beyond the pilot account (P4) |
| 10¶ | Benchmark pilot: calibrates the section 25 constants (SE multipliers, ageing rate and floor, tau floor, contributor floor, prior weights, gap-screen r values, epsilon budget) | Each proposed constant replaced by a calibrated value or kept with a recorded reason | Benchmark values leaving proposed (P4) |
Phases and exit gates¶
- P0
- P1
- P2
- P3
- P4
- Not tied to a phase
Three studies must report before any client-facing use: the estimand round trip (1), the acceptance-bar benchmark (2) and the agent evaluation harness (5). They are marked client-facing above.
Phase dates overlap where workstreams run in parallel. A phase closes only when its gate evidence is filed as an artifact.
Phase dates overlap where workstreams run in parallel; a phase closes only when its gate evidence is filed as an artifact.¶
What we need. One to two platform engineers for the core and gates; data engineering time for intake parsers; in-house modelling time for interval lift and geo-set lift tests; a measurement lead to own the estimand registry and studies 1 to 3; an agent host steward; a key custodian; named reviewers and a planning champion on the pilot account.¶