Experiments (closes gap 8)
Every test is pre-registered, reported in one standard shape, classed by admissibility before it can touch a model, and kept up to date as it ages. Platform A/B tests stay out of incrementality calibration because they measure something else; the calibration goes ahead on the evidence that remains.¶
Test plan (locked before launch; an unlocked plan still runs, labelled, G-EXP-01). Question and decision it informs; primary metric and estimand; design (matched markets, randomised geos, user holdout, ghost ads); test and control groups with buffers; planned spend and arms; assurance (at least 80%, proposed floor) (the probability of meeting the plan’s recorded success criterion, with the design prior equal to the posterior of the model version the plan was sized from) with MDE from t-quantiles (two-sided alpha and power recorded on the plan; proposed defaults 0.10 and 0.8), plus the smallest effect of interest (SESOI); placebo calibration on null windows; carryover floor computed from the same posterior draw as the lift band (G-EXP-04); analysis method and decision rule; the model version it was sized from. Plans from Google’s geo planning tool import with tool, version and settings.¶
Admissibility classes. Braun and Schwartz (2025) and Burtch et al. (2025) show that platform A/B tools deliver each arm to a different, algorithmically chosen mix of users, so a “winning” ad confounds creative with targeting. UMP therefore classes every result before use (G-EXP-05; excluded evidence never stops the step):¶
| Class | Examples | May calibrate incrementality | Default trust |
|---|---|---|---|
| A: Randomised holdout we control¶ | Randomised geo test, user-level holdout with assignment log, ghost ads | Yes | Own-experimental |
| B: Platform holdout lift¶ | Meta Conversion Lift, Google Conversion Lift with a holdout | Yes, with method recorded and a power-prior discount | Platform-reported |
| C: Quasi-experimental¶ | Matched-market DiD, TBR, synthetic control | Yes; without placebo-calibrated intervals or with weak pre-period fit, it enters with a heavier discount and a wider interval | Own-experimental, discounted on weak fit |
| D: Platform A/B (divergent delivery)¶ | Creative or audience split tests in platform tools | No; library only, for within-platform creative comparison | Not applicable |
| E: Survey brand lift¶ | Platform brand lift studies | Brand family only, not sales MMM | Platform-reported |
- A Randomised holdout we control Calibrates incrementality: yes. Trust: own-experimental.
- B Platform holdout lift Calibrates: yes, with the method recorded and a power-prior discount. Trust: platform-reported.
- C Quasi-experimental Calibrates: yes. Weak pre-period fit or no placebo-calibrated intervals means a heavier discount and a wider interval.
- D Platform A/B (divergent delivery) Calibrates: no. Library only, for comparing creative within one platform.
- E Survey brand lift Brand family only, not sales MMM. Trust: platform-reported.
Every result is classed before it can touch a model. Classes A, B and C may move a model's estimate of what ads cause. Class D never does, and class E only informs brand models.
Excluded evidence never stops the work. It is set aside and the step carries on with what remains (G-EXP-05).
Platform A/B test (class D). The platform picks who sees each ad, so each arm reaches a different mix of people. A "winning" ad mixes up the creative with the targeting.
Randomised holdout (class A). One audience is split at random, so both groups hold the same mix. The only difference left is the ads.
Filled and hollow dots stand for two kinds of people. Illustrative.
This table is the single definition of admissibility classes. An implementation MUST take the admissibility class from this table; where another section or the source type registry differs, this table governs.¶
Spillover and SUTVA. Geo designs MUST record buffer zones or commuting-zone clustering, adjacency and any national-auction exposure (G-EXP-03). After readout, buffer geos are tested for a shift; a significant shift flags contamination. A cross-geo posterior correlation is kept as a diagnostic, not as the test.¶
Estimator choice. Chosen at design time by geo count and tail behaviour, and recorded:¶
- Few geo pairs with heavy tails: Trimmed Match.¶
- A single treated unit or region: augmented synthetic control, with placebo-in-space.¶
- Many geos with stable pre-periods: TBR or DiD with AR(1) design effects and placebo calibration (Bertrand, Duflo and Mullainathan, 2004).¶
- User-level: ITT, with CACE only when the exposure log supports it.¶
Test result. Summary; plan as locked; what actually ran (dates, delivered spend, disruptions); primary result with range as incremental KPI, lift and iROAS; checks (pre-period fit, balance, achieved power, spillover, sensitivity to analysis choices); where it applies (media, spend range, geos, period); how it was used. The core payload is modeling-api’s impact group (actual, counterfactual draws, effect, cumulative effect).¶
Outcomes. Outcomes are classified against the locked plan’s interval level and SESOI. A result is positive or negative when the interval excludes zero on that side. It is “no detectable effect” when the interval lies inside (-SESOI, +SESOI), and the record states the largest effect ruled out. Otherwise it is inconclusive. Nulls and negatives are stored exactly like positives. Cost-per-incremental-outcome ratios use Fieller intervals, and decisions near zero lift use net monetary benefit, so unstable ratios never drive a choice.¶
Ageing. Use-by dates come from the decay clock with a measured channel half-life; media-type defaults apply only at cold start. Large changes in spend level, creative, targeting, pricing or platform algorithm trigger review before the use-by date. Corrections publish a new version; every model calibrated with the old one and every scenario using those models is flagged. Nothing is deleted.¶
Library. One searchable table, with the required fields mapped onto test results (estimand, spend level, provenance tier, methodology, sample size, measurement date, schema version, validity flag).¶