Benchmark database

§25NormativeDraft 2.3Latest draft

The benchmark store is a cross-brand database built from UMP evidence rows. It pools only numbers that answer the same registered question. It feeds the Measurement Console: a portfolio overview of coverage and maturity, client detail, alerts, a benchmark explorer (vertical × channel, over time, peer placement, moderators, evidence intake, release and privacy, health), the calibration pipeline, and a plan builder that also issues starter priors for new clients. Weaker evidence enters wider and labelled. Evidence known to be wrong for the question is kept on file but never pooled.¶

25.1 Benchmark estimands

Two questions are registered first. Every row in the store is mapped to one or both, or kept on file.¶

The columns use the section 5 vocabulary.¶

Id	Intervention	Baseline	Horizon	Unit	Point rule	Interval	`schema_version`
`E-BM-AVG-ROI`¶	`remove_all` on the channel (canonical: against `current_plan`)	`current_plan`	`total_to_decay`	Net revenue per $	`estimator_point` for REML pooling, or `posterior_median` when a Bayesian hierarchical model is declared; `scale: log`, on the cell mean	`{method: confidence, level: 0.95}` (HKSJ) on the pooled mean, or `eti` 0.95 under a declared Bayesian model; prediction interval 0.95 for a new brand	Registry version of these definitions
`E-BM-MARG-ROI`¶	`add`, size relative +0.10, on the channel	`current_plan`	`total_to_decay`	Net revenue per $	As above	As above	As above

Reporting scale: log. The cell mean is estimated and reported on log iROAS. Contributing rows enter on their reported scale, per 25.3 rule 4a. point_rule carries scale: log in its {rule, scale} form (section 5), which is part of the estimand.¶

25.2 How each source enters

The admissibility class of each source type is the one in the section 9 table and the source type registry. This table only cites that class; where they differ, section 9 governs.¶

Source	What it reports	What UMP does	Enters as	Note
Meridian, PyMC-Marketing, in-house frameworks¶	ROI against zero spend, net, draws	Matches average ROI; re-scored at +10% per draw for marginal ROI	Brand estimate, latest refit only	`ESTIMAND_ALIGNED` for marginal
Robyn¶	ROAS on gross revenue, bootstrap interval	Gross to net with the registered fee rate; interval read as log-normal; marginal re-scored from exported replicates	Brand estimate	`UNCERTAINTY_IMPUTED`
Any MMM not yet calibrated¶	As above	SE widened ×1.5 (proposed)	Brand estimate, wider	`UNCALIBRATED_WIDENED`
Calibrated MMM¶	As above, with tests in its likelihood	Tests filed at or before the refit are marked absorbed	Brand estimate	`absorbed_by` on the test rows
Randomised geo test, Class A¶	Lift in the test window, adstock tail excised	Carryover extended with the brand’s adstock posterior; with no brand model, the vertical adstock prior with SE ×1.2 (proposed). Partial cuts need a brand curve	Brand estimate	`ESTIMAND_ALIGNED`
Matched-market DiD, TBR or synthetic control, Class C¶	Lift in the test window, adstock tail excised	Carryover handled as for a randomised geo test; without placebo-calibrated intervals, or with weak pre-period fit, the class C discount of section 9 applies	Brand estimate	`ESTIMAND_ALIGNED`
Platform conversion lift with a holdout, Class B¶	Platform-attributed conversions, 28 days	Revenue via the registered AOV; carryover from the prior; SE ×1.3 (proposed) design discount	Average ROI only	`ESTIMAND_ALIGNED`
Platform A/B test, Class D¶	ROAS gap between delivery-optimised arms	Excluded	On file	`EVIDENCE_EXCLUDED`
Vendor curve¶	ROI by spend level, baseline not stated	Attach only until it declares a baseline and passes outcomes analysis (`G-VEN-01`)	On file	`ESTIMAND_UNDECLARED`, `VENDOR_UNVALIDATED`
Audience model¶	Uplift by propensity segment	Shown beside the pooled view; no method reconciles the two	On file	None
Scenario planner¶	Forecasts and approved decisions	Stored so the readout can score the forecast	`decision_score`, never pooled	None

25.3 Pooling rules

One estimate per client, channel and window. The window is the four quarters to the vintage date. Only the latest MMM refit counts, because refits share data. Independent tests and lift studies filed after that refit combine with it by inverse variance. Each item’s SE is multiplied by 1.15^q, where q is its age in quarters (proposed), after the source multipliers of 25.2. All adjustments are multiplicative, and the result is floored at 0.05 on log iROAS (proposed). Each multiplier applied is recorded on the row. Older evidence is down-weighted rather than dropped.
¶
One spend scale, pooled response curves. Spend is measured as intensity: channel spend as a share of the client’s revenue, so clients of very different sizes sit on one axis. Every MMM exports its channel response curve through UMP as joint draws of half-saturation, shape and level, and must declare the revenue (or other scale) it was fitted against; a curve with no declared scale cannot be pooled. Each cell pools half-saturation, relative to the cell’s median intensity, and shape across client curves with random effects, borrowing from all verticals when a cell has fewer than three. The curve’s level is anchored to the cell’s pooled iROAS at the median intensity. Every number is moved along the pooled curve to the median intensity before pooling, and the benchmark can be read at any budget level (25th, 50th and 75th percentile intensity, or a multiple of the median), for average iROAS and marginal ROI alike.
¶
Quoted at a reference operating point. Each vintage fits a weighted meta-regression on the latest evidence, with vertical × channel means held fixed and moderators for retargeting share above 40%, spend intensity, peak weeks, platform self-measurement and creative refresh. Items are adjusted to the reference point (below 40% retargeting, non-peak, independent) before pooling; spend intensity is handled by the pooled curves in rule 2 rather than by the regression’s average slope. The same regression is run on numbers as reported, with estimand differences (window, revenue basis, carryover) as extra terms; they should vanish after alignment, which is the check that the converters work.
¶
Across clients, random effects on log iROAS. REML is the default τ² estimator. The CI on μ uses Hartung–Knapp–Sidik–Jonkman (t with k − 1 degrees of freedom, variance multiplier floored at 1). DerSimonian–Laird with a Wald CI, or a Bayesian hierarchical model, MAY be declared on the cell. Each cell reports k, n, μ, SE, τ, I², the estimator and CI method, the 95% CI, and the 95% prediction interval using t with k − 2 degrees of freedom. Predictive uses floor τ at 0.15 (proposed), because τ̂ can be zero with few clients.

4a. Non-positive rows. Rows with non-positive estimates, or a lower bound at or below 0, are kept, never dropped, floored or clipped, and carry note NONPOSITIVE_ESTIMATE. A cell containing any such row is pooled with a likelihood on the reported iROAS scale (each row normal with its reported SE) and a log link on the cell mean, fitted by REML or the declared Bayesian model. The delta-method log transform is used only when every row’s 95% lower bound is above the registered floor (proposed $0.10 iROAS). The cell records which construction was used.
¶
Client estimates shrink toward the cell. Client-level views combine the client’s own estimate with the cell’s predictive distribution, shifted to the client’s moderators and spend intensity, so a stale or uncalibrated model is never less certain than having no model at all.
¶
Vintages are immutable. A vintage uses only rows the store held at its date and is sealed in the Merkle log (section 18). A late study enters the next vintage.
¶
Publication bias is audited, not corrected. Each vintage runs an Egger check on test residuals and carries a flag when asymmetric.
¶
Weights follow precision. Whether spend should also weight a client is open (25.9).
¶

All numeric constants in section 25 are proposed registry parameters (benchmark-params/0.x). Conformance tests read them from the registry, so a calibrated value is a registry version bump, not a spec edit.¶

25.4 Gates and fallbacks

These use the outcomes in section 15. None adds a hard stop. Brand-level privacy is enforced by withholding the cell, which no setting can override, and the view falls back to the parent cell.¶

Gate	Checks	Outcome	What happens instead	Note
`G-BM-01`¶	The row maps to a registered benchmark estimand	Fall back	Kept on file as attach-only	`ESTIMAND_UNDECLARED`
`G-BM-02`¶	Admissibility class allows pooling	Exclude and continue	Class D kept on file and counted	`EVIDENCE_EXCLUDED`
`G-BM-03`¶	The model is calibrated on this channel	Fall back	SE widened ×1.5 (proposed)	`UNCALIBRATED_WIDENED`
`G-BM-04`¶	Cell has at least 5 (proposed) contributing clients, none above 60% (proposed) of the cell’s spend, none above 60% (proposed) of the precision weight	Fall back	Cell held; view shows the parent cell (all verticals for that channel); the published reason is only “insufficient contributors”	`VERTICAL_SUPPRESSED`, `PARENT_CELL`
`G-BM-05`¶	The client has usable evidence on the channel	Fall back	The cell’s prediction interval, shifted to the client’s moderators, is used as a labelled best guess	`BEST_GUESS_VERTICAL`
`G-BM-06`¶	The row is a forecast, not a measurement	Fall back	Stored in `decision_score`	`ROUTED_TO_DECISION_SCORE`
`G-BM-07`¶	The study has an interval, a confidence level and a value basis	Exclude and continue	Held from the shared pool, kept on file with the reason; the client’s own model may still use it with an imputed interval	`HELD_FROM_POOL` (pool); `UNCERTAINTY_IMPUTED` (client model)
`G-BM-08`¶	Every contributor to a publishable cell has ε left in its annual budget	Fall back	Cell stays internal this year	`RELEASE_BUDGET_SPENT`
`G-BM-09`¶	A third-party row’s licence allows pooled use	Fall back	Kept out of every pooled cell; still feeds the licensing client’s own estimates, plans and gap list (25.10)	`LICENSE_CLIENT_ONLY`

G-BM-04 and G-BM-08 withhold the cell. Neither a reviewer, an account policy nor on_gate_fail can release it. The fallback output (the parent cell, or an internal-only view) is the only permitted release.¶

The note codes of this section (UNCALIBRATED_WIDENED, VERTICAL_SUPPRESSED, PARENT_CELL, BEST_GUESS_VERTICAL, HELD_FROM_POOL, RELEASE_BUDGET_SPENT, ROUTED_TO_DECISION_SCORE, NONPOSITIVE_ESTIMATE) are rows of the registry table in section 19.¶

25.5 Maturity roadmap and calibration pipeline

Each client sits on a five-stage maturity roadmap. Stages are recorded independently, so a client can be experimenting before it is calibrated.¶

Stage	Reached when
Foundations¶	Data feeds, taxonomy and consent in place; past studies filed through UMP
Model live¶	A mix model refit within the last 180 days
Experimenting¶	A causal test in at least three of the last four quarters
Calibrated¶	At least one channel calibrated on the client’s own tests in the latest refit
Reallocating¶	Budget moved on the calibrated numbers, with a decision record

Maturity score, 0 to 100 = 25 for a model refit within 90 days (12.5 within 180, 5 if older) + 25 × min(tests in the last four quarters / 4, 1) + 30 × share of channels calibrated + 20 × share of spend with a test or calibrated model in the last four quarters. The weights are placeholders for the measurement lead to set.¶

Work moves through a six-stage calibration pipeline: T0 Fit, T1 Prioritise (value-of-information grid), T2 Design, T3 Running, T4 Calibrate (fold readouts into the model), T5 Reallocate. A readout is banked when it has landed but no calibrated refit has absorbed it; banked readouts raise an alert, because absorbing them needs no fieldwork. Every readout is also scored against the 90% interval the recommending model quoted before the test; coverage well below 90% flags overconfidence across the practice.¶

25.6 Onboarding priors

A new client’s starter prior on E-BM-AVG-ROI is a robust mixture on log iROAS: the cell’s predictive distribution N(μ + moderator shift, τ² + SE²) with weight 0.8 (proposed), plus a wide component with SD 1.0 (proposed). The moderator shift places the prior at the client’s own spend level and retargeting share rather than the reference point. On a parent-cell fallback the weight drops to 0.6 (proposed). Each prior is stored as a prior_card with its source cell and vintage, and the plan builder shows them for any client without a model.¶

25.7 Gap screen

The gap screen ranks brand and channel pairs for the next measurement. It is a screen; the EVOI step in the measurement loop decides what runs.¶

\text{value} = c \times \text{spend} \times \tilde{\sigma} \times L\!\left(|m - h| / \tilde{\sigma}\right), \qquad \tilde{\sigma} = \sigma \sqrt{r}, \qquad L(z) = \varphi(z) - z\,(1 - \Phi(z))

Here m and σ are the client’s marginal ROI and its SD on the dollar scale, after shrinkage toward the cell (25.3, rule 5); h is the client’s hurdle, the spend-weighted marginal ROI of its other channels; r is the expected fraction of the prior variance of m that the next measurement removes, set by the channel’s evidence state: 0.55 (proposed) for a Class A test (0.45, proposed, Class B) where there is no causal evidence, 0.35 (proposed) where it is over a year old, 0.10 (proposed) where it is fresh; and c is the share of the channel’s spend that the decision moves, proposed 0.3. The value is in dollars of expected loss avoided per planning period. These defaults hold until the pilot calibrates them against realised interval shrinkage.¶

25.8 Privacy and release

Internal leadership views see exact cell estimates. Anything that crosses a client boundary is a release, and every release is noised and ledgered.¶

Mechanism. A released cell value is the mean of per-client log iROAS, each clipped to [−1.39, 2.08] ($0.25 to $8), plus Laplace noise with scale (2.08 + 1.39) / (k_b × ε). Contributor counts are released in bands (5–9, 10+). Neighbouring datasets differ by replacing one client’s contribution (bounded DP). The noise scale uses the floor of the released contributor band, k_b = 5 for 5 to 9 and k_b = 10 for 10+, so the scale is (2.08 + 1.39) / (k_b × ε) and does not reveal the exact k. This bounds the sensitivity of the clipped mean, 3.47/k, from above.¶
Budget. Each release costs every contributing client ε = 1.0 (proposed). A vertical’s mean, clipped dispersion and retargeting contrast cost 1.0 each. Each client has 6.0 ε a year (proposed), summed over every release it appears in. Budgets add under basic sequential composition.¶
Planner. Vertical statistics release first, then cells in order of contributor count, until a contributor’s budget would be exceeded; the rest stay internal (G-BM-08).¶
Refresh. A cell is refreshed between releases only if two or more contributors changed, so successive releases can’t be differenced to isolate one client. This rule is a heuristic against differencing; it is not part of the ε guarantee.¶
Stress tests. Every release is replayed against insider collusion, differencing across vintages and significance-gated archiving on synthetic data with known truth.¶

In the 5 to 9 band at ε = 1.0 (proposed), the Laplace scale is 0.69 and the noise SD 0.98 in log iROAS. In the 10+ band, they are 0.35 and 0.49. Both are larger than the between-client spread in most cells. A released cell is a direction rather than a number to plan on. Spending more budget on fewer cells, or releasing only vertical-level statistics, are the options under review (25.9).¶

25.9 Tables and open questions

Eight tables hold the store: evidence (with moderators and hold reasons), estimand_registry, client_maturity, vintage (with the moderator estimates), benchmark_cell (with spend and precision shares), release_ledger, prior_card and decision_score. Client names live only in the client-scoped table; pooled cells and ledgers carry the salted reference.¶

25.10 Metric families and third-party data

Some clients already buy third-party measurement, and not every useful result is a sales return. The store therefore holds several metric families. A family is a set of estimands that share a unit and may be pooled together. Results from different families are never pooled or converted into each other unless a registered link between them has been estimated for that client.¶

Family	Estimand	Unit	Pooled
Sales return¶	E-BM-AVG-ROI, E-BM-MARG-ROI	Incremental revenue per dollar	Yes, as in 25.2 to 25.5
Brand awareness¶	E-BM-BRAND-CPLP	Media cost per person lifted on aided awareness	Yes, in its own cells, same gates
Ad recall, consideration¶	E-BM-BRAND-CPLP-RECALL, -CONSID	Cost per person lifted on that metric	Held until each family has k ≥ 5 (proposed)
Delivery quality¶	None (descriptive)	Viewable share, attentive seconds, brand safety	No; shown beside, moderator candidate

Brand lift. Platform-run and independent panel studies (for example Kantar or Upwave) are converted to media cost per person lifted, from the study’s lift in percentage points, the exposed reach and the spend. Platform-run studies are self-measured, so they carry a registered offset (0.25 in log cost, placeholder until estimated from paired studies) and their SE is inflated by 1.3 (proposed). Lower cost per person lifted is better, so views reverse their shading. Brand trackers are filed as context and are not converted into either family.¶

Third-party sales return. A vendor MMM (for example Nielsen or Analytic Partners) is mapped from the vendor’s methodology note to an estimand at Tier C or D (section 14). Until it has been validated against the client’s own tests it carries VENDOR_UNVALIDATED and an SE inflation of 1.5 (proposed). Only one model per client and window enters the pool, ranked: a calibrated own model first, then a validated vendor model, then any other model. A second model of the same data is shown beside, never averaged in, because both would count the same sales twice. Panel sales lift studies (for example Circana or NielsenIQ) are Class C, matched exposed and unexposed households, unless households were randomised to exposure, which makes them Class A. Results are projected to the client’s total sales and converted to revenue at the registered price.¶

Licences. Every third-party row carries licence (pooled use, or client only) and feed_cost. Data licensed for one client feeds that client’s estimates, plans and gap list, and is excluded from every pooled cell by gate G-BM-09 LICENSE_CLIENT_ONLY (Fall back: the row stays in the client’s views and the ledger records why it is absent from the pool). Licence costs belong to the client and are passed through at cost.¶

Attention and verification. These describe how media was delivered, not what it caused. They are filed as delivery-quality rows and offered to the meta-regression (25.3, rule 3) as a moderator candidate once enough model and test pairs exist to estimate it.¶

The evidence table gains three fields to carry this: metric_family, licence and feed_cost.¶