1 · What a benchmark is for

Three jobs, all of them narrow, and each of them a real answer to a question that comes up in the first week of an engagement.

  • Sanity-checking a new fit. A model reports a paid-search elasticity of 0.31 for a beverage brand. Nothing inside that model can tell you whether 0.31 is a plausible number or an artifact of a collinear control, because the model has exactly one brand to learn from. A vertical pool has dozens.
  • Onboarding. A new client with eighteen months of data and no experiment history has to start somewhere. Starting from what similar brands have measured is strictly better than starting from a default that a framework author picked once.
  • Vertical prior presets. The formal version of onboarding: the pooled population feeds the ROI-parameterized default media priors for the next model in that vertical. This is the most useful thing a benchmark can do and also the most dangerous, which is why it has its own gate (§4).

Now the failure mode. “Channel ROI” is not one quantity. Across contributors it varies by numerator basis (revenue, gross margin, LTV-adjusted value), denominator basis (working media or fully loaded), attribution scope, and whether it is an average over historical spend or a marginal at current spend. Those last two differ by the whole curvature of the response curve and routinely differ by 2–3×. Averaging them produces a number that is the answer to no question, and most real-world benchmark databases fail here first, silently.

The fix is boring and non-negotiable: a versioned catalog of quantities that are defined to be comparable across brands, and refusal to ingest anything that does not declare its basis. The catalog is short on purpose. 

from mmm_framework.benchmarks import CATALOG_VERSION, all_estimands

CATALOG_VERSION                             # 'bench-est-v1'
sorted(e.name for e in all_estimands())     # the seven below, and nothing else
EstimandWhat it isComes from
bench_elasticity % KPI per % spend at the brand’s operating point. Dimensionless and invariant to market size, currency and price — the flagship. model, experiment, calibrated model
bench_iroas Incremental ROAS from a randomized lift study, at a declared value basis, quoted at a reference operating point. experiment only
bench_mroi_at_half_sat Marginal ROI at 50% and 100% of the channel’s half-saturation spend — a fixed operating point, so brands compare like-for-like. model, experiment, calibrated model
bench_adstock_halflife Carryover half-life in weeks. Mechanics, not earnings. model
bench_half_sat_ratio Half-saturation spend over the brand’s median weekly channel spend — curve shape on a brand-relative scale. model
bench_efficiency_per_mille Margin-basis effect per 1,000 impressions, for impression-measured channels. Separates media performance from media price. model, experiment, calibrated model
bench_contribution_share % of KPI attributed to the channel. Flagged non-performance: it is a context descriptor, and readouts say so. model, experiment, calibrated model

Ratio quantities pool on the log scale, because brand effects are multiplicative and a symmetric-normal meta-model on the raw scale gets dragged around by the right tail. Cells never pool across catalog generations: a definition change bumps CATALOG_VERSION and the old contributions stay in their own generation rather than quietly merging with the new one.

The absence you should notice

Average ROI is not in that table, and its absence is deliberate rather than pending. §4 says why.

2 · Why k-anonymity is not privacy

The usual posture for a shared benchmark is a minimum cell size. Five contributors or more and the cell publishes; fewer and it is suppressed. It reads like privacy and it is not, and the argument against it is arithmetic rather than statistics.

Suppose the cell publishes an exact mean over five contributors. Four of them compare notes. Each knows its own contributed number, so the four know their own sum, and the cell has just told them the total. Subtraction returns the fifth brand’s number to machine precision. No inference, no approximation, no statistical assumption — the exact value, recoverable by anyone who can add. Raising k does not help: it just requires more colluders, and in a small vertical the other contributors are the handful of competitors most motivated to collude. 

import numpy as np
from mmm_framework.benchmarks import PrivacyPolicy

policy = PrivacyPolicy()      # k_min=5, epsilon_per_release=1.0, epsilon_year=12.0
per_contributor = np.array([0.42, 0.61, 0.55, 0.38, 0.70])   # clipped per-brand means
target, colluders = per_contributor[0], per_contributor[1:]
n = per_contributor.size

# An exact cell mean hands the held-out brand's number straight back.
exact_mean = float(per_contributor.mean())        # 0.532
n * exact_mean - colluders.sum()                  # 0.42 — exactly the target

# The same attack against the Laplace-released mean.
b = (policy.clip_width / n) / policy.epsilon_per_release   # sensitivity / epsilon
rng = np.random.default_rng(41)
replays = n * (exact_mean + rng.laplace(0.0, b, 4000)) - colluders.sum()

float(np.median(np.abs(replays - target)))        # 2.44 — typical recovery error
float(per_contributor.std())                      # 0.12 — between-brand spread

The last two lines are the whole point. Against the noised release, the insider’s recovery is off by twenty times the spread of the pool: the attack returns a number worse than guessing the pool average, so running it buys nothing. The Laplace noise is what makes the goal statement (“no client’s results are recoverable by another client”) true. The cell-size gate only decides which cells are worth releasing at all.

It is a test, not a design-doc paragraph

The insider attack runs in CI, on the real store and the real release path: tests/benchmarks/test_publish_priors.py::TestInsiderAttack asserts both halves — that n−1 colluders recover the target exactly from an exact mean (to 1e-9), and that against the DP release their median recovery error exceeds the between-contributor spread. A privacy claim nobody can execute is a claim about intentions. This one fails the build if it stops being true.

The cell-size gates still exist, and they do a different job — deciding whether a cell carries enough independent evidence to be worth spending privacy budget on. There are three, applied per (vertical × channel × estimand):

  • k — at least 5 distinct contributors, counted by salted hash rather than by record, so one contributor with six models does not look like six brands.
  • Spend dominance — no single contributor may account for more than 60% of the cell’s spend.
  • Precision dominance — no single contributor may hold more than 60% of the cell’s inverse-variance weight. This gate exists because the pool is precision-weighted: a contributor reporting a suspiciously tiny standard error would otherwise be the cell, and an SE floor is applied at the source for the same reason.

Two details of the client-facing surface follow from taking the insider seriously. The suppression reason a client sees is always the generic insufficient contributors, because a specific reason (“dominance 67%”) is itself a disclosure about the cell’s structure. And contributor counts publish as bands — 5–9, 10–19, 20+ — because exact counts across two vintages difference into membership facts.

3 · What crosses the boundary

Privacy is enforced at a single place, which is why the package is layered the way it is. contribute and ingest run inside the client’s project and see raw posteriors. store, meta_model and publish run in the central benchmark tenant. Only publish’s ledgered releases ever reach another client.

Project side: eligibility, coarsening, consent

A contribution is a summary, never a posterior: a mean, a standard error, five quantiles, and a coarsened covariate set. Covariates leave as bands and tiers, never as exact values, and the coarsening rules live in exactly one module so they cannot drift per caller. The contributor identity is a salted hash; the raw tenant id never leaves the project. 

import numpy as np
from mmm_framework.benchmarks import build_contribution, budget_band, contributor_hash

record = build_contribution(
    estimand="bench_elasticity",
    channel_code="paid_search",
    vertical="beverage",
    draws=np.random.default_rng(0).normal(-2.5, 0.3, 4000),   # log scale, per the catalog
    contributor=contributor_hash("tenant-42", salt="rotating-secret"),
    covariates={"budget_band": budget_band(38_000_000), "audience": "prospecting"},
    consent=True,      # no opt-in, no record — the default raises
)

record.mean, record.sd                    # -2.5, 0.3 — the summary, not the draws
record.covariates["budget_band"]          # '$10–100M' — a band, never $38,412,900
record.contributor_hash                   # '7bce25abb879f18c'

Consent is a required argument rather than a settings flag, and the record carries the consent version it was made under, so a later revocation can be honored retroactively by deleting on contributor hash. Two eligibility gates run before any of this, and both refuse loudly: 

from types import SimpleNamespace

from mmm_framework.benchmarks import EligibilityError, check_model_eligibility

try:
    check_model_eligibility(
        SimpleNamespace(approximate=True, converged=True, panel=None),
        channel="paid_search",
    )
except EligibilityError as err:
    print(err)
# approximate fit (MAP/ADVI/Pathfinder/Laplace): uncertainty is not calibrated,
# and the meta-model consumes the SE as data

try:
    check_model_eligibility(
        SimpleNamespace(approximate=False, converged=True, panel=None),
        channel="paid_search",
        contraction=0.04,     # prior -> posterior contraction for this channel
    )
except EligibilityError as err:
    print(err)
# channel 'paid_search' posterior is prior-dominated (contraction 0.04 < 0.1):
# contributing it would launder the default prior into the benchmark

An approximate fit is excluded because the meta-model treats the contributed SE as data, and a MAP or ADVI interval is not a calibrated statement about uncertainty. The contraction gate is the first half of the echo-chamber defense in §4: a channel whose posterior never moved off its prior would contribute the prior, which would then become the next default prior. There is also a minimum panel length, an identification floor rather than a privacy one.

Central side: clipped statistics under Laplace noise

Differential privacy is applied to clipped simple statistics and nothing else. Per (vertical × estimand) the release set is deliberately small and orthogonal:

  • a clipped population mean over per-contributor means (unweighted within a contributor and clipped to the policy range, so the sensitivity of the cell mean is exactly clip_width / n);
  • a clipped mean absolute deviation as the spread — a naive DP variance is hopeless at cell sizes of five to ten, while the MAD’s sensitivity survives;
  • clipped contrasts for declared binary splits, such as retargeting minus prospecting, or one channel minus the other when a group has exactly two.

Per-channel cell centers are then post-processing of the released mean and the released channel contrast. Post-processing is free under differential privacy, so publishing more cells never re-bills the same contributors for the same information — those cells are marked derived and enter the ledger at ε = 0. 

import numpy as np
from mmm_framework.benchmarks import (
    BenchmarkStore, build_contribution, contributor_hash, publish_vintage,
)

rng = np.random.default_rng(7)
store = BenchmarkStore()          # ':memory:' by default; the central tenant passes a path
for i in range(6):                # six brands, one vertical, one channel
    store.add_contribution(
        build_contribution(
            estimand="bench_elasticity",
            channel_code="paid_search",
            vertical="beverage",
            draws=rng.normal(-2.5 + 0.3 * rng.normal(), 0.3, 2000),
            contributor=contributor_hash(f"tenant-{i}", salt="rotating-secret"),
            covariates={"audience": "prospecting", "spend": 1.0},
            consent=True,
        )
    )

artifact = publish_vintage(store, vintage="2026Q3", rng=rng)
cell = artifact.cells[0]

cell.n_band, cell.center          # '5–9', -2.14   (the pool's true center is -2.5)
cell.spread, cell.noise_scale     # 0.05, 0.58     — the noise dwarfs the spread here
cell.epsilon_spent                # 2.0            — mean + MAD, one release each
store.ledger().worst_case("2026") # 2.0            — worst-case annual spend, any member

That published center sits 0.36 away from the pool it came from, which is the honest arithmetic of six contributors at ε = 1 and not a bug. The system’s answer to a cell this thin is the usability verdict in §5, not an extra decimal place.

The ε ledger and the annual cap

Every boundary crossing lands in one append-only ledger, and a contributor’s privacy loss is the sum of ε over every release it appears in. The annual cap (12.0 by default, twelve unit releases) is a hard gate, not a report: a release that would push any member past it is refused, and the cell freezes at its last published values with the staleness disclosed on the cell itself. Budget exhaustion surfacing as visible staleness is the system working rather than failing.

Refresh discipline follows from the same accounting. Republishing last vintage’s noised values is free, because only touching the data spends budget, so a quarter with no membership change costs nothing. The refresh trigger is therefore membership churn — metadata the store knows without reading a single value. A value-dependent trigger (“republish when the mean moved”) would leak through the decision to republish and would itself have to be accounted for, which is a sparse-vector problem and deliberately out of scope.

4 · Three refusals

Each of these is a capability the system could plausibly offer and does not. They are the design.

The meta-model posterior stays internal

The pooling engine is a Bayesian random-effects meta-analysis: contributed means with known SEs pooled hierarchically, between-contributor heterogeneity τ estimated per vertical, moderators for market structure and curve position, and a provenance bias term identified by the contributors who supply both a model read and an experimental read of the same channel. It is the best estimate in the building, and none of it is published.

The reason is the same one that motivates clipped statistics: the sensitivity of a partially pooled, precision-weighted posterior functional to one contributor’s data has no clean bound. You cannot say how much a single brand’s withdrawal would move a shrunken posterior mean without solving a much harder problem than the benchmark is worth, so a DP release of that posterior would be a number with a privacy claim nobody can defend. The posterior’s job is the internal view, the moderator science, and the shrinkage; publication happens only through publish.

Average ROI is absent from the catalog

It is the number every stakeholder asks for and it is not comparable across brands, however carefully it is pooled. Response curves saturate, so a brand’s measured ROI depends on where its budget puts it on its curve: the predictable artifact is that big spenders show lower ROI and a naive reader concludes big brands buy worse, when they are simply further along the same curve. Stack on top of that the basis problem from §1 — margin versus revenue, working versus loaded media, with or without halo — and an “industry average ROI of 3.2” is a summary of four confounded things at once.

What survives is quantities that are dimensionless (bench_elasticity), quantities that describe the curve itself (bench_half_sat_ratio, bench_adstock_halflife), or quantities evaluated at a standardized operating point (bench_mroi_at_half_sat). bench_iroas is in the catalog only because it is pooled with a log-spend moderator and its published center is quoted at a declared reference spend — raw average pooling of iROAS is banned; moderated reference-point iROAS is not.

The echo chamber is gated at both ends

The loop to fear is model → benchmark → prior → model. Models fitted under a default prior contribute to a pool; the pool sets the next default prior; the next generation of models is fitted under it and contributes again. After a few cycles a vertical has convinced itself of something no experiment ever measured, and the posteriors look tighter every year.

The upstream gate is the prior-contraction check in §3 — a prior-dominated channel cannot contribute. The downstream gates live in the prior-feedback stage:

  • Predictive width, never the SE of the mean. As n grows the standard error of the population mean shrinks toward zero, while the between-brand variance does not. A new client is a draw from the population, not a re-measurement of its mean, so the preset’s width is sqrt(spread² + noise²) with both the between-brand spread and the release noise included. This single confusion produces most overconfident benchmark applications.
  • Experimental anchoring. A preset may only tighten below a floor of 0.5 (pooled log scale) when the cell’s precision-weighted share of non-model evidence clears 0.3. A benchmark assembled purely from models is not permitted to sharpen the priors those models were fitted under.

The gate index

SurfaceRefuses whenInstead of
build_contribution no explicit consent; a finance-basis estimand with no declared value_basis; an estimand fed the wrong provenance pooling numbers whose dollar means different things
check_model_eligibility approximate fit, failed or unassessable convergence, panel below the identification floor, prior-dominated channel consuming an uncalibrated SE as data, or laundering a prior
gate_cell fewer than 5 contributors, or one contributor over 60% of spend or of precision weight publishing a cell that is really one brand
Client-facing suppression notice always — the specific reason is withheld disclosing cell structure through the error message
publish_vintage the release would push any member past εyear spending past the cap; the cell freezes and says so
meta_model asked for a client-visible number a DP release with no defensible sensitivity bound
Estimand catalog asked for average ROI, or any quantity without an operating point a benchmark that confounds margin, price, scale and curve position
vertical_prior_preset the cell is unanchored — too little experimental evidence tightening priors on the strength of models fitted under them
placement release noise exceeds the between-brand spread a point percentile that is noise theater

5 · Reading a benchmark

A published cell is a center plus a between-brand spread, and the spread is the headline rather than a caveat. “Centered at 0.20, brands genuinely range from −0.15 to 0.55” is the honest statement; the center on its own invites every reader to treat the population mean as their own expectation.

Placement is a range, with a verdict

The natural question is where does my brand sit? At small cells and tight privacy budgets, the release noise can exceed the between-brand spread entirely, and a point percentile would be an elaborate way of reporting the noise draw. placement reports the range across replicate releases instead, plus a usability verdict and the cell size a stable placement would need. 

from mmm_framework.benchmarks import n_band, placement
from mmm_framework.benchmarks.schema import PublishedCell

thin = PublishedCell(
    vertical="insurance", channel_code="meta", estimand="bench_iroas",
    n_band=n_band(7), center=0.20, spread=0.15, noise_scale=0.42, epsilon_spent=4.0,
)
placement(0.5, thin)
# {'p_low': 0.0, 'p_high': 1.0, 'usable': False, 'n_needed': 66, 'noise_sd': 0.59}

deep = PublishedCell(
    vertical="beverage", channel_code="meta", estimand="bench_iroas",
    n_band=n_band(14), center=0.20, spread=0.35, noise_scale=0.05, epsilon_spent=2.0,
)
placement(0.5, deep)
# {'p_low': 0.66, 'p_high': 0.89, 'usable': True, 'n_needed': 29, 'noise_sd': 0.07}

The thin cell returns the only true answer available: somewhere between the 0th and 100th percentile, usable=False, and a note that roughly 66 contributors would be needed before the placement stabilizes. The deeper cell places the brand in the 66th to 89th percentile and says so. Both answers cost nothing extra in privacy budget — the replicate analysis runs over hypothetical noise draws of an already-published cell, and post-processing is free.

Turning a cell into a prior

The last stage of the loop, with the §4 gate visible in the numbers: 

from mmm_framework.benchmarks import vertical_prior_preset

model_only = vertical_prior_preset(deep, vintage="2026Q3", experimental_share=0.05)
model_only.sd, model_only.anchored     # 0.5, False — floored, the pool is all models

anchored = vertical_prior_preset(deep, vintage="2026Q3", experimental_share=0.45)
anchored.sd, anchored.anchored         # 0.36, True — randomized evidence earns the width

Same cell, same center, two different widths. The cell built from models alone is held at the floor no matter how tight the pool looks, because a tight pool of models fitted under one prior is evidence about the prior. Once randomized evidence carries enough of the precision weight, the preset is allowed to narrow to its predictive width. Every preset records its experimental_share, its anchored flag and its source_vintage, so a prior can always be traced back to the evidence that justified it.

6 · Where it lives

In Augur the client-facing surface is Haruspex, at /benchmarks: read a vertical’s channel benchmark, place a brand against it, contribute a fitted model or an archived lift study, and audit what has been spent on your behalf. The REST surface behind it splits along the same boundary as the package.

EndpointDoes
GET /benchmarks The ops console, internal: per-cell contribution and gate status, the named forest per vertical, the full ε ledger with per-member annual accounting, publication history. Real suppression reasons appear here and only here.
GET /benchmarks/client-view One contributor’s view: their own studies, the published cells they can see, their placements, generic suppression notices.
POST /benchmarks/studies Ingest archived lift studies, normalized into the catalog.
POST /benchmarks/publish Publish a vintage: gate every cell, spend the budget, write the ledger.
DELETE /benchmarks/contributors/{hash} Honor a revocation by contributor hash.
GET /portfolio-benchmark The within-tenant Constellation view, comparing brands you already own. Admin-only since 1.5.0; non-admin sessions receive a redacted payload with exact cross-brand distributions and percentile ranks removed, backed by the published benchmark vintage instead.

A benchmark is a population claim

Everything on this page describes the pool, and applying a population number to one brand is the ecological fallacy with a dashboard around it. The pool is also not a random sample of its vertical: contributions come from brands that run sophisticated measurement, spend enough to justify it, and opted in. Channels that performed badly get defunded and stop generating data, so surviving observations skew toward the cases where the channel worked. Use a benchmark to notice that a number is surprising, then go measure your own.