ump_spec.rules.benchmark

module ump-spec

Benchmark pooling maths (spec §25.3 rule 4) and the benchmark parameters registry (spec §21).

Across clients, random effects on log iROAS: REML is the default tau-squared estimator and the CI on mu uses Hartung-Knapp-Sidik-Jonkman (t with k - 1 degrees of freedom, variance multiplier floored at 1). DerSimonian-Laird with a Wald CI MAY be declared instead. The 95% prediction interval for a new brand uses t with k - 2 degrees of freedom and tau floored at the registered value (0.15, proposed).

Also here, so core, the console and onboarding share one copy:

• rule 4a, non-positive rows: nonpositive_row, cell_construction (delta-method log transform only when every row’s 95% lower bound clears $0.10), to_analysis_scale and pool_on_construction (the reported-scale likelihood with a log link, pool_reported_scale);
• rule 7, egger_test (a flag, never a correction);
• §25.4 public intervals: band_floor_intervals (t on the band floor’s df, never k’s);
• §25.8 release: dp_band_floor and dp_noise_scale (scale from the band floor k_b, never the exact k), contributors_changed and refresh_allowed (item 4);
• §25.7 gap screen: gap_screen_value with sigma~ = sigma sqrt(r), and unit_normal_loss.

Pure Python (no numpy, no scipy): Student t quantiles come from the regularised incomplete beta.

Classes

PooledMean

class

class PooledMean

One cell’s random-effects summary on the analysis scale (log iROAS for benchmark cells).

Fields

Name	Type	Default	Description
k	int
mu	float
se	float
tau2	float
tau	float
i2	float
q	float
tau2_estimator	Tau2Estimator
ci_method	CiMethod
level	float
ci	tuple[float, float]
ci_df	int | None
hksj_multiplier	float | None
pi	tuple[float, float] | None
pi_df	int | None
tau_floor_applied	bool

Methods and properties (1)

init constructor

PooledMean(
    self,
    k: int,
    mu: float,
    se: float,
    tau2: float,
    tau: float,
    i2: float,
    q: float,
    tau2_estimator: Tau2Estimator,
    ci_method: CiMethod,
    level: float,
    ci: tuple[float, float],
    ci_df: int | None,
    hksj_multiplier: float | None,
    pi: tuple[float, float] | None,
    pi_df: int | None,
    tau_floor_applied: bool,
) -> None

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 161

Functions

benchmark_params

function

def benchmark_params() -> dict[str, Any]

{version, params: {name: {value, proposed?, status, spec_ref, description?}}} from registries/benchmark_params.yaml (version benchmark-params/0.1.0).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 38

benchmark_param

function

def benchmark_param(name: str) -> Any

One parameter’s value; raises KeyError for an unknown name.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 47

t_cdf

function

def t_cdf(t: float, df: float) -> float

Student t CDF.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 92

t_quantile

function

def t_quantile(p: float, df: float) -> float

Student t quantile by bisection on the CDF (absolute error below 1e-10).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 98

dl_tau2

function

def dl_tau2(y: Sequence[float], v: Sequence[float]) -> float

DerSimonian-Laird moment estimator.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 125

reml_tau2

function

def reml_tau2(
    y: Sequence[float],
    v: Sequence[float],
    *,
    tol: float = 1e-10,
    max_iter: int = 500,
) -> float

REML tau-squared by Fisher scoring from the DL start (truncated at 0), as in metafor.

With P = W - w w’ / sum(w): score step = (y’PPy - tr P) / tr(PP).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 136

reml_hksj

function

def reml_hksj(
    y: Sequence[float],
    se: Sequence[float],
    *,
    tau2_estimator: Tau2Estimator | None = None,
    ci_method: CiMethod | None = None,
    level: float | None = None,
) -> PooledMean

Pool rows y with standard errors se (spec §25.3 rule 4).

Defaults (from the registry): REML tau-squared, HKSJ CI with k - 1 df and the variance multiplier floored at 1, level 0.95. tau2_estimator="dl" with ci_method="wald" gives DerSimonian-Laird with a normal CI. i2 is tau2 / (tau2 + s2) with s2 the typical within-row variance (equal to the Q-based I-squared under DL). The prediction interval uses t with k - 2 df and tau floored at the registered tau_floor; it is None when k < 3.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 183

nonpositive_row

function

def nonpositive_row(value: float, se: float, scale: str) -> bool

Spec §25.3 rule 4a: a non-positive estimate, or a 95% lower bound at or below 0, on the row’s reported scale (note NONPOSITIVE_ESTIMATE). A row reported on log iROAS is positive by construction.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 265

cell_construction

function

def cell_construction(rows: Sequence[tuple[float, float, str]]) -> Construction

Which construction a cell uses (spec §25.3 rule 4a). rows are (value, se, scale) on each row’s reported scale. The delta-method log transform (log_delta) is used only when every row’s 95% lower bound is above the registered floor (log_transform_floor_iroas, $0.10, proposed); otherwise the cell is pooled with a likelihood on the reported iROAS scale and a log link on the cell mean (reported_scale_log_link).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 273

to_analysis_scale

function

def to_analysis_scale(
    value: float,
    se: float,
    scale: str,
    construction: str,
) -> tuple[float, float]

A row moved to its cell’s analysis scale: log iROAS under log_delta (delta method from the dollar scale), the dollar scale under reported_scale_log_link (delta method from log).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 284

floored_se

function

def floored_se(y: float, se: float, construction: str) -> float

The §25.3 rule 1 SE floor (se_floor_log, 0.05 on log iROAS, proposed) on the analysis scale. Under the log link the floor is carried to the dollar scale by the delta method (0.05 x |value|).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 299

pool_reported_scale

function

def pool_reported_scale(
    y: Sequence[float],
    se: Sequence[float],
    *,
    tau2_estimator: Tau2Estimator | None = None,
    ci_method: CiMethod | None = None,
    level: float | None = None,
    max_iter: int = 200,
    tol: float = 1e-10,
) -> PooledMean

Random effects with a likelihood on the reported (dollar) scale and a log link on the cell mean (spec §25.3 rule 4a): y_i ~ N(exp(theta_i), se_i^2), theta_i ~ N(mu, tau^2). Fitted by penalised quasi-likelihood: each step linearises the link at the current mu (working value mu + (y - e^mu) / e^mu, working SE se / e^mu) and runs reml_hksj on the working rows, until mu settles. Rows are never dropped, floored or clipped. The returned mu, CI and prediction interval are on log iROAS. Raises ValueError when the weighted mean of the rows is not positive, since a log-link cell mean then does not exist.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 308

pool_on_construction

function

def pool_on_construction(
    y: Sequence[float],
    se: Sequence[float],
    construction: str,
    **kw: Any,
) -> PooledMean

Pool contributors already on the analysis scale of construction (to_analysis_scale).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 340

egger_test

function

def egger_test(y: Sequence[float], se: Sequence[float]) -> tuple[bool, float | None]

Egger regression test for small-study asymmetry (spec §25.3 rule 7: audited, not corrected): regress y / se on 1 / se and test the intercept with t on k - 2 df at 5%. Returns (flag, p); (False, None) when k < 3.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 347

dp_band_floor

function

def dp_band_floor(k: int) -> int

k_b, the floor of the released contributor band (dp_band_floors, 5 and 10, proposed). The exact k is never reported. Raises ValueError below the lowest band, which is never released (G-BM-04).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 376

band_floor_intervals

function

def band_floor_intervals(mu: float, se: float, tau: float, k: int) -> dict[str, Any]

Public intervals of a cell (spec §25.4: the exact k never leaves the query layer). The HKSJ confidence interval uses t on (k_b - 1) df and the prediction interval t on (k_b - 2) df, with k_b the floor of the contributor band (dp_band_floor) and tau floored at tau_floor. With the k-based df, (ci95.hi - mu) / se would equal t(0.975, k - 1) and give k; with k_b <= k the intervals are wider, so the label is conservative. Raises ValueError below the lowest band (such a cell is suppressed and shows no interval).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 385

dp_noise_scale

function

def dp_noise_scale(k: int, epsilon: float) -> float

Laplace scale (clip_hi - clip_lo) / (k_b x epsilon) (spec §25.8 item 1). It bounds the sensitivity of the clipped mean, (clip_hi - clip_lo) / k, from above and does not reveal the exact k.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 401

contributors_changed

function

def contributors_changed(previous: Mapping[str, str], current: Mapping[str, str]) -> int

How many contributors changed between two releases of a cell: joined, left, or with a different contribution. Each map is {client_ref: digest of its contribution}.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 408

refresh_allowed

function

def refresh_allowed(changed: int) -> bool

Spec §25.8 item 4: a cell refreshes between releases only when at least refresh_min_changed_contributors (2, proposed) contributors changed. A heuristic against differencing, outside the epsilon guarantee.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 415

unit_normal_loss

function

def unit_normal_loss(z: float) -> float

L(z) = phi(z) - z (1 - Phi(z)).

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 424

gap_screen_value

function

def gap_screen_value(
    spend: float,
    m: float,
    sigma: float,
    h: float,
    r: float,
    c: float | None = None,
) -> float

Spec §25.7: value = c x spend x sigma~ x L(|m - h| / sigma~), sigma~ = sigma sqrt(r). c defaults to the registered gap_scale (0.3, proposed). Dollars of expected loss avoided per planning period; a screen, not EVOI. Zero when spend, sigma or r is not positive.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 430

pool_contributors

function

def pool_contributors(
    y: Sequence[float],
    se: Sequence[float],
    construction: str = 'log_delta',
) -> dict[str, Any]

One cell across contributors (spec §25.3 rule 4), the dict core and the console report: k, mu, se, tau, i2, q, ci95, pi95, pi_df, method, ci_method, construction, ci_df, hksj_multiplier, precision_weights, egger_flag, egger_p, tau_floor_applied`. `y` and `se` are on the analysis scale of `construction`; `mu and the intervals are on log iROAS. The prediction interval uses t on k - 2 df, or t on 1 df when k < 3 (very wide, the honest answer), with tau floored at tau_floor. A single contributor gets its own estimate with a Wald CI. precision_weights are 1 / (se^2 + tau^2) on log iROAS (log-link rows by the delta method at the pooled mean). Raises ValueError when no log-link cell mean exists.

Source: packages/ump-spec/src/ump_spec/rules/benchmark.py, line 441

Constants

Name	Type	Value	Description
Tau2Estimator		Literal['reml', 'dl']
CiMethod		Literal['hksj', 'wald']
Construction		Literal['log_delta', 'reported_scale_log_link']
RowScale		Literal['log', 'identity']