Skip to content
Universal Model ProtocolDocs

01 · Estimands and shown numbers

You will learn

  • Describe a number with its seven-field estimand
  • Make numbers that always carry an interval
  • Check whether two numbers can be combined (G-EST-02)
  • Add channels up honestly (G-DEP-01) and find your fallback rung

You need

  • Python 3.12 and uv
  • The notebooks project (babelfin comes with it)

Runs offline. Uses babelfin (import babelfin as fin), UMP’s analyst-facing API.

Northwind Outdoor’s media lead has two numbers on her desk the Thursday before budget lock. The agency’s MMM says social returns $1.40 per dollar. Last spring’s geo test says $0.95. Which is right?

Possibly both. The MMM number is the return on the last dollar over a year; the test number is the return from adding spend for six weeks. Before anyone averages them, UMP asks what each number means. That is the estimand, and every number UMP exchanges carries one.

This notebook shows how to:

  1. describe what a number means, starting from a registered question;
  2. make a number that always carries its interval;
  3. check whether two numbers can be combined (the G-EST-02 ladder);
  4. add channels up honestly (the G-DEP-01 envelope);
  5. find the fallback-ladder rung for what you have.

Values marked proposed are starting points pending the pre-rollout studies (spec §22).

In [1]:
import babelfin as fin
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd

1. What does the number mean?

Section titled “1. What does the number mean?”

Start from a registered estimand and change only what differs. "mroi" is the registry’s marginal ROI at the current plan; "iroas" is a geo test’s incremental ROAS. Aliases: roi, mroi, iroas, lift, or any registry id.

In [2]:
mmm_q = fin.estimand("mroi", channels="social", weeks=52, interval="hdi")
test_q = fin.estimand("iroas", channels="social", weeks=6, level=0.8)
mmm_q
Out [2]:
revenue per dollar, net from a small increase in spend (+10%), geo_set, versus the current plan, over 52 weeks, posterior median, 90% hdi
custom estimand
details
intervention{'type': 'marginal', 'size': 0.1, 'size_kind': 'relative', 'arms': [], 'population': 'geo_set'}
baselinecurrent_plan
horizon{'type': 'window', 'weeks': 52}
unit{'kpi': 'revenue', 'per': 'spend', 'cost_basis': 'net', 'currency': 'USD'}
point_ruleposterior_median
interval{'method': 'hdi', 'level': 0.9}
schema_versionest-registry/1.2.0
applies_to{'media_ids': ['social'], 'geo_ids': []}
metric_familysales_return
In [3]:
test_q
Out [3]:
revenue per dollar, net from adding spend, geo_set, versus the control counterfactual, over 6 weeks, estimator point, 80% confidence
custom estimand
details
intervention{'type': 'add', 'arms': ['treatment'], 'population': 'geo_set'}
baselinecontrol_counterfactual
horizon{'type': 'window', 'weeks': 6}
unit{'kpi': 'revenue', 'per': 'spend', 'cost_basis': 'net', 'currency': 'USD'}
point_ruleestimator_point
interval{'method': 'confidence', 'level': 0.8}
schema_versionest-registry/1.2.0
applies_to{'media_ids': ['social'], 'geo_ids': []}
metric_familysales_return

Changing a scope field (channels, geos, client) keeps the registry id. Changing one of the seven fields (here weeks and the interval) makes it a custom question, and the registry id is dropped so nobody mistakes it for the registered one.

In [4]:
print(fin.estimand("mroi", channels="social").registry_id, "|", mmm_q.registry_id)
Out [4]:
E-MMM-MROI | None

2. Numbers carry their interval

Section titled “2. Numbers carry their interval”

fin.number(value, (lo, hi), estimand). There is no way to make a bare point; in Jupyter the number renders as a card with its meaning one click away.

In [5]:
mmm = fin.number(1.40, (1.05, 1.82), mmm_q, label="Social mROI (MMM)")
test = fin.number(0.95, (0.61, 1.30), test_q, label="Social iROAS (geo test)")
mmm
Out [5]:
Social mROI (MMM): 1.40
likely between $1.05 and $1.82 · medium confidence
details
meansrevenue per dollar, net from a small increase in spend (+10%), geo_set, versus the current plan, over 52 weeks, posterior median, 90% hdi
interval90% hdi 1.05 to 1.82
trust level2
In [6]:
test
Out [6]:
Social iROAS (geo test): 0.95
likely between $0.61 and $1.30 · low confidence
details
meansrevenue per dollar, net from adding spend, geo_set, versus the control counterfactual, over 6 weeks, estimator point, 80% confidence
interval80% confidence 0.61 to 1.30
trust level2

A vendor PDF that gives “1.2” and nothing else gets an imputed band, labelled, with trust level 1.

In [7]:
fin.imputed_number(1.2, "roi", label="Vendor social ROI")
Out [7]:
Vendor social ROI: 1.20
likely between $0.60 and $1.80 · low confidence
details
meansrevenue per dollar, net from removing all spend, geo_set, versus zero spend, total including carryover, posterior median, 90% eti
interval90% imputed 0.60 to 1.80
trust level1
G-UNC-01 UNCERTAINTY_IMPUTEDtier-default band (proposed); interval imputed

3. Can they be combined?

Section titled “3. Can they be combined?”

fin.compare walks the ladder: match, converter, re-ask the model, assumed alignment with a widened interval, side by side. It never silently averages two different questions.

In [8]:
fin.compare(mmm, test)
Out [8]:
Combinable with a wider interval
differs on intervention, baseline, horizon, point_rule, interval
details
whymismatch limited to assumable fields
interval after0.61 to 1.82
field first second
intervention type marginal, size 0.1, size_kind relative, arms , population geo_set type add, arms treatment, population geo_set
baseline current_plan control_counterfactual
horizon type window, weeks 52 type window, weeks 6
point_rule posterior_median estimator_point
interval method hdi, level 0.9 method confidence, level 0.8
In [9]:
fin.compare(mmm, test).to_frame()
Out [9]:
field first second
0 intervention type marginal, size 0.1, size_kind relative, a... type add, arms treatment, population geo_set
1 baseline current_plan control_counterfactual
2 horizon type window, weeks 52 type window, weeks 6
3 point_rule posterior_median estimator_point
4 interval method hdi, level 0.9 method confidence, level 0.8

One pair is never assumed aligned: an average ROI against a last-dollar mROI. Those answer different questions, so they go side by side.

In [10]:
fin.compare(fin.estimand("roi"), fin.estimand("mroi"))
Out [10]:
Different questions: show side by side
differs on intervention, baseline
details
whya remove-all (ROI) and a marginal (mROI) intervention answer different questions
field first second
intervention type remove_all, size -1.0, size_kind relative, arms , population geo_set type marginal, size 0.1, size_kind relative, arms , population geo_set
baseline zero_spend current_plan

For a number going into a sealed decision, a labelled best guess may not be enough. strict=True turns the fallback into a stop.

In [11]:
try:
    fin.compare(mmm, test, strict=True)
except fin.Stopped as stop:
    print(stop.code, "-", stop.detail)
Out [11]:
ESTIMAND_ALIGNED - estimands differ on ['intervention', 'baseline', 'horizon', 'point_rule', 'interval']; strict mode stops at assumed_alignment

4. Adding channels up honestly

Section titled “4. Adding channels up honestly”

A vendor table gives per-channel intervals but no joint draws, and display has no interval at all. How the channels’ errors move together is unknown, so the total is the envelope between the independent sum (narrowest) and the comonotone sum (widest), labelled.

In [12]:
roi = fin.estimand("roi")
contributions = {
    "tv": fin.number(4.1e6, (2.9e6, 5.6e6), roi),
    "search": (3.2e6, 2.7e6, 3.8e6),   # a (value, lo, hi) tuple works too
    "social": (1.6e6, 0.9e6, 2.4e6),
    "display": 0.7e6,                  # no interval: imputed and labelled
}
vendor_total = fin.total(contributions)
vendor_total
Out [12]:
Total 9,600,000, 6,850,000 to 12,850,000
envelope between independent and comonotone sums
details
G-UNC-01 UNCERTAINTY_IMPUTEDdisplay tier-default band +/-50% (proposed); interval imputed
G-DEP-01 DEPENDENCE_ASSUMEDindependence; interval is envelope of independent and comonotone
sum lo hi
independent 8.082601e+06 1.143644e+07
comonotone 6.850000e+06 1.285000e+07
reported 6.850000e+06 1.285000e+07

With joint draws (one fit, many draws per channel) the total uses them directly: no assumption, no fallback. Pass a draws-by-channels DataFrame.

In [13]:
rng = np.random.default_rng(7)
draws = pd.DataFrame(rng.multivariate_normal([4.1, 3.2, 1.6, 0.7], np.diag([0.6, 0.3, 0.4, 0.2]) ** 2 + 0.05,
                                             size=4000) * 1e6, columns=list(contributions))
joint_total = fin.total(draws)
joint_total
Out [13]:
Total 9,628,790, 7,680,340 to 11,642,236
from joint draws, no assumption needed
details
sum lo hi
independent 7.680340e+06 1.164224e+07
comonotone 7.680340e+06 1.164224e+07
reported 7.680340e+06 1.164224e+07
In [14]:
fig, ax = plt.subplots(figsize=(7, 2.2))
rows = {"vendor table (envelope)": vendor_total, "joint draws": joint_total}
for i, (name, t) in enumerate(rows.items()):
    ax.plot([t.lo / 1e6, t.hi / 1e6], [i, i], lw=6, solid_capstyle="butt")
    ax.plot(t.value / 1e6, i, "k|", ms=14)
ax.set_yticks(range(len(rows)), list(rows))
ax.set_xlabel("total incremental revenue, $M (90% interval)")
plt.tight_layout()
Out [14]:
Two horizontal intervals for total incremental revenue in $M (90% interval). The vendor table envelope runs from about 6.9 to 12.8; joint draws run from about 7.7 to 11.6. Both centre near 9.6.

5. The fallback ladder

Section titled “5. The fallback ladder”

When the full model is not available UMP steps down: full model, curve parameters, spend table, summary (warned), benchmarks (warned), not available.

In [15]:
pd.DataFrame([{"you have": ", ".join(have) or "nothing", "rung": d.step, "warned": d.warned}
              for have in (["full_model", "curve_params"], ["spend_table"], ["summary"], [])
              for d in [fin.ladder(have)]])
Out [15]:
you have rung warned
0 full_model, curve_params full_model False
1 spend_table spend_table False
2 summary summary True
3 nothing benchmarks True

Going deeper

Section titled “Going deeper”

Every babelfin object keeps the contract model on .spec (mmm.spec is a ump_spec ShownNumber), and .explain() prints the details as text. The full contract is in ump_spec when you need it.

Next: 02 · Validate and gate a document offline.