Vendor-Graded Homework
Every fitted model in this framework keeps a running report card. For each channel's predicted ROI, it checks what a later experiment actually found and asks whether the truth landed inside the credible interval. Roll that up across a portfolio and you get one summary number: coverage, the share of intervals that held. That's a real question, answered honestly: were the intervals right about as often as advertised? A report card seems to promise a sharper question. Was the forecaster honest? Did it report the interval it actually believed, or the one that was convenient to report? A calibration frequency and an honesty check are two different instruments. Below, a small simulation makes the difference concrete. A forecaster whose interval covers the truth 99.5% of the time, better than an honest forecaster's own 82.5%, scores 23% worse on the one rule built to catch exactly this kind of hedging.
No model is ever a finished, perfectly correct account of the world. It's a simplification built to be useful for a decision, and it stays useful only as long as someone keeps asking it sharper questions. Coverage was a real, good answer to a real question worth asking of a scorecard. The next question worth asking goes past whether the interval was right, all the way to whether the forecaster was being straight about how sure it was. That is the kind of question a measurement practice starts asking once the first one is already answered well, and asking it is a sign the current scorecard is doing its job.
The Only Number That Counts
The mechanism lives in a small module, platform/scorecard.py, whose own docstring states the goal plainly: nothing builds a model's credibility faster than watching its calls come true, or seeing it honestly own the misses. For each experiment whose outcome is known, the scorecard looks up the channel's predicted ROI (a mean, plus a lower and upper bound), works out how far off the realized value was, and flags whether that value landed inside the stated interval. Roll those flags up across a portfolio and the result is three numbers: how many predictions had an interval, how many of those hit, and the ratio of the two. That ratio is coverage, served at GET /projects/{project_id}/scorecard.
The join matches each calibrated or completed experiment to the run that recommended it or, absent that, the most recent run carrying any prediction for that channel (scorecard.py:64-66). The full aggregate (n_with_interval, hits, coverage) is built at scorecard.py:127-131.
The per-row error is computed and stored, and then never used again. No mean absolute error, no sense of how far off the model tends to be when it misses, just a frequency, a count of hits over trials. That's a calibration statistic, and calibration is a real, useful thing to have. Left on its own, it treats every hit the same regardless of how much room the forecaster left itself to be right, and that is a door worth opening.
A quick definition: what makes a scoring rule "proper"
The plain-language version first. A scoring rule is proper if the best way to score well is to report what you actually believe, with no hedging and no padding of your uncertainty for effect. It's strictly proper if honest reporting is the only way to get the best expected score, not merely one option among several that tie.
More formally, a scoring rule \( S(F, y) \) assigns a number to a predictive distribution \( F \) once the outcome \( y \) is observed, lower being better. Gneiting & Raftery (2007) call \( S \) proper if, for the forecaster's true belief \( G \),
$$ \mathbb{E}_{Y \sim G}\!\big[S(G, Y)\big] \;\le\; \mathbb{E}_{Y \sim G}\!\big[S(F, Y)\big] \quad \text{for every } F, $$and strictly proper if that inequality is strict whenever \( F \ne G \). The logarithmic score and the continuous ranked probability score (CRPS) are both strictly proper. A quantile (pinball) loss is strictly proper for the specific quantile it targets.
Coverage doesn't have this property. Many different forecasters, honest and shaded alike, can share the very same stated interval endpoints and get identical coverage, no matter how differently they actually distribute belief inside that interval, and no matter whether those endpoints reflect real uncertainty or a number chosen to look defensible. A metric with no unique honest optimum can't penalize dishonesty. The best it can do is catch a forecaster whose interval landed somewhere else entirely.
The next figure makes that idea concrete at the smallest possible scale. Two forecasters watch the same five realized values. One reports an honestly wide interval, the other a narrower one it privately knows understates its uncertainty. At the numbers below, both intervals happen to cover exactly three of the five realizations, a tie on coverage, which is the frequency the scorecard would report. Drag the slider to change how narrow the second forecaster's interval is relative to the honest one's, then toggle the readout between coverage and a proper score.
Same coverage, different honesty
Five fixed, illustrative realized ROI values (not a real fit): 1.04, 0.94, 1.10, 1.65, 0.42, against a common central forecast of 1.0. The honest forecaster reports a stated width matching the true spread. The overconfident one reports a fraction of that width. Coverage is the share of the five that land inside each stated interval. The proper score is each forecaster's own mean CRPS against the same five realizations, the correct scoring rule for a full predictive distribution rather than its endpoints alone.
At the default width (a third of the honest forecaster's), both intervals cover exactly 3 of 5 rounds, a flat tie at 60% coverage. The proper score breaks that tie: the overconfident forecaster's mean CRPS (0.2456) is 9.7% worse than the honest one's (0.2239). Drag the fraction toward 1.0 and the two forecasters converge to the same number, because at that point they are reporting the same belief. Drag it toward 0.15 and coverage itself starts to diverge too, but only well after the score already told you something was wrong.
What Propriety Buys You
That definition is the whole reason a proper scoring rule exists as a category. Under a strictly proper rule, a forecaster's best strategy is simply to report what it believes. Shade the interval narrower than your honest uncertainty, and your own expected score gets worse. There's no way to game a strictly proper rule by hedging, sandbagging, or reporting a "safer-looking" number instead of your real one.
That's exactly what the figure above shows at small scale. On this particular draw of five outcomes, the overconfident forecaster's tighter band happened to land inside the truth just as often as the honest one's did. A small sample produces that coincidence easily. Coverage, which only counts hits, can't see past it. A proper score doesn't depend on the coincidence going the overconfident forecaster's way on every round. It prices in the shape of the whole distribution being reported, and averaged over enough rounds, it stops rewarding anyone for claiming less uncertainty than they actually have.
Coverage is far from useless. A model that is actually miscalibrated, one whose interval is off in scale or badly placed, will eventually show up as bad coverage too, and the earlier post on simulation-based calibration and recovery coverage covers that side of the story well. This post takes up a narrower, complementary question that sits alongside that one. Coverage tells you whether the interval was about right. It stays silent on whether the forecaster was being honest about how wide that interval needed to be. Those are two different things worth checking, and a scorecard that only checks the first has a natural next check to add.
Three Opportunities Worth Building Next
Read scorecard.py as a set of incentives rather than a set of computations, and three specific opportunities open up. None of them requires anyone to act in bad faith. Each is simply a place where the code, as written today, can't yet distinguish an honest report from a strategically shaped one.
First, sharpness (how wide the interval is) doesn't yet factor into the aggregate. The portfolio number tracks only whether an interval hit, never how wide it was. Two rows with the same realized value and the same hit/miss outcome score identically regardless of whether one interval was twice as wide as the other. A model that widens every interval until it can hardly miss will see its coverage climb toward 100%, with nothing yet in this file to flag that as a cost rather than a virtue. That's exactly the pattern the next section's simulation puts a number on. (scorecard.py:127-131 is where the aggregate itself gets built, for anyone who wants to see it.)
Second, there's an opening to tighten the fallback match. When an experiment doesn't carry an explicit link to the run that recommended it, the join falls back to the most recently created run with any prediction for that channel (scorecard.py:64-66), chosen purely by date, with no check on whether that run was fit before or after the experiment's outcome was already known. A run fit today, after everyone in the room already knows how a geo test came out, can currently be matched against that same test as though it were a prospective call. Giving that match visible provenance, and flagging when it's retrospective, is a natural next addition.
Third, there's no denominator yet for what never got tested. The join walks the list of experiments, not the list of channels with a live prediction. A model could carry a predicted ROI for ten channels, have only the two cheapest or most-favored of them ever put to an experiment, and the scorecard would report 100% coverage on a sample of two. Nothing yet flags that eight other predictions were never checked against anything. Adding a second ratio, tested channels over channels with a live prediction, would close that out.
A nearby module already does something adjacent: reporting/evidence.py's EvidenceTier, which labels a number as experiment-validated, model-identified, or prior-dominated based on experiment coverage plus how much a prior contracted toward the posterior. That's a useful companion instrument. It answers where a number's credibility comes from, so a client can tell a validated ROI from an assumption wearing a model's clothes. But it answers a different question than this post does. It never reads a realized experiment value against a prediction, and it doesn't touch the honesty question here at all. Both are worth having, and neither substitutes for the other.
Who Looks Safest, Who Scores Best
The two-forecaster toy makes the mechanism easy to see. It doesn't show how that mechanism plays out at portfolio scale, under realistic noise, across a range of market conditions. The next figure runs three archetypes against 400 simulated rounds of a common true ROI distribution, using the same random draws for all three so the comparison is fair. An honest forecaster's stated width always tracks the true volatility. The overconfident one holds its width fixed and narrow no matter what, and the hedging one holds its width fixed and wide no matter what. The slider sets how volatile that true distribution is. Watch how the coverage ranking and the proper-score ranking agree at low volatility, then start to disagree as it climbs.
Coverage rewards caution; CRPS does not
Illustrative simulation, not real framework output: 400 simulated rounds per volatility setting (mulberry32 seed 4242), true ROI drawn from a Normal centered at 1.0 with the slider's standard deviation. Honest reports the true standard deviation every round. Overconfident always reports 0.15, and Hedging always reports 0.75. Coverage is against each forecaster's own nominal 80% interval. The score is mean CRPS per round, lower is better.
At the default volatility, coverage reads Hedging 99.5%, Honest 82.5%, Overconfident 43.5%. A coverage-only reviewer would call Hedging the safest performer in the portfolio and Overconfident the clear problem. The proper score disagrees about which of the two is worse. Overconfident's mean CRPS is 10.1% above Honest's, but Hedging's is 23.1% above it, which is close to a full rank inversion. Hedging's coverage stays at or above nominal across the entire slider range, so a "coverage ≥ 80% passes" gate would never flag it at any volatility tested, even though it is never the best-scoring forecaster of the three.
The direction of the disagreement is the interesting part. At low volatility, an interval fixed by convention rather than by belief can happen to land about right, and both metrics agree there's nothing to flag. As the environment gets noisier, the honest forecaster's stated width tracks it and the other two don't. They miss in opposite directions, and a coverage-only readout treats those as opposite verdicts. Too narrow reads as a clear miss. Too wide reads as safety. A proper score treats both as a cost, because both are one. It prices the hedge's wasted caution the same way it prices the narrow forecaster's overreach: as distance from the belief the data actually supported.
Where This Shows Up in Our Own Scorecard
None of this is a hypothetical borrowed from another domain. It's live, shipped code in this framework, and the trust claim it sits under is right there in the module's own docstring: nothing builds a model's credibility faster than watching its calls come true, or seeing it honestly own the misses (scorecard.py:3-4). That line names honesty as the thing being built, alongside accuracy, and the three opportunities above are exactly the three places a scorecard could satisfy that line's letter (a clean-looking coverage number) without yet fully living up to its spirit. A forecaster shading its intervals strategically, a "prediction" selected after the fact, or a 100% record built on a hand-picked slice of what it actually claimed to know would each currently clear this scorecard without a flag.
None of that is a bug in the join logic. The join computes exactly what it was built to compute. What it does not do yet is measure honesty directly, and building that measure is the increment worth taking next.
Scoring the Prediction, Not Just the Interval
None of this calls for new statistics. It calls for pointing an old one at data this module already has. Three additions follow, each answering one of the opportunities above.
Score the reported quantiles, not only whether the point fell between them. Every row already carries a predicted mean, a lower bound, and an upper bound, which together imply two specific quantiles. A pinball (quantile) loss evaluated at those two stated quantiles is strictly proper for exactly what's already stored: it costs a forecaster something, in expectation, to report an interval narrower than its true belief, in a way a hit/miss flag simply can't. Add the portfolio's mean pinball loss (or CRPS, wherever the full posterior survives) alongside coverage rather than in place of it, and coverage keeps answering whether the stated rate was hit while the proper score answers whether hitting it was earned or engineered.
Badge the post-hoc match. A row built from the fallback match, rather than an explicit link to its recommending run (scorecard.py:64-66), could carry that provenance visibly and compare the matched run's fit date against the experiment's readout date. A match from a run fit after the outcome was known describes history. Badge it "retrospective match, not a forecast", keep it out of any headline number, and the honest cases separate cleanly from the retrospective ones. That is how this framework already keeps underpowered refutation tests from silently counting as a clean pass.
Add the missing denominator. The framework already knows every channel carrying a live ROI prediction. Track that set against the channels that ever get matched to a realized experiment and you get a second ratio, tested channels over channels with a prediction, to sit next to the hit-rate coverage. A portfolio number that says "100% coverage" is a lot more useful once it also says what share of its claims that number actually covers.
💡 None of this requires re-deriving anything
Pinball loss, CRPS, and the logarithmic score are all closed-form, decades-old, and already used elsewhere in applied forecasting practice. The work here is entirely about what a small join chooses to aggregate, not about inventing new statistical machinery.
The next figure makes the third opportunity concrete, using the same shape of numbers this section describes: a model reporting a predicted ROI for eight channels, only some of which were ever put to an experiment.
The coverage the scorecard reports vs. the coverage that exists
Illustrative example, structured on scorecard.py's actual fields: eight channels each carry a contribution_roi prediction, and channels are tested in order of ease/confidence, cheapest and most-favored first. "Reported coverage" is what calibration.coverage would show, computed only over channels actually tested. "Testing coverage" is tested channels divided by all eight channels with a live prediction, the ratio the module never computes.
At the default of 2 tested channels (both hits), the scorecard's own field reads coverage: 1.0 on n_with_interval: 2, a clean 100%. The corrected testing-coverage ratio for the same portfolio is 25% (2 of 8). Drag the slider to 8 (every channel tested) and reported coverage settles at 62.5%, because two of the six untested channels turn out, once checked, to have been misses. The 100% figure was never false about the two channels it covered. It was silent about the six it didn't, and silence read as a clean bill of health.
A Next Step, Not a Verdict
Quoting a specific file and a specific docstring can read like an accusation, and that isn't the intent here. This is a single-maintainer module that solved a real, previously-unsolved problem: joining predicted and realized ROI at all, so that a model's track record is even visible in the first place. Every opportunity described above is a property of the metric it currently computes, not evidence that anyone has shaded an interval or gamed a match. Nobody has been caught doing anything with this scorecard. The point is that, as specified today, it couldn't yet tell you if someone had. Giving it that ability is a natural next increment, the kind of work a healthy module keeps generating.
The idea generalizes past this codebase, too. "Vendor" here means whoever is being graded: an internal model a team maintains, an external MMM vendor going through a bake-off, an analyst whose calls get reviewed at quarter's end. A commercial vendor grading itself on coverage alone has the same opening to build toward honesty-scoring as this framework's own module does. Nothing here claims that vendors specifically behave badly. The claim is only that a coverage-only scorecard leaves the same door open to anyone being scored by it, this project included.
A Scorecard Is an Institution
Frame the problem this way and it stops being a statistics question and becomes a question of incentive design. The mechanism-design literature made that shift explicit decades before MMM existed as a discipline. Prelec (2004) built a "Bayesian truth serum," which rewards a respondent for giving an answer more common than they expect others to give, so that honest reporting pays off even for a belief the respondent thinks is a minority view. Miller, Resnick, and Zeckhauser (2005) built the peer-prediction method for almost exactly this setting (eliciting honest evaluations of "opportunities, products, and vendors") by scoring a rater's stated belief about another rater's report with a proper rule, so honest reporting becomes the rational move rather than a hope. Both are solving the same underlying problem. A report is easy to collect, and a report worth trusting needs a scoring rule under which honesty is the strategy that pays, so that the forecaster has a reason of its own to be straight with you.
There isn't yet a body of literature applying this specifically to marketing-measurement scorecards. This post borrows decision theory that's forty to sixty years old (propriety traces back further still, to Osband's 1985 UC Berkeley thesis on cost-forecasting incentives, cited inside Gneiting & Raftery themselves) into a domain that hasn't applied it yet. The fix is not a research problem. Every tool it needs has existed since before most MMM software did, and nobody has yet pointed a proper scoring rule at this particular join. What is open here is the opportunity, not the math.
The point worth carrying past this one module: a scorecard sits inside the system it measures, as a working part of it, and it quietly rewards whatever it happens to measure. A coverage-only scorecard doesn't need anyone to act in bad faith to end up favoring confident theater over honest uncertainty. It only needs enough time, and enough attention paid to the number it publishes. That's a different question from the one covered in this blog's earlier post on holdout error, which is about grading a causal estimand with the wrong kind of metric entirely. This post is about a metric of the right kind (interval calibration is a legitimate thing to want) that simply isn't fully incentive-compatible on its own yet. "Measure the right thing" and "measure it in a way nobody has reason to game" are two different design requirements, and closing the second is exactly the kind of work that keeps a useful model becoming more useful, one honest layer at a time.
Takeaways
- This framework's
platform/scorecard.pyjoins predicted channel ROI to realized experiment readouts and reports one aggregate: coverage, hits over trials (scorecard.py:127-131). Per-row error is computed and then set aside. Nothing yet rolls it up. - Coverage isn't a proper scoring rule: forecasters with very different honesty can share the same stated interval endpoints and get identical expected coverage. Only a strictly proper rule (CRPS, pinball loss, the log score) makes truthful reporting the forecaster's uniquely best strategy (Gneiting & Raftery, 2007).
- Three concrete opportunities follow: interval width doesn't yet enter the aggregate (sharpness is invisible), the fallback match (
scorecard.py:64-66) lets a run fit after an outcome is known stand in for a forecast, and there's no denominator yet for channels with a live prediction that were never tested at all. - In a seeded simulation, a hedging forecaster's coverage read 99.5%, better than an honest forecaster's own 82.5%, while scoring 23.1% worse on mean CRPS. An overconfident forecaster with 43.5% coverage scored only 10.1% worse. A coverage-only read and a proper-score read rank these two in opposite order.
- None of this is a flaw to pin on a single-maintainer module or on anyone using it. Read "vendor" as whoever is being graded: an internal model, an external vendor, or an analyst's own track record.
- The fix borrows mechanism-design tools (Prelec, 2004; Miller, Resnick & Zeckhauser, 2005) that predate MMM as a discipline: score the stated quantiles with a proper rule, badge the post-hoc matches, and report tested-channel coverage alongside hit-rate coverage.
- None of this makes the scorecard finished, and it never will be. It's one more increment toward a model that's more useful for the decision it actually supports: knowing who, and what, to trust.
References
- Gneiting, T., & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association, 102(477), 359–378.
- Prelec, D. (2004). A Bayesian Truth Serum for Subjective Data. Science, 306(5695), 462–466.
- Miller, N., Resnick, P., & Zeckhauser, R. (2005). Eliciting Informative Feedback: The Peer-Prediction Method. Management Science, 51(9), 1359–1373.
- Osband, K. (1985). Providing Incentives for Better Cost Forecasting. Ph.D. thesis, University of California, Berkeley.