Scenarios, optimisation and decision records
A scenario is a saved question asked of specific model revisions; a decision record captures the business choice made from scenarios and is checked against what actually happened, with an allowance for the optimiser’s curse.¶
Scenario. Plan by media line, region and period with cost basis; endpoint alias and revision (never a file) and view used (full, curves, table); results with range, estimand and fallback step; assumptions (lookups, splits, calibration choices, dependence assumption if any); author, purpose, status. Deltas between scenarios use common random numbers and the shared thinning index (G-SCN-04). Related scenarios form a group pinned to the same revisions.¶
Staleness. When an endpoint is promoted, scenarios scored against the earlier revision are marked stale. Staleness is driven by monotonic artifact versions, so a late event cannot clear it (section 19).¶
Decision record. Question; options (each a scenario); chosen option; estimand of the comparison; expected outcome per draw (median, interval, probability of beating the current plan, expected regret); audit level (lowest input); gate results; fallbacks used, each acknowledged by the approver; drafted by; approved by; review-after date; integrity fields (section 18).¶
Optimiser’s curse. Choosing the option with the best estimated return selects for estimates that happen to be high, so realised returns fall short on average (Smith and Winkler, 2006). Draft 2 makes this explicit:¶
- Each decision record stores the raw expected uplift and a curse-adjusted figure, shrunk by a factor estimated from past decision records (study 7). Until that study reports, the adjusted figure uses posterior-predictive shrinkage within the model and is labelled provisional.¶
- At the review-after date, realised versus expected is computed and stored. A shortfall within the expected curse band is recorded as calibration data, not model failure; a shortfall beyond it opens a model review.¶
- The running ratio of realised to expected uplift by channel is a monitored metric (section 15, outcomes analysis).¶
Estimate the model reports True value, unknown when choosing
The curse-adjusted figure is provisional until study 7 reports; until then it uses posterior-predictive shrinkage within the model. Band width here is illustrative.
Draw 1: Plan 1 had the best estimate, so it was picked. Its true value, unknown when choosing, was 90% of that estimate. Picking the top estimate tends to pick the one that got lucky.
At review it delivered 92% of the raw estimate and 100% of the curse-adjusted figure. That shortfall is inside the curse band, so it is recorded as calibration data, not model failure.
Illustrative simulation: a fixed seed, so every visitor sees the same draws. Scale has no units and no numbers from any real account.