The Optimizer's Curse

Matthew Reda· July 26, 2026· MMM

A budget optimizer built on a fitted MMM does something more specific than report a noisy ROI. It searches a whole range of possible allocations and hands back the one that scores best. In the jargon that winner is its argmax, the option that comes out on top. That search is not neutral arithmetic sitting on top of the posterior. Picking the best-looking option among many noisy candidates makes that option's own reported value look better than it really is, and the same statistical mechanism runs whether the candidates are three consulting studies, ten named channels, or the many spend levels a budget allocator walks through on its way to a recommendation. This framework already takes one flavor of that problem seriously. It re-optimizes under each posterior draw and reports how unstable the resulting plan is. What it has not yet attached an interval to are the two numbers a budget decision actually runs on: shadow_price and marginal_roas, the marginal return of the next dollar and the per-channel funding line. Both are read straight off the optimizer's own recommended point as plain two-decimal numbers. Simulating this framework's exact allocation mechanism at its own default sample size shows the plan's own reported value overstating its honest, out-of-sample value by 0.3–0.5%, before sampling error even enters the picture. A shadow price reported as "1.40" can carry a 90% interval that puts an 18% chance on the marginal dollar actually losing money.

None of that makes the model wrong, exactly. It makes it a model. Every fitted MMM is a simplification built to be useful for a decision, not a finished, perfectly correct description of the world, and there is always a sharper question left to ask of it. Finding one is what a measurement practice getting more mature looks like, not a bug report. This post is about one specific, buildable next question: how confident should the shadow price and marginal ROAS actually sound?

Picking a Winner Makes It Look Better Than It Is

Start with the everyday version. Three consultants each estimate how much value a project will add, and you go with whichever number is highest. Even if all three are equally skilled and equally unbiased, the number you picked is, on average, too optimistic. The consultant who produced it did not lie. The optimism came from the rule you used to choose. You filtered for the biggest number, and big numbers are exactly what noise produces when you go looking for them.

The same thing happens to a budget optimizer. Rather than comparing three named alternatives, it searches a whole continuum of possible spend levels, judging each one against a finite sample of posterior draws, and it hands back the plan whose estimated value happens to look best in that sample. That plan's own reported value is, for the same reason, an overstatement of what it will actually deliver.

In this framework, that search runs on a fixed sample of posterior draws (200 by default, fewer for some callers) averaged into a single response curve per channel, which a greedy step-by-step allocator then walks across to find the highest-scoring plan (planning/budget.py). Nothing about that is wrong. Greedy marginal allocation is the right algorithm for a concave response curve, and averaging over posterior draws is the obvious thing to do with a Bayesian fit. The module's own documentation is honest about the half of the mechanism it already handles well. Re-optimizing under each posterior draw turns parameter uncertainty into decision uncertainty, how stable the plan is rather than just how wide the individual ROIs are. optimize_budget does this for real, and reports expected_regret: the KPI a person would be leaving on the table, on average, by committing to this one plan instead of the truly best one.

Smith and Winkler didn't stop at theory. They reanalyzed Clemen and Kwit's 38 published Eastman Kodak decision-analysis studies, a portfolio that claimed 487 million dollars in value added, and added plausible estimation noise of 5%, 10% and 25% to the studies' own numbers. At those three noise levels the claimed value overstated the true value by 14%, 39% and 128% respectively. That is the size of the effect in a setting with far fewer options to choose from than a continuous ad-spend allocation offers.

Fitting Alongside “Advertising Is a Stock”

Advertising Is a Stock already covers the decision-instability side of this story well, in a section titled "Not the Optimizer's Curse." That post says, correctly, that this framework re-optimizes under each posterior draw to turn parameter uncertainty into decision uncertainty. Alongside that, it reports the odds the plan beats the current allocation and prices the expected KPI left on the table by committing to one plan. Those are the right instruments, and they all accept the objective and question the estimate.

That's still true, and this post doesn't walk it back. This is the piece that naturally comes next. expected_regret, expected_uplift, and the per-channel spread between a plan's high and low posterior spend levels (allocation_instability) are honest, correctly-averaged statements about how much the recommended plan could move. What they don't cover yet is the two specific numbers a person actually reads off that plan to decide whether one channel's next dollar is worth spending: the shadow price (the marginal value of the next dollar, the level greedy allocation equalizes across channels) and marginal ROAS, the per-channel funding line. Both are computed once, at the recommended plan, and neither carries a distribution yet. The fields sitting right next to them already report a spend-level range across draws, and no equivalent range exists for the shadow price. The instability of where the money goes is already priced. What deserves the same treatment next is the pair of numbers that justify why.

So the two posts stack rather than compete. Every point the sibling post makes survives being handed the model's exact response curves with zero uncertainty, because it concerns the objective rather than the estimate. This post is the mirror image, entirely about the estimate, and specifically about the two estimates the existing decision-uncertainty machinery doesn't reach yet. A plan can carry a well-characterized expected_regret and a shadow price with no error bar at all, at the same time. They answer different questions about the same optimizer.

Two Ways of Describing the Same Effect

Smith and Winkler's result is about choosing among a handful of named options: three consulting studies, ten channels. A budget optimizer does something related but not identical. It searches a continuous range of spend levels, using a finite sample of posterior draws to judge each point along the way. The intuition carries over directly. Taking the best of many noisy guesses and taking the average of many noisy guesses are different operations, and they don't commute. The best-of tends to run high. Statisticians who study this kind of search (a Sample Average Approximation, or SAA, meaning you optimize an average built from a limited sample rather than the true underlying quantity) have shown the same optimistic bias shows up here too, and that it shrinks, but never fully disappears, as the sample of draws grows (Mak, Morton & Wood, 1999).

These are two ways of describing the same regression-to-the-mean-at-selection effect: one for choosing among named alternatives, one for choosing the best point in a continuous space from a finite sample. A budget optimizer walking a spend simplex on 200 posterior draws is doing the second thing. The first is the easiest way to build intuition for it, so the figure below runs the discrete version directly, MMM-flavored: several channels share an identical true marginal ROI, and nothing separates the winner from the pack except which one happened to draw favorable noise.

One more related post is worth placing carefully, because it looks like the same idea but works differently. Hundreds of Models, One Winner is also about a winner looking better than it is, and its fix is hierarchical shrinkage across a short list of named alternatives (geo ROIs, model specs), borrowing strength across a finite set of parallel estimates before anyone ranks them. A continuous budget simplex has no finite set of channels to pool across at the resolution the optimizer actually searches. It walks hundreds of increments of a simplex rather than comparing a handful of named options. So that post's shrinkage machinery doesn't carry over mechanically here. The diagnosis is the same family, but the remedy that actually fits a continuous search is the Mak-Morton-Wood one above, and the fix section below builds on it directly.

Why This Shows Up in Every Fit, Not Just Bad Ones

Nothing pathological is required for this mechanism to bite. Every response curve the optimizer builds is already a finite posterior sample (120 to 200 draws depending on the caller) evaluated on a coarse grid and averaged. The greedy allocator walks toward whichever point on that averaged curve currently looks best. That's the SAA setup exactly: optimize an average built from a sample, and the sample's own quirks get baked into the winning allocation before anyone checks whether the win would survive a fresh sample.

The figure below runs that mechanism for real, on a synthetic six-channel world with known concave response curves, using this framework's own greedy-allocation logic. For each posterior sample size it draws that many noisy curve realizations, with multiplicative noise calibrated to this framework's own default media-coefficient prior, Gamma(mu=1.5, sigma=1), whose coefficient of variation of 0.82 sets the noise slider's upper range. It averages those realizations the same way objective_curves does and runs the identical greedy walk to pick an allocation. The figure then compares two numbers for that plan: the in-sample value (what the mean curve says the plan is worth, which is what the framework would report) against the true value (the same plan scored against the exact population curve the draws came from, which is what a fresh, independent look would find).

Nothing about this world is stacked against the framework. The response curves are smooth, concave, and correctly specified, and the noise is centered exactly on the truth, so no channel's posterior is biased. What drives the gap is that a mean curve built from a finite sample differs from the population curve, and the greedy allocator finds the sample's own idiosyncratic high points before it finds the population's. That is the ordinary case, not an edge case. It happens every time this optimizer runs on a real fit, because every fit's response curves are a finite posterior sample by construction.

The Number With No Error Bar

The part of this that actually reaches a person is narrower than "the reported uplift is a bit optimistic." The two numbers used to justify a specific keep-or-cut call come out bare, with nothing attached to them. When the agent reports a budget plan, the shadow price shows up as a plain two-decimal figure (the marginal return of the next dollar), and the per-channel marginal ROAS is a plain dict of floats with the same shape (agents/model_ops.py).

That asymmetry is the specific opening worth building next. The level already gets an interval. Nothing similar exists for the slope, the number that actually answers "should I add or cut the next dollar here," even though the per-draw values needed to build it already sit right above the line where they get collapsed to a single mean.

What This Doesn't Claim

This isn't a claim that the framework's decision-uncertainty reporting is hollow. expected_uplift, expected_regret and allocation_instability are legitimately honest, correctly-averaged statements, and the credit the sibling post gives them stands. This post's claim adds to that. Those statements describe the plan, and they do not yet describe the shadow price and marginal ROAS numbers a reader actually cites to justify moving one channel's spend.

Nor is it a claim that more posterior draws would fix it. Figure 2's gap shrinks with sample size, matching the Mak-Morton-Wood bound, but it doesn't vanish at any draw count tested, and doubling max_draws doesn't change what kind of number shadow_price is: a point estimate at the argmax, carrying a shrinking but real residual optimism.

The Kleywegt-Shapiro-Homem-de-Mello (2002) reference is cited narrowly. Their paper's formal object is a finite feasible set, so it's cited here for the general SAA method rather than for the continuous-simplex bias claim itself. That claim belongs to Shapiro (2003) and Mak-Morton-Wood (1999), both cited above.

The simulations are demonstrations, not a measurement of any one real fit's exposure. Both figures use stylized worlds (six channels, known response curves, noise calibrated to this framework's own default prior scale) built to make the mechanism visible and roughly size it. A real fit has more channels, more collinearity, and priors shaped by real data rather than free-floating noise. The 0.3–0.5% and 18% figures above describe these particular simulated worlds, not universal constants.

One more current-research note. Berman, Zhang and Zhao's 2026 arXiv preprint on correcting for winner-selection bias in randomized experiments isn't peer-reviewed yet. It's cited here only to note that this whole family of questions, whether and how to correct a number for having been selected as the best-looking, is an active area of current statistical research, not a settled question that's been sitting unaddressed.

This isn't a gap unique to this framework, either. Google's own Meridian documentation and API reference say much the same thing about the industry more broadly. Its scenario-planning and budget-optimization guide doesn't discuss uncertainty propagation, a credible interval, or a probability of outperforming for the optimizer's recommended allocation, and its OptimizationResults API class carries spend, ROI, mROI, incremental outcome, and CPIK as plain point fields, with no confidence or probability field on any of them, despite the rest of Meridian being fully Bayesian end to end. A secondhand summary claiming Meridian's optimizer reports a probability of outperforming didn't hold up against the primary documentation, so it isn't repeated here. None of this is a knock on Meridian. That documentation is evidence that a bare point estimate at the argmax is the industry's current default for this exact output, not something one framework overlooked. If anything, this framework's own expected_regret and allocation_instability already go a step further than the point of comparison does.

A Good Next Upgrade

None of this argues for less optimization. Equimarginal allocation under a concave response is still the right target, and greedy marginal allocation is still the right algorithm for finding it. The upgrade worth building is giving the number the algorithm hands back the same epistemic honesty already built for the allocation's location. Three changes would get there, each cheap relative to a full re-architecture.

Attach a distribution to the two numbers that don't have one yet. per_draw_alloc and the curve's per-draw contributions already carry every draw optimize_budget needs (lines 594-610). The per-draw marginal value at the recommended spend is the same quantity already computed once on the mean curve, and it could be computed once per draw instead, at essentially no additional posterior sampling. shadow_price_p5/_p95 and per-channel marginal-ROAS intervals would just be a reuse of arrays the optimizer already holds in memory, not a new fit.

Evaluate the committed plan against an independent sample, not the sample that picked it. This mirrors Mak-Morton-Wood's own recommended practice for SAA: draw one sample to solve for a candidate plan, and a separate, independent sample to estimate how far it sits from the true optimum. In practice, split the available posterior draws (or resample in two disjoint halves), pick the allocation on one half, and report expected_uplift and the shadow price evaluated on the other half. The moment the evaluation sample isn't the selection sample, the reported numbers stop being circular.

Shrink the mean curve itself before optimizing. This is the direct translation of Smith and Winkler's own prescribed fix. Their Bayesian-shrinkage correction blends the raw winning estimate toward a population or prior curve, and is provably curse-free. A smoothness prior or light regularization on the response curve, applied before the greedy walk runs, would do the same job here, damping the finite sample's idiosyncratic high points before the walk can lock onto them. That is cheaper and better targeted than simply adding more posterior draws.

None of this changes what the optimizer recommends most of the time. The mean curve and the true curve usually point toward a similar allocation, and the figures above show overstatement measured in fractions of a percent to a handful of percent, not a reversal of the ranking. What changes is the reading. Someone deciding whether a channel's marginal dollar is worth spending would get those two numbers with the same honest uncertainty this framework already built for the plan they sit inside.

Take those three changes as one buildable increment in the same direction the framework is already headed, rather than a verdict on the current optimizer. No model, this one included, is ever going to be a finished, perfectly correct description of the world. A model gets more useful for the decisions people actually need to make one honest increment at a time, and a shadow price with a real interval attached is a good next one.

References

• Smith, J. E., & Winkler, R. L. (2006). The Optimizer's Curse: Skepticism and Postdecision Surprise in Decision Analysis. Management Science, 52(3), 311–322.
• Mak, W. K., Morton, D. P., & Wood, R. K. (1999). Monte Carlo Bounding Techniques for Determining Solution Quality in Stochastic Programs. Operations Research Letters, 24(1), 47–56.
• Kleywegt, A. J., Shapiro, A., & Homem-de-Mello, T. (2002). The Sample Average Approximation Method for Stochastic Discrete Optimization. SIAM Journal on Optimization, 12(2), 479–502.
• Shapiro, A. (2003). Monte Carlo Sampling Methods. In A. Ruszczyński & A. Shapiro (Eds.), Handbooks in Operations Research and Management Science, Vol. 10, pp. 353–425. Elsevier.
• Berman, R., Zhang, W. W., & Zhao, H. (2026). Valuing Winners: When and How to Correct for Selection Bias in Randomized Experiments. arXiv:2605.18887 [stat.ME]. A 2026 working paper, cited here as background on active research, not as a settled result.
• Google. Scenario planning and future budget optimization and OptimizationResults API reference. Meridian documentation, developers.google.com/meridian. Checked directly (July 2026). Documentation can change.