Workshop 05: From Draws to Decisions

Your CMO doesn't want a posterior. She wants three sentences: which channel do we cut, where does the next million go, and how sure are we? Workshops 00–04 built the machinery; this capstone is the bridge from machinery to those three sentences. The bridge has exactly one rule — compute per draw, summarize last — and by the end of this page that one rule will have ranked the channels, priced the next dollar, and told us, with a number, how confident to be about a budget reallocation. Because the model is fit on a synthetic world with a sealed answer key, every claim gets graded.

Everything below is arithmetic on one fit — the same model, settings, and seed as workshop 03: NumPyro NUTS, 500 draws × 2 chains, parametric adstock, on the dgp.build("clean") world from mmm_framework.synth. The fit took 10 s with max r-hat 1.009, zero divergences, min bulk ESS 355. One units note: this world's KPI is in sales units, so "ROAS" here means incremental KPI units per dollar — a value of 0.8 says a dollar buys 0.8 units. The answer key, recorded when the world was built:

1 — The doctrine: \( f(\mathbb{E}[\theta]) \neq \mathbb{E}[f(\theta)] \)

A derived quantity (any business number computed from model parameters — ROAS, an uplift, a ranking — rather than estimated directly) can be computed two ways. The plug-in shortcut (collapse the posterior to one "best" parameter value, then compute) is the tempting one. The notebook demonstrates why it quietly misreports, using TV's saturation curve (diminishing returns: each extra dollar buys a bit less than the last):

\[ \text{weekly effect} \;=\; \beta_{TV}\,\bigl(1 - e^{-\lambda_{TV}\, x}\bigr) \]

The posterior knows \( \beta_{TV} \) and \( \lambda_{TV} \) only as clouds of draws — 1,000 plausible values each, correlated with each other (workshop 04's joint structure: a flatter curve with a bigger \( \beta \) explains the same data as a steeper curve with a smaller \( \beta \)). Two recipes for "TV's effect at a mid-curve spend level": average the parameters first and push the averages through the curve (plug-in), or push each (β, λ) pair through and average the 1,000 answers (per-draw). If the curve were a straight line they'd agree. It isn't, and they don't — measured from the notebook's posterior at \(x_0 = 0.5\):

Illustrative recreation (seeded), live in your browser — a correlated (β, λ) cloud pushed through the same saturation formula as the notebook. The thin green curves are individual posterior draws; the solid line is their per-draw average; the dashed red line is the plug-in curve, which sits above the honest curve at every positive spend level — it is a curve no posterior draw believes in. Drag the slider to move the comparison point. The measured notebook numbers at \(x_0=0.5\): plug-in 1.5556 vs per-draw 1.4098, a +10.3% overstatement.

Why this is THE transferable skill. Every number your CMO asks for — ROAS, an uplift, "which channel is best" — is a nonlinear function of parameters (ratios, curves, rankings, max's). For all of them the plug-in shortcut is biased, and — worse — it produces a single number, hiding how sure you are. The per-draw recipe fixes both at once: the spread of the 1,000 answers is your uncertainty, for free.

2 — ROAS is a distribution (and we can grade it)

The fitted model carries a per-draw array called channel_contributions: for every posterior draw, every week, every channel, the KPI units that channel's spend generated. The ROAS recipe: per draw, sum a channel's weekly contributions and divide by its actual total spend → that draw's ROAS; summarize last — 1,000 draws → a distribution per channel → median plus a 90% HDI (highest-density interval — the narrowest range holding 90% of the draws; workshop 04). Because the world is synthetic, the table gets a column real analysts never see — the truth, and whether the interval caught it:

Illustrative recreation (seeded) — per-channel ROAS densities from seeded draws matched to the measured medians and 90% HDIs above; the ink diamonds mark the measured TRUE ROAS from the answer key. Every channel's ROAS carries real width — remember that width when ranking. Intervals caught the truth for 4/4 channels.

3 — Decision probabilities: numbers your CFO can argue with

Once ROAS is 1,000 draws instead of one number, questions that classically require a statistics consult become counting. "Does Search clear our breakeven multiple?" → count the draws above the threshold: (ROAS_search > t).mean(). That is a decision probability (the posterior probability that a business statement is true). "Is Social actually better than Display?" → count the draws where it is — computed within each draw, so the comparison automatically respects everything the two estimates share. No p-values, no significance rituals: "there's an 81% chance Social out-earns Display per dollar" is a sentence a CFO can argue with, budget against, or demand more data about.

One honesty note: in this world the answer key says every channel's true ROAS sits below 1. A good model should agree — and it does. Measured P(ROAS > 1.0): TV 0.000, Search 0.001, Social 0.070, Display 0.022. And from the measured pairwise "beats" matrix: P(Social beats Display) = 0.811, P(Social beats TV) = 0.998, while P(Search beats Display) = 0.446 — an honest coin flip.

Interactive, illustrative (seeded) — P(ROAS > t) per channel, counted from seeded draws approximating the notebook's posterior (matched to the measured medians/HDIs). Drag the bar like a finance team would: fully-loaded margin, contribution margin, "must beat last year's blended 0.7"… The probabilities move; the draws never change. Measured anchor at t = 1.0: TV 0.000, Search 0.001, Social 0.070, Display 0.022 — the model correctly refuses to claim anyone clears 1.

4 — Marginal ROAS: the next dollar is not the average dollar

The §2 ROAS — total contribution ÷ total spend — is the average ROAS: what each dollar earned on average, historically. But your CMO's second question was about the next dollar. Under saturation those differ, always in the same direction: on a curve that flattens, the next dollar lands on a flatter part than the historical average dollar did. Marginal ROAS (the return on a small additional dollar of spend, at today's operating point) is therefore below average ROAS for every channel — and the gap differs by channel, which is exactly what makes rankings flip. Geometrically: average ROAS is the slope of the secant (the line from zero to today's operating point); marginal ROAS is the slope of the tangent (the line that just kisses the curve there).

The framework's compute_marginal_contributions(compute_uncertainty=True) simulates a +10% nudge per channel through paired posterior draws (same seed for baseline and nudged prediction, so shared noise cancels draw-by-draw — the per-draw doctrine again). The measured table, with the payoff in the rank columns:

The two rankings genuinely disagree on this world. By average ROAS, Search out-earns TV (0.730 vs 0.569). By the next-dollar yardstick the order flips: TV 0.428 vs Search 0.404. Search keeps only 55% of its average return at the margin — it operates deepest into its curve — while TV keeps 75%. And because the flip was computed per draw, it comes with its own confidence: measured P(TV's next dollar beats Search's) = 0.62 — a real but modest call, not a certainty.

Measured — average ROAS (solid) vs marginal ROAS (hatched) per channel, both with measured 90% HDIs from the notebook. The solid bars rank Social > Display > Search > TV; the hatched bars rank Social > Display > TV > Search. Only the marginal number is about the future.

5 — Response curves with bands, and what-if scenarios

Tangent-vs-secant was one channel at one point. The full deliverable is each channel's response curve (expected weekly contribution as a function of weekly spend) drawn per draw, so it comes with a credibility band, not just a line. Read each curve with two questions: where is the operating point (diamond = today's average week), and is the curve still climbing there, or flat? Steep → headroom; flat → saturated — more budget mostly buys confirmation that the curve is flat. The notebook's measured saturation-at-operating-point (share of the estimated ceiling already consumed in a typical week): Search 0.46, TV 0.29, Display 0.25, Social 0.20. Search has eaten almost half its curve; Social has barely started. For Search, the measured secant slope is 0.894 against a tangent slope of 0.629 — the next dollar earns about 30% less than the picture the historical average paints.

Illustrative recreation — steady-state weekly response curve per channel, anchored to the measured operating points, contributions, and saturation-at-op fractions from the notebook; band width scaled to each channel's measured ROAS interval. The secant runs from zero to the operating diamond (its slope ≈ average return); the tangent kisses the curve there (its slope ≈ the next dollar's return). Switch to Social to see a curve still climbing; switch to Search to see one going flat. The notebook's own caveat applies: this is a steady-state simplification — trust the §4 table for the decision numbers.

What-if scenarios: uplift is a distribution too. "What if we put 20% more into Social next year?" The one-liner is mmm.what_if_scenario({"Social": 1.2}) — measured point answer: +644 KPI units (+0.94%) for $941 more Social spend. Useful, but it summarizes to a point. The doctrine version is two paired mmm.predict() calls (same seed, so each draw's simulated noise cancels in the difference) and a per-draw subtraction. The measured +20% uplift distributions:

Measured — mean uplift of +20% spend per channel with 90% HDIs, from the notebook's paired predictions (P(uplift > 0) = 1.0 for all four). Absolute lift rewards big budgets — TV's +20% is the biggest check ($1,839) and buys the biggest lift (769). Per extra dollar, the ordering reshuffles to the marginal story: Social 0.684, Display 0.548, TV 0.418, Search 0.389. And cuts bite harder than raises on a concave curve: the notebook's measured Social −20% scenario loses a mean 695 units with P(loss) = 100% — the dollars you remove were earning more than the dollars you'd add.

6 — A simple reallocation, with honesty

The capstone scenario: move budget, keep the total constant. Section 4 nominated the donor and the recipient — Search's next dollar earns the least (most saturated), Social's earns the most. So: cut Search 20% and give those dollars to Social. And, as a control on ourselves, the plausible-looking mistake: rank by average ROAS instead. On that table Search out-earns TV, so a naive analyst happily moves TV money into Search — into the most saturated curve on the books. Both scenarios get the full per-draw treatment; both get graded the same way.

Illustrative recreation (seeded) — the two reallocations' gain distributions, from seeded draws matched to the measured means and 90% HDIs in the table. Same total dollars, two rankings: the marginal-driven move clears zero with P = 0.98; the average-ROAS-driven move is a coin flip tilted to a loss, P(gain > 0) = 0.18. This chart is the CMO answer: what to move, what it's worth, and how sure we are.

7 — What this means for real budget meetings

The deliverable that survives a budget meeting is never "the ROAS is 0.73." It is the §6 chart plus three sentences shaped like the CMO's questions: cut the most saturated channel's next dollars, not the lowest-average one's; the next million goes where the marginal curve is steepest; and here is the probability we're right. Concretely, the habits this page bakes in:

• Never rank channels by average ROAS when deciding increments. The average grades the past; the margin prices the future. On this world they disagree, and only one of the two resulting reallocations wins (P = 0.98 vs 0.18).
• Attach a probability to every recommendation. "81% chance Social out-earns Display" invites a productive argument; "Social is better" invites a political one.
• Say "coin flip" out loud when it is one. P(Search beats Display) = 0.446 — the honest answer is that the data can't separate them yet, which is itself a decision input (it prices an experiment).
• Keep the caveat attached: these are the model's own error bars, conditional on its causal assumptions. The pressure-testing scorecard is the required reading before trusting any of this on real data.

Where to go next: Aurora 05 · Unified Workflow runs this exact doctrine at story scale — a full analyst engagement where per-draw decision quantities drive a real budget recommendation; the budget-optimization workflow guide covers the production tooling for the constrained-allocation version of §6; and workshop 04 is the prequel if any of "draws, HDIs, joint structure" felt shaky.

Glossary