Workshop 00 — Thinking in Distributions

Your team just ran an A/B test on two ad creatives. The new creative (B) got a 5.4% click-through rate; the old one (A) got 5.1%. Your media buyer asks the only question that matters: "Is B actually better — and how sure are we?" "5.4 is bigger than 5.1" is not an answer: both numbers came from a finite sample, and rerunning the test would produce different ones. The honest answer is a probability — "there's an X% chance B is better, and if it is, the upside is roughly Y dollars a month, give or take Z." This page teaches the machinery that produces sentences like that, with no Bayesian background assumed and every computation running live in your browser.

Why does this matter for Marketing Mix Modeling (MMM)? Because every MMM decision — shift budget to Search, cut Display, scale TV — is a bet placed under uncertainty. The whole point of the Bayesian approach in mmm_framework is that the model's output is not a single ROAS number but a distribution of plausible ROAS values, and decisions get made from the whole distribution. This is workshop 0 of 6; everything here is plain arithmetic — no MCMC, no PyMC, no framework code. Next stop: Workshop 01 — Priors.

1 — Probability as belief: distributions over things you don't know

The click-through rate of creative B is a fixed fact about the world — you just don't know it. Classical statistics treats it as an unknown constant and only lets you make probability statements about data. The Bayesian move is simpler and more natural: use probability to describe your uncertainty about the quantity itself. So instead of one number for B's CTR, you hold a probability distribution — a curve over all possible values, where height means plausibility. A random variable here is just a quantity you're uncertain about (a CTR, a channel's ROAS, next quarter's sales lift), and the distribution is your current state of knowledge about it.

The shape of the curve is the message:

• A wide distribution says "could be anywhere in this range" — you know little. Acting on its peak alone is gambling.
• A narrow distribution says "pinned down to here" — you know a lot.

Below: three different states of knowledge about a CTR, all centered on the same best guess of 5%. Hover over the curves — same peak position, wildly different business implications.

Computed live in your browser — the same math as the notebook. Three Beta distributions with identical mean 5% and standard deviations 0.048, 0.015 and 0.005 (notebook-measured: 0.0476 / 0.0154 / 0.0049). Wide blue = a brand-new creative where 2% or 10% are live possibilities; narrow berry = a mature campaign whose CTR is known to within a few tenths of a point.

Every Bayesian analysis is a machine for moving wide curve → narrow curve: start with what you believe, feed in data, end with a sharper belief. The rule that does the moving is next.

2 — Bayes' rule, visually

Three ingredients, three names you'll use for the rest of this series:

• Prior — your belief about the quantity before seeing the new data, expressed as a distribution. Ours: a CTR around 5%, loosely held — "new creatives in this account usually land near 5%".
• Likelihood — for each candidate value, how probable is the data you actually observed? If the true CTR were 1%, seeing 11 clicks in 200 impressions would be a shock; if it were 5.5%, it'd be typical. The likelihood scores every candidate this way.
• Posterior — your updated belief after the data: prior reweighted by likelihood.

Bayes' rule is just the recipe for combining them:

\[ \underbrace{P(\theta \mid \text{data})}_{\text{posterior}} \;\propto\; \underbrace{P(\text{data} \mid \theta)}_{\text{likelihood}} \times \underbrace{P(\theta)}_{\text{prior}} \]

where \( \theta \) ("theta") is the unknown CTR and \( \propto \) means "proportional to — multiply the curves pointwise, then rescale so the result is a proper distribution". That's the whole rule. Here it is running on the creative-B data: 11 clicks out of 200 impressions (a 5.5% observed rate), against a Beta(2, 38) prior centered at 5%.

Computed live in your browser — the same math as the notebook. Read it like a negotiation: the prior (blue, dotted) votes "around 5%, flexible"; the likelihood (amber, dashed) votes "the 200 impressions point at 5.5%"; the posterior (berry, solid) is the settlement. Notebook-measured: prior mean 5.000%, observed 5.500%, posterior mean 5.417% — strictly between the two — and the posterior sd (0.0146) is less than half the prior sd (0.0340), because two agreeing sources of information beat one.

Live exploration — watch the posterior sharpen

With only 200 impressions the prior still has real influence. What happens as data piles up? Drag impressions (the observed conversion rate is set by the second slider; clicks are recomputed each time). Watch two things: the posterior narrows, and it migrates from the prior's center toward the observed rate.

Computed live in your browser — the same math as the notebook (the notebook's ipywidgets slider, ported; the grid-Bayes update reruns on every drag). Notebook-measured posterior sd at 5.5% observed: n = 10 → 0.0333, n = 100 → 0.0195, n = 1,000 → 0.0071, n = 10,000 → 0.0023 — roughly a 10× sharpening, and by n = 10,000 the posterior mean sits within 0.01 percentage points of the data.

3 — The grid trick: there is no magic

Notice what the page actually did to compute every posterior so far:

1. lay down a fine grid of candidate CTR values (2,001 points from 0 to 1),
2. at each point, multiply prior × likelihood,
3. rescale so the curve integrates to 1.

Three lines of numpy (or, here, three lines of JavaScript). No theorems, no special functions, no sampler. This is worth dwelling on, because Bayesian inference has a reputation for being mysterious — and for one unknown quantity, it's brute-forceable on a laptop in microseconds.

It happens that for this particular model there's also a pencil-and-paper shortcut: a Beta prior updated with click/no-click data gives a Beta posterior with the counts just added on — \( \text{Beta}(a, b) \rightarrow \text{Beta}(a + \text{clicks},\; b + \text{non-clicks}) \). Statisticians call this conjugacy — a happy algebraic coincidence where the posterior stays in the prior's family. It's a convenience, not a requirement: the grid gives the same answer with zero algebra. Proof:

Computed live in your browser — the same math as the notebook. The closed-form Beta(2+11, 38+189) curve (green band) with the brute-force grid posterior overlaid as × markers. Notebook-measured: largest pointwise difference 1.7×10⁻¹³ against curves that peak around 28 — identical to numerical precision. The grid is the math.

4 — Priors matter (and then stop mattering)

The prior is the part of Bayes that makes newcomers nervous: "isn't that just injecting opinion?" Let's stress-test it. Three analysts look at the same creative before launch:

• Skeptic (berry): "new creatives in this category flop — CTR around 2%." Beta(2, 98).
• Optimist (green): "the brief was great — CTR around 10%." Beta(10, 90).
• Agnostic (blue): a flat prior — every CTR from 0% to 100% equally plausible — "I refuse to assume anything." Beta(1, 1).

Then all three watch the same data come in. A prior has a location (where it's centered) and a strength (how many "pseudo-impressions" of conviction back it). Two things to try with the sliders:

1. Crank prior strength up with little data — the three posteriors stay apart. This is what "the priors are driving the results" looks like.
2. Now crank impressions — watch the three posteriors fuse into one curve regardless of strength. Data overwhelms priors at a rate you can see.

Computed live in your browser — the same math as the notebook (its prior-strength × sample-size ipywidgets cell, ported; observed rate fixed at 5.5%). Notebook-measured at default strength 100 with 200 impressions: the skeptic lands at 4.33%, the agnostic at 5.94%, the optimist at 7.00% — every posterior mean strictly between its prior and the data, and the spread of opinions collapses from 48.0 to 2.7 percentage points after one batch of data. At strength 1,000 the disagreement is 6.67% with n = 200 but only 0.38% with n = 20,000.

Computed live in your browser — the same math as the notebook. Posterior-mean disagreement between the three analysts (strength 100) as data grows. Notebook-measured stops: n = 20 → 6.67%, n = 200 → 2.67%, n = 2,000 → 0.38%, n = 20,000 → 0.04% — monotone convergence: priors matter exactly until the data outweighs them.

5 — From distribution to decision

Back to the opening question. Full data from the A/B test:

A point estimate — a single-number summary like the mean or the peak — says "B: 5.4% beats A: 5.1%" and throws away everything we just learned to care about: the shape. The Bayesian answer keeps the shape and answers the buyer's actual question directly: what is the probability that B's true CTR exceeds A's? The recipe is disarmingly simple: build a posterior for each creative (flat prior + data → Beta posteriors, via the conjugate shortcut), then sample from both — draw thousands of plausible (CTR_A, CTR_B) pairs — and count how often B wins.

Computed live in your browser — the same math as the notebook. The two posteriors. Note the overlap: B's curve sits to the right, but a fat slice of A's plausible values exceeds a fat slice of B's. That overlap is the entire reason "5.4 > 5.1" is not an answer.

Computed live in your browser with seeded sampling — the same Monte Carlo as the notebook (200,000 paired draws from the two posteriors). The histogram of CTR_B − CTR_A: mass to the right of zero is "B wins". Notebook-measured: P(B beats A) = 0.7252 sampled, 0.7260 by exact grid integration — "B is probably better, but it's not a lock", now quantified.

Credible intervals and the HDI

To report the size of an effect with its uncertainty, Bayesians use a credible interval — a range that contains the true value with a stated probability, given your model and data. The version used throughout this series is the HDI (highest-density interval) — the narrowest such range: every value inside it is more plausible than every value outside it. A "90% HDI of 5.0%–6.1%" means exactly what it sounds like: "90% chance the true CTR is between 5.0% and 6.1%." Drag the probability slider and watch the shaded region grow and shrink on creative B's posterior — you'll meet HDIs again on every MMM parameter in Workshop 04.

Computed live in your browser — the same math as the notebook (its HDI ipywidgets slider, ported; the interval is recomputed from the same 200,000 sorted draws on every drag). Notebook-measured at 90%: HDI [4.82%, 6.00%], width 1.18 percentage points, containing 90.0% of the posterior mass — and never wider than the equal-tailed alternative ([4.85%, 6.02%]).

6 — Monte Carlo as a superpower: from CTRs to dollars

Here's the trick that powers the entire back half of this series. The buyer doesn't actually care about CTR — she cares about revenue. Suppose the campaign serves 2,000,000 impressions a month and each click is worth $0.50 in expected revenue. The monthly revenue uplift from switching to B is:

\[ \text{uplift} = (\text{CTR}_B - \text{CTR}_A) \times 2{,}000{,}000 \times \$0.50 \]

This is a derived quantity — a number computed from unknowns rather than estimated directly. And here is the superpower, called Monte Carlo — answering questions about distributions by drawing many random samples and just counting: push every posterior draw through the formula, and you get the full distribution of the derived quantity, for free. No new model, no calculus — arithmetic on the draws you already have.

Computed live in your browser with seeded sampling — the same Monte Carlo as the notebook (every one of the 200,000 paired CTR draws pushed through the revenue formula). Notebook-measured: median uplift $2,981/month, 90% interval −$5,229 to $11,248, P(uplift > 0) = 72.5% — the same event as "B beats A", because a monotone formula preserves every probability statement. A point estimate would have said "switch, you'll make the median amount". The distribution shows the red tail: a real chance the switch loses money.

Live exploration — does the upside clear the switching cost?

Switching creatives isn't free (production, trafficking, QA). Drag the cost slider and watch the decision quantity — the probability the first-month uplift beats the cost — move. The question changed from statistics ("is B's rate higher?") to business ("how many dollars, with what risk?") and the method didn't change at all: same draws, one line of arithmetic.

Computed live in your browser with seeded sampling — the same Monte Carlo as the notebook (its switching-cost ipywidgets slider, ported: the net histogram and the probability readout recompute from the same draws on every drag). Notebook-measured stops: P(net gain) at $0 → 73%, $2k → 58%, $6k → 27%, $12k → 4%.

Computed live in your browser — the same math as the notebook. The full decision curve: probability the switch pays for itself in month one, at every cost from $0 to $12,000, with the four slider stops marked. Notebook-measured: the switch is better than even money up to a cost of about $3,000. Every budget question is a vertical slice of one posterior.

This per-draw arithmetic is exactly how mmm_framework turns posterior parameter draws into ROAS distributions, budget-shift scenarios, and "probability this channel clears its hurdle rate" — the entire subject of Workshop 05 — From Draws to Decisions.

7 — What you can now say

You started with "5.4% vs 5.1% — is it better?" and ended with a probability that B wins, a credible range for its CTR, a dollar-denominated uplift distribution, and a cost threshold below which switching is better than even money — all from two Beta curves and counting. The terms you picked up (the site-wide glossary has the full vocabulary for the rest of the framework):