#The setting

Upworthy was a viral-news site that, for every story, wrote several candidate headlines (and sometimes several images) and showed them at random to visitors on its home page and in boxes on other pages. The package with the most clicks per impression (the click-through rate, CTR) became the headline everyone saw afterwards. Between 2013 and 2015 it ran tens of thousands of such tests; the archive (Matias et al. 2021) releases them, split into an exploratory sample, a confirmatory sample and a holdout, and asks researchers to explore on the first. We use the exploratory sample.

The archive only has final totals per package (impressions and clicks), not the order in which they arrived. So wherever we need a test in progress (peeking, stopping rules, bandits) we simulate it, with CTRs and effect sizes taken from the archive and the model fitted to it. Every simulated result is labelled as such.

#The data

Clicks are rare (about 1 in 80 impressions) and each arm gets a few thousand impressions, so a typical arm has 40-50 clicks. Binomial noise on 45 clicks is about $\pm 15\%$ of the CTR, which is the same size as many of the differences a headline test hopes to find.

A free A/A check. In 51 tests, two or more arms are identical in every creative field (headline, image, excerpt, lede, share text). These are natural A/A tests: their CTRs should differ by binomial noise only. If the randomisation or the counting added extra variation, the standardised differences would be wider than a standard normal.

Three things to take forward. Baseline CTRs differ by an order of magnitude between stories, so effects should be modelled within a test, relative to that test's own level. Within tests, arms differ by a lot: a spread of about $\pm 0.3$ on the log-odds scale, part real effect and part noise. The natural A/A pairs look like pure binomial noise (sd close to 1, well inside the chi-square band), which supports a plain binomial likelihood with no extra noise term. Fifty-one pairs is not many, so this check can only rule out large departures.

Part A · One test¶

#A1 · Four headlines for one story

A November 2014 test with four headlines for the same video and the same image:

Upworthy's dashboard showed a "significance" of 100 for arm 3 and the editor declared it the winner. What does a standard analysis say?

A p-value answers "how surprising would data like these be if all headlines were equal?". The editor's question is different: which headline should I run, and what does it cost me if I pick wrong? A Bayesian analysis answers that directly.

#A2 · Beta-Binomial posteriors, P(best) and expected loss

Give each arm's CTR $p_k$ a flat Beta(1, 1) prior. With $k$ clicks out of $n$ impressions the posterior is Beta$(1 + k, 1 + n - k)$ - no sampler needed. From draws of the four posteriors we get:

• P(best): the share of draws in which arm $k$ has the highest CTR;
• expected loss of choosing arm $k$: $E[\max_j p_j - p_k]$, the clicks we expect to give up per impression by committing to $k$, in clicks per 1,000 impressions. The expected loss of the best choice is also the expected value of perfect information (EVPI): the most that any further testing could gain per future impression.

The flat prior deserves one comment: it says a 60% CTR is as plausible as 1%, which is absurd. It barely matters for each arm's level here (3,700 impressions dominate it) but, as part B shows, it matters a great deal for the differences.

With flat priors, arm 3 is best with probability about 0.94 and is estimated to beat the best alternative by roughly 25% (the 90% interval just touches zero). The p-values disagree with each other: "all four equal" is rejected (p = 0.014), but arm 3 against the runner-up is not (p = 0.10). The expected loss has no such ambiguity: committing to arm 3 costs, in expectation, 0.085 clicks per 1,000 impressions; committing to any other arm costs 6-10. That number is the natural stopping rule: stop when the expected loss of the leader falls below what you are willing to give up (a threshold of caring, say 0.1 clicks per 1,000 at a CTR of about 2%, i.e. 0.5% of the CTR), not when a p-value crosses 0.05. With that threshold this test would stop and ship arm 3. Multiplying the EVPI by the traffic still to come gives the most that more testing could be worth: 0.085 per 1,000 over another 100,000 impressions is under 10 clicks.

Two warnings before we trust any of this. First, arm 3's 25% lift was picked because it was the largest of four noisy estimates: the winner's curse says it is probably too big. Second, the flat prior treats a 25% lift as no more surprising than a 1% one. Both are questions about what headline effects usually look like, and the archive can answer them.

Part B · What do headline effects really look like? A hierarchical model over 1,000 tests¶

#B1 · The model

For arm $k$ of test $j$ with $n_{jk}$ impressions and $y_{jk}$ clicks:

$$y_{jk} \sim \text{Binomial}(n_{jk}, \operatorname{logit}^{-1}(a_j + \delta_{jk})), \qquad \delta_{jk} = r_{jk} - \bar r_j, \qquad r_{jk} \sim \text{Student-}t(\nu, 0, \tau).$$

$a_j$ is the test's average log-odds (each test has ~15,000 impressions, so a separate weakly informative Normal(-4.2, 1.5) per test is enough; no pooling needed). The arm effects are centred within their test, so $\delta_{jk}$ is "how much better than this test's average arm", and every contrast between two arms is $r_{jk} - r_{jl}$, unaffected by the centring. The centring also removes a ridge between $a_j$ and the mean of the effects that otherwise slows the sampler. The question is the distribution of effects: its scale $\tau$ and its tail weight $\nu$. A Normal version ($\nu \to \infty$) is the standard assumption; the $t$ lets most headlines be close and a few very different. Priors: $\tau \sim$ HalfNormal(0.3) (effects of up to about $\pm 60\%$ in odds are plausible), $\nu \sim$ Gamma(2, 0.1) (mean 20, so the Normal is not excluded).

To keep the notebook fast we fit a random sample of 1,000 tests (about 4,650 arms), leaving out the part-A test so that it can play the role of a new test later.

#B2 · Prior predictive check

Before fitting: do the priors produce data that look like headline tests? We compare two statistics of simulated data with the observed ones: each arm's CTR, and each arm's deviation from its test's mean on the log-odds scale (the quantity in the middle panel of the first figure).

The prior predictive CTRs span from below 0.01% to tens of percent and cover the observed ones, and the within-test deviations are wider than observed (sd 0.37 vs 0.28), with heavier tails. The prior allows the data without forcing them. (The small spike at 0.0001 is arms with zero clicks, shifted so they show on the log axis.)

#B3 · Fitting: Normal effects, and a sampling failure for the $t$

Three fits, all with nutpie defaults: Normal effects with the usual non-centred parameterisation ($r = \tau z$), and $t$ effects both non-centred and centred.

All three fits run without divergences, but the $t$ model's hyperparameters mix much worse in the non-centred form: $\tau$ and $\nu$ have ESS of about 130-140 and the worst r_hat is 1.02; the centred version roughly triples their ESS (350-450) in the same time, with the worst r_hat over all 5,600 parameters at 1.013. The usual rule "non-centre a hierarchy" assumes the data per group are weak. Here each arm's ~45 clicks pin its effect to about $\pm 0.15$, the same size as $\tau$ itself, so neither form is ideal, and for the $t$ the non-centred $z$ carries the heavy tail: a $z$ in the tail must move together with $\nu$. We keep the centred $t$ and the non-centred Normal (whose ESS is fine).

The estimates themselves are what the question was about:

The tail index $\nu$ is about 3: headline effects are heavy-tailed. Most pairs of headlines differ by modest amounts, but a few differ enormously. The two models agree on the typical size of an effect but not on the tails. Which one fits better?

#B4 · Posterior predictive check: the tails decide

The Normal model reproduces the centre of the distribution but not the tails: 2.1% of arms lie more than 0.75 from their test's mean and 0.45% more than 1.0, while the Normal model predicts 0.7-1.2% and 0.1-0.3%. The $t$ model's 90% ranges contain the observed fractions beyond 0.5 and 1.0 and just miss at 0.75 (1.2-1.8% vs 2.1%) and at 0.25, so it is better but not perfect. (We do not use PSIS-LOO here: every arm has its own effect, so leaving an arm out removes all information about that effect and the importance weights are unreliable. The split-half check below is a sharper out-of-sample test of what we need.)

#B5 · The winner's curse, and a real replication

In each test the arm with the highest observed CTR is the "winner". Its observed advantage is biased upwards: it won partly because its noise was favourable. The hierarchical posterior shrinks every arm's effect towards 0; a heavy-tailed prior shrinks small effects hard and leaves large, well-measured effects mostly alone.

Is the shrinkage right? The archive has no repeated tests, but a binomial count can be split. If each impression is an independent Bernoulli trial, drawing half of an arm's impressions at random (a hypergeometric draw of the clicks) gives two independent half-samples of the same arm. We fit both models to half 1 only, pick each test's winner on half 1, and ask half 2 how good the winner really is. Half 2 did not take part in the selection, so its average is an unbiased estimate of the winners' true effect.

This is the winner's curse measured on real data: the winners' raw half-1 advantage is about twice what the same arms achieve in fresh impressions. Both hierarchical models predict the replication almost exactly. Across all arms, the raw estimates replicate with a slope of about 0.45 (they are too extreme by a factor of two), the $t$ posterior means with a slope close to 1: they are calibrated. The Normal model does nearly as well on the average winner; its weakness is in the tails (B4), which matter for the large-effect decisions.

#B6 · The population as the prior for a new test

Back to the part-A test, which was not in the fitted sample. Instead of flat priors on each arm's CTR, we use what 1,000 earlier tests learned about how headlines differ: the Student-t effect prior with $\tau$ and $\nu$ fixed at their posterior medians (their posterior is narrow, so plugging them in loses little).

Arm 3 is still the most probable winner, but everything moves towards "less sure, smaller": P(best) drops from 0.94 to about 0.91, the median lift from +25% to +18% (90% interval about -4% to +47%), and the expected loss of shipping arm 3 rises from 0.085 to about 0.125 clicks per 1,000 impressions - above the 0.1 threshold, so under the population prior this test would keep running a little longer. The +18% is what a forecast of extra clicks from the new headline should use, and the population prior is the one whose estimates the split-half check just showed to be calibrated.

Part C · Looking at a running test (simulated tests)¶

In practice nobody waits: the dashboard updates every few minutes and it is tempting to stop as soon as it says "significant". The archive has only final totals, so we simulate two-arm tests with realistic numbers: each simulated test takes the CTR of a randomly drawn real test as its baseline, gets 250 impressions per arm between looks, and runs for at most 20 looks (5,000 impressions per arm, a bit more than an average Upworthy arm).

With clicks this rare, the pair of click counts carries almost all the information about the difference through one number: given $m = y_A + y_B$ total clicks on equal traffic, $y_B \sim \text{Binomial}(m, \operatorname{logit}^{-1}(d))$ where $d$ is the log rate ratio (essentially the log-odds difference at CTRs near 1%). This lets us compute exact posteriors for 20,000 tests at every look in NumPy:

• flat prior: uniform prior on the click share, so P(B > A) = P(Beta$(1 + y_B, 1 + y_A) > 1/2)$;
• population prior: $d$ = difference of two draws from the fitted Student-t effects (integrating over the posterior of $\tau$, $\nu$), posterior on a grid.

Decision rules: the frequentist "p < 0.05" (two-proportion z-test), and the Bayesian "P(B > A) > 0.975 or < 0.025", the one-sided analogue of the same 5% error budget.

Checking a p-value at every look and stopping at the first "p < 0.05" turns a 5% false-positive rate into about 25% over 20 looks, and it keeps climbing with more looks. The Bayesian rule with a flat prior is no better (22%): with a flat prior, P(B > A) > 0.975 is numerically almost the same event as a one-sided p < 0.025, and it inherits the same inflation. The population prior helps, because it expects most differences to be small and so needs more evidence, but its A/A "winner" rate still rises from 1.5% at a single look to about 4% with peeking.

This is the honest statement about Bayesian stopping: the posterior is not affected by the stopping rule (the likelihood principle), so a posterior computed after peeking means exactly what it would mean without peeking. That guarantee is about calibration - among all the tests where you claim "97.5% sure B is better", B is better about 97.5% of the time - when the tests come from the population your prior describes. It is not a guarantee about the error rate in a world where every difference is exactly zero, which is what the A/A figure measures. A/A tests are not drawn from our prior (the prior puts no mass exactly at 0), so there the claim "97.5% sure" is just wrong in the way any claim would be.

Let us check the calibration claim where it applies: tests whose true effect is drawn from the fitted population (simulated from the model, so this is the ideal case for the population prior).

With effects drawn from the population, claims made with the population prior are right about 98-99% of the time with or without peeking (each claim was made at P > 0.975, so that is what calibration promises): the posterior probability survives optional stopping, and peeking buys somewhat earlier decisions. The flat prior's claims are right 98% of the time when it looks once, but only about 91% of the time when it peeks - below the 97.5% each claim announced - because a flat prior takes early noisy differences at face value. Calibration is a property of prior + model, not of the word "Bayesian".

#C2 · Stopping on expected loss

A/B tests are not really about claims but about choices, and a wrong choice between two nearly equal headlines costs almost nothing. The expected-loss rule stops as soon as the leader's expected loss (under the population prior) is below a threshold of caring $\varepsilon$, in clicks per 1,000 impressions, and picks the leader. With baseline CTR $p_0$, choosing A when the truth is $d$ costs $p_0 \max(e^{d} - 1, 0)$ per impression (and symmetrically for B).

The threshold trades time for loss, in units a product owner understands. A coin flip loses about 1.9 clicks per 1,000 impressions; running every test to the 5,000 cap loses 0.22. With $\varepsilon = 0.5$ the average test stops after about 1,300 impressions per arm and loses 0.39; with $\varepsilon = 0.2$, three quarters of tests stop early and the loss is within 0.02 of running to the cap. Among the tests that stop early the realised loss is below the threshold, as it should be when the prior matches the population (the "all tests" column is higher because tests that hit the cap were the hard, close ones). From $\varepsilon = 0.1$ down the overall loss barely moves while more and more tests run to the cap: smaller thresholds only cost time. In A/A tests the rule also stops and picks an arbitrary arm, which costs exactly nothing: a "false positive" is harmless when the arms are equal. What the rule gives is not a claim that the chosen arm is better, but a choice whose expected cost is small.

Part D · Adapting while testing: bandits (simulated)¶

A fixed-split test shows the worse headlines to half or more of the traffic for the whole test. A multi-armed bandit shifts traffic towards arms that look good while it learns. Thompson sampling (TS) does this by showing each arm with the probability that it is the best, computed from its Beta posterior - so exploration fades exactly as fast as uncertainty.

We simulate 2,000 four-headline tests from the fitted population: each draws a baseline CTR from the archive and four arm effects from the Student-t population (with $\tau$, $\nu$ from their posterior). Traffic arrives in batches of 100 impressions for 20,000 impressions (about an Upworthy test's size), and the policies are:

• fixed split: 25% each throughout (the classic A/B/n test);
• explore-then-commit: fixed split for the first 1,000 impressions per arm, then all traffic to the leader;
• epsilon-greedy: 90% to the current leader, 10% spread evenly;
• Thompson sampling with flat Beta(1, 1) priors.

Regret is the number of clicks lost compared with showing the truly best arm to everyone.

The fixed split loses, by construction, the whole difference between the best and the average arm for every impression of the test (81 clicks per 20,000). The three adaptive policies lose about a third as much, 26-27 clicks, and over 20,000 impressions they are nearly tied. Thompson sampling needs no tuning constant, has the lightest bad tail (90% of tests below 49 clicks) and ends with the most reliable final choice (the true best arm in 72% of tests, vs 65% for the others). Its regret curve keeps bending over as it stops exploring, while epsilon-greedy pays for its 10% exploration forever and explore-then-commit's cost depends on a test length chosen in advance - which is the next question.

#D2 · The decision: how long should a headline test run?

Upworthy did not run bandits: it tested, then showed the winner to everyone for the rest of the story's life. The right test length depends on how much traffic comes after the test: long tests waste impressions on bad headlines, short tests pick the wrong one more often and pay for it on all later traffic. With the fitted population we can compute the expected total regret of "test $n$ impressions per arm, then commit" for a story that will get $N$ impressions in total (a preposterior analysis: averaged over effects drawn from the population and over the data a test would produce). We also compute two planning numbers from the same simulation: the chance the test picks the best arm, and the classical power of a two-arm z-test at $\alpha = 0.05$ for realistic effects.