Guide · doing the work
Deciding what to measure
The standard question is “what sample size do I need for 80% power?” It has a precise answer, and it is the wrong question — because it never asks what the study is for. An experiment is a purchase. The thing you are buying is a change in what you would decide.
Start here
Power against what, and why does that number matter?
Conventional sizing asks you to nominate a minimum detectable effect and then sizes to detect it. Where does the MDE come from? Usually from a convention, a previous study, or the number that made the sample size affordable.
None of those is the number the decision turns on. That number is a boundary: the effect above which you would act and below which you would not. It comes from the economics of the decision — what the intervention costs, what an outcome unit is worth — and it is knowable before any data exists.
Sizing against the boundary rather than a convention changes the answer, and it changes it in both directions. Sometimes you need far more units than convention suggested. Often, as below, you need far fewer, or you discover the experiment was never worth running.
The reframe
Not: “can I detect an effect of
size d?”
But: “after this study, how much more
often will I take the right action than I would have without it — and is that
worth what it costs?”
Which makes the prior load-bearing
You cannot answer the second question without saying what you currently believe. That is not a weakness of the approach; it is the approach noticing that conventional sizing also depends on a prior and simply does not tell you which one. See priors that mean something.
A real example from this build
When the conventional study is buying something you already have
The design page sizes an experiment properly and finds, first, that the conventional target is not in doubt.
The prior, before the study
what the team already believes about the effect
Mass already above the conventional MDE
The conventional design would spend real money to establish that the effect exceeds 2 — a proposition already held with 99.9% confidence. Whatever that experiment is for, it is not that.
Sized against what the decision turns on
units for 81% power against an effect of 4.19 — against a proposal of 80. Same decision, less than half the study.
This is the most common finding in honest design work and it is almost never reported, because nobody runs the analysis that would reveal it. The conventional path produces a defensible-looking number, the study runs, and the question of whether it was worth running is never posed in a form that could have been answered “no”.
Putting a price on it
What is this experiment worth, in the currency of the decision?
If a study changes what you would do, its value is the value of the better decisions. That is computable before it runs, from the prior, the boundary and the stakes.
from the ledger: one unit is worth 400 USD (stated by the decision owner, 2026)
Two things follow immediately, and both are uncomfortable in the right way.
First, a study whose EVSI is below its cost should not be run, however good its power. Power measures the study's statistical properties; EVSI measures whether anybody's behaviour will change. They come apart more often than not.
Second, the value depends on the decision's economics, so it forces a conversation that is otherwise deferred forever: what is an outcome unit worth here? Answering it badly is better than not answering it, because a stated exchange rate can be argued with and revised. An implicit one cannot.
Note also that these are expected values under your current beliefs. If the prior is wrong, so is the valuation — which is one more reason to state the prior in units somebody can dispute.
Before you run it
Decide the stopping rule now, not later
“We'll look at the data as it comes in and stop when it's clear” is a procedure with no error rate. Peeking repeatedly at accumulating data and stopping when the result crosses a threshold will produce a crossing eventually, whether or not the effect is real.
The fix is old and well understood: decide the review schedule in advance, and spend your error budget across those looks — an O'Brien–Fleming boundary is conservative early and relaxes as evidence accumulates. What matters is that the rule exists before the first look and is recorded with the design.
The HYPER-3 case study runs exactly this: a monitoring plan whose stopping boundary is set before the trial, and which fires at the first scheduled review in one stratum.
Design is where transportability is cheapest to fix
Every mismatch in the transportability guide — wrong population, wrong window, wrong level — is an expensive correction after the fact and a free choice beforehand. If the decision is per person over a year, measure per person over a year. Designing the study against the target estimand rather than the convenient one removes the whole problem.
And write it down before it runs
The estimand, the graph, the prior, the boundary and the stopping rule, hashed and stored. That is a pre-registration, it takes an afternoon, and it is the difference between a result and a story. See provenance.