Pillar 02 · axiom.design
What should I measure next?
Sample-size calculators answer a narrow question: how many units to detect an effect of a size you nominated. axiom asks the wider one. What do you already believe? What would the experiment actually change your mind about? What is that worth, what does withholding treatment cost, and after both — is this experiment worth running at all?
Step one
Start from what you already believe
A pilot has been run and fitted. Before choosing a size, axiom looks at the posterior for the contrast you care about and asks whether the conventional threshold is even in question.
Posterior for the contrast, before the new experiment
Shaded bars are draws above the conventional minimum detectable effect of 2.0.
axiom.estimands.realize(estimand, fit, keep_draws=True) over a Laplace posterior · 2,000 draws · seed 0
What the pilot already says
90% equal-tailed · posterior sd 1.15 · the simulation’s truth is 3.54
Powering for the conventional threshold would waste the study
The pilot already puts 99.9% of posterior mass above 2.0. Designing a study to detect 2.0 would spend money confirming something you are already sure of. axiom anchors instead to the 4.19 at the sceptical end of what you believe — the effect that is genuinely still in play.
Step two
Then size it
With the effect to power for settled at 4.19 and an outcome sd of 4.55, the arithmetic is ordinary — and axiom will also tell you what a size someone has already proposed actually buys.
Power against effect size, at four sample sizes
Two-arm difference in means, equal allocation, α = 0.05.
axiom.design.power_curve(n, sd=4.55, effects=…) · α = 0.05, two-sided
Over-powering is not free. Every extra unit in the control arm is a unit not receiving a treatment you mostly believe works, and that cost shows up explicitly in the next section rather than being left as a footnote.
Step three
What is the answer worth?
An experiment is only worth running if it can change a decision. Here the programme scales up if the contrast clears 6.0, each outcome unit is worth 400 USD, and the programme covers 500 units. The prior sits just under the threshold with real spread — which is exactly when information has value.
What a more precise experiment is worth
Expected value of sample information as the experiment’s standard error falls, against the ceiling set by knowing the truth outright.
axiom.design.evoi_gaussian(decision, prior_mean, prior_sd, experiment_se) over a grid of 40 precisions
Value of perfect information
USD · the most any study on this decision could ever be worth
Value of this study
USD · at se 1.478, capturing 0.24 nats
Where the value comes from
The curve is steep at the right and flat at the left: buying precision is worth a great deal until the decision stops being in doubt, and almost nothing after. Had the threshold been 1.0 instead — comfortably below the prior — the whole curve would sit near zero, and the correct recommendation would be to run no experiment and act now.
Step four
Subtract what it costs to find out
Three candidate designs, scored on the same decision. Net value is the information they buy minus the treatment withheld from the control arm minus the fixed cost of running them.
Net value of each design
All three clear zero, so all three are worth running — the question is only which is best. Hover for the breakdown.
axiom.design.evaluate_candidate(candidate, decision, prior_mean, prior_sd, economics) · winner by net value: proposed_holdout
| design | EIG (nats) | EVSI | opportunity cost | fixed cost | net value | power | Pareto |
|---|---|---|---|---|---|---|---|
| proposed_holdout | 0.409 | 35,265 | 12,378 | 550 | 22,337 | 1.000 | on front |
| switchback | 0.331 | 31,096 | 16,186 | 420 | 14,490 | 0.997 | on front |
| powered_holdout | 0.235 | 24,530 | 12,378 | 300 | 11,852 | 0.966 | on front |
The recommendation is stable
axiom re-runs the whole comparison over a grid of each economic input. Scaling the value per outcome across 0.25, 0.5, 1.0, 2.0, 4.0× leaves the winner unchanged, and so does scaling the experiment’s standard error across 0.5, 0.75, 1.0, 1.5, 2.0×. No tipping points were found in either sweep — the answer does not hinge on a number someone guessed.
The price is on the record
The value per outcome is not a hard-coded constant. It is a stated input that writes its own ledger line: “one unit is worth 400 USD (stated by the decision owner, 2026)” — so a reader six months later can see who claimed it and challenge that rather than the arithmetic.
What is in axiom.design
165 public symbols — the largest subpackage. Power and MDE, cluster and panel designs, Fisher information and D-optimal probes, method calibration and leaderboards, economic inputs and value of information, alpha-spending and sequential boundaries, and the monitoring machinery the case study runs on.
Next: fold the answer back in
Once the experiment has run, its result should change the observational model you already had — and the record should say exactly what that transfer assumed.