One question, carried all the way · step 4 of 8
Designing the experiment
How big does the trial need to be, and is the answer worth its cost?
Four questions in order: what effect to power for, how many depots that takes, what the answer is worth, and which concrete design buys it cheapest. Powering for the effect you hope for is how trials get commissioned that cannot fail informatively.
The effect worth detecting is the break-even, not the point estimate — the experiment has to distinguish 'worth funding' from 'not', and that boundary is where the precision has to land.
step 4
from axiom.design import DecisionSpec, ValuePerOutcome, evoi_gaussian, sample_size
from axiom.calibrate import carryover_window_factor
from axiom.surface import GeometricCarryover as GC
# The experiment reads a first week; the decision lives in the steady
# state. `carryover_window_factor` says how much of the effect has
# landed by then, which is what converts the break-even onto the
# scale the trial can actually measure.
lam = float(biased.posterior.flat("lam_a").mean())
share = carryover_window_factor(GC(max_lag=MAX_LAG), {"lam_a": lam}, 1, treatment="a")
threshold_first_week = BREAK_EVEN * share.counterfactual
print(f"{share.counterfactual:.3f} of the effect lands in the first week")
print(f"break-even on the experiment's scale: {threshold_first_week:.3f}")
print()
decision = DecisionSpec(
name="raise_the_schedule",
threshold=threshold_first_week,
value_per_outcome_unit=VALUE_PER_POINT * 500 * 52,
numeraire="USD",
)
sd = float(observed_panel.frame["y"].std())
ss = sample_size(effect=threshold_first_week, sd=sd, power=0.8, alpha=0.05)
print(f"powering for the boundary, not the hope: {threshold_first_week:.3f}")
print(f" depots per arm : {ss.n}")
first_week = realize(
experiment_estimand, biased, assume_identified=True, mass=0.9, seed=SEED
)
ev = evoi_gaussian(
decision,
float(first_week.summary.mean),
float(first_week.summary.sd),
0.35,
)
print()
print(f"perfect information would be worth : {ev.evpi:,.0f} USD")
print(f"this experiment, at se=0.35 : {ev.evsi:,.0f} USD")
what it printed
0.572 of the effect lands in the first week break-even on the experiment's scale: 3.863 powering for the boundary, not the hope: 3.863 depots per arm : 23 perfect information would be worth : 567,331 USD this experiment, at se=0.35 : 392,538 USD
Every line above was captured by running this code at build time. The steps share one namespace, so the code on this page is the code you would type after the steps before it.