Reproducibility levels
Every artifact states how far it can be audited, a result is only as auditable as its weakest input (G-AUD-01), and “close rerun” now has a statistical definition stated on the decision quantities.¶
| Level | Meaning | Typical artifacts |
|---|---|---|
| 4 Exact rerun¶ | Code, environment, data and seeds saved; rerun is bitwise identical | Python pipelines, saved BigQuery jobs, seed-deterministic fits |
| 3 Close rerun¶ | Rerun agrees within the tolerance below | MCMC fits with non-deterministic samplers, GPU runs |
| 2 Logged¶ | Inputs, outputs and steps recorded; cannot rerun | Registered Excel steps, DataRobot, hosted endpoints with logged calls |
| 1 Taken on trust¶ | Provider’s word only | Vendor tables and reports |
Close-rerun tolerance. Raw draws will differ between runs, so the test is on the estimands a decision uses. For each compared quantity theta (each channel’s ROI and mROI posterior mean, and each channel’s recommended allocation share, all on the estimand of the decision they feed), z = |theta_1 - theta_2| / sqrt(MCSE_1^2 + MCSE_2^2), with MCSE computed as in Vehtari et al. (2021), method mean. With one compared quantity, the rerun is close when z <= 2 (proposed). With two or more, the two-sided normal p-values of z are combined with Holm’s procedure at family-wise alpha = 0.05 (proposed), and the rerun is close unless Holm rejects. The recommended allocation’s expected KPI MUST also differ by less than 1% (proposed) of the account’s standard materiality threshold. Conformance fixtures carry a fixed materiality_threshold in their metadata for this purpose. The check runs in CI on the conformance fixtures and on every independent rerun for a decision at the Full materiality tier.¶
Environment capture. A Level 3 or 4 claim MUST record a lockfile and a container image digest, alongside the code commit, dataset hashes and seeds. An artifact without them MUST be labelled Level 2, regardless of what else was saved. A Level 4 claim is checked by rerunning from the recorded lockfile, image digest and seeds and comparing the content hashes of the outputs.¶
Where current work sits. Pimentel et al. (2019) found only 24% of 863,878 GitHub notebook executions ran without errors and 4% reproduced their results. Notebooks register only after a restart-and-run-all in CI; existing notebook work is expected to start at Level 2 and move up at its next refresh.¶