Guide · writing the question down
Provenance: the paper trail
Six months after an analysis, the number survives and everything around it evaporates. The slide says 12%. Nobody can say which data, which window, which adjustment set, or which of the four versions of the model produced it. At that point it is not evidence. It is a rumour with a decimal point.
The problem
Analyses decay in a specific, predictable way
Nobody sets out to lose the context. It goes in a fixed order, and it goes fast. First the alternatives disappear: the three specifications you tried before this one, and why you rejected them. Then the assumptions: the adjustment set, the window, the population, all of which felt too obvious to write down at the time.
Then the question goes. The number gets quoted in a context slightly different from the one it was computed for, then again from there, and by the third hop it is answering something nobody ever measured. Nothing dishonest happened at any step.
What is left is a number and a confident memory. That is the state most organisations are in about most of their evidence, and it is why the same questions get re-analysed every eighteen months.
The design response
Make the trail a property of the object rather
than a discipline of the analyst. Anything that depends on discipline is gone by
the second deadline. In axiom every configuration object is a
Spec: frozen, versioned, and content-hashed, and every transfer of
evidence across a boundary appends a ledger line — not because somebody
remembered to, but because the alternative does not typecheck.
This site is built the same way
Every figure and statistic on these pages was produced by running axiom, and every figure names the call that produced it. The numbers on this page were computed while it was being generated.
Mechanism one
A content hash: the same question has the same name
Every spec hashes its own canonical contents. The hash does not depend on dictionary
insertion order, on PYTHONHASHSEED, or on which machine computed it — so
it is an address for the question rather than an artefact of a process.
adaf9c9abfa4b2fddea9998b92d6ddb3193ff6ff8142626539b65a7a7b276137
adaf9c9abfa4b2fddea9998b92d6ddb3193ff6ff8142626539b65a7a7b276137
identical
854b16405ece6c638495c9f148cc3a5b7ec38d13bb2718e664aa5240dea7cc31
different
round-trip stable: True · one changed field changes the hash: True
The second row is the one that matters. A spec serializes to JSON and back to an equal object, and the hash survives the round trip — so a question stored in a database in March is provably the same question in September.
The third row is the other half. Changing the window from twelve periods to thirteen is a change most workflows would not even record, and it produces a completely different address. You cannot accidentally present the answer to Tuesday's question as the answer to Monday's.
And when two hashes differ, you do not have to hunt for why:
spec='axiom.estimands.spec:Estimand' changed={'window.stop': (12, 13)}
Versioned, so old records stay readable
Each spec class carries a
SCHEMA_VERSION (this one is
1) and serializes through one
envelope — data, schema_version, spec. Loading an older
version applies a registered migration, or raises naming both versions. A stored
analysis does not silently become unreadable because the library moved on, and it
does not silently load wrong either.
Mechanism two
A ledger that refuses to be incomplete
A hash records what the question was. A ledger records what happened to the answer — every boundary it crossed and what was assumed to get it across.
s_admissibility — challenged by overlap; moderator interaction in meta satisfied
carryover_contained — challenged by half-life against window length unverified
linear_aggregation — challenged by the Jensen gap under a nonlinear response unverified
s_admissibility — challenged by a mechanism difference between the populations the diagram omits satisfied
Three properties make this more than logging.
It is complete by construction. The ledger starts from the
transfer plan with one line per differing facet, marked uncorrected. Each
correction replaces a line rather than adding one. So a facet nobody
corrected is still visible — as an explicitly uncorrected facet — instead of being
absent and unnoticed. check_complete then verifies it:
every differing facet accounted for exactly once (population, window, level).
Facts and assumptions look different. A registered unit conversion carries no assumption. An assumption carries a name, a state, and the thing that would refute it. You can read a ledger and see immediately which lines are exposed.
It is addressable. The ledger has its own content hash —
14c74efa68608cace20fd0b80f04d0aae03cac6df32908ce2ad3847ca22bfc36 — so a number in a report can be
traced back to the exact chain that produced it.
How many of these lines are load-bearing
still unverified. That is the honest state of
this transfer, and it is why the overall status is
downgraded rather than
identified.
Downgraded is not a failing grade
It is a description of the evidence. Most real
work is downgraded, permanently and correctly. The alternative — a tool that
reports identified because nothing checked — is not a better
result, it is a worse instrument.
What to take from this
Provenance is what makes evidence reusable instead of disposable
Write the question down and store its hash with the answer
This is the whole practice in one sentence, and it is cheap. Everything else on this page is machinery for making it hard to skip.
Record the alternative you rejected, not only the choice you made
“We used a geometric carryover” is half a record. “We used a geometric carryover rather than a delayed one, because the half-life implied by the second was longer than the observation window” is a record somebody can audit — and revisit when a longer window arrives.
Never let a number and its interval definition be separated
An interval means nothing without knowing what kind it is and at what mass. On
this site they always travel together — e.g.
[1.9849,
2.0404] (95% WALD) — never a bare
pair of numbers.
Make the trail the default path, not the diligent one
If producing the record is a separate step, it will be skipped exactly when it matters most — under deadline, on the analysis that turns out to be important.