Guide · doing the work
Priors that mean something
The objection to Bayesian methods is that priors are subjective. The reply is not that they are objective. It is that the alternative — a procedure with no prior — is not neutral either; it is a procedure whose assumptions are simply not written down anywhere you can attack them.
What a prior actually is
A statement about magnitudes, in units
A useful prior is not “I am fairly uncertain.” It is a sentence like: a filter in a household reduces reported illness by somewhere between one and four days per twelve weeks; two would not surprise me; ten would astonish me.
That sentence is checkable. Somebody who has run three of these programmes can tell you it is wrong, and why. It also has consequences — it says that an estimate of thirty days is more likely a data problem than a discovery, which is a claim you want your analysis to make rather than one you want to make manually after the fact.
A prior stated as “normal(0, 10) on the coefficient” is not that sentence. It may encode the same belief or it may encode something wild, and you cannot tell without knowing the scale of the outcome, the scale of the treatment, and what the coefficient multiplies.
The test
Can you state your prior as a claim about the outcome, in the outcome's own units, to somebody with domain knowledge and no statistics? If not, you have a regularization setting, not a prior — and you should not be surprised when it does something you did not intend.
“Weakly informative” can be a strong claim
A wide prior on a parameter that enters multiplicatively, or through a saturating transform, can put most of its mass on physically impossible responses. Wide is not the same as humble. The only way to know is to push the prior through the model and look at the outcomes it implies.
The check that catches most problems
Simulate from the prior and look at what it predicts
Before any data. Draw parameters from the prior, push them through the model, and look at the outcomes. If it routinely produces effects nobody in the room believes are possible, the prior is wrong — and you have found out for free.
A prior, pushed through the model
Effects implied by draws from the prior on the design page. The prior is a claim about this distribution, not about a coefficient.
axiom.design prior draws · mean 5.59 · sd 1.15
What this prior believes
Central mass between 3.73 and 7.43 on an outcome whose own standard deviation is 4.55 — an effect around one standard deviation, which is a strong claim that somebody should be made to defend.
And it has a consequence you can act on
This prior puts 99.9% of its mass above the conventional minimum detectable effect of 2. In other words, the experiment as conventionally sized would be spending money to establish something the team already believes. That is a design finding, and it came from taking the prior seriously — see deciding what to measure.
Where priors come from
Four sources, in descending order of comfort
axiom.meta is for, and
the pooled interval it produces is usually wider than any single study's — which
is the correct and unwelcome answer.
strong
Whichever source, record it
A prior derived from an experiment should carry the line that says so. On the calibrate page the prior derived from a measurement arrives with its provenance attached, so the posterior can be traced back past the fit to the experiment that shaped it. A number whose prior came from nowhere recorded is a number with a missing premise.
Afterwards
Two checks, and one thing not to do
Did the data move it?
Compare prior and posterior. If they are identical, the data said nothing about this parameter and any conclusion is your prior wearing a disguise. If the posterior is far outside the prior's plausible range, either the prior or the model is wrong — investigate before celebrating.
Does it change the decision?
Refit with the two other priors your colleagues would have chosen. If the decision is the same under all three, the prior was not load-bearing and the argument about it is a waste of a meeting. If it flips, that is the finding — and it tells you what evidence is worth buying.
Do not tune it to the answer
Adjusting a prior after seeing the posterior, until the result looks right, is not Bayesian updating. It is the same failure as p-hacking with more machinery. Writing the prior down and hashing it before the fit makes this hard to do by accident and visible when done on purpose.