Guide · doing the work
What would have to be true to overturn this?
Every causal estimate rests on assumptions that cannot be checked from the data — that is what makes them assumptions. The productive response is not to defend them. It is to work out how wrong they would have to be to matter, and report that.
The reframe
Stop arguing about whether there is an unmeasured confounder
“Have you accounted for everything?” is unanswerable, and every meeting that asks it ends in a draw. Nobody can prove a negative about variables nobody measured.
The answerable version is quantitative: how strong would an unmeasured confounder have to be — in its association with both the treatment and the outcome — to move this estimate past the boundary the decision turns on?
Now the conversation has content. The answer is a number, and the room can judge it against what they know: is a confounder that strong plausible here, given everything already adjusted for? Sometimes obviously yes, and you have learned the result is fragile. Sometimes obviously no, and the objection is settled on the merits rather than by seniority.
From this build: the robustness value
A confounder would have to explain 86.2% of the residual variation in both the treatment and the outcome to reduce this estimate to zero. The estimate is 2.726; the truth in this simulation is 2, so the actual bias is +0.726.
Read that number against something
A robustness value is not high or low in the abstract. Benchmark it: how much of the residual variation does the strongest covariate you did measure explain? If an unmeasured confounder would have to be three times as strong as the strongest thing you know about, the result is robust. If half as strong would do it, it is not.
Three families of check
What to run, and what each one can tell you
Sensitivity — how much unmeasured confounding would it take?
Parameterize the confounder you did not measure by its strength, and trace what happens to the estimate as that strength grows. The output is a tipping point: the confounder strength at which your conclusion flips. Report it alongside the estimate, always.
this build: bias vanishes at a confounder explaining 86.2% of residual variation in both
Refutation — does the method find effects that cannot exist?
Replace the treatment with random noise; the estimated effect should be zero. Add an unrelated common cause; the estimate should not move. Use an outcome that precedes the treatment; there should be nothing there. These do not validate the result — they catch pipelines that would have produced a confident answer from anything at all, which is more common than it should be.
axiom.dynamics ships these as refutation tests
against a fitted graph
Specification — does the conclusion survive the choices you could have made?
Refit under the other defensible graph, the other adjustment set, the other functional form, the prior your colleague would have picked. If the decision is stable across all of them, say so — that is a stronger claim than any single fit. If it is not, you have found what the answer actually depends on.
two valid adjustment sets are a check, not a choice — see identification
How to report it
A result is the strength of the objection it survives
The weak form: “The effect is 2.73 (95% CI 2.69–2.76).” Precise, and it tells a reader nothing about whether to believe it.
The strong form: “The effect is 2.73 (95% CI 2.69–2.76), identified via the back door adjusting on Z, under the assumption of no unmeasured common cause. That assumption fails only if a confounder explains 86% of the residual variation in both treatment and outcome; benchmarked against the strongest covariate we did measure, that is [stronger / weaker]. The conclusion is unchanged under the alternative graph in which W confounds rather than mediates.”
The bracketed clause is the one people skip, and it is the one that makes the percentage mean anything. Fill it in.
The second version is longer, harder to write, and much harder to overturn in a meeting — because every objection somebody could raise has already been raised and quantified by the person who did the work.
Report the fragile ones too
“This result flips if a confounder half as strong as the strongest one we measured exists” is a finding, not a failure. It names the risk in terms somebody can weigh, and it tells the organisation not to bet on this number, and it names the study that would settle it. That is worth more than a confident estimate nobody stress-tested.
And check the transfer, last
A result can survive every sensitivity check listed here and still be wrong where it is used, because none of them asks whether the answer is for the right population. That check is transportability, and it is the one the others cannot substitute for.
The end of the shelf
What all twelve guides were driving at
Causal work is not a technique applied to data; it is a chain of commitments — about mechanism, about the question, about who the answer is for — with a number at the end of it. The technique is the easy part. What makes an answer usable six months later, in a room you are not in, is that every one of those commitments was written down where somebody can find it, check it, and disagree with it.
That is what the rest of this site is: the same commitments, made explicit and executable. The tutorials carry five problems all the way through, the examples do it in twelve fields, and the pillar pages show what each part of the library actually does.
Guide 12 of 12