P-Hacking and the Thin Evidence of a P-Value

Matthew Reda Practice

The p-value has two problems, and they compound each other. The first is that a result just under the sacred 0.05 line carries far less evidence than its reputation suggests — a "significant" finding can leave the null hypothesis with something like a one-in-four chance of being true. The second is that, precisely because the bar is low, a determined or even well-meaning analyst can clear it by chance, through choices that each look innocent. Weak evidence that is easy to manufacture is a bad foundation for a decision. This post is about why, and what to do instead.

What a P-Value Isn't

Start with the definition, because almost every misuse is a substitution of something the p-value is not. A p-value is the probability, computed under a specified statistical model assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one you got. That is all it is. In 2016 the American Statistical Association took the unusual step of issuing a formal statement of principles (Wasserstein and Lazar, 2016), because the misinterpretations had become a public-health problem for science. Three of its six principles are worth stating plainly, because each is a thing people constantly believe a p-value tells them and it does not.

⚠️ Three things a p-value does not measure

It is not the probability the null is true. A p-value conditions on the null; it cannot also be the probability of the null. \( p = P(\text{data} \mid H_0) \) is not \( P(H_0 \mid \text{data}) \), and confusing them is the transposed conditional — as wrong as equating the chance a person is dead given they were hanged with the chance they were hanged given they are dead. It is not the effect size or its importance. A trivial effect measured with enough data yields a tiny p-value; a large, decision-relevant effect measured noisily yields a big one. And by itself it is not a good measure of evidence. The ASA's sixth principle says this outright.

The gap between \( P(\text{data} \mid H_0) \) and \( P(H_0 \mid \text{data}) \) is not a pedantic quibble — it is the whole story of the second half of this post. But first, the easier failure: even taken at face value, the 0.05 threshold is a bar low enough to trip over on purpose.

Manufacturing Significance

Every analysis involves choices. Which observations are outliers? Should the outcome be logged? Which covariates go in the model? Where does the time window start? Do you stop collecting data now or after another week? Each choice is defensible in isolation. The problem, named by Simmons, Nelson, and Simonsohn (2011) as researcher degrees of freedom, is that when the choice is made after seeing which version gives a smaller p-value, the reported 5% error rate is a fiction.

Their demonstration is deservedly famous. Exploiting four common, individually-innocent degrees of freedom — two dependent variables, an optional covariate, optional extra observations, and a subgroup split — they drove the probability of finding at least one "significant" result to 61%, from a nominal 5%. Then they turned the machine on a real experiment and "proved," at \( p \lt 0.05 \), that listening to the Beatles' "When I'm Sixty-Four" made people about a year and a half younger — not younger-feeling, but younger in chronological age, a logical impossibility. The point was not that the song is magic; it was that the method will certify anything.

💡 You don't have to be cheating

The unsettling version, which Gelman and Loken (2014) call the garden of forking paths, needs no p-hacking at all. Even an honest analyst who runs a single test inflates their error rate if the test they would have run depends on the data — a different pattern in the numbers would have led to a different, equally reasonable, analysis. The multiple comparisons are latent: they are all the analyses you didn't run but would have, given other data. This is why "but I only ran one regression" is not a defense, and why the problem surfaces even for a hypothesis posited in advance. It is a close cousin of the winner's curse across model specifications, and both are treated at more length in that companion post.

# illustrative
# The garden of forking paths: significance from choices, not effects.
# None of these steps is individually dishonest.
for outcome in [y_raw, log(y)]:            # which outcome scale?
    for controls in candidate_control_sets:  # which covariates?
        for window in [full, last_2y, ex_covid]:  # which period?
            p = fit(outcome, controls, window).pvalue
            if p < 0.05:
                report(outcome, controls, window)  # stop and publish
# With enough forks, a p < 0.05 is near-certain even if the effect is zero.

The garden, simulated

Every point is one honest analysis of data where the true effect is exactly zero. Under the null, p-values are uniform on [0, 1], so each dot lands anywhere with equal chance. Keep the smallest one and call it a finding: with enough forks, a dot slips into the red 5% band almost every time. Drag the number of paths and hit Resample — nothing here is cheating, and significance still appears.

12
P(≥1 “significant”)—
False positives this draw—
Smallest p—
Verdict—

The curve 1 − 0.95k reaches the Simmons–Nelson–Simonsohn figure of ~61% at around 18 forks. Their real demonstration hit 61% with just four everyday degrees of freedom — two outcomes, an optional covariate, optional extra data, a subgroup split.

How Much Evidence Is p = 0.05?

Now the deeper problem. Suppose there is no p-hacking at all — one pre-registered test, honestly reported, coming back at exactly \( p = 0.05 \). How much should that move your belief? Much less than the ritual implies, and there is a clean way to see it. Because \( p = P(\text{data} \mid H_0) \) is not \( P(H_0 \mid \text{data}) \), converting one to the other requires Bayes' theorem — and that conversion has a startling lower bound.

Berger and Sellke (1987) asked how small the posterior probability of the null can be pushed when \( p = 0.05 \). For any reasonable prior — one that is unimodal and symmetric about the null — a two-sided test leaves the null with a posterior probability of at least about 0.30; even the absolute floor, minimizing over every prior including ones gerrymandered to favor an effect, only lowers it to roughly 0.13. Sellke, Bayarri, and Berger (2001) sharpened this into a calibration you can compute on a napkin. The most evidence a p-value can possibly carry against the null is bounded by a Bayes factor

$$ \overline{B}(p) \;=\; -e\,p\,\ln p, \qquad p \lt 1/e, $$

and, starting from even odds, the null's posterior probability is at least

$$ \underline{\alpha}(p) \;=\; \left(1 + \frac{1}{-e\,p\,\ln p}\right)^{-1}. $$

Plug in the usual thresholds and the ritual looks threadbare.

p-valueMax Bayes factor for \( H_1 \)Max odds against \( H_0 \)Min posterior \( P(H_0) \), even prior
0.050.412.5 : 1≈ 29%
0.010.138 : 1≈ 11%
0.0050.0714 : 1≈ 7%
0.0010.0253 : 1≈ 2%

The napkin calculator

The Sellke–Bayarri–Berger bound turns any p-value into the most evidence it can carry. Drag your p-value and read off the best-case Bayes factor and the minimum posterior probability the null still holds — starting from even odds and tuning the prior maximally in the effect’s favor. The shaded band is the region where the null keeps a hefty chance of being true no matter what your p-value did.

0.050
Max Bayes factor (H₁)—
Best odds against H₀—
Min posterior P(H₀)—
Reading—

At p = 0.05 the null still holds a ~29% posterior chance and the evidence is barely 2.5 : 1 — the case for the 0.005 threshold. This is a lower bound on P(H₀): realistic priors leave the null even more likely.

Read the top row again. A result that just cleared the most celebrated threshold in science offers, at best, odds of about two and a half to one against the null — and leaves the null with roughly a 29% chance of being true even before you account for any of the flexibility in the previous section. Two-and-a-half-to-one is the kind of edge you would not bet a mortgage on. It is emphatically not "the effect is real." The bound is generous, too: it is the most evidence \( p = 0.05 \) can supply, achieved only by tuning the prior in the alternative's favor; realistic priors give less.

Deep diveWhere \( -e\,p\,\ln p \) comes from

The calibration treats the observed p-value itself as the datum and asks how its density behaves under the null versus the best-case alternative. Under the null, p-values are uniform on \( [0,1] \), so the null density is \( 1 \). If one entertains alternatives under which small p-values are more likely — modeling the p-value's density as \( \mathrm{Beta}(\xi, 1) \) with \( \xi \lt 1 \) — the maximum-likelihood member of that family, evaluated at the observed \( p \), has density \( -e\,p\ln p \) at its most favorable. The ratio of that best-case alternative density to the null density is the Bayes-factor bound \( \overline{B}(p) \); feeding it through Bayes' theorem with equal prior odds gives \( \underline\alpha(p) \). The construction is deliberately charitable to the alternative, which is what makes it a lower bound on how much the null survives — the real posterior on the null is generally larger, not smaller.

The False-Positive Risk

The calibration above assumed a coin-flip prior. In practice, whether a "significant" result is a false alarm depends on how often the hypotheses you test are true in the first place — the base rate — and on the power of the test. This is the same arithmetic that governs a medical screen: a positive result on a rare disease is mostly false positives, no matter how good the test.

Definition: False-positive risk

Let \( \pi \) be the prior probability that a hypothesis is true, \( \alpha \) the significance threshold, and \( 1-\beta \) the power. Among all the times you declare significance, the fraction that are false alarms is

$$ \mathrm{FPR} \;=\; \frac{\alpha\,(1-\pi)}{\alpha\,(1-\pi) + (1-\beta)\,\pi}. $$

The value of a "significant" result is not fixed at 5% — it slides with the base rate \( \pi \) and the power.

Where the false alarms come from

Run 1,000 tests. The base rate sets how many hypotheses are actually true; power sets how many true ones you catch; the threshold sets how many nulls sneak through. The bars show the split among the results you’d declare significant — real effects versus false alarms. In exploratory work, where true effects are rare and power is mediocre, the red bar wins.

0.10
0.50
0.050
Real effects found—
False alarms—
False-discovery rate—

This is Ioannidis’ arithmetic. At π = 0.1 and 50% power, nearly half of your “p < 0.05” findings are already false — and dropping the power or the base rate pushes it toward three in four.

Ioannidis (2005) made this the centerpiece of the most-downloaded paper in the history of PLoS Medicine, "Why Most Published Research Findings Are False." His positive-predictive-value framing adds a bias term, but the skeleton is the formula above: when true effects are uncommon and power is mediocre — the normal condition in exploratory work — most claimed discoveries are false, and no amount of hitting \( p \lt 0.05 \) rescues them. Colquhoun (2014) put concrete numbers on the everyday case. A result observed sitting right at \( p \approx 0.05 \) carries a false-positive risk of at least about 26% even under a generous even-odds prior — reassuringly close to the 29% from the Sellke–Bayarri–Berger calibration by a completely different route. And if only, say, one in ten of your tested hypotheses is actually true, the false-discovery rate among your \( p \lt 0.05 \) "findings" climbs to roughly three in four.

💡 "p < 0.05" and "p = 0.049" are different numbers

A subtlety that trips up even careful analysts: the false-positive rate averaged over all results with \( p \lt 0.05 \) is much lower than the risk attached to a result that landed right at the boundary. The pooled set includes all the strongly-significant results that pull the average down. But when you report \( p = 0.049 \), you have the boundary result, not the average one — and that is the ~26–30% case, not the comfortable single-digit one. The number to quote for a specific finding is the "p-equals" risk, not the "p-less-than" rate.

Redefine, or Retire?

The field's response has split into two camps, and both are instructive. One camp says the threshold is simply in the wrong place. Benjamin and 71 co-authors (2018) proposed lowering the default bar for claiming a new discovery from 0.05 to 0.005, relabeling the 0.005-to-0.05 zone as merely "suggestive." Their justification is precisely the calibration above: \( p = 0.05 \) corresponds to a Bayes factor of only about 2.4 to 3.4 — weak — whereas \( p = 0.005 \) buys a Bayes factor around 14 to 26, which reaches the "substantial-to-strong" range. Move the line and you move the evidence a significant result must carry.

The other camp says the line itself is the problem. Amrhein, Greenland, and McShane (2019), in a Nature comment endorsed by more than 800 signatories, argued to "retire statistical significance" entirely — not to lower the threshold but to stop dichotomizing at any threshold. The dichotomy is what manufactures the garden of forking paths, the file drawer, and the illusion that \( p = 0.049 \) and \( p = 0.051 \) describe different worlds. In the same special issue, the ASA's own editors called for "a world beyond \( p \lt 0.05 \)" (Wasserstein, Schirm, and Lazar, 2019): report effect sizes and their uncertainty, treat the p-value as one continuous piece of evidence among many, and stop letting a threshold make binary decisions that the data cannot support.

These are not really in conflict. Both agree that a single p-value near 0.05 is thin, that thresholds invite gaming, and that the honest output of an analysis is an estimate with its uncertainty, not a verdict. Which is exactly where a Bayesian workflow was already standing.

What It Means for Media

Marketing measurement is a garden of forking paths with unusually lush foliage. A media mix model asks the analyst to choose an adstock form and its decay, a saturation curve and its shape, the lag length, the control variables, the outlier treatment, the transformations, the training window — and then, in most commercial workflows, to select the "best" of hundreds or thousands of candidate model runs. Every one of those is a researcher degree of freedom, and the last one — picking the winner from a large search — is the winner's curse in its purest form. If the reporting boiled down to a significance star next to a channel's coefficient, it would be worth almost nothing, for every reason above.

Three practices follow directly, and they are the framework's posture rather than afterthoughts.

Pre-specify the analysis. The cleanest defense against forking paths is to commit to the model structure, the controls, and the transforms before seeing which choice flatters which channel — the same logic as pre-registration in science, which is why this framework treats a pre-specified analysis as reducing researcher degrees of freedom by construction rather than by discipline. A prior predictive check lets you interrogate that specification before it ever touches the outcome.

Report the posterior, not a verdict. A Bayesian estimate is an effect size with a credible interval — a distribution over how much a channel drove, with its uncertainty attached. That is the "continuous evidence, not a dichotomy" prescription made concrete. A channel whose ROI posterior is wide and straddles break-even is reported as uncertain, not laundered into a binary "significant / not." No threshold to hack, no boundary result to over-read.

Anchor to experiments. The only thing that resolves the base-rate problem — is this apparent effect real, or a false alarm from a low-prior hypothesis? — is a causal design. A geo lift test supplies exactly the kind of high-quality evidence a lone p-value cannot, and folding it in as a prior pulls the whole model toward something an experiment actually verified. The p-value's weakness is, in the end, an argument for the measurement loop: don't ask a single observational test to carry a decision it was never strong enough to bear.

Takeaways

  • A p-value is \( P(\text{data} \mid H_0) \), not \( P(H_0 \mid \text{data}) \), not an effect size, and — by the ASA's own statement — not by itself a good measure of evidence.
  • Researcher degrees of freedom drive the real false-positive rate far above 5% (Simmons et al. hit 61%); the garden of forking paths does it even without conscious p-hacking.
  • At \( p = 0.05 \), the most evidence possible is a Bayes factor of about 2.5:1, leaving the null with roughly a 29% posterior probability even when the comparison is stacked in the alternative's favor (Sellke–Bayarri–Berger).
  • The false-positive risk of a just-significant result is ~26–30% at even odds and climbs to ~75% when true effects are rare — it depends on the base rate and power, not on the threshold.
  • The reform consensus: lower the bar to 0.005 for discovery claims, or retire the dichotomy entirely and report effect sizes with uncertainty.
  • For MMM: pre-specify the analysis, report the full posterior instead of a significance verdict, and anchor to experiments — the weakness of a single p-value is the case for the whole measurement loop.

References