Proving Nothing Happened: Testing for No Difference

Matthew Reda· July 23, 2026· Practice

The most expensive sentence in measurement is usually some version of "the difference wasn't significant, so there's no difference." A brand pauses a channel in ten markets for six weeks, the holdout–control gap comes back at p = 0.41, and the deck concludes the channel does nothing. The cheaper creative "performed the same" as the expensive one. The new bidding strategy "didn't hurt" conversion. Every one of these claims asserts an absence, and the test that produced it is structurally incapable of supporting that claim. A significance test can reject "no difference." It can never affirm it, because a large p-value conflates two states the decision desperately needs to distinguish: the difference is genuinely negligible and the data are too noisy to tell. Distinguishing them is a solved problem, twice over. Frequentist statistics flips the hypotheses: pre-specify the largest difference that would still count as "the same," then reject the hypothesis that the difference exceeds it (equivalence testing, TOST). Bayesian statistics puts posterior mass on the claim itself, either the probability that the difference lies inside a region of practical equivalence or a Bayes factor quantifying how much the data favor the null. Both routes share an uncomfortable feature the naive reading hides: proving nothing happened takes more evidence than proving something did.

A Machine That Cannot Say "Same"

The asymmetry is built into the logic of significance testing. Careless users did not put it there. A null-hypothesis test starts by assuming no difference, computes how surprising the observed gap would be under that assumption, and reports the surprise as p. Small p: the data are hard to reconcile with "no difference," reject. Large p: the data are compatible with no difference, along with a continuum of non-zero differences the same data are equally compatible with. Fisher (1935), who built the machine, was explicit about its one-way valve: "the null hypothesis is never proved or established, but is possibly disproved, in the course of experimentation." The test is an alarm. An alarm that stays silent is no certificate that nothing is burning, because the battery may simply be dead.

The battery, here, is power. When a test has an 80% chance of detecting the effect you care about, silence carries some information. When it has a 20% chance, silence is the expected outcome whether or not the effect exists, and reading it as absence is reading tea leaves. Altman and Bland (1995) compressed the point into the most-cited three-word slogan in applied statistics: absence of evidence is not evidence of absence. Greenland and colleagues (2016) rank the inversion ("no significant difference" read as "no difference") among the most common and most damaging p-value misinterpretations, and it survives because the sentence sounds like a finding. It is a description of the test, not the world.

There is also a subtler failure mode on the same road: treating "A was significant, B was not" as if the pair of verdicts established that A and B differ. That is the Gelman–Stern fallacy, covered in a companion post. This post is about the direct version: one comparison, one non-significant result, and the leap from silence to sameness.

What a Null Result Rules Out

The first honest question to ask of any non-significant result is: what could this test have seen? At 80% power the minimum detectable effect (MDE) is roughly \( 2.8 \times \mathrm{SE} \), the sum of the significance threshold (1.96) and the power requirement (0.84), times the standard error. Say a holdout test's standard error on lift is 4 percentage points. The test was built to notice a 11-point swing, so "no significant loss" is compatible with losing 8 points of sales, an outcome that would be catastrophic for the decision the test is supposedly informing. Stating the MDE next to every null result is the cheapest inferential hygiene available, and it is remarkable how many "the test showed no impact" slides cannot survive it.

Advertising is the canonical domain where this bites, because the signal-to-noise ratio of ad effects on sales is brutally low. Lewis and Rao (2015) analyzed twenty-five large digital ad experiments (millions of users each, run at firms with excellent data) and showed that even at that scale, confidence intervals on advertising ROI routinely spanned from strongly negative to wildly positive. Individual-level purchase variance is enormous relative to any plausible per-impression effect, so the experiments that would settle "does this campaign pay back" need sample sizes that most advertisers cannot buy. The corollary for smaller tests is grim. A six-week, ten-market holdout is almost guaranteed to return "no significant difference" for any channel that isn't heroic or terrible, and that silence gets reported as a finding.

One tempting patch does not work: computing "observed power" after the fact, meaning the power the test would have had if the observed estimate were the truth. Hoenig and Heisey (2001) showed that observed power is a deterministic function of the p-value and adds exactly nothing. Every non-significant result has low observed power, by arithmetic, so "the result was null and observed power was low" is one fact stated twice. Power calculations belong to design, run against effect sizes that matter for the decision. After the data arrive, the honest summary of what the study ruled out is the confidence interval, and if that interval contains both zero and differences that would change the decision, the study is inconclusive, full stop.

Flipping the Null: TOST

If "no difference" is the claim you actually want to establish, frequentist logic has a clean fix: make "there is a difference" the null hypothesis, so that the machinery's one-way valve points in the direction you need. The catch is that "no difference exactly" is unprovable. No finite sample can distinguish a true difference of zero from one of 0.0001. So the claim has to be made falsifiable first, by choosing an equivalence margin \( \Delta \): the largest difference that would still count, for this decision, as "the same." Then test two one-sided null hypotheses (Schuirmann, 1987):

each at level \( \alpha \). Reject both and you have affirmatively concluded \( -\Delta \lt \delta \lt +\Delta \): the difference, whatever it is, is too small to matter. This is the two one-sided tests procedure, TOST, and it is the standard of evidence behind every generic drug on the market. Bioequivalence approval requires demonstrating that generic and brand pharmacokinetics agree within a pre-registered margin (the FDA's 80–125% window), not that a difference test came back silent. Lakens (2017; Lakens, Scheel, and Isager, 2018) did the field service of translating the machinery into a practical workflow for everyday two-group comparisons.

TOST has a convenient dual form. Rejecting both one-sided hypotheses at \( \alpha = 0.05 \) is exactly the statement that the 90% confidence interval lies entirely inside \( (-\Delta, +\Delta) \). One picture then holds the whole inferential situation: an interval, a zero line, and two margin lines. Out of that picture come four possible verdicts instead of two. A difference can be significant and equivalent, which means a real but trivial gap, precisely measured. Significant and not equivalent is a difference that matters. Non-significant and equivalent is the honest "no difference." And non-significant and not equivalent is the underpowered quadrant, where the only defensible conclusion is that the data don't know. The naive reading collapses the last two quadrants into one, and they could not be more different.

The Price of Proving Nothing

Equivalence has an arithmetic that surprises people who are used to sizing difference tests: it usually costs more. To detect a difference of size \( \Delta \) with 80% power you need \( \Delta / \mathrm{SE} \approx 2.8 \). To declare equivalence within \( \pm\Delta \) when the true difference is exactly zero you need \( \Delta / \mathrm{SE} \approx 2.93 \), because the 90% interval, of half-width \( 1.645\,\mathrm{SE} \), must fit inside the band with room for sampling noise. Similar, so far. But the true difference is rarely exactly zero, and here the cost curve turns vicious: the 90% interval has to fit into the band left over between the truth and the nearer margin, so the required sample grows like the inverse square of that remaining gap. That is roughly three times the detection budget when the truth sits at half the margin, and more than twelve times at three-quarters. Proving that a difference is negligible is cheap only when it is negligible with room to spare.

That last property deserves a sentence of appreciation, because it is the whole point. Under the naive reading, shrinking the sample helps the "no difference" conclusion: noise manufactures silence, and silence gets read as sameness. Under TOST, shrinking the sample destroys the conclusion, because with \( 1.645\,\mathrm{SE} \gt \Delta \) the 90% interval cannot fit inside the band no matter where it lands, and the procedure simply refuses to certify. A method for proving absence must get harder to satisfy as the data get worse. That is the test of whether your "no difference" pipeline is real.

ROPE, HDIs, and Bayes Factors

The Bayesian toolkit attacks the same problem from the other side: instead of engineering a rejection machine that points the right way, put a posterior distribution on the difference and interrogate it directly. Two instruments dominate practice, and they answer subtly different questions.

The first is the Bayesian sibling of TOST. Define a region of practical equivalence (ROPE) around zero (the same decision-derived \( \pm\Delta \) as before) and read the posterior against it (Kruschke, 2018). The continuous summary is \( P(|\delta| \lt \Delta \mid \text{data}) \), the posterior probability that the difference is negligible, which is a number a stakeholder can use directly. The discrete decision rule mirrors TOST's geometry: accept practical equivalence when the 95% highest-density interval falls entirely inside the ROPE, reject it when the HDI falls entirely outside, and withhold judgment otherwise. With flat priors the two procedures nearly coincide. That correspondence is reassuring, because it means the real split was never frequentist against Bayesian. It was bounding the difference against reading silence.

The second instrument has no frequentist counterpart at all. The Bayes factor compares how well two models predicted the data: a point null \( \delta = 0 \) against an alternative that spreads \( \delta \) over a prior (Rouder et al., 2009; Jeffreys, 1961). Where a p-value can only fail to reject the null, \( \mathrm{BF}_{01} \) can accumulate evidence for it, because data tightly concentrated near zero are better predicted by the null model than by an alternative that wagered on effects that never materialized. For the normal case the computation collapses to one line via the Savage–Dickey ratio, the posterior density at zero over the prior density at zero (Wagenmakers et al., 2010). Dienes (2014) makes the practical case. A Bayes factor turns a non-significant result from a shrug into a graded statement that separates "the data support no effect" (\( \mathrm{BF}_{01} \gt 3 \)) from "the data are simply insensitive" (\( \mathrm{BF}_{01} \approx 1 \)).

Which Tool for Which Claim

The instruments line up by the claim you need to make, and by what you must specify in advance to be entitled to it.

Two disciplines apply across all four. The margin is a decision quantity, fixed before the data are seen. A margin tuned after the fact inherits every pathology of the garden of forking paths. And a null claim earned in one analysis does not survive selection across many: screen twenty channels for "no loss when cut" and the winning null is as cursed as the winning effect.

# illustrative
# WRONG: silence read as sameness
p = ttest(holdout, control).p          # p = 0.41
# "no significant loss: cut the channel"   <- underpowered silence, not absence

# RIGHT, floor: say what the test could see
mde = 2.8 * se                          # the smallest loss this test could detect
ci  = (d - 1.96 * se, d + 1.96 * se)    # spans break-even?  then it's inconclusive

# RIGHT, frequentist: equivalence within a pre-registered margin
p_tost = max(1 - cdf((d + margin) / se),    # H01: loss beyond -margin
             cdf((d - margin) / se))        # H02: gain beyond +margin
equivalent = p_tost < 0.05             # == 90% CI inside (-margin, +margin)

# RIGHT, Bayesian: probability the difference is negligible
p_rope = P(abs(delta) < margin | data)  # posterior mass in the ROPE
bf01   = post_pdf_at_0 / prior_pdf_at_0 # Savage-Dickey: evidence FOR the null

What It Means for Media

Media measurement produces claims of absence at an industrial rate, and almost all of them are the naive kind. "We went dark in Denver and nothing happened." "The cheaper creative performed the same." "Cutting the retargeting budget had no effect on conversions." Each one is a budget decision wearing a p-value, and each one should be re-read with the machinery above, starting with the question the geo test's design already answered: what loss could this test have detected? Six weeks and ten markets typically buys an MDE of several points of sales, so a channel driving a 2–3% lift (a perfectly good channel) will read "no significant difference" most of the time. Lewis and Rao's arithmetic says this is the default condition of ad measurement rather than a fixable sloppiness. The tests that would separate "nothing" from "normal-sized effect" are bigger than the tests people run. Silence is cheap. Bounded silence costs markets and weeks.

The saving grace is that media, unlike psychology, gets its equivalence margin handed to it by accounting. Pausing a channel saves its spend \( S \). The pause is profitable exactly when the forgone contribution margin is smaller than \( S \), so the break-even lift \( \Delta^* = S / m \) (with \( m \) the margin per incremental unit) is the non-inferiority margin. Media reads that number off the ledger the CFO already owns, with no convention borrowed from pharmacology. "Going dark was fine" then becomes a testable, one-sided claim: the lower confidence bound on the loss stays inside break-even, or the posterior probability that the loss exceeds \( \Delta^* \) is small. This framework runs the Bayesian version natively. The experiment feeds the model, and the model's posterior prices the pause against the saved spend, so a holdout can come out net-positive once saved dollars are counted. Before a single market goes dark, the design tools state the assurance of reaching a verdict. An underpowered test gets declined rather than run and misread. And when the posterior can only say "somewhere between fine and painful," the honest output is that interval plus the longer, wider test that would shrink it, because adstock smears a pause's cost past a short window and biases exactly these tests toward comforting silence.

The deepest habit worth breaking is grammatical. "No significant difference" is a sentence about a test. "The difference is between −1.2 and +0.8 points, inside break-even" is a sentence about the world. Decisions deserve the second kind. If the analysis cannot produce one, with no margin, no bound, and no posterior, then it has not found that nothing happened. It has found nothing, which is not the same thing at all.

References

• Altman, D. G., & Bland, J. M. (1995). Absence of Evidence Is Not Evidence of Absence. BMJ, 311(7003).
• Fisher, R. A. (1935). The Design of Experiments. Oliver & Boyd.
• Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical Tests, P Values, Confidence Intervals, and Power: A Guide to Misinterpretations. European Journal of Epidemiology, 31(4).
• Hoenig, J. M., & Heisey, D. M. (2001). The Abuse of Power: The Pervasive Fallacy of Power Calculations for Data Analysis. The American Statistician, 55(1).
• Schuirmann, D. J. (1987). A Comparison of the Two One-Sided Tests Procedure and the Power Approach for Assessing the Equivalence of Average Bioavailability. Journal of Pharmacokinetics and Biopharmaceutics, 15(6).
• Lakens, D. (2017). Equivalence Tests: A Practical Primer for t Tests, Correlations, and Meta-Analyses. Social Psychological and Personality Science, 8(4).
• Lakens, D., Scheel, A. M., & Isager, P. M. (2018). Equivalence Testing for Psychological Research: A Tutorial. Advances in Methods and Practices in Psychological Science, 1(2).
• Wellek, S. (2010). Testing Statistical Hypotheses of Equivalence and Noninferiority (2nd ed.). Chapman & Hall/CRC.
• Kruschke, J. K. (2018). Rejecting or Accepting Parameter Values in Bayesian Estimation. Advances in Methods and Practices in Psychological Science, 1(2).
• Kruschke, J. K., & Liddell, T. M. (2018). The Bayesian New Statistics: Hypothesis Testing, Estimation, Meta-Analysis, and Power Analysis from a Bayesian Perspective. Psychonomic Bulletin & Review, 25(1).
• Rouder, J. N., Speckman, P. L., Sun, D., Morey, R. D., & Iverson, G. (2009). Bayesian t Tests for Accepting and Rejecting the Null Hypothesis. Psychonomic Bulletin & Review, 16(2).
• Wagenmakers, E.-J., Lodewyckx, T., Kuriyal, H., & Grasman, R. (2010). Bayesian Hypothesis Testing for Psychologists: A Tutorial on the Savage–Dickey Method. Cognitive Psychology, 60(3).
• Dienes, Z. (2014). Using Bayes to Get the Most out of Non-Significant Results. Frontiers in Psychology, 5.
• Rouder, J. N. (2014). Optional Stopping: No Problem for Bayesians. Psychonomic Bulletin & Review, 21(2).
• Tendeiro, J. N., & Kiers, H. A. L. (2019). A Review of Issues About Null Hypothesis Bayesian Testing. Psychological Methods, 24(6).
• Lewis, R. A., & Rao, J. M. (2015). The Unfavorable Economics of Measuring the Returns to Advertising. The Quarterly Journal of Economics, 130(4).
• Jeffreys, H. (1961). Theory of Probability (3rd ed.). Oxford University Press.