A Smaller P-Value Is Not More Signal
Every measurement team has seen the slide. Search shows p < 0.001 in this year's revenue model, TV managed only p = 0.21 in last year's model, so search "carries more signal" and the budget should follow. The comparison feels rigorous because both numbers came from regressions. It is meaningless, for two separate reasons. First, a p-value measures how incompatible one estimate is with zero, on a scale set by that regression's own sample size, outcome noise, predictor variation, and collinearity. How much signal a variable carries is nowhere in that list, and the scale itself changes every time the model does. Comparing p-values across regressions compares the rulers, not the things measured. The second reason bites even when the rulers happen to match, because a pair of p-values is still no comparison at all. The difference between "significant" and "not significant" is not itself statistically significant. And in the regime where the ranking is most tempting, two weak variables with the null entirely plausible for both, the contest degenerates completely. Which one posts the lower p-value is, exactly, a coin toss.
What a t-Statistic Is Made Of
Start with what the number is actually made of, because the argument is arithmetic before it is philosophy. In a linear regression, the p-value attached to a coefficient \( \beta_j \) is a monotone transform of its t-statistic. Sorting variables by p-value is sorting them by \( |t| \). And the t-statistic factors cleanly:
$$ t_j \;=\; \frac{\hat\beta_j}{\widehat{\mathrm{SE}}(\hat\beta_j)}, \qquad \widehat{\mathrm{SE}}(\hat\beta_j) \;=\; \frac{\hat\sigma}{\sqrt{n}\; s_{x_j} \sqrt{1-R_j^2}}, $$up to a degrees-of-freedom correction, where \( \hat\sigma \) is the residual standard deviation of the outcome, \( n \) the number of observations, \( s_{x_j} \) the standard deviation of the predictor in the sample, and \( R_j^2 \) how well the other covariates predict this one. That last piece is the collinearity term, whose reciprocal \( 1/(1-R_j^2) \) is the familiar variance inflation factor. Rearranged:
$$ t_j \;=\; \underbrace{\frac{\hat\beta_j}{\hat\sigma}}_{\text{effect per unit of noise}} \;\times\; \underbrace{s_{x_j}}_{\text{how much } x_j \text{ varied}} \;\times\; \underbrace{\sqrt{n}}_{\text{how long you watched}} \;\times\; \underbrace{\sqrt{1-R_j^2}}_{\text{how un-collinear it was}}. $$⚠️ Five dials, one effect
The t-statistic (and therefore the p-value) moves with five quantities: the effect size \( \hat\beta_j \), the outcome noise \( \hat\sigma \), the sample size \( n \), the predictor's variation \( s_{x_j} \), and its collinearity with everything else \( R_j^2 \). Only the first is the thing you mean by "signal." The other four are properties of the design: what you measured, for how long, how much the variable happened to move, and what else you put in the model. Two regressions differ in all four, so the ordering of two p-values can freely invert the ordering of the two effects.
Within a single model, sorting the coefficient table by significance stars at least holds \( \hat\sigma \) and \( n \) fixed. It still rewards the variables that happened to vary more and penalizes the collinear ones. And it still doesn't test any variable against another. Across two models, nothing is held fixed. A variable with double the effect can post a p-value a thousand times larger simply because its regression had a noisier outcome, a shorter window, or a co-moving covariate. The figure below makes that concrete: channel B's effect is 1.5× channel A's, and its p-value is worse by three orders of magnitude.
The bigger effect loses
Two channels from two different regressions, both predictors scaled to unit variance so the coefficients are in comparable units. Channel A is fixed: effect 0.30, a three-year weekly window (n = 156), quiet outcome, no collinearity, p ≈ 0.0002. Channel B starts with a bigger effect (0.45) but one year of data, a noisier outcome, and a co-flighted partner channel. Drag B's design dials and watch its p-value swing wildly while its effect never changes.
Normal-approximation tests for clarity. At the defaults, B's effect is 1.5× A's while its p-value is ~1,500× larger. Every bit of the reversal is design: a quarter of the data, twice the noise, and a variance inflation factor of 2.5. Nothing about the channels themselves differs in the direction the p-values suggest.
Significant vs. Not Is Not a Test
Suppose the designs really were identical: same outcome, same window, same controls, same variation. Reading "A significant, B not, therefore A > B" is still wrong, and this failure has its own literature. Gelman and Stern (2006) put it in the title of their paper, the difference between "significant" and "not significant" is not itself statistically significant. Their example takes ten seconds to absorb. One study estimates an effect of 25 with standard error 10, giving z = 2.5, comfortably significant. A second, independent study estimates 10 with standard error 10, giving z = 1.0, nowhere near the line. The comparison everyone wants to make is "the first effect is real, the second isn't." But the difference between the two estimates is \( 15 \pm \sqrt{10^2 + 10^2} \approx 15 \pm 14 \), barely one standard error from zero, p ≈ 0.29. The data are entirely consistent with the two effects being identical. The verdicts differ. The evidence for a difference is nil.
This is not a rare slip by careless analysts. Nieuwenhuis, Forstmann, and Wagenmakers (2011) audited every article across five top journals (Science, Nature, Nature Neuroscience, Neuron, and the Journal of Neuroscience) over two years. Of 157 papers that compared two effects, 78, almost exactly half, did it by comparing significance verdicts rather than testing the difference. Half of them, in the most selective journals in science, made a comparison the arithmetic does not support. There is no reason to expect marketing analytics to do better, and some reason to expect worse: media models produce more coefficients per analysis and face more pressure to rank them.
The Gelman–Stern demonstration
Two independent estimates, each with standard error 10. The curves are their sampling distributions, and the shaded one is the distribution of their difference (standard error √200 ≈ 14). At the defaults one estimate is significant and the other is not, yet the difference curve comfortably straddles zero. Drag the estimates and watch how far apart they must be before the comparison itself clears the bar.
At 25 vs 10, the verdicts split (“significant” vs “not”) while the test of the difference reads p ≈ 0.29. No evidence the effects differ at all. With SE = 10 on both sides, the estimates need to sit roughly 28 apart before the comparison itself is significant.
Deep diveThe test the two p-values are impersonating
For two coefficients in the same regression, the difference has variance \( \mathrm{Var}(\hat\beta_1 - \hat\beta_2) = V_{11} + V_{22} - 2V_{12} \). The covariance term matters, and with collinear predictors it can dominate. For coefficients from two models fit to the same sample (say, the same outcome with and without a covariate), the estimates are correlated across models, and the honest comparison needs the machinery of Clogg, Petkova, and Haritou (1995). Only for genuinely independent samples does the simple form apply, \( z = (\hat\beta_1 - \hat\beta_2)\,/\,\sqrt{\mathrm{SE}_1^2 + \mathrm{SE}_2^2} \), the estimator Paternoster and colleagues (1998) had to remind an entire field to use. A related cousin: judging the comparison by whether two 95% confidence intervals overlap is also wrong, just in the conservative direction (Schenker and Gentleman, 2001). Overlapping intervals can hide a significant difference, and split verdicts can hide an absent one. The fix in every case is to compute a statistic for the difference, with its own uncertainty.
Different Regressions, Different Rulers
There is a deeper problem than the missing difference test, and it is what makes the cross-model version of this mistake unfixable by a formula. A p-value has exactly one calibration: under its own null hypothesis, within its own model, it is uniformly distributed. That is the entire promise. Carry the number outside that model and the guarantee evaporates. There is no sense in which p = 0.01 "means the same thing" in a regression with 52 noisy observations as in one with 500 quiet ones. Goodman (2008) lists this among his dozen p-value misconceptions: studies with the same p-value do not provide the same evidence. Greenland and colleagues (2016), in their catalogue of misinterpretations, single out precisely the practice this post is about, treating two results as "conflicting" because their p-values land on opposite sides of 0.05.
The cleanest way to feel this is to hold the true effect fixed and let the design move. Because \( t = \delta\sqrt{n} \) for a standardized effect \( \delta \), the p-value is partly a sample-size readout: a weak effect observed patiently will always out-significance a strong effect observed briefly. Push it further and you reach Lindley's (1957) paradox. At a large enough sample size, a result sitting exactly at p = 0.05 is evidence for the null under any reasonable Bayesian reading, because the effect required to produce that p-value has shrunk to something the alternative never predicted. Same digits, opposite conclusion. A number whose meaning inverts with n cannot rank variables across models with different n.
A p-value is partly a sample-size readout
Two true effects, one 2.5× stronger than the other, and the p-value each would produce as the sample grows. The strong effect is measured on a short window, the weak one on a long window. Drag the two sample sizes. The weaker signal “wins” whenever it is watched long enough, which is precisely why the digits cannot be compared across designs.
At the defaults the 2.5×-weaker effect posts p ≈ 0.014 while the stronger one posts p ≈ 0.11. Ranking by p-value here ranks patience, not signal. Both curves eventually cross any threshold you like. Every nonzero effect becomes “significant” if you wait.
And even this understates the problem, because two regressions rarely differ only in n and σ. They differ in covariate sets, and conditioning changes what the coefficient is. A channel's coefficient adjusted for branded search answers a different causal question than the same channel's coefficient adjusted for price and seasonality alone. That is the Table 2 fallacy, and it means that across specifications, even a formally correct difference test may be comparing answers to different questions. The p-value comparison fails twice over: the rulers differ, and so do the quantities being measured.
Choosing Variables by Coin Toss
This ranking does its worst damage well before the slide, inside model construction. Stepwise procedures, the habit of dropping insignificant controls, and the rule of keeping whichever of two candidate variables looks stronger all use p-values to decide which variables to include. If a p-value measured information content, this is exactly where it would earn its keep. Instead the procedure degenerates in the one regime where variable selection is genuinely in doubt, among candidates that carry little information and for which the null is entirely plausible. There the choice is a fair coin, in the exact arithmetic sense.
⚠️ The coin-toss regime
Recall the p-value's one calibration: under its own null, it is uniformly distributed. Now let that guarantee work against you. If two candidate variables are both true nulls, whether in the same regression or in two different ones, their p-values are two draws from Uniform(0, 1), and for independent draws
$$ P(p_1 \lt p_2) \;=\; \tfrac{1}{2}, $$exactly. No approximation, no asymptotics, and regardless of how the two designs differ, because uniformity is the one property both numbers share. The same symmetry holds for two exchangeable null candidates within one model. Which variable posts the lower t-statistic is a fair coin, and picking the "winner" for inclusion is flipping it. The perversity is that the coin is fairest precisely when the decision is hardest. Thinner information in the candidates means a contest closer to pure chance. And the analyst who reaches for a p-value to break a tie gets the tie broken by a coin toss, out of sight.
Signal buys surprisingly little edge. Suppose variable A has a genuine effect while B is pure noise, and A's design gives it an expected t-statistic of 1.0, a real but weak signal, the everyday case for marginal controls and small channels. The probability A wins the p-value contest is only about 64%. At an expected t of 0.5 it is 54%, a coin with a nearly invisible bend. Even a variable whose expected t is 2.0, averaging right at the significance line, loses to pure noise about one contest in seven. A selection rule that picks the noise variable over the real one 15% to 46% of the time is sampling its own error term.
The contest, replayed 700 times
Each dot is one replication of the same two regressions: A's p-value against B's. Dots above the diagonal are contests A "wins" (its p-value is the smaller one), and dots below it are B's. Set both true signals to zero and the cloud splits the diagonal exactly, a fair coin whatever the designs. Give A a real but weak signal and watch how little the odds move. The exact win probability is computed alongside the draw.
At the defaults (a real signal with expected t = 1.0 against pure noise) the real variable wins only ~64% of contests. Both signals at zero is the theorem: exactly 50–50, because two null p-values are two independent uniforms no matter what the two regressions look like.
With more than two candidates the lottery compounds into something worse. It manufactures winners. The smallest of \( m \) null p-values follows a \( \mathrm{Beta}(1, m) \) distribution, with expectation \( 1/(m+1) \). Screen twenty pure-noise candidates and the best p-value averages 0.048, arriving just under the threshold on schedule, out of nothing. Freedman (1983) built the canonical demonstration: 100 observations, 50 regressors of pure noise, every true coefficient zero. The full regression "explains" \( R^2 \approx 0.5 \), exactly what the overfitting arithmetic \( E[R^2] = k/(n-1) \) predicts. Drop the variables with weak t-statistics, refit the survivors, and the second-pass equation looks publishable: a highly significant overall F-test and individually significant coefficients, in data that contained no relationships at all. Every step used the p-value exactly as the significance-based selection recipe prescribes.
And the damage outlives the selection step. The surviving model's standard errors and p-values are computed as if the specification had been fixed in advance, when it was in fact chosen by the data, so the reported inference is invalid and generically too optimistic (Berk et al., 2013). The surviving coefficients are biased away from zero, because clearing a significance bar selects for lucky noise on top of any real effect. That is the winner's curse again, now baked into the model rather than the slide. Harrell (2015) catalogues the full wreckage of stepwise selection: inflated \( R^2 \), coefficients biased away from zero, standard errors biased low, p-values without their nominal meaning. P-value selection fails coming and going. Near the null the choice is a coin toss, and far from it the reporting that follows the choice is broken.
The alternatives are the same ones the rest of this post keeps arriving at. Choose variables on causal and substantive grounds, so that a control enters because the causal structure says it must rather than because a star appeared. Shrink instead of selecting: a hierarchical or regularizing prior keeps every candidate in the model and lets the data pull the weak ones toward zero, which replaces a sequence of coin-toss in/out verdicts with partial pooling. And when inclusion is genuinely about predictive value, measure that directly. The cross-validated \( \Delta \)ELPD with its standard error is a comparison statistic, and a pair of significance verdicts is nothing of the kind.
What "More Signal" Could Mean
"Which variable carries more signal?" is a fair question. It just needs to be asked precisely, because it has at least three defensible meanings, each with its own honest metric. None of them is a p-value.
Definition: three versions of "more signal"
A bigger effect. If signal means causal importance, the metric is the effect size on a decision-relevant scale (an elasticity, an incremental-sales-per-dollar) with its uncertainty interval. Comparable only when the two effects are expressed in the same units against the same kind of outcome.
More variance explained. If signal means "how much of this dataset's movement the variable accounts for," the metrics are partial \( R^2 \), \( \Delta R^2 \), or a Shapley-style variance decomposition that handles correlated predictors gracefully (Grömping, 2007). Note what these measure: a joint property of the effect and how much the variable moved in this sample. A variable held nearly constant explains almost nothing, however strong its causal effect.
Better prediction. If signal means out-of-sample predictive value, the metric is the change in expected log predictive density from adding or dropping the variable, estimated by cross-validation with a standard error on the difference (Vehtari, Gelman, and Gabry, 2017). That is a paired comparison on shared data, which is exactly what two marginal p-values fail to be.
The distinctions matter because they come apart in practice. An always-on channel can have a large causal effect (meaning one), explain almost no variance in the training window because it never moved (meaning two), and add nothing to prediction for the same reason (meaning three). A p-value collapses all three meanings into a single design-contaminated number and invites you to read whichever meaning flatters the narrative. The ASA's statement on p-values (Wasserstein and Lazar, 2016) says the quiet part as principle five: a p-value does not measure the size of an effect or the importance of a result. Importance-ranking is precisely the job people keep handing it.
How to Compare Properly
If the comparison matters enough to put on a slide, it matters enough to estimate. The right tool depends on how the two variables relate to the data.
Same dataset: put both variables in one model and estimate the contrast \( \Delta = \beta_1 - \beta_2 \) directly, with a standard error that includes the covariance. In a Bayesian fit, read the posterior of \( \Delta \) off the joint draws instead. Same sample, different specifications: the estimates are correlated across models, so use the Clogg–Petkova–Haritou machinery rather than pretending independence. Genuinely independent datasets: the two-sample z-statistic on the difference, and better still a hierarchical model that treats both estimates as draws from a common structure, which partial-pools the comparison and damps the winner's-curse exaggeration that raw pairwise contrasts inherit. Different outcomes: translate both effects onto one decision scale first (per-dollar incremental revenue, say), or concede that no comparison exists. A p-value from a revenue model and a p-value from an awareness model share no scale on which "smaller" means anything.
# illustrative
# WRONG: rank "signal" by stars from two separate fits
m1 = fit(revenue ~ search + controls_2024) # search: p = 0.001
m2 = fit(awareness ~ tv + controls_2023) # tv: p = 0.21
# "search matters more": this compares rulers, not effects
# RIGHT (same data): one model, estimate the contrast itself
m = fit(kpi ~ search + tv + controls)
diff = m.beta["search"] - m.beta["tv"]
se_diff = sqrt(V["search","search"] + V["tv","tv"] - 2 * V["search","tv"])
# RIGHT (independent samples): test the difference, not the verdicts
z = (b1 - b2) / sqrt(se1**2 + se2**2)
# RIGHT (Bayesian): the comparison is one line of the joint posterior
p_search_beats_tv = mean(draws["beta_search"] - draws["beta_tv"] > 0)The Bayesian version deserves the emphasis it gets in this framework, because it makes the correct comparison the easy one. A joint posterior over all channels prices in every covariance automatically. "Does search beat TV, and by how much?" is a subtraction over draws, reported as a distribution. There is no intermediate step where each channel gets a lone verdict against zero, so there is nothing to launder into a ranking.
What It Means for Media
Media measurement is unusually exposed to this mistake because it is unusually fond of cross-model comparisons. This year's MMM against last year's. The agency's model against the in-house one. Brand A's TV coefficient against brand B's. A channel's significance in the revenue model against another's in the awareness model. Every one of these crosses a boundary across which p-values do not travel: different windows (n), different KPIs (σ), different control sets (different estimands).
Two design terms from the t-statistic's anatomy do disproportionate damage in MMMs. The first is \( s_{x_j} \), the predictor's variation: an always-on channel with near-constant spend has almost no variance in the training window, so its standard error explodes and its p-value is large no matter what the channel truly does. Its "insignificance" is a fact about the media plan, not the media. The cure is designed variation, either flighting the channel through deliberate levels or calibrating to a geo experiment that supplies the variation the plan never did. The second term is \( R_j^2 \). Channels that flight together (a launch week that lights up TV, social, and display at once) inflate one another's variance until every one of them reads "not significant," even when all of them work. Reading those p-values as "little signal in each channel" gets the diagnosis exactly backwards. The data contain good evidence about the channels' sum and poor evidence about the split, which is an argument for de-correlating future spend rather than for cutting the channels.
This framework's posture follows from the arithmetic. Channels are estimated jointly, so any comparison between them is a posterior contrast with the covariance priced in, reported as "the probability search out-earns TV per incremental dollar, with the interval," rather than as a pair of verdicts. Effects live on the ROI scale, where two channels' numbers share units by construction. And when a comparison is genuinely under-identified, by collinear flights or constant spend, the honest output is a wide contrast interval and a recommendation for the experiment that would narrow it, rather than a significance table whose ordering quietly re-ranks the design. Companion posts cover the p-value's cousin failures, how little evidence one carries and what selecting the best of many does to it. This one's lesson is narrower and, in practice, violated more often: whatever a single p-value is worth, two of them from two regressions are not a comparison.
Takeaways
- A p-value is a monotone readout of \( t = (\hat\beta/\hat\sigma) \cdot s_x \cdot \sqrt{n} \cdot \sqrt{1-R_j^2} \): one part effect, four parts design. Across two regressions, the design terms differ freely, so the p-value ordering can invert the effect ordering.
- Even under identical designs, "A significant, B not" is not evidence that A and B differ: the difference between significant and not significant is not itself significant (Gelman–Stern). Half of the papers in top neuroscience journals that made this comparison made it wrong.
- P-values are calibrated only within their own model, and the same digits carry different evidence at different sample sizes (Goodman, Lindley). There is no scale on which p-values from different regressions are comparable.
- When both variables are plausibly null, the contest is exactly a fair coin: two null p-values are independent Uniform(0,1) draws, so \( P(p_1 \lt p_2) = 1/2 \) regardless of the designs. A real signal with expected t = 1.0 beats pure noise only ~64% of the time, and even expected t = 2.0 loses one contest in seven.
- Selecting variables by significance therefore selects noise: the best of 20 null p-values averages 0.048 (a Beta(1, m) order statistic), Freedman's 50-noise-variable regression yields a "publishable" model after screening, and post-selection standard errors and p-values are invalid anyway (Berk et al., Harrell). Choose variables causally, shrink instead of selecting, and judge predictive value by ΔELPD.
- "More signal" has three honest meanings (bigger effect, more variance explained, better prediction) with three honest metrics: effect sizes with intervals on a shared scale, partial \( R^2 \)/Shapley decompositions, and \( \Delta \)ELPD with its standard error. None is a p-value.
- To compare two variables, estimate the comparison: a contrast in a joint model, a Clogg-style cross-model test, a two-sample z on the difference, or the posterior of \( \beta_1 - \beta_2 \).
- In MMMs, "not significant" is often a fact about the plan (constant spend, co-flighted channels) rather than the channel. The fix is designed variation and joint posterior contrasts, not a significance ranking.
References
- Gelman, A., & Stern, H. (2006). The Difference Between "Significant" and "Not Significant" Is Not Itself Statistically Significant. The American Statistician, 60(4).
- Nieuwenhuis, S., Forstmann, B. U., & Wagenmakers, E.-J. (2011). Erroneous Analyses of Interactions in Neuroscience: A Problem of Significance. Nature Neuroscience, 14(9).
- Wasserstein, R. L., & Lazar, N. A. (2016). The ASA Statement on p-Values: Context, Process, and Purpose. The American Statistician, 70(2).
- Greenland, S., Senn, S. J., Rothman, K. J., Carlin, J. B., Poole, C., Goodman, S. N., & Altman, D. G. (2016). Statistical Tests, P Values, Confidence Intervals, and Power: A Guide to Misinterpretations. European Journal of Epidemiology, 31(4).
- Goodman, S. N. (2008). A Dirty Dozen: Twelve P-Value Misconceptions. Seminars in Hematology, 45(3).
- Lindley, D. V. (1957). A Statistical Paradox. Biometrika, 44(1/2).
- Freedman, D. A. (1983). A Note on Screening Regression Equations. The American Statistician, 37(2).
- Berk, R., Brown, L., Buja, A., Zhang, K., & Zhao, L. (2013). Valid Post-Selection Inference. The Annals of Statistics, 41(2).
- Harrell, F. E. Jr. (2015). Regression Modeling Strategies (2nd ed., §4.3 on stepwise selection). Springer.
- Clogg, C. C., Petkova, E., & Haritou, A. (1995). Statistical Methods for Comparing Regression Coefficients Between Models. American Journal of Sociology, 100(5).
- Paternoster, R., Brame, R., Mazerolle, P., & Piquero, A. (1998). Using the Correct Statistical Test for the Equality of Regression Coefficients. Criminology, 36(4).
- Schenker, N., & Gentleman, J. F. (2001). On Judging the Significance of Differences by Examining the Overlap Between Confidence Intervals. The American Statistician, 55(3).
- Grömping, U. (2007). Estimators of Relative Importance in Linear Regression Based on Variance Decomposition. The American Statistician, 61(2).
- Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and WAIC. Statistics and Computing, 27(5).