Common Pitfalls in Statistical Modeling

Matthew Reda· April 9, 2026· Practice

Most modeling failures are not exotic — they are the same eight mistakes, recurring across industries and decades, each producing output that looks exactly like success. This is a field guide: what each pitfall looks like in practice, why it sails through review, and what fixes it.

The unsettling property these failure modes share is that none of them announce themselves. A confounded regression converges cleanly; an overfit model fits beautifully in-sample; a forking-paths specification arrives with an entirely sincere analyst attached. The diagnostics that would catch each one are rarely the ones anyone asks for — which is why the pitfalls persist.

1. Mistaking Correlation for Cause

What it looks like. A regression of revenue on media spend reports, with a clean fit and a significant coefficient, that paid search "drives" a startling share of sales. It is mostly measuring the media plan: budgets were set against demand forecasts, so the demand that causes the sales also caused the spend. Three mechanisms produce this family of errors. Confounding: a common cause moves both predictor and outcome, the way regional culture drives both Waffle House density and divorce rates in McElreath's running example. Selection: the raw treated-versus-untreated gap is the true effect on the treated plus a selection term, which can be large enough to flip the sign — why hospitals look dangerous in observational data. Omitted variables: the bias from a left-out variable is the product of its effect on the outcome and its correlation with what you kept — predictable in direction, and rarely small.

Why it survives review. Significance and fit look identical whether an association is causal or spurious. Reviewers audit the regression's internals — standard errors, residuals, robustness to covariate swaps — none of which know anything about the assignment mechanism. The one question that matters, why did the treated units get treated?, is not on most checklists.

The remedy. Draw the causal graph before writing the model, and name the estimand before estimating anything. Randomize where possible; otherwise use an explicit identification strategy — instruments, differences-in-differences, discontinuities — and state the assumption each one rents. A DAG will not make an unidentifiable effect identifiable, but it makes the failure visible before the coefficient does damage.

2. The Garden of Forking Paths

What it looks like. A media mix analyst tries a grid of adstock half-lives, two saturation forms, three control sets, and a couple of date windows, then reports the winning specification's intervals as though it were the only model ever fit. Nothing was hidden and nothing was hacked. Gelman and Loken's point is that intent is irrelevant: the analysis you ran is one path through a garden of analyses you would have run had the data come out differently, and the reported p-value knows nothing about the rest of the garden. Five binary analytic choices already give thirty-two paths; the chance that at least one clears the conventional significance bar is far above five percent. Simmons, Nelson, and Simonsohn catalogued how much flexibility ordinary analyses contain — enough to present nearly anything as significant.

Why it survives review. The forks are invisible in the deliverable. A reviewer sees one clean analysis, not the thirty-one counterfactual ones, and each individual decision reads as competent judgment rather than as a degree of freedom. The paper trail only records the path taken.

The remedy. Shrink the garden or walk all of it. Pre-specification removes the forks; hierarchical models with partial pooling estimate all the branches at once and shrink them toward each other — why Gelman, Hill, and Yajima argue multilevel modeling dissolves the multiple-comparisons problem rather than correcting for it. For the specification-search version, regularizing priors and principled variable selection — projection predictive methods in particular — replace "fit many, report one" with a procedure whose selection step is itself accounted for.

3. Overfitting and the R-Squared Trap

What it looks like. McElreath frames model complexity as a strait between two monsters: Scylla, the overfit model that learns the noise, and Charybdis, the underfit model that misses the signal. The current pushes toward Scylla, because in-sample fit always improves with more parameters — even randomly generated predictors raise R-squared. The demonstration: a fifth-degree polynomial through seven data points achieves an R-squared of 0.99 and predicts negative brain volumes between the observations. The marketing version: enough flexible controls that the fit chart is immaculate while ROAS estimates swing wildly between refits — flexibility spent memorizing this dataset's accidents.

Why it survives review. Because the number reviewers ask for is the one that rewards the failure. "How well does it fit?" is answered in-sample, and in-sample the overfit model wins every time. Absent an out-of-sample discipline, complexity is indistinguishable from insight.

McElreath’s hominin data: brain volume vs. body mass for seven species. The degree-6 polynomial threads every point exactly — R² = 1.00 — yet it swings to negative brain volumes between the observations (shaded region below zero). The straight line explains far less in-sample and is far more trustworthy out of it. Perfect fit is a symptom, not a triumph.

The remedy. Two families. First, regularizing priors — informative, mildly skeptical priors on effect sizes, the Bayesian counterpart of ridge and lasso. Second, score models on estimated out-of-sample predictive accuracy rather than fit: WAIC, or in practice PSIS-LOO cross-validation, whose Pareto-k diagnostic also flags observations where the approximation itself is straining (Vehtari, Gelman, and Gabry, 2017).

4. Ignoring How the Data Went Missing

What it looks like. Rows with gaps are silently dropped — usually by a software default rather than a decision — and the analysis proceeds on "complete cases." Whether that is harmless depends entirely on the causal mechanism generating the missingness:

A tracking outage that blanks three weeks of impressions is plausibly MCAR. Competitor spend a vendor only reports in covered markets is MAR at best. Survey responses that go missing because of the attitude being surveyed are MNAR, and no naive fix recovers the truth. Worse, complete-case analysis conditions on being observed — draw the missingness indicator into the causal graph as a node and that conditioning can open collider paths that bias estimates even under MAR.

Why it survives review. Deleted rows leave no trace. The model summary looks the same whether ten rows or forty percent of the panel quietly vanished, and because dropping is the default, nobody ever made a visible choice to defend.

The remedy. Ask what causes the missingness, then treat missing values as latent quantities in the model — Bayesian imputation assigns them priors and infers them jointly, so uncertainty about what was missing propagates into the parameters instead of being deleted with the rows.

5. Statistical Tests as Golems

What it looks like. McElreath opens Statistical Rethinking with the golem of Prague: powerful, obedient, and without judgment about whether its instructions make sense. Classical tests are pre-fabricated golems — a t-test, a chi-squared, an ANOVA chosen from a flowchart and set loose on data they may or may not suit. The deeper problems: hypotheses are not models (the hypothesis-to-model mapping is many-to-many, so rejecting a null tells you far less than it feels like it does), and clean falsification is rare in a world of measurement error and continuous hypotheses.

Why it survives review. Ritual reads as rigor: a named test with a p-value looks like due diligence precisely because it required no modeling judgment. The most common casualty is the comparison of significance levels — channel A's lift is significant, channel B's is not, so the deck concludes A outperforms B. But the difference between significant and non-significant is itself often not significant; the two intervals can overlap almost entirely.

The remedy. Build models rather than choosing tests: constructing, criticizing, and revising an explicit generative model keeps the assumptions where you can see them. And when the question is a comparison, compute the comparison — the posterior contrast between the two effects — rather than comparing two verdicts about zero.

6. Skipping the Model Check

What it looks like. The model converges, the intervals are tight, the deck ships. Nobody asks whether the fitted model could plausibly have generated data resembling what it was fit to. Posterior predictive checking asks exactly that: simulate replicated datasets from the model and compare them to the observed one. If the replicas never show the seasonality spikes or the variance pattern the real series shows, the model is missing structure — however confident its intervals look. The same logic runs forward: prior predictive checks simulate data from the priors before fitting, catching priors that imply negative sales while they are still cheap to fix.

Why it survives review. Checking is the only step of the workflow with no deliverable. A fitted model looks finished, criticism of it looks like delay, and the incentives all point toward shipping. It is also the step that requires taste: focus, as Gelman and colleagues advise, on severe tests — checks likely to fail if the model would mislead you on the questions you care about.

The remedy. Make checking a loop, not a gate: find misfit, expand the model, check again, in the iterative spirit of the Bayesian workflow. Leave-one-out diagnostics round out the toolkit, flagging observations the model finds surprising and checking that nominal 90% intervals cover about 90% of the time.

7. Blaming the Sampler

What it looks like. The chains diverge, mixing is glacial, and the response is mechanical: raise the target acceptance rate, run longer, switch samplers. The folk theorem of statistical computing — Gelman's phrase — says the diagnosis is usually elsewhere: when you have computational problems, often there is a problem with your model. The sampler is the messenger.

Why it survives review. Because tuning sometimes silences the warning without touching the pathology, and a quiet sampler looks like a healthy model. A funnel that stops producing divergences after aggressive step-size tuning is still a funnel — explored badly, just less loudly.

The remedy. Debug from both ends: simplify the failing model until it works, and build the trivial working model up until it fails. The pathology lives where the two meet — nearly always a modeling choice, not an algorithmic one.

8. Significant, but the Wrong Sign and Size

What it looks like. An underpowered analysis produces a significant, impressive effect — and the impressiveness is the artifact. When the true effect is near zero, an estimator with a large standard error is more likely to produce large point estimates, and conditioning on significance selects precisely those exaggerated draws — inflated by construction, sometimes with the wrong sign outright. This is the winner's curse: the most impressive number in any set of comparisons — the standout channel in a noisy media mix model, the best cell in a sprawling test matrix — is likely the most exaggerated, not the most real.

Why it survives review. Because the filter meant to protect against noise manufactures the error: in a low-power setting, significance functions as a selection effect for overestimates, not a certificate of reliability. The bigger the claimed effect, the more the room nods.

The remedy. Frame questions as sign and magnitude, not "is it zero?" Use multilevel models: partial pooling shrinks noisy estimates toward the group, controlling Type S and Type M error rates automatically — the same property that dissolved the multiple-comparisons problem in pitfall 2. And cultivate the reflex: a large effect from a small sample is a reason for suspicion, not a press release.

A Field Checklist

The eight pitfalls compress into a discipline you can run in order:

• Before modeling — draw the causal DAG, including the missingness mechanism; name the target estimand.
• Building — start simple and grow modularly; use regularizing priors; run prior predictive checks first.
• Fitting — watch the convergence diagnostics; when computation fails, suspect the model before the algorithm.
• Evaluating — posterior predictive checks aimed at severe tests; LOO cross-validation with its diagnostics; calibration.
• Comparing — score on estimated out-of-sample accuracy, never in-sample fit or p-values; consider averaging over plausible models rather than crowning one, and make the selection step part of the accounted-for procedure.
• Reporting — full posterior uncertainty, contrasts computed directly, no verdicts laundered through significance stars.

None of this is exotic either — that is the point. What separates teams that avoid these pitfalls is not sophistication; it is the willingness to run checks whose only possible output is bad news about your own model.

References

• McElreath, R. (2020). Statistical Rethinking: A Bayesian Course with Examples in R and Stan (2nd ed.). CRC Press.
• Gelman, A., & Loken, E. (2013). The Garden of Forking Paths: Why Multiple Comparisons Can Be a Problem, Even When There Is No "Fishing Expedition" or "p-Hacking" and the Research Hypothesis Was Posited Ahead of Time. Department of Statistics, Columbia University.
• Gelman, A., Hill, J., & Yajima, M. (2012). Why We (Usually) Don't Have to Worry About Multiple Comparisons. Journal of Research on Educational Effectiveness, 5(2).
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant. Psychological Science, 22(11).
• Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A., & Rubin, D. B. (2013). Bayesian Data Analysis (3rd ed.). CRC Press.
• Gelman, A., Vehtari, A., Simpson, D., Margossian, C. C., Carpenter, B., Yao, Y., Kennedy, L., Gabry, J., Bürkner, P.-C., & Modrák, M. (2020). Bayesian Workflow. arXiv:2011.01808.
• Angrist, J. D., & Pischke, J.-S. (2009). Mostly Harmless Econometrics: An Empiricist's Companion. Princeton University Press.
• Lewis, R. A., Rao, J. M., & Reiley, D. H. (2011). Here, There, and Everywhere: Correlated Online Behaviors Can Lead to Overestimates of the Effects of Advertising. Proceedings of WWW.
• Vehtari, A., Gelman, A., & Gabry, J. (2017). Practical Bayesian Model Evaluation Using Leave-One-Out Cross-Validation and WAIC. Statistics and Computing, 27(5).