Second Brain

Index: Model Validation and Interpretability

9 min read

Model Validation and Interpretability

Routing Summary

This topic covers how to tell whether a predictive or measurement model can be trusted before it informs decisions. It spans false discovery and overfitting, the biases and instability of feature importance, why SHAP and other explanations are not causal levers (and how optimized metrics drift from their goals), validation of time-series models, known-truth simulation of statistical and marketing methods, and model risk governance. It holds 6 sub-topics, 21 source folders and 186 notes: 185 source notes plus one cross-source synthesis, not counting 27 index files. All 21 sources were ingested with full capture on 2026-10-13.

  • Need the big picture, the eight failure mechanisms, the six remedies, or a validation checklist? → Model Validation and Interpretability - Overview
  • Need why significant findings and fitted models fail on new data (PPV, researcher degrees of freedom, sample size for prediction models, the two cultures)? → Overfitting and False Discovery
  • Need whether a variable-importance score or selected variable set can be trusted (RF importance bias, conditional importance, permute-and-predict, Rashomon sets, stability selection)? → Feature Importance and Stability
  • Need what SHAP / Shapley values compute, conditional vs interventional, why they are not causal effects, or Goodhart’s Law? → Explanations, Causality and Proxies
  • Need spurious regression, cointegration, or when K-fold / rolling-origin CV is valid for time series? → Time Series Validation
  • Need to design a known-truth simulation study (ADEMP, Monte Carlo SEs), the AMSS marketing simulator, or why MMMs cannot be validated by fit? → Simulation-Based Method Validation
  • Need the model-risk governance view (SR 11-7 effective challenge, validation gate, NIST AI RMF GOVERN/MAP/MEASURE/MANAGE)? → Model Risk Governance
  • Need the ad-experiment ground truth that observational measurement is checked against? → User-Level Ad Experiments (Gordon et al. 2019, Lewis & Rao 2015, Blake et al. 2015)

Sub-topics

  • Overfitting and False Discovery — COVERS: Ioannidis (2005) PPV 2x2 framework, bias , many-teams erosion, six corollaries, Table 4; Simmons, Nelson & Simonsohn (2011) researcher degrees of freedom, Table 1 (61% FPR), optional stopping (22%), disclosure requirements; Breiman (2001) two cultures, Rashomon effect, Occam dilemma, random-forest permutation importance, Cox/Efron/Hoadley/Parzen discussion; Riley et al. (2020) sample-size criteria for prediction models (shrinkage , optimism, , pmsampsize). 35 notes, 4 source folders.
  • Feature Importance and Stability — COVERS: Strobl et al. (2007) split-selection and bootstrap bias in RF importance, cforest; Strobl et al. (2008) conditional permutation importance and its null hypotheses; Hooker, Mentch & Zhou (2021) permute-and-predict extrapolation, conditional and relearning (LOCO) importance; Fisher, Rudin & Dominici (2019) model reliance as a U-statistic, Rashomon sets, model class reliance bounds and computation; Meinshausen & Bühlmann (2010) stability selection, PFER bound, randomised Lasso. 51 notes, 5 source folders.
  • Explanations, Causality and Proxies — COVERS: Kumar et al. (2020) Shapley/SHAP value functions, conditional vs interventional catch-22, additivity limits, normative use; Janzing et al. (2020) interventional = marginal expectation, irrelevant-feature counterexample, KernelSHAP vs SHAPR; Dillon et al. SHAP vs true causal effects, DML under observed and unobserved confounding; Manheim & Garrabrant (2018) regressional, extremal, causal and adversarial Goodhart. 31 notes, 4 source folders.
  • Time Series Validation — COVERS: Granger (2003) spurious regression from Yule to Granger-Newbold, cointegration and ECM, Granger causality, Monte Carlo in Python; Bergmeir, Hyndman & Koo (2018) Theorem 1 validity of K-fold CV for AR models, Monte Carlo design and results, Ljung-Box filter; synthesis note on out-of-sample validation (nested rolling origin, blocked CV with gaps, null baselines, random-ranking permutation test). 16 notes, 2 source folders.
  • Simulation-Based Method Validation — COVERS: Morris, White & Crowther (2019) ADEMP, performance measures and Monte Carlo SEs, , seeds, reporting, marketing application; Zhang & Vaver (2017) AMSS hidden-state consumer model, media and paid-search modules, counterfactual ROAS ground truth, MMM evaluation; Chan & Perry (2017) MMM data limits (156 rows), selection bias, five equal-fit models, opportunities, simulation-based validation. 29 notes, 3 source folders.
  • Model Risk Governance — COVERS: SR 11-7 (2011) model definition and effective challenge, development and use, validation independence and gate, conceptual soundness, ongoing monitoring, outcomes analysis and back-testing, vendor models, governance, inventory, marketing application; NIST AI RMF 1.0 (2023) framing, lifecycle and TEVV, trustworthiness characteristics, GOVERN/MAP/MEASURE/MANAGE, profiles, marketing-analytics application. 23 notes, 2 source folders.

Cross-Cutting Concepts

Notes

  • Model Validation and Interpretability - Overview — CONTAINS: cross-source synthesis; eight failure mechanisms (too few rows, best-of-many, selection by importance mass, spurious time-series fits, biased/unstable importance, importance ≠ intervention effect, Goodhart, observational ad measurement error); six remedies (forward-in-time validation, null/random-ranking baselines, retest stability, ADEMP/AMSS simulation, experiments, SR 11-7/NIST gate); 12-item validation checklist; mermaid concept map; spurious-driver selection simulation (77% significant, naive wins 72%).

Sources

  • Bergmeir Hyndman Koo 2018 - A Note on the Validity of Cross-Validation for Evaluating Autoregressive Time Series Prediction — Bergmeir, Hyndman & Koo (2018), Computational Statistics & Data Analysis 120: K-fold CV validity for purely autoregressive models.
  • Breiman 2001 - Statistical Modeling The Two Cultures — Breiman (2001), Statistical Science 16(3), with comments by Cox, Efron, Hoadley, Parzen and rejoinder.
  • Chan Perry 2017 - Challenges and Opportunities in Media Mix Modeling — Chan & Perry (2017), Google white paper on MMM challenges.
  • Dillon et al - Be Careful When Interpreting Predictive Models in Search of Causal Insights (SHAP docs) — Dillon, LaRiviere, Lundberg, Roth & Syrgkanis (Microsoft), SHAP documentation case study “Be careful when interpreting predictive models in search of causal insights”.
  • Federal Reserve and OCC 2011 - Supervisory Guidance on Model Risk Management (SR 11-7) — Board of Governors of the Federal Reserve System & OCC (2011), SR 11-7 / OCC 2011-12.
  • Fisher Rudin Dominici 2019 - All Models Are Wrong but Many Are Useful — Fisher, Rudin & Dominici (2019), JMLR 20: model reliance and model class reliance.
  • Granger 2003 - Time Series Analysis Cointegration and Applications (Nobel Lecture) — Granger (2003), Nobel lecture on time series, cointegration and spurious regression.
  • Hooker Mentch Zhou 2021 - Unrestricted Permutation Forces Extrapolation — Hooker, Mentch & Zhou (2021), Statistics and Computing 31: permute-and-predict extrapolation.
  • Ioannidis 2005 - Why Most Published Research Findings Are False — Ioannidis (2005), PLoS Medicine 2(8): e124.
  • Janzing et al 2020 - Feature Relevance Quantification in Explainable AI - A Causal Problem — Janzing, Minorics & Blöbaum (2020), AISTATS: interventional vs observational feature relevance.
  • Kumar et al 2020 - Problems with Shapley-Value-Based Explanations — Kumar, Venkatasubramanian, Scheidegger & Friedler (2020), “Problems with Shapley-value-based explanations as feature importance measures”, ICML 2020.
  • Manheim and Garrabrant 2018 - Categorizing Variants of Goodhart’s Law — Manheim & Garrabrant (2018), arXiv:1803.04585.
  • Meinshausen Buhlmann 2010 - Stability Selection — Meinshausen & Bühlmann (2010), JRSS-B 72(4) (arXiv version).
  • Morris White Crowther 2019 - Using Simulation Studies to Evaluate Statistical Methods — Morris, White & Crowther (2019), Statistics in Medicine 38: ADEMP.
  • NIST 2023 - AI Risk Management Framework (AI RMF 1.0) — NIST AI 100-1 (2023), AI Risk Management Framework 1.0.
  • Riley et al 2020 - Calculating the Sample Size Required for Developing a Clinical Prediction Model — Riley et al. (2020), BMJ 368: m441.
  • Simmons Nelson Simonsohn 2011 - False-Positive Psychology — Simmons, Nelson & Simonsohn (2011), Psychological Science 22(11).
  • Strobl et al 2007 - Bias in Random Forest Variable Importance Measures — Strobl, Boulesteix, Zeileis & Hothorn (2007), BMC Bioinformatics 8: 25 (Europe PMC text).
  • Strobl et al 2008 - Conditional Variable Importance for Random Forests — Strobl, Boulesteix, Kneib, Augustin & Zeileis (2008), BMC Bioinformatics 9: 307 (web text).
  • Strobl et al 2008 - Figure 1 Selection Rates — Strobl et al. (2008) Figure 1, variable selection rates.
  • Strobl et al 2008 - Figure 2 Permutation Schemes — Strobl et al. (2008) Figure 2, permutation schemes.
  • Strobl et al 2008 - Figure 3 Permutation Importance — Strobl et al. (2008) Figure 3, permutation importance.
  • Strobl et al 2008 - Figure 4 Peptide Binding — Strobl et al. (2008) Figure 4, peptide-binding application.
  • Zhang Vaver 2017 - Introduction to the Aggregate Marketing System Simulator — Zhang & Vaver (2017), Google technical report on AMSS.

See Also

Graph View

Backlinks