Scientific Statistical Modeling

A principled approach to building, evaluating, and iterating on statistical models. This guide establishes the philosophical foundations that underpin rigorous quantitative analysis— applicable to Marketing Mix Models and beyond.

What is a Statistical Model?

Models are not photographs of reality—they are useful simplifications. A map of New York City is not the city itself, but it can help you navigate. Similarly, a model of sales response to advertising is not the actual cognitive and behavioral processes involved, but it can help you make better decisions.

This perspective is liberating: we don't need to find the "true" model (which doesn't exist). We need to find models that are useful for our purposes while being honest about their limitations.

All Models Are Wrong

"All models are wrong, but some are useful." — George E.P. Box

Box's aphorism captures a fundamental truth. Every model makes simplifying assumptions that do not perfectly match reality. The question is not "is this model true?" but rather:

• Is the model useful for the decisions I need to make?
• Are its predictions accurate enough for my purposes?
• Am I aware of its limitations and their implications?

A model that captures the main drivers of sales may be highly useful even if it ignores subtle effects that contribute only 2% of variance. The key is knowing which simplifications matter for your question and which don't.

The Generative Perspective

The generative perspective offers several advantages:

Questions Drive Models

Scientific modeling begins with a question, not with data or techniques. The question determines what model structure is appropriate, what data are needed, and how to evaluate success.

Marketing Mix Models typically aim to answer causal questions: "What is the effect of advertising on sales?" This is the hardest type of question to answer from observational data, and requires explicit assumptions about the data-generating process.

The Generative Story

Every model should have a generative story: a narrative description of how you believe the data came to be. For an MMM, this might be:

Writing down this story forces you to be explicit about your assumptions. Each element of the story corresponds to a component of the mathematical model.

Model Components

Choosing a Likelihood

The likelihood should match the nature of the outcome:

Iteration as Learning

Model building is inherently iterative. We build, check, revise, and expand—learning about both the data and the phenomenon through this cycle. This is not a failure of method; it is the method.

Each iteration teaches us something:

• Prior predictive failures reveal that our encoded assumptions imply impossible outcomes
• Computational failures often indicate model misspecification or identification problems
• Posterior predictive failures show where the model fails to capture data patterns
• Sensitivity analysis reveals which conclusions are robust and which are fragile

Honest Iteration vs. Specification Shopping

There is a crucial distinction between legitimate scientific iteration and problematic "specification shopping." Both involve changing models, but they differ fundamentally in what drives the changes.

Pre-Specification: The Solution

The solution to specification shopping is pre-specification: commit to your model structure before looking at results. This doesn't prevent iteration—you can still revise models—but it requires documenting and justifying changes, making the distinction between planned and exploratory analyses explicit.

Expanding Models

When a model fails predictive checks or sensitivity analysis reveals fragility, we may need to expand it. Model expansion should be driven by the nature of the failure, not by a desire for different results.

Predictive Checking

The primary tool for evaluating models is predictive checking: comparing model predictions to observed data. There are two forms:

Prior Predictive Checks

Before fitting to data, sample from priors and push through the model:

# Prior predictive check
with model:
    prior_pred = pm.sample_prior_predictive(samples=500)

# Examine the implied distribution of predictions
y_prior = prior_pred.prior_predictive["y"].values

# Check: are these values plausible for sales data?
# - All positive? (sales can't be negative)
# - Reasonable range? (not implying billion-dollar weeks)
# - Sensible variation? (not identical across all scenarios)

Posterior Predictive Checks

After fitting, generate replicated datasets and compare to observations:

# Posterior predictive check
with model:
    post_pred = pm.sample_posterior_predictive(trace)

# Compare observed and replicated data
y_rep = post_pred.posterior_predictive["y"].values
y_obs = data["sales"].values

# Visual checks
# - Does distribution of y_rep match distribution of y_obs?
# - Do time-series patterns match?
# - Are extreme values reproduced?

Sensitivity Analysis

Sensitivity analysis asks: how much do conclusions change if we change assumptions? Conclusions that are robust to reasonable alternative assumptions are more credible than those that depend on specific choices.

What to Vary

• Prior distributions (especially on key parameters)
• Functional forms (linear vs. nonlinear, different saturation curves)
• Lag structures (different adstock parameters)
• Control variable sets (within theoretically justified bounds)
• Do NOT vary until you get desired results

External Validation

The strongest test of a model is its performance on data it has never seen—especially data generated by experimental intervention. For MMMs, this typically means comparing model predictions to results from geo-lift experiments or randomized holdout tests.

Starting Simple

A common mistake is to start with a complex model. Better practice:

• Start with the simplest model that could possibly work
• Add complexity only when diagnostics indicate it's needed
• Document each addition and why it was made
• Compare simpler and more complex models on held-out data

A simple model that you understand deeply is more valuable than a complex model that produces mysteriously "better" results.

When to Stop

Model building could continue forever. Practical stopping rules:

• Predictive adequacy: Posterior predictive checks pass for features that matter
• Computational stability: MCMC diagnostics are acceptable
• Diminishing returns: Additional complexity improves fit marginally
• Resource constraints: Time and computational budget exhausted

Importantly, stopping is not the same as claiming the model is "correct." It means the model is adequate for current purposes, given current constraints.

Communicating Models

How you communicate model results is as important as the modeling itself. Key principles:

• Report uncertainty honestly—credible intervals, not just point estimates
• Distinguish robust findings from fragile ones
• Explain assumptions and their implications for conclusions
• Acknowledge limitations without burying the signal in caveats
• Do not present specification-shopped results as definitive

Summary: The Scientific Modeling Mindset

Further Reading

• Gelman, A., et al. (2020). Bayesian Workflow. arXiv:2011.01808. [Link]
• Box, G. E. P. (1976). Science and Statistics. Journal of the American Statistical Association.
• McElreath, R. (2020). Statistical Rethinking (2nd ed.). CRC Press. [Link]
• Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-Positive Psychology. Psychological Science. [Link]