Does Your Model Work Before It Sees Data?

Matthew Reda Bayesian workflow

Most model validation waits for the data: fit, then check residuals, coverage, out-of-sample error. But two of the most powerful checks in the Bayesian toolkit run before a single real observation is used. A prior predictive check asks whether your priors imply data that could plausibly exist. Simulation-based calibration asks whether your inference software recovers the parameters it claims to. Both catch failures that a post-fit diagnostic quietly passes — and both matter most for exactly the weakly-identified models, like an MMM, where the prior does the heavy lifting.

The Checks Before the Data

A Bayesian model is a joint distribution, \( p(y, \theta) = p(y \mid \theta)\,p(\theta) \). That factorization contains a fact that is easy to forget: the model can generate data. Fix nothing, observe nothing — just draw parameters from the prior and push them through the likelihood, and out comes a synthetic dataset. This generative property is what makes pre-data checking possible, and it splits into two distinct questions.

The first is about the model: do the priors, once combined with the likelihood, describe a world that could exist? That is the prior predictive check. The second is about the computation: given data the model itself generated, does your sampler return a posterior that actually contains the parameters that produced it? That is simulation-based calibration. Neither question needs your real data, and neither can be answered by staring at the priors one coefficient at a time. Both are cheap relative to the cost of trusting a broken model — and both are standard steps in what Gelman and colleagues (2020) codified as the Bayesian workflow.

Prior Predictive Checks

The mechanics are almost trivially simple, which is part of why the check is so underused.

Definition: Prior predictive distribution

With a proper prior, the model implies a marginal distribution over datasets before any data are seen,

$$ p(y) \;=\; \int p(y \mid \theta)\, p(\theta)\, d\theta. $$

A prior predictive check samples from it — draw \( \theta \sim p(\theta) \), then \( y \sim p(y \mid \theta) \) — and inspects the simulated datasets against domain knowledge. If the priors are sensible, the fake datasets should look like data that could plausibly arise in your application; if they don't, the priors need work.

The contrast with the more familiar posterior predictive check is exactly one word in the sampling step. A posterior predictive check draws parameters from \( p(\theta \mid y) \) — the data-informed posterior — and simulates replicated datasets to compare against the observed one. As Gabry and colleagues (2019) put it, because the posterior version conditions on real data, "we can be much more stringent in our comparisons." The prior version is looser by design: it is not asking whether the model fits, but whether the model, as specified, is even in the right universe. You run the prior predictive check first, precisely because it can veto a specification before you spend a night of sampling on it.

# illustrative
# Prior predictive check: does the model imply plausible data?
for s in range(n_sims):
    theta = sample_prior()               # draw params from p(theta)
    y_sim = simulate_likelihood(theta)   # draw a fake dataset y ~ p(y | theta)
    record(summary_stats(y_sim))         # e.g. range, max, share, growth
# Then: are these simulated datasets physically possible?
#   negative sales? a channel explaining 500% of revenue? inspect and reject.

Innocuous Priors, Absurd Data

The payoff is that a prior which looks harmless on the parameter scale can be deranged on the outcome scale — and you cannot see this by looking at the priors individually. The canonical demonstration is Gabry and colleagues' (2019) air-pollution model, where the outcome is on a log scale. A vague prior, coefficients \( \sim \mathcal{N}(0, 100) \), generates prior predictive datasets they describe as "completely impossible for this application." Tightening to a weakly informative prior helps but is not automatically safe: even priors that look reasonable per parameter, when simulated jointly, still produced pollution concentrations above 22,000 μg/m³ — a physically absurd number. The workflow paper (Gelman et al., 2020) paraphrases the failure memorably as fake datasets "where the pollution is denser than a black hole."

An innocuous coefficient prior, simulated forward

A one-channel model on an index where baseline sales sit at 100. The channel enters as \(y_t = 100 + \beta\,z_t\), with a standardized spend series \(z_t\) and a coefficient prior \(\beta \sim \mathcal{N}(0,\,\sigma)\). Widen the prior and watch the simulated sales fan (median, 50% and 90% bands over 240 prior draws) balloon past every sane bound — negative sales, a single channel swinging revenue by multiples of the base. No one prior looks guilty on the \(\beta\) scale; the crime shows up only on the outcome scale.

8
Implied sales (90% band)—
Peak channel share of base—
Draws implying impossible data—
Verdict—

“Impossible” here means a draw whose synthetic series goes negative somewhere, or whose channel contribution exceeds the entire baseline — more than 100% of revenue from one channel. A weakly-informative σ keeps the fan physically plausible; a vague one does not.

The lesson is structural, not anecdotal. The prior predictive distribution reflects the interplay between the prior and the likelihood, so you cannot understand it by fixing all but one parameter and varying that one — you have to assess the prior as a joint, multivariate object. A related trap that only appears in aggregate: many independently "weak" priors can be jointly strong. Gelman and colleagues (2020) note that independent priors on a few dozen regression weights can imply a prior that strongly favors high \( R^2 \) — the model is quietly convinced, before seeing anything, that its predictors explain almost all the variance. No single prior looks guilty; the crime is in the sum.

💡 Elicit priors on things you can actually picture

The deeper reason prior predictive checks work is that humans have good intuition about outcomes — plausible sales, plausible lift, plausible ROI — and terrible intuition about the scale of a standardized regression coefficient after it passes through an adstock transform and a saturation curve. Simulating forward converts an un-pictureable parameter prior into a pictureable data distribution you can accept or reject. This is also why putting priors directly on interpretable quantities (an ROI, a half-life) beats putting them on opaque internal coefficients.

The Self-Consistency Identity

Prior predictive checks interrogate the model. The second check interrogates the inference machinery, and it rests on an identity that is worth stating carefully, because the whole method falls out of it. Draw a ground-truth parameter from the prior, \( \tilde\theta \sim p(\theta) \), simulate data from it, \( \tilde y \sim p(y \mid \tilde\theta) \), and compute the posterior \( p(\theta \mid \tilde y) \). Now average that posterior over all the ways the data could have come out. Talts and colleagues (2018) show you get the prior back exactly:

$$ p(\theta) \;=\; \iint p(\theta \mid \tilde y)\, p(\tilde y \mid \tilde\theta)\, p(\tilde\theta)\; d\tilde y\, d\tilde\theta. $$

In words: the data-averaged posterior equals the prior. This holds for any model, with no assumptions beyond a proper prior and a correct posterior. It is a statement of pure consistency — if you generate parameters from the prior and infer them back correctly, you cannot, on average, end up anywhere other than the prior. The identity traces back to Geweke (2004) and Cook, Gelman, and Rubin (2006); its power is the contrapositive. Any discrepancy between the data-averaged posterior and the prior signals an error — either the posterior is being computed inaccurately, or the model as coded is not the model you think you wrote.

SBC and the Rank Statistic

You could check the identity directly by comparing two densities, but Talts and colleagues (2018) found a sharper instrument. Instead of averaging distributions, look at where each ground-truth value falls among its own posterior draws.

Definition: The SBC rank statistic

For a simulated ground truth \( \tilde\theta \) and \( L \) draws \( \theta_1, \dots, \theta_L \) from its posterior, and any one-dimensional quantity \( f \), the rank statistic counts how many draws fall below the truth,

$$ r \;=\; \sum_{l=1}^{L} \mathbb{I}\!\left[\, f(\theta_l) \lt f(\tilde\theta) \,\right] \;\in\; \{0, 1, \dots, L\}. $$

Theorem (Talts et al., 2018): if the \( \theta_l \) are independent draws from the exact posterior, this rank is uniformly distributed over the \( L+1 \) integers \( \{0, \dots, L\} \), for any joint distribution \( p(y, \theta) \).

The intuition is that the true value should be a "typical" member of its own posterior — no more likely to sit at the bottom of the cloud of draws than in the middle or at the top. Repeat the whole process over many simulated datasets, bin the ranks into a histogram, and calibrated inference produces a flat histogram. Every departure from flatness has a named meaning. The procedure generalizes the older idea of benchmarking against a single simulated truth: one truth-point tells you almost nothing, but thousands of ranks, aggregated, expose the systematic tendencies of the sampler.

# illustrative
# Simulation-based calibration: one rank per simulated dataset
ranks = []
for s in range(n_sims):
    theta_true = sample_prior()                       # ground truth from the prior
    y_sim      = simulate_likelihood(theta_true)      # data it generates
    draws      = fit_and_sample(y_sim, L=100)         # L posterior draws (thinned!)
    ranks.append(sum(d < theta_true for d in draws))  # rank in {0, ..., L}
histogram(ranks)   # flat = calibrated; shape = a specific, diagnosable failure

Reading the Rank Histogram

The diagnostic value of SBC is that the shape of the deviation tells you what went wrong, not merely that something did. Talts and colleagues (2018) catalogue the vocabulary, borrowed from the ensemble-forecast rank-histogram literature.

Histogram shapeWhat the posterior is doing
Flat (uniform)Calibrated — ranks consistent with correct computation
\( \cap \)-shape (dome)Over-dispersed: the posterior is on average too wide (under-confident)
\( \cup \)-shape (bathtub)Under-dispersed: the posterior is on average too narrow (overconfident)
Sloped / tilted (ramp)Bias — the posterior is systematically shifted; ranks pile up on the opposite side
Spikes at ranks 0 and \( L \)Autocorrelation — the MCMC draws are not thinned to near-independence

Reading the rank histogram, by miscalibration type

1,200 simulated fits of a one-parameter model, each ranking the true value among 59 posterior draws. When the posterior is exactly right the ranks are uniform — every bar inside the shaded binomial band. Distort the posterior and each pathology draws its signature shape: too narrow a posterior pushes the truth into the tails (∪); too wide traps it in the middle (∩); a shifted posterior tilts the histogram into a ramp.

1.00
0.0
Histogram shape—
Bins outside the band—
Diagnosis—

A flat histogram is the only pass. Set the width to 1.0 and the bias to 0 for the calibrated case; every other setting is a computation whose posterior is systematically wrong in a way SBC names precisely.

Two of Talts and colleagues' worked examples are worth internalizing because they recur constantly in practice. First, a deliberately misspecified prior — generate with a wide prior, fit with a narrow one — produces a textbook \( \cup \)-shape, the posterior too confident because the fitting prior refuses to let it spread. Second, and more consequentially, the classic "eight schools" hierarchical model fit in its centered parameterization produces a biased, asymmetric histogram: the sampler cannot navigate the funnel-shaped geometry near the group-variance parameter, so it systematically mis-estimates it. Switch to the non-centered parameterization and the histogram flattens. Same model, same data-generating process, different code — and only SBC catches that the first version was silently wrong. This is the single most important thing SBC buys you: hierarchical funnels are exactly the geometry a geo-hierarchical MMM lives in.

Deep diveFrom histograms to ECDF bands

Binned histograms have a real weakness: the conclusion can depend on the bin width. Säilynoja, Bürkner, and Vehtari (2022) show a striking case where the same 250-value sample sits comfortably inside a 95% band with 20 bins but breaches it with 50 bins — the binning choice flips the verdict, and re-binning until you like the answer is just multiple testing in disguise.

Their fix drops binning entirely. Apply the probability integral transform to the ranks and plot the empirical CDF of the resulting values against the uniform CDF, wrapped in simultaneous confidence bands that jointly hold at the stated coverage across every evaluation point at once. Subtracting the diagonal gives an ECDF-difference plot, which stretches the vertical scale so that small deviations at the extreme ranks — the ones a coarse histogram smears away — become visible. The bands come from the binomial distribution of the scaled ECDF with a one-time coverage adjustment, so there is no bin-width knob to fiddle and no post-hoc re-binning temptation. Talts and colleagues had already recommended pairing the histogram with an ECDF view; Säilynoja and colleagues supplied the rigorous band that makes it a formal test.

A few numbers from the original experiments calibrate expectations. Talts and colleagues used \( L = 100 \) posterior draws per fit over as many as 10,000 replications, with a rule of thumb of roughly 20 counts per bin to tame Monte-Carlo noise, and a variability band drawn from the binomial distribution of the bin counts. Because MCMC draws are autocorrelated — and autocorrelation shows up as those boundary spikes — you thin each fit down to near-independent draws before ranking. A convenient trick is to sample a number of draws one short of a power of two, say 1023, so that \( L + 1 = 1024 \) divides evenly when you re-bin.

What SBC Cannot See

SBC is a computational check, and it is disciplined about its own limits. In Talts and colleagues' words, it "offers no guarantee that the posterior will cover the ground truth for any single observation or that the model will be rich enough to capture the truth at all." It certifies that your sampler correctly infers the model you actually coded under data that model generates. Whether that model resembles reality is the job of posterior predictive checks on real data; whether a single fit is robust is the job of a sensitivity analysis. SBC and the real world never meet.

⚠️ Parameter-only SBC has a blind spot: the posterior that equals the prior

Almost all SBC in practice ranks individual parameters. Modrák and colleagues (2023) prove this misses an entire class of failures — most alarmingly, a broken computation whose posterior simply stays at the prior passes parameter-by-parameter SBC cleanly, because a posterior that never updates is trivially consistent with the prior on every marginal. The fix is to allow the ranked quantity \( f(\theta, y) \) to depend on the data, not just the parameters. Their headline recommendation: include the joint likelihood of the data as a default test quantity alongside the parameters. In their case studies, when the posterior wrongly equals the prior, every parameter passes but the likelihood-based quantity flags the problem "after just a handful of simulations"; when mean-field variational inference drops posterior correlations, the individual parameters look fine and the joint likelihood is the first to scream.

Modrák and colleagues also clear up a confusion the literature repeatedly makes: SBC (which constrains the quantile function of a test quantity) is not the same check as "the data-averaged posterior equals the prior" (which constrains its density). They are formally different, and conflating them is how the posterior-equals-prior blind spot went unnoticed for so long. The practical takeaway is a short recipe: rank the parameters and the joint log-likelihood, read the histograms or ECDF-difference plots, and spend your simulation budget on roughly as many datasets as posterior draws per dataset — with at least about a hundred effectively-independent draws per fit so that convergence is real and not an artifact of a stuck chain.

The Checks on a Media Mix Model

An MMM is almost a worst case for trusting a fit blindly, which makes it close to a best case for these two checks. Three features conspire.

The priors live far from the outcome scale. A coefficient prior in an MMM does not act on sales directly — it acts through a geometric adstock decay and a saturation curve, at the specific spend levels in your data. A prior that looks modest on the raw coefficient can, once pushed through those transforms, imply that a channel drives a wild share of revenue, or none. This is the Gabry air-pollution lesson transplanted to marketing: the only reliable way to see it is to simulate forward. A prior predictive check draws priors, builds a synthetic sales series, and lets you reject specifications that over- or under-credit media before fitting. It is also the natural home for ROI-scale priors — because an ROI prior is defined on the decision-relevant, pictureable scale, the prior predictive is the check that the implied prior ROI distribution is sensible: roughly break-even-centered, with a spread you would actually defend. The mechanics of carryover and shape, and why their parameters are hard to pin down, are the subject of a companion post.

Key parameters are weakly identified. The adstock decay rate and the saturation half-point are notoriously hard to recover from observational spend, which slides in lockstep with seasonality and demand. When the data are that uninformative, the prior does most of the work — so a bad prior is not a minor perturbation, it is the answer. Fake-data recovery is the honest upper bound here: if a model cannot recover a planted adstock rate from data it generated itself, it has no hope of recovering it from real, confounded data. That is a reason to be humble about point estimates, and a reason to check.

The geometry is a hierarchical funnel. A geo-level or nested MMM partially pools effects across regions, and partial pooling means the eight-schools funnel — the exact pathology where SBC earns its keep. R-hat and effective sample size can look pristine while a centered parameterization quietly mis-estimates the pooling variance; only calibration against known truth exposes it. And per Modrák and colleagues, rank the joint likelihood too — a parameter-only SBC on a geo model could pass while the sampler mis-handles cross-geo correlations or effectively ignores part of the panel. None of this is exotic; it is why the framework ships prior-predictive and SBC tooling as first-class steps rather than afterthoughts.

Where They Sit in the Workflow

These two checks are not standalone tricks; they are stations on a loop. The Bayesian workflow as Gelman and colleagues (2020) lay it out runs roughly: specify a model, prior predictive check, fit (cheaply first — "fit fast, fail fast"), simulation-based calibration and fake-data recovery, then posterior predictive checks and cross-validation on real data, then model expansion and comparison — and back around.

The connective tissue is what Gelman and colleagues call the folk theorem of statistical computing: when you have computational problems, often there is a problem with your model, not just your sampler. A chain that will not mix is frequently a chain trying to explore a pathological or nonsensical region — the correct response is often to fix the model or the prior, not to crank up the iteration count. SBC operationalizes this. A failing rank histogram does not just say "sample more"; its shape points at whether you have an over-confident prior, a biased parameterization, or an autocorrelation problem. Prior predictive checks close the loop from the other end, catching the pathological region before you ever try to sample it.

Takeaways

  • A Bayesian model is generative, so you can validate it before it sees real data. Prior predictive checks test the model; simulation-based calibration tests the computation.
  • Priors that look innocuous per parameter can imply impossible data jointly — simulate forward and inspect on the outcome scale, because that is the scale you have intuition for.
  • SBC rests on one identity: the data-averaged posterior equals the prior. The rank of each simulated truth among its posterior draws is uniform if and only if the computation is calibrated.
  • Read the histogram shape: \( \cup \) means overconfident, \( \cap \) means under-confident, a slope means bias, boundary spikes mean autocorrelation. ECDF-difference plots with simultaneous bands avoid the bin-width trap.
  • Rank the joint likelihood, not just parameters — parameter-only SBC cannot detect a posterior that wrongly stays at the prior (Modrák et al., 2023).
  • MMMs are the ideal customer: priors far from the outcome scale, weakly identified adstock and saturation, and hierarchical geo funnels where a clean R-hat can still hide a mis-computed posterior.

References