Uncertainty Quantification for ABM Calibration

Summary

Calibrating ABMs requires explicitly identifying and quantifying four sources of uncertainty: parameter uncertainty (unknown parameters), model discrepancy (model is an imperfect abstraction), ensemble variance (stochastic variation across runs), and observation uncertainty (imperfect real-world measurement). All three variance components feed directly into the History Matching implausibility score and the ABC threshold $\epsilon$.

Overview

Standard calibration methods (GA, simulated annealing) minimize error without acknowledging that the model is stochastic and imperfect. This produces overconfident point estimates. UQ-based calibration explicitly measures each source of uncertainty and incorporates it into the calibration process.

Four Sources of Uncertainty

Definition: Parameter Uncertainty

Arises from not knowing which parameter values to use. The model may be insensitive to different parameter values — the identifiability / equifinality problem: many different parameter combinations may produce similarly acceptable outputs. Addressed by the calibration process itself (HM narrows the space; ABC quantifies the distribution).

Definition: Model Discrepancy ( $V_m^r$)

The difference between the best possible model output and the true data-generating mechanism. Even with perfect parameters, an ABM is an abstraction — it omits real-world processes, simplifies behaviour, and cannot perfectly replicate the target system.

$$V_m^r=\frac{1}{N-1}\sum _{n=1}^N{\left(d(z^r,f^r(x_n))-E^r(X)\right)}^2$$

Key implication: Model discrepancy cannot be reduced by better calibration. It must be acknowledged honestly — ignoring it inflates the implausibility score and causes the HM procedure to retain too few parameter sets.

Definition: Ensemble Variance ( $V_s^r$)

Arises from stochastic variation across runs with identical parameters. ABMs use random number generators, so each run produces a different output trajectory.

$$V_s^r=\frac{1}{N}\sum _{n=1}^N\left[\frac{1}{K-1}\sum _{k=1}^K{\left(d(z^r,f_k^r(x_n))-E_K^r(x_n)\right)}^2\right]$$

Choose the ensemble size $K$ by plotting variance vs. $K$ and selecting the smallest $K$ at which variance stabilises (e.g., $K=200$ for SugarScape, $K=30$ for birds).

Definition: Observation Uncertainty ( $V_o$)

Arises from imperfect measurement of the real-world system being modelled. Sources:

• Direct observations: If multiple measurements are available, use their variance directly
• Indirect measurements: Expert judgment or quantiles of the expected output characterise the uncertainty
• No uncertainty: Toy models like SugarScape where the “observation” is generated by an identical twin model have $V_o=0$

The Total Uncertainty Budget

All three variance components feed into both HM and ABC:

HM implausibility denominator (Eq. 1):

$$I^r(x)=\frac{d^2(z^r,f^r(x))}{V_s^r+V_o^r+V_m^r}$$

ABC threshold (Eq. 5):

$$\epsilon =3(V_o+V_s^r+V_m^r)$$

These ensure that the calibration criteria are scaled to the actual uncertainty in the system, preventing both over-rejection (if uncertainties are underestimated) and under-rejection (if uncertainties are overestimated).

Why Traditional Calibration Fails Without UQ

Consider a GA that minimises $d(z^r,f^r(x))$ without accounting for uncertainty:

• It treats every improvement in fit as meaningful, even if it’s within the noise floor set by $V_s^r+V_o+V_m^r$
• It produces a single “best” parameter set with no sense of how much better it is than nearby sets
• It cannot distinguish true model inadequacy from random variation

UQ-based calibration — HM + ABC — acknowledges these limitations and produces calibrated uncertainty estimates.

Connections

• The three variance components directly extend ABM Calibration Overview’s discussion of calibration challenges
• Comparison to Bayesian Workflow - Overview — similar decomposition of uncertainty in Bayesian modeling (prior, likelihood, model checking)
• Compare to Genetic Algorithm Calibration for ABM — GA ignores $V_m^r$ and $V_s^r$ in its fitness function

See Also

• History Matching for ABMs — uses all three variance components in the implausibility score
• Approximate Bayesian Computation for ABMs — uses total uncertainty to set $\epsilon$
• HM-ABC Calibration Framework — the pipeline these uncertainties enable
• Global Sensitivity Analysis - Overview — variance attribution complements UQ