ABM Calibration Case Studies

Summary

McCulloch et al. (2022) validate the HM+ABC framework on three ABMs of increasing complexity: SugarScape (toy, 2 continuous parameters), territorial birds (2 continuous parameters, compared against alternative calibration methods), and RISC Scottish cattle farms (16 binary model variants, real-world policy application). All three demonstrate that HM successfully narrows the parameter space and ABC provides a meaningful posterior within it.

Case Study 0: SugarScape (Toy Example)

Example: SugarScape Step-by-Step (Section 4)

Model: Epstein & Axtell (1996). Grid environment with sugar resource. Agents move to maximise sugar intake; agents die if sugar falls to zero. Parameters:

• Maximum metabolism: $\in [1,4]$ (integer)
• Maximum vision: $\in [1,16]$ (integer)

Observation: Sustain a population of 66 agents. Generated by an “identical twin” model with metabolism=4, vision=6. Observation uncertainty $V_o=0$.

Ensemble: Variance stabilises at $K=200$ runs.

• Measured variance: $\{1,1\}\to 13.0$; $\{2,10\}\to 12.05$; $\{4,7\}\to 18.34$

HM (10 waves): Wave 1 identifies large implausible region (dark grey in figure); wave 10 narrows to upper-right corner (high metabolism + high vision). Space stops shrinking at wave 10.

ABC: Posterior concentrated at $\{$metabolism=4, vision=7$\}$. True parameters $\{4,6\}$ also obtain high probability. HM successfully eliminated low-metabolism, low-vision region before ABC.

Lesson: Even in this discrete, small parameter space, HM eliminates ~half the grid as implausible, making ABC sampling more efficient.

Case Study 1: Territorial Birds (Comparing Calibration Methods)

Example: Birds Model (Section 5.1 — Railsback & Grimm 2019)

Model: 25 territories in a 1D ring. Birds decide each month whether to scout for a new territory (scouting probability) and whether to survive the attempt (survival probability). Simulation runs 22 years (first 2 ignored). Three fitting criteria:

1. Average total number of birds
2. Standard deviation of total birds
3. Average number of locations lacking an alpha bird

Parameters: Scouting probability $\in [0,0.5]$; Survival probability $\in [0.95,1]$.

UQ:

• Observation uncertainty = empirical data range (interval, not a single value); error $e(f^r(x),z^r)=0$ if within range, else squared deviation from range boundary
• Ensemble variance: stabilises at $K=30$; one outlier sample excluded (out of 50 LHS samples)

HM (3 waves):

• Wave 1: 18/50 samples non-implausible
• Wave 3: Non-implausible space stabilises at scouting $\approx [0.2,0.5]$; survival $[0.9606,0.9925]$
• Model more sensitive to scouting than survival probability

ABC (1000 particles):

• HM-informed prior: Posterior concentrated at scouting $\in [0.2,0.5]$, survival $\approx 0.98$
• Uninformed prior: Posterior much broader — HM gives substantial information
• Results similar to Thiele et al. (2014) using ABC alone

Efficiency comparison:

• HM alone: 420 total model runs (80–320 + ensemble variance estimation)
• HM+ABC: 3,185 total model runs
• ABC alone (Thiele et al. 2014): 11,000+ runs for comparable posterior
• Simulated annealing: 256 runs (point estimate only)
• Evolutionary algorithms: 290 runs (point estimate only)

Accuracy (100 test cases): HM+ABC produces slightly narrower 95% CIs than ABC alone (smaller MAE: 0.130 vs 0.132 for scouting), but true parameter contained in CI ~90% of the time vs ~92% for ABC alone.

Case Study 2: RISC Scottish Cattle Farms (Real-World Application)

Example: RISC Model (Section 5.2 — Ge et al. 2018)

Model: Rural Industries Supply Chain (RISC) ABM. Simulates changes in Scottish cattle farm size 2000–2012. Each of 13 annual time steps, each farm owner makes decisions affecting farm size (small/medium/large). Model outputs: number of farms in each size category over time.

Parameters: 4 binary (on/off) decision factors:

• Succession: Owner has an heir to take over
• Leisure: Farming is a secondary income source
• Diversification: Farm diversifies into tourism
• Industrialisation: Professional manager could be employed

→ 16 model variants (all binary combinations). Goal: identify which variants are plausible.

Error metric — MASE (Mean Absolute Scaled Error, Eq. 8):

$$d^2(z^r,f^{rj}(x))=\frac{\frac{1}{n}{\sum }_{t=2}^T|f_t^{rj}(x)-z_t^r|}{\frac{1}{n-2}{\sum }_{t=2}^T|z_{t-1}^r-z_{t-2}^r|}$$

where $n=13$ (years), $j\in \{1,\dots ,16\}$ model variants. MASE normalises error by naive forecast variability.

UQ:

• Observation uncertainty negligible (mandatory survey, typographical errors assumed negligible)
• Ensemble variance estimated across 16 variants; one outlier excluded
• Model discrepancy quantified per Section 3.14

HM results: Multiple waves eliminate implausible model variants. The plausible variants are those that can recreate observed farm size polarisation (disappearance of medium-size farms).

ABC results: Posterior assigns probabilities to the 16 model variants, identifying which parameter combinations (which social factors are “on”) best explain Scottish farm size trends.

Key insight for Pattern-Oriented Modelling (POM): The framework applies naturally to POM, where the goal is to retain only model structures that can recreate observed patterns. HM eliminates implausible structures; ABC quantifies relative probabilities among the plausible ones.

Cross-Case Lessons

Aspect	SugarScape	Birds	RISC
Parameter type	Continuous (integer)	Continuous	Binary (16 models)
Complexity	Low	Medium	High (real policy data)
HM waves needed	10	3	Multiple
Key finding	HM narrows half the grid	2,047 fewer runs vs ABC alone	Identifies plausible policy scenarios
UQ challenge	None ($V_o=0$)	Interval observations	Multiple model outputs

Connections

• All three examples implement HM-ABC Calibration Framework
• RISC extends ABM Methodology and Principles to policy-relevant national-scale ABMs
• The birds comparison quantifies when Genetic Algorithm Calibration for ABM is faster but less informative
• RISC application relates to Pattern-Oriented Modelling discussed in ABM Calibration Overview

See Also

• HM-ABC Calibration Framework — the pipeline all examples implement
• History Matching for ABMs — the HM step in each case
• Approximate Bayesian Computation for ABMs — the ABC step
• Uncertainty Quantification for ABM Calibration — how uncertainties were quantified per case