Index: Calibration and Validation

Calibration and Validation

Routing Summary

This topic covers the challenges and methods for calibrating and validating agent-based models. Contains 4 sub-topics and 15 total notes.

• For calibration methods (GA, HM+ABC, uncertainty quantification) → Calibration Methods
• For experimental design and parameter sensitivity → Experimental Design
• For global (variance-based) sensitivity analysis — Sobol, Morris, FAST → Sensitivity Analysis
• For validation challenges and standards → Validation

Sub-topics

• Calibration Methods — COVERS: ABM calibration overview (3 approaches), GA calibration (chromosome encoding, GA operators, convergence), Result-Analysis Module (macro/micro fitness), HM+ABC framework (implausibility score $I^r(x)=d^2/(V_s+V_o+V_m)$, wave-based pruning, ABC rejection sampling, $\epsilon =3(V_o+V_s+V_m)$), uncertainty quantification (4 sources: parameter uncertainty, model discrepancy, ensemble variance, observation uncertainty), case studies (SugarScape 10 waves, territorial birds 3,185 vs 11,000+ runs, RISC Scottish farms 16 binary variants + POM)
• Experimental Design — COVERS: parameter initialization distributions, one-at-a-time experimental design, 100-replication strategy, WOM toggle, benchmark configuration, sensitivity findings
• Sensitivity Analysis — COVERS: global (variance-based) sensitivity analysis — Sobol first-order & total-effect indices, the ANOVA/HDMR decomposition, Morris elementary-effects screening (${\mu }^{\ast },\sigma$), Saltelli sampling & FAST, local-vs-global pitfalls, SALib (5 notes; GSA review 2024)
• Validation — COVERS: Merson’s plausibility criterion, Troitzsch’s systematic validation difficulty, input-output mismatch, stochastic variation, the plausibility standard

Cross-Cutting Concepts

• Micro-macro gap: All sub-topics address the challenge of connecting agent-level parameters to population-level observables — calibration searches for the right micro parameters, experimental design explores their effects, and validation assesses whether the mapping is correct
• Stochasticity management: Replication (Karakaya, 100 runs), population-level fitness (Ben Said, GA), analytical baselines (Bonabeau), and ensemble variance quantification (McCulloch HM+ABC, $K$ runs per parameter set until variance stabilises) are all strategies for handling stochastic variation
• Uncertainty acknowledgment: The HM+ABC framework makes explicit what GA/SA/EA methods ignore — model discrepancy, ensemble variance, and observation uncertainty must be quantified and incorporated into the calibration criterion, not minimized away
• Point estimate vs. posterior: GA/simulated annealing/evolutionary algorithms find the best-fitting parameter set (computationally cheap, ~256–290 runs); HM+ABC finds the full posterior distribution of plausible parameters (more informative, ~3,185 runs); both are appropriate depending on whether uncertainty quantification is required

Sources

• abm_human_behaviour.pdf — GA calibration and RAM (Ben Said et al. 2002)
• abm_consumer.pdf — Experimental design and sensitivity (Karakaya et al. 2011)
• abm_word_of_mouth.pdf — Analytical baselines and validation discussion (Bonabeau 2002)
• calibration_ABM.pdf — HM+ABC framework with UQ (McCulloch et al. 2022, JASSS 25(2))