Index: Probabilistic Numerics

Probabilistic Numerics

Routing Summary

Ingest of the textbook Hennig, Osborne & Kersting, Probabilistic Numerics: Computation as Machine Learning (Cambridge, 2022). PN recasts numerical tasks — integration, linear algebra, optimisation, ODEs — as Bayesian inference returning a calibrated posterior, and treats a solver as a decision-making agent. The unifying result: many classical methods (trapezoid/Gauss quadrature, Conjugate Gradients, BFGS, Runge–Kutta) are the posterior mean of a specific Gaussian inference procedure. Contains 36 notes across 5 sub-topics.

• The big picture / where to start? → Probabilistic Numerics - Overview
• The Gaussian inference toolbox everything is built on? → Foundations
• Integration & Bayesian quadrature? → Integration
• Solving linear systems $Ax=b$ as inference? → Linear Algebra
• Local optimisation, Bayesian optimisation, acquisition functions? → Optimisation
• Solving ODEs with filters/smoothers? → Differential Equations

Sub-topics

• Foundations — COVERS (Part I): the shared machinery — the PN thesis and agent view; Gaussian algebra (conditioning/affine maps); GP regression (RKHS, worst-case error); SDE priors (IWP, Matérn, IOUP); Kalman filter & RTS smoother as $\mathcal{O}(N)$ GP regression; scale calibration. (7 notes.)
• Integration — COVERS (Part II): the quadrature task; Bayesian quadrature (GP-integral posterior, BQ weights); kernel means & worst-case error; Clenshaw–Curtis as posterior means; convergence & scale inference; active BQ, WSABI, Bayesian Monte Carlo; transferable PN lessons. (7 notes.)
• Linear Algebra — COVERS (Part III): the spd solve & matrix-vector observations; conjugacy; matrix-variate Gaussians; the general probabilistic solver; CG = BayesCG posterior mean; [[Computational Constraints on Probabilistic Solvers|keeping $\mathcal{O}(\cdot )$ cost]]; scale calibration. (7 notes.)
• Optimisation — COVERS (Parts IV–V): local — noisy gradient minimisation, probabilistic line search & Wolfe conditions, BFGS-as-inference, probabilistic gradients; global — expensive black-box optimisation, the BO loop, value-of-information & entropy search, KG closed forms, AutoML. (8 notes.)
• Differential Equations — COVERS (Parts VI–VII): the IVP as regression; multistep as posterior means; EKF1 ODE filter + RTS smoother; convergence, A-stability, calibration; additive-noise solvers; inverse problems, PDEs, the frontier. (6 notes.)

Cross-Cutting Concepts

Concepts that span multiple sub-topics:

• Classical method = Gaussian posterior mean — the book’s signature result, instantiated in Classical Quadrature as Inference (trapezoid/Gauss), Conjugate Gradients as Probabilistic Inference (CG), First- and Second-Order Optimisation Methods (BFGS), and Classical ODE Solvers as Regression (Runge–Kutta).
• Bayesian filtering & smoothing as the engine — Bayesian Filtering and Smoothing underlies both state-space GP regression and the ODE filter; the same IWP/Matérn Gauss–Markov priors appear in ODEs and in $\mathcal{O}(N)$ quadrature (Classical Quadrature as Inference).
• Gaussian conditioning — the master formula from Gaussian Distributions and Algebra produces every closed-form posterior in Bayesian Quadrature, Probabilistic Linear Solvers - Algorithmic Scaffold, and Gaussian Process Regression.
• Active evaluation / value of information — choosing the next evaluation to shrink posterior uncertainty: Active Bayesian Quadrature and Bayesian Monte Carlo, Acquisition Functions, Value Loss and Entropy Search, and probabilistic-line-search node selection (Probabilistic Step-Size Selection and Line Searches).
• Runtime scale calibration — the same Gauss–Gamma conjugate trick calibrates uncertainty in Hierarchical Inference in Gaussian Models, Convergence and Priors in Bayesian Quadrature, and Uncertainty Calibration for Linear Solvers.
• Consolidating numerics with statistics — the keystone payoff: a probabilistic solver inside an inverse problem reduces bias — Further Topics in ODE Solvers (§41.3).

Concept Dependency Chain

Computation as Probabilistic Inference / The Numerical Agent → Gaussian Distributions and Algebra → {Gaussian Process Regression, Gauss-Markov Processes and SDEs → Bayesian Filtering and Smoothing} → Hierarchical Inference in Gaussian Models; then each application branch:

• Integration: The Integration Problem → Bayesian Quadrature → Kernel Quadrature and Kernel Means → Classical Quadrature as Inference → Convergence and Priors in Bayesian Quadrature → Active Bayesian Quadrature and Bayesian Monte Carlo → Lessons from Integration.
• Linear algebra: The Linear Algebra Problem and Evaluation Strategies → Classic Linear Solvers - A Review → Gaussian Priors over Matrices and the Symmetric Kronecker Product → Probabilistic Linear Solvers - Algorithmic Scaffold → Conjugate Gradients as Probabilistic Inference → Computational Constraints on Probabilistic Solvers → Uncertainty Calibration for Linear Solvers.
• Optimisation: The Local Optimisation Problem → Probabilistic Step-Size Selection and Line Searches → First- and Second-Order Optimisation Methods; The Global Optimisation Problem → Bayesian Optimisation → Value Loss and Entropy Search → Acquisition Functions → Further Topics in Global Optimisation.
• ODEs: Solving ODEs as Inference → Classical ODE Solvers as Regression → ODE Filters and Smoothers → Theory of ODE Filters and Smoothers → Perturbative ODE Solvers → Further Topics in ODE Solvers.

Sources

• ProbabilisticNumerics.pdf — Philipp Hennig, Michael A. Osborne, Hans P. Kersting, Probabilistic Numerics: Computation as Machine Learning, Cambridge University Press, 2022. (Draft/pre-publication copy, 412 pp.)

See Also

• Bayesian Statistics — the inference, GP, and conjugate-prior foundations PN reuses
• Bayesian Experimental Design — the expected-information-gain logic mirrored in active evaluation and acquisition functions
• Econometrics · Market Response Models — downstream users of GP regression and Bayesian optimisation