Heath 2015 - Regression-Based EVPPI (GAM and Gaussian Process)

Summary

Oakley and Strong et al. rewrite the inner conditional expectation of Eq. (1) as a regression: $N{B}_t({\theta }^s)=g_t({\varphi }^s)+{\epsilon }^s$. Fitting $g_t$ with a GAM or a Gaussian process to the PSA sample gives EVPPI for any subset $\varphi$ from a five-step recipe with no extra model runs. The paper judges this “the best method for calculating the EVPPI efficiently”.

Overview

Because ${\mathbb{E}}_{\psi \mid \varphi }[N{B}_t(\varphi ,\psi )]$ is a function of $\varphi$ only, the noisy simulated values $N{B}_t({\theta }^s)$ are unbiased observations of it at the points ${\varphi }^s$. Any flexible smoother then estimates the inner expectation; the noise ${\epsilon }^s\sim N(0,{\sigma }_{\epsilon }^2)$ is the contribution of $\psi$ given $\varphi$.

Main Content

Regression model for the inner expectation

$$N{B}_t({\theta }^s)={\mathbb{E}}_{\psi \mid {\varphi }^s}[N{B}_t({\varphi }^s,{\psi }^s)]+{\epsilon }^s=g_t({\varphi }^s)+{\epsilon }^s,{\epsilon }^s\sim N(0,{\sigma }_{\epsilon }^2).$$

The ${\varphi }^s$ are “observed covariates” and $N{B}_t({\theta }^s)$ the “observed response”, $s=\mathrm{1..}S$ simulations.

Regression EVPPI estimator (Strong et al. 2014)

Fit ${\hat{g}}_t$ for each option $t$ and let ${\hat{g}}_t({\varphi }^s)$ be the fitted values. Then

$$\widehat{\text{EVPPI}}=\frac{1}{S}\sum _{s=1}^S\underset{t}{\max }{\hat{g}}_t({\varphi }^s)-\underset{t}{\max }\frac{1}{S}\sum _{s=1}^S{\hat{g}}_t({\varphi }^s).$$

The five steps: (1) sample ${\theta }^s$; (2) compute $N{B}_t({\theta }^s)$; (3) regress each $N{B}_t$ on $\varphi$; (4) take fitted values; (5) apply the formula. Strong et al. supply R code with GAM (spline smoothers; gam chooses basis dimension automatically) and Gaussian process regression (a specific structure with analytic results, so no user-supplied model).

Implementations. R functions of Strong et al., the web app SAVI (Sheffield Accelerated Value of Information) with standardised graphics and tables, and the R package BCEA.

Properties. GAM: negligible cost for one parameter; user must specify interaction structure for larger subsets; “mathematically unstable for larger subsets of parameters ($\ge 6$)“. GP: feasible for large subsets, but compute is proportional to $N^3$ for a PSA of size $N$; an extension using geostatistics (INLA/SPDE, ref. [20]) is “being investigated”.

Examples

Verified toy example (run in this session): option 0 has $NB=0$, option 1 has $NB=0.3+0.5\varphi +0.5\psi$ with independent standard normals, $S=200,000$. A degree-5 polynomial regression stands in for the GAM.

import numpy as np
from scipy import stats
rng = np.random.default_rng(1); S = 200_000
phi, psi = rng.normal(size=S), rng.normal(size=S)
NB = np.stack([np.zeros(S), 0.3 + 0.5*phi + 0.5*psi])
base = NB.mean(1).max()
evpi = NB.max(0).mean() - base                     # 0.1575 (analytic 0.1571)
g = np.stack([np.polyval(np.polyfit(phi, NB[t], 5), phi) for t in range(2)])
evppi = g.max(0).mean() - g.mean(1).max()          # 0.0845 (analytic 0.0843)

Analytic values use $\mathbb{E}[\max (m+sZ,0)]=s\phi (m/s)+m\Phi (m/s)$ with $m=0.3$ and $s=\sqrt{0.5}$ (EVPI) or $s=0.5$ (EVPPI).

PyMC-style sketch (untested): draw the posterior with pm.sample, compute net-benefit arrays from idata.posterior, then call a GP/GAM regression (e.g. pm.gp.Marginal or pygam) on the chosen parameter column; the five-step formula above is pure NumPy on the fitted values.

Connections

• Estimating EVPPI is a regression of decision value on a parameter, so it inherits regression pathologies (extrapolation at tails, overfit with small $S$); report to 1-2 decimals as the paper advises.
• Marketing: with posterior draws from an MMM, run this for each channel’s response parameters to rank where better data pays; this is the decision-oriented complement of a sensitivity analysis.
• GP regression machinery: Gaussian Process Regression, and for large $N$ Hilbert Space Gaussian Processes.

See Also

• Heath 2015 - Single-Parameter EVPPI Strong-Oakley and Sadatsafavi
• Heath 2015 - Review of Expected Value of Information Methods - Overview