Heath 2015 - Single-Parameter EVPPI Strong-Oakley and Sadatsafavi

Summary

Two methods that estimate EVPPI for a scalar $\varphi$ from PSA draws alone by sorting on $\varphi$ and exploiting local constancy of the best decision. Strong-Oakley averages net benefit in $M$ equal blocks; Sadatsafavi et al. cut at the points where the optimal decision flips. Both depend on a tuning input ($M$ or the number of decision changes) that can badly bias the answer.

Overview

Both methods rest on two ideas stated in Sec. 3.2: (i) provided the optimal decision stays constant, the first EVPPI term can be approximated by the average known-distribution net benefit within that set; (ii) an option optimal at one parameter value stays optimal “close to” it. With $S$ PSA draws $({\theta }^1,\dots ,{\theta }^S)$ this yields an estimator provided the decision is constant on the chosen sets.

Main Content

Strong-Oakley blocking estimator

1. Draw ${\theta }^1..{\theta }^S$.
2. Compute $N{B}_t({\theta }^s)$ for every option $t$.
3. Sort the draws so $\min \varphi ={\phi }_1<\cdots <{\phi }_S=\max \varphi$.
4. Reorder $N{B}_t({\theta }^s)$ to follow that order, $N{B}_t({\phi }_s)$.
5. Split into $m=\mathrm{1..}M$ sub-lists $L_m$ of length $L=S/M$ and average ${\overline{NB}}_t^m=\frac{1}{L}{\sum }_{j\in L_m}N{B}_t({\phi }_j)$.
6. $\widehat{\text{EVPPI}}=\frac{1}{M}{\sum }_{m=1}^M{\max }_t{\overline{NB}}_t^m-{\max }_t{\overline{NB}}_t$ (the second term is the overall best mean).

Tuning. “If $M$ is large, the EVPPI is close to the overall EVPI estimate. At the other end of the scale, if $M$ is small, the EVPPI is approximately 0.” Strong and Oakley use a normal approximation to the upward bias and choose $M$ as the largest number of subsets such that the bias is below a threshold, for example 0.1. In the vaccine case study that gave $M$ between 2 and 20 depending on the parameter.

Sadatsafavi et al. segmentation estimator

Same steps 1-4, then split the sorted list at $M$ cut points ${\phi }_{L_1}..{\phi }_{L_M}$ (the segmentation vector) so that sub-list $m$ holds all $\phi$ with ${\phi }_{L_m}\le \phi <{\phi }_{L_{m+1}}$; set

$$\widehat{\text{EVPPI}}=\underset{({\phi }_{L_1},\dots ,{\phi }_{L_M})}{\max }\frac{1}{M}\sum _{m=1}^M\underset{t}{\max }{\overline{NB}}_t^m-\underset{t}{\max }{\overline{NB}}_t.$$

It is asymptotically unbiased if the search uses the true number of decision changes.

Choosing the number of changes. A visual tool plots the cumulative sum of the difference in known-distribution net benefit between two options (ordered by $\varphi$); a change in the optimal decision appears as an extremum. The paper warns it “can be difficult to identify extrema making EVPPI estimates using this method unreliable in certain situations”, and that a parameter judged to have no decision changes is by that choice assigned EVPPI $=0$ (“visual indications are used to determine the EVPPI rather than objective analysis”).

Examples

Sketch (my code, untested on the paper’s data): for $\varphi$ scalar and two options,

import numpy as np
def evppi_blocks(phi, nb, M):
    """nb: (T, S) known-distribution net benefit; Strong-Oakley blocking with M blocks."""
    o = np.argsort(phi); nbs = nb[:, o]
    L = nb.shape[1] // M
    blk = nbs[:, :M*L].reshape(nb.shape[0], M, L).mean(axis=2)   # (T, M)
    return blk.max(axis=0).mean() - nb.mean(axis=1).max()

For the toy model in Heath 2015 - Net Benefit, EVPI and EVPPI Definitions, small $M$ underestimates and large $M$ drifts toward EVPI, as the quoted text predicts.

Connections

• Both are black-box in the sense of using only the PSA sample, but not black-box in the sense of needing no tuning; this is why Sec. 5 says they are “largely superseded” by Heath 2015 - Regression-Based EVPPI (GAM and Gaussian Process).
• Marketing: with a draw matrix from an MMM posterior, a one-parameter block estimator is a fast screening check on which single channel parameter matters.

See Also

• Heath 2015 - Review of Expected Value of Information Methods - Overview