Model Class Reliance - Toy Example and Bootstrap Coverage Simulations

Summary

§9.1 uses a 2-D classification toy problem (one class inside, one on a ring) with degree-3 polynomial classifiers and hinge loss to contrast AR (refit without $X_1$), MR ($\widehat{MR}(f_{\text{ref}})=3.47$) and MCR (the range of MR over 15+ near-optimal classifiers). §9.2 is a Monte Carlo study of misspecification. The true function is quadratic with nonlinearity $\gamma \in \{0,\dots ,0.5\}$ and is fit with a linear class. Bootstrap MCR intervals (“MCR-Linear”) keep 95% coverage of $MR(f_0)$ up to $\gamma =0.3$, where a standard bootstrap of one fitted model’s MR breaks down. Both fail for $\gamma \ge 0.4$. MCR intervals are about 2-4 times wider, larger $n$ makes misspecification coverage worse, and the remaining coverage errors come from the class not containing an MR-matching model (Condition 20), not from the bootstrap (Condition 21).

Overview

The two experiments have different jobs. The toy example builds intuition for the three importance notions in a single sample. The simulation tests the central claim: MCR can cover the reliance of an unknown true function $f_0$ even when the class is misspecified, provided some good model in the class matches $f_0$‘s reliance. Read through ADEMP: the Aim is coverage of $MR(f_0)$; the Data-generating mechanisms vary $\gamma$ and $n$; the Estimand is $MR(f_0)$ for $X_1$ and $X_2$; the Methods are MCR-Linear and Standard-Linear; the Performance measure is coverage of nominal 95% CIs.

Main Content

§9.1 Illustrative toy example (pp. 34-36)

Framing (§9.1, p. 34). The goal is intuition for how MR, MCR and AR differ, so all three are shown in a single sample and compared as empirical quantities ($\widehat{MR}$, ${\widehat{MCR}}_{-}$, ${\widehat{MCR}}_{+}$ against AR). The reason given is that AR is typically interpreted either as an in-sample measure (Breiman 2001) or as an intermediate step toward variable rankings (Gevrey et al. 2003; Olden et al. 2004), not as an estimate of a population quantity. So the comparison is like-for-like only at the sample level.

Design of the polynomial-classifier toy example (§9.1)

Data.

• $X=(X_1,X_2)\in {\mathbb{R}}^2$ and $Y\in \{-1,1\}$.
• $X\mid Y=-1\sim$ independent bivariate normal with means 0 and variances $\frac{1}{9}$.
• $X\mid Y=1$: the same draw plus $(C_1,C_2)=(\cos U,\sin U)$ with $U\sim \text{Uniform}[-\pi ,\pi ]$, a point uniform on the unit circle. So class $+1$ is a noisy ring around class $-1$.
• Two samples of size 300: one for training, one for testing.

Prediction and loss. Predict $\mathrm{sign}f(X_1,X_2)$. Use hinge loss $(1-yf{)}_{+}$, a convex surrogate for 0-1 loss $1[y\ne \mathrm{sign}f]$.

Class. Degree-3 polynomial classifiers

$${\mathcal{F}}_{d3}=\{f_{\theta }:f_{\theta }={\theta }_1+{\theta }_2{x}_1+{\theta }_3{x}_2+{\theta }_4{x}_1^2+{\theta }_5{x}_2^2+{\theta }_6{x}_1{x}_2+{\theta }_7{x}_1^3+{\theta }_8{x}_2^3+{\theta }_9{x}_1^2{x}_2+{\theta }_{10}{x}_1{x}_2^2;\Vert {\theta }_{[-1]}{\Vert }_2^2\le r_{d3}\},$$

with $r_{d3}$ chosen by 10-fold CV on the training data.

Reference and threshold. ${\mathcal{A}}_{d3}$ minimizes hinge loss over the convex feasible region. $f_{\text{ref}}={\mathcal{A}}_{d3}(\text{training})$, and $ϵ=0.10\times$ the CV loss of ${\mathcal{A}}_{d3}$, so $\mathcal{R}(ϵ)$ contains models within about 10% of $f_{\text{ref}}$‘s loss. AR, MR and MCR are computed on the test sample, with $\widehat{MR}$ via ${\hat{e}}_{\text{divide}}$.

Results.

• AR (Fig. 6-A). Refitting without $X_1$ gives ${\hat{f}}_2$ (blue dashed); refitting with only $X_1$ gives ${\hat{f}}_1$ (solid blue). Comparing ${\hat{f}}_2$ with $f_{\text{ref}}$ gives AR-necessity; comparing ${\hat{f}}_1$ with $f_{\text{ref}}$ gives AR-sufficiency. Single-input boundaries are straight lines.
• MR (Fig. 6-C). $\widehat{MR}({\hat{f}}_2)=1$, as expected for a model that never sees $X_1$, and $\widehat{MR}(f_{\text{ref}})=3.47$. “Each $\widehat{MR}$ value has an interpretation contained to a single model” (p. 35). It compares one model across data distributions, not different models on marginals of one joint distribution.
• MCR (Fig. 6-B). Fifteen well-performing models in $\hat{\mathcal{R}}(ϵ,f_{\text{ref}},{\mathcal{F}}_{d3})$ appear as grey oval-like boundaries. ${\hat{f}}_{+,ϵ}$ (solid green) and ${\hat{f}}_{-,ϵ}$ (dashed green) approximately maximize and minimize empirical reliance on $X_1$. In Fig. 6-C, every model in $\hat{\mathcal{R}}(ϵ)$ has $\widehat{MR}$ between $\widehat{MR}({\hat{f}}_{-,ϵ})$ and $\widehat{MR}({\hat{f}}_{+,ϵ})$, up to small approximation error.
• Takeaway: “unlike AR, MCR is only a function of models that fit the data well” (p. 35).

Figure 6 (p. 36): AR, MCR and MR for polynomial classifiers

Panels (A) and (B) show the same 300 draws from the simulated data set.

• (A) For AR: points (x for $Y=1$, o for $Y=-1$) with the $f_{\text{ref}}$ boundary (dotted black) and the single-feature models formed by dropping a covariate, ${\hat{f}}_1$ and ${\hat{f}}_2$. The caption explains that these are drawn as straight lines because each takes only a single input.
• (B) For MCR: the same points with the classification boundaries of several two-feature models with low in-sample loss (grey). The model with minimal dependence on $X_1$, ${\hat{f}}_{-,ϵ}$, is the dashed green oval, and the model with maximal dependence, ${\hat{f}}_{+,ϵ}$, is the solid green oval.
• (C) A strip plot of empirical MR on $X_1$ (y-axis about 1.0-4.0) for ${\hat{f}}_1$, ${\hat{f}}_2$, $f_{\text{ref}}$, the good models, ${\hat{f}}_{+,ϵ}$ and ${\hat{f}}_{-,ϵ}$. No good model exceeds ${\widehat{MCR}}_{+}(ϵ)$ or falls below ${\widehat{MCR}}_{-}(ϵ)$.

§9.2 Bootstrap CIs under misspecification (pp. 36-38)

Conditions 20-21 (p. 36)

• Condition 20 (nearly correct model class): $\mathcal{F}$ contains a well-performing $\tilde{f}\in \mathcal{R}(ϵ,f_{\text{ref}},\mathcal{F})$ with $MR(\tilde{f})=MR(f_0)$. Equivalently, $MR(f_0)$ is “well supported” by $\mathcal{F}$ even if $f_0\notin \mathcal{F}$.
• Condition 21 (bootstrap coverage): bootstrap CIs for empirical MCR give appropriate coverage of population MCR.

Condition 20 makes $[MC{R}_{-},MC{R}_{+}]\ni MR(f_0)$. Condition 21 makes that interval estimable in finite samples. Coverage of $MR(f_0)$ requires both.

Goals of §9.2 (p. 37). The primary goal is to assess whether CIs derived from MCR give appropriate coverage of $MR(f_0)$, which depends on both conditions. As a secondary goal, the authors want to assess Conditions 20 and 21 individually. That is why the design needs population-level MCR (computed on the finite population of 20,000), so that the two failure sources can be told apart. The result (item 5 below) is that coverage failures trace to Condition 20, while Condition 21 holds conservatively.

Source typo. The sentence interpreting Condition 20 reads that “the model reliance value of $MR(f_c)$ is ‘well supported’ by the class $\mathcal{F}$, even if $\mathcal{F}$ does not contain $f_0$” (p. 37). Given Condition 20’s statement, $MR(f_c)$ should read $MR(f_0)$. No $f_c$ is defined in §9.2.

Simulation design: MCR-Linear vs Standard-Linear (§9.2, pp. 36-37)

Population. A finite population of 20,000 observations, so population MCR can be computed exactly. Samples are drawn with replacement from it.

Data-generating model. For $\gamma \in \{0,0.1,0.2,0.3,0.4,0.5\}$:

• $Y=f_0(X)+E$ with $f_0(x)={\sum }_{j=1}^p(j{x}_{[j]}-\gamma x_{[j]}^2)$ and $p=2$, so $X_2$ has twice the linear coefficient of $X_1$;
• $X\sim$ bivariate normal with $\mathbb{E}{X}_1=\mathbb{E}{X}_2=0$ and $\mathrm{Cov}(X_1,X_2)=1/4$. The paper prints $\mathrm{Var}(X_1)=\mathrm{Var}(X_2)=0$, an evident typo since a nonzero covariance requires positive variances; unit variances are the natural reading;
• $E\sim N(0,{\sigma }_E^2)$ with ${\sigma }_E^2:=\mathrm{Var}(f_0(X))$, i.e. signal-to-noise 1.

Sample sizes. $n=400$ with $n_{tr}=200$ training, and $n=800$ with $n_{tr}=300$.

MCR-Linear.

1. Split sample ${\mathcal{Z}}_s$ into training and analysis subsets.
2. Fit $f_{\text{ref},s}$ on the training subset, within ${\mathcal{F}}_{\text{lm}}=\{f_{\beta }:f_{\beta }(x)={\beta }_1+{\sum }_{j=1}^2{x}_{[j]}{\beta }_{j+1},\beta \in {\mathbb{R}}^3\}$.
3. Set $ϵ=0.1\times {\sigma }_E^2$.
4. Draw 500 bootstrap samples from the analysis subset. In each, compute ${\widehat{MCR}}_{-}(ϵ)$ and ${\widehat{MCR}}_{+}(ϵ)$ by optimizing over $\hat{\mathcal{R}}(ϵ,f_{\text{ref},s},{\mathcal{F}}_{\text{lm}})$.
5. CI = [2.5th percentile of ${\widehat{MCR}}_{-}$, 97.5th percentile of ${\widehat{MCR}}_{+}$].
6. Repeat for $X_1$ and for $X_2$.

Standard-Linear (comparator).

1. Draw 500 bootstrap samples from ${\mathcal{Z}}_s$.
2. In each, train $f_b\in {\mathcal{F}}_{\text{lm}}$ on $n_{tr}$ points and compute $\widehat{MR}(f_b)$ on the rest.
3. CI = 2.5th-97.5th percentiles of $\widehat{MR}(f_b)$.

Estimand and performance. Coverage of $MR(f_0)$ by nominal 95% CIs.

Results (§9.2.1, pp. 37-38; Fig. 7)

1. MCR-Linear gives proper coverage up to moderate misspecification ($\gamma =0.3$), where Standard-Linear begins to break down.
2. For $\gamma \ge 0.4$, both MCR-Linear and Standard-Linear fail to reach 95% coverage.
3. Larger $n$ generally worsens coverage under misspecification: “methods become more certain of biased results” (p. 37).
4. MCR-Linear intervals are roughly 2-4 times wider than Standard-Linear, partly because they target a range $[MC{R}_{-},MC{R}_{+}]$ (defined using $f_{\text{ref},s}$) rather than a point.
5. MCR-Linear’s coverage errors come mostly from violations of Condition 20. Condition 21 holds conservatively in every scenario: at least 95.9% of bootstrap CIs contained population MCR.

Figure 7 (p. 38): MR coverage

• Layout: two panels, reliance on $X_1$ (left) and on $X_2$ (right). The x-axis is the degree of nonlinearity $\gamma \in \{0,\dots ,0.5\}$ and the y-axis is coverage from about 0.7 to 1.0, with a dashed line at the 95% target.
• Series: colour = method (Standard vs MCR); line style = sample size (400 vs 800).
• Pattern: MCR curves stay at or near 1.0 longer than the Standard curves. All curves drop at high $\gamma$, and the $n=800$ curves drop further.

Exact coverage values are shown only graphically.

Interpretation (p. 38). MCR is generic. It assumes no specific route for misspecification, so it “is less powerful than sensitivity analyses which make that assumption correctly”. It still adds robustness, and with a strong enough signal it returns an informative interval, as in the COMPAS analysis.

Examples

Why larger samples hurt under misspecification

Standard-Linear’s CI shrinks around $MR({\hat{f}}_{\text{lin}})$. Under curvature, the best linear fit splits signal between $X_1$ and $X_2$ differently from $f_0$, so its MR is biased. More data makes the CI tighter around the wrong value. MCR’s interval also tightens, but it keeps every nearly optimal linear fit, and some of these may match $MR(f_0)$. That is Condition 20. The same logic applies when a linear MMM is fit to saturating media effects: more weeks of data do not fix a misspecified response curve (Shape of the Marketing Response Function).

Connections

• Bootstrapping a single selected model ignores model-selection uncertainty, one face of the Garden of Forking Paths. MCR’s bootstrap over the Rashomon set partly restores it.
• Simulation-design checklist: Designing Simulated-Data Experiments, Simulation Studies for Evaluating Statistical Methods (ADEMP), Morris White Crowther 2019 - Simulation Studies (ADEMP) - Overview. The paper reports no Monte Carlo SEs for coverage and gives no explicit number of simulation repetitions in the text.
• Coverage under misspecification is a recurring validation theme: Cross Validation Checking, Simulation-Based Calibration - Overview.

See Also

• Model Class Reliance - Recidivism Case Study: the same bootstrap applied to real data
• Model Class Reliance - Outer and Inner Finite-Sample Bounds: why theory alone is not used for CIs
• Fisher Rudin Dominici 2019 - Model Class Reliance - Overview