Rashomon-Set Confidence Intervals and Profile Likelihood

Summary

Section 5 generalizes the MCR argument from model reliance to any model characteristic $\varphi :\mathcal{F}\to \mathbb{R}$, such as a coefficient norm or a prediction at a covariate profile. Proposition 7: with ${ϵ}^{\prime }=ϵ+2B_{\text{ref}}\sqrt{\log (2{\delta }^{-1})/2n}$, the interval $[{\min }_{\hat{\mathcal{R}}({ϵ}^{\prime })}\varphi ,{\max }_{\hat{\mathcal{R}}({ϵ}^{\prime })}\varphi ]$ contains $\{\varphi (f):f\in \mathcal{R}(ϵ)\}$ with probability at least $1-\delta$. This holds in finite samples, without asymptotics and without a correct likelihood. When $L$ is a negative log-likelihood, the classic profile-likelihood interval is exactly a min/max over an empirical Rashomon set with threshold ${\chi }_{1,1-\delta }^2/2n$. The Rashomon-set threshold shrinks more slowly than that, so it is more conservative.

Overview

The idea behind Theorem 4, isolated here, is that “models with near-optimal performance in the population tend to have relatively good performance in random samples” (§5, p. 15). That turns finite-sample inference into in-sample optimization. To bound a property of the good models, optimize that property over a slightly enlarged empirical Rashomon set.

Main Content

Proposition 7: Finite-sample CIs from Rashomon sets (§5.1, p. 17)

Let $\varphi :\mathcal{F}\to \mathbb{R}$ be a characteristic of interest that can be computed exactly for any $f$. Let

$${ϵ}^{\prime }:=ϵ+2B_{\text{ref}}\sqrt{\frac{\log (2{\delta }^{-1})}{2n}},{\hat{\varphi }}_{-}({ϵ}^{\prime }):=\underset{f\in \hat{\mathcal{R}}({ϵ}^{\prime })}{\min }\varphi (f),{\hat{\varphi }}_{+}({ϵ}^{\prime }):=\underset{f\in \hat{\mathcal{R}}({ϵ}^{\prime })}{\max }\varphi (f).$$

If Assumption 2 holds for all $f\in \mathcal{R}(ϵ)$, then

$$\mathbb{P}(\{\varphi (f):f\in \mathcal{R}(ϵ)\}\subseteq [{\hat{\varphi }}_{-}({ϵ}^{\prime }),{\hat{\varphi }}_{+}({ϵ}^{\prime })\left]\right)\ge 1-\delta .$$

The CI has a rigorous coverage rate and a coherent in-sample reading: it is the range of $\varphi$ over models whose empirical loss is not much above that of $f_{\text{ref}}$.

Examples of $\varphi$: for $f_{\beta }(x)=x^{\prime }\beta$, $\varphi (f_{\beta })=\Vert \beta {\Vert }_2^2$ or $\varphi (f_{\beta })=f_{\beta }(x_{\text{new}})$, the risk assigned to a given covariate profile.

Lemma 26: One good model is captured (App B.6, p. 62)

Let ${ϵ}_1^{\prime }:=ϵ+2B_{\text{ref}}\sqrt{\log ({\delta }^{-1})/2n}$. For any $f_1\in \mathcal{R}(ϵ)$ satisfying Assumption 2, $\mathbb{P}\{\varphi (f_1)\in [{\hat{\varphi }}_{-}({ϵ}_1^{\prime }),{\hat{\varphi }}_{+}({ϵ}_1^{\prime })]\}\ge 1-\delta$.

Proof. The event holds whenever $f_1\in \hat{\mathcal{R}}({ϵ}_1^{\prime })$, which fails with probability at most $\delta$ by Lemma 23.

Source note. As printed, Lemma 26 defines ${ϵ}_1^{\prime }:=2B_{\text{ref}}\sqrt{\log ({\delta }^{-1})/2n}$ without the leading $ϵ$. The proof invokes Lemma 23, whose ${ϵ}_1^{\prime }$ is $ϵ+2B_{\text{ref}}\sqrt{\log ({\delta }^{-1})/2n}$, so the $ϵ+$ form above is the one the proof supports.

Proof of Proposition 7 (App B.6.1, p. 63). Let $f_{-,ϵ,\varphi }$ and $f_{+,ϵ,\varphi }$ attain the min and max of $\varphi$ over $\mathcal{R}(ϵ)$. The set $\{\varphi (f):f\in \mathcal{R}(ϵ)\}$ escapes the interval only if one of these two endpoints does. Lemma 26 at $\delta /2$ for each, plus a union bound, gives $\delta /2+\delta /2=\delta$. That is why $\log (2{\delta }^{-1})$ appears in ${ϵ}^{\prime }$.

Corollary (§5.1, p. 17): All best-in-class models

Setting $ϵ=0$ in Proposition 7 gives a finite-sample CI covering $\varphi (f^{\star })$ simultaneously for every $f^{\star }\in \arg {\min }_{f\in \mathcal{F}}\mathbb{E}L(f,Z)$.

Corollary 27: Unique best-in-class model (App B.6.2, p. 63)

Let ${ϵ}_0^{\prime }:=2B_{\text{ref}}\sqrt{\log ({\delta }^{-1})/2n}$, and let ${\hat{\varphi }}_{\pm }({ϵ}_0^{\prime })$ be the min and max of $\varphi$ over $\hat{\mathcal{R}}({ϵ}_0^{\prime })$. If $f^{\star }$ uniquely minimizes $e_{\text{orig}}$ over $\mathcal{F}$ and satisfies Assumption 2, then $\mathbb{P}\{\varphi (f^{\star })\in [{\hat{\varphi }}_{-}({ϵ}_0^{\prime }),{\hat{\varphi }}_{+}({ϵ}_0^{\prime })]\}\ge 1-\delta$.

Proof. $f^{\star }\in \mathcal{R}(0)$, so apply Lemma 26. Uniqueness means only one model must be captured, so no union bound is needed and the threshold uses $\log ({\delta }^{-1})$ instead of $\log (2{\delta }^{-1})$. This is a slightly tighter statement.

Caveat: characteristics that must be estimated (§5.1, p. 17)

Proposition 7 assumes $\varphi (f)$ is known exactly once $f$ is known. That fails for $\varphi (f)=MR(f)$ or $\varphi (f)=\mathrm{Var}\{f(X_1,X_2)\}$, which depend on the unknown population. Then a correction for the estimation error of $\varphi$ is needed, such as the ${\mathcal{Q}}_{\text{out}}$ term of Theorem 4.

Profile likelihood as a Rashomon set (§5.1 pp. 17-18; App A.6)

Profile-likelihood intervals are min/max over an empirical Rashomon set (App A.6, Eq A.5, pp. 46-47; Coker et al. 2018)

Setup.

• Models $f_{\theta }$ are indexed by $\theta =(\gamma ,\psi )\in \Theta$, with scalar parameter of interest $\gamma$ and nuisance $\psi$.
• $e_{\text{orig}}(f_{\theta })$ has a unique minimizer ${\theta }^{\star }=({\gamma }^{\star },{\psi }^{\star })$.
• If $s_{\theta }:=\int \exp \{-L(f_{\theta },z)\}dz<\infty$, define the likelihood $\mathcal{L}(z;\theta )=\exp \{-L(f_{\theta },z)\}/s_{\theta }$. Then $\hat{\theta }=\arg \min {\hat{e}}_{\text{orig}}(f_{\theta })$ is the MLE.
• Let $\varphi (f_{(\gamma ,\psi )})=\gamma$.

Result. The $(1-\delta )$ profile-likelihood interval is

$$\mathrm{PLI}(\delta )=\{\gamma :\log \mathcal{L}(\mathbf{Z};\hat{\theta })-\log \mathcal{L}(\mathbf{Z};{\hat{\theta }}_{\gamma })\le \frac{{\chi }_{1,1-\delta }}{2}\},{\hat{\theta }}_{\gamma }=\arg \underset{\{\theta :\varphi (f_{\theta })=\gamma \}}{\max }\mathcal{L}(\mathbf{Z};\theta )$$ $$=\{\gamma :\exists {\hat{\theta }}_{\gamma },\varphi (f_{{\hat{\theta }}_{\gamma }})=\gamma ,f_{{\hat{\theta }}_{\gamma }}\in \hat{\mathcal{R}}(\frac{{\chi }_{1,1-\delta }}{2n},f_{\hat{\theta }},\mathcal{F}\left)\right\}.(A.5)$$

Here ${\chi }_{1,1-\delta }$ is the $1-\delta$ quantile of ${\chi }_1^2$. If $\mathrm{PLI}$ is a contiguous interval, it equals $[{\hat{\varphi }}_{-}({\chi }_{1,1-\delta }/2n),{\hat{\varphi }}_{+}({\chi }_{1,1-\delta }/2n)]$, an empirical Rashomon set with $f_{\text{ref}}=\hat{\theta }$.

Advantages of Proposition 7 over profile likelihood (p. 17).

• It does not need asymptotics.
• It does not need the likelihood to be known up to a parametric form.
• It covers the whole near-optimal set $\mathcal{R}(ϵ)$, not a single, possibly misspecified $f^{\star }$. This is useful “when different near-optimal models accurately describe different aspects of the underlying data generating process, but none capture it completely.”

Disadvantage. The threshold ${ϵ}^{\prime }=ϵ+2B_{\text{ref}}\sqrt{\log (2{\delta }^{-1})/2n}$ shrinks at rate $n^{-1/2}$, while the profile-likelihood threshold ${\chi }_{1,1-\delta }/2n$ shrinks at rate $n^{-1}$. So the Rashomon CI is wider. §4.1 has the same disadvantage, so the authors use these results mainly to motivate point estimates.

Generality (p. 18). Rashomon sets “reframe a wide set of finite-sample inference problems as in-sample optimization problems.” The CIs need not be in closed form. Existing profile-likelihood machinery might be reused for finite-sample results. Approximate optimizers used for complex classes, such as MCMC for BART (Chipman et al. 2010), could be repurposed for approximate finite-sample inference.

Examples

Range of predicted risk for one profile

Take $\varphi (f)=f(x_{\text{new}})$ for a particular customer or market profile. Proposition 7 gives the range of predictions that any near-optimal model could make there, with finite-sample coverage. This is an honest “model-multiplicity” interval for a forecast or a scenario response, as opposed to the sampling interval of one fitted model.

Marketing illustration: ROI range across acceptable MMM fits

Let $\varphi$ be the implied marginal ROI of a channel in a regularized linear response model, a linear functional of $\beta$. The Rashomon-set CI is the min and max of that ROI over all fits within ${ϵ}^{\prime }$ of the reference loss. With a ridge-type class this is a QP1QC (see Model Class Reliance for Linear and RKHS Model Classes). The result says how much the ROI conclusion depends on which acceptable model was chosen. Compare posterior intervals in Bayesian Media Mix Modeling - Overview.

Connections

• Profile likelihood is the classical frequentist device for nuisance parameters. Proposition 7 is its finite-sample, misspecification-tolerant cousin.
• Conformal Prediction - Overview also gives finite-sample, distribution-free intervals, but for outcomes rather than for model characteristics.
• In sensitivity analysis for observational studies, ranges over “plausible” models play a similar role (Sensitivity Analysis in Observational Studies).

See Also

• Model Class Reliance - Outer and Inner Finite-Sample Bounds: the MR-specific version with the extra $\mathcal{Q}$ correction
• Rashomon Sets and Model Class Reliance - Definitions
• Fisher Rudin Dominici 2019 - Model Class Reliance - Overview
• Permute-and-Predict Extrapolation in Variable Importance — same profile-CI reading of MCR from an independent source