Stein’s Paradox and Risk Dominance

Summary

Stein’s paradox (1955-1961): for $N\ge 3$, the James-Stein estimator everywhere dominates the MLE ${\hat{\mathbf{\mu }}}^{(MLE)}=\mathbf{z}$ in total squared-error risk — its risk is strictly smaller for every value of $\mathbf{\mu }$, making the century-old MLE inadmissible in dimension $\ge 3$. The result is frequentist, not Bayesian: it holds regardless of any prior belief. The proof rests on Stein’s unbiased estimate of risk (an integration-by-parts identity). The paradox is that estimating unrelated quantities jointly (Clemente’s average, Munson’s average) beats estimating each alone.

Overview

Charles Stein’s 1955 result shocked statisticians: maximum likelihood estimation for Gaussian models — in routine use for over a century — is inadmissible beyond one or two dimensions. The MLE has constant risk $R^{(MLE)}(\mathbf{\mu })=N$ for every $\mathbf{\mu }$, treating every point of the parameter space equally, which seems reasonable for general estimation. Yet for $N\ge 3$ it can be uniformly beaten.

The shock did not come from the empirical Bayes risk comparison (eqs. 1.24-1.25). Those are based on the zero-centric Bayesian model where shrinkage toward $0$ is naturally expected to help. The “rude surprise” came from a theorem with no prior at all: James and Stein (1961) proved domination for every fixed $\mathbf{\mu }$. (Stein 1956 first showed ${\hat{\mathbf{\mu }}}^{(0)}=\mathbf{z}$ could be improved; the explicit form 1.23 came with his student Willard James in 1961.)

Main Content

James-Stein dominance / inadmissibility of the MLE ^js-dominance

For $N\ge 3$, the James-Stein Estimator everywhere dominates the MLE in expected total squared error (Efron eq. 1.26):

$$E_{\mathbf{\mu }}\left\{\Vert {\hat{\mathbf{\mu }}}^{(JS)}-\mathbf{\mu }{\Vert }^2\right\}<E_{\mathbf{\mu }}\left\{\Vert {\hat{\mathbf{\mu }}}^{(MLE)}-\mathbf{\mu }{\Vert }^2\right\}\text{for every choice of }\mathbf{\mu }.$$

This is frequentist: it implies the superiority of ${\hat{\mathbf{\mu }}}^{(JS)}$ no matter what one’s prior beliefs about $\mathbf{\mu }$ may be. Since versions of the MLE underlie linear regression and ANOVA, its apparent uniform inferiority was a cause for alarm.

Stein's unbiased risk estimate (the proof identity) ^sure

The proof starts from the algebraic identity (eq. 1.27)

$$({\hat{\mu }}_i-{\mu }_i{)}^2=(z_i-{\hat{\mu }}_i{)}^2-(z_i-{\mu }_i{)}^2+2({\hat{\mu }}_i-{\mu }_i)(z_i-{\mu }_i).$$

Summing and taking expectations under $\mathbf{z}\sim {\mathcal{N}}_N(\mathbf{\mu },I)$, integration by parts on the normal density gives the key covariance identity

$${\mathrm{cov}}_{\mathbf{\mu }}({\hat{\mu }}_i,z_i)=E_{\mathbf{\mu }}\left\{\frac{\partial {\hat{\mu }}_i}{\partial z_i}\right\}$$

(valid whenever ${\hat{\mu }}_i$ is continuously differentiable in $\mathbf{z}$), which reduces the risk to (eq. 1.30)

$$E_{\mathbf{\mu }}\Vert \hat{\mathbf{\mu }}-\mathbf{\mu }{\Vert }^2=E_{\mathbf{\mu }}\left\{\Vert \mathbf{z}-\hat{\mathbf{\mu }}{\Vert }^2\right\}-N+2\sum _{i=1}^N{E}_{\mathbf{\mu }}\left\{\frac{\partial {\hat{\mu }}_i}{\partial z_i}\right\}.$$

Exact risk of James-Stein ^js-exact-risk

Applying the risk identity (1.30) to ${\hat{\mathbf{\mu }}}^{(JS)}$ (eq. 1.31) yields

$$E_{\mathbf{\mu }}\left\{\Vert {\hat{\mathbf{\mu }}}^{(JS)}-\mathbf{\mu }{\Vert }^2\right\}=N-E_{\mathbf{\mu }}\left\{\frac{(N-2{)}^2}{S}\right\},S=\sum z_i^2.$$

Since the subtracted term is strictly positive whenever $N>2$, the JS risk is strictly below the MLE risk $N$ for all $\mathbf{\mu }$ — proving the theorem.

Risk is total, not individual ^total-vs-individual

The theorem concerns total squared-error loss $\sum ({\hat{\mu }}_i-{\mu }_i{)}^2$, with no guarantee for individual cases. Most individual effects are improved, but genuinely unusual cases can be made worse — JS over-shrinks outliers toward the crowd. This is why standalone MLE methods remain popular (protecting individual inferences), and why compromises like the limited-translation estimator exist (see James-Stein Estimator).

Examples

Simulation showing the individual-vs-total tradeoff (Efron Table 1.2, $N=10$)

One thousand simulations of $\mathbf{z}\sim {\mathcal{N}}_{10}(\mathbf{\mu },I)$, with ${\mu }_{10}=4$ a far outlier:

$i$	${\mu }_i$	${\text{MSE}}_i^{(MLE)}$	${\text{MSE}}_i^{(JS)}$
1	$-.81$	.95	.61
4	$-.08$	.99	.58
9	1.89	1.00	.88
10	4.00	1.08	2.04!!
Total Sqerr		10.12	8.13

JS beats the MLE for the nine ordinary cases but has nearly twice the error for the outlier ${\mu }_{10}$. Overall the total mean squared error still favors ${\hat{\mathbf{\mu }}}^{(JS)}$ — as it must, by the theorem.

The values are consistent with ${\mu }_i\stackrel{ind}{\sim }\mathcal{N}(0,A)$; the total error $8.13$ matches the empirical Bayes risk prediction (Exercise 1.5).

The "paradox" in the baseball data

Clemente (top of Table 1.1) performs independently of Munson (near the bottom). Why should Clemente’s good early performance change our prediction for Munson? It does for ${\hat{\mathbf{\mu }}}^{(JS)}$ — mainly through the grand mean $\bar{z}$ in eq. 1.35 — but not for ${\hat{\mathbf{\mu }}}^{(MLE)}$. There is indirect evidence lurking among the players, supplementing each player’s own average. Formal Bayes supplies this via a prior; for empirical Bayes “the prior may exist only as a motivational device.” Note Clemente was genuinely an extraordinary hitter and should not have been shrunk so far toward his cohort — exactly the individual-case risk the theorem does not protect.

Connections

• James-Stein Estimator — the estimator whose risk is bounded here.
• Empirical Bayes - Overview — Stein’s branch of the EB initiative.
• Empirical Bayes Interpretation of Shrinkage — why dominance and shrinkage are two faces of estimating the prior.
• Partial Pooling as Multiple Comparisons Correction — pooling reduces total risk by the same mechanism.
• Asymptotics and Frequentist Connections — admissibility/inadmissibility as a frequentist decision-theory notion.

See Also

• Hierarchical Models
• Multiple Comparisons - Bayesian Perspective
• _Index