James-Stein Estimator

Summary

The James-Stein (JS) estimator shrinks the vector of observed values $\mathbf{z}$ toward a center by an empirically estimated factor: ${\hat{\mu }}^{(JS)}=\left(1-\frac{N-2}{S}\right)\mathbf{z}$ with $S=\Vert \mathbf{z}{\Vert }^2$. It is exactly the Bayes estimator ${\hat{\mu }}^{(Bayes)}=(1-\frac{1}{A+1})\mathbf{z}$ with the unknown shrinkage term $1/(A+1)$ replaced by its unbiased estimate $(N-2)/S$ formed from the marginal distribution. The general “shrink toward the grand mean” form pulls each $z_i$ toward $\bar{z}$. This is empirical Bayes in action: the prior is estimated from the $N$ parallel cases.

Overview

Consider $N$ parallel normal estimation problems (Efron eq. 1.7):

$${\mu }_i\sim \mathcal{N}(0,A)\text{and}{z}_i\mid {\mu }_i\sim \mathcal{N}({\mu }_i,1),i=1,\dots ,N,$$

with total squared-error loss $L(\mathbf{\mu },\hat{\mathbf{\mu }})=\Vert \hat{\mathbf{\mu }}-\mathbf{\mu }{\Vert }^2={\sum }_{i=1}^N({\hat{\mu }}_i-{\mu }_i{)}^2$ and risk $R(\mathbf{\mu })=E_{\mathbf{\mu }}\{L\}$.

The obvious estimator — used implicitly in every regression and ANOVA — is the MLE ${\hat{\mathbf{\mu }}}^{(MLE)}=\mathbf{z}$, with constant risk $R^{(MLE)}(\mathbf{\mu })=N$ for every $\mathbf{\mu }$. If the prior were known, Bayes rule (eq. 1.10) gives posterior $\mathbf{\mu }\mid \mathbf{z}\sim {\mathcal{N}}_N(B\mathbf{z},BI)$ with $B=A/(A+1)$, and the Bayes estimator (eq. 1.16) is

$${\hat{\mathbf{\mu }}}^{(Bayes)}=B\mathbf{z}=\left(1-\frac{1}{A+1}\right)\mathbf{z}.$$

With $A=1$ this shrinks the MLE halfway toward $0$. But if $A$ is unknown we cannot use it — this is precisely where empirical Bayes enters (see Empirical Bayes - Overview).

Main Content

Estimating the shrinkage factor from the marginal ^marginal-estimate

Integrating the prior out, the marginal distribution of $\mathbf{z}$ (eq. 1.20) is

$$\mathbf{z}\sim {\mathcal{N}}_N(0,(A+1)I).$$

Hence $S=\Vert \mathbf{z}{\Vert }^2\sim (A+1){\chi }_N^2$, which yields the key unbiased estimate of the shrinkage term:

$$E\left\{\frac{N-2}{S}\right\}=\frac{1}{A+1}.$$

The unknown Bayes quantity $1/(A+1)$ is thus estimable directly from the pooled data — see Robbins Formula and Poisson Empirical Bayes for the analogous nonparametric move.

James-Stein estimator (shrink toward 0) ^js-estimator

Substituting the unbiased estimate $(N-2)/S$ for $1/(A+1)$ in the Bayes rule gives the James-Stein estimator (Efron eq. 1.23):

$${\hat{\mathbf{\mu }}}^{(JS)}=\left(1-\frac{N-2}{S}\right)\mathbf{z},S=\Vert \mathbf{z}{\Vert }^2=\sum _{i=1}^N{z}_i^2.$$

The name “empirical Bayes” is apt: the Bayes estimator (1.16) is itself empirically estimated from the data. This is only possible because $N$ similar problems $z_i\sim \mathcal{N}({\mu }_i,1)$ are under simultaneous consideration.

James-Stein estimator (shrink toward the grand mean) ^js-grand-mean

We need not shrink toward $0$. Starting from the more general prior ${\mu }_i\stackrel{ind}{\sim }\mathcal{N}(M,A)$, $z_i\mid {\mu }_i\stackrel{ind}{\sim }\mathcal{N}({\mu }_i,{\sigma }_0^2)$ (eq. 1.32), the Bayes rule ${\hat{\mu }}_i^{(Bayes)}=M+B(z_i-M)$ with $B=A/(A+{\sigma }_0^2)$ has empirical Bayes form (eq. 1.35):

$${\hat{\mu }}_i^{(JS)}=\bar{z}+\left(1-\frac{(N-3){\sigma }_0^2}{S}\right)(z_i-\bar{z}),$$

with $\bar{z}=\sum z_i/N$ and $S=\sum (z_i-\bar{z}{)}^2$. Each $z_i$ is pulled toward the grand mean $\bar{z}$ by a data-estimated factor; the risk-dominance theorem holds now for $N\ge 4$ (one degree of freedom is spent estimating $\bar{z}$).

Overall Bayes risk and the modest EB penalty ^js-risk

The overall Bayes risk of ${\hat{\mathbf{\mu }}}^{(Bayes)}$ is $R_A^{(Bayes)}=NA/(A+1)$, versus $R_A^{(MLE)}=N$. The James-Stein estimator pays only a small penalty for not knowing $A$ (eqs. 1.24-1.25):

$$R_A^{(JS)}=N\frac{A}{A+1}+\frac{2}{A+1},\frac{R_A^{(JS)}}{R_A^{(Bayes)}}=1+\frac{2}{N\cdot A}.$$

For $N=10$, $A=1$, $R_A^{(JS)}$ is only 20% greater than the true Bayes risk — almost all the Bayesian savings are recovered without knowing the prior.

Limited-translation compromise ^limited-translation

To protect genuinely unusual cases from being over-shrunk, Efron’s limited-translation estimator ${\hat{\mu }}_i^{(D)}$ (eq. 1.37) follows the JS estimate but never deviates more than $D{\sigma }_0$ from $z_i$:

$${\hat{\mu }}_i^{(D)}=\{\begin{array}{ll}\max \left({\hat{\mu }}_i^{(JS)},{\hat{\mu }}_i^{(MLE)}-D{\sigma }_0\right)& \text{for }{z}_i>\bar{z},\\ \min \left({\hat{\mu }}_i^{(JS)},{\hat{\mu }}_i^{(MLE)}+D{\sigma }_0\right)& \text{for }{z}_i\le \bar{z}.\end{array}$$

Taking $D=1$ in the baseball data costs only ~10% of the overall JS advantage while sharply limiting damage to outliers like Clemente.

Examples

Baseball batting averages (Efron Table 1.1, $N=18$)

Early-1970-season batting averages $z_i={\hat{\mu }}_i^{(MLE)}$ (hits/45 at-bats) predict true season averages ${\mu }_i$. With grand average $\bar{z}=0.265$ and ${\sigma }_0^2=\bar{z}(1-\bar{z})/45$ (binomial variance), the JS estimates (1.35) shrink each player toward $0.265$:

Player	hits/AB	${\hat{\mu }}^{(MLE)}=z_i$	true ${\mu }_i$	${\hat{\mu }}^{(JS)}$
Clemente	18/45	.400	.346	.294
F. Robinson	17/45	.378	.298	.289
Munson	8/45	.178	.316	.247
Alvis	7/45	.156	.200	.242
Grand Average		.265	.265	.265

The ratio of total prediction errors is

$$\frac{{\sum }_1^{18}({\hat{\mu }}_i^{(JS)}-{\mu }_i{)}^2}{{\sum }_1^{18}({\hat{\mu }}_i^{(MLE)}-{\mu }_i{)}^2}=0.28,$$

a roughly 3.5x accuracy gain for the empirical Bayes estimates. (The $z_i$ are binomial here, violating the exact normal theorem conditions, but the JS effect is quite insensitive to the model.)

Regression-based shrinkage (kidney data)

Combining covariate information with shrinkage (eq. 1.39), JS shrinks toward a fitted regression line ${\hat{\mu }}_i^{(reg)}={\hat{M}}_0+{\hat{M}}_1\cdot {\text{age}}_i$ rather than toward $\bar{z}$:

$${\hat{\mu }}_i^{(JS)}={\hat{\mu }}_i^{(reg)}+\left(1-\frac{(N-4){\sigma }_0^2}{S}\right)(z_i-{\hat{\mu }}_i^{(reg)}),S=\sum (z_i-{\hat{\mu }}_i^{(reg)}{)}^2.$$

See Empirical Bayes Interpretation of Shrinkage.

Connections

• Empirical Bayes - Overview — places JS in the broader EB program.
• Robbins Formula and Poisson Empirical Bayes — the nonparametric sibling; both estimate the prior from the marginal.
• Stein’s Paradox and Risk Dominance — the proof that JS dominates the MLE for $N\ge 3$.
• Empirical Bayes Interpretation of Shrinkage — why "$(N-2)/S$" is an estimated prior; link to hierarchical Bayes.
• Partial Pooling as Multiple Comparisons Correction — shrinkage toward $\bar{z}$ as partial pooling.
• Hierarchical Models — the fully Bayesian generalization.

See Also

• Multiple Comparisons - Bayesian Perspective
• Overfitting and Information Criteria
• _Index