In the 1950s, the statistical orthodoxy was clear: to estimate the mean of a Gaussian, use the sample mean. It is the maximum-likelihood estimator, the minimum-variance unbiased estimator, and (in one dimension) admissible: no other estimator beats it for every value of the parameter.

In 1961, Charles Stein and Willard James proved that in three or more dimensions, the sample mean is not admissible. There is a different estimator that beats it everywhere: for every choice of the true mean $\theta \in \mathbb{R}^d$, simultaneously, in mean squared error. Even when the components are completely unrelated.

The estimator is famous and, when first encountered, jarring. It is also a precursor of ridge regression, empirical Bayes, and a great deal of modern machine learning.

The setup

You observe a single sample $X \sim \mathcal{N}(\theta, I_d)$ in $\mathbb{R}^d$ and want to estimate $\theta$. The sample mean (here, just $X$) has

Doesn’t depend on $\theta$. Just $d$, no matter what.

The James–Stein estimator

Define

The shrinkage factor $\bigl(1 - (d-2)/\|X\|^2\bigr)$ pulls $X$ toward the origin. (When $\|X\|^2$ is small, the factor can go negative; the positive-part version $\hat\theta_{\mathrm{JS}^+} = \max\{0, 1 - (d-2)/\|X\|^2\}\,X$ is a strict improvement, and is what people use in practice.)

Theorem (James and Stein, 1961). For every $\theta \in \mathbb{R}^d$ and every $d \ge 3$,

The sample mean is dominated uniformly. For $d \le 2$ the estimator degenerates and the dominance disappears. The phase transition at $d = 3$ is sharp.

The geometric reason

Why should shrinkage help?

If $X \sim \mathcal{N}(\theta, I_d)$, then $\|X\|^2$ has a non-central chi-squared distribution with non-centrality $\|\theta\|^2$, so

The sample $X$ is on average further from the origin than $\theta$ is, by an amount that grows with $d$. This is exactly the Gaussian shell phenomenon: noise in $d$ dimensions has typical magnitude $\sqrt{d}$, and adding it to $\theta$ pushes outward.

Shrinking $X$ toward the origin partially undoes this outward push. In high dimensions, the bias introduced by shrinkage is a much smaller error than the variance you save. The dimension threshold $d = 3$ becomes natural in this picture: in $d = 1, 2$, the noise radius $\sqrt{d}$ isn’t much, and shrinkage doesn’t pay off; in $d \ge 3$, it does, uniformly.

The exact value at $\theta = 0$

A clean computation. If $\theta = 0$ then $\|X\|^2 \sim \chi^2_d$, and for $d > 2$, $\mathbb{E}[1/\|X\|^2] = 1/(d - 2)$. Plugging in,

The James–Stein MSE at the origin is exactly $2$, for every $d \ge 3$.

The MLE has MSE $d$. So at $\theta = 0$, JS is a factor of $d/2$ better. For $d = 100$, MSE drops from $100$ to $2$. A factor of $50$.

Slide $d$ to see the full curve as $\|\theta\|$ varies:

The MLE curve sits flat at $d$. The James–Stein curve starts at $\approx 2$ for $\theta = 0$ and rises smoothly toward $d$ as $\|\theta\|$ grows: with the true mean far from the origin, the shrinkage benefit shrinks, but the MLE never wins at any $\theta$ in $d \ge 3$. At $d = 1$ the JS curve sits above the MLE (the formula inflates rather than shrinks); at $d = 2$ they coincide.

What is genuinely weird

Stein’s original example is the part that breaks people’s intuition.

Suppose you want to estimate three independent things: the speed of light, the price of tea in China, and the average height of redwoods. For each, you have one Gaussian-noisy measurement. The natural recipe: estimate each one independently, by its own measurement. Stein says: shrink all three toward zero (or toward any common point) and your total MSE drops.

The three estimation problems have nothing to do with each other. The three measurements are independent. And yet, coupling the three estimators by shrinking together reduces the joint risk. This is profoundly counterintuitive: the cost of independence in MSE terms is real.

The reason this confounds intuition is that unbiasedness was the wrong objective. MSE = bias$^2$ + variance, and the sample mean trades zero bias for high variance. In $d \ge 3$, accepting a bit of bias (shrinkage) buys a much larger reduction in variance, every time.

What this led to

Stein’s paradox is the seed of a huge amount of modern statistics and ML:

• Empirical Bayes. Learn the prior from data; shrink toward what the data says is the typical $\theta$.
• Ridge regression. Penalize the squared norm of the regression coefficients; the regularization parameter is a shrinkage knob.
• Hierarchical Bayesian models. Multi-level shrinkage across groups.
• L2 regularization in deep learning. Weight decay is, at heart, a shrinkage estimator.

All of these are descendants of: the sample mean is wrong in $\ge 3$ dimensions; do something biased instead.

References

• William James, Charles Stein. Estimation with Quadratic Loss. Berkeley Symposium on Mathematical Statistics and Probability, 1961.
• Bradley Efron, Carl Morris. Stein’s Paradox in Statistics. Scientific American, 1977. A beautifully written exposition.
• Roman Vershynin. High-Dimensional Probability. Cambridge University Press, 2018. Chapter 8 has the geometric setup and a clean proof of the key inequality.