Manheim Garrabrant - Regressional Goodhart and Tails Coming Apart

Summary

Regressional Goodhart: when you select on a proxy $M=G+\text{noise}$, you select for the noise as well as for the goal. The states with the highest $M$ are, on average, those whose noise is most positive, so their true $G$ falls predictably short of $M$. Selection still helps ($E[G\mid M>c]$ rises with $c$), but the proxy overstates what you get. In the Gaussian case the gap grows linearly in $c$. The paper calls this “the most fundamental” Goodhart effect: it “cannot be avoided” with any inexact metric. Statisticians know it as regression to the mean and the winner’s curse. It is the mechanism behind optimistic in-sample model selection and inflated “best-model” effect sizes.

Overview

This is the first of the four categories (p.2). It needs only selection, with no intervention, no change of regime and no adversary. It also goes by “Tails come apart”, after a 2014 essay by Gregory Lewis (ref. [5]): even when two variables are strongly correlated, the most extreme case on one is rarely the most extreme case on the other.

Main Content

Regressional Goodhart (§1, p.2)

“When selecting for a proxy measure, you select not only for the true goal, but also for the difference between the proxy and the goal. This is also known as ‘Tails come apart.‘”

Simple model, eq. (1) (p.2)

$$M=G+\text{normal}(\mu ,{\sigma }^2)\text{\hspace{1em}(1)}$$

Here $G$ is the goal, $M$ the proxy, and the additive noise $\epsilon =M-G\sim N(\mu ,{\sigma }^2)$ is independent of $G$.

The paper’s reasoning: “Due to the noise, a point with a large M value will likely have a large G value, but also a large noise value. Thus, when M is large, you can expect G to be predictably smaller than M. Despite the lack of bias, for large values of c the values of G when M > c is expected to be higher than otherwise.”

Claim: unavoidability (§1, p.2)

“While this is the simplest Goodhart effect, it is also the most fundamental: it cannot be avoided. No matter what measure is chosen for optimization, an inexact metric necessarily leads to a divergence between the goal and the metric in the tail.”

The paper states this informally and gives no proof. The Gaussian derivation below is a vault addition that makes the claim exact.

Gaussian regressional shrinkage (vault derivation of eq. (1))

Let $G\sim N({\mu }_G,{\tau }^2)$ and $\epsilon \sim N(\mu ,{\sigma }^2)$ be independent, with $M=G+\epsilon$. Then

$$E[G\mid M=m]={\mu }_G+\underset{\lambda \in (0,1)}{\underbrace{\frac{{\tau }^2}{{\tau }^2+{\sigma }^2}}}(m-\mu -{\mu }_G),$$

so the expected shortfall $m-E[G\mid M=m]=(1-\lambda )(m-{\mu }_G)+\lambda \mu$ grows linearly in $m$.

For threshold selection,

$$E[G\mid M\ge c]={\mu }_G+\lambda (E[M\mid M\ge c]-\mu -{\mu }_G),E[M\mid M\ge c]={\mu }_M+s_M\frac{\varphi (z_c)}{1-\Phi (z_c)},$$

where ${\mu }_M={\mu }_G+\mu$, $s_M^2={\tau }^2+{\sigma }^2$ and $z_c=(c-{\mu }_M)/s_M$. Two consequences follow:

1. Selection still pays. $E[G\mid M\ge c]$ increases in $c$, which is the paper’s “the values of G when M > c is expected to be higher than otherwise”.
2. The proxy over-promises by a growing amount. The gap $E[M-G\mid M\ge c]$ grows without bound as $c\to \infty$.

The proxy also gets worse at ranking within the selected set, because conditioning on $M\ge c$ cuts the variance of $M$ (^sim-regressional has the numbers).

Proof sketch. $(G,M)$ is bivariate normal with $\mathrm{Cov}(G,M)={\tau }^2$. The conditional mean of a bivariate normal is linear with slope $\mathrm{Cov}/\mathrm{Var}(M)=\lambda$. Then take expectations over $M\mid M\ge c$, using the inverse Mills ratio for the truncated normal mean.

Interpretation

• Bias and Goodhart are different things. With $\mu =0$ the proxy is unbiased unconditionally, $E[M]=E[G]$, and yet it is biased conditional on being selected. The paper’s “Despite the lack of bias” makes exactly this point. Debiasing a metric does not remove regressional Goodhart.
• The cure is shrinkage, not a better point estimate. The gap is $(1-\lambda )$ times the distance from the mean, so the best predictor of $G$ given a selected $M$ shrinks $M$ towards the mean with the reliability ratio $\lambda$. This is the logic of James-Stein Estimator, Empirical Bayes Interpretation of Shrinkage and Partial Pooling as Multiple Comparisons Correction.
• Optimization power matters. Picking the maximum of $K$ candidates is equivalent to $c$ at the $1-1/K$ quantile. The expected overstatement grows roughly like $s_M(1-\lambda )\sqrt{2\ln K}$, so a search over thousands of variants overstates far more than a search over two.

Examples

Best-of- $K$ model selection (vault illustration)

An analyst evaluates $K$ candidate models by a hold-out score $M_k=G_k+{\epsilon }_k$, where $G_k$ is the model’s true out-of-sample skill and ${\epsilon }_k$ is the hold-out sampling error. With a short weekly hold-out (8-13 weeks), $\sigma$ is large relative to the true spread $\tau$ of skill across reasonable specifications, so $\lambda$ is small. The winning model’s hold-out score then mostly reflects its luck. This is why Model Selection and Overfitting recommends nested validation, or reporting the winner’s score on fresh data. It is also why Cross-Validation for Time Series Prediction warns against tuning on the same folds used for reporting.

Inflated effect sizes in significant findings

Filtering studies on $p<0.05$ is threshold selection on a noisy proxy for the true effect. Significant estimates exaggerate magnitude: Type M error (Type S and Type M Errors). The same mechanism drives Ioannidis 2005 - Why Most Published Findings Are False - Overview and the false positives in Simmons Nelson Simonsohn 2011 - False-Positive Psychology - Overview.

Connections

• Selection on outcome-correlated variables: The Selection Problem.
• Choosing the analysis that “works” is regressional Goodhart on the analyst’s own pipeline: Researcher Degrees of Freedom, Garden of Forking Paths.
• Stability selection limits it by asking for selection across subsamples: Meinshausen Buhlmann 2010 - Stability Selection - Overview.
• Rashomon sets: many near-equal models exist, so the argmax among them is largely noise-driven: Fisher Rudin Dominici 2019 - Model Class Reliance - Overview.

See Also

• Manheim Garrabrant - Extremal Goodhart, Model Insufficiency and Regime Change — the next category, where the relationship itself changes in the tail
• Manheim Garrabrant - Non-Causal Goodhart Effects in Causal Systems — ignored causes that worsen regressional Goodhart
• Manheim Garrabrant 2018 - Variants of Goodhart’s Law - Overview