Riley 2020 - Shrinkage and Optimism Criteria for Overfitting

Summary

Steps 3-4 are the overfitting criteria and usually bind. Step 3: choose $n$ so that the expected uniform shrinkage factor is $S\ge 0.9$ ($\le 10\%$ shrinkage): $n=P/[(S-1)\ln (1-R_{cs}^2/S)]$ (Fig 3). For $P=20$, $R_{cs}^2=0.1$ this gives 1698, which is EPP 8.5 at prevalence 0.1 but EPP 25.5 at prevalence 0.3. Step 4: choose $n$ so that the apparent Nagelkerke $R^2$ is optimistic by at most $\delta =0.05$. Set $S=R_{cs}^2/(R_{cs}^2+\delta \max (R_{cs}^2))$ and reuse the Fig 3 formula (Fig 5). For $R_{cs}^2=0.2$, $\varphi =0.05$, $P=20$ this gives 1079. The authors aim to avoid needing shrinkage, because estimated shrinkage is least reliable exactly when it is most needed.

Overview

The paper sets this step up around one definition and one paradox. Small samples cause overfitting. Shrinkage (penalisation, regularisation) is the textbook remedy. But the amount of shrinkage is itself estimated from the same small sample. So the paper designs the sample so that little shrinkage is needed to begin with.

Main Content

Overfitting and shrinkage (p. 3)

Definition: Overfitting and shrinkage (p. 3)

Overfitting: a developed model’s predictions are more extreme than they ought to be for individuals in a new dataset from the same target population. For a binary outcome, predicted probabilities are too close to 1 for above-average-risk individuals and too close to 0 for below-average-risk individuals. It occurs notably when $P$ is large relative to $n$ (continuous outcomes) or to the number of events (binary, time-to-event). Consequence: apparent performance (in the development data) is optimistic, and actual performance in new data is worse.

Shrinkage (penalisation, regularisation): methods that reduce the variability of predictions, pulling extreme predictions back toward the overall average. A uniform shrinkage factor $S\in (0,1]$ multiplies all predictor effects (${\beta }_j\to S{\beta }_j$, intercept re-estimated). $1-S$ is the proportional shrinkage.

The shrinkage paradox (p. 3)

Shrinkage is not guaranteed to fix overfitting:

1. Shrinkage and penalty factors are estimated from the development data. In small samples they are imprecise and can mis-size the correction (van Houwelingen 2001, ref 30).
2. Estimated required shrinkage is negatively correlated with apparent performance. When apparent fit is excellent by chance, required shrinkage is estimated too low.

Verbatim (p. 3): “ironically, in those situations when overfitting is of most concern (and thus shrinkage is most urgently needed), the prediction model developer has insufficient assurance in selecting the proper amount of shrinkage to cancel the impact of overfitting.”

Implication: target a sample size and candidate-parameter count at which only a small amount of shrinkage ($\le 10\%$) is expected.

Step 3: the shrinkage criterion (Fig 3)

Criterion B3 / T2: sample size for expected shrinkage $S$ (Fig 3, p. 10; Riley et al. 2019 Part II)

For binary or time-to-event outcomes, the number of participants needed for an expected uniform shrinkage factor $S$ is

$$n=\frac{P}{(S-1)\ln \left(1-\frac{R_{cs}^2}{S}\right)}$$

where $P$ = number of candidate predictor parameters, $R_{cs}^2$ = anticipated (optimism-adjusted) Cox-Snell $R^2$ of the new model (definition), and the recommended target is $S\ge 0.9$.

Requirements: prespecify $P$; for binary/survival outcomes the anticipated outcome proportion or rate (to translate $n$ into events and to bound $R_{cs}^2$); a conservative anticipated $R_{cs}^2$ (Fig 4). Note $R_{cs}^2<S$ is needed for the log to exist.

Proof sketch (reconstruction; the PDF states only the formula). (i) The van Houwelingen-le Cessie heuristic shrinkage is $S=1-P/\text{LR}$, where LR is the model’s likelihood-ratio ${\chi }^2$. (ii) The Cox-Snell definition gives $R_{cs,\text{app}}^2=1-\exp (-\text{LR}/n)$, so $\text{LR}=-n\ln (1-R_{cs,\text{app}}^2)$. (iii) The optimism-adjusted $R_{cs}^2$ is approximately $S\cdot R_{cs,\text{app}}^2$, so $R_{cs,\text{app}}^2=R_{cs}^2/S$. (iv) Substitute: $S=1+P/[n\ln (1-R_{cs}^2/S)]$. Solving for $n$ gives the formula.

Fig 3 worked example (p. 10)

$P=20$, $R_{cs}^2=0.1$, $S=0.9$:

$$n=\frac{20}{(0.9-1)\ln (1-0.1/0.9)}=1698.$$

• Outcome proportion 0.1: $1698\times 0.1=169.8$ events, EPP $=169.8/20=8.5$.
• Outcome proportion 0.3: EPP $=25.5$.

The required EPP changes greatly because $R_{cs}^2=0.1$ is a much smaller fraction of max($R_{cs}^2$) at prevalence 0.3 (max 0.71) than at 0.1 (max 0.48). The same absolute $R_{cs}^2$ therefore means weaker relative signal at the higher prevalence.

Signal:noise intuition (p. 3). High anticipated $R_{cs}^2$ (near 1, possible for continuous outcomes) means true patterns are easy to detect, overfitting is less of a concern, and more parameters can be estimated. Low $R_{cs}^2$ (near 0) means patterns are hard to find, overfitting risk is larger, and fewer parameters can be estimated reliably. Required $n$ is very sensitive to $R_{cs}^2$, so choose it conservatively.

Continuous outcome (p. 3; supplementary S4)

For a continuous outcome, $R_{cs}^2$ is the ordinary $R^2\in [0,1]$. With up to $P=30$ predictor parameters and anticipated $R^2=0.7$, 206 participants give expected shrinkage of 10%, about 7 participants per parameter.

Caveat recorded on ingest: plugging these values into the Fig 3 binary formula gives about 199.5, not 206. The continuous-outcome criterion in S4 (Riley et al. 2019 Part I) is a separate expression that is not included in the PDF. Use pmsampsize, type(c) rather than the Fig 3 formula for continuous outcomes.

Step 4: the optimism criterion (Fig 5)

Optimism in apparent model fit (p. 4)

The apparent $R_{Nagelkerke}^2$ $(=R_{cs}^2/\max (R_{cs}^2))$ is computed on the development data. The optimism-adjusted value is an approximately unbiased estimate of fit in the target population. Optimism is apparent minus adjusted. Target a small optimism, e.g. $\delta \le 0.05$.

Criterion B4 / T3: sample size for optimism $\delta$ (Fig 5, p. 12; continuous C4 uses supplementary S6 instead)

First find the shrinkage factor corresponding to expected optimism $\delta$ in $R_{Nagelkerke}^2$:

$$S=\frac{R_{cs}^2}{R_{cs}^2+\delta \max (R_{cs}^2)}.$$

Then substitute $S$ into the Fig 3 formula $n=P/[(S-1)\ln (1-R_{cs}^2/S)]$.

Proof sketch (reconstruction). Apparent $R_{cs}^2\approx R_{cs}^2/S$, so optimism in the Nagelkerke scale is $(R_{cs}^2/S-R_{cs}^2)/\max (R_{cs}^2)=\delta$. Solving for $S$ gives the expression above.

Worked example: logistic model, anticipated $R_{cs}^2\ge 0.2$, $\varphi =0.05$ so $\max (R_{cs}^2)=0.33$, $\delta =0.05$:

$$S=\frac{0.2}{0.2+0.05\times 0.33}=0.924,n=\frac{20}{(0.924-1)\ln (1-0.2/0.924)}=1078.9,$$

so at least 1079 participants for $P=20$.

Numerical note: using unrounded $S=0.92379$ gives $n\approx 1075.6$. The paper’s 1078.9 comes from rounding $S$ to 0.924. Either way, rounding up to the next integer is the convention.

Which binds? Steps 3 and 4 both translate into a required $S$. B4’s $S$ is below 0.9 whenever $R_{cs}^2<9\delta \max (R_{cs}^2)$, i.e. $R_{Nagelkerke}^2<0.45$ for $\delta =0.05$, and then B3 (with $S=0.9$) binds. That covers most clinical models, where $R_{Nagelkerke}^2$ is typically 0.1-0.2 (Fig 4). B4 binds only for models with high relative fit.

Examples

Verified computation (python)

import math
def n_shrink(P, R2cs, S=0.9):
    return P / ((S - 1) * math.log(1 - R2cs / S))
def S_optimism(R2cs, maxR2, delta=0.05):
    return R2cs / (R2cs + delta * maxR2)
print(round(n_shrink(20, 0.1)))                  # 1698
S = S_optimism(0.2, 0.33); print(round(S, 3))   # 0.924
print(round(n_shrink(20, 0.2, 0.924), 1))       # 1078.9

Marketing-analytics reading

How many parameters can 104 weeks support? (illustrative, not from the paper)

Treat a weekly sales regression as a continuous-outcome development problem and, as a rough guide only, use the Fig 3 expression (see the continuous caveat above). For $P=36$ candidate parameters and $S=0.9$:

anticipated out-of-sample $R^2$	required $n$ (weeks)
0.3	about 888
0.5	about 444
0.7	about 239

Two years of weekly data (104 rows) fall well short unless the honest, out-of-sample $R^2$ is very high. In-sample $R^2$ from a model with trend and seasonality is not the right input; it is the optimistic quantity this criterion guards against. Autocorrelated rows carry less information than independent ones, so the true shortfall is larger. Practical responses mirror the paper’s: reduce candidate parameters (fix adstock forms from prior evidence), use informative priors as structured shrinkage (Bayesian Estimation and Priors for MMM), or pool across markets or segments.

Connections

• Uniform shrinkage $S$ is the frequentist counterpart of the posterior pull-to-the-mean in Empirical Bayes Interpretation of Shrinkage. Bayesian shrinkage priors (Global-Local Shrinkage Priors, Regularized Horseshoe (Finnish Horseshoe)) replace one global $S$ with structured, partly data-driven shrinkage. That is still subject to the paradox when the data are weak.
• Optimism (apparent minus adjusted fit) is the quantity that Overfitting and Information Criteria corrects with an effective-parameter penalty, and that bootstrap or Cross Validation Checking estimates empirically.
• The paradox (shrinkage least reliable when most needed) parallels the instability of selected subsets in Meinshausen Buhlmann 2010 - Stability Selection - Overview.

See Also

• Riley 2020 - Anticipating the Cox-Snell R2 and Its Maximum — the key input to both formulas
• Riley 2020 - Four-Step Procedure, pmsampsize and Worked Examples — B3 binds in Example 1 and T2 in Example 2
• Model Selection and Overfitting
• Model Selection Using Predictive Performance