The Horseshoe Prior

Summary

The horseshoe prior (Carvalho, Polson & Scott, 2009/2010) is the global-local shrinkage prior with half-Cauchy local scales ${\lambda }_j\sim {\mathrm{C}}^{+}(0,1)$. Its name comes from the $\text{Beta}(\frac{1}{2},\frac{1}{2})$, horseshoe-shaped density it induces on the shrinkage factor ${\kappa }_j$: mass piles up at $\kappa =0$ (signals untouched) and $\kappa =1$ (noise crushed). The heavy Cauchy tails give tail-robustness — strong signals are barely shrunk — which is usually an asset but becomes a liability under weak likelihoods.

Overview

The horseshoe is the most prominent member of the Global-Local Shrinkage Priors family and the starting point for the Regularized Horseshoe (Finnish Horseshoe). It has shown performance comparable to the gold-standard Spike-and-Slab Prior for Covariate Selection across many examples while remaining a simple continuous prior that samples in Stan. This note covers its definition, the horseshoe-shaped $\kappa$ density, tail-robustness, and the connection to spike-and-slab.

Main Content

Horseshoe prior

For the linear Gaussian regression $y_i={\mathbf{\beta }}^{\mathsf{T}}{\mathbf{x}}_i+{\epsilon }_i$, ${\epsilon }_i\sim \mathrm{N}(0,{\sigma }^2)$:

$${\beta }_j\mid {\lambda }_j,\tau \sim \mathrm{N}\left(0,{\tau }^2{\lambda }_j^2\right),{\lambda }_j\sim {\mathrm{C}}^{+}(0,1),j=1,\dots ,D.$$

The global scale $\tau$ pulls all coefficients toward zero; the thick half-Cauchy tails of ${\lambda }_j$ allow some coefficients to escape. Large $\tau$ gives diffuse priors with little shrinkage; $\tau \to 0$ shrinks all ${\beta }_j$ to zero. An intercept ${\beta }_0$, if present, gets a flat prior.

Horseshoe-shaped density on ${\kappa }_j$

With ${\lambda }_j\sim {\mathrm{C}}^{+}(0,1)$, the shrinkage factor ${\kappa }_j=(1+n{\sigma }^{-2}{\tau }^2{s}_j^2{\lambda }_j^2{)}^{-1}$ follows, at fixed $\tau ,\sigma$,

$$p({\kappa }_j\mid \tau ,\sigma )=\frac{1}{\pi }\frac{a_j}{(a_j^2-1){\kappa }_j+1}\frac{1}{\sqrt{{\kappa }_j}\sqrt{1-{\kappa }_j}},a_j=\tau {\sigma }^{-1}\sqrt{n}{s}_j.$$

For $a_j=1$ this is exactly $\text{Beta}(\frac{1}{2},\frac{1}{2})$, the symmetric U-shaped “horseshoe” with unbounded density at both ${\kappa }_j=0$ and ${\kappa }_j=1$. A priori we therefore expect both relevant variables (${\kappa }_j\approx 0$, no shrinkage) and irrelevant variables (${\kappa }_j\approx 1$, complete shrinkage).

Tail-robustness. Because the half-Cauchy local scale has Cauchy tails, the marginal prior on a large ${\beta }_j$ has heavy tails too. The shrinkage factor satisfies ${\kappa }_j\to 0$ for large signals, so ${\bar{\beta }}_j\to {\hat{\beta }}_j$ — strong coefficients are essentially not shrunk. Carvalho–Polson–Scott view this as a key strength: signals are not over-shrunk. The downside (motivating the Regularized Horseshoe (Finnish Horseshoe)) is that there is no way to regularize the largest coefficients: under a weak/flat likelihood (separable logistic regression), the heavy tails let $|{\beta }_j|\to \infty$ and posterior means can vanish, sharing the pathologies of the Cauchy prior.

Relation to spike-and-slab

Writing the spike-and-slab with $\epsilon =0$ as ${\beta }_j\mid {\lambda }_j,c\sim \mathrm{N}(0,c^2{\lambda }_j^2)$, ${\lambda }_j\sim \text{Ber}(\pi )$, the shrinkage factor takes only two values: ${\kappa }_j=(1+n{\sigma }^{-2}{s}_j^2{c}^2{)}^{-1}$ (slab, prob. $\pi$) and ${\kappa }_j=1$ (spike, prob. $1-\pi$). Letting $c\to \infty$ puts all mass at $\kappa =0$ and $\kappa =1$ — the discrete analogue of the horseshoe’s continuous U-shape. This is why the two priors perform similarly.

Examples

• $\text{Beta}(\frac{1}{2},\frac{1}{2})$ intuition: the density $\propto {\kappa }^{-1/2}(1-\kappa {)}^{-1/2}$ is the arcsine distribution; almost all prior mass is near the two endpoints, encoding “each coefficient is either clearly in or clearly out.”
• Default-$\tau$ pitfall: the popular default $\tau \sim {\mathrm{C}}^{+}(0,1)$ ignores that $a_j$ depends on $n$ and $\sigma$; it implies an implausibly large effective model size unless $\tau$ is strongly identified by data (see Choosing the Global Scale and Effective Nonzeros).

Connections

• A specific instance of Global-Local Shrinkage Priors (half-Cauchy local scales).
• Extended to fix tail problems by the Regularized Horseshoe (Finnish Horseshoe).
• Its $\tau$ is set via $m_{\text{eff}}$ in Choosing the Global Scale and Effective Nonzeros.
• The continuous analogue of Spike-and-Slab Prior for Covariate Selection.
• Prior on regression coefficients of Bayesian Linear Regression; the shared global $\tau$ mirrors Hierarchical Linear Models and the multiplicity control of Partial Pooling as Multiple Comparisons Correction.

See Also

• Global-Local Shrinkage Priors — the parent framework
• Regularized Horseshoe (Finnish Horseshoe) — the slab-regularized fix
• Choosing the Global Scale and Effective Nonzeros — setting $\tau$
• Spike-and-Slab Prior for Covariate Selection — the discrete gold standard
• Horseshoe and Regularized Horseshoe Priors — overview hub
• Partial Pooling as Multiple Comparisons Correction — shrinkage as multiplicity control