Neyman Orthogonality

Summary

A score $\psi (W;\theta ,\eta )$ identifying ${\theta }_0$ via ${\mathbb{E}}_P[\psi (W;{\theta }_0,{\eta }_0)]=0$ is Neyman orthogonal if its pathwise (Gateaux) derivative with respect to the nuisance $\eta$ vanishes at the truth: ${\partial }_{\eta }{\mathbb{E}}_P\psi (W;{\theta }_0,{\eta }_0)[\eta -{\eta }_0]=0$. The moment condition is then locally insensitive to nuisance error, so noisy, regularized ML estimates ${\hat{\eta }}_0$ can be plugged in without first-order damage: the remaining bias is second order (a product or square of nuisance errors), and $\sqrt{N}$-inference on ${\theta }_0$ needs only $\Vert {\hat{\eta }}_0-{\eta }_0\Vert =o(N^{-1/4})$. The idea is due to Neyman (1959, 1979; the $C(\alpha )$ test); Chernozhukov et al. (2018) make it, together with cross-fitting, one of the two generic keys of DML.

Overview

Think of the population moment $M(\theta ,\eta )={\mathbb{E}}_P[\psi (W;\theta ,\eta )]$ as a surface over $(\theta ,\eta )$. The estimator ${\tilde{\theta }}_0$ solves $\hat{M}(\theta ,{\hat{\eta }}_0)=0$. A Taylor expansion in the nuisance direction gives

$$M({\theta }_0,{\hat{\eta }}_0)=\underset{=0}{\underbrace{M({\theta }_0,{\eta }_0)}}+\underset{\text{first order}}{\underbrace{{\partial }_{\eta }M({\theta }_0,{\eta }_0)[{\hat{\eta }}_0-{\eta }_0]}}+\underset{\text{second order}}{\underbrace{\frac{1}{2}{\partial }_{\eta }^{2}M[{\hat{\eta }}_0-{\eta }_0{]}^2}}+\dots$$

After scaling by $\sqrt{N}$, the first-order term is $\sqrt{N}\times O(N^{-\phi })\to \infty$ for any ML learner with $\phi <1/2$ — that is term $b$ in Regularization Bias and the Partially Linear Model. Orthogonality sets the first-order term to zero by construction, leaving $\sqrt{N}\times O(N^{-2\phi })$, which vanishes for $\phi >1/4$.

Main Content

Pathwise derivative and Neyman orthogonality (Def. 2.1) ^def-neyman-orth

Let $T$ be a convex nuisance space and ${\mathcal{T}}_N\subset T$ a nuisance realization set containing ${\hat{\eta }}_0$ with high probability (a shrinking neighborhood of ${\eta }_0$). Define

$$D_r[\eta -{\eta }_0]:={\partial }_r{\mathbb{E}}_P[\psi (W;{\theta }_0,{\eta }_0+r(\eta -{\eta }_0))],r\in [0,1),$$

and write ${\partial }_{\eta }{\mathbb{E}}_P\psi (W;{\theta }_0,{\eta }_0)[\eta -{\eta }_0]:=D_0[\eta -{\eta }_0]$. The score $\psi$ obeys the orthogonality condition at $({\theta }_0,{\eta }_0)$ with respect to ${\mathcal{T}}_N$ if ${\mathbb{E}}_P\psi (W;{\theta }_0,{\eta }_0)=0$ and

$${\partial }_{\eta }{\mathbb{E}}_P\psi (W;{\theta }_0,{\eta }_0)[\eta -{\eta }_0]=0\text{for all }\eta \in {\mathcal{T}}_N.$$

Near-orthogonality (Def. 2.2) ^def-near-orth

$\psi$ is ${\lambda }_N$ near-orthogonal if $\Vert {\partial }_{\eta }{\mathbb{E}}_P\psi (W;{\theta }_0,{\eta }_0)[\eta -{\eta }_0]\Vert \le {\lambda }_N$ for all $\eta \in {\mathcal{T}}_N$, with ${\lambda }_N=o(N^{-1/2})$. This covers scores whose orthogonalizing parameter is itself regularized (e.g. a lasso-estimated ${\mu }_0$ satisfying $\Vert J_{\theta \beta }-\mu J_{\beta \beta }{\Vert }_q\le r_N$).

Check on the PLR model. For the naive score $\phi =(Y-\theta D-g(X))D$, a perturbation ${\delta }_g=g-g_0$ gives ${\partial }_g\mathbb{E}\phi [{\delta }_g]=-\mathbb{E}[{\delta }_g(X)D]=-\mathbb{E}[{\delta }_g(X)m_0(X)]\ne 0$. For the DML score $\psi =(Y-D\theta -g(X))(D-m(X))$ with $\eta =(g,m)$,

$${\partial }_{\eta }\mathbb{E}\psi [{\delta }_g,{\delta }_m]=-\mathbb{E}[{\delta }_g(X)V]-\mathbb{E}[U{\delta }_m(X)]=0,$$

because $\mathbb{E}[V\mid X]=0$ and $\mathbb{E}[U\mid X,D]=0$. The second derivative is $2\mathbb{E}[{\delta }_g(X){\delta }_m(X)]$ — the product structure that produces term $b^{\ast }$.

Constructing orthogonal scores (§2.2)

Neyman's construction for (quasi-)likelihood problems (Lemma 2.1) ^thm-neyman-construction

Let $({\theta }_0,{\beta }_0)$ maximize ${\mathbb{E}}_P[\ell (W;\theta ,\beta )]$ with finite-dimensional nuisance $\beta$, and let $J={\partial }_{({\theta }^{\prime },{\beta }^{\prime })}{\mathbb{E}}_P[{\partial }_{({\theta }^{\prime },{\beta }^{\prime }{)}^{\prime }}\ell ]$ be partitioned into $J_{\theta \theta },J_{\theta \beta },J_{\beta \theta },J_{\beta \beta }$. The score

$$\psi (W;\theta ,\eta )={\partial }_{\theta }\ell (W;\theta ,\beta )-\mu {\partial }_{\beta }\ell (W;\theta ,\beta ),\eta =({\beta }^{\prime },\mathrm{vec}(\mu {)}^{\prime }{)}^{\prime },$$

with ${\mu }_0=J_{\theta \beta }{J}_{\beta \beta }^{-1}$ solving $J_{\theta \beta }-\mu J_{\beta \beta }=0$, is Neyman orthogonal at $({\theta }_0,{\eta }_0)$ — with respect to both $\beta$ and $\mu$. When $\ell$ is the true log-likelihood, $\psi$ is also the efficient score (Remark 2.2).

In words: project the $\theta$-score off the span of the nuisance scores. Applied to the high-dimensional linear model $Y=D{\theta }_0+X^{\prime }{\beta }_0+U$, $D=X^{\prime }{\gamma }_0+V$ (Example 2.1), one gets $J_{\theta \beta }=-\mathbb{E}[D{X}^{\prime }]$, $J_{\beta \beta }=-\mathbb{E}[X{X}^{\prime }]$, ${\mu }_0={\gamma }_0^{\prime }$ and

$$\psi (W;\theta ,\eta )=(Y-D\theta -X^{\prime }\beta )(D-\mu X),$$

the “double lasso”/partialling-out score. The paper gives parallel constructions for GMM problems (Lemma 2.3: $\psi =\mu m(W;\theta ,\beta )$ with $\mu$ chosen to annihilate the nuisance Jacobian $G_{\beta }$), for M-estimation with infinite-dimensional nuisance via concentrating-out (Lemma 2.5: profile ${\beta }_{\theta }=\arg {\max }_{\beta }\mathbb{E}\ell (W;\theta ,\beta )$ and differentiate $\ell (W;\theta ,{\beta }_{\theta })$ totally in $\theta$), and for conditional moment restrictions (Lemma 2.6).

Orthogonal score = original score + influence-function adjustment (§2.2.5) ^thm-if-adjustment

If a first-step estimator ${\hat{\beta }}_0$ admits the expansion $\int \phi (w;{\theta }_0,{\hat{\beta }}_0)dP(w)=\frac{1}{n}{\sum }_i\varphi (W_i;{\theta }_0,{\eta }_0)+o_P(n^{-1/2})$ (Newey 1994), then

$$\psi (W;\theta ,\eta )=\phi (W;\theta ,\beta )+\varphi (W;\theta ,\eta )$$

is Neyman orthogonal, and $\psi (W;{\theta }_0,{\eta }_0)$ is the influence function of the limit of $n^{-1}{\sum }_i\phi (W_i;{\theta }_0,{\hat{\beta }}_0)$. For PLR: the derivative of $\phi =D(Y-D\theta -g(X))$ in $g$ has conditional mean $-m_0(X)$; multiplying by the nonparametric residual gives $\varphi =-m_0(X)\{Y-D\theta -g(X)\}$ and hence $\psi =\{D-m_0(X)\}\{Y-D\theta -g(X)\}$.

This is why the AIPW/doubly-robust score is orthogonal (DML Estimators for ATE and the Interactive Model): it is the plug-in $g(1,X)-g(0,X)-\theta$ plus the influence-function correction for estimating $g$.

What orthogonality buys (§3.2)

Rate requirements under orthogonality ^thm-rates

For linear scores $\psi ={\psi }^a(W;\eta )\theta +{\psi }^b(W;\eta )$, Assumptions 3.1–3.2 require (near-)orthogonality with ${\lambda }_N\le {\delta }_N{N}^{-1/2}$, identification ($J_0={\mathbb{E}}_P[{\psi }^a(W;{\eta }_0)]$ with singular values in $[c_0,c_1]$), and statistical rates

$$r_N,r_N^{\prime }\le {\delta }_N,{\lambda }_N^{\prime }:=\underset{r\in (0,1),\eta \in {\mathcal{T}}_N}{\sup }\Vert {\partial }_r^2{\mathbb{E}}_P[\psi (W;{\theta }_0,{\eta }_0+r(\eta -{\eta }_0))]\Vert \le {\delta }_N/\sqrt{N}.$$

In smooth problems $r_N,r_N^{\prime }\lesssim {\epsilon }_N$ and ${\lambda }_N^{\prime }\lesssim {\epsilon }_N^2$ where $\Vert {\hat{\eta }}_0-{\eta }_0{\Vert }_{P,2}\lesssim {\epsilon }_N$, so the crude requirement is ${\epsilon }_N=o(N^{-1/4})$ (eq. 3.8). When the second derivative vanishes identically, ${\lambda }_N^{\prime }=0$ — the optimal-instrument problem, PLR with known $m_0$, and treatment-effect problems with known propensity score (RCTs) — the requirement collapses to mere consistency, ${\epsilon }_N=o(1)$.

The $o(N^{-1/4})$ rate is attainable: ${\ell }_1$-penalized methods under approximate sparsity, $L_2$-boosting in sparse linear models, certain trees/forests (Wager & Walther 2016), and classes of neural nets (Chen & White 1999) — the paper’s list on p. 26. Given these conditions, Theorem 3.1 delivers $\sqrt{N}{\sigma }^{-1}({\tilde{\theta }}_0-{\theta }_0)=\frac{1}{\sqrt{N}}{\sum }_i\bar{\psi }(W_i)+O_P({\rho }_N)⇝\mathcal{N}(0,I_d)$ uniformly over $P\in {\mathcal{P}}_N$, with influence function $\bar{\psi }=-{\sigma }^{-1}{J}_0^{-1}\psi (\cdot ;{\theta }_0,{\eta }_0)$ and ${\sigma }^2=J_0^{-1}{\mathbb{E}}_P[\psi {\psi }^{\prime }](J_0^{-1}{)}^{\prime }$. Uniformity — robustness to perturbations of $P$ along sequences — “can be shown to fail for methods not based on orthogonal scores.”

Two caveats the paper states explicitly: orthogonality is “a joint property of the score, the true parameter value ${\eta }_0$, the parameter set $T$, and the distribution of $W$” rather than of a model for $\theta$; and orthogonal scores need not be efficient (Remark 2.3: a regularized ${\mu }_0$ gives up efficiency), though if $\psi$ is the efficient score the DML estimator attains the semiparametric bound (Corollary 3.2).

Examples

Numerical intuition. Suppose $N=10,000$ and both nuisances are learned at rate $N^{-0.35}\approx 0.040$. Non-orthogonal score: bias in $\sqrt{N}(\hat{\theta }-{\theta }_0)$ is of order $\sqrt{N}\cdot N^{-0.35}=N^{0.15}\approx 4.0$ — four standard-error-scale units of bias, so a nominal 95% CI has near-zero coverage. Orthogonal score: $\sqrt{N}\cdot N^{-0.70}=N^{-0.20}\approx 0.16$ — negligible relative to the $O(1)$ Gaussian term.

Catalogue of orthogonal scores in the DML paper.

Model	Score $\psi (W;\theta ,\eta )$	Nuisance $\eta$
PLR (4.3)	$\{Y-D\theta -g(X)\}(D-m(X))$	$(g,m)$
PLR, Robinson (4.4)	$\{Y-\ell (X)-\theta (D-m(X))\}(D-m(X))$	$(\ell ,m)$
Partially linear IV (4.7)	$(Y-D\theta -g(X))(Z-m(X))$	$(g,m)$, $m_0=\mathbb{E}[Z\mid X]$
PLIV, Robinson (4.8)	$(Y-\ell (X)-\theta (D-r(X)))(Z-m(X))$	$(\ell ,m,r)$
ATE (5.3)	AIPW score	$(g,m)$
ATTE (5.4), LATE (§5.2)	see DML Estimators for ATE and the Interactive Model	$(g,m,p)$; $(\mu ,m,p)$

Connections

• Regularization Bias and the Partially Linear Model — the motivating failure that orthogonality repairs.
• Cross-Fitting and Sample Splitting — orthogonality handles bias from regularization; splitting handles bias from overfitting. Both are needed.
• DML Estimators for ATE and the Interactive Model — doubly-robust scores as the canonical orthogonal scores for treatment effects.
• Frequentist Causal Estimation — “double robustness” (consistency if either nuisance model is right) and “Neyman orthogonality” (first-order insensitivity to both) are two faces of the same product-form remainder.
• Doubly-Robust Estimands for ATT(g,t) — Callaway & Sant’Anna’s DR estimand is an orthogonal score for the group-time ATT.
• Generalized Random Forests - Local Moment Equations — local centering orthogonalizes the forest’s local moment; R-Learner and Orthogonal CATE Estimation turns the orthogonal PLR score into a loss.
• GMM Estimation and Instruments for Price Endogeneity — GMM moments with first-stage nuisance can be orthogonalized by the Lemma 2.3 construction.

See Also

• Instrumental Variables — optimal-instrument estimation is the prototypical case with ${\lambda }_N^{\prime }=0$.
• Plausible GMM - Overview — another relaxation of exact moment conditions.
• The Experimental Ideal — with a known randomization probability, orthogonal scores need only consistent outcome models.