Regression and the Conditional Expectation Function

Regression and the CEF

Summary

The population regression function is the best linear approximation to the conditional expectation function (CEF). This relationship holds regardless of whether the CEF is actually linear, giving regression a robust interpretation even when functional form assumptions fail.

The CEF

The conditional expectation function $E[Y_i|X_i]$ is:

• The best predictor of $Y_i$ given $X_i$ (minimizes mean squared error)
• A function that decomposes any random variable: $Y_i=E[Y_i|X_i]+{\epsilon }_i$ where ${\epsilon }_i$ is uncorrelated with any function of $X_i$

Three Justifications for Regression

1. Linear CEF Theorem

If the CEF is linear, then the population regression function equals the CEF.

2. Best Linear Predictor Theorem

The regression function $X_i^{\prime }\beta$ is the best linear predictor of $Y_i$ given $X_i$ in the MMSE sense.

3. Regression-CEF Theorem (the key one)

Even when the CEF is nonlinear, regression provides the MMSE linear approximation to it:

$$\beta =\arg \underset{b}{\min }E\{(E[Y_i|X_i]-X_i^{\prime }b{)}^2\}$$

Regression Anatomy

The coefficient on regressor $k$ in a multivariate regression:

$${\beta }_k=\frac{Cov(Y_i,{\tilde{x}}_{ki})}{V({\tilde{x}}_{ki})}$$

where ${\tilde{x}}_{ki}$ is the residual from regressing $x_{ki}$ on all other covariates. This is the Frisch-Waugh result: each multivariate coefficient equals the bivariate coefficient after “partialling out” other variables.

Robust Standard Errors

The heteroskedasticity-consistent (robust) covariance matrix:

$$E[X_i{X}_i^{\prime }{]}^{-1}E[X_i{X}_i^{\prime }{e}_i^2]E[X_i{X}_i^{\prime }{]}^{-1}$$

Always Use Robust Standard Errors

Since regression approximates a possibly nonlinear CEF, heteroskedasticity is the natural state of affairs. Robust and conventional standard errors that differ by more than 30% may indicate a problem.

Saturated Models

A saturated model has a separate parameter for every possible covariate combination — it fits the CEF perfectly and is inherently linear. Example: with two dummies $x_1,x_2$, the saturated model includes both main effects and their interaction.

See Also

• Conditional Independence Assumption
• Omitted Variables Bias
• Mostly Harmless Econometrics - Overview
• Bayesian Linear Regression — the Bayesian perspective on regression, with priors providing natural regularization
• Asymptotics and Frequentist Connections — Bayesian posteriors converge to OLS estimates under flat priors
• Data Collection Models — Bayesian ignorability as the prerequisite for regression’s causal interpretation
• Local Average Treatment Effects — IV/LATE as the estimand when regression cannot recover the full ATE due to non-compliance
• Directed Acyclic Graphs — DAGs identify the adjustment set that gives regression a causal interpretation
• Spurious Association and Confounds — Statistical Rethinking’s treatment of multivariate regression and confounds, the Bayesian parallel to MHE’s CEF analysis