A-learning and Robustness

Summary

A-learning (“A” for advantage; Blatt, Murphy & Zhu 2004) exploits the fact that deducing the optimal regime needs only the treatment contrast $C_k({\bar{s}}_k,{\bar{a}}_{k-1})=Q_k(\cdot ,1)-Q_k(\cdot ,0)$, not the full Q-function. It models $C_k$ parametrically along with the propensity ${\pi }_k$ (and a nuisance function $h_k=Q_k(\cdot ,0)$), and estimates them by g-estimation (Robins 2004). This yields double robustness: consistent estimation of the optimal regime if the contrast is correct and either the propensity or the nuisance $h_k$ is correct — a weaker requirement than Q-learning’s “all Q-functions correct.” The price is some efficiency loss when everything is correctly specified. The paper’s simulations map this bias–variance / robustness trade-off.

Overview

A-learning is the robust alternative to Q-learning within the dynamic-programming framework. The insight: the arg-max defining the optimal rule depends only on the contrast between treatments, so the parts of the outcome regression that are common across treatments need not be modeled correctly. This note gives the contrast/advantage functions, the g-estimation equations, the double-robustness property, and the simulation findings comparing the two methods.

Main Content

Contrast and advantage functions

Contrast / advantage (regret) function (§5.2)

For binary options ${\Psi }_k=\{0,1\}$, the contrast function is

$$C_k({\bar{s}}_k,{\bar{a}}_{k-1})=Q_k({\bar{s}}_k,{\bar{a}}_{k-1},1)-Q_k({\bar{s}}_k,{\bar{a}}_{k-1},0).$$

Any Q-function decomposes as $Q_k({\bar{s}}_k,{\bar{a}}_k)=h_k({\bar{s}}_k,{\bar{a}}_{k-1})+a_k{C}_k({\bar{s}}_k,{\bar{a}}_{k-1})$ with $h_k=Q_k(\cdot ,0)$, so the optimum is $a_k=I\{C_k>0\}$ — it depends only on $C_k$. The related advantage / regret is $C_k[I\{C_k>0\}-a_k]$ (Murphy 2003): the loss from not taking the optimal treatment. $C_k$ is also the optimal blip-to-zero function (Robins 2004; Moodie et al. 2007).

G-estimation

Contrast-based A-learning estimating equations (§5.2, Eqs. 30-31)

Posit models $C_k({\bar{s}}_k,{\bar{a}}_{k-1};{\psi }_k)$ for the contrast, $h_k({\bar{s}}_k,{\bar{a}}_{k-1};{\beta }_k)$ for the nuisance, and ${\pi }_k({\bar{s}}_k,{\bar{a}}_{k-1};{\phi }_k)$ for the propensity $\mathrm{pr}(A_k=1\mid \text{history})$. At decision $k$ (response ${\bar{V}}_{(k+1)i}$, with ${\bar{V}}_{(K+1)i}=Y_i$), solve jointly in $({\psi }_k,{\beta }_k,{\phi }_k)$:

$$\sum _{i=1}^n{\lambda }_k({\bar{S}}_{ki},{\bar{A}}_{(k-1)i};{\psi }_k)\{A_{ki}-{\pi }_k({\bar{S}}_{ki},{\bar{A}}_{(k-1)i};{\phi }_k)\}\{{\bar{V}}_{(k+1)i}-A_{ki}{C}_k(\cdots ;{\psi }_k)-h_k(\cdots ;{\beta }_k)\}=0,$$ $$\sum _{i=1}^n\frac{\partial h_k}{\partial {\beta }_k}\{{\bar{V}}_{(k+1)i}-A_{ki}{C}_k(\cdots ;{\psi }_k)-h_k(\cdots ;{\beta }_k)\}=0,$$

together with the binary-regression score for ${\phi }_k$. The estimated rule is ${\hat{d}}_k^{\text{opt}}=I\{C_k({\bar{s}}_k,{\bar{a}}_{k-1};{\hat{\psi }}_k)>0\}$, fit backward as in Q-learning (Eq. 32). A practical choice is ${\lambda }_k=\partial C_k/\partial {\psi }_k$; the variance-optimal ${\lambda }_k$ is complex.

Double robustness (§5.2)

The factor $\{A_{ki}-{\pi }_k\}$ multiplying the residual is the key: as long as the contrast $C_k$ is correctly specified, the estimating equation has mean zero — and hence yields a consistent estimator of ${\psi }_k$ (and the optimal regime) — if at least one of the propensity ${\pi }_k$ or the nuisance $h_k$ is correctly specified. This is the double-robustness property. By contrast, Q-learning requires correct specification of the entire Q-function (here both $h_k$ and $C_k$).

Q vs. A-learning: the trade-off

Efficiency vs. robustness, and simulation findings (§5.3, §6, Figs. 1-6)

• All models correct: Q-learning is more efficient (e.g. at $K=1$ with correct variance model, the Q-learning estimating equation is the optimal form; A-learning is generally not, so it is relatively inefficient).
• Propensity misspecified, contrast correct: A-learning still yields consistent inference on $\psi$ and the optimal regime, whereas Q-learning (which doesn’t use the propensity) can be inconsistent if its Q-function is wrong. Simulations (Figs. 1, 4): A-learning maintains high value-efficiency $R({\hat{d}}^{\text{opt}})$ across propensity misspecification.
• Q-function misspecified: A-learning’s robustness shows most clearly — its value-efficiency stays high while Q-learning’s degrades as misspecification grows (Figs. 2, 5).
• Both misspecified: neither dominates uniformly; performance depends on the direction/magnitude of misspecification (Figs. 3, 6).
• Performance metrics: MSE ratio (A/Q) of $\hat{\psi }$ components (>1 favors Q-learning) and the v-efficiency $R({\hat{d}}^{\text{opt}})=\mathbb{E}\{H({\hat{d}}^{\text{opt}})\}/H(d^{\text{opt}})$ — the fraction of the true optimal regime’s population value achieved.

Interpretability angle. A-learning is a middle ground: it allows flexible/nonlinear modeling of the nuisance $h_k$ while keeping a simple, interpretable parametric contrast $C_k$ — plausible when the science suggests a complex outcome surface but a simple optimal decision boundary. Under the null of no treatment effect in a SMART, contrast functions are zero and correctly specified by design, so A-learning gives consistent inference.

Connections

• The robust counterpart to Q-learning within the backward-induction framework; both estimate the same $d^{\text{opt}}$.
• G-estimation derives from Robins’s structural nested mean models; double robustness parallels that of doubly-robust estimators in single-stage causal inference.
• Demonstrated on the STAR*D depression study (§7) as well as the simulations summarized here.

See Also

• Q-learning — the full-outcome-model alternative
• Q- and A-learning - Overview — head-to-head comparison
• Time-Varying Treatments and G-computation — related sequential-treatment estimation