Linzer 2013 - Reverse Random-Walk Model and Priors

Summary

Poll counts are binomial in a state-day vote share ${\pi }_{ij}$ whose logit is a slow state trend ${\beta }_{ij}$ plus a fast national campaign effect ${\delta }_j$. Both follow random walks that run backwards from Election Day, where ${\beta }_{iJ}$ gets an informative normal prior centred on the fundamentals forecast and ${\delta }_J=0$.

Overview

Notation: states $i=1,\dots ,50$; campaign days $j=1,\dots ,J$ with $J$ = Election Day; polls $k=1,\dots ,K_j$ with $n_k$ respondents expressing a major-party preference, $y_k$ of whom support the Democrat; $h_i$ the historical forecast of Democratic two-party share in state $i$. Undecided respondents are excluded. A poll is dated at its last field day.

Main Content

Observation model (Eq 1)

$y_k\sim \text{Binomial}({\pi }_{i[k]j[k]},n_k)$, where $i[k],j[k]$ are the poll’s state and day. Real polls are overdispersed (house effects, other non-sampling error); the paper argues the resulting overconfidence is minimal, that estimating firm effects is impractical because most pollsters run few polls, and that deflating $n_k$ would underestimate volatility and worsen overconfidence.

Decomposition (Eq 2)

${\pi }_{ij}={\text{logit}}^{-1}({\beta }_{ij}+{\delta }_j)$. ${\beta }_{ij}$: long-term state-level dynamics; ${\delta }_j$: national short-term departure from the forecast (${\delta }_j<0$ means the Democrat is polling below the forecast). Separating them lets states with no polls borrow the common pattern. Identification: ${\delta }_J\equiv 0$, so ${\pi }_{iJ}={\text{logit}}^{-1}({\beta }_{iJ})$.

Informative prior from fundamentals (Eq 3)

${\beta }_{iJ}\sim N(\text{logit}(h_i),{\tau }_i^{-1})$ with analyst-chosen precision ${\tau }_i=s_i^{-2}$. Smaller ${\tau }_i$ upweights polls; larger ${\tau }_i$ keeps forecasts near $h_i$. Sensitivity (Sec 4.4): ${\tau }_i$ should not generally exceed 20.

Reverse random walks

${\beta }_{ij}\mid {\beta }_{i,j+1}\sim N({\beta }_{i,j+1},{\sigma }_{\beta }^2)$ and ${\delta }_j\mid {\delta }_{j+1}\sim N({\delta }_{j+1},{\sigma }_{\delta }^2)$ for $j<J$, with uniform priors on ${\sigma }_{\beta }$ and ${\sigma }_{\delta }$. The walk “begins” on Election Day (as in Strauss 2007), so after the last poll the path reverts toward the prior mean. Reason (Gelman & King 1993): fundamentals predict where opinion ends up but not the path.

Behaviour (qualitative, from the text)

The forecast of ${\pi }_{iJ}$ is a compromise between the latest polls and $h_i$. If the election is close in time, ${\delta }_j\approx 0$ and ${\beta }_{ij}$ has little time to revert, so polls dominate. If it is far, the post-poll path depends on the prior: with large ${\tau }_i$ ${\beta }_{ij}\to \text{logit}(h_i)$ quickly; with small ${\tau }_i$ the path is driven by ${\delta }_j$ returning to 0, so a candidate behind the forecast (${\delta }_j<0$) gains and one ahead loses.

Windowing. To speed MCMC when forecasting far ahead, split the campaign into $J/W$ windows of $W$ days ($W=3$-5) and replace ${\beta }_{ij}$ by ${\beta }_{it[j]}$; ${\delta }_j$ stays daily because it fluctuates daily whereas $\beta$ changes gradually. Estimation. WinBUGS/R; 3 chains of 200,000 iterations, burn-in half, thinning 300; $\hat{R}\approx 1$.

Poll error sources (Sec 2.1). Sampling variability (half or more of the total variation), house effects, timing (early polls weakly correlated with the outcome), and campaign events such as conventions.

Examples

Scale of the reverse random walk

With ${\sigma }_{\beta }=0.02$ per day, the state trend moves by $0.02\sqrt{30}\approx 0.11$ on the logit scale over 30 days (about 2.7 points of vote share near 50%), whereas a prior with $\tau =10$ has sd $\approx 0.32$ (about 8 points). So 30 days from Election Day the prior still dominates ${\beta }_{iJ}$ unless polls say otherwise, and only the common ${\delta }_j$ (daily sd ${\sigma }_{\delta }$, typically larger) lets sparsely polled states learn the national mood.

# PyMC 5 sketch of the reverse random walk (untested)
import pymc as pm, numpy as np, pytensor.tensor as pt
with pm.Model():
    sb = pm.HalfNormal("sb", 0.05); sd = pm.HalfNormal("sd", 0.05)
    # reverse RW: build forward in reversed time, then flip
    beta_rev = pm.GaussianRandomWalk("beta_rev", sigma=sb, shape=(S, T),
        init_dist=pm.Normal.dist(logit_h[:, None], 1/np.sqrt(tau)))
    delta_rev = pm.GaussianRandomWalk("delta_rev", sigma=sd, shape=T)
    delta = delta_rev - delta_rev[0]            # delta_J = 0 (index 0 = election day)
    beta, d = beta_rev[:, ::-1], delta[::-1]
    p = pm.math.invlogit(beta[state_idx, day_idx] + d[day_idx])
    pm.Binomial("y", n=n_resp, p=p, observed=y_dem)

Connections

Compare with filtering/smoothing in Linear-Gaussian State-Space Models and The RTS Smoother: the reverse walk is a smoother-style prior anchored at the terminal time.

See Also

Linzer 2013 - Dynamic Bayesian Forecasting of Presidential Elections - Overview