Two Ways to Price a Confounder You Cannot See

Matthew Reda Causal inference

Budgets go up when someone expects demand to go up. That single habit puts a latent driver of sales into the spend column of every media mix model, and no amount of data will tell you it is there. The fit will be good. The intervals will be tight. What you can do instead is put a price on the assumption: state how much hidden confounding your conclusion could absorb before it stops being your conclusion. Two research traditions do this, they ask different questions, and one of the two most-cited tools does not apply to an ROI at all.

The Assumption With No Test

Identification in an observational MMM rests on there being no unmeasured common cause of spend and the KPI, and for media it is always the same candidate, the one this framework's documentation calls unobserved demand. A planner who raises budget into an expected upswing has made spend a function of something that also drives sales. Adjusting for what you measured does not remove what you did not.

Goodness of fit cannot see this. A confounded model and an unconfounded one can produce the same residuals and the same posterior width, and both pass a posterior predictive check. The holdout post makes the sharper version of that argument: predictive score has near-zero resolving power over the causal specifications you actually want to choose between. So the question has to change. Rather than asking whether the assumption holds, ask what it would take to break the decision that rests on it.

Where this sits next to the other posts in the series

Demand proxies and proximal inference gives a falsification test with real power: a negative control that detects confounding from data. Sensitivity analysis is the complement, for the ordinary case where no valid negative control exists. It prices confounding without new data, and it can never detect any.

Causal estimates from observational data states the general principle and defines the E-value in one box. This post is about what happens when you try to use these devices on a media mix model, where one of them refuses to run.

How Strong Would It Have to Be

The older tradition is frequentist and asks a single question: how strong would an unmeasured confounder have to be, in association with both treatment and outcome, to overturn what you observed? Everything else is refinement of that question.

It starts with a six-author consensus paper on smoking and lung cancer. Cornfield and colleagues (1959) showed that for a binary confounder to explain away an observed relative risk, its association with the exposure must exceed that relative risk. Schlesselman (1978) supplied the matching condition on the outcome side. The pair are now called the classical Cornfield conditions, and their appeal is that they need almost no assumptions to state.

They are also necessary conditions, and only that. Ding and VanderWeele (2016) give the counterexample that settles it. Take an observed risk ratio of 3. The classical threshold asks both confounder associations to exceed 3, and the sharper bound they derive asks the larger of the two to exceed 5.45. A confounder associated 5.5 with the exposure and 3.1 with the outcome clears both thresholds. Its joint bounding factor is

$$ \frac{\mathrm{RR}_{EU} \times \mathrm{RR}_{UD}}{\mathrm{RR}_{EU} + \mathrm{RR}_{UD} - 1} \;=\; \frac{5.5 \times 3.1}{5.5 + 3.1 - 1} \;=\; 2.24, $$

which is smaller than the 3 it would have to reach. Such a confounder passes both tests and still cannot explain the finding away. Reading a necessary condition as a sufficient one is the single most common error in this literature, and it runs in the dangerous direction: it declares results explained away that are not.

VanderWeele and Ding (2017) turned the bound into one number. The E-value is the smallest association, on the risk-ratio scale, that a confounder would need with both treatment and outcome to fully explain the observed effect:

$$ \text{E-value} \;=\; \mathrm{RR} + \sqrt{\mathrm{RR} \times (\mathrm{RR} - 1)}. $$

Their worked example takes an observed risk ratio of 3.9 and reports an E-value of 7.2. The number is easy to communicate, which is most of why it spread. They ask for two of them, one for the point estimate and one for the confidence limit nearer the null, and the second is exactly 1 whenever the interval already crosses the null.

Cinelli and Hazlett (2020) rebuilt the same question for ordinary least squares, on the partial \( R^2 \) scale, which is where a marketing model actually lives. Their robustness value answers how strong a confounder associated equally with treatment and outcome must be to shrink the estimate by a given fraction \( q \). With \( f_q = q \, |t| / \sqrt{\mathrm{df}} \),

$$ \mathrm{RV}_q \;=\; \tfrac{1}{2}\left( \sqrt{f_q^4 + 4 f_q^2} \;-\; f_q^2 \right). $$

This framework has computed that per channel for some time, in validation/sensitivity_unobserved.py, using the posterior mean over the posterior standard deviation in place of a \( t \) statistic and a deliberately generous nominal degrees of freedom. A channel at \( t = 4.7 \) with 140 degrees of freedom scores a robustness value of 0.33. Read it as a demand: an omitted variable would need to explain a third of the residual variation in both spend and sales.

The E-Value Does Not Apply to an ROI

Here the transplant fails. The E-value is defined for a ratio of risks, rates, or prevalences, all of which are ratios of probabilities. A media ROI is a ratio of two continuous quantities, incremental revenue over spend, and the bounding-factor algebra behind the E-value does not hold for it. Nothing about the formula complains when you feed it 1.32. It returns a number, and the number means nothing.

So the framework refuses. evalue() in diagnostics/bias_sensitivity.py accepts exactly three measures and declines anything else with its reason attached:

# illustrative
from mmm_framework.diagnostics.bias_sensitivity import evalue

evalue(3.9, measure="risk_ratio").point      # 7.26
evalue(1.32, measure="roi").available        # False

⚠️ The refusal carries the reason, not just a failure

The returned object explains itself: the E-value "is defined for ratio measures on a rate or binary outcome (prevalence_ratio, rate_ratio, risk_ratio), not for 'roi'. An ROI or ROAS is a ratio of continuous quantities, not a risk ratio", and it points the caller at the tipping point instead. Odds ratios and hazard ratios are refused too, by that same generic message. Why they sit outside the whitelist is documented in the source comment above it, that converting them to the risk-ratio scale depends on how rare the outcome is.

A vendor scorecard quoting an E-value for a channel ROI is quoting a quantity with no derivation behind it. That is worth checking, because the number is cheap to compute and looks authoritative.

Make It a Parameter

The Bayesian tradition asks a different question. Rather than solving for the confounder strength that would overturn the result, it makes the bias itself a parameter and puts a prior on it, then reports what that does to the conclusion. Greenland (2009) states what the object is, that a bias is the discrepancy between the target parameter and the analogous one the data actually identify. His earlier multiple-bias modelling paper (2005) is blunt about the consequence, writing that such parameters are typically not identified by the analysis data and that results then depend completely on the priors placed on them.

The PyMC Labs treatment that this framework follows is the current, readable version of the idea. Nathaniel Forde's notebook takes a checkout experiment whose randomization leaked, writes the observed effect as a causal part plus a bias part, and sweeps the prior on the bias:

$$ \hat{d} \;=\; \tau + \beta, \qquad \beta \sim \mathcal{N}(\mu, \sigma^2). $$

That one line asks you for two numbers the data cannot supply. A direction, \( \mu \), which for demand-chasing media is positive because the observed effect is the overstated one. A width, \( \sigma \), for how uncertain you are about the size. The value the estimate has to clear to matter is the third input, and the decision that prompted the model already supplies it.

In this framework the arithmetic is closed form, which matters more than it sounds. The de-biased posterior is a Gaussian mixture over draws that already exist, so

$$ P(\tau \gt r) \;=\; \frac{1}{D}\sum_{d} \Phi\!\left( \frac{x_d - \mu - r}{\sigma} \right) $$

evaluates exactly, with no sampling and no seed to reproduce. A tipping point bisected inside that mixture is trustworthy at small draw counts, and the identical arithmetic runs in JavaScript, which is how the figure below recomputes it in your browser. The framework's own report transports the curve the scan already produced, so the line it draws can never disagree with the tipping point it reports.

What a tipping point reads like

Take a channel whose contribution ROI posterior has mean 1.32 and standard deviation 0.18. Before any bias is assumed it clears break-even with probability 0.977. Widen an unsigned bias commitment and that probability falls: 0.931 at a spread of 10% of the estimate, 0.842 at 20%, 0.769 at 30%.

The tipping point is where it crosses the decision threshold. At 90% confidence this channel tips at a bias spread of 14% of its own size. Hold it to 95% and it tips at 7%. Committing to a direction rather than a width brings the conclusion down far faster, because a signed bias moves the whole posterior instead of spreading it: a mean bias of 10% with a narrow spread already drops the probability to 0.839.

Figure 1. Two commitments, two rates of collapse

The same posterior under the two things an analyst can commit to. One curve widens an unsigned bias. It says only that confounding of some size could be present, without committing to which way it runs. The other commits to a direction and holds the spread narrow at 5%. Drag the slider to change the confidence you demand. The tipping point for each commitment is the bias where its curve crosses that line.

0.90
Tips on width at…
Tips on direction at…
Illustrative Price benchmark…
Verdict at that benchmark…

Committing to a direction costs more than twice as much conclusion as committing to a width of the same size. At the default 90% threshold the unsigned curve tips at 14%, the signed one at 6%. The dashed line marks an illustrative benchmark, a confounder as strong as Price implying a 9% bias. Both curves are the closed-form mixture the framework computes, over a fixed 128-point quantile grid of the posterior described above, so the numbers here are the numbers in the text.

The Number That Never Settles

An analyst meeting this for the first time usually asks when more data will resolve it. It will not, and the precise statement is worth carrying because the loose version is wrong.

Gustafson (2005) reparameterizes a non-identified model into an identified part \( \phi_I \) and a non-identified part \( \phi_N \), chosen so the likelihood depends only on \( \phi_I \). Standard asymptotics then apply to \( \phi_I \), and the conditional distribution satisfies

$$ f(\phi_N \mid \phi_I, \text{data}) \;=\; f(\phi_N \mid \phi_I). $$

Conditional on the identified part, the data have said everything they can say, and no sample size changes that. Greenland (2009) puts the same point in terms of the profile likelihood being flat in the bias parameter.

⚠️ The tempting shorthand is false

"For a non-identified parameter the posterior is just the prior" is the version people repeat, and Gustafson rebuts it in the same paragraph that establishes the result. The equality above is conditional on \( \phi_I \). Marginally there is still indirect learning whenever the prior makes the two parts dependent, so the marginal posterior for the bias generally does move. What does not happen is concentration: it never sharpens toward a point the way an identified parameter does.

This is the honest reason a sensitivity analysis is a reporting device. It estimates nothing. An unfalsifiable assumption becomes a statement about which conclusions survive which commitments, and the commitments stay visible.

A Tipping Point Needs a Benchmark

On its own a tipping point is a slider. Saying that a channel tips at 14% invites the obvious question: is 14% a lot? Without an answer the analyst supplies one from intuition, which is where sensitivity analysis earns its reputation as a rhetorical exercise.

Cinelli and Hazlett's second contribution closes that gap. Price a hypothetical confounder against a covariate you did measure. If a confounder as strong as Price implies a bias of 9%, and the conclusion tolerates 14%, the argument now has a referent that a reader can dispute on its merits. This framework implements the bound as a port of sensemakr, in validation/confounding_sensitivity.py:

$$ |\text{bias}| \;=\; \mathrm{se} \times \sqrt{\mathrm{df}} \times \sqrt{ \frac{R^2_{Y \sim Z \mid D,X} \; R^2_{D \sim Z \mid X}}{1 - R^2_{D \sim Z \mid X}} }. $$

What makes this checkable rather than merely cited: when \( Z \) really is the omitted confounder, the identity holds with equality. Fit the same data with and without \( Z \), and the predicted bias equals the realized one. The test suite asserts that to a relative tolerance of 1e-9, and the parametrized cases meet it at machine precision, between 1e-16 and 1e-13. That is a stronger guarantee than any reference number remembered from a paper.

Two implementation details are load-bearing and neither is obvious. The validity condition on the bound is \( R^2_{D \sim X_j \mid X} \lt 1 / (1 + k_d) \), so at \( k_d = 1 \) the real limit is 0.5 rather than the 1 a casual reading suggests. Past it the formula square-roots a negative number and returns NaN, and NaN compares false against every fragility threshold, which would read as "not fragile". The framework refuses and names the largest admissible \( k_d \) instead of clipping. Separately, benchmark covariates default to controls the DAG has labelled as confounders, because benchmarking against a mediator or a collider produces a bound with no causal meaning.

Two Caveats That Point the Same Way

Both devices went into the same codebase, and they turned out to fail in the same direction for opposite reasons.

The robustness value is computed from \( |t| = |\text{mean}| / \text{sd} \). A tighter prior shrinks the posterior standard deviation, which raises \( t \), which raises the robustness value. Tightening a prior therefore makes a channel look more robust to confounding, on a scale where nothing about confounding has changed.

The Cinelli–Hazlett benchmark has the opposite mechanism. Its identity is calibrated on an OLS fact, that \( \mathrm{se} \times \sqrt{\mathrm{df}} = \lVert y_{\text{res}} \rVert / \lVert d_{\text{res}} \rVert \). Substitute a Bayesian posterior standard deviation and the identity breaks by whatever the prior contributed. Because this framework's media priors are informative with positive support, the posterior standard deviation is the smaller quantity, so the implied bias comes out too small and the model reports robustness the data never supplied. That is why benchmark_bias_priors refits ordinary least squares rather than reusing the posterior it already has.

One mechanism inflates the numerator, the other deflates the bound. They are opposite operations and they arrive at the same place: a tighter prior makes both numbers more reassuring. An analyst who has internalised one caveat will reasonably assume the other path inherits it, and the assumption is backwards.

The framework's answer is to measure prior contraction and refuse to quote resilience below a threshold of 0.20. Under that, the report says the posterior is mostly its prior and that the tipping point reflects prior tightness rather than evidence about confounding. The companion post on priors works through what that looks like when it happens.

What This Does Not Buy

Sensitivity analysis never detects confounding. It reports the exposure and leaves the judgement where it belongs.

Two of the limits are about what the words mean, and they matter most when a tipping point gets quoted in a deck. The verdict vocabulary here tops out at resilient rather than robust, and the difference is deliberate: the claim is only that a conclusion survived the range that was actually scanned, and every result carries the widest bias it was tested against. A benchmark is an argument about plausibility, not a bound on the truth, since the real confounder is under no obligation to resemble the covariates you happened to collect.

The last limit is arithmetic. Independence between the bias and the causal part is an assumption baked into the closed form. That is exactly right when the input draws come from a flat-prior posterior of \( \tau + \beta \), and only approximately right for an MMM posterior carrying an informative ROI prior, where the effective prior on \( \tau \) becomes the media prior convolved with the bias prior. Coherent, and undeclared. So the report asks what the same commitment implied before any data arrived, and says so out loud when a conclusion's prior had already cleared the threshold.

What would improve this most is a bias prior that comes from a measurement rather than a ladder of named guesses. Placebo spreads from matched-market designs already give one for experiments, where the excess of realized over analytic variance is a defensible width. Doing the same for observational media would mean finding a period where spend moved for a reason unrelated to demand, and treating the gap as the measurement. That is a harder search than it sounds, and it is the right next problem.

Takeaways

  • No-unobserved-confounding has no test. Sensitivity analysis replaces the untestable question with a reportable one: how much hidden bias would this conclusion absorb before changing?
  • The Cornfield conditions are necessary, not sufficient. A confounder can clear both thresholds and still fail to explain the result away, as Ding and VanderWeele's (5.5, 3.1) example does against an observed risk ratio of 3.
  • The E-value is defined for risk, rate, and prevalence ratios. An ROI is a ratio of continuous quantities, so this framework refuses to compute one and points at the tipping point instead.
  • A Bayesian bias parameter is not identified, and more data does not sharpen it. Conditional on the identified part of the model, the data have already said everything they can.
  • A tipping point without a benchmark is a slider. Pricing a hypothetical confounder against a measured covariate is what turns it into an argument someone can dispute.
  • A tighter prior inflates the robustness value and deflates the implied bias bound. Two opposite mechanisms, both making a channel look safer than the data warrant.

References