Frame the question
The model answers whatever you hand it.
A propensity model will fit on any set of columns and return a lift number with a confidence interval. Whether that number means anything was decided earlier, by what you asked and what you let into the model.
Same classifier. Different job.
Most of the habits that make a good targeting model make a bad propensity analysis. The code looks the same, which is why the habits carry over unnoticed.
A raw difference is two numbers added together.
Exposed respondents differ from controls for two reasons. The campaign changed them, and they were different people to begin with. The raw gap cannot tell these apart.
Random assignment makes the second term zero by construction. Without it, adjustment can only remove the part of selection bias that runs through variables you measured.
Say who the answer is about.
“What was the lift?” is not yet a question. An estimand names a population, an exposure, a comparison and an outcome. Change any one and the right number changes, even with the same data.
Your question
, in
What this commits you to
Brand lift is almost always an ATT question: did the campaign move the people it reached? That is the package default. Pick something else on purpose, and before you see results.
How the estimands differ in practice Define the estimand first
A variable earns its place by where it sits, not by how well it predicts.
Below is a small brand-lift world where the true effect is known. Choose what goes into the propensity model and watch the estimate. The set with the best AUC gives the worst answer.
Each dot is the estimate from one simulated survey of 1,200 respondents, using inverse probability weights from a logistic model on the variables you chose. Scores are clipped to 0.02–0.98.
Four kinds of variable. Four different answers.
Causes of both exposure and the outcome.
Age, media habits, prior relationship with the brand, category involvement. These are the confounders, and the analysis exists to balance them. Be generous here: leaving one out biases the estimate, while including a weak one costs little.
Causes of the outcome only.
They do not predict exposure, so they barely change the scores. They still tighten the estimate, because they soak up outcome variation that would otherwise be noise.
Causes of exposure only.
Targeting flags, ad-server quirks, which panel a respondent came from. They raise the AUC, thin out the overlap and widen the interval. If a confounder is still missing, they make its bias larger.
Anything the exposure could have changed.
Ad recall, site visits, post-campaign attitudes, anything measured after the first impression that the ad might have moved. Adjusting for it removes real effect or manufactures a false one.
Every answer in a post-campaign survey was given after exposure. “Prior brand usage” asked today is a memory, and seeing the ad can change what people remember. Treat a covariate as pre-exposure only when exposure could not plausibly have altered the answer: demographics, long-run media habits, purchases dated before the flight.
The importance chart describes who got exposed. It says nothing about what drives awareness or intent, and the coefficients of adjustment variables are not effects of those variables. Epidemiologists call this the Table 2 fallacy.
Decide before you look at lift.
Choosing features, a caliper, a trimming rule and a method is design work, and it can all be done without opening the outcome columns. Tune those choices while watching the lift number and you are searching for a result. Five yes-or-no choices already make 32 different analyses.
Write the question down.
Population, exposure, comparison, outcome. One sentence, agreed with whoever asked for the number.
Describe how people came to be exposed.
Who decided who saw the ad: targeting rules, placements, the respondent's own media habits. This is the list your covariates have to cover.
List the common causes, then check when each was fixed.
For every driver of exposure, ask whether it also moves the outcome, whether you measured it, and whether exposure could have changed the answer.
Name what you could not measure.
There is always something. Writing it down tells readers which direction the estimate is likely to be wrong in.
Fix the estimand, the methods, the trimming rule and the primary KPI.
Run several methods and report all of them. Agreement between methods is reassuring; picking the best-looking one is not.
Check overlap and balance. Then open the lift results.
If the design fails those checks, change the design. Do it before the outcomes are in view.
Background reading
The ideas on this page are developed in the Second Brain knowledge base. These notes are the closest sources.
- The selection problemWhy a naive difference equals the effect on the treated plus selection bias.Econometrics
- EstimandsDefine the causal quantity before choosing a statistical model.Research methodology
- DAGs and causal identificationForks, chains, colliders and the back-door criterion.Econometrics
- Conditional independence assumptionSelection on observables, and why controls must be pre-treatment.Econometrics
- Logic of regression adjustmentThe adjustment set as a deliberate choice, not a kitchen sink.Research methodology
- Table 2 fallacyCoefficients on adjustment variables are not effects.Research methodology
- Garden of forking pathsHow data-dependent analysis choices inflate false positives without any fishing.Research methodology
- The propensity score and the balancing propertyVariable selection aims at balance, not at the c-statistic.Econometrics