Frame the question

The model answers whatever you hand it.

A propensity model will fit on any set of columns and return a lift number with a confidence interval. Whether that number means anything was decided earlier, by what you asked and what you let into the model.

Same classifier. Different job.

Most of the habits that make a good targeting model make a bad propensity analysis. The code looks the same, which is why the habits carry over unnoticed.

Targeting or response model
Propensity analysis
What it is for
Predict who will be exposed, click or buy.
Make exposed and control respondents comparable, so their difference can be read as an effect.
What good looks like
High AUC on a holdout set.
Balanced covariates and enough overlap. A high AUC is a warning that the groups barely overlap.
Which variables go in
Anything that improves the fit.
Only things fixed before exposure that cause both exposure and the outcome.
How you validate
Compare predictions with what happened.
You cannot. What exposed respondents would have said without exposure is never observed.
How it fails
Visibly: accuracy drops out of sample.
Silently: the estimate is biased and every diagnostic still looks clean.

A raw difference is two numbers added together.

Exposed respondents differ from controls for two reasons. The campaign changed them, and they were different people to begin with. The raw gap cannot tell these apart.

Exposed − controlwhat the survey shows
=
Effect on the exposedwhat the campaign did (the ATT)
+
Selection biashow the two groups would have differed with no campaign at all

Random assignment makes the second term zero by construction. Without it, adjustment can only remove the part of selection bias that runs through variables you measured.

The selection problem, in full

Say who the answer is about.

“What was the lift?” is not yet a question. An estimand names a population, an exposure, a comparison and an outcome. Change any one and the right number changes, even with the same data.

Whose outcomes are you asking about?
What counts as exposed?
Which outcome?
The config this question implies

Your question

, in

What this commits you to

    Brand lift is almost always an ATT question: did the campaign move the people it reached? That is the package default. Pick something else on purpose, and before you see results.

    How the estimands differ in practice Define the estimand first

    A variable earns its place by where it sits, not by how well it predicts.

    Below is a small brand-lift world where the true effect is known. Choose what goes into the propensity model and watch the estimate. The set with the best AUC gives the worst answer.

    Each dot is the estimate from one simulated survey of 1,200 respondents, using inverse probability weights from a logistic model on the variables you chose. Scores are clipped to 0.02–0.98.

    In your propensity model
    Click a variable here or in the diagram.
    0.50model AUC
    0bias in the estimate
    0spread across surveys

    Four kinds of variable. Four different answers.

    Include

    Causes of both exposure and the outcome.

    Age, media habits, prior relationship with the brand, category involvement. These are the confounders, and the analysis exists to balance them. Be generous here: leaving one out biases the estimate, while including a weak one costs little.

    Include if you have it

    Causes of the outcome only.

    They do not predict exposure, so they barely change the scores. They still tighten the estimate, because they soak up outcome variation that would otherwise be noise.

    Leave out

    Causes of exposure only.

    Targeting flags, ad-server quirks, which panel a respondent came from. They raise the AUC, thin out the overlap and widen the interval. If a confounder is still missing, they make its bias larger.

    Never

    Anything the exposure could have changed.

    Ad recall, site visits, post-campaign attitudes, anything measured after the first impression that the ad might have moved. Adjusting for it removes real effect or manufactures a false one.

    The survey ran after the campaign.

    Every answer in a post-campaign survey was given after exposure. “Prior brand usage” asked today is a memory, and seeing the ad can change what people remember. Treat a covariate as pre-exposure only when exposure could not plausibly have altered the answer: demographics, long-run media habits, purchases dated before the flight.

    Do not read the model's feature importance as a finding.

    The importance chart describes who got exposed. It says nothing about what drives awareness or intent, and the coefficients of adjustment variables are not effects of those variables. Epidemiologists call this the Table 2 fallacy.

    Decide before you look at lift.

    Choosing features, a caliper, a trimming rule and a method is design work, and it can all be done without opening the outcome columns. Tune those choices while watching the lift number and you are searching for a result. Five yes-or-no choices already make 32 different analyses.

    1. Write the question down.

      Population, exposure, comparison, outcome. One sentence, agreed with whoever asked for the number.

    2. Describe how people came to be exposed.

      Who decided who saw the ad: targeting rules, placements, the respondent's own media habits. This is the list your covariates have to cover.

    3. List the common causes, then check when each was fixed.

      For every driver of exposure, ask whether it also moves the outcome, whether you measured it, and whether exposure could have changed the answer.

    4. Name what you could not measure.

      There is always something. Writing it down tells readers which direction the estimate is likely to be wrong in.

    5. Fix the estimand, the methods, the trimming rule and the primary KPI.

      Run several methods and report all of them. Agreement between methods is reassuring; picking the best-looking one is not.

    6. Check overlap and balance. Then open the lift results.

      If the design fails those checks, change the design. Do it before the outcomes are in view.