Identification
What has to be true for the number to mean something.
Identification is the argument that connects what you can compute to the effect you want. It rests on three assumptions. The data can speak to one of them. The other two you have to defend from what you know about the campaign.
The quantity you want was never recorded.
Every respondent has two possible answers: one if exposed, one if not. The survey records one of them. The effect is the difference between the two, so half of every comparison is missing.
| Respondent | Group | Aware if exposed | Aware if not exposed | Effect |
|---|---|---|---|---|
| A | Exposed | Yes | ? | ? |
| B | Exposed | Yes | ? | ? |
| C | Exposed | No | ? | ? |
| D | Control | ? | Yes | ? |
| E | Control | ? | No | ? |
| F | Control | ? | No | ? |
A propensity analysis fills the question marks for the exposed group by borrowing answers from controls who resemble them. The three assumptions below are the conditions under which that borrowing is legitimate.
Assumption one: unconfoundedness
You measured why people were exposed.
Among respondents who look the same on everything in your model, who ended up exposed must be as good as random. It also goes by conditional independence, ignorability, or selection on observables. If something you did not measure drives both exposure and the outcome, the estimate is biased, and nothing in the output will show it.
Covariate balance
Estimated effect
Argue it from the assignment.
Write down how respondents came to be exposed and check that each driver is in the model. This is a subject-matter argument. A statistic cannot make it for you.
Run a placebo outcome.
Add a KPI the campaign could not have moved, such as awareness of an unrelated brand. Adjusted lift on a placebo should be near zero. If it is not, something is still unbalanced.
Ask how strong a hidden cause would need to be.
Sensitivity analysis turns “there might be confounding” into a size. The package does not compute it; the reading list below covers E-values and Rosenbaum bounds.
Assumption two: overlap
Comparable people exist.
For every kind of exposed respondent, there have to be controls of the same kind. Also called positivity or common support. Drag the slider to make exposure more predictable, and watch what a “better” model does to the comparison.
A high AUC is a warning, not a result.
An AUC of 0.95 means you can nearly always tell exposed from control by their profile. So there are almost no controls who look like the exposed, and the estimate rests on a handful of unusual people. Overlap has to hold in the population. Fitting a better model does not create it.
What the package reports.
The common-support interval, counts of respondents in the control_only, both and exposed_only regions, and the overlap coefficient. Roughly: above 0.8 is good, 0.5 to 0.8 is workable, below 0.5 is poor.
What it warns about.
An AUC above 0.9, more than 5% of exposed respondents scoring above every control, and exposed respondents left unmatched by the caliper. Warnings are in result.warnings and on each method.
What you can do.
Trim to common support, use overlap weights, or tighten the caliper. Each one changes who the answer is about: dropping exposed respondents means the estimate no longer covers everyone the campaign reached.
ov = result.overlap
print(ov.overlap_coefficient) # 1 = identical distributions, 0 = disjoint
print(ov.common_support_lower, ov.common_support_upper)
for r in ov.regions: # control_only, both, exposed_only
print(r.region, r.n_exposed, r.n_control, f"{r.pct_exposed:.1f}% of exposed")
# Per-feature gaps: levels or ranges that only one group has
for f in result.feature_overlap:
if f.pct_exposed_unsupported > 0:
print(f.feature, f.pct_exposed_unsupported, f.exposed_only_levels)
Assumption three: consistency
“Exposed” means one thing.
The outcome you observe for an exposed respondent has to be the outcome of the exposure you are asking about, and one person's exposure must not change another person's answer. Statisticians bundle these as SUTVA: no hidden versions of the treatment, and no interference. Both are decided by how the exposure flag was built.
Where the flag came from.
Ad-server logs and panel matching record exposure. “Do you remember seeing this ad?” records an outcome. People who liked the ad remember it, so a recall-based flag sorts respondents by their reaction to the campaign and then credits the campaign for the difference.
Controls who saw it anyway.
A control exposed on another device, channel or household screen is counted as unexposed. The two groups become more alike than their labels say, and the measured lift shrinks towards zero.
One flag, many doses.
One impression and twenty are both “exposed”. The estimate is an average over whatever frequency mix your sample has, and it will not transfer to a campaign with a different mix. If frequency matters, define exposure by it and say what happens to everyone in between.
Effects that travel.
Shared households, word of mouth and social sharing let exposure reach people coded as controls. A respondent-level comparison then measures only the part of the effect that stayed with the person who saw the ad.
What each diagnostic can tell you, and where it stops.
Every check in the results is useful. None of them certifies the analysis. Each one checks something narrower than its name suggests.
| Diagnostic | What it shows | What it cannot show |
|---|---|---|
| Balance (SMD) | The measured covariates have similar distributions in the two groups after adjustment. | Anything about covariates that are not in the table. |
| Overlap coefficient, common support | Comparable respondents exist, as far as the measured covariates can tell. | Whether those covariates are the ones that matter. |
| AUC | How separable the groups are. | Model quality. Higher is worse for the comparison, and a low AUC does not mean confounders are missing. |
| Calibration | The scores behave like probabilities, so the weights built from them are sensible. | That the model includes the right variables. |
| Effective sample size | How much information survives the weighting. | Bias. A large effective sample can be precisely wrong. |
| Confidence interval, p-value | Sampling noise, given that the assumptions hold. | Error from a violated assumption. The interval does not widen to cover it. |
| Agreement across methods | The answer does not hinge on one way of using the score. | Correctness. All five methods use the same covariates and share the same blind spot. |
Sometimes the honest answer is that this data cannot say.
Propensity methods work when exposure was driven by things you measured. Online ad delivery often is not: people who are active online on a given day both see more ads and do more of everything else. In a randomised study of this pattern, the control group showed the same spike in activity on campaign day as the exposed group.
Exposure followed behaviour you cannot see.
Retargeting, look-alike audiences and bid optimisation select on signals that predict the outcome and are not in your survey.
Most of the exposed have no counterpart.
An overlap coefficient under 0.5, or a large exposed_only region, means the comparison exists for a small and unusual subset.
The flag depends on the respondent's reaction.
Recall-based or completion-based exposure cannot be repaired by adjustment. It needs a different exposure measure or a randomised holdout.
Background reading
- Potential outcomes frameworkPotential outcomes, SUTVA, unconfoundedness and overlap, stated formally.Causal inference
- Balancing theorem and failure modesWhy small standardised differences can sit alongside a biased estimate.Propensity score matching
- Common support and overlapExtrapolation, trimming, and how dropping treated units changes the estimand.Propensity score matching
- Sensitivity analysis in observational studiesCornfield's inequality, E-values and Rosenbaum bounds.Causal inference
- Doubly robust estimators and overlapOverlap as an assumption, and why a highly predictive propensity model signals trouble.Causal machine learning
- Experimental benchmarks for observational ad measurementHow far observational estimates landed from randomised ones across hundreds of campaigns.Market response models