Each recipe gives the config, what to check afterwards, and the catch. They assume df is loaded and a base cfg exists, as in the walkthrough.
Brand lift for one campaign
Situation
One exposed group, one control group, a handful of KPIs. You want the effect on the people the campaign reached.
Check afterwards
The overlap coefficient is above 0.5, no variable is imbalanced after adjustment, and ipw and aipw agree.
The catch
The answer is only as good as the covariate list. Write down what drove exposure before choosing features.
cfg = AnalysisConfig(
dataset_id="survey",
treatment_column="group",
exposed_values=["Exposed"], control_values=["Control"],
outcome_columns=["aided_awareness", "favorability", "consideration"],
model_features=["age", "gender", "region", "tv_hours_week",
"social_media_hours_day", "prior_brand_usage"],
estimand="ATT",
methods=["ipw", "aipw", "nn_matching"],
)
result = run_analysis(df, cfg)
print(result.overlap.overlap_coefficient)
for m in result.methods:
print(m.label, m.balance_summary.n_imbalanced_after)
A sample that was quota-matched
Situation
Controls were recruited to mirror the exposed group on age band, gender and region. You want the analysis to respect those cells.
Check afterwards
matching.n_exposed_unmatched. Exact cells plus a caliper can leave many exposed respondents without a match.
The catch
Quota matching balances the quota variables and nothing else. Media habits and brand history still need to be in model_features.
cfg = cfg.model_copy(update={
"matching_features": ["age_band", "gender", "region"], # exact-match cells
"include_matching_in_model": True, # also feed them to the score
"methods": ["nn_matching", "ipw"],
"match_ratio": 2, # two controls per exposed respondent
"caliper": 0.2,
})
result = run_analysis(df, cfg)
info = result.methods[0].matching
print(info.n_exposed_matched, info.n_exposed_unmatched,
info.n_unique_controls_used, info.n_exact_strata)
Text scales and top-two box
Situation
Purchase intent is a five-point text scale and the business reads it as “definitely or probably will”.
How outcomes are read
Yes/No, True/False and 0/1 columns become proportions. Numeric scales are reported as means. A text scale needs its positive levels listed, and is then recoded to 1 and 0.
The catch
Choose the box before you see results. Top box and top-two box can tell different stories, and picking afterwards is a fork.
cfg = cfg.model_copy(update={
"outcome_columns": ["aided_awareness", "favorability", "purchase_intent"],
"outcome_positive_values": {
"purchase_intent": ["Definitely will", "Probably will"],
},
})
result = run_analysis(df, cfg)
# The run records how each outcome was interpreted
print(result.warnings)
for c in result.methods[0].contrasts:
print(c.outcome, c.outcome_type)
Several exposure cells, or messy group codes
Situation
The group column holds more than two values: several creative cells, or codes such as “Unknown”. The example uses a column named cell.
How rows are assigned
Rows matching exposed_values are exposed and rows matching control_values are control. Everything else is dropped and counted. If control_values is left unset, every other non-missing value becomes control, with a warning.
The catch
Each cell's ATT describes that cell's own audience. Two cells can differ because they reached different people, so do not rank creatives on these numbers alone.
results = {}
for cell in ["Video", "Display", "Video + Display"]:
cfg_cell = cfg.model_copy(update={
"name": cell,
"treatment_column": "cell",
"exposed_values": [cell],
"control_values": ["Control"], # always name the controls explicitly
})
results[cell] = run_analysis(df, cfg_cell)
for cell, r in results.items():
print(cell, r.sample.n_exposed, r.sample.n_control,
"dropped:", r.sample.n_dropped_treatment_missing)
Missing answers
Situation
Some respondents skipped income, or a KPI question.
What happens by default
With missing_strategy="impute", numeric features are median-imputed and gain a missing-indicator column, so the two groups are also balanced on who skipped the question. A missing categorical answer is treated as its own level. A respondent missing a KPI is dropped for that KPI only.
The catch
"drop" removes every respondent with any missing feature. That is simpler to explain and can change who the sample represents.
profile = profile_dataframe(df, dataset_id="survey", filename="survey.xlsx")
print([(c.name, c.n_missing) for c in profile.columns if c.n_missing])
strict = run_analysis(df, cfg.model_copy(update={"missing_strategy": "drop"}))
print(strict.sample.n_dropped_feature_missing)
# n differs by outcome when KPI answers are missing
for c in strict.methods[0].contrasts:
print(c.outcome, c.n_exposed, c.n_control)
Thin overlap
Situation
The overlap coefficient is low, a few controls carry most of the weight, or the run warns that exposed respondents score above every control.
Options
Trim to the shared score range, switch to overlap weights, or both. Compare the effective number of controls before and after.
The catch
All of these narrow the population. Report how many exposed respondents were removed, and describe the estimate as applying to the comparable ones.
base = run_analysis(df, cfg)
print(base.overlap.overlap_coefficient, base.warnings)
trimmed = run_analysis(df, cfg.model_copy(update={
"trim": "common_support", # or "crump": keep scores 0.1 to 0.9
"methods": ["ipw", "overlap_weights"],
}))
print(trimmed.overlap.n_trimmed_exposed, trimmed.overlap.n_trimmed_control)
for m in trimmed.methods:
print(m.label, m.weights.ess_control, m.weights.max)
Doubting the model's shape
Situation
Exposure may depend on the covariates in ways a logistic model misses: thresholds, interactions, frequency caps.
Options
Add aipw, which is consistent when either the score model or the outcome model is right. Rerun with gradient_boosting and compare balance.
The catch
Both guard against the wrong functional form. Neither helps with a variable that is missing. With tree models, bootstrap intervals hold the scores fixed and the run says so.
for model_type in ["logistic", "gradient_boosting"]:
r = run_analysis(df, cfg.model_copy(update={
"model_type": model_type,
"methods": ["ipw", "aipw"],
}))
print(model_type, "AUC", round(r.model.auc, 2))
for m in r.methods:
c = m.contrasts[0]
print(" ", m.method, m.balance_summary.max_abs_smd_after,
f"{c.lift:+.3f} [{c.ci_low:+.3f}, {c.ci_high:+.3f}]")
A placebo outcome
Situation
You want evidence about confounding beyond the balance table.
How
Add a KPI the campaign could not have changed: awareness of an unrelated brand, or an attitude the creative never touched. Its adjusted lift should be close to zero.
The catch
A clean placebo is encouraging and proves nothing. A placebo that moves is a clear sign something is unbalanced. In the sample data, unaided_awareness plays this role.
placebo = "unaided_awareness" # no true effect in the sample data
r = run_analysis(df, cfg.model_copy(update={
"outcome_columns": list(dict.fromkeys([*cfg.outcome_columns, placebo])),
}))
for m in r.methods:
c = next(c for c in m.contrasts if c.outcome == placebo)
print(m.method, f"raw {c.raw_diff:+.3f} adjusted {c.lift:+.3f}",
"significant" if c.significant else "not significant")
The effect across the whole audience
Situation
The decision is about extending the campaign to everyone, so you want the average effect over exposed and unexposed alike.
Check afterwards
Effective sample size in both groups, and the largest weight. The ATE needs exposed respondents who look like typical controls.
The catch
You are extrapolating the effect to people the campaign did not reach. If those people differ from the reached in how they respond, the data cannot tell you.
r = run_analysis(df, cfg.model_copy(update={
"estimand": "ATE",
"methods": ["ipw", "aipw"],
# "weight_clip": 0.99, # only if a few weights dominate; it trades variance for bias
}))
for m in r.methods:
w = m.weights
print(m.label, "ESS exposed", round(w.ess_exposed), "ESS control", round(w.ess_control),
"max weight", round(w.max, 1), "clipped", w.n_clipped)
Lift by segment
Situation
Stakeholders want lift by gender, age band or region.
How
List the columns in subgroup_columns. Numeric columns with more than 10 values are cut into quintiles. Levels with fewer than 10 respondents in either group are skipped.
The catch
The weights were built to balance the whole sample. Inside a small segment they may not. Read segment results as descriptive, and name the segments you care about in advance. Before using them to choose an audience, read picking audiences from lift.
r = run_analysis(df, cfg.model_copy(update={
"subgroup_columns": ["gender", "age_band"],
"methods": ["ipw"],
}))
for s in r.methods[0].subgroups:
if s.outcome == "aided_awareness":
print(f"{s.subgroup_column}={s.subgroup_level:12} n={s.n_exposed}/{s.n_control} "
f"lift {s.lift:+.3f} [{s.ci_low:+.3f}, {s.ci_high:+.3f}]")
Does the answer depend on my settings?
Situation
You chose a caliper and a model type, and want to know how much rides on those choices.
How
Rerun over a small, pre-declared grid and report the range. Set n_bootstrap=0 for speed while exploring.
The catch
This is a robustness report for a result you already committed to. Used to pick the setting with the best-looking lift, it is a search for significance.
rows = []
for model_type in ["logistic", "gradient_boosting"]:
for caliper in [0.1, 0.2, 0.25]:
r = run_analysis(df, cfg.model_copy(update={
"model_type": model_type, "caliper": caliper,
"methods": ["nn_matching", "ipw"], "n_bootstrap": 0,
}))
for m in r.methods:
rows.append((model_type, caliper, m.method, m.contrasts[0].lift))
lifts = [lift for *_, lift in rows]
print(f"primary KPI lift ranges from {min(lifts):+.3f} to {max(lifts):+.3f}")
Many surveys from the command line
Situation
A tracker delivers a new file every wave, with the same columns.
How
Keep one cfg.json under version control and run the CLI per file. dataset_id defaults to the file name.
The catch
A fixed config is the point: the analysis was decided before the data arrived. Keep it fixed, and still read each wave's overlap and warnings.
# Inspect a new file's columns
uv run propensity-lift profile waves/2026-10.xlsx --sheet Respondents
# Run every wave with the same config
for f in waves/*.xlsx; do
uv run propensity-lift run "$f" --config cfg.json \
--out "results/$(basename "$f" .xlsx).json" --quiet
done
Over HTTP
Situation
Another system needs to submit analyses and collect results.
How
Upload a file, post an AnalysisConfig that references the returned dataset_id, poll until the job completes, then read the result or download the Excel export. Analyses run in a process pool, so several can run at once.
The catch
The config is validated when you submit it. A column that does not exist, or an empty group, comes back as a 422 with a message.
A model of who converts is not a model of who to convince.
In much of marketing, “propensity model” means something else: a classifier fitted on demographics and first-party data to predict who will convert, with no exposed or control group at all. It is a legitimate and useful model. It is not what this package does, and it cannot tell you whether advertising to the people it ranks highly will change anything.
Propensity to convert
Propensity score (this package)
What it predicts
Whether a person converts, buys or signs up.
Whether a person was exposed to the campaign.
Data it needs
Customers and non-customers, with demographics and first-party behaviour. No exposure flag.
Exposed and unexposed people, the outcome for both, and what drove exposure.
Ordering leads for a sales team, sizing next quarter's volume, and setting budgets all need a calibrated chance of conversion. No causal claim is involved.
Good for suppression and seeds.
Leaving out people who already converted, or seeding a lookalike audience, uses the score as a description of who your customers resemble.
Silent on incrementality.
The highest scorers are the people most likely to convert without you. Whether an ad moves them is a different quantity, and this model never sees it.
Conversion rate among the people you target
The number a campaign dashboard reports.
Extra conversions the ads caused, per 1,000 targeted
The number the budget is actually buying.
Of 20,000 people, ranked by each strategy.
Who the ad persuades
0conversion rate, targeting by score
0extra conversions per 1,000, by score
0extra conversions per 1,000, at random
The conversion model here is perfect: it knows each person's true chance of converting without the ad. A real one does worse. “By true uplift” is what you would get if you could see each person's response to the ad, which only an experiment can estimate.
Where it goes wrong
Reporting the top decile's conversion rate as success.
Targeted people convert at a high rate because they were chosen for it. That rate says nothing about what the campaign added. Only a comparison with similar people who were not contacted does.
Features that leak the outcome.
Account created, cart visits or last login recorded inside the conversion window are partly the conversion itself. The model looks brilliant offline and fails live. Build every feature from data before a fixed prediction date.
Learning your old media plan.
Past campaigns decided who was contacted, and contacted people convert more. A model trained on that history partly learns who you targeted last time, and recommends them again.
Judging it by AUC alone.
AUC measures ranking. Budgets need probabilities that are right on average. Check calibration and conversions by score decile on a later period than the one you trained on.
Reading feature importance as cause.
App users converting more does not mean pushing the app creates conversions. Importance describes the prediction, not what would happen if you changed the feature.
The feedback loop.
Target the high scorers, retrain on what they did, and the low scorers never get a chance to prove the model wrong. The score hardens around your first guess.
Doing it properly
Fix a prediction date.
Features come only from data before it; the outcome comes from a window after it. Anything recorded in between is off limits.
Validate on a later period.
Train on one quarter, test on the next. Report calibration and conversions by decile next to the AUC.
Keep a random holdout in every campaign.
Leave a random share of each score band uncontacted. The difference between contacted and held-out people in a band is the lift there, and the holdout gives the next model unbiased training data.
Target where the lift is, not where conversion is highest.
If the top band converts at 30% with or without the ad and the middle band moves from 8% to 12%, the middle band is the better buy.
No holdout, but exposure was recorded?
Then this package applies. Code who was exposed, use the conversion model's inputs that were fixed before the campaign as covariates, and follow the walkthrough. Every assumption on the identification page comes with it.
Targeting the next flight
Aim at the people the ad moves, not the people it reaches.
Once a lift study is done, the next question is usually who to buy next time. The model you just fitted looks like it should answer that. It answers a different question, and the obvious ways of reusing it send budget to people who would have responded anyway.
The propensity score says who you already reach.
It ranks respondents by how likely the media plan was to expose them. Targeting on it buys more of the audience you already have, where comparable controls are scarcest. It is never a targeting score.
A response model says who will say yes.
Ranking by predicted awareness or intent finds people who are already aware or already buying. Many would have answered the same way without the ad, so the spend shows up in the outcome and not in the lift.More on conversion models.
Lift by segment says where the ad changed something.
The adjusted difference between exposed and comparable controls inside a segment is the only one of the three that measures the ad. It is also the noisiest, and that is where the trouble starts.
Twelve segments, ranked by estimated lift
One simulated survey. The orange row is the segment a ranking would pick. The dashed line is the true average lift across all twelve.
A 3,000-respondent survey cut twelve ways leaves about 120 exposed and 120 controls per segment.
At “not at all”, every segment has the same true lift of +4 points.
0estimated lift of the picked segment, points
0its true lift, points
0average overstatement of the pick in points, over 2,000 surveys
0surveys where the pick really is the best segment
Where it goes wrong
Taking the top segment.
With a dozen segments, the highest estimate is usually the luckiest sample, not the strongest response. Its lift shrinks when you buy it. The lab above shows by how much.
Segments with few controls.
The segment the plan was bought on has the fewest unexposed respondents. Lift there rests on a handful of heavily weighted controls. Read the counts and the interval before the number.
Segments the ad defined.
“People who remember the ad” or “site visitors” are selected on the response. Segment only on what was fixed before the flight, and on what a media buyer can actually target.
Assuming the effect carries over.
The ATT describes people the campaign reached, at the frequency they got. A new audience is, by definition, people it did not reach. Unmeasured differences, another frequency or another creative can all change the response.
“Significant here, not there.”
Significant for 18–34 and not for 35+ is not evidence that the two differ. That needs the interval of the difference between their lifts. Heavily overlapping intervals rarely support a difference.
Slicing until something shows.
Every cut is another draw. Ten columns of five levels is fifty segments, and at the 95% level two or three will look significant when no segment differs at all.
Doing it properly
Name the segments before you look.
A handful, each with a reason the ad should work differently there. Put them in subgroup_columns before the outcomes are in view.
Use columns fixed before the flight that you can buy against.
Demographics, region, long-run media habits, category purchases dated before the campaign.
Read counts and intervals with every lift.
The package skips levels with fewer than 10 respondents in either group. That is a floor, not a target. An interval wider than the lift itself means the segment is unread.
Discount the winner.
Expect the top segment to land closer to the overall lift next time. A segment that beats the overall estimate by less than its own interval width is not shown to be different.
Confirm with a holdout before scaling.
Run the targeted flight with a randomised control group inside the new audience. That measures the lift you are buying, without any of the assumptions on this page.
r = run_analysis(df, cfg.model_copy(update={
"subgroup_columns": ["age_band", "region"], # named in advance, fixed before the flight
"methods": ["ipw"],
}))
m = r.methods[0]
overall = next(c for c in m.contrasts if c.outcome == "aided_awareness")
print(f"overall lift {overall.lift:+.3f} [{overall.ci_low:+.3f}, {overall.ci_high:+.3f}]")
for s in m.subgroups:
if s.outcome != "aided_awareness":
continue
unread = (s.ci_high - s.ci_low) > abs(s.lift)
print(f"{s.subgroup_column}={s.subgroup_level:10} n={s.n_exposed}/{s.n_control} "
f"lift {s.lift:+.3f} [{s.ci_low:+.3f}, {s.ci_high:+.3f}]" + (" unread" if unread else ""))