Walkthrough
One analysis, start to finish.
This page runs the package on its bundled sample survey: 3,000 respondents, 51 columns, with exposure that depends on age, media habits and prior brand usage. The data are simulated, so the true effects are known. Every chart and number below comes from an actual run.
Install
The repository is a uv workspace. Cloning it gives you the library, the web app and the sample data.
git clone https://github.com/redam94/propensity-modeling.git
cd propensity-modeling
uv syncTo use only the library in another project, install it straight from the repository. It needs Python 3.10 or later.
pip install "git+https://github.com/redam94/propensity-modeling.git#subdirectory=packages/propensity_lift"Step 1 Write the question before opening the data.
The steps after this one are mechanical. This one is not, and it decides what the rest will mean. For the sample survey:
| Question | Among respondents who were exposed to the campaign, how much higher is each brand KPI than it would have been without exposure? |
| Estimand | ATT, the effect on the exposed. |
| Exposure | The group column, Exposed against Control. |
| How exposure happened | Younger respondents, heavy social and streaming users, and current brand users were more likely to be reached. Device and ad blocking mattered too. |
| Covariates | Demographics, media habits, prior brand usage and category purchase frequency. All describe the respondent before the campaign. |
| Left out on purpose | The attitude scales and the brand loyalty score. In a post-campaign survey the ad could have moved them. |
| Primary KPI | Aided awareness. The others are secondary. |
| Placebo check | unaided_awareness, which the simulation gives no true effect. Adjusted lift on it should be near zero. |
Step 2 Load and profile the data.
load_table reads Excel or CSV into a DataFrame, one row per respondent. profile_dataframe classifies every column and suggests a role for it.
from propensity_lift import load_table, profile_dataframe
df = load_table("sample_data/brand_lift_sample.xlsx")
# No file to hand? The same data: from propensity_lift import make_brand_lift_sample; df = make_brand_lift_sample(n=3000, seed=0)
profile = profile_dataframe(df, dataset_id="sample", filename="brand_lift_sample.xlsx")
for col in profile.columns:
print(col.name, col.kind, col.n_missing, col.suggested_role)The first rows and columns of df
The profile, all 51 columns
The profiler suggests model feature for the attitude scales and brand_loyalty_score because they look like covariates. It cannot know they were measured after exposure. Deciding what belongs in the model is your job. Which variables belong
Step 3 Turn the question into a config.
An AnalysisConfig holds every decision from step 1. Each block below maps to one line of the framing table.
from propensity_lift import AnalysisConfig
cfg = AnalysisConfig(
dataset_id="sample",
# Who is exposed and who is control
treatment_column="group",
exposed_values=["Exposed"],
control_values=["Control"],
# What is measured. Yes/No columns are read as proportions automatically;
# a text scale needs its positive levels named (here, top-two box).
outcome_columns=["aided_awareness", "unaided_awareness", "favorability",
"consideration", "purchase_intent", "ad_recall"],
outcome_positive_values={"purchase_intent": ["Definitely will", "Probably will"]},
# What drove exposure. Everything here was fixed before the campaign.
model_features=[
"age", "income_band", "education", "employment", "marital_status", "household_size",
"children_in_household", "urbanicity", "ethnicity", "household_income_k",
"tv_hours_week", "social_media_hours_day", "streaming_hours_week", "radio_hours_week",
"podcast_listener", "primary_device", "ad_blocker", "social_platforms_used",
"prior_brand_usage", "category_purchase_freq",
],
matching_features=["age_band", "gender", "region"], # the survey's quota cells
subgroup_columns=["gender"],
# How to adjust
estimand="ATT",
model_type="logistic",
methods=["nn_matching", "ipw", "overlap_weights", "stratification", "aipw"],
n_bootstrap=200,
)propensity_lift.sample_data.sample_analysis_config() builds the same config with four methods. It leaves out aipw, so pass methods=[...] to it to get all five. Options left at their defaults: a 0.2 caliper, 1:1 matching with replacement, no trimming, no weight clipping, alpha=0.05. Every option
Step 4 Run it.
from propensity_lift import run_analysis
result = run_analysis(df, cfg, progress=lambda frac, stage: print(f"{frac:4.0%} {stage}"))The run analysed 1,208 exposed and 1,792 control respondents. Read the warnings before anything else. They are the package telling you where the analysis departs from what you asked for.
print(result.sample)
print(result.warnings)
for m in result.methods:
print(m.label, m.warnings)Two of these matter. Matching left 123 exposed respondents without a control inside the caliper and its exact-match cells, so its estimate covers the rest. And overlap weights answer for the overlap population, not the exposed.
Step 5 Check overlap before any lift number.
If comparable respondents do not exist, nothing downstream can be trusted. Look at where the two groups sit on the propensity score.
The overlap coefficient of 0.71 is in the workable range, and nearly every exposed respondent sits inside the range the controls cover. The groups differ, visibly, but controls exist across the whole exposed distribution. The analysis can proceed without trimming.
Overlap is also reported feature by feature, which is where you find a level or a range that only one group has.
for f in sorted(result.feature_overlap, key=lambda f: f.overlap_coefficient)[:6]:
print(f.feature, f.overlap_coefficient, f.pct_exposed_unsupported, f.pct_control_unsupported)Step 6 Inspect the propensity model.
You are checking two things: that the scores behave like probabilities, and that the model has not separated the groups so sharply that overlap is gone. An AUC of 0.69 is unremarkable, which here is good news.
Calibration
Each dot is a tenth of the sample. On the diagonal, predicted and actual exposure rates agree.
What predicts exposure
Share of total importance, top ten features.
It tells you who the campaign reached. Use it to sanity-check the story from step 1: if a variable you expected to drive exposure is missing from the top, ask why. Do not report it as what drives awareness.
Step 7 Check balance, method by method.
Before adjustment, 19 variables differ between the groups by more than 0.1 standard deviations. The largest gap is 0.39. Each method should pull every dot to the left of the dashed line.
Weighting leaves nothing above the threshold. Matching leaves one variable just over it and keeps far fewer effective controls, which is why its intervals will be the widest. Overlap weights balance almost exactly, as they are built to.
This table covers the variables in the model. It cannot show imbalance on anything that was not measured. Why that matters
Step 8 Read the lift.
Now open the outcomes. For each KPI, compare the raw difference with what each method reports. A hollow marker means the interval includes zero.
What these results say
- Adjustment shrinks every KPI. Consideration goes from a raw +14.0 pp to +8.0 pp under IPW. The gap between them is the part of the raw difference that the measured covariates explain.
- The weighting methods agree. IPW, overlap weights, stratification and AIPW land within about a point of each other on most KPIs. That says the result does not depend on how the score is used. It says nothing about whether the covariates are sufficient.
- The placebo behaves. Unaided awareness shows a raw +3.6 pp that is pure confounding. After adjustment IPW puts it at −3.1 pp, the other weighting methods land between −2.6 pp and −3.4 pp, and all four intervals include zero.
- One false positive. Matching reports the same placebo as significant, p = 0.046. With six KPIs and five methods, one result at that level is expected by chance. It is also the method that dropped 123 exposed respondents.
The simulation sets its effects on a latent scale (log-odds), which does not convert to one fixed number of percentage points. Unaided awareness is the exception: its true effect is zero on any scale.
for m in result.methods:
for c in m.contrasts:
print(f"{m.method:16} {c.outcome:18} raw {c.raw_diff:+.3f} lift {c.lift:+.3f} "
f"[{c.ci_low:+.3f}, {c.ci_high:+.3f}] p={c.p_value:.3f} n_eff={c.effective_n_control:.0f}")Step 9 Break it out by subgroup, carefully.
Columns in subgroup_columns get their own contrasts, using the same weights or matches as the main analysis. A level needs at least 10 exposed and 10 control respondents to be reported.
The small group has an interval several times wider than the others. A subgroup difference is only worth reporting when the intervals clearly separate, and each extra breakout is another chance to find a pattern in noise. Decide which subgroups matter before you run.
ipw = next(m for m in result.methods if m.method == "ipw")
for s in ipw.subgroups:
print(s.subgroup_column, s.subgroup_level, s.outcome, f"{s.lift:+.3f}", s.n_exposed, s.n_control)Step 10 Report it so a reader can judge it.
A lift number without its assumptions invites the reader to treat it as an experiment. Put these alongside it:
- The question. Population, exposure definition, estimand.
- The sample. How many exposed and control, and how many were dropped, trimmed or left unmatched.
- What was adjusted for, and what you believe drove exposure but could not measure.
- Overlap and balance. The overlap coefficient, the largest standardised difference after adjustment, the effective number of controls.
- Every method you ran, not only the one you lead with.
- The placebo result, if you ran one.
The result object serialises to the same JSON the HTTP API returns, and the CLI runs the whole analysis from a config file.
from pathlib import Path
Path("result.json").write_text(result.model_dump_json(indent=2))
Path("cfg.json").write_text(cfg.model_dump_json(indent=2))uv run propensity-lift profile sample_data/brand_lift_sample.xlsx
uv run propensity-lift run sample_data/brand_lift_sample.xlsx --config cfg.json --out result.jsonThe same run in the web app.
The repository includes a FastAPI service and a React interface over the same library. Start both, open http://localhost:5173 and upload sample_data/brand_lift_sample.xlsx.
make dev-backend # API on http://localhost:8000, interactive docs at /docs
make dev-frontend # interface on http://localhost:5173| Page | What you do there | Step above |
|---|---|---|
| Upload | Drop the file and pick a sheet. The profile is computed on upload. | 2 |
| Configure | Assign a role to each column, choose exposed and control values, pick methods. | 3 |
| Runs | Watch progress. Several analyses can run at once. | 4 |
| Results | Tabs for Lift, Overlap, Balance, Model, Subgroups and Config, with export to Excel. | 5 to 10 |
The Results page opens on the Lift tab. Read Overlap and Balance first anyway.