Walkthrough

One analysis, start to finish.

This page runs the package on its bundled sample survey: 3,000 respondents, 51 columns, with exposure that depends on age, media habits and prior brand usage. The data are simulated, so the true effects are known. Every chart and number below comes from an actual run.

Install

The repository is a uv workspace. Cloning it gives you the library, the web app and the sample data.

git clone https://github.com/redam94/propensity-modeling.git
cd propensity-modeling
uv sync

To use only the library in another project, install it straight from the repository. It needs Python 3.10 or later.

pip install "git+https://github.com/redam94/propensity-modeling.git#subdirectory=packages/propensity_lift"

Step 1 Write the question before opening the data.

The steps after this one are mechanical. This one is not, and it decides what the rest will mean. For the sample survey:

QuestionAmong respondents who were exposed to the campaign, how much higher is each brand KPI than it would have been without exposure?
EstimandATT, the effect on the exposed.
ExposureThe group column, Exposed against Control.
How exposure happenedYounger respondents, heavy social and streaming users, and current brand users were more likely to be reached. Device and ad blocking mattered too.
CovariatesDemographics, media habits, prior brand usage and category purchase frequency. All describe the respondent before the campaign.
Left out on purposeThe attitude scales and the brand loyalty score. In a post-campaign survey the ad could have moved them.
Primary KPIAided awareness. The others are secondary.
Placebo checkunaided_awareness, which the simulation gives no true effect. Adjusted lift on it should be near zero.

How to frame your own

Step 2 Load and profile the data.

load_table reads Excel or CSV into a DataFrame, one row per respondent. profile_dataframe classifies every column and suggests a role for it.

from propensity_lift import load_table, profile_dataframe

df = load_table("sample_data/brand_lift_sample.xlsx")
# No file to hand? The same data: from propensity_lift import make_brand_lift_sample; df = make_brand_lift_sample(n=3000, seed=0)

profile = profile_dataframe(df, dataset_id="sample", filename="brand_lift_sample.xlsx")
for col in profile.columns:
    print(col.name, col.kind, col.n_missing, col.suggested_role)

The first rows and columns of df

The profile, all 51 columns

The suggested role is a guess about a column's shape.

The profiler suggests model feature for the attitude scales and brand_loyalty_score because they look like covariates. It cannot know they were measured after exposure. Deciding what belongs in the model is your job. Which variables belong

Step 3 Turn the question into a config.

An AnalysisConfig holds every decision from step 1. Each block below maps to one line of the framing table.

from propensity_lift import AnalysisConfig

cfg = AnalysisConfig(
    dataset_id="sample",

    # Who is exposed and who is control
    treatment_column="group",
    exposed_values=["Exposed"],
    control_values=["Control"],

    # What is measured. Yes/No columns are read as proportions automatically;
    # a text scale needs its positive levels named (here, top-two box).
    outcome_columns=["aided_awareness", "unaided_awareness", "favorability",
                     "consideration", "purchase_intent", "ad_recall"],
    outcome_positive_values={"purchase_intent": ["Definitely will", "Probably will"]},

    # What drove exposure. Everything here was fixed before the campaign.
    model_features=[
        "age", "income_band", "education", "employment", "marital_status", "household_size",
        "children_in_household", "urbanicity", "ethnicity", "household_income_k",
        "tv_hours_week", "social_media_hours_day", "streaming_hours_week", "radio_hours_week",
        "podcast_listener", "primary_device", "ad_blocker", "social_platforms_used",
        "prior_brand_usage", "category_purchase_freq",
    ],
    matching_features=["age_band", "gender", "region"],   # the survey's quota cells
    subgroup_columns=["gender"],

    # How to adjust
    estimand="ATT",
    model_type="logistic",
    methods=["nn_matching", "ipw", "overlap_weights", "stratification", "aipw"],
    n_bootstrap=200,
)

propensity_lift.sample_data.sample_analysis_config() builds the same config with four methods. It leaves out aipw, so pass methods=[...] to it to get all five. Options left at their defaults: a 0.2 caliper, 1:1 matching with replacement, no trimming, no weight clipping, alpha=0.05. Every option

Step 4 Run it.

from propensity_lift import run_analysis

result = run_analysis(df, cfg, progress=lambda frac, stage: print(f"{frac:4.0%} {stage}"))
Output

        

The run analysed 1,208 exposed and 1,792 control respondents. Read the warnings before anything else. They are the package telling you where the analysis departs from what you asked for.

print(result.sample)
print(result.warnings)
for m in result.methods:
    print(m.label, m.warnings)

Two of these matter. Matching left 123 exposed respondents without a control inside the caliper and its exact-match cells, so its estimate covers the rest. And overlap weights answer for the overlap population, not the exposed.

Step 5 Check overlap before any lift number.

If comparable respondents do not exist, nothing downstream can be trusted. Look at where the two groups sit on the propensity score.

0.71overlap coefficient
0.69model AUC
99.6%of exposed inside common support
5exposed above every control

The overlap coefficient of 0.71 is in the workable range, and nearly every exposed respondent sits inside the range the controls cover. The groups differ, visibly, but controls exist across the whole exposed distribution. The analysis can proceed without trimming.

Overlap is also reported feature by feature, which is where you find a level or a range that only one group has.

for f in sorted(result.feature_overlap, key=lambda f: f.overlap_coefficient)[:6]:
    print(f.feature, f.overlap_coefficient, f.pct_exposed_unsupported, f.pct_control_unsupported)

What overlap is and why AUC misleads

Step 6 Inspect the propensity model.

You are checking two things: that the scores behave like probabilities, and that the model has not separated the groups so sharply that overlap is gone. An AUC of 0.69 is unremarkable, which here is good news.

Calibration

Each dot is a tenth of the sample. On the diagonal, predicted and actual exposure rates agree.

What predicts exposure

Share of total importance, top ten features.

The importance chart describes exposure, not the outcome.

It tells you who the campaign reached. Use it to sanity-check the story from step 1: if a variable you expected to drive exposure is missing from the top, ask why. Do not report it as what drives awareness.

Step 7 Check balance, method by method.

Before adjustment, 19 variables differ between the groups by more than 0.1 standard deviations. The largest gap is 0.39. Each method should pull every dot to the left of the dashed line.

Weighting leaves nothing above the threshold. Matching leaves one variable just over it and keeps far fewer effective controls, which is why its intervals will be the widest. Overlap weights balance almost exactly, as they are built to.

This table covers the variables in the model. It cannot show imbalance on anything that was not measured. Why that matters

Step 8 Read the lift.

Now open the outcomes. For each KPI, compare the raw difference with what each method reports. A hollow marker means the interval includes zero.

What these results say

  • Adjustment shrinks every KPI. Consideration goes from a raw +14.0 pp to +8.0 pp under IPW. The gap between them is the part of the raw difference that the measured covariates explain.
  • The weighting methods agree. IPW, overlap weights, stratification and AIPW land within about a point of each other on most KPIs. That says the result does not depend on how the score is used. It says nothing about whether the covariates are sufficient.
  • The placebo behaves. Unaided awareness shows a raw +3.6 pp that is pure confounding. After adjustment IPW puts it at −3.1 pp, the other weighting methods land between −2.6 pp and −3.4 pp, and all four intervals include zero.
  • One false positive. Matching reports the same placebo as significant, p = 0.046. With six KPIs and five methods, one result at that level is expected by chance. It is also the method that dropped 123 exposed respondents.
Why there is no “true effect” line on most KPIs.

The simulation sets its effects on a latent scale (log-odds), which does not convert to one fixed number of percentage points. Unaided awareness is the exception: its true effect is zero on any scale.

for m in result.methods:
    for c in m.contrasts:
        print(f"{m.method:16} {c.outcome:18} raw {c.raw_diff:+.3f}  lift {c.lift:+.3f} "
              f"[{c.ci_low:+.3f}, {c.ci_high:+.3f}]  p={c.p_value:.3f}  n_eff={c.effective_n_control:.0f}")

Step 9 Break it out by subgroup, carefully.

Columns in subgroup_columns get their own contrasts, using the same weights or matches as the main analysis. A level needs at least 10 exposed and 10 control respondents to be reported.

The small group has an interval several times wider than the others. A subgroup difference is only worth reporting when the intervals clearly separate, and each extra breakout is another chance to find a pattern in noise. Decide which subgroups matter before you run.

ipw = next(m for m in result.methods if m.method == "ipw")
for s in ipw.subgroups:
    print(s.subgroup_column, s.subgroup_level, s.outcome, f"{s.lift:+.3f}", s.n_exposed, s.n_control)

Step 10 Report it so a reader can judge it.

A lift number without its assumptions invites the reader to treat it as an experiment. Put these alongside it:

  • The question. Population, exposure definition, estimand.
  • The sample. How many exposed and control, and how many were dropped, trimmed or left unmatched.
  • What was adjusted for, and what you believe drove exposure but could not measure.
  • Overlap and balance. The overlap coefficient, the largest standardised difference after adjustment, the effective number of controls.
  • Every method you ran, not only the one you lead with.
  • The placebo result, if you ran one.

The result object serialises to the same JSON the HTTP API returns, and the CLI runs the whole analysis from a config file.

from pathlib import Path

Path("result.json").write_text(result.model_dump_json(indent=2))
Path("cfg.json").write_text(cfg.model_dump_json(indent=2))
uv run propensity-lift profile sample_data/brand_lift_sample.xlsx
uv run propensity-lift run sample_data/brand_lift_sample.xlsx --config cfg.json --out result.json

The same run in the web app.

The repository includes a FastAPI service and a React interface over the same library. Start both, open http://localhost:5173 and upload sample_data/brand_lift_sample.xlsx.

make dev-backend     # API on http://localhost:8000, interactive docs at /docs
make dev-frontend    # interface on http://localhost:5173
PageWhat you do thereStep above
UploadDrop the file and pick a sheet. The profile is computed on upload.2
ConfigureAssign a role to each column, choose exposed and control values, pick methods.3
RunsWatch progress. Several analyses can run at once.4
ResultsTabs for Lift, Overlap, Balance, Model, Subgroups and Config, with export to Excel.5 to 10

The Results page opens on the Lift tab. Read Overlap and Balance first anyway.