Methods

Five ways to use one score.

Every method here starts from the same propensity scores and builds a comparison group from them. They differ in who ends up in the comparison, so they can differ in which question they answer.

The score is a summary, not a prediction.

The propensity score is the probability of being exposed given the covariates. Its useful property: among respondents with the same score, the measured covariates are balanced between exposed and control. So you can compare on one number where you would otherwise need to match on dozens.

Three model types.

logistic (the default) is L2-regularised on standardised inputs. gradient_boosting and random_forest are there for exposure rules a linear model cannot follow.

Tree scores are out of sample.

Boosting uses 5-fold cross-fitting and the forest uses out-of-bag predictions. In-sample tree scores are over-confident and make the groups look more separated than they are.

Features are prepared for you.

Categoricals are one-hot encoded with levels under 1% pooled. Numeric gaps are median-imputed with a missing-indicator column, so missingness itself gets balanced.

Judge the model by balance.

Pick the model type that leaves covariates balanced with a reasonable effective sample size. A flexible model that pushes scores to 0 and 1 has fitted exposure better and made the comparison worse.

Methods disagree when they answer for different people.

In this simulation the campaign works better on people who were likelier to see it. So the effect on the exposed (ATT), on everyone (ATE) and on the overlap population (ATO) are three different true numbers. Switch methods and watch which one each lands on.

Who is being compared

Bars show each group after adjustment. The dashed outline is the same group before.

What the estimate lands on

Weight by score

0effective controls
0effective exposed
0largest control weight, relative to average
0imbalance left on the covariate

When your methods disagree on real data, check this first. A gap between IPW and overlap weights often means the effect varies across people, and the two are reporting on different people.

The five methods, one at a time.

Set any combination in methods. They run on the same scores and report in the same shape, so results line up row for row.

Nearest-neighbour matching

How it works
Each exposed respondent is paired with the closest controls on the logit of the score. A control further away than the caliper is not used. Matching is exact within strata formed by the matching_features.
Answers for
The matched exposed respondents (ATT). With estimand="ATE" it matches in both directions.
Watch
Exposed respondents with no control inside the caliper are dropped and a warning says how many. The estimate then describes the matched exposed only.
OptionDefaultMeaning
caliper0.2Maximum distance, in standard deviations of the logit score. None turns it off.
match_ratio1Controls per exposed respondent, 1 to 10.
replacementTrueWhether a control can be matched more than once.
matching_features[]Columns to match exactly on. Numeric columns with more than 10 values are cut into quintiles.

Inverse probability weighting

How it works
Everyone stays in the sample and is reweighted. For the ATT, exposed respondents get weight 1 and controls get e / (1 − e). For the ATE, the weights are 1 / e and 1 / (1 − e). Weights are normalised within each group.
Answers for
The exposed (ATT) or everyone (ATE), as configured.
Watch
Scores near 0 or 1 produce very large weights. Check the effective sample size and the largest weight before trusting the interval.
OptionDefaultMeaning
estimand"ATT""ATT" or "ATE".
weight_clipNoneCap weights at this quantile within each group, for example 0.99. Off by default, because clipping trades variance for bias. The run warns when clipping moves an estimate by more than a quarter of a standard error.

Overlap weights

How it works
Exposed respondents get weight 1 − e and controls get e. Weights are bounded between 0 and 1 and are largest where the groups mix.
Answers for
The overlap population (ATO): people who could plausibly have been in either group. The result is filed under the configured estimand with a warning that the target is the ATO.
Watch
Balance is close to exact with a logistic score, which looks excellent. The population has changed, though. Say so when you report the number.
OptionDefaultMeaning
No options of its own. Clipping does not apply, because the weights cannot blow up.

Stratification

How it works
Respondents are grouped into bands of the score. For the ATT the bands are quantiles of the exposed group's scores. Controls are reweighted within each band to match the exposed share.
Answers for
The exposed (ATT) or everyone (ATE), as configured.
Watch
Within a band the groups can still differ, so some imbalance usually survives. More strata reduce it and leave fewer controls per band.
OptionDefaultMeaning
n_strata5Number of score bands, 2 to 20.

Doubly robust AIPW

How it works
An outcome model predicts each KPI from the model features, and the IPW weights then correct that model's residuals. Binary KPIs use a ridge logistic regression and scales use least squares, whatever model_type the propensity model uses.
Answers for
The same population as ipw. For the ATT, only the control outcome model is fitted.
Watch
The estimate stays consistent if either the propensity model or the outcome model is right. Both use the same covariates, so it gives no protection against a missing confounder.
OptionDefaultMeaning
estimand"ATT"Same as ipw.
weight_clipNoneSame weights and clipping as ipw.

Which to run.

Run more than one and report them together. Use this table to decide which to lead with.

Your situationLead withWhy
Standard brand lift, reasonable overlapipw and aipwThey keep every respondent and target the ATT directly. AIPW is usually the more precise of the two.
The sample was quota-matched on demographicsnn_matching with matching_featuresExact matching within the quota cells respects the design, with the score handling everything else.
Thin overlap, a few very large weightsoverlap_weights, or trimStable estimates for the comparable population. State that the population changed.
A small control groupipw or overlap_weightsMatching reuses the same few controls and discards the rest.
You need the effect for the whole audienceipw and aipw with estimand="ATE"Requires overlap in both directions, so check the low-score exposed respondents too.
You doubt the propensity model's functional formaipw, and try gradient_boostingThe outcome model gives a second chance at getting the adjustment right.

Trimming

Trimming removes respondents from the analysis before any method runs. Set it with trim.

RuleKeepsUse when
"none"Everyone. The default.Overlap is good, or the method handles thin regions itself.
"common_support"Scores inside the range both groups share.A few respondents sit outside the other group's range.
"crump"Scores from 0.1 to 0.9.Weights are unstable at the extremes.
Trimming exposed respondents changes the question.

Dropping controls who resemble no one exposed costs an ATT nothing. Dropping exposed respondents means the answer no longer covers everyone the campaign reached. The counts are in result.overlap.n_trimmed_exposed and n_trimmed_control; report them with the lift.

Balance

After adjustment, every covariate should have about the same distribution in both groups. The measure is the standardised mean difference: the gap between the exposed and control means, divided by a pooled standard deviation.

The package divides by the pooled standard deviation from before adjustment in both the before and after columns. That keeps the two comparable: a weighting scheme cannot make balance look better just by inflating the variance.

  • |SMD| above 0.1 is flagged as imbalanced. balance_summary.n_imbalanced_after counts them.
  • Variance ratios are reported for numeric covariates. Between 0.5 and 2 is acceptable.
  • Matching features always appear in the balance table, whether or not they are in the model.
  • There are no p-values. Balance is a property of this sample, and a significance test would mostly measure sample size.
for m in result.methods:
    s = m.balance_summary
    print(m.label, s.max_abs_smd_before, "->", s.max_abs_smd_after, "imbalanced:", s.n_imbalanced_after)
    worst = max(m.balance, key=lambda row: abs(row.smd_after))
    print("  worst:", worst.variable, round(worst.smd_after, 3))

See a real balance plot

Uncertainty

With n_bootstrap above zero (the default is 200), respondents are resampled within each group and the matching or weights are rebuilt on every resample. The interval is the lift plus or minus a normal quantile times the bootstrap standard error, and the p-value uses the same standard error, so the interval, the p-value and the significant flag always agree.

  • For weighting methods with a logistic model, each replicate resamples the full sample, refits the propensity model and reapplies the trim rule. The interval then includes the uncertainty in the scores.
  • For gradient boosting and random forest, refitting is too slow. Replicates resample the already-trimmed sample with the scores held fixed, and the run adds a warning that this tends to overstate the standard errors.
  • For matching, the scores are held fixed and the matching is redone in each replicate. The plain bootstrap is an approximation here, so treat borderline results from matching with extra caution.
  • For AIPW, the outcome models are refitted in every replicate.
  • With n_bootstrap=0, analytic standard errors are used and a warning notes that they ignore the estimation of the scores.

Each contrast records which was used in ci_method.

Six KPIs and five methods is thirty tests.

At a 5% threshold, one or two will cross it by chance. In the bundled sample data, a KPI with no true effect comes out “significant” under matching. Name a primary KPI and method beforehand, and read the rest as supporting evidence.

Reading a contrast

Each method returns one OutcomeContrast per KPI.

FieldMeaning
exposed_meanThe exposed group's mean. A proportion from 0 to 1 for binary KPIs.
control_mean_adjustedThe control mean after matching or weighting. This is the estimate of what the exposed group would have scored without the campaign.
liftExposed minus adjusted control. Multiply by 100 for percentage points on a binary KPI.
relative_liftLift as a percentage of the adjusted control mean.
raw_diffExposed minus unadjusted control. The gap between this and lift is what adjustment removed.
ci_low, ci_high, p_value, significantInterval and test for the absolute lift at the configured alpha.
effective_n_controlHow many equally weighted controls the adjusted control group is worth. A small number means a few respondents carry the estimate.

Every field Read real results