Dharmateja Priyadarshi Uddandarao, Senior Statistician – Data Scientist, Amazon
Executive summary
Inverse Probability Weighting (IPW), often implemented via propensity-score weighting, is a causal measurement framework for estimating incremental impact (e.g., lift in conversions or spend) from non-randomized marketing campaigns by reweighting observed data to mimic a randomized experiment.
IPW targets marginal (population-average) causal estimands such as the Average Treatment Effect (ATE) by constructing a “pseudo-population” in which treatment assignment is (approximately) independent of measured confounders.
Correct causal interpretation relies on assumptions that must be stated and interrogated, especially in targeted marketing:
(i) consistency/SUTVA (well-defined interventions and no interference),
(ii) conditional exchangeability (a.k.a. ignorability/unconfoundedness) given measured covariates, and
(iii) positivity/overlap (nonzero probability of receiving each treatment level for covariate patterns in the analysis set).
Diagnostics are central. In practice, credible IPW analyses emphasize (a) covariate balance after weighting, (b) weight stability (variance, maxima, effective sample size), and (c) overlap checks and potentially trimming/truncation when weights become extreme, recognizing a bias–variance trade-off.
A synthetic credit-card offer example below shows why this matters. In a realistic targeted campaign, the naïve (unadjusted) difference in conversion rates can materially overstate the incremental impact. After IPW balancing, the ATE estimate moves toward the simulated ground truth, and covariate imbalance (standardized mean differences) collapses toward zero.
Causal estimand and assumptions
Setup and notation
Consider a set of customers i=1,2,3…n. Let:
A – 0 to n : treatment indicator (e.g., received a credit-card marketing offer vs. not).
- Y: outcome (e.g., conversion in 60 days, incremental spend, net revenue).
- L: vector of measured pre-treatment covariates (demographics, credit risk, prior behavior, channel affinity, etc.).
Define potential outcomes Y(1) and Y(0): the outcome customer i would exhibit if treated or untreated, respectively. The ATE is:
In credit-card marketing, is rarely random: targeting often uses prior spend, engagement, risk, and expected response, exactly the variables that also predict outcomes, creating confounding if not addressed.
Identification assumptions
IPW identifies the ATE from observational data under a standard identification set:
Consistency / SUTVA. If Ai = a, then the observed outcome equals the corresponding potential outcome:
Yi SUTVA also requires (i) no interference (one customer’s outcome does not depend on others’ treatment) and (ii) no hidden “versions” of treatment that meaningfully differ without being represented in . In marketing, “versions” might include different channels, frequencies, creatives, or offer terms; if these vary, treatment must be redefined accordingly (e.g., multi-valued
).
Conditional exchangeability (ignorability / unconfoundedness).
Intuitively: after controlling for measured covariates , treatment assignment is as good as random. This is strong in marketing and unmeasured intent (e.g., imminent large purchase) or unlogged targeting rules can violate it.
Positivity (overlap). For covariate values with positive density in the target population,
This can fail in heavily targeted campaigns where some segments always (or never) receive offers, leading to extreme weights and unstable estimates.
When these assumptions are plausible, IPW provides a design-oriented route to causal measurement: specify and diagnose the weighting design (via balance and overlap) before focusing on outcome estimation, echoing the separation between “design” and “analysis” emphasized in propensity-score workflows.
IPW formulation and estimator derivation
Propensity score and balancing intuition
Define the propensity score:
A core result is that conditioning on e(L) can balance the distribution of measured covariates between treated and untreated groups; IPW uses this idea through weighting rather than matching or stratification.
IPW assigns each unit a weight inversely proportional to the probability of receiving its observed treatment. This upweights “surprising” assignments (e.g., low-propensity treated customers) and downweights “obvious” ones (e.g., high-propensity treated customers), constructing a pseudo-population that resembles one with randomized treatment.
This estimator family is conceptually related to classical inverse-probability ideas in survey sampling (the Horvitz–Thompson approach), which motivates the “weight by inverse assignment probability” logic.
Deriving the IPW estimand expressions
Assume conditional exchangeability and consistency. Then:
Taking expectations over yields:
Similarly,
Hence the ATE can be written:
Sample estimators
In practice, e(L) is unknown and is estimated as e^(L), often via logistic regression (or more flexible models), yielding estimated weights.
A common implementation estimates:
and
This “ratio” form (often called Hájek-style normalization) tends to be numerically stable in finite samples. [16]
Stabilized weights
Unstabilized ATE weights are:
Stabilized weights multiply by marginal treatment probabilities:
Stabilization is widely used in marginal structural model practice to reduce variance of weights without changing the estimand (noting that within-group weighted means are unchanged up to constants).
Diagnostics and practical implementation
A credible IPW analysis treats the propensity/weighting model as part of the study design. Diagnostics emphasize whether the weighted sample behaves like a randomized experiment with respect to measured covariates.
Flowchart
A[Define causal question and estimand: ATE] –> B[Choose pre-treatment confounders L]
B –> C[Estimate propensity score e(L)=P(A=1|L)]
C –> D[Compute IPTW / stabilized weights]
D –> E[Diagnose: covariate balance + overlap + weight stability]
E –> F[If needed: refine e(L), trim/truncate weights, restrict to overlap region]
F –> G[Estimate ATE using weighted means or weighted regression; compute SE/CI]
G –> H[Interpret results; assess sensitivity to unmeasured confounding]
Covariate balance
Balance is typically assessed by comparing covariate distributions between treatment groups after weighting. A widely used scalar diagnostic is the standardized mean difference (SMD) for each covariate :
with weighted analogs using weighted means and weighted variances. Rules of thumb often treat SMD <= 0.1 as acceptable balance, though context matters.
Balance diagnostics are essential because good predictive performance of the propensity model (e.g., high c-statistic) does not guarantee covariate balance.
Weight stability, effective sample size, and overlap
Because weights scale variance, it is important to inspect:
- Distribution of weights: mean, variance, maxima, tail behavior.
- Effective sample size (ESS): unequal weights reduce the information content relative to n; severe reductions can signal instability.
- Propensity score overlap: regions where e^(L) is near 0 or 1 produce extreme weights and unstable ATE estimation.
Limited overlap is not just a technical nuisance; it changes what can be learned from the data. Formal work shows that lack of overlap can make ATE estimators sensitive and imprecise; simple rules such as excluding observations with propensity scores outside can be a practical approximation to more formal overlap-based criteria in some settings.
Alternative weighting schemes (e.g., overlap weights) explicitly target the “overlap population,” producing bounded weights and often improved efficiency when overlap is limited at the cost of changing the target estimand away from the full-population ATE.
Trimming / truncation
A routine operational mitigation is to truncate weights at a high percentile (e.g., 99th) or cap them at a maximum value. This can reduce variance but may introduce bias and can implicitly change the effective target population (especially if extreme weights correspond to sparse covariate regions).
Simulation evidence also suggests trimming can sometimes improve accuracy/precision when propensity models are misspecified, but it is not a substitute for improving the propensity model itself; the “best” trimming level is context dependent.
Credit-card marketing campaign example with synthetic data
Scenario and data schema
Suppose a card issuer runs an observational campaign offering a product upgrade (e.g., premium rewards card). The marketing team targets customers predicted to respond: higher prior spend, higher digital engagement, better credit score, and lower delinquency risk. This “targeted nature of advertising” is a canonical driver of selection bias in observational measurement.
We create a realistic synthetic dataset (for illustration; not from a specific bank) with:
- Unit: customer account in a monthly snapshot.
- Treatment
: received offer (1) vs. not (0).
- Outcome
: conversion within 60 days (1/0).
- Covariates
(pre-treatment):
- age
- credit_score
- income_k (annual income, \$k)
- tenure_mo (months with issuer)
- prior_spend (monthly, \$)
- digital_eng (app sessions/month)
- delinq (recent delinquency indicator)
We then (i) estimate e^(L) using logistic regression, (ii) compute stabilized IPW weights, (iii) assess balance, and (iv) estimate the ATE as a weighted difference in conversion probabilities, with a robust standard error from a weighted regression (a marginal structural model with regressed on
using IPW).
Balance and effect estimates table
The table uses SMDs to summarize covariate imbalance before weighting and after stabilized IPW weighting (weighted SMD). It also reports the campaign ATE estimate.
| Metric | Unweighted | IPW-weighted (stabilized) |
| SMD(age) | -0.076 | +0.003 |
| SMD(credit_score) | +0.385 | +0.009 |
| SMD(income_k) | +0.017 | +0.002 |
| SMD(tenure_mo) | +0.179 | -0.008 |
| SMD(prior_spend) | +0.435 | -0.000 |
| SMD(digital_eng) | +0.112 | -0.012 |
| SMD(delinq) | -0.085 | +0.001 |
| Estimated ATE (conversion pp) | +3.72 (CI +2.36, +5.07) | +1.94 (CI +0.56, +3.32) |
Interpretation: the unweighted data show major imbalance on the key targeting variables (credit score and prior spend), making the naïve comparison non-causal. After IPW, covariates are balanced to near zero SMD, consistent with the pseudo-population logic.
Weight distribution chart
Stabilized weights had mean ~1.00, standard deviation ~0.38, and maximum ~5.33. An effective sample size calculation (Kish-style) for the whole sample was ~4,374 (vs. n = 5000), indicating moderate but not severe variance inflation from weighting.
Trimming illustration
Truncating stabilized weights at the 99th percentile (cap ≈ 2.39) produced a very similar ATE (+1.97 pp with a similar CI), which is consistent with good overlap in this synthetic setting and a relatively light right tail of weights. In real credit-card targeting, heavier tails are common when targeting rules are strict, and trimming can be more consequential.
Limitations and alternatives
Unmeasured confounding remains the central risk
IPW only adjusts for measured covariates . If marketing assignment depends on unmeasured intent, channel inventory constraints, or agent discretion that also affects outcomes, conditional exchangeability fails and IPW can be biased even when balance on observed
is perfect.
This is not a theoretical edge case in ad/marketing measurement. Large-scale industry evidence comparing randomized experiments to observational estimators shows that observational methods can fail to recover experimental effects even with extensive behavioral and demographic controls, emphasizing the practical difficulty of observational marketing measurement.
Sensitivity analysis frameworks (often associated with Rosenbaum’s work) are commonly recommended to quantify how strong hidden bias would need to be to materially change conclusions.
Model misspecification and propensity estimation
Classic IPTW relies on a correctly specified propensity model for e^(L). Misspecification can yield residual imbalance and biased ATE estimates, and it can also create extreme weights. This is why balance diagnostics, not just propensity model fit, are central.
Flexible propensity models (trees, boosting, ensembles) can improve balance but require the same diagnostic discipline; different propensity estimators interact differently with trimming and stability.
Positivity violations and changing estimands
When overlap is limited, the ATE for the full population may simply be poorly supported by observed data. In these cases, restricting to an overlap region or switching estimands (e.g., overlap weights) can yield more stable and relevant estimates, while explicitly changing the causal question to “effect for customers with genuine treatment ambiguity.”
Trimming is a bias–variance trade-off, not a free fix
Weight truncation/trim rules can reduce variance but may introduce bias if the trimmed observations correspond to covariate regions with different treatment effects or if trimming effectively excludes certain subpopulations. The method is best treated as part of a transparent robustness analysis rather than a default “repair.”
Practical marketing constraints: small effects and noisy outcomes
Even with correct causal design, marketing incrementality is often small relative to outcome variance. Large field-experiment evidence highlights how wide confidence intervals can remain even with substantial samples, and how selection bias can cripple observational approaches when targeting is strong.
References
Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424. https://doi.org/10.1080/00273171.2011.568786
Cole, S. R., & Hernán, M. A. (2008). Constructing inverse probability weights for marginal structural models. American Journal of Epidemiology, 168(6), 656–664. https://doi.org/10.1093/aje/kwn164
Crump, R. K., Hotz, V. J., Imbens, G. W., & Mitnik, O. A. (2009). Dealing with limited overlap in estimation of average treatment effects. Biometrika, 96(1), 187–199. https://doi.org/10.1093/biomet/asn055
Gordon, B. R., Zettelmeyer, F., Bhargava, N., & Chapsky, D. (2019). A comparison of approaches to advertising measurement: Evidence from big field experiments at Facebook. Marketing Science, 38(2), 193–225.
Hernán, M. A., & Robins, J. M. (2020). Causal inference: What if. Chapman & Hall/CRC.
Horvitz, D. G., & Thompson, D. J. (1952). A generalization of sampling without replacement from a finite universe. Journal of the American Statistical Association, 47(260), 663–685.
Imbens, G. W. (2004). Nonparametric estimation of average treatment effects under exogeneity: A review. Review of Economics and Statistics, 86(1), 4–29.
Lee, B. K., Lessler, J., & Stuart, E. A. (2011). Weight trimming and propensity score weighting. PLoS ONE, 6(3), e18174.
Lewis, R. A., & Rao, J. M. (2015). The unfavorable economics of measuring the returns to advertising. Quarterly Journal of Economics, 130(4), 1941–1973.
Li, F., Morgan, K. L., & Zaslavsky, A. M. (2018). Balancing covariates via propensity score weighting. Journal of the American Statistical Association, 113(521), 390–400.
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
Rosenbaum, P. R. (2002). Observational studies (2nd ed.). Springer.