Dharmateja Priyadarshi Uddandarao, Senior Statistician , Amazon
Executive summary
Operational cybersecurity teams increasingly want causal answers when choosing between incident response strategies: Is the new containment playbook genuinely effective in reducing time-to-contain, or is it primarily utilized for less complex incidents? Real incident response is usually not fully randomized. High-severity incidents, certain attack vectors, and on-call load all influence which protocol gets chosen, and those same factors also influence outcomes. That creates confounding.
Propensity score prebalancing means you set up your study by making sure the groups you are comparing look similar based on certain factors before you look at the results. The propensity score, which is the chance of getting a specific treatment based on what we already know, helps create balance by using methods like trimming, matching, subclassification, or weighting.
Unlike many “run-a-regression-and-hope” workflows, prebalancing insists on explicit (and auditable) balance diagnostics (e.g., standardized mean differences) and on keeping the balancing stage blind to outcomes to protect objectivity.
This article explains the basic idea behind prebalancing, how it is different from regular propensity score methods and other ways to find cause-and-effect relationships, and how to use it when responding to incidents that have changing factors, limited randomization, shifting priorities, and a realistic incident response. An A/B or quasi-experiment example is provided with simulated data showing how prebalancing can dramatically reduce bias in estimated treatment effects (from “the new playbook looks much worse” to “it’s close to the true improvement,” under stated assumptions).
Propensity score prebalancing in plain English
What the propensity score is
The propensity score is the conditional probability that a unit receives the treatment, given observed covariates. In simpler terms, Paul R. Rosenbaum and Donald B. Rubin describe the propensity score as the chance of getting a specific treatment based on a set of known factors, and it helps ensure that treated and control groups are similar in those factors.
In cyber incident response, the “unit” could be an incident ticket (or incident episode), and the “treatment” could be a response protocol e.g., Protocol B = “rapid containment playbook with automated host isolation,” versus Protocol A = “standard manual triage.” The propensity score is then the probability that an incident gets Protocol B, conditional on what you knew before selecting the protocol (initial severity, asset criticality, alert source, on-call load, etc.).
What “prebalancing” means
“Prebalancing” is best understood as a design discipline rather than a single estimator:
- Design stage (prebalancing/preprocessing): create a dataset for analysis where the treatment and control groups are similar based on the characteristics measured before treatment, like using the propensity score to help match and weight. This stage should be done without using outcome data to choose models or tune decisions, echoing the principle that “design should be separated from analysis.”
- Analysis stage: estimate the causal effect in the prebalanced sample (often with a simple estimator, possibly with additional outcome modeling or robust variance).
The point is not that propensity methods are new. The novelty is the workflow guarantee: before you look at “time-to-contain,” you commit to a design that (a) checks covariate balance and (b) restricts inference to regions of covariate overlap where comparisons are defensible.
The intuition: “Make observational data look like a randomized drill.”
A clean mental model is, “If I could rerun history, how might I approximate a randomized trial of two playbooks?” Prebalancing tries to reconstruct the assignment mechanism, like why incidents got Protocol A vs. B, using only pre-treatment information, to remove systematic differences in the covariates that drove that assignment.
This philosophy is aligned with Donald B. Rubin’s argument that objective causal inference should aim to replicate the appealing features of randomized experiments by separating design from analysis and avoiding outcome-driven design choices.
Comparing causal approaches for cyber incident strategy
How prebalancing differs from “standard” propensity workflows
In practice, many teams “do propensity scores” as a black box:
- fit a propensity model (often a logistic regression),
- weight/match
- Run an effect estimate
- Move on
Prebalancing is stricter: it treats the propensity model as acceptable only if it balances covariates. That is, the goal of the propensity model is not prediction accuracy; it is balance. The research clearly states that even small mistakes in the propensity model can lead to significant bias, which encourages methods that focus on achieving balance instead of just fitting the data well.
A related family of approaches, such as covariate balancing propensity scores (CBPS) by Kosuke Imai and Marc Ratkovic builds balance into the estimation of the propensity score itself.
Similarly, entropy balancing is a method that adjusts weights before analysis to ensure that certain balance requirements for covariate moments are met, and it is specifically designed to lessen reliance
A comparison table (methods, assumptions, and best cyber-IR fit)
Comparison of causal methods for operational cyber incident response
A few interpretation notes for cyber operations:
- Incident response “A/B tests” often aren’t pure A/B. Even if you intend randomization, real-time escalation rules, staffing, and safety overrides create deviations. Prebalancing is useful both in quasi-experiments and in “messy” experiments where randomization goes awry.
- Time-varying decisions are common. If “treatment” is a sequence (escalate, isolate, rotate credentials, engage forensics) instead of a single binary choice, then baseline propensity scores are insufficient. Marginal structural models were created to handle treatments that change over time and are influenced by earlier treatments.
- Instrumental variables are rare but not impossible in cyber. They require a credible instrument that shifts protocol choice without directly affecting outcomes except through the protocol, an assumption that is often harder to defend than it first appears.
How to implement prebalancing in operational cyber incident experiments
This section assumes you have an incident tracking system (tickets/cases), telemetry (SIEM/EDR), and a definable set of response protocols. Given the absence of a specified dataset, the following details are based on assumptions about the records your organization maintains and the potential for operational modifications.
Define the unit, treatment, and outcome like an incident handler would
Definitions that align with operational reality determine the success or failure of a prebalancing project.
Unit (what gets “treated”):
The response team typically handles a single incident ticket or episode. In complex breaches, you may need to define the unit as a “case episode” bounded by a start time and a containment milestone to avoid mixing multiple related events.
Treatment (response protocol):
Define treatment as a policy applied within a fixed decision window, e.g.:
- Protocol B = “Automated containment in first 15 minutes: immediate host isolation + scripted credential reset + parallel triage,”
- Protocol A = “Standard manual triage in the first 15 minutes.”
This “decision window” matters because it anchors what counts as pre-treatment covariates versus post-treatment mediators.
Outcome (measurable and aligned with guidance):
Operationally meaningful outcomes often include time-to-containment, time-to-eradication, service restoration time, recurrence, or business impact categories. The National Institute of Standards and Technology’s incident lifecycle highlights stages (preparation, detection & analysis, containment/eradication/recovery, and post-incident activity), which directly relate to time outcomes (for example, “when containment
For recovery/resilience framing and metrics, NIST’s recovery guidance emphasizes planning, playbooks, testing, and improvement, and explicitly notes metrics and realistic test scenarios as part of resilience improvement.
Data requirements: what covariates you need and where they come from
Prebalancing only addresses observed confounding. That makes the covariate set a first-class design artifact.
A practical starting point is the incident data elements recommended in the Computer Security Incident Handling Guide, which calls for standardized incident-related data elements including timestamps (start, detected, reported, resolved), incident category/vectors, affected resources, prioritization factors, and actions performed.
For propensity prebalancing in IR strategy experiments, covariates usually include
- The initial severity/priority signal, which is available at or before protocol choice, includes factors such as the triage score, the business criticality of the affected asset, and scope indicators.
- The initial evidence strength (false positive likelihood) and attack surface/detection channel (EDR vs. SIEM correlation vs. user report) are also considered.
- Consider operational load and context, including on-call shifts, open incident queues, and measurable staffing levels before making a choice. NIST emphasizes that real-time availability matters because delays increase potential damage, which makes operational load a plausible confounder in practice.
- System state/environment controls at baseline (network segmentation, backup posture, whether critical data is in immutable storage), if they influence both playbook choice and outcomes.
The timing of when covariates are measured distinguishes between a “confounder” and a “mediator.”
A common mistake in incident analytics is mistakenly adjusting for variables that occur because of the playbook (after treatment), which can prevent us from seeing the full causal effect.
Prebalancing solves this mainly through discipline:
- Freeze covariates at incident report time or at a clearly defined “protocol selection time.”
- Treat subsequent information (e.g., deeper forensic findings) as post-treatment unless it was genuinely available before protocol selection.
Randomization constraints: where experiments break in security operations
True randomization is ideal but often constrained:
- You should ethically/randomly provide a known “best” playbook for high-severity incidents.
- Playbooks may be gated by regulatory constraints or by required escalation procedures.
- Teams often override the assignment for safety.
A pragmatic pattern is constrained experimentation: randomize only within an eligibility region (e.g., moderate severity incidents or incidents matching a particular vector), then use prebalancing to improve comparability within that region and to adjust for residual imbalances and overrides.
For organizations that can randomize more broadly, the experimentation literature emphasizes trustworthiness, choosing metrics carefully, validating assumptions, and avoiding pitfalls like carryover effects.
Handling time-varying confounders and multi-stage “treatments”
Incident response is inherently sequential: early containment actions affect later system state, which feeds back into subsequent decisions. When factors change over time and are affected by earlier actions, standard adjustment methods can be confusing; marginal structural models (MSMs) were designed for this situation and are calculated by using inverse-probability weighting of treatment histories.
Operational translation:
- If your “treatment” is a policy sequence (“isolate host at t0,” “rotate creds at t1,” “engage IR retainer at t2”), define a treatment regime and estimate weights at each decision point using the covariate history available at that point.
- If you collapse the treatment into a single binary label, you risk conditioning on post-treatment variables or missing the causal structure.
A minimal prebalancing workflow you can audit
A defensible, auditable workflow is
- Pre-register your estimand and eligibility. Decide whether you want ATE (average effect across eligible incidents) or ATT (effect among incidents that received the new protocol).
- Fit an initial propensity model for protocol assignment using only pre-treatment covariates.
- Assess balance using standardized differences and distribution checks; avoid relying on propensity-score-only diagnostics (which can be uninformative).
- Iterate the design (still outcome-blind): add nonlinear terms/interactions or use balancing-focused methods (e.g., CBPS; entropy balancing) until covariate balance is acceptable.
- Enforce overlap/positivity by trimming extreme propensities when necessary; limited overlap can otherwise inflate variance and sensitivity.
- Estimate the effect in the prebalanced sample and report uncertainty with operational clustering in mind (e.g., by team/shift, if relevant).
- The final section discusses sensitivity analysis and its limitations.
A realistic incident-response experiment with simulation
Scenario and assumptions
Assume your SOC introduces Protocol B: a structured rapid containment playbook that triggers automated isolation and parallel evidence collection in the first 15 minutes. Protocol A is the existing manual triage approach.
In practice, Protocol B is usually chosen more for serious incidents and during night shifts because responders believe that automation helps contain the situation faster. Those same factors also increase time-to-containment. This is the confounding problem in one sentence: “The playbook you choose depends on the kind of incident you got.”
We simulate:
- pre-treatment covariates: severity score, asset criticality, shift (night), on-call workload, alert source,
- treatment assignment: a nonlinear function of those covariates (including severity² and severity×shift),
- The outcome is time-to-contain (in hours), with Protocol B improving this metric by an average of 2 hours.
What “prebalancing” looks like in the simulation
We compare:
- Unadjusted difference in means (what you get from a dashboard),
- IPTW with a naive propensity model (misses key nonlinearities/interactions),
- IPTW uses a prebalanced propensity model that includes nonlinearities and interactions, which were chosen because they balance covariates.
This reflects the practical lesson in the balancing literature: model misspecification can produce biased estimates, and balance diagnostics (not a c-statistic) should guide correctness.
Simulation results
In this simulation (600 runs, 2000 incidents per run), the true effect is −2.0 hours (Protocol B is faster). The unadjusted estimator is severely biased and suggests Protocol B is worse, which is exactly the kind of strategic mistake that confounding enables.
Simulation summary table
| Estimator | Mean Estimate | Std Dev | Bias | RMSE |
|---|---|---|---|---|
| Unadjusted | 1.3500 | 0.1992 | 3.3500 | 3.3559 |
| IPTW (Naive PS) | -0.9453 | 0.3604 | 1.0547 | 1.1144 |
| IPTW (Prebalanced PS) | -1.9637 | 0.4441 | 0.0363 | 0.4452 |
Key takeaways:
- Without prebalancing, IPTW improves on the dashboard estimate but remains meaningfully biased (about +1 hour bias in this scenario).
- With prebalancing, bias is near zero, but variance increases which is an expected tradeoff when you stop extrapolating into regions with poor overlap and instead focus on comparable incidents.
Covariate balance (one representative simulated dataset)
Prebalancing is not magic; it is verified. Below is a representative dataset showing how prebalancing improves standardized mean differences (SMDs) between Protocol A and B groups.
Covariate balance before and after IPTW
Balance diagnostics are a core part of correct propensity workflows: you validate that treated and control have comparable covariate distributions after adjustment, rather than assuming they do.
Simulation code (Python)
The code below reproduces the simulation logic and generates
(i) a results summary table and
(ii) a matplotlib visualization. It is intentionally self-contained and uses only NumPy and Matplotlib.
import numpy as np
import matplotlib.pyplot as plt
def sigmoid(x):
return 1.0 / (1.0 + np.exp(-x))
def simulate_dataset(n=2000, seed=0):
rng = np.random.default_rng(seed)
# Covariates (pretreatment)
severity = rng.beta(2, 2, size=n) # 0..1
asset_crit = rng.integers(1, 6, size=n) # 1..5
shift_night = rng.binomial(1, 0.25, size=n) # 0/1
workload = rng.poisson(2, size=n) # count
source = rng.choice(4, size=n, p=[0.35, 0.35, 0.20, 0.10]) # 0=EDR,1=SIEM,2=User,3=Cloud
# One-hot-ish helpers
src_edr = (source == 0).astype(float)
src_user = (source == 2).astype(float)
src_cloud = (source == 3).astype(float)
# Treatment assignment: Protocol B (T=1)
# True propensity includes severity^2 and severity*shift interaction.
intercept = -4.5
lp_true = (
intercept
+ 6.5 * severity
+ 1.4 * severity**2
+ 0.7 * (asset_crit - 3)
+ 0.3 * workload
+ 0.9 * shift_night
+ 2.0 * severity * shift_night
- 0.6 * src_edr
+ 0.25 * src_cloud
)
ps_true = sigmoid(lp_true)
T = rng.binomial(1, ps_true, size=n)
# Outcome: time-to-contain (hours)
tau_true = -2.0
noise = rng.normal(0, 3.0, size=n)
Y = (
10
+ 12.0 * severity
+ 1.3 * (asset_crit - 3)**2
+ 0.5 * workload
+ 1.5 * shift_night
+ 1.0 * src_user
+ tau_true * T
+ noise
)
return {
"Y": Y, "T": T, "tau_true": tau_true,
"severity": severity, "asset_crit": asset_crit,
"shift_night": shift_night, "workload": workload,
"src_edr": src_edr, "src_cloud": src_cloud
}
def ate_unadjusted(Y, T):
return Y[T==1].mean() - Y[T==0].mean()
def ate_iptw(Y, T, ps, trim=(0.01, 0.99)):
lo, hi = trim
mask = (ps > lo) & (ps < hi)
Y, T, ps = Y[mask], T[mask], ps[mask]
p_t = T.mean() # stabilized weights
w = np.where(T==1, p_t/ps, (1-p_t)/(1-ps))
mu1 = (w*T*Y).sum() / (w*T).sum()
mu0 = (w*(1-T)*Y).sum() / (w*(1-T)).sum()
return mu1 - mu0
def ps_naive(data):
# Naive propensity model omits severity^2 and severity*shift interaction
intercept = -4.5
lp = (
intercept
+ 6.5 * data["severity"]
+ 0.7 * (data["asset_crit"] - 3)
+ 0.3 * data["workload"]
+ 0.9 * data["shift_night"]
- 0.6 * data["src_edr"]
+ 0.25 * data["src_cloud"]
)
return sigmoid(lp)
def ps_prebalanced(data):
# Prebalanced model includes the nonlinearities/interactions that restore balance
intercept = -4.5
sev = data["severity"]
lp = (
intercept
+ 6.5 * sev
+ 1.4 * sev**2
+ 0.7 * (data["asset_crit"] - 3)
+ 0.3 * data["workload"]
+ 0.9 * data["shift_night"]
+ 2.0 * sev * data["shift_night"]
- 0.6 * data["src_edr"]
+ 0.25 * data["src_cloud"]
)
return sigmoid(lp)
# --- Run simulation ---
M = 600
n = 2000
ests = {"Unadjusted": [], "IPTW (naive PS)": [], "IPTW (prebalanced PS)": []}
tau_true = None
for m in range(M):
d = simulate_dataset(n=n, seed=10_000 + m)
Y, T = d["Y"], d["T"]
tau_true = d["tau_true"]
ests["Unadjusted"].append(ate_unadjusted(Y, T))
ests["IPTW (naive PS)"].append(ate_iptw(Y, T, ps_naive(d)))
ests["IPTW (prebalanced PS)"].append(ate_iptw(Y, T, ps_prebalanced(d)))
# Summary
print(f"M={M}, n={n}, tau_true={tau_true}")
for k, v in ests.items():
v = np.array(v)
bias = v.mean() - tau_true
rmse = np.sqrt(np.mean((v - tau_true)**2))
print(f"{k:22s} mean={v.mean(): .4f} sd={v.std(ddof=1): .4f} bias={bias:+.4f} rmse={rmse:.4f}")
# Visualization: distribution of estimated effects
labels = list(ests.keys())
values = [np.array(ests[k]) for k in labels]
plt.figure(figsize=(10, 5))
plt.boxplot(values, labels=labels, showfliers=False)
plt.axhline(tau_true, linestyle="--")
plt.ylabel("Estimated treatment effect (hours)")
plt.title("Incident response simulation: effect estimates with and without prebalancing")
plt.xticks(rotation=15, ha="right")
plt.tight_layout()
plt.show()Assumptions, limitations, and failure modes
The big identification assumptions (and why cyber ops stress them)
- No unmeasured confounding (conditional exchangeability):
Propensity methods assume that, after conditioning on observed covariates, treatment assignment is independent of potential outcomes. In practice, this means you captured all variables that simultaneously influence playbook choice and outcomes.
In cyber IR, plausible “missing confounders” include responder expertise, hidden incident complexity, or business pressure that is not logged. - Positivity/overlap:
Every covariate profile of interest must have a nonzero probability of receiving each protocol. In operational terms, if Protocol B is never used for certain incident types, you cannot credibly estimate its effect there without extrapolation. Lack of overlap is known to inflate variance and sensitivity, motivating trimming or alternative estimands. - SUTVA / no interference:
Incident response can violate the principle of “no interference”; the outcome of one incident can be affected by other incidents due to shared responders and tooling bottlenecks. The original propensity-score framework discusses stable unit treatment values as a foundational assumption.
In practice, consider cluster designs (randomize by shift or team) or include workload-based interference modeling, but recognize this is a hard problem.
Operational limitations and how to be honest about them
- Telemetry quality and definitional drift: NIST recommends standard incident data elements and emphasizes consistent incident handling; if logging is inconsistent across teams or time, your covariates and outcomes will not be reliable.
- Outcome-driven protocol selection: If analysts intensify response after seeing partial outcomes (e.g., early containment failure triggers escalation), then “treatment” becomes time-varying, and baseline propensity scores can be biased. MSM-style approaches (or explicit multi-stage modeling) are required.
- External validity after trimming: Overlap enforcement can change the population you are estimating effects for. Reporting should explicitly state the eligible/overlap population, not just the overall SOC population.
- Causal ≠ optimal: Even a causal estimate of average improvement may hide heterogeneous effects by attack vector, asset class, or shift. Prebalancing helps you get credible averages; policy optimization typically requires structured heterogeneity analysis (ideally preplanned).
Governance and best-practice alignment
Finally, experimentation in incident response should align with incident-response governance. The #StopRansomware guidance developed with the Cybersecurity and Infrastructure Security Agency and co-authors emphasizes maintaining and regularly exercising an incident response plan and communications plan, keeping offline access, and including notification procedures as an operational backbone for any experimentation program because it stabilizes processes and ensures safety overrides are explicit.
For managing incident processes, the International Organization for Standardization’s ISO/IEC 27035-1:2023 describes a structured approach across preparation, detection, reporting, assessment, response, and lessons learned—useful as the “process map” for defining comparable treatments and outcomes.
Reference
Rosenbaum, P. R., & Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika, 70(1), 41–55.
Rubin, D. B. (2007). The design versus the analysis of observational studies for causal effects: Parallels with the design of randomized trials. Statistics in Medicine, 26(1), 20–36.
Imai, K., & Ratkovic, M. (2014). Covariate balancing propensity score. Journal of the Royal Statistical Society: Series B, 76(1), 243–263.
Hainmueller, J. (2012). Entropy balancing for causal effects: A multivariate reweighting method to produce balanced samples in observational studies. Political Analysis, 20(1), 25–46.
Austin, P. C. (2011). An introduction to propensity score methods for reducing the effects of confounding in observational studies. Multivariate Behavioral Research, 46(3), 399–424.
Robins, J. M., Hernán, M. A., & Brumback, B. (2000). Marginal structural models and causal inference in epidemiology. Epidemiology, 11(5), 550–560.
Angrist, J. D., Imbens, G. W., & Rubin, D. B. (1996). Identification of causal effects using instrumental variables. Journal of the American Statistical Association, 91(434), 444–455.
Kohavi, R., Tang, D., & Xu, Y. (2020). Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. Cambridge University Press.
National Institute of Standards and Technology (NIST). (2012). Computer Security Incident Handling Guide (SP 800-61 Rev. 2).
National Institute of Standards and Technology (NIST). (2016). Guide for Cybersecurity Event Recovery (SP 800-184).
Cybersecurity and Infrastructure Security Agency (CISA). (2023). #StopRansomware Guide.
International Organization for Standardization (ISO). (2023). ISO/IEC 27035-1: Information security incident management.