Why rule-based systems hit a wall
Traditional transaction monitoring relies on deterministic rules: flag any transfer above a threshold, any payment to a high-risk jurisdiction, any rapid sequence of deposits just under a reporting limit. These rules are transparent and easy to defend to a regulator, but they are brittle. They generate enormous numbers of alerts, the overwhelming majority of which are false positives, with rates exceeding 95 percent in many deployed systems. Each alert consumes analyst time, so the operational cost is staggering, and because the thresholds are known or guessable, sophisticated launderers simply structure their activity to stay underneath them.
This is the gap machine learning fills. Modern aml transaction monitoring increasingly layers statistical models on top of, or in place of, static rules, learning the difference between normal and suspicious behaviour from data rather than from a fixed rulebook. The goal is not only to catch more genuine crime but to drown out far fewer legitimate transactions in the process.
Supervised learning: triaging and scoring alerts
The most established application is supervised classification. Given a history of alerts labelled by their eventual outcome, did this case result in a Suspicious Activity Report or not, a model learns to predict the risk that a new alert is genuinely suspicious. Gradient-boosted trees such as XGBoost are workhorses here, valued for their performance on tabular financial features and their relative interpretability, while sequence models like LSTMs capture temporal patterns in a customer’s transaction history.
A common and pragmatic design is the triage model that sits behind the existing rule engine. Rather than replacing the rules, it scores the alerts they produce, pushing likely false positives down the queue and surfacing the riskiest cases first. The results can be dramatic. One widely cited study, Anti-Money Laundering Alert Optimization Using Machine Learning with Graphs, demonstrated a triage model on a real-world banking dataset that cut false positives by 80 percent while still detecting more than 90 percent of true positives. The catch, familiar to any data scientist, is label quality: confirmed laundering cases are rare, labels are delayed and noisy, and the positive class is a tiny fraction of the data, which makes careful handling of class imbalance and evaluation metrics beyond accuracy essential.
Unsupervised learning: finding the unknown unknowns
Supervised models can only learn the typologies they have seen labelled. Launderers innovate constantly, which is why unsupervised methods are the second pillar. Anomaly detection algorithms such as Isolation Forests and autoencoders learn what normal looks like and flag transactions that deviate sharply from it, without needing any labelled examples of fraud. Clustering techniques like DBSCAN and k-means group customers and behaviours, making it possible to spot an account that suddenly behaves unlike its peer group.
The power of unsupervised approaches is that they can surface emerging schemes the labelled data has never captured. The trade-off is precision: an anomaly is not the same as a crime, and unsupervised flags require careful thresholding and human review to avoid simply generating a new flood of alerts. In practice, the strongest systems combine both worlds, using unsupervised signals as features or candidate generators feeding a supervised triage layer.
Graph and network analysis: following the money
Some of the most important signals in AML are not visible at the level of a single transaction or account. Money laundering is fundamentally about relationships: layering funds through chains of intermediaries, structuring deposits across many accounts, and orchestrating networks of mule accounts. Representing transactions as a graph, with entities as nodes and payments as edges, lets models reason about that structure directly.
Graph-based features, such as a node’s centrality, the density of its local neighbourhood, or the existence of circular payment paths, capture behaviours that entity-level features miss entirely. Graph neural networks can learn directly on this structure, and classic graph algorithms can trace multi-stage smurfing chains across several intermediaries. The optimisation study mentioned above showed precisely this, that combining entity-centric engineered features with graph-based features characterising inter-entity relationships, computed over time-windowed dynamic graphs, materially outperformed either approach alone. For a data scientist, the engineering challenge is real: building and updating these graphs at the scale and latency a payment system demands requires serious attention to time and space efficiency.
Feature engineering is where the work lives
As with most applied ML, model architecture matters less than the features fed into it. Effective AML models rely heavily on engineered features: transaction velocity and volume over rolling windows, deviations from a customer’s historical baseline, geographic and counterparty risk, the ratio of round-number transactions, peer-group comparisons, and the graph features described above. Behavioural profiling, learning each customer’s normal rhythm and measuring departures from it, is often more predictive than any single transaction attribute. Much of a successful AML modelling effort is spent constructing these signals and the temporal aggregations behind them, not tuning the final estimator.
The hard problems: labels, drift and explainability
Three challenges dominate production AML modelling. The first is label scarcity and bias: the only labels available are the outcomes of alerts the existing system already raised, so the model never sees the laundering that slipped through undetected, a classic selection-bias trap. The second is concept drift. Criminal typologies evolve deliberately to evade detection, so a model trained on last year’s patterns degrades, demanding continuous monitoring and retraining rather than a fire-and-forget deployment.
The third, and arguably the most consequential in this domain, is explainability. A regulator will not accept a SAR decision justified by an opaque score, and the people accountable for compliance need to understand why a case was flagged. This is why techniques like SHAP are not optional niceties but core infrastructure in AML modelling, used to attribute each prediction to its contributing features. It also explains why interpretable models such as gradient-boosted trees remain popular even as deep learning advances, and why model risk governance, documentation, validation and ongoing testing is taken so seriously. The whole effort sits inside a regulatory framework built on the risk-based approach set out by the Financial Action Task Force, which expects firms to concentrate scrutiny where risk is highest and to be able to demonstrate that their controls work.
Putting it into production
The mature pattern is not machine learning replacing rules wholesale but augmenting them. Rules provide a transparent, regulator-friendly baseline that catches clear-cut violations; ML adds the nuance, scoring and prioritising alerts, surfacing novel anomalies, and exposing networked behaviour. Humans stay firmly in the loop, investigating escalated cases and providing the feedback that, over time, becomes new training labels. Around all of this sits the operational machinery any data scientist will recognise: data pipelines for real-time feature computation, model monitoring for drift, periodic retraining, and rigorous validation before anything touches a production decision.
Conclusion
Machine learning has turned AML transaction monitoring from a brittle, rule-bound exercise into a genuine data science discipline. Supervised triage models slash false positives, unsupervised methods catch emerging schemes, and graph analysis exposes the relationships at the heart of laundering. Yet the field rewards humility: labels are scarce and biased, the adversary adapts, and every model must justify itself to a regulator. For data scientists, that combination of high stakes, messy data and hard constraints is exactly what makes AML one of the most interesting problems in applied machine learning today.