ING and TU Delft's LightGBM Catches 93% of Changes That Break Production
A bank replaced hand-coded change risk rules with gradient boosted trees plus SHAP explanations, lifting recall from 56% to 93%.
PRO- ING and TU Delft replaced a rule-based change risk scorer with LightGBM, lifting weighted recall from 56% to 93% on 175k real change tickets.
- Boosted trees were chosen because they handle missing values as signal and pair cleanly with SHAP for auditable, per-prediction explanations.
- Textual change descriptions, CI name, and assignment group dominated SHAP importance; most legacy rule-based factors did not make the top 15.
- Adding aggregated team metrics (release success rate, change frequency) modestly improved precision and recall but slightly lowered AUC.
- The stack is designed to satisfy DORA and the EU AI Act with mandatory human-in-the-loop review.
- No code or model is released; the paper is a proprietary case study with feature statistics withheld for confidentiality.
Somewhere between 70% of Google's production outages and roughly all of the pain in a typical enterprise ops team, one culprit keeps showing up: the change that looked safe on the ticket. A new ICSE-SEIP paper from ING and TU Delft replaces the bank's rule-based change risk score with gradient boosted tree classifiers, and shows the ML version is both more accurate and interpretable enough to survive financial regulators.
Why change tickets are a machine learning problem
Change-induced incidents are a well-known scourge. Google has reported that roughly 70% of outages in its live systems stem from changes, infrastructure failures can cost around $100,000 per hour, and critical application failures can reach $1 million per hour. Banks feel this twice: once from the downtime, and again from regulators.
That regulatory layer is what makes this problem interesting. In 2023 the European Parliament adopted the Digital Operational Resilience Act (DORA), which took effect in 2025 and requires documented policies, procedures, and controls based on a risk assessment approach, integral to the financial entity's change management process, ensuring all changes are recorded, tested, assessed, approved, implemented, and verified in a controlled manner. Translation: you cannot just point a black box at a change ticket and let it veto deployments. Every decision has to be auditable.
The baseline that wasn't working
ING already had a risk score. Its IT Service Management tool uses business rules that consider factors like scope of impacted IT services, deployment complexity, incident history, Confidentiality/Integrity/Availability ratings, SOx criticality, and recoverability. The output is a score from 0 to 100, mapped to low (0-33), medium (34-59), and high (60-100) risk levels.
On a two-month test set, that rule-based scorer achieved a weighted recall of 0.56 and an AUC of 0.55, close to random performance. It flagged plenty of changes as high risk, but most flagged tickets never caused incidents, and it missed nearly half the ones that did.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
