---
title: "ING and TU Delft's LightGBM Catches 93% of Changes That Break Production"
subtitle: "A bank replaced hand-coded change risk rules with gradient boosted trees plus SHAP explanations, lifting recall from 56% to 93%."
url: "https://alphasignal.ai/news/ing-and-tu-delft-s-lightgbm-catches-93-of-changes-that-break-production"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-15T00:00:00.000Z"
updated: "2026-08-23T23:02:38.175Z"
type: "Paper"
topics:
  - "Infra"
  - "Business"
subtopics:
  - "Monitoring"
  - "Inference Optimization"
upvotes: 3122
source_url: "https://arxiv.org/abs/2604.13462"
comments: 0
truncated: true
---

# ING and TU Delft's LightGBM Catches 93% of Changes That Break Production

> A bank replaced hand-coded change risk rules with gradient boosted trees plus SHAP explanations, lifting recall from 56% to 93%.

Somewhere between 70% of Google's production outages and roughly all of the pain in a typical enterprise ops team, one culprit keeps showing up: the change that looked safe on the ticket. A new [ICSE-SEIP paper from ING and TU Delft](https://arxiv.org/abs/2604.13462) replaces the bank's rule-based change risk score with gradient boosted tree classifiers, and shows the ML version is both more accurate and interpretable enough to survive financial regulators.

## Why change tickets are a machine learning problem

Change-induced incidents are a well-known scourge. Google has reported that roughly 70% of outages in its live systems stem from changes, infrastructure failures can cost around $100,000 per hour, and critical application failures can reach $1 million per hour. Banks feel this twice: once from the downtime, and again from regulators.

That regulatory layer is what makes this problem interesting. In 2023 the European Parliament adopted the Digital Operational Resilience Act (DORA), which took effect in 2025 and requires documented policies, procedures, and controls based on a risk assessment approach, integral to the financial entity's change management process, ensuring all changes are recorded, tested, assessed, approved, implemented, and verified in a controlled manner. Translation: you cannot just point a black box at a change ticket and let it veto deployments. Every decision has to be auditable.

## The baseline that wasn't working

ING already had a risk score. Its IT Service Management tool uses business rules that consider factors like scope of impacted IT services, deployment complexity, incident history, Confidentiality/Integrity/Availability ratings, SOx criticality, and recoverability. The output is a score from 0 to 100, mapped to low (0-33), medium (34-59), and high (60-100) risk levels.

On a two-month test set, that rule-based scorer achieved a weighted recall of 0.56 and an AUC of 0.55, close to random performance. It flagged plenty of changes as high risk, but most flagged tickets never caused incidents, and it missed nearly half the ones that did.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Learning from Change: Predictive Models for Incident Prevention in a Regulated IT Environment](https://arxiv.org/abs/2604.13462)
- [LightGBM Documentation](https://lightgbm.readthedocs.io/)
- [SHAP GitHub Repository](https://github.com/shap/shap)
- [Digital Operational Resilience Act (Regulation EU 2022/2554)](https://eur-lex.europa.eu/eli/reg/2022/2554/oj)