Skip to main content
Back

King's College London Catches Bad Medical AI Labels by Watching Gradients

A new study finds U-Nets shrug off most annotation noise in heart ultrasound, but a gradient-variance trick can rescue models under heavy systematic errors.

Paper, Full Access
  • King's College London team tests U-Net robustness to annotation errors on the CAMUS echocardiography dataset.
  • Variance of Gradients (VOG) flags mislabeled samples with 0.86-1.0 sensitivity, beating loss-based detection.
  • Three synthetic error types tested: incomplete masks, boundary dilation/erosion, and merged structure labels.
  • Baseline U-Net stays surprisingly robust up to 50% systematic label corruption without any intervention.
  • Pseudo-label refurbishment via 5-epoch prediction averaging helps most under heavy systematic noise.
  • Method extends VoG and SELFIE ideas from classification into multi-class medical segmentation.

Medical image segmentation lives or dies by the quality of its labels, but the humans drawing those masks are not infallible. A group at King's College London put that assumption under a microscope, testing how much annotation noise a U-Net can absorb before it breaks, and whether a lightweight training-time trick called Variance of Gradients can catch the bad labels before they poison the model.

The paper focuses on echocardiography, the noisy 2D ultrasound view of the heart used to compute biomarkers like ejection fraction. Segmenting the left ventricle, myocardium, and left atrium is tedious and inconsistent even for experts, which makes it a good stress test for label-noise research.

The dirty little secret of gold-standard labels

Most of the label-noise literature has focused on classification, where a single wrong class flips a scalar. Segmentation errors are spatial, they involve boundaries, and they can look plausible even when they are wrong. The authors simulate three realistic failure modes on the CAMUS echo dataset:

  • Incomplete labels: part of a structure gets left as background.
  • Boundary distortion: masks get dilated or eroded, mimicking annotator style drift.
  • Merged labels: one structure gets accidentally painted as another, like the myocardium collapsing into the ventricle.

Each error type is injected either randomly (any structure, any sample) or systematically (always the same structure, the same direction), with the latter simulating a consistent annotator bias rather than sloppiness.