Skip to main content
Back

Politecnico di Milano's TGAD Benchmark Exposes AI Defect Detection as Largely Fake

A new benchmark exposes that multimodal anomaly detection models barely respond to text instructions, with performance collapsing below chance on realistic industrial data.

Paper, Full Access
  • TGAD benchmark reveals that text prompts in multimodal anomaly detection models barely influence decisions, exposing a fundamental gap between claimed and actual capabilities.
  • On the new Assembled Panel Dataset (APD), one model scores 31.5 I-AUROC, below the 50 random-chance baseline.
  • Removing just the object noun from a prompt drops AnomalyGPT's image-level accuracy from 97.4 to 82.6, showing models anchor on object names, not full instructions.
  • Component-level instructions (e.g., "inspect only the left capacitor") are ignored once defects outside the instructed region are present, with scores falling from 90.3 to 66.3.
  • Three model paradigms tested: generative (AnomalyGPT), training-free discriminative (LogSAD), and embedding-adaptive (AA-CLIP) -- all fail to genuinely follow language instructions.
  • The paper calls for a new benchmark protocol as a prerequisite before deploying language-controlled inspection systems in real industrial settings.

There is a growing class of industrial inspection systems that promise something compelling: point the model at a circuit board, tell it in plain English which component to check, and it will flag only the defects you care about. The pitch is text-guided anomaly detection, and it has attracted serious research attention. A new paper from researchers at Politecnico di Milano argues that this promise is largely illusory, and they have the benchmark to prove it.

The paper introduces TGAD (Text-Guided Anomaly Detection), a structured benchmark designed to answer one deceptively simple question: does the text you provide actually change what the model decides? The answer, across every model and every scenario tested, is mostly no.

Why the old benchmarks were blind to this

Industrial anomaly detection has long been dominated by MVTec AD, the de facto standard dataset. MVTec AD is a dataset for benchmarking anomaly detection methods with a focus on industrial inspection, containing over 5000 high-resolution images divided into fifteen different object and texture categories. It is a rigorous visual benchmark, but it was designed for unimodal models that only look at images.

When multimodal vision-language models arrived, researchers simply plugged them into the same evaluation protocol. The problem: traditional AI research for automatic vision inspection typically relies on discriminative models within the conventional deep learning paradigm. These models can only perform trained detection tasks and cannot provide detailed reports like quality inspection workers. Additionally, when production lines or requirements change, traditional methods necessitate retraining or redevelopment. The new multimodal models were supposed to fix this by accepting natural language instructions, but the benchmarks never checked whether those instructions were actually doing anything.

The core flaw is that existing evaluation protocols hold the text prompt constant across all test images. If the prompt never changes, you cannot measure whether the model is responding to it. High scores on MVTec AD could reflect powerful pretrained visual features, not genuine language conditioning. The TGAD benchmark is built specifically to break that ambiguity.