NUS's DeceptionDecoded Catches 14 Frontier AI Models Failing to Spot Fake News Intent
A new 12,000-sample benchmark shows top vision-language models still fail to spot deliberately misleading multimodal news, even when the manipulation is deliberate.
PRO- NUS researchers released DeceptionDecoded, a 12,000-sample benchmark for detecting misleading creator intent in multimodal news.
- Intent is modeled explicitly during generation as desired influence plus execution plan, not inferred by readers.
- 14 VLMs including GPT-4o, Claude 3.7, and Gemini 2.5 Pro struggle, especially on image manipulations and desire inference.
- Models are fooled by internal image-text consistency, polished writing style, and one-line prompt hints about authenticity.
- Fine-tuning on 6,000 samples gave Qwen2.5-VL-7B a 30-point macro-F1 gain on MMFakeBench.
- Nano Banana and GPT-image-1 safety guardrails were bypassed by framing prompts as creator personas instead of misinformation requests.
Detecting fake news used to mean checking whether an image matched its caption or whether a date lined up. A new benchmark from the National University of Singapore argues that this framing misses the actual problem. Modern misinformation is rarely about broken facts; it is about the story a creator is trying to plant in your head. DeceptionDecoded is built to test whether vision-language models can see that intent, and the answer, across 14 frontier systems, is mostly no.
DeceptionDecoded contains 12,000 multimodal news instances created through an intent-guided synthesis pipeline. Each instance is anchored in a trustworthy news context drawn from verified articles and paired with a predefined creator intent configuration. The dataset and evaluation code are on GitHub, though the specific generation prompts are withheld to prevent misuse.
Why intent beats inconsistency as a target
Prior benchmarks rely on heuristic strategies such as CLIP-based mismatching or sentiment substitutions, which fail to capture the complexity of real-world multimodal misinformation. The tell in real disinformation is subtler. A picture of icebergs, paired with a caption attributing the collapse to secret underwater nuclear tests, is technically consistent image-to-text but engineered to erode trust in institutions. Earlier efforts such as EMU, NewsINT, and MRF began to consider news creator intent, but they remain limited to unimodal settings and typically define intent from readers' subjective perspectives, which may diverge from the actual communicative purpose behind a piece.
DeceptionDecoded takes a different route by simulating the creator directly and recording their intent as ground truth before any manipulation happens.
How the benchmark is built
The pipeline starts from verified reporting in the VisualNews corpus and layers on a simulated creator. Drawing on strategic communication theories, the authors conceptualize creator intent along two core dimensions: desired influence and execution plan. Desired influence specifies the societal sectors the creator seeks to affect, selected from a predefined taxonomy (up to three per instance). The execution plan is open-ended text describing how the creator aims to achieve that influence.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
