Skip to main content
Back

Tsinghua's SynCred-Bench Reveals AI Fakes Fool Humans 37% of the Time

A new benchmark reveals that AI-generated fake official documents fool humans 37% of the time and defeat nearly every automated detector on the market

Paper, Full Access
  • New benchmark: SynCred-Bench introduces 600 AI-generated fake official documents (notices, certificates, receipts) to test detection systems.
  • Terrible detection rates: Under a 5% false-positive constraint, 15 MLLMs hit only 10.5% true positive rate; open-source detectors score below 5%.
  • Humans also fail: Human annotators correctly identified synthetic credibility images only 63% of the time.
  • Commercial APIs best, but still weak: Best-in-class commercial detection APIs reached 57.6% TPR, missing over 4 in 10 fakes.
  • New threat concept defined: "Synthetic credibility" -- AI images that mimic official seals, stamps, and layouts -- is framed as a distinct, underexplored misinformation category.
  • Why models fail: MLLMs reason about surface credibility cues (seals, letterheads) rather than generation artifacts, making them easy to fool.

The latest generation of image models can now produce fake government notices, medical certificates, and official receipts that are nearly indistinguishable from the real thing. This isn't just about photorealistic faces or dreamlike landscapes anymore. Deepfakes have crossed a critical threshold, eliminating earlier tell-tale glitches and becoming accessible to anyone with a smartphone. The threat has quietly shifted to something more insidious: documents that look credible, not just realistic.

A new paper from researchers at Tsinghua University introduces SynCred-Bench, the first benchmark specifically designed to measure this threat. The core concept they're probing is called synthetic credibility: the ability of AI-generated images to carry the visual hallmarks of trustworthy, official sources -- seals, letterheads, handwriting, stamps, and worn paper textures -- even when the content is entirely fabricated.

Why existing detectors are flying blind

Prior work on AI-generated image detection focused on visual artifacts -- the telltale blurring, warped fingers, or noise patterns that generative models leave behind. That approach fails completely here. Today, many AI-generated documents are often unconvincing under close inspection, but the quality is improving rapidly. The real danger isn't pixel-level fakery; it's that a document looks like it came from a real institution.

This is the gap SynCred-Bench is designed to expose. The benchmark tests whether a system can detect that an image is AI-generated when that image has been deliberately crafted to look like a trustworthy artifact -- not just a pretty picture.

Example of an AI-generated official education bureau notice with annotated credibility cues like seals, handwriting, and paper texture

What's actually in the benchmark

The dataset has two components. The main set, SynCred-Bench, contains 600 AI-generated misinformation images. The companion set, FP450, contains 450 real images used to measure false positives -- how often a detector incorrectly flags a genuine document as fake. This pairing is critical: a detector that flags everything as fake is useless in practice.