---
title: "PatMD Teaches AI to Spot Its Own Meme Moderation Mistakes"
subtitle: "A new retrieval-augmented framework teaches multimodal LLMs to spot the specific ways they usually misread memes, lifting F1 scores by 8.30% on average."
url: "https://alphasignal.ai/news/patmd-teaches-ai-to-spot-its-own-meme-moderation-mistakes"
author: "AlphaSignal Newsroom"
author_url: "https://alphasignal.ai/editorial-team"
company: "AlphaSignal"
company_url: "https://alphasignal.ai/company/alphasignal"
published: "2026-04-16T00:00:00.000Z"
updated: "2026-08-23T20:56:06.869Z"
type: "Paper"
topics:
  - "Image"
  - "Security"
subtopics:
  - "Vision Language"
  - "Red Teaming"
upvotes: 1278
source_url: "https://arxiv.org/abs/2510.15946"
comments: 0
truncated: true
---

# PatMD Teaches AI to Spot Its Own Meme Moderation Mistakes

> A new retrieval-augmented framework teaches multimodal LLMs to spot the specific ways they usually misread memes, lifting F1 scores by 8.30% on average.

Sarcasm, metaphor, and visual irony are the rhetorical devices that make memes funny, and the same devices that trip up multimodal LLMs trying to flag harmful content. A new approach called [PatMD](https://arxiv.org/abs/2510.15946) tries a different angle: instead of teaching the model what harmful content looks like, it teaches the model how it tends to get things wrong.

PatMD detects harmful memes by learning from and proactively mitigating potential misjudgment risks, moving beyond content-level matching to identify underlying failure patterns that steer MLLMs away from known pitfalls. Tested across five detection tasks on 6,626 memes, it delivers an average 8.30% improvement in F1-score and 7.71% improvement in accuracy over strong baselines.

## Where content-based retrieval breaks down

Most modern harmful meme detectors, including retrieval-augmented generation (RAG) systems that pull similar historical memes for context, share the same weakness: they match on surface content. When novel memes use entirely different templates to convey similar implicit rhetoric like sarcasm or metaphor, these methods struggle. Prior work also fails to formalize the MLLM's inherent reasoning pitfalls into generalized, reusable patterns that explain why misjudgments happen.

The authors give a clear example. An MLLM can look at a meme and correctly identify a human and a monkey in the image, yet fail to capture why and how their juxtaposition constitutes a racially hateful analogy. The pieces are recognized; the composition is missed.

## Cataloging failure, not content

PatMD builds a knowledge base where each meme is annotated with a short explanation of the specific way an MLLM might misread it. These fall into two buckets:

* **False negative risks** for harmful memes: how compositional elements subtly encode harmful intent that a model would likely overlook.

_Preview only. Full text requires an AlphaSignal Pro account._

## Sources

- [Fall into a Pit, Gain in a Wit: Cognitive-Guided Harmful Meme Detection via Misjudgment Risk Pattern Retrieval](https://arxiv.org/abs/2510.15946)
- [PatMD full paper (HTML)](https://arxiv.org/html/2510.15946)
- [GOAT-Bench: Safety Insights to Large Multimodal Models through Meme-based Social Abuse](https://arxiv.org/abs/2401.01523)