PatMD Teaches AI to Spot Its Own Meme Moderation Mistakes
A new retrieval-augmented framework teaches multimodal LLMs to spot the specific ways they usually misread memes, lifting F1 scores by 8.30% on average.
PRO- PatMD improves harmful meme detection by retrieving patterns explaining why MLLMs typically misjudge, not just similar content. Paper
- Average gains of 8.30% F1 and 7.71% accuracy across five tasks on GOAT-Bench's 6,626 memes.
- Sarcasm detection sees the biggest jumps: +29.54% F1 for InternVL3-14B, +30.81% for GPT-4o-mini.
- A four-level hierarchical harm tree (category, subject, technique, instantiation) drives structural reranking beyond content similarity.
- Generalizes to 500 unseen political memes without knowledge base updates, gaining 5.27% F1 over the base model.
- Adds roughly 0.7 seconds latency per meme; pattern generation still relies on GPT-4o and human correction.
Sarcasm, metaphor, and visual irony are the rhetorical devices that make memes funny, and the same devices that trip up multimodal LLMs trying to flag harmful content. A new approach called PatMD tries a different angle: instead of teaching the model what harmful content looks like, it teaches the model how it tends to get things wrong.
PatMD detects harmful memes by learning from and proactively mitigating potential misjudgment risks, moving beyond content-level matching to identify underlying failure patterns that steer MLLMs away from known pitfalls. Tested across five detection tasks on 6,626 memes, it delivers an average 8.30% improvement in F1-score and 7.71% improvement in accuracy over strong baselines.
Where content-based retrieval breaks down
Most modern harmful meme detectors, including retrieval-augmented generation (RAG) systems that pull similar historical memes for context, share the same weakness: they match on surface content. When novel memes use entirely different templates to convey similar implicit rhetoric like sarcasm or metaphor, these methods struggle. Prior work also fails to formalize the MLLM's inherent reasoning pitfalls into generalized, reusable patterns that explain why misjudgments happen.
The authors give a clear example. An MLLM can look at a meme and correctly identify a human and a monkey in the image, yet fail to capture why and how their juxtaposition constitutes a racially hateful analogy. The pieces are recognized; the composition is missed.
Cataloging failure, not content
PatMD builds a knowledge base where each meme is annotated with a short explanation of the specific way an MLLM might misread it. These fall into two buckets:
- False negative risks for harmful memes: how compositional elements subtly encode harmful intent that a model would likely overlook.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
