YesBut Benchmark Exposes GPT-4 and Claude-3's Blind Spot on Comic Humor
A new NeurIPS benchmark called YesBut shows that even top vision-language models still fumble the punchlines of two-panel comics built on contradiction.
Takeaways
- YesBut is a NeurIPS 2024 benchmark testing whether vision-language models understand humor from two-panel contradiction comics.
- Four tasks: literal description, contradiction generation, philosophy selection, and title matching.
- Best scores: Claude-3 hits 84.1% on philosophy, LLaVA-1.6 34B leads title matching at 63.3%.
- Text-only LLMs with oracle descriptions match open-source VLMs, pinning bottleneck to visual grounding.
- Failure modes: visual misreads, weak cross-panel comparison, and hallucinated cultural context.
- Code on GitHub, dataset on Hugging Face.
Multimodal models can describe a photograph, answer questions about a chart, and reason across video frames. Ask one to explain why a two-panel comic is funny, though, and it often falls flat. A new benchmark called YesBut, presented as an oral at NeurIPS 2024, puts numbers behind that gap and shows that humor built on juxtaposition remains a hard problem for GPT-4, Claude-3, and their open-source counterparts.
The paper, from researchers at Hong Kong Polytechnic University and Case Western Reserve University, argues that current models cannot process information through nonlinear, deep thinking effectively, since the autoregressive paradigm of language models limits bidirectional reasoning. Comics that hinge on a setup-and-subversion structure make a clean stress test: you cannot understand panel two without silently re-reading panel one.
Why two panels are harder than they look
A YesBut comic pairs a straightforward first panel (the yes) with a second panel that quietly contradicts it (the but). Picture a driver braking to let ducks cross the road, then in the next panel pulling into a fast-food restaurant. The joke lives in the space between the panels, not inside either one.

Getting the joke requires two skills a vision-language model rarely has to combine at once: social reasoning about human behavior, and nonlinear logical reasoning that compares the two panels for what changed and what stayed the same. Literal captioning is not enough.