Skip to main content
Back

MIDI Benchmark Catches GPT-5.2 Failing Basic Idioms Across 18 Languages

A new 18-language benchmark exposes a critical gap: LLMs can recognize figurative idioms but consistently fail when those same idioms are used literally

Paper, Full Access
  • MIDI benchmark launched: 18-language idiom dataset curated by native speakers, spanning high-, medium-, and low-resource languages with sentence and dialogue contexts.
  • LLMs near chance on low-resource languages: Average accuracy drops to 50% (4-way MCQ) for low-resource languages, barely above random.
  • Literal idioms harder than figurative: Across all resource tiers, models consistently fail more on literal uses of idioms than figurative ones -- a memorization artifact.
  • Memorization drives figurative success: Representation probing reveals models get figurative readings right via pattern-matching, not contextual reasoning.
  • Conversational context helps but doesn't fix it: Multi-turn dialogue improves scores but does not close the literal-figurative gap or the low-resource deficit.
  • Frontier models fare better but still struggle: Gemini 2.5 Pro and GPT-5.2 outperform open-source models but show the same systematic biases.

Ask GPT-5.2 what "kick the bucket" means and it will tell you correctly. Now put that same idiom in a sentence where someone is literally kicking a bucket, and watch the model stumble. This is the central tension that a new benchmark called MIDI was built to expose -- and the results are more troubling than the headline numbers suggest.

MIDI is a multilingual idiom dataset spanning 3 high-, 3 medium-, and 12 low-resource languages, curated by native speakers. Idiomatic expressions pose a major challenge for multilingual NLP because their meanings shift between figurative and literal usage, often requiring context for accurate interpretation. The dataset is the first of its kind to test idiom comprehension not just in isolated sentences, but inside realistic multi-turn conversations.

Why idioms break models

One of the most challenging aspects of human language for LLMs to comprehend is figurative language, which includes similes, idioms, and metaphors. Figurative language significantly enriches human communication by facilitating the implicit expression of complex ideas and emotions. Unlike literal expressions, figurative language often involves rich cultural references and judgments that vary considerably across different cultures.

With social media, this informal language has become more easily observable to people and trainers of large language models (LLMs) alike. While the advantage of large corpora seems like the solution to all machine learning and NLP problems, idioms and figurative language continue to elude LLMs. The core problem is that an idiom like "spill the beans" has a figurative meaning (reveal a secret) that is completely disconnected from the literal meaning of its words -- and the same phrase can appear in both senses in real text.