Skip to main content
Back

European Universities Built EuropeMedQA to Expose Medical AI's English Bias

A new European benchmark taps official medical licensing exams from four countries to test whether multimodal LLMs actually handle non-English clinical reasoning.

Paper, Full Access
  • EuropeMedQA sources questions from official medical licensing exams in Italy, France, Spain, and Portugal.
  • First European multilingual and multimodal medical benchmark designed to resist training data contamination.
  • Evaluates GPT-5-mini, Claude 3.5 Haiku, and Claude Sonnet 4.5 under strict zero-shot prompting.
  • Answer options are shuffled with fixed seed 42 to eliminate positional shortcuts.
  • French subset preserves multi-answer scoring and hierarchical two-image clinical cases from the ECN exam.
  • Dataset and code will be released publicly following FAIR and SPIRIT-AI reporting standards.

Medical LLMs keep passing the USMLE, but that success has always come with an asterisk: the questions are English, the diseases skew American, and the exams have been on the public internet long enough to leak into training corpora. A team from Italian and French universities is trying to close that gap with EuropeMedQA, a multilingual, multimodal benchmark built from official regulatory exams in Italy, France, Spain, and Portugal.

The paper is a study protocol, meaning it describes the dataset construction and evaluation pipeline in detail before the full results drop. The design choices themselves are what warrant attention, because they target failure modes that most existing medical benchmarks quietly ignore.

Why English-only medical benchmarks keep flattering models

Models like Med-PaLM 2 and GPT-4 have achieved passing scores on the USMLE, but that headline hides two problems the authors want to force into the open. LLMs produce varying outcomes when evaluated on items from different countries and contexts, owing to disparities in disease prevalence, clinical guidelines, terminologies, and question formats across regions. GPT-4 excels on USMLE-style questions, yet performance drops notably on non-English exams like Polish medical licensing tests or Spanish benchmarks, revealing biases toward English-centric training data.

Contamination is the second issue. Numerous studies have assessed model accuracy on established datasets like MedQA and MedMCQA, but the widespread online availability of these resources poses a significant risk of training data contamination, artificially inflating performance metrics. Synthetic questions have been proposed as a workaround, but they tend to strip out the cultural and clinical nuance that real licensing exams enforce.