EurekaBench tests whether AI agents can explain the data they fit

Frontier AI agents came within 1.4 percentage points of the human baseline on EurekaBench’s predictive-accuracy metric. Their best scientific-insight score remained 27.3 points behind, and several systems violated basic domain constraints on more than half the tasks. The results suggest that benchmarks centered on prediction can overstate an agent’s ability to conduct scientific research.

Designed with ten doctoral-level scientists, the EurekaBench paper presents 26 research problems across neuroscience, geophysics, astrophysics, computer science, plasma physics, and chemistry. Each task supplies observations that challenge existing explanations, along with a simulator for testing hypotheses. The agent must submit a proposed mechanism, working Python code, and a record of its experiments.

Why a close fit can mislead

Most AI-scientist benchmarks emphasize accuracy on held-out data. A model can achieve that accuracy with correlations, tuned parameters, or flexible equations that reveal little about the system’s behavior. EurekaBench defines a mechanism as an executable account that respects domain knowledge, predicts unseen observations, and supports testable conclusions.

The benchmark scores submissions along three axes:

• Scientific constraints (SC): Does the mechanism respect expert-specified physical, biological, or computational assumptions?
• Predictive accuracy (PA): How accurately does the submitted Python implementation predict held-out data?
• Scientific insights (SI): Can a separate judge use the unchanged mechanism and additional simulator experiments to answer expert-authored questions about the system?

EurekaBench turns scientific insight into a measurable score through 306 yes-or-no questions. Domain experts wrote each question around a conclusion that a strong mechanism should support, such as how a system responds under a new condition or which process drives an observed effect.