CMU's IdeaTaste Reveals AI Research Agents Hit Only 18% of Human Judgment
A new CMU benchmark freezes the literature at 2025 and checks whether LLM research proposals match what human communities actually published in 2026.
- IdeaTaste benchmarks LLM research proposals against what communities actually published the following year.
- Across 90,000+ proposals, agents hit 26% of human level on citations and 18% on ideas.
- Models match problems and motivations but rarely the concrete approaches humans pursue.
- Scaling population size or giving richer field context barely closes the gap.
- Blind reviewers scored AI proposals 20 points below human papers on expected success.
- Open leaderboard, code and dataset released: project page, GitHub.
Researchers at Carnegie Mellon University and MBZUAI have introduced IdeaTaste, a benchmark that tests whether large language model agents can propose the same research directions a field later pursues. Across six models, four research areas, and more than 90,000 proposals, agents reached about 26% of the human baseline for citation matching and 18% for idea matching. The results point to project selection and method design as major weaknesses in autonomous research systems.
Many agent benchmarks evaluate execution after a human has defined the task. IdeaTaste evaluates the earlier decisions: what to read, which problems to pursue, and which methods to test. It treats later publication activity as a measurable proxy for research taste.
Taste becomes a prediction task
For each research area, papers available through the end of 2025 form the corpus that agents can search and cite. Papers published from January through August 2026 form the human reference set. The benchmark measures how closely the agents’ proposals anticipate those later papers.
Both corpora are expanded through citations from 20 to 22 hand-picked landmark papers per area, with surveys removed. Comparisons among the 2026 papers establish the human baseline used to normalize each agent population’s score.
The scoring pipeline converts each proposal or paper into two representations: its cited papers and a structured idea summary. Each summary contains five slots covering contribution type, problem, motivation, proposed approach, and research plan.
This story is for Pro members
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.
Comments
