OpenAI's GeneBench-Pro Exposes That Top AI Fails Real Biology 70% of the Time
OpenAI's new GeneBench-Pro tests whether AI agents can handle the messy judgment calls of real computational biology research — and current models solve fewer than a third of problems.
- GeneBench-Pro launches: OpenAI releases a 129-problem research-level benchmark testing AI agents on messy, judgment-heavy computational biology tasks.
- Best score is 31.5%: GPT-5.6 Sol (Pro mode) leads all models; GPT-5 scored below 5% when the original GeneBench was first built.
- Open-source release: 10 representative problems with data files, ground-truth answers, and a reference grader are available on Hugging Face under CC-BY-4.0.
- Key failure mode: Models spot data issues but don't act on them — they notice diagnostic signals and then pick the wrong analysis path anyway.
- Massive cost gap: Human experts need 20–40 hours per problem (~$4,000–$8,000); AI inference costs only a few dollars per problem.
- Independent evaluation coming: A 50-question subset will go to Artificial Analysis for third-party benchmarking; full benchmark may be saturated by year-end.
Most AI benchmarks in biology test whether a model knows facts or can run a standard pipeline. GeneBench-Pro tests something harder: can an agent look at a messy, real-world genomics dataset, figure out what is wrong with it, choose the right analysis strategy, and arrive at a conclusion that a researcher could actually act on? That is a very different bar, and today's best models clear it less than a third of the time.
The benchmark gap nobody was measuring
Existing biology benchmarks mostly measure knowledge retrieval, execution of routine pipelines, or a single analysis step. They do not capture what actually occupies most of a computational scientist's time: cleaning and normalizing data, exploratory analysis, statistical model selection, diagnostic iteration, and producing a conclusion that informs a downstream scientific or translational decision.
GeneBench-Pro is a challenging, research-level benchmark for testing whether models can handle the kind of judgment-heavy analysis that real-world computational biology requires. It is the successor to the original GeneBench, and it deliberately raises the difficulty floor. OpenAI calls the target skill "research taste" , the chains of judgment calls that shape an analysis: which questions the data can actually support, how early warning signs should change your model, and when your initial plan needs to be thrown out.
129 problems, no shortcuts allowed
GeneBench-Pro covers 129 questions across genomics, quantitative biology, and translational medicine, capturing the complexity, iterative nature, and ambiguity of scientific research in computational biology. The 10 domains include:
- Statistical and population genetics
- Regulatory omics and functional genomics
- Proteomics and biomarkers