OpenAI's BioBasicsBench Exposes How AI Misreads Real Lab Evidence
OpenAI and biotech firm Ligo released a 100-task benchmark that tests whether AI agents can actually interpret messy wet-lab evidence.
- BioBasicsBench is a new 100-task benchmark from OpenAI and Ligo for wet-lab AI agents.
- Tasks use raw gel images, colony plates, microscopy, and instrument files across five biology domains.
- GPT-6 Astra led with 66.7% mean partial-credit reward at high reasoning effort.
- Each task was reviewed by Ligo scientists plus an external PhD-level wet-lab reviewer.
- Models still make confident mistakes: wrong colony picks, 12% colony overcounts, z-stack confusion.
- Positioned as a measurable baseline for lab assistance progress, biosafety, and biosecurity tracking.
OpenAI and enzyme-design startup Ligo have released BioBasicsBench, a 100-task evaluation of how AI agents interpret raw wet-lab evidence. Tasks use gel images, colony plates, microscopy data and instrument readouts. Models must extract observations, check controls and limit conclusions to what the evidence supports.
Biology evaluations often measure factual recall, multiple-choice performance, protocol reasoning or bioinformatics. BioBasicsBench targets the interpretive step between collecting data and deciding what to do next. Its scope covers analysis after data collection; physical work such as pipetting, instrument operation and robotic execution remains outside the benchmark.
Messy evidence, five lab domains
The tasks span five areas of routine wet-lab work:
- Protein methods
- Imaging and cytometry
- Quantification and assays
- Nucleic acid methods
- Molecular cloning
Agents work from images and instrument files that preserve the ambiguity of real experiments. A response may need to distinguish a biological signal from an imaging artifact, identify a mismatched control or conclude that the available data cannot support the proposed interpretation.
The tasks underwent review by another Ligo scientist and review by an external wet-lab PhD.
Astra leads on score and brevity
OpenAI evaluated seven models at a high reasoning-effort setting, giving each model five attempts per task. Mean rubric reward includes partial credit for incomplete answers. The 95% confidence interval reflects statistical uncertainty around each mean, and output tokens per task indicate average response length.
| Model | Mean reward | 95% CI | Output tokens per task |
|---|---|---|---|
| GPT-6 Astra | 66.7% | 60.3–73.1% | 4,704 |
| GPT-6.1 Sol | 62.0% | 55.6–68.4% | 5,238 |
| GPT-5.6 Sol | 43.8% | 37.6–50.0% | 9,099 |
| GPT-6 Luna | 38.3% | 32.3–44.5% | 13,120 |
| GPT-5.5 | 35.4% | 29.7–41.0% | 11,135 |
| GPT-5.4 | 33.2% | 28.3–38.4% | 13,717 |
| GPT-5.2 | 31.2% | 26.2–36.3% | 27,620 |
GPT-6 Astra led at 66.7%, 22.9 percentage points above GPT-5.6 Sol. Astra averaged 4,704 output tokens per task, compared with 9,099 for GPT-5.6 Sol, and produced the shortest responses listed.
Partial-credit scoring means a 66.7% average can include many incomplete answers. Task-level completion rates would answer a different reliability question. The table also compares seven OpenAI models under one configuration; cross-provider and human comparisons require separate baselines. A deployment cost analysis would need input-token counts, latency and model pricing.
Where observation turns into error
OpenAI’s qualitative analysis shows several ways a plausible observation can produce an unreliable laboratory decision:
- Focus change misread as biology: A microscope depth scan, or z-stack, captures the same sample at multiple focal planes. One response initially attributed the visual change to a biological event.
- Gel identities misassigned: The model distinguished separate band patterns, then attached the wrong experimental identities and selected the wrong colonies.
- Plate count inflated: One colony-counting response overcounted by about 12% after missing some colonies, merging neighboring colonies and marking empty regions.
- Conflicting evidence ignored: A microscopy response noticed variation with focus, then retained a generalization contradicted by lighter regions in the same field.
GPT-6 Astra also caught a plate labeled with a different antibiotic from the one used in the experiment, a mismatch that would invalidate a control. In another task, it recognized that overlapping microscopy detection channels left a cell-death conclusion unsupported. The report presents these as qualitative examples; per-category accuracy would be needed to estimate how reliably each behavior occurs.
The evidence-reading layer
Earlier evaluations cover adjacent stages of scientific work. BenchBench-Protocol focuses on reasoning about protocol modifications, while ABC-Bench tests agents on broader biology tasks that include novel bioinformatics work. BioBasicsBench isolates the interpretation of evidence after an experiment has produced data.
That scope makes the benchmark relevant to lab copilots that review images, verify labels, assess controls or draft preliminary interpretations. Its failure cases also expose a central deployment risk: a model can describe an artifact correctly and still recommend the wrong action.
Design around the failure modes
Developers building laboratory assistants can translate the benchmark’s errors into concrete system requirements:
- Record observations separately from interpretations and recommended actions.
- Preserve raw images, instrument metadata and control conditions for review.
- Check labels, antibiotics, sample identities and control assignments before analysis.
- Escalate cases with overlapping channels, poor image quality or conflicting evidence.
- Require scientist approval for colony selection, experimental conclusions and follow-up procedures.
OpenAI also connects these capabilities to biosecurity because the same evidence-reading skills can support beneficial enzyme research and harmful biological projects. Deployment therefore requires access controls, logging and review processes alongside accuracy testing.
The reported scores and failure cases support a supervised copilot role in which scientists review recommendations and retain authority over laboratory decisions.
Comments
