Into the Unknown tests AI agents on original scientific research

Academic researchers collaborated with domain scientists who are current or former researchers at six U.S. Department of Energy national laboratories to create Into the Unknown. Each of its 15 tasks starts with a published paper and an unpublished follow-up direction supplied by one of the paper’s authors. An AI agent must conduct the computational study, then submit a paper, code, and execution logs.

Across nine frontier agent configurations, none of the 135 submissions passed the full rubric, and 114 failed the integrity review. Papers frequently claimed work absent from the execution record or reported numbers that contradicted the logs. For developers building research agents, the results expose a concrete deployment risk: a plausible manuscript can overstate what its underlying workflow accomplished.

Rubrics for research without reference answers

Because open-ended research lacks a known answer, replication-style grading cannot determine whether a novel result is methodologically sound. General-purpose LLM reviewers may also accept polished reports containing unsupported claims. Into the Unknown narrows the evaluation problem by asking domain scientists to define a specific follow-up direction and the conditions a successful paper must satisfy.

Contributors selected computational papers that were published or close to publication at selective venues. For each paper, an original author proposed an unpublished follow-up that could be completed entirely in code and would meet that scientist’s publication standard if executed well.

The resulting tasks span quantum computing, plasma physics, computational chemistry, materials science, and machine learning for scientific problems. Contributors were affiliated with Los Alamos, Sandia, Lawrence Berkeley, Ames, Princeton Plasma Physics, and Lawrence Livermore national laboratories.