Harvard and MIT's Science Sandbox Catches Claude Beating Humans While Missing the Point

A team from the Broad Institute, Harvard, MIT, and Yale has proposed a new way to measure whether AI agents can actually do science, rather than just optimize a benchmark. Their paper introduces science sandboxes, a closed-loop framework where an agent runs experiments, gets feedback, and revises hypotheses, then puts three frontier coding agents through two biology tasks. The headline finding: today's best agents can match or beat expert humans on the numbers while completely missing the rules that generate those numbers.

The gap benchmarks cannot see

Most AI evaluations report a single score, which works when the task has a verifiable answer. Empirical science is different. Unlike software or mathematics, where candidate solutions can be checked through unit tests or formal proofs, empirical sciences like biology and chemistry lack scalable, automated verifiers. A system that fits data with epicycles is not the same as one that discovers gravity, and traditional benchmarks reward the former. The authors' fix is to preserve the entire investigative loop and grade the reasoning, not just the final metric.

How a science sandbox works

A science sandbox is a controlled environment in which an AI agent conducts experiments to learn the rules governing a phenomenon. In each round, the agent chooses what to test based on its current hypotheses, receives feedback from a sealed oracle, revises those hypotheses, and selects the next experiment. Every sandbox has three components:

• Specimens: what the agent submits (DNA sequences, proteins, strings).
• Assays: the measurement applied to each specimen.
• Oracle: a sealed mechanism that runs the assay and returns a report.

Oracles span three levels of physical grounding. Wet oracles use actual physical experiments. Damp oracles use computational models trained on empirical data to approximate experimental results. Dry oracles apply invented rules specified by the sandbox designer, with no necessary relationship to the natural world. The dry setting is the clever part: because the designers know the rule and the agent does not, they can tell whether the agent inferred it or just climbed a hill.