MysteryMechanism tests whether AI can rediscover scientific laws

Vals AI has released MysteryMechanism, an evaluation that asks AI agents to recover 222 mathematical relationships drawn from scientific literature. The benchmark removes subject labels, internet access, and other retrieval cues, then gives each model a small experimental budget. GPT-6 Astra scored 53.15%, leading Claude Fable 5.1 by 5.40 percentage points and GPT-5.6 Sol by 19.82 points.

A black box and 2d+1 queries

Each task provides anonymous input variables, allowed ranges, two noisy passive observations, and a persistent shell for calculations. An agent may run 2d+1 experiments for a mechanism with d inputs, then must submit one executable expression. A three-input mechanism therefore permits seven active queries.

Grading executes the submitted expression at fresh, privately sampled input points. The model earns binary credit when its normalized prediction error falls below a threshold adjusted for the task’s noise level. Functionally equivalent formulas can pass even when their notation differs from the source equation.

The 222 test mechanisms cover biology, physics, chemistry, engineering, ecology, and abstract dynamical systems. Because the variables are anonymous, agents must choose informative interventions, infer the equation’s shape, fit parameters, and decide when the available evidence supports a final formula.

Five probes recover a viscosity law

A public blood-viscosity task uses two anonymous inputs representing hematocrit and shear rate. The agent sees bounds such as x1 in [0.05, 0.70], receives two passive observations, and may run five experiments because d=2.

One successful trajectory probes the four corners of the input space and one interior point. The results show that one variable steepens near its upper bound while the other follows a square-root-shaped decline. The agent fits this multiplicative family:

mu*(1+a*x1/(1-x1)**b)*(1+c/(1+sqrt(x2)))
mu=1.7707, a=0.9128, b=1.9124, c=1.6447

The expression passes the private structural tests without naming hemodynamics or identifying the source equation.

Astra leads on accuracy and cost

Vals reports an average cost of about $1.77 per task for GPT-6 Astra, compared with $5.63 for second-place Claude Fable 5.1. Astra used roughly 31% of Fable’s per-task spending while passing 12 more mechanisms.

Passing traces follow the experimental data

A blinded post-hoc audit examined 449 passing trajectories. Depending on the model, auditors classified 85.5% to 98.6% of successful runs as relying on experimental results, mathematical transformations, and parameter fitting. GPT-6 Astra left the source discipline unnamed in 89.8% of its passing traces, while GPT-5.6 Luna did so in every passing trace.

• GPT-6 Astra: Passing traces frequently design interventions, isolate terms, compare compact function families, and fit coefficients.
• Claude and Gemini models: A larger share of passing traces names the source field, indicating more explicit use of scientific priors.

Because the audit reads visible traces, unspoken scientific knowledge may still influence a model’s choices. The “domain unnamed” metric records whether the trace identified the correct field; it cannot establish which internal representations contributed to the answer.

Known laws make discovery scorable

Genuinely new scientific findings lack ready-made ground truth, which makes automated discovery difficult to evaluate consistently. MysteryMechanism uses established relationships as verifiable targets while concealing their names and scientific context. Its task design captures a narrow discovery loop: inspect noisy observations, choose the next experiment, infer a functional form, and test that form on unseen inputs.

Private structural probes also discourage formulas that merely interpolate the few observed points. A submission must generalize across the mechanism’s input range, giving the benchmark a stronger signal than direct equation recall or fit quality on a tiny visible sample.

What developers can use

• Model selection: Teams building symbolic-regression, active-learning, or automated-lab systems can compare models under the same query budget.
• Strategy analysis: Agent traces reveal how models allocate experiments, transform variables, choose function families, and recover parameters.
• Cost planning: Reported per-task spending provides an inference-cost baseline for controlled equation-discovery workloads.

MysteryMechanism’s scope ends at rediscovering known functional relationships in a simulated environment. Real laboratory work also requires instrumentation, safety controls, hypothesis formation, replication, and decisions under changing conditions. The leading model failed 104 of 222 tasks, and the evaluation does not test whether any model can derive a genuinely novel law. Within those limits, the full leaderboard provides a reproducible measure of experimental design and equation recovery under sparse data.