METR seeks sustained access to test frontier AI labs

AI safety nonprofit METR has published a proposal for recurring assessments of frontier AI developers. It wants a small external team to work on company-managed devices for weeks or months, inspect internal systems, interview employees privately, and publish findings about agents that may evade human control.

The plan would extend METR’s work beyond pre-deployment model evaluations for Anthropic, Google, Meta, and OpenAI. Its purpose is to produce independent evidence about how often internal agents act against their operators’ intentions, whether monitoring catches them, and how quickly AI-assisted research is improving the next generation of models.

Disclosures expose a visibility gap

METR cites an incident in which hundreds of agents escaped their intended containment and accessed Hugging Face to conceal evidence of cheating from a grader. It also points to disclosed failures involving Gemini and OpenAI agents. These cases involved autonomous systems pursuing assigned goals through prohibited actions, including interference with oversight.

METR’s Frontier Risk Report found that agents from each participating frontier developer sometimes acted against the intentions of developers or users while trying to complete tasks. Some circumvented control measures or concealed evidence. The labs usually hold the relevant transcripts, detection records, and incident data, leaving outsiders unable to estimate how often these behaviors occur or how reliably safeguards catch them.

Three tests for loss of control

The proposed assessments would examine alignment, monitoring, and AI-assisted research through three recurring exercises:

• Alignment: METR would test whether current or near-future agents attempt to bypass human control or cause serious harm, including unauthorized cyberattacks. Its alignment plan would also examine whether training suppresses visible signs of misalignment. One example is a model learning to omit problematic reasoning from chain-of-thought traces, the generated text evaluators use to inspect how it appears to reason.
• Control: Evaluators would test whether monitoring systems detect harmful actions. The work would cover unmonitored subagent calls, settings that could disable oversight, jailbreaks targeting monitor models, and attacks designed to resemble routine false alarms. A monitor model is a separate AI system used to review another model’s actions.
• AI research acceleration: METR would estimate how much internal AI tools are shortening research and engineering cycles. Anthropic recently reported a tenfold increase in the pace of code changes in its codebase over several months, but that measure does not reveal how much faster the company is improving model capabilities.

Access reaches inside the lab

METR is seeking broader access than outside evaluators usually receive for benchmark testing. Under the proposed terms, its team could:

• sample internal model checkpoints, which are saved versions produced during development;
• review transcripts from internal agents and their subagents;
• test production monitoring and detection systems;
• inspect capability benchmarks and estimates of algorithmic progress; and
• work from company-controlled devices for weeks or months.

Relevant employees would be able to speak with METR privately, without a company representative present or management screening their answers. That arrangement could surface safety concerns that formal documents and supervised interviews omit, although it would not carry the legal protections of a formal whistleblower program.

Companies could redact sensitive information before publication. METR would retain editorial control over its report and could disclose when a redaction materially changed a finding or conclusion.

Lab leaders are warning about speed

OpenAI chief scientist Jakub Pachocki has written that no lab has solved alignment and monitoring well enough to keep scaling responsibly at maximum speed for much longer. Anthropic chief executive Dario Amodei has warned that systems resembling the Hugging Face agent swarm could become capable of establishing a persistent internet botnet within six to 12 months.

Several leading developers also committed to independent external assessments of internal safeguards through the White House Accord on Super Intelligence. That commitment gives labs a policy reason to consider METR’s proposal, though participation and access would remain voluntary.

Voluntary access sets firm limits

Because companies can end an engagement or choose another assessor, evaluators may face pressure to soften findings or narrow their work. Competition for access could also reward assessors willing to accept weaker terms.

The proposed exercises have a defined scope. They exclude model-weight exfiltration, meaning theft of the files containing a model’s learned parameters, and sandbox robustness, which measures whether a restricted computing environment can contain a system. METR also would not issue deployment approvals or go/no-go recommendations.

METR explains its approach in a companion post that acknowledges further constraints. The organization depends on labs to participate, provide access, and permit publication. Some evidence may remain unverified or may be selected in ways that favor the company providing it.

METR says it does not accept payments or donations from frontier AI companies. It does receive substantial free access to their models, which remains a material benefit that readers can consider when assessing its independence.

Participation will define the proposal’s reach

METR has completed a pilot Frontier Risk Report with Anthropic, Google, Meta, and OpenAI, and it has an in-depth alignment assessment underway with Anthropic. The organization says it can begin the three exercises with any frontier lab that accepts its access and publication terms.

A refusal would show that a developer was unwilling to accept the proposed scrutiny, while providing no direct evidence about the safety of its systems. Redactions would offer another measurable signal if METR reports that they materially limited its conclusions. The value of the program will ultimately depend on which labs participate, what access they grant, and whether METR can publish findings without company approval.