Anthropic proposes metrics for AI-assisted model development

Anthropic has proposed three development metrics that frontier AI labs could publish to show how quickly models are automating research, how labs monitor internal agents, and how much research compute goes to safety. The company also released internal figures from July and August 2026 as an initial baseline.

Frontier labs, which develop the most capable general-purpose models, increasingly use AI to write code, run experiments, analyze results, and coordinate work on future systems. That feedback loop could shorten development cycles while increasing the need for effective oversight. Anthropic’s proposal would give regulators and independent evaluators a common reporting framework for examining the process.

Three numbers to track

Claude leads one-quarter of measured R&D work

Anthropic’s R&D Automation Index assigns sampled research tasks to the Epoch AI scale, which runs from AL0 to AL5. The relevant thresholds are:

• AL0: The task involves no AI assistance.
• AL3: AI collaborates on the task under close human supervision.
• AL4: AI leads the task from an initial prompt while a human supervises.
• AL5: AI completes the task without a human in the loop.

As of August 2026, more than 90% of Anthropic’s measured AI R&D work ranked at AL3 or higher. Claude led 26% at AL4, while no measured category reached AL5. The AL4 share began the measured period below 1%. Anthropic projects that it could reach 80% by the end of 2026 if the observed trend continues, making that figure a conditional forecast rather than a target or measured result.

A Claude research agent created the index by reviewing each sampled employee’s week through Slack messages and internal documents. It identified roughly 15,000 tasks and organized them into a taxonomy with 542 nodes. A separate Claude judge then assigned an automation level to each task.

Because Claude helps generate both the task inventory and the scores, correlated blind spots could distort the result. Anthropic proposes using third-party or cross-lab judges to reduce that risk. The index remains an estimate derived from sampled records, rather than direct telemetry showing autonomous task completion.

Thirty thousand agents, two monitoring layers

Anthropic’s second metric covers agents running on its most-used internal research and engineering platform. About 30,000 were active concurrently as of August 2026, according to the company.

Online monitors review proposed actions before execution and can block behavior such as attempts to exfiltrate model weights. Offline monitors inspect transcripts afterward for patterns that emerge across longer sequences of activity. Anthropic proposes reporting three measurements:

• Coverage: Every agent action passes through both monitoring layers.
• Latency: Online review occurs before execution, and humans review blocked actions within one week. Offline monitoring flags about 100,000 transcripts per week, with roughly 50 escalated to human reviewers.
• Escalation rate: The online monitor blocked 0.002% of more than one billion research and engineering agent decisions in August 2026, or about one in 47,000.

Persistent identities allow each agent’s record to survive model upgrades. Agents also communicate through a shared internal message bus, giving monitors access to the conversation graph and allowing agents to cross-check one another. The reported block rate measures monitor interventions; it does not count independently confirmed security or alignment incidents.

Safety receives 6% of R&D compute

During one week in July 2026, Anthropic allocated about 6% of its AI R&D compute to safety work. Within the narrower category of AI-driven AI R&D, where models perform work that contributes to future systems, safety received about 12%.

Anthropic describes these estimates as deliberately conservative. Safety experiments also tend to consume less compute than model pretraining, so compute share cannot substitute for staffing, spending, or research output. Comparisons across labs would require a shared definition of safety work, consistent accounting rules, and access for independent auditors.

From disclosure to verification

Anthropic CEO Dario Amodei has argued for coordinated frontier pacing. Measuring internal automation would give labs and regulators a way to track whether AI-assisted research is compressing development timelines. Monitoring and compute-allocation data would show which controls and resources accompany that acceleration.

Anthropic is also developing arrangements for embedded third-party evaluators with access comparable to its internal risk teams. Such access would let evaluators inspect the underlying records, sampling methods, monitor behavior, and classification decisions instead of relying only on published totals.

Faster AI-assisted development could affect model release cadence, API migration schedules, compatibility testing, and the frequency with which developers refresh evaluations. Anthropic’s automation index provides the clearest pace indicator among the three metrics, while the oversight and safety-compute figures describe the controls surrounding that work.

Stable definitions, raw-data access, and independent replication across labs would turn the proposal into an auditable reporting standard. Until those mechanisms exist, the figures remain Anthropic’s self-reported baseline.