Anthropic's Claude Now Flags Its Own Misuse at Under 2% Extra Cost
Anthropic trained Claude to grade the safety of its own conversations right after responding, catching misuse other classifiers miss at under 2% extra compute.
- Anthropic introduced self-assessment, where Claude grades the safety of each conversation right after responding.
- Flags 78% of harmful conversations and 94% of high-priority ones at 1% false-positive rate on Fable 5.
- Adds under 2% inference compute by reusing the KV cache and emitting roughly 10 tokens.
- Scales with model size: recall rises from 16% to 96% from smallest to largest model tested.
- Weak as a blocking classifier (54% attack success vs 25% for prompted classifier) but strong as an offline monitor.
- Since June, surfaced 100k+ CSAM flags from one customer, romance scam operations, and prompt injection patterns.
Anthropic has deployed a misuse-detection technique called self-assessment, which trains Claude to produce a short safety verdict immediately after each response. Since June, self-assessment has been running in production across Claude models including Fable 5. Anthropic has now detailed its design, performance, limitations, and early findings.
After Claude replies, the system appends a hidden instruction asking the model to assess the exchange. The assessment states whether the conversation involves a potentially harmful request, what kind of harm it involves, a priority score for human review, whether Claude refused, and a one-sentence explanation of what led to the assessment. Benign conversations produce only {"flagged": false}. The system also derives a continuous risk score from the probability Claude assigns to the token true.
The KV cache keeps costs low
Claude’s full conversation context remains available in the key-value cache after it generates a response. This cache stores intermediate computations from the original inference, allowing the assessment to reuse them instead of processing the conversation again. Claude typically needs one short instruction and about 10 additional output tokens to produce its verdict.
Across production API traffic, Anthropic estimates that self-assessment adds less than 2% to inference compute. Its first Constitutional Classifiers, released in early 2025, added 23.7% to inference cost. OpenAI has reported that safety reasoning consumed as much as 16% of compute for some launches.
Self-assessment produces a free-text explanation alongside structured fields, drawing on Claude’s general knowledge of harmful activity. This design allows the model to identify behavior beyond the policy categories used during training, including previously uncatalogued misuse patterns.
Fable 5 catches the highest-risk cases
On Claude Fable 5, self-assessment detects 78% of conversations identified as potentially harmful by a stronger teacher classifier. Recall rises to 94% for high-priority cases, with a 1% false-positive rate. Here, recall measures the share of harmful conversations detected, while the false-positive rate measures benign conversations incorrectly flagged.
The teacher is a prompted classifier built on Claude Mythos Preview with Anthropic’s full internal policy documents included in its context.
| Measure | Result |
|---|---|
| Overall harmful-conversation recall | 78% |
| High-priority recall | 94% |
| False-positive rate | 1% |
| Estimated production compute overhead | Less than 2% |
Model size strongly affects detection quality. Across four models trained only with supervised fine-tuning, recall at a 1% false-positive rate rises from about 16% for the smallest model to about 96% for the largest. Area under the receiver operating characteristic curve, or AUC, increases from 0.862 to 0.995. AUC summarizes how reliably a detector ranks harmful conversations above benign ones across different thresholds; a score of 1 indicates perfect separation.
These results suggest that self-assessment can improve alongside the underlying model because both response generation and monitoring draw on the same capabilities. Fixed classifiers require separate upgrades to keep pace with more capable models.
Unseen harms remain detectable
Anthropic tested generalization by removing entire harm categories from the training data, then evaluating the model on those omitted categories. At a 1% false-positive rate, excluding fraud examples reduced recall on synthetic fraud conversations from 92.8% to 90.0%. Excluding cybersecurity examples reduced recall on synthetic cyber conversations from 92.7% to 90.3%.
| Omitted category | Recall with category | Recall without category |
|---|---|---|
| Fraud | 92.8% | 90.0% |
| Cybersecurity | 92.7% | 90.3% |
The small declines indicate that training primarily teaches Claude to express safety judgments in a consistent format. Much of its ability to recognize harmful conduct appears to come from knowledge acquired during general model training.
Synthetic data teaches the reporting format
Anthropic trains self-assessment in two stages using fully synthetic data from a pipeline similar to Constitutional Classifiers:
- Supervised fine-tuning: A prompted teacher classifier labels synthetic conversations. Training loss applies only to the assessment tokens, allowing Claude to learn how to judge harmful replies without rewarding it for generating those replies.
- Reinforcement learning: Six verifiable rewards score valid JSON, agreement on the flagged label, exact category match, exact subpolicy match, refusal agreement, and proximity to the teacher’s priority score.
The data generator creates 32 themes for each subpolicy and 32 scenarios for each theme, yielding 1,024 harmful and 1,024 benign scenarios per subpolicy. The benign examples cover similar topics, making them difficult negative cases rather than obviously safe conversations.
Attackers can still evade the monitor
Self-assessment provides limited protection when used as a real-time blocking mechanism. In a red-team harness covering 24 offensive cybersecurity tasks, attackers succeeded 79% of the time without safeguards and 54% of the time with self-assessment. A standalone prompted classifier reduced attack success to 25%.
Self-assessment increased the generated-token cost of attacks by roughly four times. An attacker with repeated access to the monitor could still search for a jailbreak that suppresses its verdict. Anthropic therefore positions the technique as one layer in a broader detection and enforcement system, while stronger classifiers handle high-risk blocking decisions.
Production monitoring finds missed abuse
Passive monitoring since June has identified abuse and policy gaps that Anthropic’s other detection systems missed:
- Child sexual abuse material: In August, self-assessment generated more than 100,000 child sexual abuse and exploitation flags tied to one API customer. Anthropic investigated the activity and reported it to the National Center for Missing and Exploited Children.
- Unclassified misuse: Roughly one-third of flags fall outside the existing harm taxonomy. Clustering those cases revealed attempts to extract Claude’s reasoning through illicit distillation, prompt-injection attacks, and agent actions that users had not authorized.
- Previously undetected organizations: More than half of the organizations surfaced by self-assessment had produced no flags from other detection sources during the previous month. One case involved an automated romance-scam operation built with Claude Code.
- Session-specific policies: Anthropic can insert a policy block into the hidden assessment prompt, allowing a verified security team to test designated software without triggering the same treatment as unauthorized intrusion attempts.
Tying misuse detection to Claude’s own judgment gives Anthropic a low-cost monitor that can recognize activity beyond a fixed taxonomy and improve as model capabilities advance. The production results also define its boundary: self-assessment is useful for discovery, prioritization, and defense in depth, while adversarial blocking still requires stronger independent safeguards.
Comments
