Anthropic uses Claude to flag potential misuse after responses

• Claude checks conversations for potential misuse after responding, a technique Anthropic says has surfaced previously missed misuse since June.
• Fable 5 recovered 78% of a stronger classifier’s flags and 94% of its high-priority flags at a 1% false-positive rate, according to testingcatalog.
• Production assessments feed later review and add less than 2% inference compute, as WesRoth reports.

A clean check can return just {"flagged": false}. During training, Anthropic masks every token outside the assessment, even when the synthetic transcript contains harmful compliance. Traffic covered by zero-data-retention agreements is excluded.

Anthropic appends a hidden system instruction after Claude responds, asking it to assess the entire conversation, including requests it refused. According to its implementation description, neither that instruction nor the assessment is visible to users or carried into subsequent conversation turns.

A flagged assessment contains:

• flagged: whether the conversation contains a potentially harmful request.
• reason: a one-sentence explanation.
• category and subpolicy: labels from Anthropic’s internal harm taxonomy.
• refused: whether Claude refused the request.
• priority: urgency for human review, from 1 to 10.

Anthropic reads the probability assigned to true in the flagged field to obtain a continuous score. Changing the threshold sets the false-positive operating point.

Reusing the conversation’s KV cache is a tidy engineering win: an assessment averages about 10 generated tokens, with estimated overhead below 2% across Anthropic’s production API traffic distribution.

Anthropic trains Claude to imitate a policy-prompted Claude Mythos Preview classifier, the distillation setup rohanpaul_ai described. The teacher reads full internal policies and follows a reasoning checklist; the student emits its assessment directly, without those policies in its prompt.

The training pipeline uses fully synthetic conversations in two stages:

1. Supervised fine-tuning: Claude learns the assessment format and teacher labels. Only assessment tokens contribute to the loss, so harmful responses in the transcripts are material to judge rather than completions to learn.
2. Reinforcement learning: Claude samples assessments on-policy and receives rewards for schema compliance and agreement with the teacher’s labels and priority scores.

Anthropic also tested withholding an entire harm category from training. It reports no substantial degradation in recall for that category, supporting its claim that the monitor can draw on knowledge beyond the assessment-training examples.

Anthropic evaluated Fable 5 on a random sample of production traffic labeled by the Mythos Preview-based teacher. At an operating point that flags 1% of teacher-labeled benign conversations, it reports:

• 78% recall of all conversations the teacher flags.
• 94% recall of high-priority conversations, defined as teacher-assigned priority 7 or higher.

Those figures measure recovery of the teacher classifier’s judgments. The evaluation description identifies model-generated labels as the reference.

Separate experiments across four model sizes found that detection improved with size. Those models received supervised fine-tuning only and used a different evaluation set, so Anthropic says their scores are not comparable with the production Fable 5 results.

Anthropic says self-assessment has operated since June and now runs in every Claude model released since Fable 5, across a large fraction of global traffic. Its production workflow uses the assessments in three ways:

1. Organization-level escalation: In a growing set of harm areas, flags are aggregated nightly. Organizations with several high-priority flags are escalated to the enforcement team for human review.
2. Classifier comparison: Flags are compared with other classifiers’ outputs to find potentially harmful conversations those systems missed.
3. Novel-misuse discovery: Flags outside the existing taxonomy are clustered to identify new kinds of misuse.

Anthropic describes assessments as preliminary signals for investigation. Its data-handling section says the system uses conversation traffic already retained under existing policies.

In a separate experiment, Anthropic tested self-assessment as a blocking classifier, rather than its deployed offline-monitoring role. An attacker model attempted 24 offensive cybersecurity tasks and could rewind and retry with a different request whenever a turn was blocked.