Goodfire publishes probe-based agent-monitoring guide with a reported 99% catch rate in a GLM 5.3 test

• Activation probes are small classifiers over internal computations, aimed at prohibited actions, CBRN requests, and reward hacking, according to GoodfireAI's definition.
• Goodfire reported a 99% catch rate for imminent prohibited tool calls by a GLM 5.3 coding agent in GoodfireAI's test result, with a stated false-positive rate below 1%.
• The proposed production layout keeps probes on for every run, then escalates flagged behavior to a more expensive judge, as GoodfireAI's monitor cascade lays out.
• Goodfire says its Silico agent can build a probe-monitoring workflow from a prompt in GoodfireAI's Silico post.

Goodfire's [new guide]Goodfire's guide treats model activations as a runtime monitoring surface. Anthropic's sleeper-agent research reported similarly strong linear-probe results in a controlled setting, while Ai2's account of separate Goodfire work describes tracing an unwanted behavior back through a post-training stack before a full run.

Goodfire defines a probe as a small detector trained on a model's internal computations. The proposed targets span user intent and agent behavior:

• intent to take a prohibited action
• CBRN-related requests
• reward hacking
• imminent tool calls

The post says probes add virtually no latency, can run continuously, and leave the model output unchanged in GoodfireAI's deployment claim. Its pitch is operational: surface a risk signal while the model is still generating or before an agent executes the next action.

Goodfire's headline number concerns a probe trained to predict an imminent prohibited tool call from a GLM 5.3 coding agent. It reported 99% detection with less than 1% false positives.

That result is a task-specific catch-rate claim. Anthropic's sleeper-agent study reports AUROC above 99% for defection probes trained on residual-stream activations in deliberately trained sleeper-agent models, using generic contrast pairs. The two reports measure different model behaviors with different metrics.

Goodfire places the probe at the cheap, always-on stage of a three-part cascade.

1. Activation probes score categories such as cybersecurity, CBRN requests, and prohibited tool calls in parallel.
2. An LLM judge reviews a flagged transcript and returns a verdict.
3. Actions can send a refusal or clarification, reroute the task to a model with stronger safeguards, or log the transcript.

Goodfire contrasts this with keyword and regex filters, which it calls brittle; LLM judges, which it calls slower and more expensive to run everywhere; and fine-tuning, which it says can have off-target effects in GoodfireAI's comparison.

Chain-of-thought monitoring and activation monitoring expose different signals. Goodfire says a model can internally represent sensitive topic classification, prompt interpretation, or an agent's next move without expressing that information in visible text, and calls for using both.

A recent mechanistic-interpretability paper likewise reports that reasoning operations such as formulation, decomposition, and deduction are separable in held-out hidden representations, with peak separability in middle layers. It supplies a research-side hint about where operation-level probe signals may be most legible.

Goodfire presents Silico as an interpretability agent that can construct a probe workflow from a natural-language prompt.

The company still lists several model- and task-specific choices behind a usable monitor:

• which layer to read
• how to pool token signals
• probe architecture
• dataset construction
• generalization testing

Goodfire's safety overview places related interpretability work in auditing and red-teaming, including feature steering for testing model behaviors.

Goodfire is also extending the same interpretability framing to training data. It says predictive data debugging can reveal and shape what a model learns before training, crediting Ai2 for open-sourcing its post-training stack in GoodfireAI's data-debugging post.

Ai2 says the work predicted behavioral changes before a full training run, traced an observed safety regression to individual preference examples, and tested targeted changes intended to reduce that regression without sacrificing broader capability gains in its September 9 writeup.