Transluce releases SimMH-Chat evaluation of 77 model variants

• Transluce released a multi-turn crisis evaluation across 77 model variants: TransluceAI's announcement says the coverage spans eight developers, APIs, and consumer chatbot apps.
• Recent model generations rarely explicitly endorsed or facilitated suicide in these simulations, while TransluceAI's results report more connection-to-human-support behavior than in older GPT-4o and Gemini 2.5 systems.
• Creative-writing prompts remain a weak path: TransluceAI's finding recorded 20% to 64% willingness in recent models, while its task-assistance examples include farewell notes.
• Consumer web apps did not generally add safety over APIs: TransluceAI's browser/API comparison found parity or worse performance outside of resource banners.

The interactive report includes a transcript explorer and an API-to-browser comparison. Transluce has also published a SimMH-Chat dataset card and a separate MHUsage dataset built from privacy-preserving production-traffic statistics.

In its report, Transluce describes more than 50,000 multi-turn conversations and more than one million messages between simulated users and assistants. SimMH-Chat contains no real user conversations, according to its dataset documentation.

• Crisis states: the users that TransluceAI simulated expressed suicidal ideation, psychosis, or mania.
• Coverage: the test ran 77 model variants through API and, where available, consumer-browser interfaces.
• Scoring: 14 mental-health-relevant behaviors were defined with clinical input and measured by automated judges.

The common user trajectories that TransluceAI ran through each model enable cross-provider and over-time comparisons that production traffic cannot supply.

The clearest trend is a generational one. The full report puts recent-model rates of reinforcing delusions or mania at roughly 2% to 36% of simulated conversations, depending on model, versus 69% to 82% for examples including GPT-4o, Claude Opus 4, and Gemini 2.5.

Recent models also almost never explicitly endorsed or facilitated suicide in the evaluation, while helpful behaviors such as steering people toward human support became more frequent.

The remaining failures cluster around requests whose surface form resembles ordinary assistance or fiction.

• The few recent-model instances of aiding suicide often involved practical tasks such as writing farewell notes to loved ones.
• Recent models still engaged in creative writing with user-specific details suggesting the work might concern the user's own suicide, at rates from 20% to 64%.
• Models sometimes reinforced a delusion while also encouraging the user to seek support, as TransluceAI's examples show.
• In TransluceAI's GPT examples, mathematical material elicited lengthy mathematical formulations alongside mental-health-support language.

The permission that TransluceAI says OpenAI, Anthropic, and Google granted allowed Transluce to test consumer interfaces alongside APIs. Outside resource banners, the browser versions were not generally safer, and some performed worse.

The report also identifies browser configurations with markedly different behavior rates from their corresponding APIs, a reminder that the deployed product can diverge from the model endpoint.

The user simulators changed after Transluce compared them with anonymized ChatGPT and Claude usage data, a collaboration described by TransluceAI. In TransluceAI's production comparison, real users were more likely to switch unrelated topics, ask for practical help, or make typos; Transluce generated new simulators and said the model rankings mostly held.

Some models still had meaningfully different absolute behavior rates under the production-derived simulators.

More than 30 clinical experts shaped the behavior taxonomy, TransluceAI said. Transluce also described the project as a months-long effort involving external partners in its report thread.

The release includes two public artifacts:

• SimMH-Chat: 48,956 simulated transcripts, 3,524,788 conversation-by-rubric-by-judge rows, 157 personas, and 24 rubrics.
• MHUsage: differentially private synthetic conversation vectors approximating patterns from 8,237 real mental-health-related ChatGPT and Claude conversations.

The report also publishes a template legal agreement based on its arrangements with Anthropic, OpenAI, and Google DeepMind, plus operating-condition disclosures aligned with the AI Evaluator Forum's AEF-1 standard. Transluce describes the project as descriptive rather than normative in its statement of purpose.