OpenAI has released MentalHealthBench — an open benchmark for evaluating the behavior of language models in psychological conversations. The dataset under an MIT license contains 1,215 synthetic dialogues and 5,262 expert criteria, written by more than 80 licensed psychologists and psychiatrists from 22 countries, and covers conversations with adults, teenagers aged 13–17, caregivers, and doctors in 11 languages — from everyday stress to acute crisis. Model responses are evaluated by the grader model GPT-5.6 Sol, and the best result is shown by GPT-6 Astra with 57.3%: even the top model performs slightly more than half of the behaviors that experts consider ideal.

image

What happened

OpenAI published the open benchmark MentalHealthBench along with the dataset, methodology, and report. The set includes 1,215 synthetic psychological dialogues and 5,262 expert evaluation criteria; the scenes cover adults, teenagers aged 13–17, caregivers, and doctors in 11 languages and are labeled by three levels of acuity: non_acute, high_acuity, and emergent. The criteria were written by more than 80 licensed psychologists and psychiatrists from 22 countries, speaking 19 languages; each dialogue was checked by at least three experts, and a criterion was kept in the set only with the agreement of at least two. The final score of a model is broken down into ten axes of behavior, including context gathering, practical recommendations, empathy, refusal of harmful help, and maintaining user autonomy. Model responses are evaluated by the grader model GPT-5.6 Sol; according to the table in the official report, GPT-6 Astra leads with 57.3%, followed by GPT-6 Sol with 53.9% and Claude Opus 5.5 with 52.4%. The dataset is available for download in JSONL format, 2.1 MB in size, and contains a canary string against training data contamination; along with it, a 30-page report with the methodology has been published.

Context

Previous open evaluations of AI in mental health focused on emergency scenarios: they checked whether a model recognizes a crisis and escalates correctly, but left out the main mass of everyday conversations — everyday stress, anxiety, questions about routine and well-being. MentalHealthBench is the first open benchmark to cover the full spectrum of acuity, from non_acute to emergent, so it measures not only behavior during a crisis but also the quality of ordinary supportive conversation. A second methodological feature is the economics of evaluation: the weights of the criteria range from −10 to +10, meaning the rubric not only rewards useful answers but also penalizes harmful ones, for example supporting a user's false beliefs or insufficient escalation during a crisis. Because of this, a high score requires both usefulness and refusal of dangerous patterns, and the breakdown of the final figure into ten axes turns it into a profile of the strengths and weaknesses of a specific model.

Why this matters for the industry

For teams building assistants and other conversational products, this is primarily a ready-made open evaluation tool, not a product: the dataset, criterion weights, and report can be taken and integrated into their own evaluation pipeline today. Practical step: download the JSONL, pin the version of the grader GPT-5.6 Sol, run their model and prompts across the ten axes, and include the criterion weights in CI as regression tests, tracking where the model fails in context gathering or escalation during a crisis. Benchmarks for comparison are already available: 57.3% for GPT-6 Astra and 52.4% for Claude Opus 5.5. For business teams, measuring safety in sensitive conversations becomes much cheaper: the question "how safe are you" turns into a verifiable figure based on open criteria, and the 30-page report serves as a ready-made source of requirements for specifications and technical tasks. The scheme "MIT dataset plus expert rubrics with negative weights plus canary string" is a reproducible evaluation template for other sensitive conversational products beyond psychology.

Why this matters for users

For the reader, the most important figure is 57.3%: even the best model, GPT-6 Astra, performs only slightly more than half of the behaviors that experts consider correct in psychological conversations, meaning top chatbots still noticeably lag behind clinical recommendations. This is a direct reason to be more cautious about AI advice in sensitive situations, especially in acute states where the cost of error is maximal. The set's scenarios cover not only adults but also teenagers aged 13–17, caregivers, and doctors — the groups that are already turning to chatbots for support today. The benchmark also provides an open way to compare models on safety before trusting them with complex conversations: the dataset and 30-page report are available to anyone, so over time these axes can be used to question the developers of the services you use.

What is still unknown / limitations

The leaderboard should be treated with caution: model responses are judged by the grader GPT-5.6 Sol, which belongs to the same company whose models top the table, so a high score is a vendor's claim about its own models, not an independent measurement. The published materials do not show validation of the judge — how well GPT-5.6 Sol's evaluations align with human judgments. The threshold of "at least two out of three" experts is the lower bound of rubric quality, and statistics on inter-expert agreement beyond that are not disclosed. The benchmark does not measure latency, inference cost, or serving reliability, has not been tested on real users, and frequent grader runs via API are an additional cost. Finally, an open dataset with public weights is a convenient target for training on the test, so the set will need updates and private holdouts; until third-party runs appear, the figures can correctly be considered data from one vendor and its grader.

Sources

Author

Look at AI, editorial team