Mind-Eval

MindEval: Benchmarking Language Models on Multi-turn Mental Health Support.

Installation

git clone https://github.com/SWORDHealth/mind-eval.git
cd mind-eval
python -m venv venv
source venv/bin/activate
poetry install

Poetry version 2.1.4 ; Python 3.10.12.

Make sure credentials to gcloud are set up.

IMPORTANT: edit CUSTOM_CLINICIAN_INTERACTION_TEMPLATE at the end of mindeval/prompts.py if you want to use a custom clinician system prompt template. To ensure a fair comparison with other models, only the arguments in MINDEVAL_CLINICIAN_TEMPLATE are allowed in any template.

Run interaction

Assuming you want to evaluate a clinician model with a custom system template on the default mindeval setting.

python mindeval/scripts/generate_interactions.py \
--profiles_path data/profiles.jsonl \
--clinician_system_template_version custom \
--clinician_model_api_params <YOUR_MODEL_API_PARAMS> \
--member_system_template_version v0_2 \
--member_model_api_params "{'model':'PROVIDER/claude-haiku-4-5@20251001','max_completion_tokens':16000,'reasoning_effort':'high','api_base':'API_BASE'}" \
--n_turns 10 \
--max_workers 50 \
--output_path interactions.jsonl

Copy the syntax of --member_model_api_params for your clinician model.

If you want to replicate the clinician setup for general-purpose models, set --clinician_system_template_version v0_1.

Adjust --max_workers according to rate limits.

Run judgments

python mindeval/scripts/generate_judgments.py \
--interactions_path interactions.jsonl \
--judge_template_version v0_1 \
--judge_model_api_params "{'model':'PROVIDER/claude-sonnet-4-5@20250929','max_completion_tokens':16000,'reasoning_effort':'high','api_base':'API_BASE'}" \
--max_workers 50 \
--output_path judgments.jsonl

Check scores

importpandasaspdfrommindeval.utilsimportload_jsonlfull_judgments=load_jsonl("judgments.jsonl")
float_judgments= [j["parsed_judgment"] forjinfull_judgments]
df=pd.DataFrame(float_judgments)
print(df.mean())

Do it all in one go (except checking scores; default MindEval settings)

bash run_benchmark.sh <YOUR_CLINICIAN_MODEL_API_PARAMS>

Meta-evaluation

Human data for patient realism and judge meta-evaluation is available.

Cite

@misc{pombal2025mindevalbenchmarkinglanguagemodels,
title={MindEval: Benchmarking Language Models on Multi-turn Mental Health Support}, author={José Pombal and Maya D'Eon and Nuno M. Guerreiro and Pedro Henrique Martins and António Farinhas and Ricardo Rei},
year={2025},
eprint={2511.18491},
archivePrefix={arXiv},
primaryClass={cs.CL},
url={https://arxiv.org/abs/2511.18491}, }