Introducing heMTEB, an open-source benchmark for health information retrieval

Montréal, October 18 – Clinia is proud to announce heMTEB (health-specific Massive Text Embedding Benchmark), a purpose-built set of health datasets to extend the MTEB suite, providing a new standard for evaluating AI models for health information retrieval. The first release of this suite introduces Clinia's CURE — Crosslingual Understanding and Retrieval Evaluation, an open-source health-specific dataset for retrieval evaluation. Developed in close collaboration with health experts, the datasets cover 10 medical disciplines in three cross-lingual settings: English-English, French-English, and Spanish-English. The launch of heMTEB underscores Clinia’s dedication to advancing open-source evaluation tools that address the unique challenges of developing real-world medical AI applications.

The Massive Text Embedding Benchmark (MTEB) is the industry standard for evaluating text embeddings, ensuring their effectiveness in powering accurate and reliable AI applications. It is a comprehensive benchmark with a wide array of datasets spanning 8 tasks, including information retrieval. Until now, MTEB has lacked the purpose-built datasets needed to fully evaluate retrieval capabilities specifically in health applications, where the nuances of complex terminology and returning complete and accurate results can have a significant impact on the health outcomes of an individual. This has made it challenging for the health community to fully assess how well their models perform in real-world scenarios and optimize AI applications for complex health and medical use cases, where accuracy, reliability, and trust are essential.

The CURE was created to address this challenge head-on and empower the health community when building reliable and trustworthy AI applications for health. Developers, researchers, academics and health institutions can now assess model performance fairly for health information retrieval tasks, giving measurable confidence in the progress of their work. Clinia’s clients will also benefit from a more robust Health-grade Search, designed to enhance their health workflows and applications from end to end.

Developed in collaboration with health professionals to meet real-world, point-of-care information retrieval needs, the CURE is designed to test how models can meet the diverse individual needs of medical professionals while also being highly specialized to cater to various health disciplines.

Key features of the CURE include:

• Supervised by health professionals across 10 specialties, the datasets ensure per-discipline granularity and cover both the specificity and diversity of the medical landscape;

• Support for varying levels of language and terminology, from layperson to medical expert, the CURE evaluates model performance in diverse communication settings involving both patients and health professionals;

• Test a model’s cross-lingual capabilities, which ensures end-users can query in their native language without compromising accuracy or losing time in translation.

The CURE is now publicly available and will be integrated as a MTEB task in the coming days, allowing developers and researchers to incorporate it into their testing and benchmarking workflows. Additionally, a public leaderboard will be live on Hugging Face, allowing for performance comparison of various open-source and proprietary models against this benchmark.

In line with the commitment to continual improvement and building for impact on health, the development of the CURE will continue to include additional medical disciplines, to help address the diverse and evolving needs of the health community.

About Clinia

Clinia helps health organizations to deploy Health-grade Search across their ecosystems, so their users can access the right health information at the right time. Each year, millions of health journeys are powered by Clinia Search infrastructure - enabling organizations to supercharge the impact of their data, empowering care teams to deliver efficient and timely care, and supporting patients to live healthier lives.

For more information visit www.clinia.com, follow us on LinkedIn or contact press@clinia.com.

Clinia will also offer onsite demonstrations at the HTLH Conference (booth V4608) on October 20-23 for those interested in exploring the capability of the benchmark and our search product suite.