Introducing HealthBench

Source: OpenAI Date: 2025-05-12 URL: https://openai.com/index/healthbench

Summary

OpenAI’s introduction of HealthBench in May 2025 — a benchmark for evaluating AI models on healthcare-relevant tasks: clinical reasoning, patient communication, medical information synthesis, and appropriate referral/safety behavior. HealthBench was developed in collaboration with medical professionals and covers both capability (can the model provide useful medical information?) and safety (does it appropriately disclaim, refer to professionals, and avoid harmful outputs?).

Implications

Healthcare AI evaluation as a credibility move. Publishing a healthcare benchmark before broad medical AI deployment is OpenAI positioning as a responsible actor in a high-stakes domain. The benchmark development with medical professionals adds credibility beyond what an in-house eval would have. It also sets the evaluation standard that OpenAI’s models will be measured against — which they designed.

The capability-safety balance in medical AI. Medical information is a domain where being wrong has severe consequences, but being unhelpfully cautious also has costs — patients who can’t access clear information about symptoms or medications. HealthBench’s dual-axis evaluation (capability + safety) reflects this tension more honestly than most benchmarks, which optimize for one dimension.

Healthcare as the next enterprise AI frontier. Medical coding, clinical documentation, patient communication, diagnostic support — these are billion-dollar market opportunities where AI can clearly help but where the failure costs are high enough that enterprise buyers need evidence of safety. HealthBench is OpenAI’s evidence-building tool for this vertical.

Watch: Whether HealthBench scores for GPT-5.x models are used by hospitals and healthcare enterprises in their AI procurement decisions, and whether third-party medical AI companies (Epic, Oracle Health) adopt HealthBench as an evaluation standard or develop their own.