arXiv:2607.15166: MedFailBench Measures HOW Medical AI Systems Fail, Not Just Accuracy, Across 44 Cases

What does MedFailBench measure differently from classic benchmarks?

MedFailBench is a benchmark built by clinicians with the goal of measuring the failure mode of medical AI systems — the specific pattern in which a system gets something wrong — instead of the usual focus solely on the percentage of correct answers. This shift in focus makes it possible to distinguish a harmless error from one that endangers a patient.

Five severity levels and six error types

Errors are categorized by severity on a scale from 1 to 5 and by six types: missed emergencies, unsafe medication doses, unsafe discharge decisions, fabricated evidence, unsafe protocol execution, and source support gaps. This kind of taxonomy reveals exactly where a model fails, not just that it failed.

What version v0.2.1 contains

The current version, v0.2.1, contains 44 clinically reviewed synthetic cases, a safety taxonomy, a clinical scoring rubric, and an automated screening pipeline that speeds up the evaluation of new models.

What MedFailBench deliberately does not contain

The authors explicitly emphasize that the benchmark contains no real patient data, clinical validation claims, or model leaderboards — it is a tool for probing safety boundaries, not a certificate of clinical usability. The project is released under the Apache-2.0 and CC-BY-4.0 licenses, with a DOI on Zenodo, which makes it more transparent for independent review compared to classic accuracy-focused benchmarks.