Synthetic Hospital: Physician-Validated Longitudinal EHR Benchmark (arxiv.org)

The introduction of the Synthetic Hospital benchmark marks a significant advancement in the development of artificial intelligence for clinical applications. Designed as an open, fully synthetic longitudinal electronic health record (EHR) benchmark, it addresses the limitations posed by the scarcity of realistic benchmarking datasets in the healthcare domain. By utilizing only public medical-education material—free of protected health information—Synthetic Hospital provides a safe and verifiable ground truth foundation, encompassing 1,268 patient records and 5,602 encounters. This benchmark is crucial as it enables AI systems to be tested in a more realistic and challenging environment, significantly advancing their applicability in clinical settings. The technical implications of Synthetic Hospital are profound. It utilizes standard ontologies such as ICD-10-CM and SNOMED CT to ensure the clinical relevance of its data. In evaluations, existing frontier models struggled to match clinician performance, with the best algorithm achieving a severity-weighted F1 score of 0.73, trailing behind a group of physicians at 0.89. This gap highlights the ongoing challenges faced by AI systems in fully understanding and processing clinical information, underscoring the necessity for effective benchmarks like Synthetic Hospital to propel the AI/ML community forward in developing more capable and reliable clinical AI solutions.