AI Diagnoses Patients More Accurately Than Doctors in Emergency Room Study

Researchers testing OpenAI’s o1 model on real emergency room records found the system outperformed physicians in diagnosing patients based on written clinical data, according to a study published April 30 in Science, highlighting the growing role of AI in frontline medical decision-making.

The study, led by researchers at Beth Israel Deaconess Medical Center and Harvard Medical School, evaluated how well the model could interpret sparse, real-world hospital documentation—an environment far less structured than traditional test cases. In one core experiment, the AI and two physicians reviewed records from 76 emergency department patients at different stages of care, including triage and admission.

The results showed the model delivered exact or near-exact diagnoses 67% of the time, compared with 50% and 55% for the two doctors. Researchers emphasized that the test used limited information typical of early patient encounters, where clinicians often make high-stakes decisions with incomplete data.

“This is the big conclusion for me: It works with the messy real-world data of the emergency department,” said study co-author Adam Rodman. “It works for making diagnoses in the real world.”

The findings build on earlier work showing strong AI performance on curated medical case studies, but extend that capability into more chaotic clinical settings. The model evaluated in the study is a preview version of OpenAI’s o1, a reasoning-focused system designed to break down problems step by step.

Researchers said the emergency room test was the most significant of several experiments, noting that the model performed strongly across other scenarios as well. Still, they cautioned that the results do not translate directly into clinical replacement.

Experts pointed to limits in how AI approaches decision-making. “When we say clinical reasoning, it doesn’t mean the same thing as moral reasoning,” said Arya Rao, a biomedical informaticist not involved in the study, noting that AI lacks the broader judgment applied in patient care.

The researchers also warned that performance could vary with larger, more complex patient datasets, such as long-term hospital stays. Future work will focus on clinical trials to determine how these systems can be integrated into real care settings.

Clinicians are already using AI tools for tasks like documentation, imaging analysis, and early disease detection. The study suggests that newer reasoning models could extend that role into diagnosis, particularly in time-sensitive environments like emergency departments.

“It’s just another tool to help us give the patient the highest quality care possible,” said physician Nour Khatib, who was not involved in the research.

This analysis is based on reporting from Smithsonian Magazine.

Image courtesy of Unsplash.

This article was generated with AI assistance and reviewed for accuracy and quality.