AI systems are already beating human doctors on some medical tests, but the bigger lesson is that the wins are narrow and the limits are still real.
Quick Take
- An OpenAI reasoning model outperformed two experienced doctors in a Harvard and Beth Israel Deaconess Medical Center study.
- The model used only the same electronic health record information the doctors had during the test.
- A large meta-analysis found no overall difference between generative artificial intelligence and physicians, but experts still did better.
- Other studies show artificial intelligence can match or beat doctors on narrow tasks, especially in controlled settings.
AI Wins in a Controlled Doctor Test
Researchers at Harvard Medical School and Beth Israel Deaconess Medical Center tested an OpenAI reasoning model on diagnosis and care-management questions. According to National Public Radio, the model outperformed two experienced physicians when both were limited to the same electronic health record information. That matters because the test did not give the machine extra clues. It compared the model with human doctors using the same facts available at the time.
The results fit a pattern that has been building across medical AI research. In this case, the system was not asked to replace bedside judgment or handle full patient care. It was asked to reason from records and choose likely diagnoses and next steps. On that kind of narrow task, the model did better than the doctors in the study. That is a striking result, but it is not the same as proving AI can run a hospital.
Why the Bigger Picture Is More Mixed
Broader evidence is less dramatic than a single headline win. A meta-analysis of 83 studies found an overall diagnostic accuracy of 52.1 percent for generative AI and no significant difference from physicians overall, while expert physicians still performed better. That means AI can look very strong in some tests and still fall short in the average comparison. The gap between benchmark success and dependable real-world use remains the key issue.
Earlier reviews point in the same direction. One systematic review found AI could perform at a level comparable to clinicians in some fields, especially image-based diagnosis, and sometimes did better than less experienced doctors. Another study found a diagnostic tool could match the clinical accuracy and safety of human doctors in a triage setting. These results help explain why hospitals and medical schools keep testing AI, even as they stay cautious about handing over final decisions.
What This Means for Patients and Doctors
The practical meaning is clear: AI is becoming a powerful second reader, not a finished replacement for physicians. The strongest studies so far tend to use fixed questions, limited data, and scored answers. That setup favors systems that can process patterns quickly and stay consistent. It also favors tests that may not capture messy real life, where symptoms overlap, records are incomplete, and patients need judgment, trust, and follow-up.
Rheumatology has a technology gap. I’m building RheumaNext to help close it.
I’m Bartosz Fiałek – a rheumatologist and Founder & CEO of RheumaNext.
Years of clinical work showed me the same problems repeatedly: delayed diagnosis, fragmented information, limited access to…
— Bartosz Fiałek (@bfialek) August 9, 2026
For families already frustrated by long waits, rising costs, and rushed appointments, the research cuts both ways. It suggests hospitals may soon use AI to reduce missed diagnoses and help doctors sort difficult cases faster. It also shows why many people worry about overpromising technology before it proves itself in everyday care. The evidence so far does not show that AI has solved medicine. It shows that the contest inside narrow medical tests has already begun, and humans are not always winning it.
Sources:
reason.com, npr.org, nature.com, pmc.ncbi.nlm.nih.gov, erictopol.substack.com, hai.stanford.edu
