ChatGPT Takes on the Tough U.S. Medical Licensing Exam

AFP-Jiji

13:00 JST, February 15, 2023

WASHINGTON (AFP-Jiji) — Dr. ChatGPT will see you soon.

The artificial intelligence system scored passing or near-passing results on the U.S. medical licensing exam, according to a study published on Feb. 9.

“Reaching the passing score for this notoriously difficult expert exam, and doing so without any human reinforcement, marks a notable milestone in clinical AI maturation,” said the authors of the study published in the journal PLOS Digital Health.

“These results suggest that large language models may have the potential to assist with medical education, and potentially, clinical decision-making,” they said.

ChatGPT, which is able to produce essays, poems and programming code within seconds, was developed by OpenAI, a California-based startup founded in 2015 with early funding from Elon Musk among others.

Microsoft invested $1 billion in OpenAI in 2019 and just inked a new multi-billion deal with the firm.

For the study, researchers at California-based AnsibleHealth tested ChatGPT’s performance on a three-part licensing exam taken by medical students and physicians-in-training in the United States.

The standardized exam tests knowledge in multiple medical disciplines from basic science to biochemistry to diagnostic reasoning to bioethics.

The AI system was tested on 350 of the 376 public questions on the June 2022 version of the exam, the study said, and the chatbot was not given any specialized training ahead of time.

Image-based questions were removed.

ChatGPT scored between 52.4% and 75% across the three parts of the exam.

A passing grade is around 60%.

According to the study, the first part of the exam, which focuses on basic science and pharmacology, is typically taken by medical students who have put in 300 hours to 400 hours of dedicated study time.

The second part is generally taken by fourth-year medical students and emphasizes clinical reasoning, medical management and bioethics.

The final section is for physicians who have completed at least six months to a year of postgraduate medical education.