General News

AI Demonstrates Passing Score on Medical Practitioner Exam in Japan

The Yomiuri Shimbun

15:27 JST, May 10, 2023

The latest version of an artificial intelligence model developed by U.S.-based OpenAI Inc. has demonstrated the ability to pass Japan’s national examination for medical practitioners, according to an international team of researchers from Japan and the United States.

However, the AI scored below the average score of human examinees, and gave some inappropriate answers, such as choosing an option to induce patients to euthanize themselves.

The team of researchers, including Jungo Kasai, a researcher at the University of Washington who specializes in natural language processing, used two of OpenAI’s interactive AI systems, ChatGPT and GPT-4, to take national medical practitioner examinations for the five years from 2018 to 2022.

ChatGPT failed all of the exams. But GPT-4, the latest system, exceeded the passing mark on all of them.

However, the AI models made blunders such as choosing drugs for pregnant patients that should not be administered to them. They also said that advice encouraging euthanasia for patients was an “appropriate handling.”

“It’s problematic that AI does not have knowledge about laws and rules of the Japanese medical field and overlooks important information such as a patient being pregnant,” Kasai said. Kasai added that these issues could be significantly improved in the future, for example, by having AI learn more Japanese language data.

In the United States, it has been reported that GPT-4 has produced passing results for a bar exam and the national medical practitioner examination.

Ryuichiro Higashinaka, a professor of informatics at Nagoya University, said: “I think the national examination for medical practitioners is relatively easy for AI to solve, as the differences among the options for an answer are easy to understand. In the future, if we patiently have AI learn the rules and regulations of the medical field and make use of them, it may help reduce the burdens on medical staff.”