Social Series

Generative AI Shock Wave / ChatGPT Can Be Tricked into Generating Malware, Bomb-making Instructions

Reuters file photo
The ChatGPT logo and a computer motherboard are seen in this illustration.

The Yomiuri Shimbun

17:36 JST, April 25, 2023

The use of generative artificial intelligence tools like ChatGPT is spreading rapidly. While forecasts indicate the technology could revolutionize society, a number of problems linked to such tools have emerged. This is the first installment in a series examining issues with generative AI ahead of the Group of Seven summit in May, where international guidelines on the technology will be discussed.

***

In online forums where criminals exchange information, discussions about “ChatGPT loopholes” and “Ways to evade censorship” are common.

Use of ChatGPT for criminal purposes is prohibited under the service’s terms of use. The system is designed to not provide information to queries linked to crime. However, ways to trick the system can be found online.

Exploiting prompts that get ChatGPT to reveal information it has been designed to restrict — such as material that can be used for criminal purposes — is referred to as “jailbreaking.”

By using certain prompts, it is possible to get ChatGPT to generate computer viruses, write text for phishing emails that can be used to steal personal information, and reveal how to make explosives.

Takashi Yoshikawa, a senior malware analyst at the Tokyo-based security firm Mitsui Bussan Secure Directions, has been investigating the risks associated with generative AI technology.

By entering jailbreak prompts found online, Yoshikawa got ChatGPT to generate the source code of ransomware, a type of malicious software designed to block access to a computer system until a sum of money is paid.

When the code was run on a computer, the software encrypted the data on the machine, rendering it unusable.

Yoshikawa, 38, said: “Some beginners with limited knowledge of viruses use forums [where such information is shared]. It is a dangerous situation.”

Over 100 million users

ChatGPT responds to questions and instructions using natural language.

Developed by U.S. startup OpenAI, the service was released to the public in November, and within two months had more than 100 million users.

However, the technology is still developing. On April 11, OpenAI announced rewards of up to $20,000 for people who discover security flaws in the service, but the company is excluding jailbreaking techniques, as “addressing these issues often involves substantial research and a broader approach.”

Regarding jailbreaking, the European Union Agency for Law Enforcement Cooperation released a report in March stating, “It is of the utmost importance that awareness is raised on this matter, to ensure that any potential loopholes are discovered and closed as quickly as possible.”

Warning that ChatGPT could be used for terrorism and other criminal activities, the report also stated, “It will become easier than ever for malicious actors to perpetrate criminal activities with no necessary prior knowledge.”

Leaks of corporate secrets?

Generative AI may affect Japanese culture and entertainment.

AI image generators are expected to be used more often in the production of manga and anime, to make characters and backgrounds, leading to concerns that the copyrights of original data used for AI learning could be violated.

When it comes to expression that uses short phrases, such as haiku, tanka and other poems, quality work might get lost amid the huge volume created by AI.

There are also concerns about the leakage of confidential information, because some settings might allow information input by users to be used as data for learning.

“Your internet access is blocked.” This message appears on the screen when someone tries to browse on a work computer at the Mizuho Financial Group, Inc.

The group has been considering using ChatGPT, but at the end of 2022 it added ChatGPT to its list of websites to which access is restricted, to prevent the leakage of confidential information. “We took extra precautions,” a security official at the group said.

Honda Motor Co. and Hitachi Ltd. cautioned employees about the issue earlier this month, while SoftBank Corp. and other companies positive about utilizing ChatGPT for business operations are creating rules related to its use.

“Many people would give instructions like, ‘Summarize this internal document.’ Some restrictions are necessary,” said Ichiro Sato, a professor specializing in information public policy at the National Institute of Informatics.

An engineer working for a company in Tokyo received a notice from his company telling him not to use ChatGPT on work computers.

“ChatGPT is still immature as a service and we’re not absolutely sure about its transparency and safety,” the engineer said, expressing his understanding of the company’s policy.

The world is taking a defensive stance toward the new technology, which appeared just half a year ago.