Harnessing digital tech for diversity

The Yomiuri Shimbun

16:49 JST, December 24, 2021

The following is excerpted from the presentation given by Natsumi Sasaki and Holly Casandra, who won the Grand Prize at the 10th Annual All Japan Student English Presentation Contest.

Holly: Did you know that around 15% of the world’s population has some type of disability?

In this era, digital technology like AI has greatly improved. Now, it is not unusual to see 3D human avatars or holograms used in many areas of our daily life, such as entertainment, online games and even customer service.

We thought it would be amazing to create a hologram to help people with disabilities. For this project, we decided to focus on people with deafness or those who are hard of hearing, because 3D human avatars can move and do hand gestures, and so we thought they could also perform sign language.

Our goal is to create a better study environment for students who are hard of hearing or deaf. To achieve this goal, we propose a holographic sign language generator for the classroom!

Sasaki: Can you imagine what this percentage represents? This is the percentage of deaf people in Japan. 0.36% of the population, or about 448,000 people, are deaf.

Holly: However, people who are hard of hearing are not included, which means, the actual number of people who are hard of hearing or deaf is certainly more than 0.36% of the population.

Some of you may be thinking, “but it’s still a very small part of the community.” However, we think these people are overlooked, and we strongly believe that they need and deserve more of our attention and support.

Sasaki: I would now like to outline what we see as the current situation of deaf or hard of hearing students in Japan.

According to data supplied by MEXT, the number of students that go to colleges/universities has increased little by little but steadily. In fact, in the last 10 years, the number of deaf or hard of hearing students attending universities has increased by about 1.4 times.

However, there are not enough sign language interpreters who can handle advanced academic subjects. With the increase of this number, schools need to provide an adequate amount of support and improve their learning environments even more.

Holly: These are some voices of college and university students who are hard of hearing or deaf. The first student, at the university she attended, said she wasn’t allowed to get a certain qualification because she had a disability. The classes that she could take were also limited.

The second student, she was supporting students with hearing disabilities as a note taker. However, she realized the awareness of the staff of the Student Support Division was low, and she also realized the lack of support and attention from the university.

Sasaki: For the third student, in the university that she went to, there were no note takers in all classes, and no voice recognition app was used. She was struggling to take the exam because she couldn’t understand the content of lectures that were provided by only oral explanation.

In fact, according to the Japan Student Support Organization, out of 1,170 schools there are only 56 schools that provide sign language translation support and 148 schools that provide note taking support for deaf students, which is far from enough.

Their voices motivated us to create something that would help them to be able to study more comfortably and hopefully expand their abilities and open up new opportunities for them.

What we came up with is a system called “i Hear U,” a way to generate holographic sign language in the classroom or lecture theater.

The special device will be a desktop portable 3D hologram generator. Now, although this system does not exist yet, the technology to create it does. We will partner with a leading holographic image producer and the best speech-to-text software developer to create this system.

Holly: Let me explain the functions.

In simple terms, this device will catch what the person says and translate it into sign language through a 3D human hologram. Initially the sign language will be based on the standard sign language used in Japan.

First, you will open the i Hear U app that is installed on your smartphone, make an account and pair it with your hologram generator. After it is paired and connected, the hologram image will appear from the projector in the classroom.

This device will catch the spoken words and translate them into sign language through a 3D hologram. In addition, it will also show the written words in English or Japanese in order to assist students who don’t know how to read sign language. Moreover, it may help them to learn grammar.

Now, can it also be used for output? The answer is YES. With your smartphone, you can type what you want to say and the text will appear above the hologram.

Sasaki: We plan to include features such as automatic mode and manual mode. In automatic mode, we can adjust the distance that we want and the device will only catch voices within the chosen distance automatically.

In manual mode, students can press the “listen” button on their own terms and timing. These two modes are to prevent the devices from catching the “unnecessary voices.” Maybe now some of you are wondering, “What if new words are created?” Our app will also include features that allow the user to register new words in sign language.

Holly: By using this app, students can communicate smoothly. Also, it will include all of the diverse students in one classroom. Therefore, it will allow them to be more interactive and creative.

This is our proposal for the digital transformation of schools. No more note takers, no more barriers due to disability and the inclusion of all the diverse students in one classroom. We want to make sure no one is left behind. And our system, i Hear U, will make it possible.