Pokémon Go style avatar to help bridge communication gap with Deaf community


An exciting virtual sign language project under development at UCD will help deaf and non deaf people communicate more smoother.

Working alongside tech giant Microsoft, researchers at University College Dublin and Lero, an SFI funded research centre, have designed a system that translates the spoken word into Irish Sign Language (ISL) and vice versa.

With the assistance of deaf students studying at University College Dublin through its Access programme, a working prototype has been developed that can interpret hand gestures used in ISL.

Using augmented reality technology similar to that found in Pokémon Go, the programme uses a digital avatar to translate ISL into computer generated speech.

When a non-deaf person wears the headset, the avatar appears on screen translating the person’s sign language into speech.

Philip Power, a deaf student studying law at UCD, said his reaction when he first used the prototype was "Wow! Absolutely fantastic!' I think it would be a great benefit for myself, for students, for everyday life. It is s a very good idea.”

According to Leah Ennis McLoughlin, another deaf student studying at UCD and helping to develop the system, “It’s great to see deaf people finally being viewed as a priority.”

Dr Anthony Ventresque from the UCD School of Computer Science and Director of the UCD Complex Software Lab, who is leading the programme, hopes the team can further refine the system over the next three to five years so that it will be able to recognise facial expressions used during communication.

Pokémon Go is an augmented reality game for iOS and Android devices that allows users to capture and train avatars of their favourite Pokémon characters

The prototype developed at UCD uses a combination of Mircosoft’s Skype and the company’s augmented reality goggles HoloLens, as well as other tools such as LUIS language understanding, Azure cognitive services and Xbox depth camera technologies.

When complete, it will be capable of using a phone or headset to translate ISL into computer-generated speech, and changing speech back into ISL.

The research project is carried out in conjunction with Microsoft through its Skype4Good initiative.

By: David Kearns, Digital Journalist / Media Officer, UCD University Relations