VR gaming will get emotion recognition using new method

Virtual Reality (VR) is the new frontier of gaming and makes for an engaging experience. The selling point of VR is that it makes for an experience where the gamer feels he/she is inside the game itself, instead of playing games on a screen. There are no frame rate drops and no screen latency.

However, it can get even more engaging, using a new method called emotion recognition. Simply put, using the new method, a game will recognize the gamer's emotion and respond accordingly.

Researchers at the Yonsei University and Motion Device Inc. have developed a deep learning technique that will enable emotion recognition during VR gaming. The findings were presented in a paper titled, "Emotion Recognition in Gamers Wearing Head-mounted Display" at the 2019 Institute of Electrical and Electronics Engineers (IEEE) Conference on virtual reality and 3D User interfaces.

Researchers have tried to do this before, but have failed repeatedly. Reading emotions requires seeing the whole face, especially the eyes. VR gaming is done using head-mounted displays (HMDs) which makes it difficult to read emotions since the upper part of the face is obscured by the headset.

What the researchers have done is they have collated a series of images showcasing people's emotions while they are gaming, wearing these headsets. They found that these images can be used by convolutional neural networks (CNNs) DenseNet, ResNet and Inception-ResNet-V2 and can be used to predict people's emotions.

They formed a Radbound Faces Database of 8,040 images, of 67 subjects with different emotions and then covered the part that is generally obscured by HMDs.

When the accuracy of the neural networks was tested, one of them was found to be 90 percent accurate.

"We successfully trained three CNN architectures that estimate the emotions from the partially covered human face images. Our study showed the possibility of estimating emotions from images of humans wearing HMDs using machine vision," the researchers state in their paper.

According to the study, emotion recognition tools can be developed by combining this method with machine learning. They will definitely enhance VR gaming techniques and make for an engaging experience.

The method is currently in the most initial stage – only a research paper has been submitted. The hope is that it can be used in the future for VR gaming headsets such as the Sony PlayStation VR.