Google boasts 'significant transition' in VR tech with new mixed reality concept

Google wants to make virtual reality a more inclusive experience by allowing spectators to see what a headset wearer is experiencing. The company is experimenting with new mixed reality technology that allows onlookers to 'see' inside a user's headset, as well as where they are in the virtual world.

Google said the tech marked a "significant transition" in virtual reality.

Anyone who has tried virtual reality before will know how isolating an experience it can be. While wearing a virtual reality headset, you are completely cut off from the outside world, experiencing sights and sounds others can't see. Meanwhile, anyone watching simply sees someone lumbering around with a clunky headset strapped to their face.

Using a combination of computer learning, machine vision and advanced 3D rendering techniques, the Google Research team and Daydream Labs were able to map a person's position in within virtual reality and place it against a green screen. When viewed on a monitor, this allowed spectators to see what was going on inside the VR headset.

Google also showed off an experimental concept called "headset removal", which allows spectators to see a person's face inside the VR headset. This is done by digitally scanning a person's face and eye movements and then building a dynamic 3D model of their head.

The 3D face model is then aligned with the visible portion of the face in the video stream, creating a see-through effect. Eye-tracking inside the headset allows the 3D model to mimic where the person is looking. It's a cool concept, if not a bit creepy in its current iteration.

"The end result is a complete view of both the virtual world and the person in it, including their entire face and where they're looking," said Google software engineer Avneesh Sud.

The technology will be made available to select YouTube creators in the near future, initially at its YouTube Spaces studios. It would certainly allow for more involving Let's Play videos, which increasingly involve YouTubers filming themselves playing games in virtual reality.

"While we have shown the potential of our technology, its applications extend beyond mixed reality," Sud said. "Headset removal is poised to enhance communication and social interaction in VR itself with diverse applications like VR video conference meetings, multiplayer VR gaming, and exploration with friends and family."