Disney develops augmented reality app that turns coloured drawings into 3D animations

Disney Research, ETH Zurich and the Swiss university EPFL have teamed up to develop an augmented reality app that can take a completed picture from its official colouring book and turn it into a real-time 3D animated character.

The augmented reality colouring app has been trained to recognise 3D animated characters from regular 2D line-art representations of the same characters found in the official colouring book that has been programmed to work with it.

When the child starts colouring the page, the camera on a mobile device placed next to the child automatically detects the character being coloured in, and displays the 3D version in real-time, exactly the way the child coloured the character in.

The app even figures out what the back of the character should look like, mirroring the colouring from the image to look just like the front of the image.

The technology has inspired the iPhone and iPad app Disney Color and Play, which allows children to colour in favourite Disney Princesses, as well as characters from popular franchises like Frozen, Big Hero Six and the Mickey Mouse Clubhouse.

In order to create the 3D characters, parents have to buy special Disney colouring books that work with the app or use free official Disney colouring pages available online, or obtain the colouring pages through in-app purchases. Other colouring books, even of Disney characters, will not work with the app.

Animated characters increased interest in colouring books

The researchers presented their research at the IEEE International Symposium on Mixed and Augmented Reality (ISMAR 2015) in Fukuoka, Japan, explaining that they had to create a "lookup map" for each character that matched pixels on the side of the drawing that was visible to the user, to the back of the character.

The researchers carried out user testing with adults, rather than children, prior to the app being released on the App Store, and they found that that most of the users were more motivated to complete pages in colouring books thanks to the app, and that 80% of the users felt more connected to the characters.

"Augmented reality holds unique and promising potential to bridge between real-world activities and digital experiences, allowing users to engage their imagination and boost their creativity," said Robert W Sumner, a principal research scientist who leads the group on animation and interactive graphics at Disney Research.

"We are thrilled to have the opportunity to present the scientific advances behind this technology," continued Sumner, "and are especially happy that it is available to consumers, thanks to our cooperation with Disney Publishing."

The open-access paper, entitled "Live Texturing of Augmented Reality Characters from Colored Drawings" is available to read on the Disney Research website.