RoboHow: Wikipedia style database translates WikiHow recipes so robots can follow them

Before robots can even begin to think about taking over our jobs or trying to rule the world, they are going to have to first learn how to do the basic everyday things that humans take for granted. But this is easier said than done, as robots find it very difficult to understand how to handle objects so they do not drop.

So RoboHow, an EU-funded programme backed by nine European universities, has been working on the equivalent of a Wikipedia for robots that takes how-to guides, recipes and other instructions, and breaks them down into very simple steps robots can understand. The project is headed by Germany's Universität Bremen (UNIHB), which is focused on researching and developing robots capable of autonomous mobile manipulation.

Breaking it down so robots can start to understand

To this end, they have spent the last three years researching different ways to break down simple cooking instructions so they can be explained to robots, such as how to pour pancake batter into a frying pan, handle a spatula, tilt the spatula to shovel a pancake on to it and flip it.

"Common sense knowledge like moving the spatula to place the pancake on the plate – it's an implicit knowledge humans have, but it's extremely hard to make that explicit for a robot," Professor Michael Beetz from UNIHB's Dept of Computer Science and head of RoboHow told IBTimes UK.

"One approach to teaching robots is kinesthetic, where you physically take the robot by the hand, put a spatula in its hand, close its hand and perform the action you want it to learn. It's like teaching a child, so the robot remembers the exact motions."

But it is not just about teaching robots to repeat movements – robots have a great difficulty in understanding depth, so if there are glass bottles on a table, the robot perceives a table and two bottle caps floating in mid-air.

Combining improved perceptions with a database of instructions

The researchers have come up with a perception system that is able to fill in the gaps in a robot's knowledge, so it understand the properties of glass and realises there are bottles on the table.

At the 14th International Conference on Autonomous Agents and Multiagent Systems (AAMAS) in April, one of the biggest robotics conferences held in the world, the RoboHow researchers presented their system, and recommended it be used in conjunction with OpenEase, the free open online database they had built.

"The big development is that you have to accumulate the knowledge for robots and share that knowledge with other robotics researchers. So you might have recipes, but you have to programme that knowledge in a way that is in-depth enough the robot is able to understand it," explained Beetz.

OpenEase, which won an award at AAMAS, functions as a database that anyone, including interested internet users, can contribute to in aiding the development of artificial intelligence, and some of the experiments can be quite fun.

One way to teach the robots, for example, was to set up a virtual reality game and then allow users to connect to it using their game consoles. The players were asked to perform everyday tasks in virtual reality and by watching the footage of the player in the game, the robot was then able to learn how to perform the action itself.

Don't count on robots taking your jobs anytime soon

However, apart from being able to do things like making pancakes or pizza, buttering toast, making popcorn, frying Bavarian sausages or conducting chemical experiments, the robot is still not able to cope with other basic household tasks, such as screwing a light bulb or pulling a window blind cord.

The researchers are now making the robot repeat its experiments but see if it can make pancakes successfully using different tools, a different location and whether it can cook without being in a fixed location.

"If we can, with our robots, flip any pancake with any tool we have, then we can probably perform most of the challenging manipulation tasks that there are in daily life," said Beetz.

"But we need to teach the robot to manipulate, build a knowledge base and then connect the two. In future, we want the robot to be able to ask, 'How do I clean this chair?' and get the answer immediately."

However, Beetz does not see robots taking this knowledge and using it to make humans obsolete, but rather, to make their lives easier. "I don't expect robots to take away our jobs. Robots will for a long time take the jobs that are unhealthy or extremely repetitive, so I don't think our jobs are in danger," he stressed.

"Instead, imagine if you have a robot at home that can fetch objects for you. This would give a lot of independence to disabled or cognitively impaired people, who usually have to wait for a caregiver to help them get what they need."