Adobe and Princeton develop software to automatically remove distracting photobombs from images

Researchers from Adobe and Princeton University have developed an advanced photo-editing system that is able to automatically remove distracting elements from images, such as cars passing by in the background, rubbish on the ground or even photobombing strangers.

When you're trying to take that perfect shot or have one taken of you, it's always a pity when everything looks perfect, but someone is walking past in the background, or the photographer wasn't able to angle the camera lens away from the empty can, bottle, fixture or junk in the background near the subject.

Of course, if you don't want to delete the shot, you could always have a go at removing distracting elements in Adobe Photoshop, using the Healing tool, Content Aware Fill or other methods – but sometimes this is quite complex, taking a lot of time and skill that casual users might not have.

So the researchers set out to find a way to instantly remove troubling elements from shots, using two data sets of images and getting help from Amazon's Mechanical Turk program, which employs lots of human workers who work remotely to help complete tasks such as testing websites, answering surveys and transcribing audio files for businesses and developers.

Instantly removing distracting elements in a photo

The researchers asked the Mechanical Turk workers to look at the first data set of 1,073 images and identify elements in the images that they felt were distracting.

Then Adobe created a free experimental iPhone app called Fixel that let users mark and remove unwanted distractions from existing photos. Many of the users agreed to donate their images to the research, and the researchers collected more than 5,000 pictures that had been retouched, exported, saved or shared.

Using the data they received, the researchers were able to create detectors and train the software to spot specific elements that should be removed. They also designed a weighting system that assigned a specific value to various different arrangements of colours and shapes in photos, which taught the program about what sort of items to remove.

Even moving objects like cars can be removed

"We have a specific car detector in the code because people often want to eliminate cars that wander into the frame," said Ohad Fried, a graduate student in computer science at Princeton University and the lead author of the research paper. "We have a face detector. If the face is large and in the centre of the photo, we probably don't want to remove it. But if it is coming in from the side, it might be a photobomb."

The results showed that the software was able to remove certain distracting elements, such as some exhibition labels next to an object in a museum, or a sign on the grass in the shadow of a tree.

However, the program still needs some work as it sometimes still identifies important parts of images as distracting elements. "In the future, it might become accurate enough to be used in a photo editing product," Fried said. "Right now, it is a very interesting research question."

Adobe's success has come from creating the market-leading photo editing software, but it only works on computers. However, with so many people now used to taking photos with smartphones and tablets, then using apps to edit them, Adobe badly needs to gain a firmer foothold in this market.

The company is believed to be preparing to launch a new mobile app called Project Rigel at its Max conference in October, but perhaps this research might eventually be part of an Adobe product.