New Computer Algorithm Makes It Possible to See the Arrow of Time

If you know anything about Albert Einstein's theory of relativity, you'll know that he envisioned time as a spatial dimension, similar to the way height, weight, width and depth is quantified.

However, in theoretical physics, something cannot be considered to be a dimension if it's either based on three dimensions, or if it only goes in one direction, and time only seems to go one way – forward.

A group of computer scientists from MIT, Oxford University and the Max Planck Institute for Intelligent Systems disagrees. They claim to have found a way to show that not only does the arrow of time exist, but it is actually possible to see it going backwards, or forwards.

Detailed in a paper entitled "Seeing the Arrow of Time", the scientists outline a new algorithm that can show if video is running backwards or forwards accurately about 80% of the time.

Reversing time

"If you see that a clock in a movie is going backward, that requires a high-level understanding of how clocks normally move, but we were interested in whether we could tell the direction of time from low-level cues, just watching the way the world behaves," said William Freeman, a professor of computer science and engineering at MIT and one of the paper's authors.

"It's kind of like learning what the structure of the visual world. To study shape perception, you might invert a photograph to make everything that's black white, and white black, and then check what you can still see and what you can't.

"Here we're doing a similar thing, by reversing time, then seeing what it takes to detect that change. We're trying to understand the nature of the temporal signal."

The algorithm divides a frame of video up into a grid of hundreds of thousands of squares, then divides each of those squares into a smaller, four-by-four grid.

Then the algorithm determines how far and in what direction clusters of pixels move from one frame to the next, and generates this data into a "dictionary" of almost 4,000 four-by-four grids, which aim to show all the different examples of particular directions and degrees of motion.

A dictionary of grids

Once the database of different "terms" is complete, the algorithm then searches through the compiled grids, hunting for visual cues that show if different combinations of the four-by-four grids indicate either forward or backward motion.

"You see things that happen together and you want to figure out: 'Did one cause the other?' There's this whole area of study within statistics on, 'How can you figure out when something did cause something else?', and that relates in an indirect way to this study as well," explained Freeman.

You might be wondering what the point of this research is – well, it's actually to try to improve video signals, by creating a statistical relationship between a mathematical model of an object's motion and the "noise" or error, in the visual signal.

"Every signal is corrupted by noise," said Shai Avidan, an associate professor of electrical engineering at Tel Aviv University who was not involved in this work.

"If you understand the statistical properties of the underlying signal, you have a better chance of 'de-noising' a video sequence or compressing a video sequence."