What have Manchester United, HFT and deep learning got in common?

Gaurav Chakravorty, co-founder of AI investment advisors qplum, likes to use sporting analogies to illustrate changing trends within finance. The way high frequency trading (HFT) seemed to work like magic in the old days reminds him of Manchester United under Sir Alex Ferguson.

Between 1993 and 2013 Manchester United won the English Premier League 13 times, an incredible record. The truth was Ferguson used a machinery that other clubs had not yet happened upon. He would scout clubs in Europe for talented youngsters and be willing to pay top dollar for young stars without a proven track record at a big club. The best example is Christiano Ronaldo. This "technology" when it worked looked amazing. But today almost every club uses the same method so it no longer works like it did for Manchester United – just look at Paul Pogba.

"So it's the same with HFTs," said Chakravorty. "In the crisis years they may have made a bunch of money. Now it's completely about using machine learning methods. They [the media] tend to make the focus of HFT all about latency, how fast you are. I mean fast helps. But it's sort of like being tall on a basketball court. It definitely helps but you don't win matches by being tall."

Chakravorty, a veteran of HFT, also says that describing it as a trading strategy is like saying that microwaving food is a recipe. Machine learning is key to what HFT actually does, which is attempting to ascertain the size of orders as they are being placed into the marketplace. "Inferring the real intent of market participants is what HFT is about. It used to be a much simpler problem back in the day when people were just placing orders without really thinking. Now it's tougher.

"So you would try to see different statistics: how long has the order been in the book, whether it has been placed just after a market event, or if it's just been placed when the market opens. And you try to gauge whether the size is abnormally large compared to other sizes. So every single feature of those orders would be passed into this robo-brain, which would give you a guess of what is going to happen."

Chakravorty explained there has now been a kind of cross-pollination where firms like Goldman Sachs Asset Management and some hedge funds have started using machine learning algorithms that HFT uses. "Sometimes they would get an ex-HFT guy to come in and redesign the execution system. So when they have to place an order of size 200,000 they break it up into small chunks of size 100 so that nobody can see that order, and place it at just the right time. Strictly speaking if everybody is using the same algorithms then there is no way to distinguish a dumb order from a smart one."

These days Chakravorty spends a lot of time explaining how deep learning algorithms can be applied to finance. We have seen an evolution from trend following in the 1980s, to more complex statistical arbitrage in the 90's, which was followed by machine learning and HFT coming to the fore around 2005. Most recently, big increases in computing power combined with the sheer amount of data available have provided the scale for deep learning approaches.

Rather than learning to predict things in a kind of hardcoded, supervised manner, deep learning techniques operate more like a human brain, filtering through data and learning the important signals. For example, a person driving down a road that hits a pothole will probably slow down; they don't have to hit 100 of them.

In order to try and encode the social science component of why a person might click on a certain advert, or trade a certain stock, deep learning algorithms use layers of nodes, some of which filter lots of data into summaries and then learn to make assumptions from these.

"Actually summarising data is an easier problem than making predictions," said Chakravorty. "It's about trying to learn how people trade. We are building this huge brain-like thing, which is looking at all the data in the markets and figuring out what are the summaries and from those summaries can I predict what will happen?"

Newsweek's AI and Data Science in Capital Markets conference on December 6-7 in New York is the most important gathering of experts in Artificial Intelligence and Machine Learning in trading. Join us for two days of talks, workshops and networking sessions with key industry players. With many of the world's leading capital market firms represented don't miss out - register now.