Deep learning can read the tea leaves in market data

Henri Waelbroeck, director of research at machine learning trade execution system Portware, says rather poetically that the system "reads the tea leaves" in market data to distinguish different sorts of orders and execute trades more efficiently.

Portware uses artificial intelligence to help traders select the best algorithm for particular market conditions, asset class, broker, venue etc., interacting with the order flow and computing a mind-boggling array of variables in real time.

Say you are buying a stock, and you predict there is likely to be more orders hitting the bid side of the spread in the next five minutes, you should be able to operate an efficient algorithm that only posts limit orders and collects the spread as it executes. Using an algorithm that crosses the spread in this instance would be wasteful since you expect order flow to be coming your way.

Waelbroeck, formerly a professor at the Institute of Nuclear Sciences at the National University of Mexico, whose specialisms include genetic algorithms and chaos theory, said: "Just throwing machine learning at problems usually doesn't give a very good answer. You need to have a good analytical understanding of what's going on and this usually gives you a baseline model and then you find opportunities to insert machine learning tactically to exploit opportunities to improve the models."

The Portware Brain, which is currently in beta, embraces an entire spectrum of complexity from high volume, low latency trading to longer horizon investing and trade scheduling. One of things it does is look at a portfolio manager's trading history: some will be trading with the trend, in which case they are going to be competing with other managers for the same trades; others might be contrarian. "Of course it's never as simple as that," says Waelbroeck, "it depends on the circumstances; they can be contrarians one moment and trend followers the next".

Deep learning

The system observes what is driving order flow, looking for things like imbalances in the market. Where portfolio managers are competing with others you are going to find adverse order flow on the market and that may be detectable. Different sorts of orders are categorised as being high or low urgency, which enables the execution of trades more efficiently.

"We look at all sorts of other sources too. You can have a system that works well most of the time and fails spectacularly sometimes, and produces no net value.

"It's important to understand when circumstances are unique, and not reflective of the statistical average. We look at data streams like news options to identify situations that warrant either a different model, or perhaps walking away and not going to a default strategy with no specific prediction at all.

"So machine learning is used in both capturing the statistical average for normal situations and also at identifying outliers to know when not to use the routine model, so to speak," said Waelbroeck.

The Portware Brain is based on the concept of interacting agents. This framework enables the deployment of deep learning techniques, essentially processing data through an architecture of agents; each processes the information at their disposal and produces an output which is then consumed by the next agent and so on.

"You can have several agents briefing things and using the other agents' outputs as inputs. You can have potentially deep stacks of systems like this, similar to deep learning.

"The agents we deploy are essentially the basic machine learning techniques like regressions, Bayesian learning trees, something we call 'consensus agents', sort of a home invention. Combining these in different ways enables you to create potentially interesting model architectures," he said.

Concept drift

There have been a lot of advancements in deep learning in areas like image processing. But predicting the movement of asset prices using AI presents its own set of challenges. Traditional approaches to finance made wide use of normal distributions in stock market analysis to maximise return rates.

Today's methods use artificial neural networks which continuously monitor themselves, allowing for a non-stationary model of market behaviour. These applications have vast amounts of data arriving continuously and the target concept about which data is being collected may change over time, known as concept drift.

Waelbroeck said: "A lot of the things we are tracking are evolving systems and the market reacts to itself. You find a useful predictive driver for PMs short term alpha - well that may not be there next year.

"So anytime we do a study for a PM we always look at concept drift. We have developed techniques to identify drivers that are robust, but also we look at system stability and plasticity to identify the optimal trading training windows and retraining frequencies."

Regulation

Another interesting question is where AI-optimised execution meets the regulatory notion of best execution. These questions come up specifically with MiFIDII in Europe. It's not entirely clear if that is actually going to promote or hinder the development of new best execution capabilities.

Waelbroeck says that to some extent regulation is a double-edged sword: "On the one hand it focuses trading desks' attentions on best execution and encourages them to adopt advanced technologies to promote best execution; on the other hand regulation has a sort of contrarian effect of forcing people to stick to established processes.

"This is because any deviation from the sort of established state of the art needs to be explained and documented in regulatory filings.

"Those explanations can be complicated and expose the firm to regulatory scrutiny. So, creative efforts to create best execution sometimes run into the challenge of requiring changes in compliance processes."

Newsweek's AI and Data Science in Capital Markets conference on December 6-7 in New York is the most important gathering of experts in Artificial Intelligence and Machine Learning in trading. Join us for two days of talks, workshops and networking sessions with key industry players.