The Kings of Data: How Dataminr uses billions of public tweets to monitor the entire world

If information is power, then Dataminr reigns supreme. With real-time access to every public Twitter post, the US-headquartered firm is uniquely placed offer its client's on-the-ground breaking news alerts from situations that really matter – terror attacks, school shootings, financial crashes.

"When you think about the advent of social media and what that has meant for the world it's completely changed how people communicate and how people publish information," Peter Bailey, Dataminr's chief strategy officer (CSO) told IBTimes UK in an exclusive interview.

Usually based in the bustle of New York, Bailey was in London to address the IBTimes/Newsweek AI and Data Science conference on 1 March. Behind the scenes, Bailey opened up about the firm's approach to data science and what the future holds.

"We are in the real time information business," he said after his keynote address. "The industry is how do we surface and detect that information right as its breaking and deliver it to professionals, operators, who need it to perform some critical function."

To put the scope of the operation into perspective his firm ingests between 500 million to a billion public tweets a day. It boasts close links to the social media platform (Twitter owns a 5% stake in the Dataminr) and Bailey says this sets his firm apart from the crowd – at least for now.

"We use a lot of different datasets but Twitter is the one that's really helpful because one of the use-cases of it is about telling the world what you are seeing," he said.

"When you think about leveraging the global army of social media users it's true that information that might first emerge on Facebook or YouTube or other platforms, but if it's something that people care about its almost instantaneous getting to Twitter.

"It really is a proxy for the real world at large."

Bailey said his firm's customers are roughly split into four main demographics: finance, government, large corporations and the media. For an undisclosed sum, Dataminr opens up the world of Twitter and delivers urgent alerts to help them stay one step ahead of the news.

"A lot of the work we do is informing investment professionals and traders about breaking events that are impactful for markets right as they are happening," he said, adding this can mean "anything unexpected that can have an impact on industries, markets or securities."

He continued: "We also sell to corporates, very large organisations that have a global footprint. Think of a large energy company or retail company, they have a complex supply chain, thousands of stores, infrastructure, pipelines, all different types of things they need to monitor.

"And we sell to the news media. We have over 12,000 journalists who use our product and that's a very exciting product what's cool about our alerting capabilities is that what we show is a tweet. So a journalist who receives an alert saying 'there's been a school shooting', the alert we are sending is the student who is in the middle of that, who is tweeting 'oh my god there's a gunman'."

Formed back in 2009, Dataminr has offices in New York City, Washington DC, London, Seattle and Montana. Like many data mining firms, it has reportedly received funding in the past by In-Q-Tel, the investment fork of the Central Intelligence Agency (CIA).

"Outside of finance we work with public sector organisations, including governments, who are learning about breaking events and rushing to the scene to try and manage and mitigate the impact of those," Bailey said. "So when people's lives are at risk the earlier the first responders can know about them the better they can protect and serve."

In October last year, a company called Geofeedia brought attention to the data mining scene after Twitter decided to take away its access to data over concerns it was enabling surveillance – a stance that went against its internal policies. But Bailey maintains Dataminr is different.

"We are passionate about the work we do with our government and security clients," he said. "We applaud Twitter for the way they think about protecting their users. I think it's very important for [the social network] to think about its users first.

"It is very familiar with what we do and we have been working with them for a long time.

"It knows what we do, that we don't surveil people, we basically are alerting on topics. So if you want to know about 'natural disasters' you can select 'natural disasters' and we will send you alerts about that. So you can't use our product to conduct surveillance."

It's a fascinating industry, one that surely likely to explode with competition as others realise its potential. Bailey remained confident his firm can remain the dominant player in the field.

"We have been injecting the full feed of tweets, all publicly available tweets we receive in real time in our back end and we have been processing that data for a number of years," he said. "We have been able to build a very unique [data] science on top of that.

"With the time and resources and the access to data someone else could so as well but that hasn't happened. So, it could, but while [...] we are the one who has access and the unique capability the pressure is on us to get it to market and become the dominant player."

And with the seemingly inevitable internet-of-things connecting the world more than ever before, Bailey said it will likely play a key role in the company's future.

"If you think about social media and how powerful it is as a signal in identifying something critical which one of our customer's needs to focus on, that is really just the front door to all the other types of data that could become relevant that's non-social," he said.

"The example that I bring up is an explosion at a chemical plant," he continued. "If that blows up and there's a plume of smoke in the sky and someone witnesses it and sends a tweet then that's something that we can detect and know this has happened.

"If you send that to a first responder trying to manage and deal with that situation, one of the questions that person is going to have is 'what's been released into the atmosphere'. Well, social media can't tell you that. But there are other big datasets which can. Whether its weather data, satellite data, IoT data, there are other datasets which are massive and growing exponentially."

The future, it seems, is data.

*This article was updated on 3 March 2017, at 16:48. Dataminr said terrorism is not an alert topic it offers clients. "That was inaccurate on our part," a spokesperson said.