Bigger faster data: Four trends that are upending the analyst world

No one needs to be reminded that the data world – and everything it touches, which is, basically, everything – is changing rapidly. For analysts, there four major trends that are driving the opportunities that will fuel their career adventure over these next few years. At the heart of these trends is a massive wave of data being generated and collected by organisations worldwide.

With this data analysts of the future can shift their focus from explaining the past to predicting the future. And in order to do this, they need to spend less time doing the same things over and over and more time doing brand new things. And accomplishing all these changes will require the analysts of the future to work together differently than they do now.

Trend 1: Bigger Larger Faster Data

You've probably already heard the fact that every two years we, as humans, are doubling the amount of data in the world. This literally exponential growth of data is impacting analysis in some big ways:

• Big data means new infrastructure: distributed computing like Hadoop.
• Large datasets mean new tools. Excel can no longer do the work it once did. We've seen analysts using Access to cut datasets down into Exceldigestible pieces. Note: this is not a sustainable strategy! If you haven't already, expect to start working with some new tools very soon.
• We are on the cusp of the real-time data revolution. Services like Kafka will enable organisations to apply their data products in real time, which will revolutionise everything from operations to customer service. The urgency of top-notch analytics will be paramount!

Brent Dykes Director, Data Strategy at Domo, said: "Traditionally, business decision makers have been accustomed to waiting days, weeks or even months to have ample information before they can make a high-quality decision based on past business performance. For fast-paced organisations like Amazon, the traditional approach to decision making is far too slow."

Trend 2: Predictive Analytics

The vast majority of time spent by the vast majority of today's analysts is on understanding data collected in the past, often in the form of reports and dashboards. Those days are coming to an end. The data and tools now available are allowing analysts to go beyond just convincing someone to do something and instead to often just do it themselves. For example:

• Using customer data to identify which customers are most likely to churn (stop being subscribers/customers) – offer them special deals automatically in order to keep them.
• Using Internet of Things (IoT) data to identify which machines in a factory are most likely to break down, and fix them before they cause a disruption to production. This is called "predictive maintenance", and not only does it reduce downtime, it can also substantially lower insurance rates.
• Using customer behaviour data to narrow down potential fraud cases for insurance companies. As the predictive model gathers more data, it becomes even better at figuring out which cases the company should focus their investigative resources on.

Allison Snow, Senior Analyst of B2B Marketing, Forrester, said: "It's key to recognise that analytics is about probabilities, not absolutes. Unlike traditional analytics, when applying predictive analytics, one doesn't know in advance what data is important. Predictive analytics determine what data is predictive of the outcome you wish to predict."

Trend 3: Automation of Tasks

Once upon a time, analysts built models in Excel, and once a month or so, they exported the model to PowerPoint and sent it to (or even printed it out for) the managers who relied on regular reports. Soon, there were too many reports, so maybe they used macros in Excel to automate the creation of reports. Or maybe they were lucky enough to have a dashboard program that had some automation functionalities built in. The future promises even more than this:

• Replicable data preparation flows/recipes that can be applied and customised easily and quickly to brand new sources of data and for brand new applications.
• Models scheduled to re-run regularly and produce a set of metrics that will determine whether or not they are performing as needed.
• Meta-reports: regular reports on the state of the many models deployed in production, so that analysts can feel comfortable and in control.

Mike Driscoll, Founder & CEO at Metamarkets, said: "Data analysts who don't organise their transformation pipelines often end up not being able to repeat their analyses, so the advice I would give to myself is the same advice often given to traditional scientists: make your experiments repeatable!"

Trend 4: Collaborative teams

Many organisations have been growing their analytics capabilities by building centralised analytics teams made up of analysts, data scientists, and subject matter experts. Perhaps you're a part of one of these teams. This structure is effective because it allows members of the team to share knowledge and expertise quickly, and because it gives the team greater visibility within the organisation so that the insights it generates can be propagated and used. However, as these organisations mature, they're finding that the best practice is to embed analytics team members within individual business units, where they face some new challenges with respect to collaboration:

• The value of collaboration is migrating from collaboration among analytics team members (horizontal collaboration) to that among business unit team members (vertical collaboration).
• More non-technical team members are engaging with data and models, so being able to involve them in the analytics process via graphical interfaces and dashboards is essential.
• Both synchronous collaboration – when multiple people work on the same project at the same time – and asynchronous collaboration – when different people can work on and refer to the same project but at various points in time – are key to a data-driven organisation.

Ben Hamner, Co-founder and CTO of Kaggle, said: "Most data scientists, and data science teams, have terrible practices for collaboration. The current default workflows have grown organically and are bad. You need to be really intentional to do a lot better, and this yields large gains in productivity and reducing painful frictions."

In conclusion, analysts come in all shapes and sizes and there is no doubt that there will be more analysts with a more diverse set of responsibilities in the world in the coming years. Change is inevitable and their future roles are being defined by the vast amount of data that is being generated today. The questions for any analyst now become: how do you prepare for the future and what role do you see yourself playing in the years to come?

To learn more download the free white paper from Dataiku titled: "The Analyst of the Future"

Jennifer Roubaud is the VP of the UK and Ireland for Dataiku, the maker of the all-in-one data science software platform Dataiku Data Science Studio (DSS), a unique advanced analytics software solution that enables companies to build and deliver their own data products more efficiently.