Donald Trump was right about the polls getting it wrong

It's a clammy feeling, one unknown to many of us. But we may have to conclude that Donald Trump may be right. Imagine that...

Now clearly he's right in some ways, because he won, and winners are always right. That's the nature of it. Equally clearly, he's wrong in the many, many ways in which he's outraged and annoyed so many of us (if not as many of us as we thought).

But where he, increasingly, was correct was in his views on the polling industry. As with all politicians, his views on the validity of the polling number varied with his performance but as election day came closer (and his chances seemed to fade), he became increasingly jaundiced, saying "we don't believe in them any more".

Not unusually for him, they became a symbol of the liberal media elite and its bias against him and the working people he claimed to represent. Unusually, there is evidence that that may be right – that the unintentional bias is the nature of the questions and the condensing of a much more fractured political landscape back into the two-party model we've all been used to – makes for bad data. And even those on the inside concede he might have a point.

But not all the predictions were false. Modesty forbids spoke the day before the election about Florida liable to go for President Trump. Or that his freak show tactics were working. Or that voters weren't horrified by Trump's antics and turning to Clinton in any significant numbers. Or that Trump's bold gambles were beating Clinton's measured approach. And so on... It wasn't all correct, but the trends over the last months were favouring Trump, and the polling companies didn't notice.

And not for the first time – the British General Election and the Brexit referendum showed the real scale of the limitations of the traditional polling industry.

And the data for all those little successes above is all freely and publicly available on social media. You just need to look.

Each of those pieces was backed by insight and analysis from Impact Social, garnered from tens of thousands of posts on social media and open news sites. And then with the professional media and political figures removed. And then you end up with the views of actual voters – in all the cases above, tens of thousands of them in the key state of Florida. That is a sample. Now, you may be relying on scale, rather than "science" to make that sample something approaching representative, but where did that science get us? By not being restricted to datasets or by trying to match the political and demographic patterns of the voters from last time around, you can let the views form the answers, rather than drive those views with a set of questions.

Actually listening to what people volunteer as their opinions and thoughts, not prompting responses through questions gets a better picture. Direct yes/no/maybe questions forces people into positions they may not mean, while monitoring opinions freely offered on open platforms in a free country with high digital penetration is, quite simply, more rewarding. And the answers more honest than the Q+A approach which pushes people into looking for the "right" answer. Any reading of social media will show that people care less and less about how their honest opinions are received.

All that generates large mounds of data. The trick is in reading the conversation and letting it tell you the answers, rather than looking for answers in the data. That's done by human, not machine, intelligence. Looking for nuance, linguistic tricks and terminologies (machine reading would deem "shit hot" negatively for instance, whereas humans wouldn't).

Understanding data and understanding politics can be different skills. Which is why so many got it wrong this election – they (they thought) understood one of the other. The voices were out there and, if you knew where to look, they were pretty clear. The challenge for the pollsters lies in learning new skills and understand new platforms.

Trump, instinctively, knew the polls were wrong. If that doesn't force the polling industry into a long, dark night of the soul, nothing will.

Jimmy Leach is a digital consultant, working on platforms and communications for governments, corporations and start-ups.