How big data is predicting the future

The last few years have typified this state of flux. In the political world, French President Emmanuel Macron scored a decisive electoral victory over Marine Le Pen, the UK voted to exit from the European Union, while Donald Trump was, against pollster’s predictions, elected to the White House. World economies have experienced similar volatility, struggling to rid themselves of the dredges of the 2008 financial crisis, with global debt continuing to grow. Technologically, progress is being made at breakneck speed, particularly in fields such as artificial intelligence.

Unpredicted shifts across the breadth of society have caused deep divisions, forcing us to adapt to new realities: new governments, new technologies, new economic models, new social norms.

The role of big data

While it’s not possible to stop the future from changing, by generating and analysing big data, prediction markets give us the next best alternative: the ability to predict the ever-changing future.

Harnessing a psychological theory known as the wisdom of the crowd – a phenomenon by which large groups of people make considerably better judgements and forecasts than lone individuals – predictive markets allow people to speculate on the probabilities of things to come; the larger the group, the larger the pool of data, the more accurate the prediction. By aggregating the opinions of thousands of participants, everything from sports outcomes to swings in the stock market can be forecast with surprising accuracy.

In the end, it’s like any prediction: Brazil has a 17 per cent chance of winning the 2018 World Cup, Democrats have a 58 per cent chance of triumphing at the next US Presidential election, and so on. It’s all based on statistical analysis.

The reliability of big data

While prediction markets aren’t infallible, they’ve shown remarkable accuracy in recent years. One of the best reflections of their accuracy is their uncanny skill in forecasting political elections.

In 2008, the prediction market Intrade forecasted that Barack Obama was going to secure 364 votes within the electoral college, and with it the US Presidency. The actual result? He won with 365 electoral-college votes.

In 2012, the same prediction market accurately forecasted the results of the second Obama win with correct predictions in 49 of the 50 states. Would Intrade have predicted the 2016 election of Donald Trump? We’ll never know for sure, as the platform was no longer operating at that point.

Over history, prediction markets perform better than even Gallup polls. For the former, average error on presidential elections is 1.4 per cent, beating the equivalent number for Gallup, at over 2 per cent.

Understanding why prediction markets work

The phenomenon of “the wisdom of the crowd” is one reason for the accuracy behind prediction markets. Anyone can join a prediction market, and the large numbers of participants are one of the factors that contribute to successful predictions. Groups of individuals, of course, aren’t always right. In general, groups of people tend to be more insightful than a single expert. This is especially true when complex subject matter and highly uncertain outcomes are involved.

Financial incentives are another factor that contributes to the accuracy of prediction markets. An analogy can be made to stock markets. Just as the stock market rewards investors for choosing stocks correctly, a prediction market financially rewards those who contribute accurate predictions.

Yet another reason for the accuracy of prediction markets is that they amplify good information and filter out the bad: when someone on social media insists that candidate X is going to win an election, there are no repercussions for an incorrect prediction. There’s also no way to qualify a person’s opinion against the opinions of others. Unless a person is a well-known expert in the field, weight is often given to whoever tweets the most or shouts the loudest.

Prediction markets, on the other hand, have a filtering mechanism, giving more creditability to those who “put their money where their mouth is.” When someone invests $1.00 to predict that candidate X will win, and another person invests $50.00 in the view that candidate Y will win, it’s generally accepted that the one risking more has better information and is more confident.

The future of prediction markets

As The Economist notes, 'The most heeded futurists these days are not individuals, but prediction markets'. Although several prediction markets have come and gone over the past few decades, blockchain technology – ideal for use in predictive markets owing to its inherent decentralisation and security – has infused new life into this valuable model.

Will political forecasters soon trust prediction markets to anticipate election outcomes? Will sports pundits eventually make the move from bookmakers to predictive markets? It’s hard to say- only time will tell. Nevertheless, the odds look good.