Winning With Crowds

James Surowiecki, in his fascinating book, The Wisdom of Crowds, explores where large numbers of people can be very good at arriving at correct answers to complex problems. He looks at situations ranging from guessing the weight of cattle at agricultural fairs to complex algorithms for finding submarines. He concludes predictions from crowds work well when four conditions are met. The crowd must have a diversity of opinion, the ability to make independent decisions, a decentralised structure, and a mechanism to turn judgements into collective decisions.

Prediction markets are increasingly popular mechanisms for turning judgements into collective decisions. In 1988, the University of Iowa’s introduced its famous Electronic Market for elections the US Presidential election. In 1996, the Hollywood Stock Exchange captured press attention by letting people buy and sell prediction shares of movies, actors, and directors based on film revenues. In 1999, Z/Yen created ExtZy as a project to help students understand finance, allowing people to trade web entities such as countries based on news hits. There are many practical examples of companies using prediction markets to estimate computer sales or the effectiveness of marketing campaigns. All of these rely on the perceived wisdom of crowds. But how good are crowds?

RBS Insurance Claims Prediction Games

RBS Insurance has used prediction games about the future at recruiting events for new staff, e.g. young actuary events. The games ask participants to estimate likely insurance claims for a few classes of business in return for potential prizes and, of course, a discussion about a possible career with RBS Insurance. We thought that it would be interesting to evaluate the wisdom of crowds in estimating insurance claims and RBS Insurance kindly shared their data.

RBS Insurance uses four simple games – space car insurance, space travel insurance, pet robot insurance and dwelling pod insurance. Obviously these are really four contemporary insurance lines – cars, travel, pets and housing – dressed up for fun. RBS Insurance puts up a large poster with five variables for the insurance and fifteen examples of the average claim histories. RBS Insurance then gives participants the five variables for two new examples and asks them to guess the claim amounts. A sample poster is Figure 1 – Space Travel Insurance – where the variables include travel method, maximum holiday period, holiday activity, most frequently visited destination, and known travel ailments. RBS Insurance had five game trials, car, travel, pod and two pet insurance. Over all five games 210 players provided two guesses each.

Figure 1: Space Travel Insurance

In order to analyse the results we needed some comparators. Two comparators came to mind. First, the sophisticated, a Support Vector Machine (SVM) based on statistical learning theory ought to provide a reasonable automated guess. The SVM may not perform well on such a sparse data set, but should form a minimal comparator. We use SVM’s regularly in our statistical processing. Second, we needed a ‘naïve’ estimator, or easy to calculate benchmark that one might use for a simple approach. We chose averaging of the historic claims, a simple historic model.

Let’s look at the travel insurance game in some detail. The table below shows statistics on the players’ answers on travel insurance:

The players haven’t done badly, but how did they fare against the naïve estimator?

The players outperformed the Simple Historic Model, so perhaps the crowd is wise. But how well did the SVM do?

As it turns out, the SVM performed very well, much better than the players.

Conclusion

This small example shows that the crowds do a good job, better than a simple averaging, however not as well as sophisticated statistical tools. While this example is very small, it is encouraging that there is some wisdom in crowds. What we’d like to learn about in future is how well RBS Insurance recruits do in real life.

Further Reading

1. SUROWIECKI, James, The Wisdom of Crowds: Why the Many Are Smarter Than the Few, Little, Brown, 2004.

http://www.forecastingprinciples.com/marketsforforecasting/index.php/research