Notes from James Surowiecki Talk at Mark Logic User Conference 2009

James started out by telling the story of an ox-weight guessing contest analyzed by Francis Galton. He thought the crowd, whose average guess was 1,197 pounds vs. a reality of 1,198, was a mix of a few smart people with a lot of dumb ones. In reality, the crowd was smart.

Under the right conditions, groups of people can be remarkably intelligent; even smarter than the smartest member of the group.

The jelly bean experiment usually produces:

  • 3-5% accuracy
  • Average better than 95%+ of anyone in the room

Who Wants to be a Millionaire example:

  • Phone-a-friend: “experts” get the right answer 2/3rds of the time
  • Audience poll: the crowd gets the right answer 90% of the time (and consider who the crowd is!)

Google’s entire business is built on the Wisdom of Crowds on the Internet — by using link structure as a voting mechanism.

Wikipedia a phenomenal example of collective labor (dk: despite the Maurice Jarre hoax revealed today)

Described another example of studying craters on Mars where groups did as well as geologists trained 5-7 years. Excerpt:

The result, in NASA’s words: “the automatically computed consensus of a large number of clickworkers is virtually indistinguishable from the inputs of a geologist with years of experience in identifying Mars craters.” And these people weren’t even being paid.

The racetrack is his favorite example of collective intelligence. The odds on horses are almost perfect predictors of race outcome. (In a study of seven-horse races at Belmont,) favorites predicted to win 33% and won 34%, fourth favored 12% of the time, won 12% of the time, et cetera. Almost perfect judgment. And the crowd of betters is not exclusively experts. A lot are not: cranks, rookies, those seeking a nice day at the track. “I only bet on chestnut-colored horses” — but somehow when you aggregate those bets, you get an accurate forecast.

There are companies starting to use these tools — e.g., prediction markets, attempts to use a market-based tool to predict outcomes. First one was done at a b-school at the University of Iowa. Idea: markets do a relatively good job (in general, not recently!!) of forecasting in a variety of circumstances. Can we use them to predict non-financial things? So they tried presidential elections. Since 1998 that market has done a better job than Gallup polls — election-eve forecasts off by 1.2%.

Now there are public markets for lots of things and lots of bets (e.g., will Michael Jackson be convicted, dk: will Lance win the 2009 Tour de France). See Intrade which call every sentate race correctly and 49/50 states in the recent presidential election.

HP, in the 1990s, set up an internal prediction market for printer sales. 25-30 people done on lunch hour, small financial incentives. That market was more accurate than the elaborate forecasting system 3/4ths of the time.

Eli Lilly doing this to forecast which drugs will make it through clinical trials. Microsoft used to predict when software projects will finish!

Reality is in big organizations information doesn’t often get from where it is to where it needs to be:

  • Hoarding: information is power
  • Fear, afraid to say what they think. Does boss want the truth?
  • Perverse incentives: budgeting systems get gamed

Obstacles get in the way. With right collective intelligence tools, the only incentive is to be right.

But it only works under certain circumstances:

  • Dysfunction: rioters, lynch mobs, market crazes
  • Corporate meetings: after 15 minutes we end up all now dumber than we were when we entered!

Three basic ingredients needed:

  • Aggregation: way to aggregate individual judgments into collective one. There are lots of way to do this: odds, markets, averages. Not talking about the suggestion box where a guy at the top selectively picks ideas.
  • Diversity: the collective opinion does not necessary equal “consensus.” The more diverse, the smarter and better the decisions. Diversity means different types of mistakes get made — uncorrelated errors. Diversity eliminates groupthink. The longer homogeneous groups talk, the dumber they get. Technique: Devil’s advocate, invented by the Catholic Church — appointed someone when considering canonization (i.e., Sainthood) process they would appoint (literally) an advocate for the Devil.
  • Independece. Want people relying on their own judgment and not immitating others. How many people wake up and say “I look forward to conforming today.” But we are nevertheless immitative beings. Experiment in Times Square: a guy gazes up at a window. When you put 5 people on a street corner looking up, 45% of others look up. With 8 people, 80% gaze up. Don’t go around the room asking people for a conclusion at the end of a meeting — want independence.

Actuary joke: three acturaries are hunting. First guy shoots at a duck and misses 20 feet to the right. Second shoots and misses 20 feet to the left. And the third guy shouts out “we got it!”

Alternate side of the street parking example. He lives in Brooklyn. If the other cars have not been moved to the other side, he doesn’t. And he’s never got a ticket.

Note that talkative people tend to dominate group discussions. This would be OK if talkative people were smart. However, there is no correlation between talkativeness and intelligence.

Submarine story: the USS Scorpion, lost at sea. Hopeless task to find, transmissions too infrequent. John Craven built a diverse team to try and locate it: scenario analysis. Asked the group to bet (using bottles of Scotch as the incentive) on scenarios and on variables (e.g., rate of descent). Ran the data through Bayes’ algorithm. Ship was eventually found 220 yards from where the model driven by Craven’s men predicted — and no one member of the team had actually predicted that spot.


Great speech!

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.