Category Archives: Wisdom of Crowds

Notes from James Surowiecki Talk at Mark Logic User Conference 2009

Posted on May 12, 2009 | Leave a comment

James started out by telling the story of an ox-weight guessing contest analyzed by Francis Galton. He thought the crowd, whose average guess was 1,197 pounds vs. a reality of 1,198, was a mix of a few smart people with a lot of dumb ones. In reality, the crowd was smart.

Under the right conditions, groups of people can be remarkably intelligent; even smarter than the smartest member of the group.

The jelly bean experiment usually produces:

  • 3-5% accuracy
  • Average better than 95%+ of anyone in the room

Who Wants to be a Millionaire example:

  • Phone-a-friend: “experts” get the right answer 2/3rds of the time
  • Audience poll: the crowd gets the right answer 90% of the time (and consider who the crowd is!)

Google’s entire business is built on the Wisdom of Crowds on the Internet — by using link structure as a voting mechanism.

Wikipedia a phenomenal example of collective labor (dk: despite the Maurice Jarre hoax revealed today)

Described another example of studying craters on Mars where groups did as well as geologists trained 5-7 years. Excerpt:

The result, in NASA’s words: “the automatically computed consensus of a large number of clickworkers is virtually indistinguishable from the inputs of a geologist with years of experience in identifying Mars craters.” And these people weren’t even being paid.

The racetrack is his favorite example of collective intelligence. The odds on horses are almost perfect predictors of race outcome. (In a study of seven-horse races at Belmont,) favorites predicted to win 33% and won 34%, fourth favored 12% of the time, won 12% of the time, et cetera. Almost perfect judgment. And the crowd of betters is not exclusively experts. A lot are not: cranks, rookies, those seeking a nice day at the track. “I only bet on chestnut-colored horses” — but somehow when you aggregate those bets, you get an accurate forecast.

There are companies starting to use these tools — e.g., prediction markets, attempts to use a market-based tool to predict outcomes. First one was done at a b-school at the University of Iowa. Idea: markets do a relatively good job (in general, not recently!!) of forecasting in a variety of circumstances. Can we use them to predict non-financial things? So they tried presidential elections. Since 1998 that market has done a better job than Gallup polls — election-eve forecasts off by 1.2%.

Now there are public markets for lots of things and lots of bets (e.g., will Michael Jackson be convicted, dk: will Lance win the 2009 Tour de France). See Intrade which call every sentate race correctly and 49/50 states in the recent presidential election.

HP, in the 1990s, set up an internal prediction market for printer sales. 25-30 people done on lunch hour, small financial incentives. That market was more accurate than the elaborate forecasting system 3/4ths of the time.

Eli Lilly doing this to forecast which drugs will make it through clinical trials. Microsoft used to predict when software projects will finish!

Reality is in big organizations information doesn’t often get from where it is to where it needs to be:

  • Hoarding: information is power
  • Fear, afraid to say what they think. Does boss want the truth?
  • Perverse incentives: budgeting systems get gamed

Obstacles get in the way. With right collective intelligence tools, the only incentive is to be right.

But it only works under certain circumstances:

  • Dysfunction: rioters, lynch mobs, market crazes
  • Corporate meetings: after 15 minutes we end up all now dumber than we were when we entered!

Three basic ingredients needed:

  • Aggregation: way to aggregate individual judgments into collective one. There are lots of way to do this: odds, markets, averages. Not talking about the suggestion box where a guy at the top selectively picks ideas.
  • Diversity: the collective opinion does not necessary equal “consensus.” The more diverse, the smarter and better the decisions. Diversity means different types of mistakes get made — uncorrelated errors. Diversity eliminates groupthink. The longer homogeneous groups talk, the dumber they get. Technique: Devil’s advocate, invented by the Catholic Church — appointed someone when considering canonization (i.e., Sainthood) process they would appoint (literally) an advocate for the Devil.
  • Independece. Want people relying on their own judgment and not immitating others. How many people wake up and say “I look forward to conforming today.” But we are nevertheless immitative beings. Experiment in Times Square: a guy gazes up at a window. When you put 5 people on a street corner looking up, 45% of others look up. With 8 people, 80% gaze up. Don’t go around the room asking people for a conclusion at the end of a meeting — want independence.

Actuary joke: three acturaries are hunting. First guy shoots at a duck and misses 20 feet to the right. Second shoots and misses 20 feet to the left. And the third guy shouts out “we got it!”

Alternate side of the street parking example. He lives in Brooklyn. If the other cars have not been moved to the other side, he doesn’t. And he’s never got a ticket.

Note that talkative people tend to dominate group discussions. This would be OK if talkative people were smart. However, there is no correlation between talkativeness and intelligence.

Submarine story: the USS Scorpion, lost at sea. Hopeless task to find, transmissions too infrequent. John Craven built a diverse team to try and locate it: scenario analysis. Asked the group to bet (using bottles of Scotch as the incentive) on scenarios and on variables (e.g., rate of descent). Ran the data through Bayes’ algorithm. Ship was eventually found 220 yards from where the model driven by Craven’s men predicted — and no one member of the team had actually predicted that spot.

Wow!

Great speech!

→ Leave a comment

Posted in #MLUC09, Wisdom of Crowds

The Madness of Mobs: Twitter and Swine Flu

Posted on April 28, 2009 | Leave a comment

In talking about web 2.0, we often think about ideas like mass collaboration, a participatory web, the web as a communication platform, and generally speaking The Wisdom of Crowds in building and establishing knowledge.

I’m a big believer in the power of functional (or wise) groups to make better decisions than even the most talented individuals. I learned this first-hand years ago when I took LDP at the Center for Creative Leadership and we did a survival exercise similar to the one detailed in table 4 of this document. In our exercise, every individual — including a Brigadier General — was outperformed by the group in prioritizing a list of items necessary for wildnerness survival.

So I believe that groups guess jellybean jar counts better than individuals, that PageRank generally works for finding web pages, that feedback (used to) work on eBay (until they said sellers can only say positive things), that Diggs are useful way to identify interesting content, that Wikipedia is a great way to build an encyclopedia (particularly a technology one), and generally most of the other stuff I’m supposed to believe as good, web 2.0, Silicon Valley guy.

I believe this so much that we invited James Surowiecki, author of The Wisdom of Crowds, to keynote our user conference coming soon on May 12-14, 2009. So I’m on board with the program.

But I also wonder about the opposite, what I’ll call The Madness of Mobs. From financial bubbles to looters to Spring Breakers to a dozen other examples, we can all find examples of where everything cuts exactly the opposite way: where a wise crowd transforms to a mad mob.

So I was quite interested to find this article, Swine Flu: Twitter’s Power to Misinform, which talks precisely about how the “mass brain” of Twitter appears to be shorting out when it comes to the topic of swine flu. Excerpt (edited for brevity, and bolding mine):

Thus, Unlike basic internet search — which has been already been used by Google to track flu trends — Twitter has introduced too much noise into the process: as opposed to search requests which are generally motivated only by a desire to learn, too many Twitter conversations about swine flu seem to be motivated by desires to fit in, do what one’s friends do, or simply gain more popularity.

In such situations this, there is some pathological about people wanting to post yet another status update containing the coveted most-searched words – only for the sake of gaining more people to follow them. And yet the bottom line is that tracking the frequency of Twitter mentions of swine flu as a means of predicting anything thus becomes useless. (However, there are plenty of non-Twitter options summed up nicely on Mashable)

Hum. I should probably cop a maybe-guilty plea on blogging on swine flu. Like moths to a flame, we bloggers are drawn to hot topics.

The article continues:

If you think that my concerns about context are overblown, here are just a few status updates from random Twitter users:

I’m concerned about the swine flu outbreak in us and mexico could it be germ warfare?

In the pandemic Spanish Flu of 1918-19, my Grandfather said bodies were piled like wood in our local town….SWINE FLU = DANGER

Good grief this swine flu thing is getting serious. 8/9 specimens tested were prelim positive in NYC. so that’s Tx, Mexico and now Nyc.

Be careful of the swine flu!!!! (may lead to global epidemic) Outbreak in Mexico. 62 deaths so far!! Don’t eat pork from Mexico!!

Swine flu? Wow. All that pork infecting people….beef and chicken have always been meats of choice

Be careful…Swine Flu is not only in Mexico now. 8 cases in the States. Pig = Don’t eat

If my reading list on Twitter was only restricted to the individuals who had produced the posts above, by now I would be extremely scared … In moments like this, one is tempted to lament the death of broadcasting, for it seems that the information from expert sources should probably be prioritized over everything else.

Now, I’m pretty sure the counter-arguments to The Madness of Mobs goes like this:

  • Not all groups are wise. The Wisdom of Crowds relies of wise groups.
  • You can’t cherry-pick the scariest contributions to argue that The Wisdom of Crowds doesn’t work. Much as the abortion page on Wikipedia is the result of a rugby scrum of passionate, oppositional forces, so will be the mass brain of Twitter on swine flu. You need to look at the whole picture.

In fact, Surowiecki outlines failures of crowd intelligence and finds root causes which include groups that are too homogeneous, too emotional, too centralized, too divided, and too imitative.

Hopefully, we’ll hear more from Jim on this topic at the user conference and, in the meantime, before enslaving yourself to The Wisdom of Crowds, ponder if your crowd is a wise one, and whether you’re actually dealing with The Madness of Mobs.

Related Information / Stories

Swine Flu Tracker Map

View H1N1 Swine Flu in a larger map

→ Leave a comment

Posted in Madness of Mobs, Swine Flu, Twitter, Web 2.0, Wisdom of Crowds

Surowiecki on Newspapers

Posted on December 26, 2008 | 1 Comment

Continuing my recent rants about newspapers, please see this interesting story in the New Yorker by James Surowiecki, author of The Wisdom of Crowds, entitled News You Can Lose.

On the source of the problems:

There’s no mystery as to the source of all the trouble: advertising revenue has dried up. In the third quarter alone, it dropped eighteen per cent, or almost two billion dollars, from last year. For most of the past decade, newspaper companies had profit margins that were the envy of other industries. This year, they have been happy just to stay in the black. Many traditional advertisers, like big department stores, are struggling, and the bursting of the housing bubble has devastated real-estate advertising. Even online ads, which were supposed to rescue the business, have declined lately, and they are, in any case, nowhere near as lucrative as their print counterparts.

While I’ve always seen publishers, and newspapers in particular, as challenged with The Innovator’s Dilemma, before reading this article for some reason I’d never associated them with another of my favorite essays, Marketing Myopia (PDF for sale), by Harvard marketing guru Theodore Levitt.

From Surowiecki:

Levitt argued that a focus on products rather than on customers led the companies to misunderstand their core business. Had the bosses realized that they were in the transportation business, rather than the railroad business, they could have moved into trucking and air transport, rather than letting other companies dominate. By extension, many argue that if newspapers had understood they were in the information business, rather than the print business, they would have adapted more quickly and more successfully to the Net.

While I love Levitt’s thoughts on marketing, the usual objection to Marketing Myopia is: say Penn Central Railroads had fully envisioned the future — just because they successfully ran a railroad, do you actually believe that Penn Central Airlines would have been a big success? Even fully informed, can you get there from here? Put differently, seeing the future and having the core competencies to compete in the future are two different things.

I have a similar objection on behalf of newspapers. It’s one thing to anticipate the whittling away of your classified ad business. It’s quite another to have the skillset and Internet savvy to come up with Craigslist. In fact, if you could travel back in time and tell the Tribune Company about their future, how much do think they could have changed?

I hate to be fatalistic here and yes, they wouldn’t have let themselves get involved in a highly leveraged buy-out — but financing strategy aside — do you think it would have changed much? Even with a fully informed visitor from the future whispering in their ear, do you think they ever could have made a Tribuneslist successful? And even if they could, what about economics. Craigslist runs a nationwide classified advertising platform and does it with about 25 staff.

Back to Surowiecki:

The peculiar fact about the current crisis is that even as big papers have become less profitable they’ve arguably become more popular. The blogosphere, much of which piggybacks on traditional journalism’s content, has magnified the reach of newspapers, and although papers now face far more scrutiny, this is a kind of backhanded compliment to their continued relevance. Usually, when an industry runs into the kind of trouble that Levitt was talking about, it’s because people are abandoning its products. But people don’t use the Times less than they did a decade ago. They use it more. The difference is that today they don’t have to pay for it.

He continues with a great soundbite:

The real problem for newspapers, in other words, isn’t the Internet; it’s us.

We want access to everything, we want it now, and we want it for free. That’s a consumer’s dream, but eventually it’s going to collide with reality: if newspapers’ profits vanish, so will their product.

This argument is right in line with the “free ride” concept that I blogged about a few days ago. He concludes:

For a while now, readers have had the best of both worlds: all the benefits of the old, high-profit regime—intensive reporting, experienced editors, and so on—and the low costs of the new one. But that situation can’t last. Soon enough, we’re going to start getting what we pay for, and we may find out just how little that is

The complete New Yorker story is here.

Related articles by Zemanta

→ 1 Comment

Posted in innovator's dilemma, Marketing, Newspaper, strategy, Wisdom of Crowds

Intrade Prediction Markets

Posted on October 8, 2008 | 5 Comments

I’m seeing increasing blogosphere references to Intrade prediction markets, so I thought I’d cruise over to intrade.com and have look. I first heard of the idea of using prediction markets in business over a decade ago from amateur economist and Business Objects co-worker Timo Elliott.

While I found the idea intriguing, Timo was way ahead of this time. In those days no “serious” business person would use anything “as random as a market” to do something important, like forecast sales. But inexplicably those same serious people were seemingly happy to have a market set a price for a share of their stock.

In any case, the whole concept was legitimized with the publication of The Wisdom of Crowds in 2004 and, seemingly overnight, using markets for predictors of all things became rather mainstream. Towards that end, businesses like Intrade appeared, allowing you to effectively buy and sell futures contracts in events, such as:

  • Will the Higgs boson be observed in 2008?
  • Will Obama win the presidency?
  • Will there be a magnitude 9.0 earthquake this year?
  • And most of all, will there be a recession in 2009?

Let’s see what Intrade has to say. On the boson, you can buy a contract that says it will be observed before the end of 2008 for only $2. (If the event happens by the date, the contract is worth $100; if it doesn’t, it expires worthless.) The price has plummeted due to the shutdown of the large Hadron collider.


On the presidential race, it seems as if the economic crisis has been good for Obama.


On the possibility of a magnitude 9.0 earthquake striking before the end of this year, while they’re only trading for $4, I’m still selling since there have only been four magnitude 9+ quakes in the last 118 years. You can buy or sell the contracts on Intrade. If you sold these at $4, the most you can make is $4/contract when it (hopefully) expires worthless on 12/31/08. By the way, buying a contract for $4 that pays $100 if an event occurs represents 24:1 odds (remember you get your bet back), which is actually quite consistent with the history (118/4 = 29.5). But remember 5/6th of the year is already past.


Finally, on whether they’ll be a recession in 2009, that contract costs nearly $80 and will deliver $100 if it happens, implying odds of 1:4. Note, that’s not 4:1, but 1:4 — i.e., a fairly certain outcome in the minds of the betters.

→ 5 Comments

Posted in Wisdom of Crowds