Polling the Crowd Within
Tuesday, June 10, 2008
By Wray Herbert
Imagine you’re a contestant on the TV trivia show Who Wants To Be a Millionaire? You’re poised to win a cool $32,000, but you’re stumped and annoyed. Who cares which country is the largest exporter of grapes? You could use your 50-50 "life line," which asks the computer to narrow the choices from four to two. But this is unlikely to help, because more often than not the computer deletes the two answers you’ve already ruled out on your own. You could use your “Phone-a-Friend” option, which means ringing your father at home. He’s sitting there with a pile of reference books waiting for your call. But he’s only got 30 seconds, and how would you look that up anyway? You can already imagine his resigned voice saying: “I’m really sorry, Mary. I’m afraid I just don’t know.”
But there’s a third option: You can use your “Ask-the-Audience" life line. You can poll the entire studio audience on the four possible answers, and their responses are instantaneously assembled into a bar graph. Invariably, this graph shows one overwhelming choice, and with rare exceptions the audience is right. “I’ll trust the audience,” you tell Regis. “Final answer.”
Good move. But why? No person in the audience is any more likely than you to know where grapes come from, yet the collective intelligence of the group is almost always a better bet than your best guess. Psychologists are very interested in this perplexing statistical phenomenon. If the crowd is always wiser than any individual, what does that say about the way knowledge is stored and arranged in our minds? And can it help us make better choices, even beyond game shows?
Psychologists Edward Vul of MIT and Harold Pashler of the University of California in San Diego decided to explore these questions in the laboratory. More specifically, they wondered if perhaps each of us carries around in our mind a “crowd”—with a range of knowledge like that of any real life crowd. If so, they reasoned, then it should be possible to get a more accurate answer by asking ourselves the same question more than once, and averaging the responses.
To test this idea, the psychologists created their own Internet quiz show. They asked participants a variety of questions about trivia, not unlike the questions one might get on Who Wants To Be a Millionaire: For example, “What percentage of the world’s airports are in the United States?” But then they asked some of the contestants to answer the same questions again, immediately. Others were asked three weeks later to answer the same questions. The idea was to sample the range of answers within an individual mind, and to see if the range changed over time. It seems logical that one’s first answer should exhaust a person’s best knowledge about the world’s airports, but what if that’s not true? What if each of us has a whole range of possible answers to that trivia question, and the best guess lies in the middle somewhere?
That’s actually what Vul and Pashler found when they ran the experiment. As reported in the July issue of the journal Psychological Science, the average of two guesses for any individual participant was better than either guess alone, regardless of the time between guesses. So polling the “crowd within” does indeed yield a statistically more accurate answer. What’s more, this internal crowd gets more independent-minded with time: Contestants who were asked to second-guess themselves three weeks later benefited even more by averaging their two guesses than did those who second-guessed themselves immediately. The psychologists speculate that the cognitive pull of the original answer loses its power and allows more mental flexibility over time. 
In other words, the researchers say, there may be some science behind the folk wisdom: “Sleep on it.”
But no individual contestant did as well as a large group. That is, second-guessing oneself always yielded an answer that was better than the first, but it was still a sampling of one mind—and no match for the wisdom of the collective mind. For that, it appears, your best option really is to ask the audience. In the real world, that’s called vox populi.
For more insights into the quirks of human nature, visit “We’re Only Human . . .” at www.psychologicalscience.org/onlyhuman. Selections from this weblog now appear in each issue of Scientific American Mind and on http://www.sciam.com/.
posted by Wray Herbert @ 1:40 PM
6 Comments:
I read elsewhere of a researcher who bought stocks based on the recommendations of random people on the street, but clearly there must be some special class of question where this effect yeilds useful results, others where it does not.
For example, if this effect is true at the face value of it, then why do crowds make such absurd or cliche choices? Why, for example, would they catapult the untalented to stardom, overwhelmingly favour McDonalds over food and elect George Bushes ... twice?
Wisdom of the Crowds vs. Groupthink
To oversimplify: crowds often make absurd choices because they fail to average.
Averaging treats each estimate as independent, while people in actual crowds tend to follow the majority, the loudest, or the perceived expert, sometimes leading a group to make a decision that each person in the group thinks is bad.
Yes it is an interesting question, and it has been studied. We know some of the conditions for creating "wisdom" versus "groupthink" .
(With luck further discussion will address substance, not flame your specific choice of examples.)
I'm still struggling with your group-wisdom idea here as opposed to the group-flock behaviour we see in markets, crowds and elections, so here's my attempt to put what you've said into different words: would you say that the three answers to the question "what is your confidence level in Candidates A, B, and C" would, when averaged, produce a better choice-weight result than letting each person go to the poll and choose their Top Pick? (even if it was the 23% vs the 12% and 8% confidence values) -- ie their ability to gage the candidates in isolation is more accurate, when summed and divided, then the total sum of their Pick One tallies?
Is there a better real-world example were we see this phenomenon at work in the wild?
I will add that I am a fan of the idea, that I want it to be true, but wishing it so doesn't mean it actually is. For example, Free Jazz is predicated on the quartet arriving at a better composition than the single planner doling out taking-turns solos, yet clearly, unless it is background, Free Jazz is not well received by the general population. That too may be a bad example :)
Read about Dr Vul and Dr Pashler’s article, “Measuring the Crowd Within: Probabilistic Representations Within Individuals”, in the Economist.
Quote: Second guesses made immediately improved accuracy by an average of 6.5%; those made after three weeks improved the accuracy by 16%.
For second guesses made 3 weeks later, is it possible respondents looked up the answers and/or asked their friends?
Chris: No. If respondents had looked up the answers, their second guesses would have been more accurate than the first guesses. But they were less accurate. The average beat both estimates. (Overall that is, not for every single person on every single question.)
mrG: I'm not sure candidate quality is a good example; we have strong feelings on both sides and no independent rating of the "right" answer.
Weighting our votes -- or using preference order voting -- would be interesting, but I don't think it's related to wisdom-of-the-crowds.
But as the article notes, on matters of fact, you should almost certainly "Ask-the-Audience". This has been demonstrated consistently in everything from estimating hog weight to predicting election results (esp. via prediction markets, which allow very fine weighting of our confidence).
Everything comes from one common informational field. We are all connected in one system, whether we are conscious of it or not. For the time being, we are in an unconscious state in relation to this system – as if sleeping. However, we are gradually awakening to the root causes of phenomenon such as this and, in fact, the roots of our entire existence. If we understand the system that we are each a part of, we can speed up the process of revelation. For more information see here: http://www.laitman.com/2008/08/the-crowd-within/
Post a Comment
Subscribe to Post Comments [Atom]
<< Home