Sometimes we get carried away with our own curiosity. Say, for example, you’re engrossed in a book and read about an experiment and wonder you could replicate its results. Well, I’ve finally got around to reading James Surowiecki’s The Wisdom of Crowds, and I’m really liking it. So much so, in fact, that I did get a little carried away, and decide to try to validate what he was saying by running a test on the wisdom of crowds up on Google+. It was a lot of work, and I finally have the results and want to share them with you here in this post.
For those of you unfamiliar with the key concepts behind the idea, I’d recommend looking at the Wisdom of Crowds Wikipedia entry. If you’re interested, then go ahead and read the book. I highly recommend it. Here’s the basic idea though: when we make it possible for people to aggregate their wisdom in independent, diverse and decentralized ways, the resulting wisdom of the crowd can be uncannily accurate.
A Runaway Experiment
This little experiment took off like wild fire; the response exceeded all my expectations many-fold. Within an hour, the post exceeded the 500-comment limit on Google+ posts. Luckily, my son was there and suggested I just post it again. So I did. Then that one hit the limit. And then another…and another. All-told, I received 2,238 valid guesses. Six hundred and forty-eight people shared the experiment with others on Google+.
It was so crazy, and I even had two or three people accuse me of intentionally architecting the experiment as a way to troll for comments and sharing of my post. I also had a number of people disagreeing that this wisdom of crowds approach was a valid form of collective intelligence.
Analyzing the Results
A week has now passed, the dust has settled and I’ve had a chance to analyze the results. It’s been a ton of work; hours and hours of inputting and then analyzing data.
Here is histogram of the results for the initial experiment:
In other words, the collective guess in the original post was off by 65 pieces of cereal.
Now, take a look at the results for the second push of data collection with the smaller set of 436 guesses. With this revised image, the collective wisdom comes much, much closer. It is now just 17 pieces of cereal away from the actual number. It makes me wonder what might have happened had I gotten as many guesses with this revised experiment as I had with the original. Would we have gotten even closer?
Either way, it’s not perfect, but certainly pretty darn good. Good enough to make me believe there is something to this whole idea of the “wisdom of crowds.”
One thing I want to call out is that out of the more than 2,600 participants, two people actually guessed “467” – the exact correct number of pieces of cereal: Ваня Колесник and Lauri Novak.
There were another two other people who were off by just one piece of cereal: Roberto Olayo and Sarah Stiles.
Calling All Statisticians
Here’s the thing – I took statistics in business school, but I’m no statistician. So, I’m posting the raw data in a Google Doc so that others can use it as they see fit. You have my permission to copy it for your own use, as long as you share the results.
I would really like for someone with a good statistical background to weigh in on a few questions that have arisen for me after doing this experiment:
- How good a guess is “450” in the first place, statistically speaking?
- Was my second implementation of the experiment statistically relevant with just the 436 responses?
- To what extent were people actually influencing one another with their answers? When you look at the way that the previous couple of comments are visible when you post a comment on the experiment’s post, you see how easy it would be to be influenced by the previous person’s guess. After inputting all those estimates by hand (with some help from my sons calling them out to me as I input them), I can tell you that there is no question that people influenced one another to some degree. You see little patterns of similar, consecutive guesses sprinkled throughout the data. I just don’t know how to measure that influence in a statistically meaningful manner. It’s all there in the data, however, and could be a very interesting question for someone to explore. It’s a big enough data set to pull some interesting conclusions about how we influence one another on social networks. Anyone?
“The wisdom of crowds” outlines three conditions for a group to be collectively intelligent: diversity, independence, and decentralization.
The design of this experiment was relatively successful on all counts, but far from perfect. The Google+ community that participated in it were probably diverse enough in terms of the way they think. My first implementation of the experiment, using comments, allowed for some degree of independence, since Google+ shields all but the most recent two responses unless you intentionally expand them, and I explicitly asked people not to look at others’ responses. But, people are naturally curious and social, and so just seeing those last two comments undoubtedly did compromise the independence variable. I corrected this in the second implementation by using a form to collect the data, but again, the data set was not nearly as large. Finally, the experiment was not as decentralized as I’d planned because, within an hour or so, it showed up on Google+ “What’s Hot“, which exposed a lot of people to my post, rather than the hundreds of other people’s versions of it – which is where I originally assumed most of the guessing would occur.
I think it’s also important to note that the accuracy of people’s guesses is probably constrained in some important ways by the fidelity of the signals they’re getting. In this case, all I gave them was an image to work with, and my initial version of that image may have caused confusion. Even putting that aside, just taking the picture at a funny angle or robbing it of the kind of context that could help hint at the vase’s size, are ways in which the sensory input one provides to the crowd would could affect its collective wisdom. In the case of the ox in the 1906 country fair, people were seeing it in person, and that’s a lot more context than seeing a picture of something on Google+. That just has to be a factor, don’t you think?
With that all said, the whole reason I was curious about this idea in the first place is that I’m looking for new systems and new approaches to helping organizations and people get much better at collaborating, coordinating and pooling wisdom. It still seems to me that there is potential to use social networking platforms, especially one like Google+ to help do that.
At the end of the day, this experiment did not prove that potential, but it does suggest there’s something worth exploring.
Cheerio image by Sean Ryan.
2 thoughts on “Testing the Wisdom of Crowds”
Hi, could I have permission to access this data (Google Docs isn’t letting me). I’d love to see them!
Yes, Annabelle. I just gave you permission. Also, I just ran into an interesting study that tests the Wisdom of Crowds across a very large set of inputs and applications:
Studying the “Wisdom of Crowds” at Scale