Wisdom of Crowds

As , the probability of correct choice by an informed individual, is increased, the number of correct choices by the committee beats that of the individual by a larger and larger margin. The dot shows the expected number correct and the arrows represent one standard deviation.
This scenario is loosely based on the Academy Awards. We suppose there are 10 awards and four nominees for each award. Assume that for each award there is only one correct choice. Each award is chosen either by an individual selected at random from the Academy or by the whole Academy. Assume that the Academy is comprised of 50 people. Of these 50 people we suppose that there are 15 informed persons and 35 who are not informed. For the informed persons, the probability of a correct choice is and for each of the other choices. For the people not informed, each nominee has probability 0.25 of being chosen.


  • [Snapshot]
  • [Snapshot]
  • [Snapshot]


The exact mean and standard deviation are easily derived for the individual case. In the committee case, exact formulas may also be obtained but it is more expedient to simulate to get accurate approximations. By default, a precalculated table based on simulations is used, but you can reset recompute=True in the initialization section to recompute the table. This is only of interest if you wish to experiment with the program by trying other parameter settings.
This Demonstration is based on [1, Figure 8.11]. In [1] it is used to illustrate how a consensus vote among bagged or bootstrap-aggregated nonlinear class predictors yields improved predictions. The wisdom of the crowds concept is useful in other contexts—see The Wisdom of Crowds.
I have found that this concept is helpful for instructors of courses with large enrollment in the evaluation and marking of multiple-choice examinations. It is important to always examine the number of students selecting each possible item in these examinations ([2]). Often, no single student scores a perfect exam yet if a majority vote for each question were used, a perfect score would have been obtained. With large classes, the only exceptions to this are when there is an error in the question or its answer, or, in hopefully rare cases, where the material was poorly taught.
[1] T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., New York: Springer, 2009.
[2] A. I. McLeod, Y. Zhang, and H. Yu, "Multiple-Choice Randomization," Journal of Statistics Education, 11(1), 2003.
    • Share:

Embed Interactive Demonstration New!

Just copy and paste this snippet of JavaScript code into your website or blog to put the live Demonstration on your site. More details »

Files require Wolfram CDF Player or Mathematica.