The Hazard of Multiple Comparisons

The need to compare means of multiple datasets arises frequently in data analysis. Procedures such as ANOVA detect a difference among means but do not determine which of them are significantly different. We may be tempted to simply run -tests on each pair of means to detect these differences, but, as this Demonstration shows, doing so leads to a dramatic inflation of the true type I error (rate of false positives). Thus, when multiple comparisons are needed, a correction for this error inflation must be used.


  • [Snapshot]
  • [Snapshot]
  • [Snapshot]


All datasets are simulated from a normal (0, 1) distribution and every pairwise comparison is made. The dashed line represents the desired type I error threshold and the individual points are the type I error rates observed in the simulation. The Bonferroni correction used here simply sets the type I threshold in the Monte Carlo simulation to where is the number of pairwise comparisons being made. Details about the Bonferroni method and other correction methods can be found at Bonferroni's method.
    • Share:

Embed Interactive Demonstration New!

Just copy and paste this snippet of JavaScript code into your website or blog to put the live Demonstration on your site. More details »

Files require Wolfram CDF Player or Mathematica.

Mathematica »
The #1 tool for creating Demonstrations
and anything technical.
Wolfram|Alpha »
Explore anything with the first
computational knowledge engine.
MathWorld »
The web's most extensive
mathematics resource.
Course Assistant Apps »
An app for every course—
right in the palm of your hand.
Wolfram Blog »
Read our views on math,
science, and technology.
Computable Document Format »
The format that makes Demonstrations
(and any information) easy to share and
interact with.
STEM Initiative »
Programs & resources for
educators, schools & students.
Computerbasedmath.org »
Join the initiative for modernizing
math education.
Step-by-Step Solutions »
Walk through homework problems one step at a time, with hints to help along the way.
Wolfram Problem Generator »
Unlimited random practice problems and answers with built-in step-by-step solutions. Practice online or make a printable study sheet.
Wolfram Language »
Knowledge-based programming for everyone.
Powered by Wolfram Mathematica © 2018 Wolfram Demonstrations Project & Contributors  |  Terms of Use  |  Privacy Policy  |  RSS Give us your feedback
Note: To run this Demonstration you need Mathematica 7+ or the free Mathematica Player 7EX
Download or upgrade to Mathematica Player 7EX
I already have Mathematica Player or Mathematica 7+