The Hazard of Multiple Comparisons

Initializing live version
Download to Desktop

Requires a Wolfram Notebook System

Interact on desktop, mobile and cloud with the free Wolfram Player or other Wolfram Language products.

The need to compare means of multiple datasets arises frequently in data analysis. Procedures such as ANOVA detect a difference among means but do not determine which of them are significantly different. We may be tempted to simply run -tests on each pair of means to detect these differences, but, as this Demonstration shows, doing so leads to a dramatic inflation of the true type I error (rate of false positives). Thus, when multiple comparisons are needed, a correction for this error inflation must be used.

Contributed by: Andy Ross (March 2011)
Open content licensed under CC BY-NC-SA



All datasets are simulated from a normal (0, 1) distribution and every pairwise comparison is made. The dashed line represents the desired type I error threshold and the individual points are the type I error rates observed in the simulation. The Bonferroni correction used here simply sets the type I threshold in the Monte Carlo simulation to where is the number of pairwise comparisons being made. Details about the Bonferroni method and other correction methods can be found at Bonferroni's method.

Feedback (field required)
Email (field required) Name
Occupation Organization
Note: Your message & contact information may be shared with the author of any specific Demonstration for which you give feedback.