p-Values Are Random Variables

We consider -values for or simulations using a random sample of size from a normal distribution with mean and unit variance to compute the two-sided -values for the test of the null hypothesis, versus using the -distribution method as implemented in the Mathematica function MeanTest. When , the -values are uniformly distributed on . With simulations, the result is obtained very quickly but there is more random variability in the histogram. Increasing to simulations takes less than three seconds on most modern computers and provides a more accurate result.
Mouseover the first rectangle to see the estimate of the probability of the power of a 5% test; the area of this rectangle represents observed probability of -values in the interval .
The slider changes the alternate hypothesis. When , the -values are no longer uniform on . The area under the first rectangle gives an estimate of the probability that the -value is less than 0.05. This is the estimated power of a two-sided test at the 5% level for
The distribution of the -values may also be visualized using a Q-Q plot in which the quantiles of the -values are plots against the corresponding quantiles from a uniform distribution.
  • Contributed by: Ian McLeod (University of Western Ontario)



  • [Snapshot]
  • [Snapshot]
  • [Snapshot]


-values are defined as the probability of observing a value as extreme or more extreme than the observed if the null hypothesis is true. Beginning students often do not realize that -values, just like confidence intervals, are random in repeated sampling and this point is often not discussed in elementary textbooks, as noted in [1].
For more on the Q-Q plot see [2].
[1] D. J. Murdoch, Y.-L. Tsai, and J. Adcock, "P-Values Are Random Variables," The American Statistician, 62(3), 2008 pp. 242–245.
[2] W. S. Cleveland, Visualizing Data, Summit, NJ: Hobart Press, 1993.
    • Share:

Embed Interactive Demonstration New!

Just copy and paste this snippet of JavaScript code into your website or blog to put the live Demonstration on your site. More details »

Files require Wolfram CDF Player or Mathematica.

Mathematica »
The #1 tool for creating Demonstrations
and anything technical.
Wolfram|Alpha »
Explore anything with the first
computational knowledge engine.
MathWorld »
The web's most extensive
mathematics resource.
Course Assistant Apps »
An app for every course—
right in the palm of your hand.
Wolfram Blog »
Read our views on math,
science, and technology.
Computable Document Format »
The format that makes Demonstrations
(and any information) easy to share and interact with.
STEM Initiative »
Programs & resources for
educators, schools & students.
Computerbasedmath.org »
Join the initiative for modernizing
math education.
Powered by Wolfram Mathematica © 2014 Wolfram Demonstrations Project & Contributors  |  Terms of Use  |  Privacy Policy  |  RSS Give us your feedback
Note: To run this Demonstration you need Mathematica 7+ or the free Mathematica Player 7EX
Download or upgrade to Mathematica Player 7EX
I already have Mathematica Player or Mathematica 7+