Wolfram Demonstrations Project

Level

confidence intervals and

-values are shown for 100 simple random samples of size

drawn from the specified population with mean

and standard deviation

. The

and

methods are compared for calculating the confidence intervals and

-values. The

method is based on the normal distribution using the estimated standard deviation in place of the unknown

. The

method is approximately valid for

large enough for all distributions considered in this Demonstration. The

method uses the Student

-distribution with

degrees of freedom; it is exact in the case of the normal distribution and is approximate for the other distributions considered in this Demonstration.

Corresponding to each confidence level for

, the two-sided

-value in the test of the null hypothesis (that the true mean equals 0) is shown to the right. These

-values are uniformly distributed between 0 and 1 in the normal case using the

method, and approximately so in the other cases.

It is interesting to do about 10,000 or more simulations to see how quickly the estimates

and

converge. To do this, click the icon beside the random seed slider, set the animation speed slower, and click the play button. Allow it to run for ten seconds or so.

This Demonstration can be used to show that the

-test is not conservative in small samples. It also illustrates the random coverage probability of confidence intervals and the random distribution

-values. Finally, the robustness of

methods can be investigated for various non-normal distributions. See the Details for further discussion.

Contributed by: Ian McLeod

Slider Zoom
Automatic Animation

Each of the distributions used has population mean 0 and variance 1, as indicated on the scale at the bottom of the left panel. The

-value is for testing

versus

. Of the three non-normal distributions considered, the

methods work almost as well for the uniform and Laplace distributions as the normal for sample sizes as small as 10, but for such small samples the exponential distribution generates

intervals and

-tests that are not conservative. As

increases, there is an improvement.

Snapshot 1: setting

and using the

-test method with the normal distribution and

, we find after some simulations that the empirical values for the estimates

and

are close to their theoretical values of 0.95 and 0.05

Snapshot 2: as in Snapshot 1, but using the

-test instead of the

-test; after a large number of simulations, we see the empirical coverage probability is about 92%, which means the confidence intervals are too narrow; similarly the type I error at 8% is well above its nominal 5% value; this demonstrates that the

-test method is not conservative in small samples

Snapshot 3: as in Snapshot 1, but with the uniform distribution; no observable difference from the normal population case

Snapshot 4: as in Snapshot 1, but with the Laplace distribution; this distribution has thicker tails than the normal, but even here the empirical coverage is not significantly different from the nominal 95% level, illustrating the robustness of the

-distribution method