Students sometimes ask about the difference between the regression line and the Karhunen–Loeve direction. The obvious answer a professor might give is that they are different animals! The object of this Demonstration is to give a more interesting answer.

The case of variables, and , is shown. It is assumed that and are bivariate normal with means zero, variances one, and covariance . For , the expected values for the regression lines on and on are shown together with the theoretical Karhunen–Loeve directions. For the estimated regressions and directions are shown along with a scatter plot of the data.

Consider the multivariate random variable with mean and covariance matrix . Then the Karhunen–Loeve directions are determined by columns of in the eigendecomposition , while the regression of on is

,

where is the matrix with the first row and column removed. In the present example, this simplifies to and similarly for the regression of on .

The related question based on data may be asked. In this case we just replace expectations by their sample estimates.