This Demonstration shows the Karhunen–Loeve ("car-ou-nen loeff") directions for bivariate normal with unit variances and correlation . Notice that the directions only depend on the sign of . The ellipsoids for the quantiles become narrower as increases. In practice, with principal components, the data are first standardized to have mean 0 and variance 1, so unless the sign of the sample correlation changes, the directions remain the same as with even when computed from the data. The principal components are the scores obtained by projecting the data in the Karhunen–Loeve directions.
This Demonstration is for variables but the idea generalizes to as well. In most principal component applications, may be much larger.
I. T. Jolliffe, Principal Component Analysis, 2nd ed., New York: Springer, 2004.
T. Hastie, R. Tibshirani, and J. Friedman, The Elements of Statistical Learning: Data Mining, Inference, and Prediction, 2nd ed., New York: Springer, 2009.