Benford's Law and Data Spread
|
 |
Benford's law is the observation that for many datasets, the distribution of their first significant digit follows a non-uniform law given by: Probability(leading digit =  ) =  . Thus the probability that the leading digit is 1 is 30%, while for the digit 7 to lead, the probability is merely 6%. An underlying reason for this is that data that spans many orders of magnitude has the errors within each order cancel out (see Details section). Thus datasets with large spread will naturally follow the law, while datasets with small spread will not. This Demonstration shows any of the scatter plots of 131 datasets derived from the data on countries in Mathematica; the points in a scatter plot are of the form (spread, Benford deviation), where spread is computed by eliminating extreme outliers and the deviation is the norm of the vector difference of the observed frequencies and the Benford predictions, normalized to lie between 0 and 1. Below the scatter plot are plots of the raw distribution, the agreement of the digit probabilities with Benford's law, and the distribution of the base-10 logarithms of the data. Note that the scatter plot supports the explanation remarkably well: all properties with large spread have small Benford deviation, and all properties with small spread have large Benford deviation. |
 |
 |
 |
 |
|
 |
|
 |
|
 The connection between Benford's law and the spread of the data is lucidly described in a paper by Rachel Fewster (see also chapter 21 of the book by Wagon). The idea is that when viewing a distribution in a base-10 log scale, the proportion of the  axis corresponding to numbers beginning with 1 is  2. Hence, so long as there are several orders of magnitude, the error between this proportion and the proportion of the area lying above these numbers should be small. R. M. Fewster, "A Simple Explanation of Benford's Law," The American Statistician, 63(1), 2009 pp. 26-32. S. Wagon, Mathematica in Action, 3rd ed., New York: Springer, forthcoming.
|
|