Zipf's Law Applied to Word and Letter Frequencies

Requires a Wolfram Notebook System

Interact on desktop, mobile and cloud with the free Wolfram CDF Player or other Wolfram Language products.

Requires a Wolfram Notebook System

Edit on desktop, mobile and cloud with any Wolfram Language product.

The frequency of words and letters in bodies of text has been heavily studied for several purposes, one being cryptography. This Demonstration performs an analysis of several texts, including fragments of popular works in several languages. It shows the distribution of frequencies, sorted from most common to least common.

[more]

Plotting word frequencies illustrates Zipf's law. This is a phenomenological law related to rank data frequencies, primarily of linguistic corpora. It says that the most frequent word will occur approximately twice as often as the second most frequent word, which will occur approximately twice as often as the fourth most frequent word. When the plot approximates a straight line it indicates that the data follows this law. It has also been shown that random bodies of text exhibit a word frequency distribution like Zipf's law, suggesting that the law is more a statistical phenomenon than specific to linguistics. The term has therefore come to be used to refer to any of a family of related power‐law probability distributions.

[less]

Contributed by: Hector Zenil (March 2011)
Open content licensed under CC BY-NC-SA


Snapshots


Details

detailSectionParagraph


Feedback (field required)
Email (field required) Name
Occupation Organization
Note: Your message & contact information may be shared with the author of any specific Demonstration for which you give feedback.
Send