A Limit Theorem from Information Theory

Requires a Wolfram Notebook System

Interact on desktop, mobile and cloud with the free Wolfram CDF Player or other Wolfram Language products.

Requires a Wolfram Notebook System

Edit on desktop, mobile and cloud with any Wolfram Language product.

How many distinct sequences can be created by rearranging binary symbols? If the binary symbols are, say, and , and if the fraction of 's is denoted , then the number of possible sequences is . When is zero or one there is only one possible sequence for any value of , but when , the number of possible sequences increases exponentially with . The logarithm of the number of possible sequences, expressed on a per symbol basis, is , and the limit can be interpreted as the average number of bits needed per symbol to describe a long binary sequence with symbol probabilities and . This Demonstration shows for with the limit .

Contributed by: John Custy (November 2008)
Open content licensed under CC BY-NC-SA


Snapshots


Details

detailSectionParagraph


Feedback (field required)
Email (field required) Name
Occupation Organization
Note: Your message & contact information may be shared with the author of any specific Demonstration for which you give feedback.
Send