Entropy is a concept in thermodynamics (see thermodynamic entropy), statistical mechanics and information theory. The concepts of information and entropy have deep links with one another, although it took many years for the development of the theories of statistical mechanics and information theory to make this apparent. This article is about information entropy, the information-theoretic formulation of entropy. Information entropy is occasionally called Shannon's entropy in honor of Claude E. Shannon.

Introduction

---

The concept of entropy in information theory describes with how much randomness (or, alternatively, 'uncertainty') there is in a signal or random event. An alternative way to look at this is to talk about how much information is carried by the signal.

For example, consider some English text, encoded as a string of letters, spaces, and punctuation (so our signal is a string of characters). Since letter frequency for some characters is not very high (e.g. 'z'), while other letters are very common (e.g. 'e'), the string of characters is not really as random as it might be. On the other hand, since we cannot predict what the next character will be, it is, to some degree, 'random'. Entropy is a measure of this randomness, suggested by Shannon in his 1948 paper "A Mathematical Theory of Communication".

Shannon offers a definition of entropy which satisfies the assumptions that:

• The measure should be proportional (continuous) - i.e. changing the value of one of the probabilities by a very small amount should only change the entropy by a small amount.
• If all the outcomes (letters in the example above) are equally likely then increasing the number of letters should always increase the entropy.
• We should be able to make the choice (in our example of a letter) in two steps, in which case the entropy of the final result should be a weighted sum of the entropies of the two steps.

(Note: The Shannon/Weaver book makes reference to Tolman (1938) who in turn credits Pauli (1933) with the definition of entropy Shannon used. Elsewhere in statistical mechanics, the literature includes references to von Neumann having derived the same form of entropy in 1927, which may explain why von Neumann favoured the use of the existing term 'entropy'.)

Formal definitions

---

Claude E. Shannon defines entropy in terms of a discrete random event x, with possible states (or outcomes) 1...n as:

$H(x)=\backslash sum\_\{i=1\}^np(i)\backslash log\_2\; \backslash left(\backslash frac\{1\}\{p(i)\}\backslash right)=-\backslash sum\_\{i=1\}^np(i)\backslash log\_2\; p(i).\backslash ,\backslash !$

That is, the entropy of the event x is the sum, over all possible outcomes i of x, of the product of the probability of outcome i times the log of the inverse of the probability of i (which is also called i's surprisal - the entropy of x is the expected value of its outcome's surprisal). We can also apply this to a general probability distribution, rather than a discrete-valued event.

Shannon shows that any definition of entropy satisfying his assumptions will be of the form:

$-K\backslash sum\_\{i=1\}^np(i)\backslash log\; p(i).\backslash ,\backslash !$

where K is a constant (and is really just a choice of measurement units).

Shannon defined a measure of entropy (H = − p₁ log₂ p₁ − … − p_n log₂ p_n) that, when applied to an information source, could determine the minimum channel capacity required to reliably transmit the source as encoded binary digits. The formula can be derived by calculating the mathematical expectation of the amount of information contained in a digit from the information source. Shannon's entropy measure came to be taken as a measure of the uncertainty about the realization of a random variable. It thus served as a proxy capturing the concept of information contained in a message as opposed to the portion of the message that is strictly determined (hence predictable) by inherent structures. For example, redundancy in language structure or statistical properties relating to the occurrence frequencies of letter or word pairs, triplets etc. See Markov chain.

Shannon's definition of entropy is closely related to thermodynamic entropy as defined by physicists and many chemists. Boltzmann and Gibbs did considerable work on statistical thermodynamics, which became the inspiration for adopting the word entropy in information theory. There are relationships between thermodynamic and informational entropy. In fact, in the view of Jaynes (1957), thermodynamics should be seen as an application of Shannon's information theory: the thermodynamic entropy is interpreted as being an estimate of the amount of further Shannon information (needed to define the detailed microscopic state of the system) that remains uncommunicated by a description solely in terms of the macroscopic variables of classical thermodynamics. (See article: MaxEnt thermodynamics). Similarly, Maxwell's demon reverses thermodynamic entropy with information; but if it is itself bound by the laws of thermodynamics, getting rid of that information exactly balances out the thermodynamic gain the demon would otherwise achieve.

It is important to remember that entropy is a quantity defined in the context of a probabilistic model for a data source. Independent fair coin flips have an entropy of 1 bit per flip. A source that always generates a long string of A's has an entropy of 0, since the next character will always be an 'A'.

The entropy rate of a data source means the average number of bits per symbol needed to encode it. Empirically, it seems that entropy of English text is between 1.1 and 1.6 bits per character, though clearly that will vary from text source to text source. Experiments with human predictors show an information rate of 1.1 or 1.6 bits per character, depending on the experimental setup; the PPM compression algorithm can achieve a compression ratio of 1.5 bits per character.

From the preceding example, note the following points:

1. The amount of entropy is not always an integer number of bits.
2. Many data bits may not convey information. For example, data structures often store information redundantly, or have identical sections regardless of the information in the data structure.

Entropy effectively bounds the performance of the strongest lossless (or nearly lossless) compression possible, which can be realized in theory by using the typical set or in practice using Huffman, Lempel-Ziv or arithmetic coding. The performance of existing data compression algorithms is often used as a rough estimate of the entropy of a block of data.

A common way to define entropy for text is based on the Markov model of text. For an order-0 source (each character is selected independent of the last characters), the binary entropy is:

$H(\backslash mathcal\{S\})\; =\; -\; \backslash sum\; p\_i\; \backslash log\_2\; p\_i,\; \backslash ,\backslash !$

where p_i is the probability of i. For a first-order Markov source (one in which the probability of selecting a character is dependent only on the immediately preceding character), the entropy rate is:

$H(\backslash mathcal\{S\})\; =\; -\; \backslash sum\_i\; p\_i\; \backslash sum\_j\; \backslash \; p\_i\; (j)\; \backslash log\_2\; p\_i\; (j),\; \backslash ,\backslash !$

where i is a state (certain preceding characters) and $p\_i(j)$ is the probability of $j$ given $i$ as the previous character (s).

For a second order Markov source, the entropy rate is

$H(\backslash mathcal\{S\})\; =\; -\backslash sum\_i\; p\_i\; \backslash sum\_j\; p\_i(j)\; \backslash sum\_k\; p\_\{i,j\}(k)\backslash \; \backslash log\; \backslash \; p\_\{i,j\}(k).\; \backslash ,\backslash !$

In general the b-ary entropy of a source $\backslash mathcal\{S\}$ = (S,P) with source alphabet S = {a₁, …, a_n} and discrete probability distribution P = {p₁, …, p_n} where p_i is the probability of a_i (say p_i = p(a_i)) is defined by:

$H\_b(\backslash mathcal\{S\})\; =\; -\; \backslash sum\_\{i=1\}^n\; p\_i\; \backslash log\_b\; p\_i\; \backslash ,\backslash !$

Note: the b in "b-ary entropy" is the number of different symbols of the "ideal alphabet" which is being used as the standard yardstick to measure source alphabets. In information theory, two symbols are necessary and sufficient for an alphabet to be able to encode information, therefore the default is to let b = 2 ("binary entropy"). Thus, the entropy of the source alphabet, with its given empiric probability distribution, is a number equal to the number (possibly fractional) of symbols of the "ideal alphabet", with an optimal probability distribution, necessary to encode for each symbol of the source alphabet. Also note that "optimal probability distribution" here means a uniform distribution: a source alphabet with n symbols has the highest possible entropy (for an alphabet with n symbols) when the probability distribution of the alphabet is uniform. This optimal entropy turns out to be $\backslash log\_b\; \backslash ,\; n$.

Another way to define the entropy function H (not using the Markov model) is by proving that H is uniquely defined (as earlier mentioned) iff H satisfies 1) - 3):

1) H(p₁, …, p_n) is defined and continuous for all p₁, …, p_n where p_i $\backslash in$^* for all i = 1, …, n and p₁ + … + p_n = 1. (Remark that the function solely depends on the probability distribution, not the alphabet.)

2) For all positive integers n, H satisfies

$$

H\underbrace{\left(\frac{1}{n}, \ldots, \frac{1}{n}\right)}_{n\ \mathrm{arguments}} < H\underbrace{\left(\frac{1}{n+1}, \ldots, \frac{1}{n+1}\right).}_{n+1\ \mathrm{arguments}}

3) For positive integers b_i where b₁ + … + b_k = n, H satisfies

$$

H\underbrace{\left(\frac{1}{n}, \ldots, \frac{1}{n}\right)}_n = H\underbrace{\left(\frac{b_1}{n}, \ldots, \frac{b_k}{n}\right)}_k + \sum_{i=1}^k \frac{b_i}{n} H\underbrace{\left(\frac{1}{b_i}, \ldots, \frac{1}{b_i}\right)}_{b_i}.

Efficiency

---

A source alphabet encountered in practice should be found to have a probability distribution which is less than optimal. If the source alphabet has n symbols, then it can be compared to an "optimized alphabet" with n symbols, whose probability distribution is uniform. The ratio of the entropy of the source alphabet with the entropy of its optimized version is the efficiency of the source alphabet, which can be expressed as a percentage.

This implies that the efficiency of a source alphabet with n symbols can be defined simply as being equal to its n-ary entropy. See also Redundancy (information theory).

Derivation of Shannon's entropy

---

Since the entropy was given as a definition, it does not need to be derived. On the other hand, a "derivation" can be given which gives a sense of the motivation for the definition as well as the link to thermodynamic entropy.

Q. Given a roulette with n pockets which are all equally likely to be landed on by the ball, what is the probability of obtaining a distribution (A₁, A₂, …, A_n) where A_i is the number of times pocket i was landed on and

$P\; =\; \backslash sum\_\{i=1\}^n\; A\_i\; \backslash ,\backslash !$

is the total number of ball-landing events?

A. The probability is a multinomial distribution, viz.

$p\; =\; \{\backslash Omega\; \backslash over\; \backslash Tau\}\; =\; \{P!\; \backslash over\; A\_1!\; \backslash \; A\_2!\; \backslash \; A\_3!\; \backslash \; \backslash cdots\; \backslash \; A\_n!\}\; \backslash left(\backslash frac1n\backslash right)^P\; \backslash ,\backslash !$

where

$\backslash Omega\; =\; \{P!\; \backslash over\; A\_1!\; \backslash \; A\_2!\; \backslash \; A\_3!\; \backslash \; \backslash cdots\; \backslash \; A\_n!\}\; \backslash ,\backslash !$

is the number of possible combinations of outcomes (for the events) which fit the given distribution, and

$\backslash Tau\; =\; n^P\; \backslash$

is the number of all possible combinations of outcomes for the set of P events.

Q. And what is the entropy?

A. The entropy of the distribution is obtained from the logarithm of Ω:

$H\; =\; \backslash log\; \backslash Omega\; =\; \backslash log\; \backslash frac\{P!\}\{A\_1!\; \backslash \; A\_2!\; \backslash \; A\_3!\; \backslash cdots\; \backslash \; A\_n!\}\; \backslash ,\backslash !$

$=\; \backslash log\; P!\; -\; \backslash log\; A\_1!\; -\; \backslash log\; A\_2!\; -\; \backslash log\; A\_3!\; -\; \backslash cdots\; -\; \backslash log\; A\_n!\; \backslash$

$=\; \backslash sum\_i^P\; \backslash log\; i\; -\; \backslash sum\_i^\{A\_1\}\; \backslash log\; i\; -\; \backslash sum\_i^\{A\_2\}\; \backslash log\; i\; -\; \backslash cdots\; -\; \backslash sum\_i^\{A\_n\}\; \backslash log\; i\; \backslash ,\backslash !$

The summations can be approximated closely by being replaced with integrals:

$H\; =\; \backslash int\_1^P\; \backslash log\; x\; \backslash ,\; dx\; -\; \backslash int\_1^\{A\_1\}\; \backslash log\; x\; \backslash ,\; dx\; -\; \backslash int\_1^\{A\_2\}\; \backslash log\; x\; \backslash ,\; dx\; -\; \backslash cdots\; -\; \backslash int\_1^\{A\_n\}\; \backslash log\; x\; \backslash ,\; dx.\; \backslash ,\backslash !$

The integral of the logarithm is

$\backslash int\; \backslash log\; x\; \backslash ,\; dx\; =\; x\; \backslash log\; x\; -\; \backslash int\; x\; \backslash ,\; \{dx\; \backslash over\; x\}\; =\; x\; \backslash log\; x\; -\; x.\; \backslash ,\backslash !$

So the entropy is

$H\; =\; (P\; \backslash log\; P\; -\; P\; +\; 1)\; -\; (A\_1\; \backslash log\; A\_1\; -\; A\_1\; +\; 1)\; -\; (A\_2\; \backslash log\; A\_2\; -\; A\_2\; +\; 1)\; -\; \backslash cdots\; -\; (A\_n\; \backslash log\; A\_n\; -\; A\_n\; +\; 1)$

$=\; (P\; \backslash log\; P\; +\; 1)\; -\; (A\_1\; \backslash log\; A\_1\; +\; 1)\; -\; (A\_2\; \backslash log\; A\_2\; +\; 1)\; -\; \backslash cdots\; -\; (A\_n\; \backslash log\; A\_n\; +\; 1)$

$=\; P\; \backslash log\; P\; -\; \backslash sum\_\{x=1\}^n\; A\_x\; \backslash log\; A\_x\; +\; (1\; -\; n)\; \backslash ,\backslash !$

Change A_x to p_x = A_x/P and change P to 1 (in order to measure the "bias" or "unevenness", in the probability distribution of the pockets for a single event), then

$H\; =\; (1\; -\; n)\; -\; \backslash sum\_\{x=1\}^n\; p\_x\; \backslash log\; p\_x\; \backslash ,\backslash !$

and the term (1 − n) can be dropped since it is a constant, independent of the p_x distribution. The result is

$H\; =\; -\; \backslash sum\_\{x=1\}^n\; p\_x\; \backslash log\; p\_x\; \backslash ,\backslash !$.

Thus, the Shannon entropy is a consequence of the equation

$H\; =\; \backslash log\; \backslash Omega\; \backslash$

which relates to Boltzmann's definition,

$\backslash mathcal\{S\}\; =\; k\; \backslash ln\; \backslash Omega$,

of thermodynamic entropy, where k is the Boltzmann constant.

Properties of Shannon's information entropy

---

We write H(X) as H_n(p₁,...,p_n). The Shannon entropy satisfies the following properties:

• For any n, H_n(p₁,...,p_n) is a continuous and symmetric function on variables p₁, p₂,...,p_n.
• Event of probability zero does not contribute to the entropy, i.e. for any n,

$H\_\{n+1\}(p\_1,\backslash ldots,p\_n,0)\; =\; H\_n(p\_1,\backslash ldots,p\_n)$.

• Entropy is maximized when the probability distribution is uniform. For all n,

$H\_n(p\_1,\backslash ldots,p\_n)\; \backslash leq\; H\_n\backslash Big(\backslash frac\{1\}\{n\},\backslash ldots,\backslash frac\{1\}\{n\}\; \backslash Big)$.

Following from the Jensen inequality,

$H(X)\; =\; E\backslash Big\backslash Big(\; \backslash frac\{1\}\{p(X)\}\backslash Big)\; \backslash Big\backslash leq\; \backslash log\_b\; \backslash Big(\; E\backslash Big\backslash frac\{1\}\{p(X)\}\; \backslash Big\backslash Big)\; =\; \backslash log\_b(n)$.

• If $p\_\{ij\},\; 1\backslash leq\; i\; \backslash leq\; m,\; 1\backslash leq\; j\; \backslash leq\; n$ are non-negative real numbers summing up to one, and $q\_i\; =\; \backslash sum\_\{j=1\}^n\; p\_\{ij\}$, then

$H\_\{mn\}(p\_\{11\},\backslash ldots,\; p\_\{mn\})\; =\; H\_m(q\_1,\backslash ldots,q\_m)\; +\; \backslash sum\_\{i=1\}^m\; q\_i\; H\_n\backslash Big(\backslash frac\{p\_\{i1\}\}\{q\_i\},\backslash ldots,\; \backslash frac\{p\_\{in\}\}\{q\_i\}\; \backslash Big)$.

If we partition the mn outcomes of the random experiment into m groups with each group containing n elements, we can do the experiment in two steps: first, determine the group to which the actual outcome belongs to; then, find the outcome in that group. The probability that you will observe group i is q_i. The conditional probability distribution function for group i is p_i1/q_i,...,p_in/q_i). The entropy

$H\_n\backslash Big(\backslash frac\{p\_\{i1\}\}\{q\_i\},\backslash ldots,\; \backslash frac\{p\_\{in\}\}\{q\_i\}\; \backslash Big)$

is the entropy of the probability distribution conditioned on group i. This property means that the total information is the sum of the information gained in the first step, H_m(q₁,..., q_n), and a weighted sum of the entropies conditioned on each group.

Khinchin in 1957 showed that the only function satisfying the above assumptions is of the form

$H\_n(p\_1,\backslash ldots,p\_n)\; =\; -k\; \backslash sum\_\{i=1\}^n\; p\_i\; \backslash log\; p\_i$,

where k is a positive constant representing the desired unit of measurement.

Extending discrete entropy to the continuous case: differential entropy

---

The Shannon entropy is restricted to finite sets. It seems that the formula

$h*=\; -\backslash int\_\{-\backslash infty\}^\{\backslash infty\}\; f(x)\; \backslash log\; f(x)\backslash ,\; dx,\backslash quad\; (*)$

where f denotes a probability density function on the real line, is analogous to the Shannon entropy and could thus be viewed as an extension of the Shannon entropy to the domain of real numbers. Formula (*) is usually referred to as the continuous entropy, or differential entropy. Although the analogy between both functions is suggestive, the following question must be set: is the Boltzmann entropy a valid extension of the Shannon entropy? To answer this question, we must establish a connection between the two functions:

We wish to obtain a generally finite measure as the bin size goes to zero. In the discrete case, the bin size is the (implicit) width of each of the n (finite or infinite) bins whose probabilities are denoted by p_n. As we generalize to the continuous domain, we must make this width explicit.

To do this, start with a continuous function f discretized as shown in the figure.

As the figure indicates, by the mean-value theorem there exists a value x_i in each bin such that

$f(x\_i)\; \backslash Delta\; =\; \backslash int\_\{i\backslash Delta\}^\{(i+1)\backslash Delta\}\; f(x)\backslash ,\; dx$

and thus the integral of the function f can be approximated (in the Riemannian sense) by

$\backslash int\_\{-\backslash infty\}^\{\backslash infty\}\; f(x)\backslash ,\; dx\; =\; \backslash lim\_\{\backslash Delta\; \backslash to\; 0\}\; \backslash sum\_\{i\; =\; -\backslash infty\}^\{\backslash infty\}\; f(x\_i)\; \backslash Delta$

where this limit and bin size goes to zero are equivalent.

We will denote

$H^\{\backslash Delta\}\; :=-\; \backslash sum\_\{i=-\backslash infty\}^\{\backslash infty\}\; \backslash Delta\; f(x\_i)\; \backslash log\; \backslash Delta\; f(x\_i)$

and expanding the logarithm, we have

$H^\{\backslash Delta\}\; =\; -\; \backslash sum\_\{i=-\backslash infty\}^\{\backslash infty\}\; \backslash Delta\; f(x\_i)\; \backslash log\; \backslash Delta\; f(x\_i)$

$=\; -\; \backslash sum\_\{i=-\backslash infty\}^\{\backslash infty\}\; \backslash Delta\; f(x\_i)\; \backslash log\; f(x\_i)\; -\backslash sum\_\{i=-\backslash infty\}^\{\backslash infty\}\; f(x\_i)\; \backslash Delta\; \backslash log\; \backslash Delta.$

As $\backslash Delta\; \backslash to\; 0$, we have

$\backslash sum\_\{i=-\backslash infty\}^\{\backslash infty\}\; f(x\_i)\; \backslash Delta\; \backslash to\; \backslash int\; f(x)\backslash ,\; dx\; =\; 1$

and so

$\backslash sum\_\{i=-\backslash infty\}^\{\backslash infty\}\; \backslash Delta\; f(x\_i)\; \backslash log\; f(x\_i)\; \backslash to\; \backslash int\; f(x)\; \backslash log\; f(x)\backslash ,\; dx.$

But note that $\backslash log\; \backslash Delta\; \backslash to\; -\backslash infty$ as $\backslash Delta\; \backslash to\; 0$, therefore we need a special definition of the differential or continuous entropy:

$h=\; \backslash lim\_\{\backslash Delta\; \backslash to\; 0\}\; \backslash left[H^\{\backslash Delta\}\; +\; \backslash log\; \backslash Delta\backslash right=\; -\backslash int\_\{-\backslash infty\}^\{\backslash infty\}\; f(x)\; \backslash log\; f(x)\backslash ,dx,$

which is, as said before, referred to as the differential entropy. This means that the differential entropy is not a limit of the Shannon entropy for n → ∞

It turns out as a result that, unlike the Shannon entropy, the differential entropy is not in general a good measure of uncertainty or information. For example, the differential entropy can be negative; also it is not invariant under continuous co-ordinate transformations.

More useful for the continuous case is the relative entropy of a distribution, defined as the Kullback-Leibler divergence from the distribution to a reference measure m(x),

$D\_\{\backslash mathrm\{KL\}\}(f(x)\backslash |m(x))\; =\; \backslash int\; f(x)\backslash log\backslash frac\{f(x)\}\{m(x)\}\backslash ,dx$

The relative entropy carries over directly from discrete to continuous distributions, and is invariant under co-ordinate reparametrisations.

See also

---

• Entropy encoding
• Kolmogorov-Sinai entropy in dynamical systems
• Rényi entropy
• Perplexity
• Theil index

External links

---

• Information is not entropy, information is not uncertainty ! - a discussion of the use of the terms "information" and "entropy".
• I'm Confused: How Could Information Equal Entropy? - a similar discussion on the bionet.info-theory FAQ.
• Java "entropy pool" for cryptographically-secure unguessable random numbers
• Description of information entropy from "Tools for Thought" by Howard Rheingold
• A cool little java applet representing Shannon's Experiment to Calculate the Entropy of English

Entropy | Information theory | Statistics | Randomness

Entropie (Informationstheorie) | Entropía (información) | Entropie de Shannon | Entropia (teoria dell'informazione) | Shannon-entrópiafüggvény | Entropie (informatietheorie) | Entropia (teoria informacji) | Информационная энтропия | เอนโทรปีของข้อมูล | 熵 (信息论)