Differential entropy (also referred to as continuous entropy) is a concept in information theory which tries to extend the idea of the Shannon entropy, a measure of average surprisal of a random variable, to continuous probability distributions. It seems that the differential entropy is a genuine extension of the Shannon entropy, but it is not; in consequence is not a measure of uncertainty and information. For details please refer to the page of the Shannon entropy.

Definition

---

Let X be a random variable with a probability density function f whose support is a set $\backslash mathbb\; X$. The differential entropy $h(X)$ or $h(f)$ is defined as

$h(X)\; =\; -\backslash int\_\backslash mathbb\{X\}\; f(x)\backslash log\; f(x)\backslash ,dx.$

As with its discrete analog, the units of differential entropy depend on the base of the logarithm, which is usually 2 (i.e., the units are bits). See logarithmic units for logarithms taken in different bases. Related concepts such as joint, conditional differential entropy, and relative entropy are defined in a similar fashion. One must take care in trying to apply properties of discrete entropy to differential entropy, since probability density functions can be greater than 1. For example, Uniform(0,1/2) has differential entropy $\backslash int\_0^\backslash frac\{1\}\{2\}\; -2\backslash log2\backslash ,dx=-1$.

The definition of differential entropy above can be obtained by partitioning the range of X into bins of length $\backslash Delta$ with associated sample points $i\backslash Delta$ within the bins, for X Riemann integrable. This gives a quantized version of X, defined by $X\_\backslash Delta\; =\; i\backslash Delta$ if $i\backslash Delta\; \backslash leq\; X\; \backslash leq\; (i+1)\backslash Delta$. Then the entropy of $X\_\backslash Delta$ is

$-\backslash sum\_i\; f(i\backslash Delta)\backslash log\; f(i\backslash Delta)\backslash Delta\; -\; \backslash sum\; f(i\backslash Delta)\backslash log(\backslash Delta)\backslash Delta$.

The first term approximates the differential entropy, while the second term is approximately $-\backslash log(\backslash Delta)$. Note that this procedure suggests that the differential entropy of a discrete random variable should be $-\backslash infty$.

Note that the continuous mutual information $I(X;Y)$ has the distinction of retaining its fundamental significance as a measure of discrete information since it is actually the limit of the discrete mutual information of partitions of X and Y as these partitions become finer and finer. Thus it is invariant under quite general transformations of X and Y, and still represents the amount of discrete information that can be transmitted over a channel that admits a continuous space of values.

Properties of differential entropy

---

• For two densities f and g, $D(f||g)\; \backslash geq\; 0$ with equality if $f\; =\; g$ almost everywhere. Similarly, for two random variables X and Y, $I(X;Y)\; \backslash geq\; 0$ and $h(X|Y)\; \backslash leq\; h(X)$ with equality if and only if X and Y are independent.
• The chain rule for differential entropy holds as in the discrete case

$h(X\_1,\; \backslash ldots,\; X\_n)\; =\; \backslash sum\_\{i=1\}^\{n\}\; h(X\_i|X\_1,\; \backslash ldots,\; X\_i-1)\; \backslash leq\; \backslash sum\; h(X\_i)$.

• Differential entropy is translation invariant, ie, $h(X\; +\; c)\; =\; h(X)$ for a constant c.
• Differential entropy is generally invariant under arbitrary invertible maps. In particular, for a constant a, $h(aX)\; =\; h(X)\; +\; \backslash log\; \backslash left|\; a\; \backslash right|$. For a vector valued random variable X and a matrix A, $h(A\backslash mathbf\{X\})\; =\; h(\backslash mathbf\{X\})\; +\; \backslash log(\backslash det\; A)$.
• If a random vector $\backslash mathbf\{X\}\; \backslash in\; \backslash mathbb\{R\}^\{n\}$ has mean zero and covariance matrix K, $h(\backslash mathbf\{X\})\; \backslash leq\; \backslash frac\{1\}\{2\}\; \backslash loge)^n\; \backslash det\{K\}$ with equality if and only if X is jointly gaussian.

Example: Exponential distribution

---

Let X be an exponentially distributed random variable with parameter $\backslash lambda$, that is, with probability density function

$f(x)\; =\; \backslash lambda\; e^\{-\backslash lambda\; x\}\; \backslash mbox\{\; for\; \}\; x\; \backslash geq\; 0.$

Its differential entropy is then

$h\_e(X)\backslash ,$	$=-\backslash int\_0^\backslash infty\; \backslash lambda\; e^\{-\backslash lambda\; x\}\; \backslash log\; (\backslash lambda\; e^\{-\backslash lambda\; x\})\backslash ,dx$
	$=\; -\backslash left(\backslash int\_0^\backslash infty\; (\backslash log\; \backslash lambda)\backslash lambda\; e^\{-\backslash lambda\; x\}\backslash ,dx\; +\; \backslash int\_0^\backslash infty\; (-\backslash lambda\; x)\; \backslash lambda\; e^\{-\backslash lambda\; x\}\backslash ,dx\backslash right)$
	$=\; -\backslash log\; \backslash lambda\; \backslash int\_0^\backslash infty\; f(x)\backslash ,dx\; +\; \backslash lambda\; E*$
	$=\; -\backslash log\backslash lambda\; +\; 1\backslash ,.$

Here, $h\_e(X)$ was used rather than $h(X)$ to make it explicit that the logarithm was taken to base e, to simplify the calculation.

Differential entropies for various distributions

---

In the table below, $\backslash Gamma(x)\; =\; \backslash int\_0^\{\backslash infty\}\; e^\{-t\}\; t^\{x-1\}\; dt$ (the gamma function), $\backslash psi(x)\; =\; \backslash frac\{d\}\{dx\}\; \backslash Gamma(x)$, $B(p,q)\; =\; \backslash Gamma(p)\backslash Gamma(q)$, and $\backslash gamma$ is Euler's constant. $\backslash ln\; 2\backslash sigma^\{2\}\backslash Gamma\backslash left(\backslash frac\{n\}\{2\}\backslash right)\; -\; \backslash left(1\; -\; \backslash frac\{n\}\{2\}\backslash right)\backslash psi\backslash left(\backslash frac\{n\}\{2\}\backslash right)\; +\; \backslash frac\{n\}\{2\}$$\backslash left(1\; +\; \backslash frac\{n\_2\}\{2\}\backslash right)\backslash psi\backslash left(\backslash frac\{n\_2\}\{2\}\backslash right)\; +\; \backslash frac\{n\_1\; +\; n\_2\}\{2\}\; \backslash psi\backslash left(\backslash frac\{n\_1\; +\; n\_2\}\{2\}\backslash right)$

Table of differential entropies.

Distribution Name	Density	Entropy in nats
Uniform	$f(x)\; =\; \backslash frac\{1\}\{b-a\}$ for $a\; \backslash leq\; x\; \backslash leq\; b$	$\backslash ln(b\; -\; a)\; \backslash ,$
Normal	$f(x)\; =\; \backslash frac\{1\}\{\backslash sqrt\{2\backslash pi\backslash sigma^2\}\}\; \backslash exp(-\backslash frac\{(x-\backslash mu)^2\}\{2\backslash sigma^2\})$	$\backslash frac\{1\}\{2\}\backslash ln(2\backslash pi\; e\; \backslash sigma^2)$
Exponential	$f(x)\; =\; \backslash frac\{1\}\{\backslash lambda\}\; \backslash exp(-\backslash frac\{x\}\{\backslash lambda\})$	$1\; +\; \backslash ln\; \backslash lambda\; \backslash ,$
Rayleigh	$f(x)\; =\; \backslash frac\{x\}\{b^2\}\; \backslash exp(-\backslash frac\{x^2\}\{2b^2\})$	$1\; +\; \backslash ln\; \backslash frac\{\backslash beta\}\{\backslash sqrt\{2\}\}\; +\; \backslash frac\{\backslash gamma\}\{2\}$
Beta	$f(x)\; =\; \backslash frac\{x^\{p-1\}(1-x)^\{q-1\}\}\{B(p,q)\}$ for $0\; \backslash leq\; x\; \backslash leq\; 1$	$\backslash ln\; B(p,q)\; -\; (p-1)-\; \backslash psi(p\; +\; q)-\; (q-1)-\; \backslash psi(p\; +\; q)\backslash ,$
Cauchy	$f(x)\; =\; \backslash frac\{\backslash lambda\}\{\backslash pi\}\; \backslash frac\{1\}\{\backslash lambda^2\; +\; x^2\}$	$\backslash ln(4\backslash pi\backslash lambda)\; \backslash ,$
Chi	$f(x)\; =\; \backslash frac\{2\}\{2^\{n/2\}\; \backslash sigma^n\; \backslash Gamma(n/2)\}\; x^\{n-1\}\; \backslash exp(-\backslash frac\{x^2\}\{2\backslash sigma^2\})$	$\backslash ln\{\backslash frac\{\backslash sigma\backslash Gamma(n/2)\}\{\backslash sqrt\{2\}\}\}\; -\; \backslash frac\{n-1\}\{2\}\; \backslash psi\backslash left(\backslash frac\{n\}\{2\}\backslash right)\; +\; \backslash frac\{n\}\{2\}$
Chi-squared	$f(x)\; =\; \backslash frac\{1\}\{2^\{n/2\}\; \backslash sigma^n\; \backslash Gamma(n/2)\}\; x^\{\backslash frac\{n\}\{2\}\; -\; 1\}\; \backslash exp(-\backslash frac\{x\}\{2\backslash sigma^2)\}$
Erlang	$f(x)\; =\; \backslash frac\{\backslash beta^n\}\{(n-1)!\}\; x^\{n-1\}\; \backslash exp(-\backslash beta\; x)$	$(1-n)\backslash psi(n)\; +\; \backslash ln\; \backslash frac\{\backslash Gamma(n)\}\{\backslash beta\}\; +\; n$
F	$f(x)\; =\; \backslash frac\{n\_1^\{\backslash frac\{n\_1\}\{2\}\}\; n\_2^\{\backslash frac\{n\_2\}\{2\}\}\}\{B(\backslash frac\{n\_1\}\{2\},\backslash frac\{n\_2\}\{2\})\}\; \backslash frac\{x^\{\backslash frac\{n\_1\}\{2\}\; -\; 1\}\}\{(n\_2\; +\; n\_1\; x)^\{\backslash frac\{n\_1\; +\; n2\}\{2\}\}\}$	$\backslash ln\; \backslash frac\{n\_1\}\{n\_2\}\; B\backslash left(\backslash frac\{n\_1\}\{2\},\backslash frac\{n\_2\}\{2\}\backslash right)\; +\; \backslash left(1\; -\; \backslash frac\{n\_1\}\{2\}\backslash right)\; \backslash psi\backslash left(\backslash frac\{n\_1\}\{2\}\backslash right)\; -$
Gamma	$f(x)\; =\; \backslash frac\{x^\{\backslash alpha\; -\; 1\}\; \backslash exp(-\backslash frac\{x\}\{\backslash beta\})\}\{\backslash beta^\backslash alpha\; \backslash Gamma(\backslash alpha)\}$	$\backslash ln(\backslash beta\; \backslash Gamma(a))\; +\; (1\; -\; \backslash alpha)\backslash psi(\backslash alpha)\; +\; \backslash alpha\; \backslash ,$
Laplace	f(x) = \frac{1}{2\lambda} \exp(-\frac{ >x - \theta	$1\; +\; \backslash ln(2\backslash lambda)\; \backslash ,$
Logisitic	$f(x)\; =\; \backslash frac\{e^\{-x\}\}\{(1\; +\; e^\{-x\})^2\}$	$2\; \backslash ,$
Lognormal	$f(x)\; =\; \backslash frac\{1\}\{\backslash sigma\; x\; \backslash sqrt\{2\backslash pi\}\}\; \backslash exp(-\backslash frac\{(\backslash ln\; x\; -\; m)^2\}\{2\backslash sigma^2\})$	$m\; +\; \backslash frac\{1\}\{2\}\; \backslash ln(2\backslash pi\; e\; \backslash sigma^2)$
Maxwell-Boltzmann	$f(x)\; =\; 4\; \backslash pi^\{-\backslash frac\{1\}\{2\}\}\; \backslash beta^\{\backslash frac\{3\}\{2\}\}\; x^\{2\}\; \backslash exp(-\backslash beta\; x^2)$	$\backslash frac\{1\}\{2\}\; \backslash ln\; \backslash frac\{\backslash pi\}\{\backslash beta\}\; +\; \backslash gamma\; -\; 1/2$
Generalized normal	$f(x)\; =\; \backslash frac\{2\; \backslash beta^\{\backslash frac\{\backslash alpha\}\{2\}\}\}\{\backslash Gamma(\backslash frac\{\backslash alpha\}\{2\})\}\; x^\{\backslash alpha\; -\; 1\}\; \backslash exp(-\backslash beta\; x^2)$	$\backslash ln\{\backslash frac\{\backslash Gamma(\backslash alpha/2)\}\{2\backslash beta^\{\backslash frac\{1\}\{2\}\}\}\}\; -\; \backslash frac\{\backslash alpha\; -\; 1\}\{2\}\; \backslash psi\backslash left(\backslash frac\{\backslash alpha\}\{2\}\backslash right)\; +\; \backslash frac\{\backslash alpha\}\{2\}$
Pareto	$f(x)\; =\; \backslash frac\{a\; k^a\}\{x^\{a+1\}\}$	$\backslash ln\; \backslash frac\{k\}\{a\}\; +\; 1\; +\; \backslash frac\{1\}\{a\}$
Student's t	$f(x)\; =\; \backslash frac\{(1\; +\; x^2/n)^\{-\backslash frac\{n+1\}\{2\}\}\}\{\backslash sqrt(n)B(\backslash frac\{1\}\{2\},\backslash frac\{n\}\{2\})\}$	$\backslash frac\{n+1\}\{2\}\backslash psi\backslash left(\backslash frac\{n+1\}\{2\}\backslash right)\; -\; \backslash psi\backslash left(\backslash frac\{n\}\{2\}\backslash right)\; +\; \backslash ln\; \backslash sqrt\{n\}\; B\backslash left(\backslash frac\{1\}\{2\},\backslash frac\{n\}\{2\}\backslash right)$
Triangular		$\backslash frac\{1\}\{2\}\; -\; \backslash ln\; 2$
Weibull	$f(x)\; =\; \backslash frac\{c\}\{\backslash alpha\}\; x^\{c-1\}\; \backslash exp(-\backslash frac\{x^c\}\{\backslash alpha\})$	$\backslash frac\{(c-1)\backslash gamma\}\{c\}\; +\; \backslash ln\; \backslash frac\{\backslash alpha^\{1/c\}\}\{c\}\; +\; 1$

See also

---

• Information entropy
• Information theory
• Self-information

References

---

• Thomas M. Cover, Joy A. Thomas. Elements of Information Theory New York: Wiley, 1991. ISBN 0471062596

External links

---

Entropy | Information theory | Statistics | Randomness