In statistics, the Cramér-Rao inequality, named in honor of Harald Cramér and Calyampudi Radhakrishna Rao, expresses an upper bound on the precision of a statistical estimator, based on Fisher information.

It states that the reciprocal of the Fisher information, $\backslash mathcal\{I\}(\backslash theta)$, of a parameter $\backslash theta$, is a lower bound on the variance of an unbiased estimator of the parameter (denoted $\backslash widehat\{\backslash theta\}$).

$$

\mathrm{var} \left(\widehat{\theta}\right) \geq \frac{1}{\mathcal{I}(\theta)} = \frac{1} { \mathrm{E} \left[ \left[ \frac{\partial}{\partial \theta} \log f(X;\theta) \right]^2 \right] }

In some cases, no unbiased estimator exists that realizes the lower bound.

The Cramér-Rao inequality is also known as the Cramér-Rao bounds (CRB) or Cramér-Rao lower bounds (CRLB) because it puts a lower bound on the variance of an estimator $\backslash widehat\{\backslash theta\}$

Example

---

Suppose X is a normally distributed random variable with known mean $\backslash mu$ and unknown variance $\backslash theta$. Consider the following statistic:

$$

T=\frac{\sum\left(X_i-\mu\right)^2}{n}.

Then T is unbiased for $\backslash theta$, as $E(T)=\backslash theta$. What is the variance of T?

$$

\mathrm{Var}(T) = \frac{\mathrm{var}(X-\mu)^2}{n}=\frac{1}{n} \left[ E\left\{(X-\mu)^4\right\}-\left(E\left\{(X-\mu)^2\right\}\right)^2 \right]

(the second equality follows directly from the definition of variance). The first term is the fourth moment about the mean and has value $3\backslash theta^2$; the second is the square of the variance, or $\backslash theta^2$. Thus

$\backslash mathrm\{var\}(T)=\backslash frac\{2\backslash theta^2\}\{n\}.$

Now, what is the Fisher information in the sample? Recall that the score V is defined as

$$

V=\frac{\partial}{\partial\theta}\log L(\theta,X)

where $L$ is the likelihood function. Thus in this case,

$$

V=\frac{\partial}{\partial\theta}\log\left^* =\frac{(X-\mu)^2}{\theta^2}-\frac{1}{2\theta}

where the second equality is from elementary calculus. Thus, the information in a single observation is just minus the expectation of the derivative of V, or

$$

I =-E\left(\frac{\partial V}{\partial\theta}\right) =-E\left(-\frac{2(X-\mu)^2}{\theta^3}+\frac{1}{2\theta^2}\right) =\frac{2\theta}{\theta^3}-\frac{1}{2\theta^2} =\frac{3}{2\theta^2}.

Thus the information in a sample of size $n$ is $\backslash frac\{3n\}\{2\backslash theta^2\}$.

The Cramer Rao inequality states that

$$

\mathrm{var}(T)\geq\frac{1}{I}.

In this case, the inequality is satisfied.

Regularity conditions

---

This inequality relies on two weak regularity conditions on the probability density function, $f(x;\; \backslash theta)$, and the estimator $T(X)$:

• The Fisher information is always defined; equivalently, for all $x$ such that $f(x;\; \backslash theta)\; >\; 0$,

$\backslash frac\{\backslash partial\}\{\backslash partial\backslash theta\}\; \backslash ln\; f(x;\backslash theta)$

is finite.

• The operations of integration with respect to x and differentiation with respect to $\backslash theta$ can be interchanged in the expectation of $T$; that is,

$$

\frac{\partial}{\partial\theta} \left[ \int T(x) f(x;\theta) \,dx \right] = \int T(x) \left[ \frac{\partial}{\partial\theta} f(x;\theta) \right] \,dx

whenever the right-hand side is finite.

In some cases, a biased estimator can have both a variance and a mean squared error that are below the Cramér-Rao lower bound (the lower bound applies only to estimators that are unbiased). See bias (statistics).

If the second regularity condition extends to the second derivative, then an alternative form of Fisher information can be used and yields a new Cramér-Rao inequality

$$

\mathrm{var} \left(\widehat{\theta}\right) \geq \frac{1}{\mathcal{I}(\theta)} = \frac{1} { -\mathrm{E} \left[ \frac{d^2}{d\theta^2} \log f(X;\theta) \right] }

In some cases, it may be easier to take the expectation with respect to the second derivative than to take the expectation of the square of the first derivative.

Multiple parameters

---

Extending the Cramér-Rao inequality to multiple parameters, define a parameter column vector

$\backslash boldsymbol\{\backslash theta\}\; =\; \backslash left\backslash theta\_1,\; \backslash theta\_2,\; \backslash dots,\; \backslash theta\_d\; \backslash right^T\; \backslash in\; \backslash mathbb\{R\}^d$

with probability density function (pdf), $f(x;\; \backslash boldsymbol\{\backslash theta\})$, that satisfies the above two regularity conditions.

The Fisher information matrix is a $d\; \backslash times\; d$ matrix with element $\backslash mathcal\{I\}\_\{m,\; k\}$ defined as

$$

\mathcal{I}_{m, k} = \mathrm{E} \left[ \frac{d}{d\theta_m} \log f\left(x; \boldsymbol(\theta)\right) \frac{d}{d\theta_k} \log f\left(x; \boldsymbol(\theta)\right) \right]

then the Cramér-Rao inequality is

$$

\mathrm{cov}_{\boldsymbol{\theta}}\left(\boldsymbol{T}(X)\right) \geq \frac {\partial \boldsymbol{\psi} \left(\boldsymbol{\theta}\right)} {\partial \boldsymbol{\theta}^T} \mathcal{I}\left(\boldsymbol{\theta}\right)^{-1} \frac {\partial \boldsymbol{\psi}\left(\boldsymbol{\theta}\right)^T} {\partial \boldsymbol{\theta}}

where

• $$

\boldsymbol{T}(X) = \begin{bmatrix} T_1(X) & T_2(X) & \cdots & T_d(X) \end{bmatrix}^T

• $$

\boldsymbol{\psi} = \mathrm{E}\left^* = \begin{bmatrix} \psi_1\left(\boldsymbol{\theta}\right) & \psi_2\left(\boldsymbol{\theta}\right) & \cdots & \psi_d\left(\boldsymbol{\theta}\right) \end{bmatrix}^T

• $\backslash frac\{\backslash partial\; \backslash boldsymbol\{\backslash psi\}\backslash left(\backslash boldsymbol\{\backslash theta\}\backslash right)\}\{\backslash partial\; \backslash boldsymbol\{\backslash theta\}^T\}$

= \begin{bmatrix} \psi_1 \left(\boldsymbol{\theta}\right) \\ \psi_2 \left(\boldsymbol{\theta}\right) \\ \vdots \\ \\ \psi_d \left(\boldsymbol{\theta}\right) \end{bmatrix} \begin{bmatrix} \frac{\partial}{\partial \theta_1} & \frac{\partial}{\partial \theta_2} & \cdots & \frac{\partial}{\partial \theta_d} \end{bmatrix} = \begin{bmatrix} \frac{\partial \psi_1 \left(\boldsymbol{\theta}\right)}{\partial \theta_1} & \frac{\partial \psi_1 \left(\boldsymbol{\theta}\right)}{\partial \theta_2} & \cdots & \frac{\partial \psi_1 \left(\boldsymbol{\theta}\right)}{\partial \theta_d} \\ \\ \frac{\partial \psi_2 \left(\boldsymbol{\theta}\right)}{\partial \theta_1} & \frac{\partial \psi_2 \left(\boldsymbol{\theta}\right)}{\partial \theta_2} & \cdots & \frac{\partial \psi_2 \left(\boldsymbol{\theta}\right)}{\partial \theta_d} \\ \\ \vdots & \vdots & \ddots & \vdots \\ \\ \frac{\partial \psi_d \left(\boldsymbol{\theta}\right)}{\partial \theta_1} & \frac{\partial \psi_d \left(\boldsymbol{\theta}\right)}{\partial \theta_2} & \cdots & \frac{\partial \psi_d \left(\boldsymbol{\theta}\right)}{\partial \theta_d} \end{bmatrix}

• $$

\frac{\partial \boldsymbol{\psi}\left(\boldsymbol{\theta}\right)^T}{\partial \boldsymbol{\theta}} = \begin{bmatrix} \frac{\partial}{\partial \theta_1} \\ \frac{\partial}{\partial \theta_2} \\ \vdots \\ \frac{\partial}{\partial \theta_d} \end{bmatrix} \begin{bmatrix} \psi_1 \left(\boldsymbol{\theta}\right) & \psi_2 \left(\boldsymbol{\theta}\right) & \cdots & \psi_d \left(\boldsymbol{\theta}\right) \end{bmatrix} = \begin{bmatrix} \frac{\partial \psi_1 \left(\boldsymbol{\theta}\right)}{\partial \theta_1} & \frac{\partial \psi_2 \left(\boldsymbol{\theta}\right)}{\partial \theta_1} & \cdots & \frac{\partial \psi_d \left(\boldsymbol{\theta}\right)}{\partial \theta_1} \\ \\ \frac{\partial \psi_1 \left(\boldsymbol{\theta}\right)}{\partial \theta_2} & \frac{\partial \psi_2 \left(\boldsymbol{\theta}\right)}{\partial \theta_2} & \cdots & \frac{\partial \psi_d \left(\boldsymbol{\theta}\right)}{\partial \theta_2} \\ \\ \vdots & \vdots & \ddots & \vdots \\ \\ \frac{\partial \psi_1 \left(\boldsymbol{\theta}\right)}{\partial \theta_d} & \frac{\partial \psi_2 \left(\boldsymbol{\theta}\right)}{\partial \theta_d} & \cdots & \frac{\partial \psi_d \left(\boldsymbol{\theta}\right)}{\partial \theta_d} \end{bmatrix}

And $\backslash mathrm\{cov\}\_\{\backslash boldsymbol\{\backslash theta\}\}\; \backslash left(\; \backslash boldsymbol\{T\}(X)\; \backslash right)$ is a positive-semidefinite matrix, that is

$x^\{T\}\; \backslash mathrm\{cov\}\_\{\backslash boldsymbol\{\backslash theta\}\}\; \backslash left(\; \backslash boldsymbol\{T\}(X)\; \backslash right)\; x\; \backslash geq\; 0\; \backslash quad\; \backslash forall\; x\; \backslash in\; \backslash mathbb\{R\}^d$

If is an unbiased estimator (i.e., $\backslash boldsymbol\{\backslash psi\}\backslash left(\backslash boldsymbol\{\backslash theta\}\backslash right)\; =\; \backslash boldsymbol\{\backslash theta\}$) then the Cramér-Rao inequality is

$$

\mathrm{cov}_{\boldsymbol{\theta}}\left(\boldsymbol{T}(X)\right) \geq \mathcal{I}\left(\boldsymbol{\theta}\right)^{-1}

Single-parameter proof

---

First, a more general version of the inequality will be proven; namely, that if the expectation of $T$ is denoted by $\backslash psi\; (\backslash theta)$, then for all $\backslash theta$

$\{\backslash rm\; var\}(t(X))\; \backslash geq\; \backslash frac\{*^2\}\{I(\backslash theta)\}$

The Cramér-Rao inequality will then follow as a consequence.

Let $X$ be a random variable with probability density function $f(x,\; \backslash theta)$. Here $T\; =\; t(X)$ is a statistic, which is used as an estimator for $\backslash theta$. If $V$ is the score, i.e.

$V\; =\; \backslash frac\{\backslash partial\}\{\backslash partial\backslash theta\}\; \backslash ln\; f(X;\backslash theta)$

then the expectation of $V$, written $\{\backslash rm\; E\}(V)$, is zero. If we consider the covariance $\{\backslash rm\; cov\}(V,\; T)$ of $V$ and $T$, we have $\{\backslash rm\; cov\}(V,\; T)\; =\; \{\backslash rm\; E\}(V\; T)$, because $\{\backslash rm\; E\}(V)\; =\; 0$. Expanding this expression we have

$$

{\rm cov}(V,T) = {\rm E} \left( T \cdot \frac{\partial}{\partial\theta} \ln f(X;\theta) \right)

This may be expanded using the chain rule

$\backslash frac\{\backslash partial\}\{\backslash partial\backslash theta\}\; \backslash ln\; Q\; =\; \backslash frac\{1\}\{Q\}\backslash frac\{\backslash partial\; Q\}\{\backslash partial\backslash theta\}$

and the definition of expectation gives, after cancelling $f(x;\; \backslash theta)$,

$$

{\rm E} \left( T \cdot \frac{\partial}{\partial\theta} \ln f(X;\theta) \right) = \int t(x) \left[ \frac{\partial}{\partial\theta} f(x;\theta) \right] \, dx = \frac{\partial}{\partial\theta} \left[ \int t(x)f(x;\theta)\,dx \right] = \psi^\prime(\theta)

because the integration and differentiation operations commute (second condition).

The Cauchy-Schwarz inequality shows that

$$

\sqrt{ {\rm var} (T) {\rm var} (V)} \geq {\rm cov}(V,T) = \psi^\prime (\theta)

therefore

$$

{\rm var\ } T \geq \frac{^*^2} \exp \left( -\frac{1}{2} \left( \boldsymbol{x} - \boldsymbol{\mu} \right)^{T} C^{-1} \left( \boldsymbol{x} - \boldsymbol{\mu} \right) \right).

The Fisher information matrix has elements

$$

\mathcal{I}_{m, k} = \frac{\partial \boldsymbol{\mu}^T}{\partial \theta_m} C^{-1} \frac{\partial \boldsymbol{\mu}}{\partial \theta_k} + \frac{1}{2} \mathrm{tr} \left( C^{-1} \frac{\partial C}{\partial \theta_m} C^{-1} \frac{\partial C}{\partial \theta_k} \right)

where "tr" is the trace.

Let $w*$ be a white Gaussian noise (a sample of $N$ independent observations) with variance $\backslash sigma^2$

$w*\backslash sim\; \backslash mathbb\{N\}\_N\; \backslash left(\; 0,\; \backslash sigma^2\; I\; \backslash right).$

Then the Fisher information matrix is 1 × 1

$$

\mathcal{I}(\sigma^2) = \frac{1}{2} \mathrm{tr} \left( C^{-1} \frac{\partial C}{\partial \theta_m} C^{-1} \frac{\partial C}{\partial \theta_k} \right) = \frac{1}{2 \sigma^2} \mathrm{tr} \left(I\right) = \frac{N}{2 \sigma^2},

and so the Cramér-Rao inequality is

$$

\mathrm{var}\left(\sigma^2\right) \geq \frac{2 \sigma^2}{N}.

Inequalities | Statistics

Cramér-Rao-Ungleichung | Disuguaglianza di Cramér-Rao