Radial basis functions are a means for interpolation in a stream of data. They differ from statistical approaches in that approximations must be performed on streams of data rather than on complete data sets. They are used in time-series prediction, control, and function approximation.

Overview

---

Radial basis functions (RBF) are powerful techniques for interpolation in multidimensional space. A RBF is a function which has built into a distance criterion with respect to a centre. Such functions can be used very efficiently for interpolation and for smoothing of data. Radial basis functions have been applied in the area of neural networks where they are used as a replacement for the sigmoidal transfer function. Such networks have 3 layers, the input layer, the hidden layer with the RBF non-linearity and a linear output layer. The most popular choice for the non-linearity is the Gaussian. RBF networks have the advantage of not being locked into local minima as do the feedforward networks.

The problem

---

The problem solved by RBF's is the development of an analytic approximation for the input/output mappings described by a deterministic, noisy, or stochastic data stream

$\backslash left\; \backslash \{\; \backslash left\backslash mathbf\{x\}(t)\; ,\; y(t)\; \backslash right:\; \backslash left\backslash mathbb\{R\}^n\; ,\; \backslash mathbb\{R\}\; \backslash right\backslash right\; \backslash \}\; \_\{t=1\}^\{\; \backslash infty\; \}$

where

$\backslash mathbf\{x\}(t)$ is the input vector at time t,

$y(t)$ is the output at time t, and

$n$ is the dimension of the input space.

In the deterministic case the data is drawn from the set

$\backslash left\; \backslash \{\; \backslash left\backslash mathbf\{x\}(t)\; ,\; y(t)\; =\; f\; \backslash big(\; \backslash mathbf\{x\}(t)\; \backslash big)\; \backslash right\backslash right\; \backslash \}\; \_\{t=1\}^\{\; \backslash infty\; \}$.

In the noisy case data is drawn from the set

$\backslash left\; \backslash \{\; \backslash left\backslash mathbf\{x\}(t)\; ,\; y(t)\; =\; f\; \backslash big(\; \backslash mathbf\{x\}(t)\; \backslash big)\; +\; \backslash epsilon(t)\; \backslash right\backslash right\; \backslash \}\; \_\{t=1\}^\{\; \backslash infty\; \}$

where $\backslash epsilon(t)$ is a partially known random process.

In the stochastic case, data is drawn from the joint probability distribution

$P\; \backslash left(\; \backslash mathbf\{x\}\; \backslash land\; y\; \backslash right\; )$.

Architecture

---

RBF architectures come in two forms, normalized and unnormalized. The forms can be expanded into a superposition of local linear models.

Unnormalized

The unnormalized radial basis function architecture, $\backslash varphi\; :\; \backslash mathbb\{R\}^n\; \backslash to\; \backslash mathbb\{R\}$ , is

$\backslash varphi\; (\; \backslash mathbf\{x\}\; )\; \backslash equiv\; \backslash sum\_\{i=1\}^N\; a\_i\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$

where $\backslash varphi$ is the approximation to the data, $\backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$ is a local function of the distance $\backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert$ between the input vector $\backslash mathbf\{x\}$ and a "basis function center"

$\backslash mathbf\{c\}\_i$ $(i=1,N)$,

and

$a\_i$ $(i=1,N)$

are weights to be determined by data. Typically the distance is taken to be the Euclidean distance and the basis function is taken to be Gaussian

$\backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \backslash propto\; \backslash exp\; \backslash left-\backslash beta\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; ^2\; \backslash right$.

The weights

$a\_i$,

and

$\backslash mathbf\{c\}\_i$,

and

$\backslash beta$

are determined in a manner that optimizes the fit between $\backslash varphi$ and the data.

Normalized

Normalized architecture

The normalized RBF architecture is

$\backslash varphi\; (\; \backslash mathbf\{x\}\; )\; \backslash equiv\; \backslash frac\; \{\; \backslash sum\_\{i=1\}^N\; a\_i\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \}\; \{\; \backslash sum\_\{i=1\}^N\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \}\; =\; \backslash sum\_\{i=1\}^N\; a\_i\; u\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$

where

$u\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \backslash equiv\; \backslash frac\; \{\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \}\; \{\; \backslash sum\_\{i=1\}^N\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \}$.

Theoretical motivation for normalization

There is theoretical justification for this architecture in the case of stochastic data flow. Assume a Stochastic kernel approximation for the joint probability density

$P\backslash left\; (\; \backslash mathbf\{x\}\; \backslash land\; y\; \backslash right\; )\; =\; \backslash sum\_\{i=1\}^N\; \backslash ,\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \backslash ,\; \backslash sigma\; \backslash big\; (\; \backslash left\; \backslash vert\; y\; -\; e\_i\; \backslash right\; \backslash vert\; \backslash big\; )$

where the weights $\backslash mathbf\{c\}\_i$ and $e\_i$ are exemplars from the data and we require the kernels to be normalized

$\backslash int\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \backslash ,\; d^n\backslash mathbf\{x\}\; =1$

and

$\backslash int\; \backslash sigma\; \backslash big\; (\; \backslash left\; \backslash vert\; y\; -\; e\_i\; \backslash right\; \backslash vert\; \backslash big\; )\; \backslash ,\; dy\; =1$.

The probability densities in the input and output spaces are

$P\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\; =\; \backslash int\; P\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash land\; y\; \backslash right\; )\; \backslash ,\; dy\; =\; \backslash sum\_\{i=1\}^N\; \backslash ,\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$

and

$P\; \backslash left\; (\; y\; \backslash right\; )\; =\; \backslash int\; P\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash land\; y\; \backslash right\; )\; \backslash ,\; d^n\; \backslash mathbf\{x\}\; =\; \backslash sum\_\{i=1\}^N\; \backslash ,\; \backslash sigma\; \backslash big\; (\; \backslash left\; \backslash vert\; y\; -\; e\_i\; \backslash right\; \backslash vert\; \backslash big\; )$

The expectation of y given an input $\backslash mathbf\{x\}$ is

$\backslash varphi\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\; \backslash equiv\; E\backslash left\; (\; y\; \backslash mid\; \backslash mathbf\{x\}\; \backslash right\; )\; =\; \backslash int\; y\; \backslash ,\; P\backslash left\; (\; y\; \backslash mid\; \backslash mathbf\{x\}\; \backslash right\; )\; dy$

where

$P\backslash left\; (\; y\; \backslash mid\; \backslash mathbf\{x\}\; \backslash right\; )$

is the conditional probability of y given $\backslash mathbf\{x\}$. The conditional probability is related to the joint probability through Bayes theorem

$P\backslash left\; (\; y\; \backslash mid\; \backslash mathbf\{x\}\; \backslash right\; )\; =\; \backslash frac\; \{P\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash land\; y\; \backslash right\; )\}\; \{P\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\}$

which yields

$\backslash varphi\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\; =\; \backslash int\; y\; \backslash ,\; \backslash frac\; \{P\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash land\; y\; \backslash right\; )\}\; \{P\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\}\; \backslash ,\; dy$.

This becomes

$\backslash varphi\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\; =\; \backslash frac\; \{\; \backslash sum\_\{i=1\}^N\; a\_i\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \}\; \{\; \backslash sum\_\{i=1\}^N\; \backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\; \}\; =\; \backslash sum\_\{i=1\}^N\; a\_i\; u\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$

when the integrations are performed.

Local linear models

It is sometimes convenient to expand the architecture to include local linear models. In that case the architectures become, to first order,

$\backslash varphi\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\; =\; \backslash sum\_\{i=1\}^N\; \backslash left\; (\; a\_i\; +\; \backslash mathbf\{b\}\_i\; \backslash cdot\; \backslash left\; (\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; )\; \backslash right\; )\backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$

and

$\backslash varphi\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\; =\; \backslash sum\_\{i=1\}^N\; \backslash left\; (\; a\_i\; +\; \backslash mathbf\{b\}\_i\; \backslash cdot\; \backslash left\; (\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; )\; \backslash right\; )u\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$

in the unnormalized and normalized cases, respectively. Here $\backslash mathbf\{b\}\_i$ are weights to be determined. Higher order linear terms are also possible.

This result can be written

$\backslash varphi\; \backslash left\; (\; \backslash mathbf\{x\}\; \backslash right\; )\; =\; \backslash sum\_\{i=1\}^\{2N\}\; \backslash sum\_\{j=1\}^n\; e\_\{ij\}\; v\_\{ij\}\; \backslash big\; (\; \backslash mathbf\{x\}\; -\; \backslash mathbf\{c\}\_i\; \backslash big\; )$

where

and

in the unnormalized case and

in the normalized case.

Here $\backslash delta\_\{ij\}$ is a Kronecker delta function defined as

.

Objective functions

---

The weights, which we signify by $\backslash mathbf\{w\}$, in the RBF architecture are found through optimization of an objective function. The most common objective function is the least squares function

$K(\; \backslash mathbf\{w\}\; )\; \backslash equiv\; \backslash sum\_\{t=1\}^\backslash infty\; K\_t(\; \backslash mathbf\{w\}\; )$

where

$K\_t(\; \backslash mathbf\{w\}\; )\; \backslash equiv\; \backslash bigy(t)\; -\; \backslash varphi\; \backslash big\; (\; \backslash mathbf\{x\}(t),\; \backslash mathbf\{w\}\; \backslash big\; )\; \backslash big^2$.

We have explicitly included the dependence on the weights. Minimization of the least squares objective function by optimal choice of weights optimizes accuracy of fit.

There are occasions in which multiple objectives, such as smoothness as well as accuracy, must be optimized. In that case it is useful to optimize a regularized objective function such as

$H(\; \backslash mathbf\{w\}\; )\; \backslash equiv\; K(\; \backslash mathbf\{w\}\; )\; +\; \backslash lambda\; S(\; \backslash mathbf\{w\}\; )\; \backslash equiv\; \backslash sum\_\{t=1\}^\backslash infty\; H\_t(\; \backslash mathbf\{w\}\; )$

where

$S(\; \backslash mathbf\{w\}\; )\; \backslash equiv\; \backslash sum\_\{t=1\}^\backslash infty\; S\_t(\; \backslash mathbf\{w\}\; )$

and

$H\_t(\; \backslash mathbf\{w\}\; )\; \backslash equiv\; K\_t\; (\; \backslash mathbf\{w\}\; )\; +\; \backslash lambda\; S\_t\; (\; \backslash mathbf\{w\}\; )$

where optimization of S maximizes smoothness and $\backslash lambda$ is known as a regularization parameter.

Training

---

Choosing weights that optimize the objective function is known as "training." Training is performed at each time step as data streams in.

Gradient descent training

The simplest training algorithm is Gradient descent. In gradient descent training the weights are adjusted at each time step by moving them in a direction opposite from the gradient of the objective function

$\backslash mathbf\{w\}(t+1)\; =\; \backslash mathbf\{w\}(t)\; -\; \backslash nu\; \backslash frac\; \{d\}\; \{d\backslash mathbf\{w\}\}\; H\_t(\backslash mathbf\{w\})$

where $\backslash nu$ is a "learning parameter."

For the case of training the linear weights, $a\_i$, the algorithm becomes

$a\_i\; (t+1)\; =\; a\_i(t)\; +\; \backslash nu\; \backslash bigy(t)\; -\; \backslash varphi\; \backslash big\; (\; \backslash mathbf\{x\}(t),\; \backslash mathbf\{w\}\; \backslash big\; )\; \backslash big\backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$

in the unnormalized case and

$a\_i\; (t+1)\; =\; a\_i(t)\; +\; \backslash nu\; \backslash bigy(t)\; -\; \backslash varphi\; \backslash big\; (\; \backslash mathbf\{x\}(t),\; \backslash mathbf\{w\}\; \backslash big\; )\; \backslash bigu\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )$

in the normalized case.

For local-linear-architectures gradient-descent training is

$e\_\{ij\}\; (t+1)\; =\; e\_\{ij\}(t)\; +\; \backslash nu\; \backslash bigy(t)\; -\; \backslash varphi\; \backslash big\; (\; \backslash mathbf\{x\}(t),\; \backslash mathbf\{w\}\; \backslash big\; )\; \backslash bigv\_\{ij\}\; \backslash big\; (\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash big\; )$

Projection operator training

For the case of training the linear weights, $a\_i$ and $e\_\{ij\}$, the algorithm becomes

$a\_i\; (t+1)\; =\; a\_i(t)\; +\; \backslash nu\; \backslash bigy(t)\; -\; \backslash varphi\; \backslash big\; (\; \backslash mathbf\{x\}(t),\; \backslash mathbf\{w\}\; \backslash big\; )\; \backslash big\backslash frac\; \{\backslash rho\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\}\; \{\backslash sum\_\{i=1\}^N\; \backslash rho^2\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\}$

in the unnormalized case and

$a\_i\; (t+1)\; =\; a\_i(t)\; +\; \backslash nu\; \backslash bigy(t)\; -\; \backslash varphi\; \backslash big\; (\; \backslash mathbf\{x\}(t),\; \backslash mathbf\{w\}\; \backslash big\; )\; \backslash big\backslash frac\; \{u\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\}\; \{\backslash sum\_\{i=1\}^N\; u^2\; \backslash big\; (\; \backslash left\; \backslash Vert\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash right\; \backslash Vert\; \backslash big\; )\}$

in the normalized case and

$e\_\{ij\}\; (t+1)\; =\; e\_\{ij\}(t)\; +\; \backslash nu\; \backslash bigy(t)\; -\; \backslash varphi\; \backslash big\; (\; \backslash mathbf\{x\}(t),\; \backslash mathbf\{w\}\; \backslash big\; )\; \backslash big\backslash frac\; \{\; v\_\{ij\}\; \backslash big\; (\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash big\; )\; \}\; \{\backslash sum\_\{i=1\}^N\; \backslash sum\_\{j=1\}^n\; v\_\{ij\}^2\; \backslash big\; (\; \backslash mathbf\{x\}(t)\; -\; \backslash mathbf\{c\}\_i\; \backslash big\; )\; \}$

in the local-linear case.

See also

---

• Artificial neural networks

References

---

• J. Moody and C. J. Darken, "Fast learning in networks of locally tuned processing units," Neural Computation, 1, 281-294 (1989). Also see Radial basis function networks according to Moody and Darken
• T. Pogio and F. Girosi, "Networks for approximation and learning," Proc. IEEE 78(9), 1484-1487 (1990).
• R. D. Jones, Y. C. Lee, C. W. Barnes, G. W. Flake, K. Lee, P. S. Lewis, and S. Qian, “Function approximation and time series prediction with neural networks,” Proceedings of the International Joint Conference on Neural Networks, June 17-21, p. I-649 (1990).

Neural networks | Information technology | Computer network analysis | Networks | Cybernetics | Artificial intelligence | Interpolation

Невронна мрежа | Radiale Basisfunktion | Red neuronal artificial | Réseau de neurones | 신경망 | neuronska mreža | ニューラルネットワーク | Sieć neuronowa | Rede neural | Reţele neuronale | Нейронная сеть | nevronska mreža | 神经网络