In mathematical optimization problems, Lagrange multipliers, named after Joseph Louis Lagrange, are a method for dealing with constraints. We seek the local extrema of a function of several variables subject to one or more constraints. This method reduces a problem in n variables with k constraints to a solvable problem in n + k variables with no constraints. The method introduces a new unknown scalar variable, the Lagrange multiplier, for each constraint and forms a linear combination involving the multipliers as coefficients.

The defence for this can be carried out in the standard way as concerns partial differentiation, using either total differentials, or their close relatives, the chain rules. The object is, for some implicit function, to find the conditions so that the derivative in terms of the independent variables of a function equal zero at some set of inputs.

Introduction

---

Consider a two-dimensional case. Suppose we have a function, f(x,y), to maximize subject to

$g\backslash left(\; x,y\; \backslash right)\; =\; c,$

where c is a constant. We can visualize contours of f given by

$f\; \backslash left(\; x,\; y\; \backslash right)=d\_n$

for various values of $d\_n$, and the contour of $g$ given by $g\; (\; x,\; y\; )\; =\; c$. Suppose we walk along the $g\; =\; c$ contour. Since, in general, the contours of $f$ and $g$ will be distinct, traversing the $g\; =\; c$ contour will generally intersect and cross many contours of $f$. In general, by moving along the line $g\; =\; c$ we can increase or decrease the value of $f$. Only when $g\; =\; c$, the contour we are following, touches tangentially, but does not cross, a contour of $f$ do we not increase or decrease the value of $f$. This occurs at the constrained local extrema and the constrained inflection points of $f$.

A familiar example can be obtained from weather maps, with their contours for temperature and pressure: the constrained extrema will occur where the superposed maps show touching lines (isopleths).

Geometrically we translate the tangency condition to saying that the gradients of $f$ and $g$ are parallel vectors at the maximum. Introducing an unknown scalar, λ, we solve

$\backslash nabla\; \backslash Big\backslash left(x,\; y\; \backslash right)\; +\; \backslash lambda\; \backslash left(g\; \backslash left(x,\; y\; \backslash right)\; -\; c\; \backslash right)\; \backslash Big=\; 0$

for λ ≠ 0.

Once values for λ are determined, we are back to the original number of variables and so can go on to find extrema of the new unconstrained equation

$F\; \backslash left(x,y\; \backslash right)=f\; \backslash left(x,y\; \backslash right)+\backslash lambda\; \backslash left(g\; \backslash left(x,y\; \backslash right)-\; c\; \backslash right)$

in traditional ways. That is, $F(x,y)\; =\; f(x,y)$ for all $(x\; ,\; y)$ satisfying the constraint because $g(x,y)-c$ equals zero on the constraint, but the zeros of ∇$F\; (\; x\; ,\; y\; )$ are all on $g(x,y)=c$.

The method of Lagrange multipliers

---

Let f be a function defined on Rⁿ, and let the constraints be given by g_k(x) = 0 (perhaps by moving the constant to the left, as in g_k(x) − c = 0). Now, define the Lagrangian, Λ, as

$\backslash Lambda(\backslash mathbf\; x,\; \backslash boldsymbol\; \backslash lambda)\; =\; f\; +\; \backslash sum\_k\; \backslash lambda\_k\; g\_k.$

Observe that both the optimization criteria and constraints g_k are compactly encoded as extrema of the Lagrangian:

$\backslash nabla\_\{\backslash mathbf\; x\}\; \backslash Lambda\; =\; 0\; \backslash Leftrightarrow\; \backslash nabla\_\{\backslash mathbf\; x\}\; f\; =\; -\; \backslash sum\_k\; \backslash lambda\_k\; \backslash nabla\_\{\backslash mathbf\; x\}\; g\_k,$

and

$\backslash nabla\_\{\backslash mathbf\; \backslash lambda\}\; \backslash Lambda\; =\; 0\; \backslash Leftrightarrow\; g\_k\; =\; 0.$

Often the Lagrange multipliers have an interpretation as some salient quantity of interest. To see why this might be the case, observe that:

$\backslash frac\{\backslash partial\; \backslash Lambda\}\{\backslash partial\; \{g\_k\}\}\; =\; \backslash lambda\_k.$

Thus, λ_k is the rate of change of the quantity being optimized as a function of the constraint variable. As examples, in Lagrangian mechanics the equations of motion are derived by finding stationary points of the action, the time integral of the difference between kinetic and potential energy. Thus, the force on a particle due to a scalar potential, F = −∇V, can be interpreted as a Lagrange multiplier determining the change in action (transfer of potential to kinetic energy) following a variation in the particle's constrained trajectory. In economics, the optimal profit to a player is calculated subject to a constrained space of actions, where a Lagrange multiplier is the value of relaxing a given constraint (e.g. through bribery or other means).

The method of Lagrange multipliers is generalized by the Karush-Kuhn-Tucker conditions.

Example

---

Suppose we wish to find the discrete probability distribution with maximal information entropy. Then

$f(p\_1,p\_2,\backslash ldots,p\_n)\; =\; -\backslash sum\_\{k=1\}^n\; p\_k\backslash log\_2\; p\_k.$

Of course, the sum of these probabilities equals 1, so our constraint is g(p) = 1 with

$g(p\_1,p\_2,\backslash ldots,p\_n)=\backslash sum\_\{k=1\}^n\; p\_k.$

We can use Lagrange multipliers to find the point of maximum entropy (depending on the probabilities). For all k from 1 to n, we require that

$\backslash frac\{\backslash partial\}\{\backslash partial\; p\_k\}(f+\backslash lambda\; (g-1))=0,$

which gives

$\backslash frac\{\backslash partial\}\{\backslash partial\; p\_k\}\backslash left(-\backslash sum\_\{k=1\}^n\; p\_k\; \backslash log\_2\; p\_k\; +\; \backslash lambda\; (\backslash sum\_\{k=1\}^n\; p\_k\; -\; 1)\; \backslash right)\; =\; 0.$

Carrying out the differentiation of these n equations, we get

$-\backslash left(\backslash frac\{1\}\{\backslash ln\; 2\}+\backslash log\_2\; p\_k\; \backslash right)\; +\; \backslash lambda\; =\; 0.$

This shows that all p_i are equal (because they depend on λ only). By using the constraint ∑_k p_k = 1, we find

$p\_k\; =\; \backslash frac\{1\}\{n\}.$

Hence, the uniform distribution is the distribution with the greatest entropy.

External links

---

For references to Lagrange's original work and for an account of the terminology see the Lagrange Multiplier entry in

• Earliest known uses of some of the words of mathematics: L

For additional text and interactive applets

• Applet
• Tutorial and applet
• Conceptual introduction (plus a brief discussion of Lagrange multipliers in the calculus of variations as used in physics)
• Lagrange Multipliers without Permanent Scarring (tutorial by Dan Klein)

Multivariate calculus | Optimization

Lagrange-Multiplikator | Multiplicadores de Lagrange | Multiplicateur de Lagrange | כופלי לגראנז' | ラグランジュの未定乗数法 | Multiplicatorul Lagrange | Метод множителей Лагранжа