Simplex algorithm

In mathematical optimization theory, the simplex algorithm of George Dantzig is a popular technique for numerical solution of the linear programming problem. An unrelated, but similarly named method is the Nelder-Mead method or downhill simplex method due to Nelder & Mead (1965) and is a numerical method for optimising many-dimensional unconstrained problems, belonging to the more general class of search algorithms.

In both cases, the method uses the concept of a simplex, which is a polytope of N + 1 vertices in N dimensions: a line segment on a line, a triangle on a plane, a tetrahedron in three-dimensional space and so forth.

Problem input

Consider a linear programming problem,

maximize

\mathbf{c}^T \mathbf{x}

subject to

\mathbf{A}\mathbf{x} \le \mathbf{b}, \, \mathbf{x} \ge 0

The simplex algorithm requires the linear programming problem to be in augmented form. The problem can then be written as follows in matrix form:

Maximize Z in:

\begin{bmatrix} 1 & -\mathbf{c} & 0 \\ 0 & \mathbf{A} & \mathbf{I} \end{bmatrix} \begin{bmatrix} Z \\ \mathbf{x} \\ \mathbf{x}_s \end{bmatrix} = \begin{bmatrix} 0 \\ \mathbf{b} \end{bmatrix}

\mathbf{x}, \, \mathbf{x}_s \ge 0

where x are the variables from the standard form, x_s are the introduced slack variables from the augmentation process, c contains the optimization coefficients, A and b describe the system of constraint equations, and Z is the variable to be maximized.

The system is typically underdetermined, since the number of variables exceed the number of equations. The difference between the number of variables and the number of equations gives us the degrees of freedom associated with the problem. Any solution, optimal or not, will therefore include a number of variables of arbitrary value. The simplex algorithm uses zero as this arbitrary value, and the number of variables with value zero equals the degrees of freedom.

Values of nonzero value are called basic variables, and values of zero values are called nonbasic variables in the simplex algorithm.

This form simplifies finding the initial basic feasible solution (BF), since all variables from the standard form can be chosen to be nonbasic (zero), while all new variables introduced in the augmented form are basic (nonzero), since their value can be trivially calculated ( $\mathbf{x}_{s\,i} = \mathbf{b}_{j}$ for them, since the augmented problem matrix is diagonal on its right half).

Revised simplex algorithm

Matrix form of the simplex algorithm

At any iteration of the simplex algorithm, the tableau will be of this form:

\begin{bmatrix} 1 & \mathbf{c}_B \mathbf{B}^{-1}\mathbf{A} -\mathbf{c} & \mathbf{c}_B \mathbf{B}^{-1} \\ 0 & \mathbf{B}^{-1}\mathbf{A} & \mathbf{B}^{-1} \end{bmatrix} \begin{bmatrix} Z \\ \mathbf{x} \\ \mathbf{x}_s \end{bmatrix} = \begin{bmatrix} \mathbf{c}_B \mathbf{B}^{-1} \mathbf{b} \\ \mathbf{B}^{-1}\mathbf{b} \end{bmatrix} where

\mathbf{c}_B

are the coefficients of basic variables in the c-matrix; and B is the columns of

\, \mathbf{I}

corresponding to the basic variables.

It is worth noting that B and $\mathbf{c}_B$ are the only variables in this equation (except Z and x of course). Everything else is constant throughout the algorithm.

Algorithm

Choose an initial BF as shown above

This implies that B is the identity matrix, and

\mathbf{c}_B

is a zero-vector.

While nonoptimal solution:
- Determine direction of highest gradient
Choose the variable associated with the coefficient in $\mathbf{c}_B \mathbf{B}^{-1}\mathbf{A} -\mathbf{c}$ that has the highest negative magnitude. This nonbasic variable, which we call the entering basic, will be increased to help maximize the problem.
- Determine maximum step length
Use the $\begin{bmatrix} \mathbf{B}^{-1}\mathbf{A} & \mathbf{B}^{-1} \end{bmatrix} \begin{bmatrix} \mathbf{x} \\ \mathbf{x}_s \end{bmatrix} = \mathbf{B}^{-1}\mathbf{b}$ subequation to determine which basic variable reaches zero first when the entering basic is increased. This variable, which we call the leaving basic then becomes nonbasic. This operation only involves a single division for each basic variable, since the existing basic-variables occur diagonally in the tableau.
- Rewrite problem
Modify B and $\mathbf{c}_B$ to account for the new basic variables. This will automatically make the tableau diagonal for the existing and new basic variables.
- Check for improvement
Repeat procedure until no further improvement is possible, meaning all the coefficients of $\mathbf{c}_B \mathbf{B}^{-1}\mathbf{A} -\mathbf{c}$ are positive. Procedure is also terminated if all coefficients are zero, and the algorithm has walked in a circle and revisited a previous state.

Description

A linear programming problem consists of a collection of linear inequalities on a number of real variables and a fixed linear functional which is to be maximized (or minimized). Some further details are given in the linear programming article.

In geometric terms we are considering a closed convex polytope P, defined by intersecting a number of half-spaces in n-dimensional Euclidean space, which lie to one side of a hyperplane. The objective is to maximise a linear functional L; if we consider the hyperplanes H(c) defined by L(x) = c as c increases, these form a parallel family. If the problem is well-posed, we want to find the largest value of c such that H(c) intersects P (if there is no such largest value of c, this isn't a reasonable question for optimization as it stands). In this case we can show that the optimum value of c is attained, on the boundary of P. Methods for finding this optimum point on P have a choice of improving a possible point by moving through the interior of P (so-called interior point methods), or starting and remaining on the boundary.

The simplex algorithm falls into the latter class of method. The idea is to move along the facets of P in search of the optimum, from point to point. Note that the optimum cannot occur at a mid-point, for example in the middle of an edge lying as a line segment on the boundary of P, unless the whole relevant 'face' is parallel to H. Therefore it is possible to look solely at moves skirting the edge of a simplex, ignoring the interior. The algorithm specifies how this is to be done.

We start at some vertex of the polytope, and at every iteration we choose an adjacent vertex such that the value of the objective function does not decrease. If no such vertex exists, we have found a solution to the problem. But usually, such an adjacent vertex is nonunique, and a pivot rule must be specified to determine which vertex to pick. Various pivot rules exist.

In 1972, Klee and Minty gave an example of a linear programming problem in which the polytope P is a distortion of an n-dimensional cube. They showed that the simplex method as formulated by Dantzig visits all 2ⁿ vertices before arriving at the optimal vertex. This shows that the worst-case complexity of the algorithm is exponential time. Similar examples have been found for other pivot rules. It is an open question if there is a pivot rule with polynomial time worst-case complexity.

Nevertheless, the simplex method is remarkably efficient in practice. Attempts to explain this employ the notion of average complexity or (recently) smoothed complexity.

Other algorithms for solving linear programming problems are described in the linear programming article.

References

Greenberg, Harvey J., Klee-Minty Polytope Shows Exponential Time Complexity of Simplex Method University of Colorado at Denver (1997) PDF download
Frederick S. Hillier and Gerald J. Lieberman: Introduction to Operations Research, 8th edition. McGraw-Hill. ISBN 007123828X
Thomas H. Cormen, Charles E. Leiserson, Ronald L. Rivest, and Clifford Stein. Introduction to Algorithms, Second Edition. MIT Press and McGraw-Hill, 2001. ISBN 0262032937. Section 29.3: The simplex algorithm, pp.790–804.

Note

Greenberg, cites: V. Klee and G.J. Minty. "How Good is the Simplex Algorithm?" In O. Shisha, editor, Inequalities, III, pages 159–175. Academic Press, New York, NY, 1972

External links

An Introduction to Linear Programming and the Simplex Algorithm by Spyros Reveliotis of the Georgia Institute of Technology.
- A step-by-step simplex algorithm solver Solve your own LP problem. Note: this does not work well. Sample problem: minimize z=x1+x2 subject to 2x1+3x2<=6 and -x1+x2<=1 should give 0 as answer. This applet gives the same answer for both maximize and minimize problem.
Java-based interactive simplex tool hosted by Argonne National Laboratory.
Simplex Algorithm by Elmer G. Wiens. Demonstrates algorithm in detail, using the simplex tableau.
Tutorial for The Simplex Method by Stefan Waner, hofstra.edu. Interactive worked example.
A simple example - step by step by Mazoo's Learning Blog.
Simplex Method A good tutorial for Simplex Method with examples (also two-phase and M-method).
General Linear Programming Demos Includes two simplex methods

Optimization algorithms | Operations research

Gourt Home::Gourt :: Simplex algorithm