A First Course in

Linear Optimization
— a dynamic book —

by
Jon Lee

Third Edition (Version 3.12)

ReEx PrEsS
 Jon Lee
2013–2018

ReEx PrEsS
 iii

This work is licensed under the

Creative Commons Attribution 3.0 Unported License

(CC BY 3.0) c b.

To view a copy of this license, visit

http://creativecommons.org/licenses/by/3.0/
where you will see the summary information below and can click through to the full
license information.
Go Forward

This is a book on linear optimization, written in LATEX. I started it, aiming it at the
course IOE 510, a masters-level course at the University of Michigan. Use it as is, or
adapt it to your course! It is an ongoing project. It is alive! It can be used, modified
(the LATEX source is available) and redistributed as anyone
pleases, subject to the terms of the Creative Commons At-
tribution 3.0 Unported License (CC BY 3.0) c b. Please
take special note that you can share (copy and redistribute
in any medium or format) and adapt (remix, transform, and
build upon for any purpose, even commercially) this mate-
rial, but you must give appropriate credit, provide a link to
the license, and indicate if changes were made. You may do
so in any reasonable manner, but not in any way that sug-
gests that I endorse you or your use. If you are interested
in endorsements, speak to my agent.
I started this material, but I don’t control so much what
you do with it. Control is sometimes overrated — and I am a control freak, so I should
know!
I hope that you find this material useful. If not, I am happy to refund what you paid
to me.

Jon Lee

Ann Arbor, Michigan

started March 2013

v
Preface

This book is a treatment of linear optimization meant for students who are reasonably
comfortable with matrix algebra (or willing to get comfortable rapidly). It is not a goal
of mine to teach anyone how to solve small problems by hand. My goals are to intro-
duce: (i) the mathematics and algorithmics of the subject at a beginning mathemati-
cal level, (ii) algorithmically-aware modeling techniques, and (iii) high-level computa-
tional tools for studying and developing optimization algorithms (in particular, MATLAB
and AMPL).
Proofs are given when they are important in understanding the algorithmics. I make
free use of the inverse of a matrix. But it should be understood, for example, that B −1 b
is meant as a mathematical expression for the solution of the square linear system of
equations Bx = b . I am not in any way suggesting that an efficient way to calculate the
solution of a large (often sparse) linear system is to calculate an inverse! Also, I avoid
the dual simplex algorithm (e.g., even in describing branch-and-bound and cutting-
plane algorithms), preferring to just think about the ordinary simplex algorithm ap-
plied to the dual problem. Again, my goal is not to describe the most efficient way to
do matrix algebra!
Illustrations are woefully few. Though if Lagrange could not be bothered1 , who am
I to aim higher? Still, I am gradually improving this aspect, and many of the algorithms
are illustrated in the modern way, with computer code.
The material that I present was mostly well known by the 1960’s. As a student at
Cornell in the late 70’s and early 80’s, I learned and got excited about linear optimiza-
tion from Bob Bland, Les Trotter and Lou Billera, using [1] and [5]. The present book
is a treatment of some of that material, with additional material on integer-linear op-
timization, mostly which I originally learned from George Nemhauser and Les. But
there is new material too; in particular, a “deconstructed post-modern” version of Go-
mory pure and mixed-integer cuts. There is nothing here on interior-point algorithms
and the ellipsoid algorithm; don’t tell Mike Todd!

Jon Lee

Ann Arbor, Michigan

started March 2013
(or maybe really in Ithaca, NY in 1979)

vii
Serious Acknowledgments

Throw me some serious funding for this project, and I will acknowledge you — seri-
ously!
Many of the pictures in this book were found floating around on the web. I am making
”fair use” of them as they float through this document. Of course, I gratefully acknowl-
edge those who own them.
Hearty thanks to many students and to Prof. Siqian Shen for pointing out typos in an
earlier version.

ix
Dedication

For students (even Ohio students). Not for publishers — not this time. Maybe next time.

xi
The Nitty Gritty

You can always get the released edition of this book (in .pdf format) from my web
page or github and the materials to produce them (LATEX source, etc.) from me.
If you decide that you need to recompile the LATEX to change something, then you
should be aware of two things. First, there is no user’s manual nor help desk. Second,
I use a lot of include files, for incorporating code listings and output produced by soft-
ware like Matlab and AMPL. Make sure that you understand where everything is pulled
from if you recompile the LATEX. In particular, if you change anything in the directory
that the LATEX file is in, something can well change in the .pdf output. For example,
Matlab and AMPL scripts are pulled into the book, and likewise for the output of those
scripts. If you want to play, and do not want to change those parts of the book, play in
another directory.
I make significant use of software. Everything seems to work with:

MATLAB R2016b
AMPL 20170711
CPLEX 12.7.1.0
Mathematica 11.1.0.0
WinEdt 10.1
MiKTeX 2.9

Use of older versions is inexcusable. Newer versions will surely break things. Nonethe-
less, if you can report success or failure on newer versions, please let me know.
I use lots of LATEX packages (which, as you may know, makes things rather fragile).
I could not possibly gather the version numbers of those — I do have a day job! (but I
do endeavor to keep my packages up to date).

xiii
Contents

1 Let’s Get Started 1

1.1 Linear Optimization and Standard Form . . . . . . . . . . . . . . . . . . . 1
1.2 A Standard-Form Problem and its Dual . . . . . . . . . . . . . . . . . . . 3
1.3 Linear-Algebra Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Modeling 11
2.1 A Production Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
2.2 Norm Minimization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.3 Network Flow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.4 An Optimization Modeling Language . . . . . . . . . . . . . . . . . . . . 15
2.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

3 Algebra Versus Geometry 23

3.1 Basic Feasible Solutions and Extreme Points . . . . . . . . . . . . . . . . . 23
3.2 Basic Feasible Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
3.3 Basic Feasible Rays and Extreme Rays . . . . . . . . . . . . . . . . . . . . 30
3.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31

4 The Simplex Algorithm 33

4.1 A Sufficient Optimality Criterion . . . . . . . . . . . . . . . . . . . . . . . 33
4.2 The Simplex Algorithm with No Worries . . . . . . . . . . . . . . . . . . 35
4.3 Anticycling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
4.4 Obtaining a Basic Feasible Solution . . . . . . . . . . . . . . . . . . . . . . 42
4.4.1 Ignoring degeneracy . . . . . . . . . . . . . . . . . . . . . . . . . . 43
4.4.2 Not ignoring degeneracy . . . . . . . . . . . . . . . . . . . . . . . . 45
4.5 The Simplex Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
4.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

5 Duality 51
5.1 The Strong Duality Theorem . . . . . . . . . . . . . . . . . . . . . . . . . . 52
5.2 Complementary Slackness . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
5.3 Duality for General Linear-Optimization Problems . . . . . . . . . . . . . 54
5.4 Theorems of the Alternative . . . . . . . . . . . . . . . . . . . . . . . . . . 57
5.5 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59

xv
xvi CONTENTS

6 Sensitivity Analysis 63
6.1 Right-Hand Side Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
6.1.1 Local analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
6.1.2 Global analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
6.1.3 A brief detour: the column geometry for the Simplex Algorithm . 67
6.2 Objective Changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
6.2.1 Local analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6.2.2 Global analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
6.2.3 Local sensitivity analysis with a modeling language . . . . . . . . 70
6.3 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70

7 Large-Scale Linear Optimization 73

7.1 Decomposition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
7.1.1 The master reformulation . . . . . . . . . . . . . . . . . . . . . . . 74
7.1.2 Solution of the Master via the Simplex Algorithm . . . . . . . . . 76
7.2 Lagrangian Relaxation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
7.2.1 Lagrangian bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
7.2.2 Solving the Lagrangian Dual . . . . . . . . . . . . . . . . . . . . . 87
7.3 The Cutting-Stock Problem . . . . . . . . . . . . . . . . . . . . . . . . . . 92
7.3.1 Formulation via cutting patterns . . . . . . . . . . . . . . . . . . . 93
7.3.2 Solution via continuous relaxation . . . . . . . . . . . . . . . . . . 93
7.3.3 The knapsack subproblem . . . . . . . . . . . . . . . . . . . . . . . 94
7.3.4 Applying the Simplex Algorithm . . . . . . . . . . . . . . . . . . . 96
7.3.5 A demonstration implementation . . . . . . . . . . . . . . . . . . 96
7.4 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101

8 Integer-Linear Optimization 103

8.1 Integrality for Free . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
8.1.1 Some structured models . . . . . . . . . . . . . . . . . . . . . . . . 103
8.1.2 Unimodular basis matrices and total unimodularity . . . . . . . . 106
8.1.3 Consequences of total unimodularity . . . . . . . . . . . . . . . . 110
8.2 Modeling Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
8.2.1 Disjunctions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
8.2.2 Forcing constraints . . . . . . . . . . . . . . . . . . . . . . . . . . . 116
8.2.3 Piecewise-linear univariate functions . . . . . . . . . . . . . . . . 118
8.3 A Prelude to Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
8.4 Branch-and-Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
8.5 Cutting Planes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
8.5.1 Pure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
8.5.2 Mixed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
8.5.3 Finite termination . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
8.5.4 Branch-and-Cut . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
8.6 Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135
CONTENTS xvii

Appendices 141
A.1 LATEX template . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
A.2 MATLAB for deconstructing the Simplex Algorithm . . . . . . . . . . . . . 149
A.3 MATLAB for Gomory cuts . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
A.4 AMPL for uncapacitated facility-location problem . . . . . . . . . . . . . . 161

End Notes 165

Bibliography 171

Index of definitions and only definitions — lazy and maintainable 174

Chapter 1

Let’s Get Started

Our main goals in this chapter are as follows:

• Introduce some terminology associated with linear optimization.

• Describe elementary techniques for transforming any linear-optimization prob-

lems to one in a ‘standard form.’

• Introduce the Weak Duality Theorem.

• Review ideas from linear algebra that we will make use of later.

1.1 Linear Optimization and Standard Form

Linear optimization is the study of the mathematics and algorithms associated with
minimizing or maximizing a real linear objective function of a finite number of real
variables, subject to a finite number of linear constraints, each being that a real linear
function on these variables be = , ≤ or ≥ a constant. A polyhedron is the solution

1
2 CHAPTER 1. LET’S GET STARTED

set of a finite number of linear constraints; so we are studying optimization of a linear

function on a polyhedron.

Definition 1.1
A solution of a linear-optimization problem is an assignment of real values to the vari-
ables. A solution is feasible if it satisfies the linear constraints. A solution is optimal
if there is no feasible solution with better objective value. The set of feasible solutions
(which is a polyhedron) is the feasible region.

It is convenient to put a general linear-optimization problem into a standard form

min c0 x
Ax = b ;
x ≥ 0,

where c ∈ Rn , b ∈ Rm , A ∈ Rm×n has full row rank m , and x ia a vector of variables in

Rn . That is, minimization of a linear function on a finite number number of non-negative
real variables, subject to a non-redundant and consistent system of linear equations. Note
that even though the system of equations, Ax = b, has a solution, the problem may not
have a feasible solution.
Through a finite sequence of simple transformations, every linear-optimization prob-
lem can be brought into an equivalent one in standard form. Specifically, we can apply
any of the follow steps, as needed, in the order presented.

• The maximum of c0 x is the same as the negative of the minimum of −c0 x .

• We can replace any non-positive variable xj with a non-negative variables x− j ,

−
substituting −xj for xj . Additionally, we can replace any unrestricted variable
xj with the difference of a pair of non-negative variables x+ −
j and xj . That is,
+ −
substituting xj − xj for xj . In this way, we can make all variables constrained to
be non-negative.

P P
• Next, if we have an inequality nj=1 αj xj ≤ γ , we simply replace it with nj=1 αj xj +
s = γ , where a real slack variable sPis introduced whichPis constrained to be non-
negative. Similarly, we can replace nj=1 αj xj ≥ γ with nj=1 αj xj −s = γ , where
a real surplus variable s is introduced which is constrained to be non-negative.

• Applying these transformations as needed results in a standard-form problem,

except possibly for the condition that the matrix of coefficients of the systems of
equations have full row rank. But we can realize this last condition by carrying out
elementary row operations on the system of equations, resulting in the elimina-
tion of any redundant equations or the identification that the system of equations
is inconsistent. In the latter case, the linear-optimization problem is infeasible.
1.2. A STANDARD-FORM PROBLEM AND ITS DUAL 3

1.2 A Standard-Form Problem and its Dual

Let c ∈ Rn , b ∈ Rm , and A ∈ Rm×n . Let x be a vector of variables in Rn . Consider the
standard-form problem
min c0 x
Ax = b ; (P)
x ≥ 0.
Let y be a vector of variables in Rm , and consider the linear-optimization problem

max y 0 b
(D)
y 0 A ≤ c0 .

It is worth emphasizing that (P) and (D) are both defined from the same data A , b
and c . We have the following very simple but key result, relating the objective values
of feasible solutions of the two linear-optimization problems.

Theorem 1.2 (Weak Duality Theorem)

If x̂ is feasible in (P) and ŷ is feasible in (D), then c0 x̂ ≥ ŷ 0 b .

Proof.
c0 x̂ ≥ ŷ 0 Ax̂ ,
because ŷ 0 A ≤ c0 (feasibility of ŷ in (D)) and x̂ ≥ 0 (feasibility of x̂ in (P)). Furthermore

ŷ 0 Ax̂ = ŷ 0 b ,

because Ax̂ = b (feasibility of x̂ in (P)). The result follows. t

u

1.3 Linear-Algebra Review

For a matrix A ∈ Rm×n , we denote the entry in row i and column j as aij . For a
matrix A ∈ Rm×n , we denote the transpose of A by A0 ∈ Rn×m . That is, the entry in
row i and column j of A0 is aji .
Except when we state clearly otherwise, vectors are “column vectors.” That is, we
can view a vector x ∈ Rn as a matrix in Rn×1 . Column j of A is denoted by A·j ∈ Rm .
Row i of A is denoted by Ai· , and we view its transpose as a vector in Rn . We will have
4 CHAPTER 1. LET’S GET STARTED

far greater occasion to reference columns of matrices rather than rows, so we will often
write Aj as a shorthand for A·j , so as to keep notation less cluttered.
For matrices A ∈ Rm×p Pp and B ∈ R
p×n , the (matrix) product AB ∈ Rm×n is defined

to be the matrix having k=1 aik bkj as the entry in row i and column j . Note that for the
product AB to make sense, the number of columns of A and the number of rows of B
must be identical. It is important to emphasize that matrix multiplication is associative;
that is, (AB)C = A(BC) , and so we can always unambiguously write the product
of any number of matrices without the need for any parentheses. Also, note that the
product and transpose behave nicely together. That is, (AB)0 = B 0 A0 . P
The dot product or scalar product of vectors x, z ∈ Rn is the scalar hx, zi := nj=1 xj zj ,
which we can equivalently see as x0 z or z 0 x , allowing ourselves to consider a 1 × 1 ma-
trix to be viewed as a scalar. Thinking about matrix multiplication again, and freely
viewing columns as vectors, the entry in row i and column j of the product AB is the
dot product h(Ai· )0 , B·j i .
Matrix multiplication extends to “block matrices” in a straightforward manner. If
   
A11 · · · A1p B11 · · · B1n
 A21 · · · A2p   B21 · · · B2n 
   
A :=  . .. ..  and B :=  .. .. ..  ,
 .. . .   . . . 
Am1 · · · Amp Bp1 · · · Bpn
where each of the Aij and Bij are matrices, and we assume that for all i and j the
number of columns of Aik agrees with the number of rows of Bkj , then
 
Pp Pp
 k=1 A1k Bk1 ··· k=1 A1k Bkn 
 
 
 Pp Pp 
 k=1 A2k Bk1 ··· 
k=1 A2k Bkn  .
AB = 
 .. .. 
 .. 
 . . . 
 
Pp Pp
k=1 Amk Bk1 · · · k=1 Amk Bkn

P
That is, block i, j of the product is pk=1 Aik Bkj , and Aik Bkj is understood as ordinary
matrix multiplication. P
For vectors x1 , x2 , . . . , xp ∈ Rn , and scalars λ1 , λ2 , . . . , λp , pi=1 λi xi is a linear
combination of x1 , x2 , . . . , xp . The linear combination is trivial if all λi = 0 . The vec-
tors x1 , x2 , . . . , xp ∈ Rn are linearly independent if the only representation of the zero
vector in Rn as a linear combination of x1 , x2 , . . . , xp is trivial. The set of all linear combi-
nations of x1 , x2 , . . . , xp is the vector-space span of {x1 , x2 , . . . , xp } . The dimension of
a vector space V , denoted dim(V ) , is the maximum number of linearly-independent
vectors in it. Equivalently, it is the minimum number of vectors needed to span the
space.
A set of dim(V ) linearly-independent vectors that spans a vector space V is a ba-
sis for V . If V is the vector-space span of {x1 , x2 , . . . , xp } , then there is a subset of
{x1 , x2 , . . . , xp } that is a basis for V . It is not hard to prove the following very useful
result.
1.3. LINEAR-ALGEBRA REVIEW 5

Theorem 1.3 (Greedy Basis Extension Theorem)

Let V be the vector-space span of {x1 , x2 , . . . , xp } . Then every linearly-independent
subset of {x1 , x2 , . . . , xp } can be extended to a basis for V using vectors from
x1 , x2 , . . . , xp .

The span of the rows of a matrix A ∈ Rm×n is the row space of A , denoted r.s.(A) :=
{y 0 A : y ∈ Rm } . Similarly, the span of the columns of a matrix A is the column space
of A , denoted c.s.(A) := {Ax : x ∈ Rn } . It is a simple fact that, for a matrix A ,
the dimension of its row space and the dimension of its column space are identical,
this common number being called the rank of A . The matrix A has full row rank if
its number of rows is equal to its rank. That is, if its rows are linearly independent.
Similarly, the matrix A has full column rank if its number of columns is equal to its
rank. That is, if its columns are linearly independent.
Besides the row and columns spaces of a matrix A ∈ Rm×n , there is another very
important vector space associated with A . The null space of A is the set of vectors
having 0 dot product with all rows of A , denoted n.s.(A) := {x ∈ Rn : Ax = 0} .
An important result is the following theorem relating the dimensions of the row
and null spaces of a matrix.

Theorem 1.4 (Rank-Nullity Theorem)

If A is a matrix with n columns, then

dim(r.s.(A)) + dim(n.s.(A)) = n .

There are some simple operations on a matrix that preserve its row and null spaces.
The following operations are elementary row operations:
1. multiply a row by a non-zero scalar;

2. interchange a pair of rows;

3. add a scalar multiple of a row to another row;

4. delete a row that is identically zero.

There is one more operation that we allow, which is really one of convenience rather
than mathematics. It is convenient to be able to permute columns while also permuting
the corresponding column indices. That is, if A ∈ Rm×n , we regard the columns as
labeled, in left-to-right order: 1, 2, . . . , n . So we have

A = [A1 , A2 , . . . , An ] .

It can be convenient to have a permutation σ1 , σ2 , . . . , σn of 1, 2, . . . , n , and then write

[Aσ1 , Aσ2 , . . . , Aσn ] .

6 CHAPTER 1. LET’S GET STARTED

This matrix is really equivalent to A , because we regard its columns as labeled by

σ1 , σ2 , . . . , σn rather than 1, 2, . . . , n . Put another way, when we write a matrix, the
order of the columns is at our convenience, but the labels of columns is determined by
the order that we choose for placing the columns.
The identity matrix Ir in Rr×r is the matrix having 1 as every diagonal element and
0 as every off-diagonal element. Via elementary row operations, any matrix A , that is
not all zero, can be transformed, via elementary row operations, into one of the form

[Ir , M ] .

Using corresponding operations on the associated system of equations, this is known

as Gauss-Jordan elimination.
For an r × r matrix B of rank r , there is a unique r × r matrix “B −1 ” such that
B B = Ir . For this reason, such a matrix B is called invertible, and B −1 is called the
−1

inverse of B . According to the definition, B −1 B = Ir , but we also have BB −1 = Ir .

Also, (B 0 )−1 = (B −1 )0 , and if A and B are both invertible, then (AB)−1 = B −1 A−1 .
Noting that,
B −1 [B, Ir ] = [Ir , B −1 ] ,
we see that there is a nice way to compute the inverse of a matrix B using elementary
row operations. That is, we perform elementary row operations on

[B, Ir ]

so that we have the form

[Ir , M ],
and the resulting matrix M is B −1 .
The Sherman-Morrison formula is a useful way to relate the inverse of a matrix to
the inverse of a rank-1 change to the matrix:

−1 B −1 uv 0 B −1
B + uv 0 = B −1 − ,
1 + v 0 B −1 u

where the r × r matrix B is invertible, u, v ∈ Rr , and it must be assumed that 1 +

v 0 B −1 u 6= 0 for otherwise B + uv 0 would not be invertible.
Next, we define the determinant of a square r × r matrix B , which we denote
det(B) . We define the determinant in a non-standard but useful manner, via a recursive
formula known as Laplace expansion.2
If r = 1 , then B = (b11 ) , and we define det(B) := b11 . For r > 1 , choose any fixed
column j of B , and we define
r
X
det(B) = (−1)i+j bij det(B ij ) ,
i=1

where B ij is the (r − 1) × (r − 1) matrix obtained by deleting row i and column j of B .

It is a fact that this is well defined — that is, the value of det(B) does not depend on the
1.4. EXERCISES 7

choice of j (taken at each step of the recursion). Moreover, we have det(B 0 ) = det(B) ,
so we can could as well choose any fixed row i of B , and we have
r
X
det(B) = (−1)i+j bij det(B ij ) ,
j=1

resulting in the same value for det(B) .

An interesting observation links det(B) with elementary row operations. Consider
performing elementary row operations on

[B, Ir ]

to obtain
[Ir , B −1 ] .
As we carry out the elementary row operations, we sometimes multiply a row by a
non-zero scalar. If we accumulate the product of all of these multipliers, the result is
det(B −1 ) ; equivalently, the reciprocal is det(B) .
Finally, for an invertible r × r matrix B and a vector b , we can express the unique
solution x̄ of the system Bx = b , via a formula involving determinants. Cramer’s rule
is the following formula:
det(B(j))
x̄j = , for j = 1, 2, . . . , r ,
det(B)
where B(j) is defined to be the matrix B with its j-th column replaced by b . It is worth
emphasizing that direct application of Cramer’s rule is not to be thought of as a useful
algorithm for computing the solution of a system of equations. But it can be very useful
to have in the proof toolbox.3

1.4 Exercises
Exercise 1.0 (Learn LATEX)
Learn to use LATEX for writing all of your homework solutions. Personally, I use
MiKTEX, which is an implementation of LATEX for Windows. Specifically, within MiKTEX,
I am using pdfLATEX (it only matters for certain things like including graphics and also
pdf into a document). I find it convenient to use the editor WinEdt, which is very LATEX
friendly. A good book on LATEX is

In A.1 there is a template to get started. Also, there are plenty of tutorials and beginner’s
guides on the web.
8 CHAPTER 1. LET’S GET STARTED

Exercise 1.1 (Convert to standard form)

Give an original example (i.e., with actual numbers) to demonstrate that you know
how to transform a general linear-optimization problem to one in standard form.

Exercise 1.2 (Weak Duality example)

Give an original example to demonstrate the Weak Duality Theorem.

Exercise 1.3 (Convert to ≤ form)

Describe a general recipe for transforming an arbitrary linear-optimization problem
into one in which all of the linear constraints are of ≤ type.

Exercise 1.4 (m + 1 inequalities)

Prove that the system of m equations in n variables Ax = b is equivalent to the system
Ax ≤ b augmented by only one additional linear inequality — that is, a total of only
m + 1 inequalities.

Exercise 1.5 (Weak duality for another form)

Give and prove a Weak Duality Theorem for

max c0 x
Ax ≤ b ; (P0 )
x ≥ 0.

HINT: Convert (P0 ) to a standard-form problem, and then apply the ordinary Weak
Duality Theorem for standard-form problems.

Exercise 1.6 (Weak duality for a complicated form)

Give and prove a Weak Duality Theorem for

min c0 x + f 0 w
Ax + Bw ≤ b ;
(P0 )
Dx = g;
x≥0 w≤0 .

HINT: Convert (P0 ) to a standard-form problem, and then apply the ordinary Weak
Duality Theorem for standard-form problems.

Exercise 1.7 (Weak duality for a complicated form — with MATLAB)

The MATLAB code below makes and solves an instance of (P0 ) from Exercise 1.6. Study
the code to see how it is works. Now, extend the code to solve the dual of (P0 ). Also, after
converting (P0 ) to standard form (as indicated in the HINT for Exercise 1.6), use MATLAB
to solve that problem and its dual. Make sure that you get the same optimal value for
all of these problems.

% DualityWithMatlab1.m // Jon Lee

%

n1=7
1.4. EXERCISES 9

n2=15
m1=2
m2=4

rng('default');
rng(1); % set seed

A = rand(m1,n1);
B = rand(m1,n2);
D = rand(m2,n1);

% Organize the situation

% so that the problem has a feasible solution
x = rand(n1,1);
w = −rand(n2,1);
b = A*x + B*w + 0.01 * rand(m1,1);
g = D*x;

% Organize the situation

% so that the dual problem has a feasible solution
y = −rand(m1,1);
pi = rand(m2,1) − rand(m2,1);
c = A'*y + D'*pi + 0.01 * rand(n1,1);
f = B'*y − 0.01 * rand(n2,1);

% Here is how the 'linprog' function works:

%
% [v,z,exitflag] = linprog(c,A,b,Aeq,beq,lb,ub,options)
% minimizes c'v ,
% with constraints A v <= b, Aeq v = beq, and
% variable bounds lb <= v <=ub.
% Some parts of the model can be null: for example,
% set Aeq = [] and beq = [] if no equalities exist.
% set ub = [] if no upper bounds exist.
% set options to choose algorithm
% [v,z,exitflag] = linprog(...) returns values:
% v = solution vector
% z = minimum objective value
% exitflag = describes the exit condition:
% 1 optimal solution found.
% −2 problem is infeasible.
% −3 problem is unbounded.
% −5 problem and its dual are both infeasible

options = optimoptions('linprog','Algorithm','dual−simplex');

% Study the following part carefully. This is how we set up

% a block problem. [c ; f] stacks vertically the vectors c and f.
% [A B] concatenates A and B horizontally. zeros(m2,n2) is an
% m2−by−n2 matrix of 0's. ones(n1,1) is an n1−vector of 1's, and
% Inf is +infinity, so Inf*ones(n1,1) is a vector of length n1
% with all components equal to +infinity. In this way we have
% infinite upper bounds on some variables.
[v,z,exitflag] = linprog([c ; f],[A B],b,[D zeros(m2,n2)],g, ...
[zeros(n1,1);−Inf*ones(n2,1)],[Inf*ones(n1,1);zeros(n2,1)],options)
10 CHAPTER 1. LET’S GET STARTED

if (exitflag < 1)
disp('fail 1: LP did not have an optimal solution');
return;
end;

% Extract the parts of the solution v to the formulation vectors

x = v(1:n1,1)
w = v(n1+1:n1+n2,1)
Chapter 2

Modeling

Our goals in this chapter are as follows:

• Learn some basic linear-optimization modeling techniques.
• Learn how to use an optimization modeling language like AMPL.

2.1 A Production Problem

We suppose that a company has m resources, available in quantities bi , i = 1, 2, . . . , m ,

and n production activities, with per-unit profits cj , j = 1, 2, . . . , n . Each unit of ac-
tivity j consumes aij units of resources i . Each production activity can be carried out

11
12 CHAPTER 2. MODELING

at any non-negative level, as long as the resources availabilities are respected. We as-
sume that any unused resource quantities have no value and can be disposed of at no
cost. The problem is to find a profit-maximizing production plan. We can formulate
this problem as the linear-optimization problem

max c0 x
Ax ≤ b ; (P)
x ≥ 0,

where b := (b1 , b2 , . . . , bm )0 , c := (c1 , c2 , . . . , cn )0 , A ∈ Rm×n is the matrix of aij , and x

is a vector of variables in Rn .
From the very same data, we can formulate a related linear-optimization problem.
The goal now is to set per-unit prices yi , for the resources i = 1, 2, . . . , m . The total cost
of purchasing the resources from the company is then y 0 b , and we wish to minimize
the total cost of obtaining the resources from the company. We want to set these prices
in such a way that the company would never have an incentive to carry out any of the
production activities versus selling Pmthe resources at the associated resources at these
prices. That is, we require that i=1 yi aij ≥ cj , for j = 1, 2, . . . , n . Because of our
assumption that the company can dispose of any unused quantities of resources at no
cost, we have yi ≥ 0 , for i = 1, 2, . . . , m . All in all, we have the linear-optimization
problem
min y 0 b
y 0 A ≥ c0 ; (D)
y ≥ 0,
Comparing this pair of linear-optimization problem with what you discovered in Ex-
ercise 1.5, we see that a Weak Duality Theorem holds: that is, the profit of any feasible
production plan is bounded above by the cost of the resources determined by any set
of prices that would render all production activities non-profitable.

2.2 Norm Minimization

“Norms” are very useful as a measure of the “size” of a vector. In some applica-
tions, we are interested in making the “size” small. There are many different “norms”
2.3. NETWORK FLOW 13

(for example, the Euclidean norm), but two are particularly interesting for linear opti-
mization.
For x ∈ Rn , the ∞-norm (or max-norm) of x is defined as

kxk∞ := max{|xj | : j = 1, 2, . . . , n}.

We would like to formulate the problem of finding an ∞-norm minimizing solution of

the system of equations Ax = b . This is quite easy, via the linear-optimization problem:

min t
t − xi ≥ 0 , i = 1, 2, . . . , n ;
t + xi ≥ 0 , i = 1, 2, . . . , n ;
Ax = b ,

where t ∈ R is an auxiliary variable. Notice how the minimization “pressure” ensures

that an optimal solution (x̂, t̂) has t̂ = maxnj=1 {|x̂j |} = kx̂k∞ . This would not work for
maximization!
The 1-norm of x is defined as
n
X
kxk1 := |xj | .
j=1

Now, we would like to formulate the problem of finding a 1-norm minimizing solu-
tion of the system of equations Ax = b . This is quite easy, via the linear-optimization
problem:
Pn
min j=1 tj
tj − xj ≥ 0 , j = 1, 2, . . . , n ;
tj + xj ≥ 0 , j = 1, 2, . . . , n ;
Ax = b ,
where t ∈ Rn is a vector of n auxiliary variables. Notice how the minimization “pres-
sure” ensures that an optimal solution (x̂, t̂) has t̂j = |x̂j | , for
Pjn = 1, 2, . . . , n (again,
this would not work for maximization!), and so we will have j=1 t̂j = kx̂k1 .

2.3 Network Flow

14 CHAPTER 2. MODELING

A finite network G is described by a finite set of nodes N and a finite set A of arcs.
Each arc e has two key attributes, namely its tail t(e) ∈ N and its head h(e) ∈ N . We
think of a (single) commodity as being allowed to “flow” along each arc, from its tail
to its head. Indeed, we have “flow” variables

xe := amount of flow on arc e ,

for e ∈ A . Formally, a flow x̂ on G is simply an assignment of any real numbers x̂e to

the variables xe , for e ∈ A . We assume that flow on arc e should be non-negative and
should not exceed
ue := the flow upper bound on arc e ,
for e ∈ A . Associated with each arc e is a cost

ce := cost per-unit-flow on arc e ,

for e ∈ A . The (total) cost of the flow x̂ is defined to be

X
ce xe .
e∈A

We assume that we have further data for the nodes. Namely,

bv := the net supply at node v ,

for v ∈ N . A flow is conservative if the net flow out of node v , minus the net flow into
node v , is equal to the net supply at node v , for all nodes v ∈ N .
The (single-commodity min-cost) network-flow problem is to find a minimum-
cost conservative flow that is non-negative and respects the flow upper bounds on the
arcs. We can formulate this as follows:
X
min ce xe
e∈A
X X
xe − xe = bv , ∀ v ∈ N ;
e∈A : e∈A :
t(e)=v h(e)=v

0 ≤ xe ≤ ue , ∀e∈A.
2.4. AN OPTIMIZATION MODELING LANGUAGE 15

2.4 An Optimization Modeling Language

Optimization modeling languages
facilitate rapid instantiation of math-
ematical optimization models with
data and the subsequent solution by
solvers. A well-known example is
AMPL (see www.ampl.com). The AMPL
book is available for free download at
www.ampl.com/BOOK/download.html.
In a disciplined approach, we set up
three files: *.mod , *.dat and *.run . The
*.mod file sets up the model, the *.dat
contains the data, and the *.run file in-
stantiates the model with the data, passes
the instance to the solver, retrieves the so-
lution from the solver, and can be used to
report on the solution. In more complex
uses, the *.run file has control commands
that allow for iterative algorithms, possi-
bly instantiating and running many mod-
els.

For a simple case, we consider the Production Problem of Section 2.1. Note that (i)
on each line of the AMPL input files, anything after a symbol “#” is treated as a comment,
and (ii) commands are terminated with “;” The *.mod file can be production.mod :

# production.mod // Jon Lee

#

param m integer > 0; # NUMBER OF ROWS

param n integer > 0; # NUMBER OF COLUMNS
set ROWS := {1..m}; # DEFINING THE SET OF ROW INDICES
set COLS := {1..n}; # DEFINING THE SET OF COLUMNS INDICES

param b {ROWS}; # RIGHT−HAND SIDE DATA

param c {COLS}; # OBJECTIVE−FUNCTION DATA
param a {ROWS,COLS}; # CONSTRAINT−MATRIX DATA

var x {j in COLS} >= 0; # DEFINING THE (NONNEGATIVE) VARIABLES

maximize z: # CHOOSE MAX/MIN AND NAME THE OBJ. FUNCTION

sum {j in COLS} c[j] * x [j]; # DEFINING THE OBJECTIVE FUNCTION

subject to Constraints {i in ROWS}: # DEFINING THE CONSTRAINT INDICES

sum {j in COLS} a[i,j] * x[j] <= b[i]; # DEFINING THE CONSTRAINTS
16 CHAPTER 2. MODELING

For a sample data set, we can have production.dat :

# production.dat // Jon Lee

#

param m := 3; # NUMBER OF ROWS

param b := # RIGHT−HAND SIDE VECTOR

# i b(i)
1 32
2 33
3 35 ;

param n := 2; # NUMBER OF COLUMNS

param c := # OBJECTIVE VECTOR

# j c(j)
1 3
2 2 ;

param a: 1 2 := # CONSTRAINT MATRIX

# a(i,j):
1 8 5
2 8 6
3 8 7 ;

Finally, we can have production.run :

# production.run // Jon Lee

#

reset; # CLEAR FROM ANY PREVIOUS SOLVE

option show_stats 1; # SET AMPL OPTION TO CONTROL OUTPUT

option presolve 0; # COMMENT OUT TO SOLVE BIGGER MODELS
option solver cplex; # AMPL OPTION TO CHOOSE AN AVAILABLE SOLVER
option cplex_options 'display=2 presolve=0'; # SETTING SOLVER OPTIONS
# DELETING "presolve=0" MAY HELP TO SOLVE BIGGER MODELS

model production.mod; # READ IN THE MODEL

data production.dat; # READ IN THE DATA AND INSTANTIATE THE MODEL
solve; # PASS THE INSTANCE TO THE SOLVER

display z > production.out; # WRITE THE OPTIMAL OBJECTIVE VALUE TO A FILE

# NOTE THAT '>' DELETES THE FILE FIRST, IF IT EXISTS
for {j in COLS}
{ # USING C−STYLE COMMANDS TO OUTPUT OPTIMAL SOLUTION TO FILE
printf "x(%i) = %f\n", j, x[j] >> production.out;
# NOTE THAT '>>' APPENDS TO THE FILE, IF IT EXISTS
};
2.4. AN OPTIMIZATION MODELING LANGUAGE 17

At the University of Michigan, College of Engineering, we have AMPL with the solver
CPLEX installed on the CAEN (Computer Aided Engineering Network: caen.engin.umich.edu)
machines running Red Hat Linux.
We check that we have the three files needed to run our Production Problem:

caen-vnc02% ls

production.dat production.mod production.run

Next, we invoke AMPL from a command prompt, and invoke the production.run
file:

caen-vnc02% ampl
ampl: include production.run;

2 variables, all linear

3 constraints, all linear; 6 non-zeros
1 linear objective; 2 non-zeros.

CPLEX 12.2.0.0: display=2

presolve=0
Parallel mode: deterministic, using up to 2 threads for concurrent optimization.
CPLEX 12.2.0.0: optimal solution; objective 12.125
3 dual simplex iterations (1 in phase I)
ampl:

Next, we quit out of AMPL from its command prompt:

ampl: quit;

And we can view the output file:

caen-vnc02% cat production.out

z = 12.125

x(1) = 3.375
x(2) = 1.000
caen-vnc02%

Next, we see a way to set up a (single-commodity min-cost) network-flow problem.

The *.mod file can be flow.mod :

# flow.mod // Jon Lee

#

set NODES;

set ARCS within NODES cross NODES;

18 CHAPTER 2. MODELING

# TELLS AMPL TO CHECK THAT EACH ARC IS AN ORDERED PAIR OF NODES

param b {NODES};

param upper {ARCS};

param c {ARCS};

var x {(i,j) in ARCS} >= 0, <= upper[i,j];

minimize z:
sum {(i,j) in ARCS} c[i,j] * x[i,j];

subject to Flow_Conservation {i in NODES}:

sum {(i,j) in ARCS} x[i,j] − sum {(j,i) in ARCS} x[j,i] = b[i];

There is a subtlety concerning the summations in the constraints. In each constraint,

the node i is fixed. So ‘sum{(i,j) in ARCS}’ is really summing over j in NODES such
that (i,j) is in ARCS ; that is, the arcs whose tail is i . Similarly, ‘sum{(j,i) in ARCS}’
is really summing over j in NODES such that (j,i) is in ARCS ; that is, the arcs whose
head is i .
The flow.dat file can be:

# flow.dat // Jon Lee

#

param: NODES: b :=
1 12
2 6
3 −2
4 0
5 −9
6 −7 ;

param: ARCS: upper c :=

1,2 6 2
1,3 8 −5
2,4 5 3
2,5 7 12
3,5 5 −9
4,5 8 2
4,6 5 0
5,6 5 4 ;

We leave it to the gentle reader to devise an appropriate file flow.run (the optimal
objective function value is 25).
2.5. EXERCISES 19

2.5 Exercises
Exercise 2.1 (Dual in AMPL)
Without changing the file production.dat , use AMPL to solve the dual of the Production
Problem example, as described in Section 2.1. You will need to modify production.mod
and production.run .
Exercise 2.2 (Sparse solution for linear equations with AMPL)
In some application areas, it is interesting to find a “sparse solution” — that is, one with
few non-zeros — to a system of equations Ax = b, on say the domain −1 ≤ xj ≤ +1,
for j = 1, 2, . . . , n.
It is empirically well known that a 1-norm minimizing solution is a good heuristic
for finding a sparse solution. The moral justification of this is as follows. We define the
function indicator function I6=0 : R 7→ R by

1, w 6= 0;
I6=0 (w) :=
0, w = 0.

It is easy to see (make a graph) that f (w) := |w| is the “best convex function Pnunder-
estimator” of I6=0 on the domain
P [−1, 1]. So we can hope that minimizing j=1 |x|j
comes close to minimizing nj=1 I6=0 (xj ) .
Using AMPL, try this idea out on several large examples, using 1-norm minimization
as a heuristic for finding a sparse solution.
HINT: To get an interesting example, try generating a random m × n matrix A of
zeros and ones, perhaps m = 50 equations and n = 500 variables, maybe with prob-
m
ability 1/2 of an entry being equal to one. Next, choose a random z̃ ∈ R 2 satisfying
−1 ≤ z̃j ≤ +1, for j = 1, 2, . . . , m/2, and z̃j = 0 for j = m/2 + 1, . . . , n. Now let
b := Az̃. In this way, you will know that there is a solution (i.e., z̃) with only m/2 non-
zeros (which is already pretty sparse). Your 1-norm minimizing solution might in fact
recover this solution (,), or it may be sparser (,,), or perhaps less sparse (/).

Exercise 2.3 (Bloody AMPL)

A transportation problem is a special kind of (single-commodity min-cost) network-
flow problem. There are certain nodes v called supply nodes which have net supply
bv > 0. The other nodes v are called demand nodes, and they have net supply bv < 0.
There are no nodes with bv = 0 , and all arcs point from supply nodes to demand nodes.
A simplified example is for matching available supply and demand of blood, in
types A, B, AB and O . Suppose that we have sv units of blood available, in types
v ∈ {A, B, AB, O} . Also, we have requirements dv by patients of different types v ∈
{A, B, AB, O} . It is very important to understand that a patient of a certain type can
accept blood not just from their own type. Do some research to find out the compatible
blood types for a patient; don’t make a mistake — lives depend on this! In this spirit, if
your model allocates any blood in an incompatible fashion, you will receive a grade of F on this
problem.
Describe a linear-optimization problem that satisfies all of the patient demand with
compatible blood. You will find that type O is the most versatile blood, then both A and
B, followed by AB. Factor in this point when you formulate your objective function,
with the idea of having the left-over supply of blood being as versatile as possible.
20 CHAPTER 2. MODELING

Using AMPL, set up and solve an example of a blood-distribution problem.

Exercise 2.4 (Mix it up)

“I might sing a gospel song in Arabic or do something in Hebrew. I want to mix it
up and do it differently than one might imagine.” — Stevie Wonder
We are given a set of ingredients 1, 2, . . . , m with availabilities bi , measured in kilo-
grams, and per kilogram costs ci . We are given a set of products 1, 2, . . . , n with mini-
mum production requirements dj , measured in kilograms, and per kilogram revenues
ej . It is required that product j have at least a fraction (by weight) of lij of ingredient i
and at most a fraction (by weight) of uij of ingredient i . The goal is to devise a plan to
maximize net profit.
Formulate, mathematically, as a linear-optimization problem. Then, model with
AMPL, make up some data, try some computations, and report on your results.

Exercise 2.5 (Task scheduling)

We are given a set of tasks, numbered 1, 2, . . . , n that should be completed in the mini-
mum amount of time. For convenience, task 0 is a “start task” and task n + 1 is an “end
task”. Each task, except for the start and end task, has a known duration di . For con-
venience, let d0 := 0 . Any number of tasks can be carried out simultaneously, except
that there are precedences between tasks. Specifically, Ψi is the set of tasks that must
be completed before task i can be started. Let t0 := 0 , and for all other tasks i , let ti be
a decision variable representing its start time.
Formulate the problem, mathematically, as a linear-optimization problem. The ob-
jective should be to minimize the start time tn+1 of the end task. Then, model the prob-
lem with AMPL, make up some data, try some computations, and report on your results.

Exercise 2.6 (Investing wisely)

Almost certainly, Albert Einstein did not say that “compound interest is the most pow-
erful force in the universe.”
A company wants to maximize their cash holdings after T time periods. They have
an external inflow of pt dollars at the start of time period t , for t = 1, 2, . . . , T . At the
start of each time period, available cash can be allocated to any of K different invest-
ment vehicles (in any available non-negative amounts). Money allocated to investment-
vehicle k at the start of period t must be held in that investment k for all remaining time
periods, and it generates income vt,tk , vk
t,t+1 , . . . , vt,T , per dollar invested. It should be
k
2.5. EXERCISES 21

assumed that money obtained from cashing out the investment at the end of the plan-
ning horizon (that is, at the end of period T ) is part of vt,T
k . Note that at the start of

time period t , the cash available is the external inflow of pt , plus cash accumulated
from all investment vehicles in prior periods that was not reinvested. Finally, assume
that cash held over for one time period earns interest of q percent.
Formulate the problem, mathematically, as a linear-optimization problem. Then,
model the problem with AMPL, make up some data, try some computations, and report
on your results.
Chapter 3

Algebra Versus Geometry

Our goals in this chapter are as follows:

• Develop the algebra needed later for our algorithms.
• Develop some geometric understanding of this algebra.

Throughout, we refer to the standard-form problem

min c0 x
Ax = b ; (P)
x ≥ 0.

3.1 Basic Feasible Solutions and Extreme Points

A basic partition of A ∈ Rm×n is a partition of {1, 2, . . . , n} into a pair of ordered sets,
the basis β = (β1 , β2 , . . . , βm ) and the non-basis η = (η1 , η2 , . . . , ηn−m ), so that the basis

23
24 CHAPTER 3. ALGEBRA VERSUS GEOMETRY

matrix Aβ := [Aβ1 , Aβ2 , . . . , Aβm ] is an invertible m×m matrix. The connection with the
standard “linear-algebra basis” is that the columns of Aβ form a “linear-algebra basis”
for Rm . But for us, “basis” almost always refers to β.
We associate a basic solution x̄ ∈ Rn with the basic partition via:

x̄η := 0 ∈ Rn−m ;
x̄β := A−1
β b ∈R .
m

We can observe that x̄β = A−1β b is equivalent to Aβ x̄β = b , which is the unique way
to write b as a linear combination of the columns of Aβ . Of course this makes sense,
because the columns of Aβ form a “linear-algebra basis” for Rm .
Note that every basic solution x̄ satisfies Ax̄ = b , because
n
X X X  
Ax̄ = Aj x̄j = Aj x̄j + Aj x̄j = Aβ x̄β + Aη x̄η = Aβ A−1
β b + Aη 0 = b .
j=1 j∈β j∈η

A basic solution x̄ is a basic feasible solution if it is feasible for (P). That is, if x̄β =
A−1
β b≥0.
It is instructive to have a geometry for understanding the algebra of basic solutions,
but for standard-form problems, it is hard to draw something interesting in two di-
mensions. Instead, we observe that the feasible region of (P) is the solution set, in Rn ,
of
xβ + A−1 β Aη xη = Aβ b ;
−1

xβ ≥ 0 , xη ≥ 0 .

Projecting this onto the space of non-basic variables xη ∈ Rn−m , we obtain

 
A−1
β A η xη ≤ A−1
β b;
xη ≥ 0 .

Notice how we can view the xβ variables as slack variables.

Example 3.1
For this system, it is convenient to draw pictures when n−m = 2 , for example n = 6 and
m = 4 . In such a picture, the basic solution x̄ ∈ Rn maps to the origin x̄η = 0 ∈ Rn−m ,
but other basic solutions (feasible and not) will map to other points.
Suppose that we have the data:
 
1 2 1 0 0 0
 3 1 0 1 0 0 
A := 
 3/2 3/2 0
 ,
0 1 0 
0 1 0 0 0 1
b := (7, 9, 6, 33/10)0 ,
β := (β1 , β2 , β3 , β4 ) = (1, 2, 4, 6) ,
η := (η1 , η2 ) = (3, 5) .
3.1. BASIC FEASIBLE SOLUTIONS AND EXTREME POINTS 25

Then
 
1 2 0 0
 3 1 1 0 
Aβ = [Aβ1 , Aβ2 , Aβ3 , Aβ4 ] =  3/2 3/2 0
,
0 
0 1 0 1
 
1 0
 0 0 
Aη = [Aη1 , Aη2 ] = 
 0
 ,
1 
0 0
xβ = (x1 , x2 , x4 , x6 )0
xη := (x3 , x5 )0 .

We can calculate
 
−1 4/3
 1 −2/3 
Aβ−1 Aη =  
 2 −10/3  ,
−1 2/3
A−1 0
β b := (1, 3, 3, 3/10) ,

and then we have plotted this in Figure 3.1. The plot has xη1 = x3 as the abscissa, and
xη2 = x5 as the ordinate. In the plot,
 besides the non-negativity of the variables x3 and
x4 , the four inequalities of Aβ−1 Aη xη ≤ A−1 β b are labeled with their slack variables —
these are the basic variables
 x1 ,x2 , x4 , x6 . The correct matching of the basic variables
to the inequalities of A−1 −1
β Aη xη ≤ Aβ b is simply achieved by seeing that the i-th
inequality has slack variable xβi .
The feasible region is colored cyan, while basic feasible solutions project to green
points and basic infeasible solutions project to red points. We can see that the basic
solution associate with the current basis is feasible, because the origin (corresponding
to the non-basic variables being set to 0) is feasible.

A set S ⊂ Rn is a convex set if it contains the entire line segment between every pair
of points in S . That is,

λx1 + (1 − λ)x2 ∈ S , whenever x1 , x2 ∈ S and 0 < λ < 1 .

It is simple to check that the feasible region of every linear-optimization problem is a

convex set — do it!
For a convex set S ⊂ Rn , a point x̂ ∈ S is an extreme point of S if it is not on the
interior of any line segment wholly contained in S . That is, if we cannot write

x̂ = λx1 + (1 − λ)x2 , with x1 6= x2 ∈ S and 0 < λ < 1 .

26 CHAPTER 3. ALGEBRA VERSUS GEOMETRY

Figure 3.1: Feasible region projected into the space of non-basic variables

Theorem 3.2
Every basic feasible solution of standard-form (P) is an extreme point of its feasible
region.

Proof. Consider the basic feasible solution x̄ with

x̄η := 0 ∈ Rn−m ;
−1
x̄β := Aβ b ∈ Rm .

If
x̄ = λx1 + (1 − λ)x2 , with x1 and x2 feasible for (P) and 0 < λ < 1 ,
3.1. BASIC FEASIBLE SOLUTIONS AND EXTREME POINTS 27

then 0 = x̄η = λx1η + (1 − λ)x2η and 0 < λ < 1 implies that x1η = x2η = 0 . But then
Aβ xiβ = b implies that xiβ = A−1
β b = x̄β , for i = 1, 2 . Hence x̄ = x = x (but we
1 2

needed x1 6= x2 ), and so we cannot find a line segment containing x̄ that is wholly

contained in S . t
u

Theorem 3.3
Every extreme point of the feasible region of standard-form (P) is a basic solution.

Proof. Let x̂ be an extreme point of the feasible region of (P). We define

ρ := {j ∈ {1, 2, . . . , n} : x̂j > 0} .
That is, ρ is the list of indices for the positive variables of x̂ . Also, we let
ζ := {j ∈ {1, 2, . . . , n} : x̂j = 0} .
That is, ζ is the list of indices for the zero variables of x̂ . Together, ρ and ζ partition
{1, 2, . . . , n} .
Our goal is to construct a basic partition, β, η , so that the associated basic solution
is precisely x̂ .
The first thing that we will establish is that the columns of Aρ are linearly indepen-
dent. We will do that by contradiction. Suppose that they are linearly dependent. That
is, there exists zρ ∈ R|ρ| different from the zero vector, such that Aρ zρ = 0 . Next we ex-
tend zρ to a vector z ∈ Rn , by letting zζ = 0 . Clearly Az = Aρ zρ + Aζ zζ = 0 + Aζ 0 = 0 ;
that is, z is in the null space of A . Next, let
x1 := x̂ + z
and
x2 := x̂ − z ,
with  chosen to be sufficiently small so that x1 and x2 are non-negative. Because z is
only non-zero on the ρ coordinates (where x̂ is positive), we can choose an appropriate
 . Notice that x1 6= x2 , because zρ and hence z is not the zero vector. Now, it is easy to
verify that Ax1 = A(x̂ + z) = Ax̂ + Az = b + 0 = b and similarly Ax2 = b . Therefore,
x1 and x2 are feasible solutions of (P). Also, 12 x1 + 21 x2 = 12 (x̂ + z) + 21 (x̂ − z) = x̂ . So
x̂ is on the interior (actually it is the midpoint) of the line segment between x1 and x2 ,
in contradiction to x̂ being an extreme point of the feasible region of (P). Therefore, it
must be that the columns of Aρ are linearly independent.
In particular, we can conclude that |ρ| ≤ m , since we assume that A ∈ Rm×n has
full row rank. If |ρ| < m , we choose (via Theorem 1.3) m−|ρ| columns of Aζ to append
to Aρ in such a way as to form a matrix Aβ having m linearly-independent columns —
we note that such a choice is not unique. As usual, we let η be a list of the n − m indices
not in β . By definition, the associated basic solution x̄ has x̄η = 0 , and we observe that
it is the unique solution to the system of equations Ax = b having xη = 0 . But x̂η = 0
because x̂η is a subvector of x̂ζ = 0 . Therefore, x̂ = x̄ . That is, x̂ is a basic solution of
(P). t
u
28 CHAPTER 3. ALGEBRA VERSUS GEOMETRY

Taken together, these last two results give us the main result of this section.

Corollary 3.4
For a feasible point x̂ of standard-form (P), x̂ is extreme if and only if x̂ is a basic
solution.

3.2 Basic Feasible Directions

For a point x̂ in a convex set S ⊂ Rn , a feasible direction relative to the feasible

solution x̂ is a ẑ ∈ Rn such that x̂ + ẑ ∈ S , for sufficiently small positive  ∈ R .
Focusing now on the standard-form problem (P), for ẑ to be a feasible direction relative
to the feasible solution x̂ , we need A(x̂ + ẑ) = b . But

b = A(x̂ + ẑ) = Ax̂ + Aẑ = b + Aẑ ,

so we need Aẑ = 0 . That is, ẑ must be in the null space of A .

Focusing on the standard-form problem (P), we associate a basic direction z̄ ∈ Rn
with the basic partition β, η and a choice of non-basic index ηj via

z̄η := ej ∈ Rn−m ;
z̄β := −A−1
β Aη j ∈ Rm .

Note that every basic direction z̄ is in the null space of A :

 
Az̄ = Aβ z̄β + Aη z̄η = Aβ −A−1 β A ηj + Aη ej = −Aηj + Aηj = 0 .

So
A (x̂ + z̄) = b ,
3.2. BASIC FEASIBLE DIRECTIONS 29

for every feasible x̂ and every  ∈ R . Moving a positive amount in the direction z̄ corre-
sponds to increasing the value of xηj , holding the values of all other non-basic variables
constant, and making appropriate changes in the basic variables so as to maintain sat-
isfaction of the equation system Ax = b .
There is a related point worth making. We have just seen that for a given basic
partition β, η , each of the n − m basic directions is in the null space of A — there is one
such basic direction for each of the n − m choices of ηj . It is very easy to check that
these basic directions are linearly independent — just observe that they are columns of
the n × (n − m) matrix  
I
.
−A−1
β Aη
Because the dimension of the null space of A is n − m , these n − m basic directions
form a basis for the null space of A .
Now, we focus on the basic feasible solution x̄ determined by the basic partition
β, η . The basic direction z̄ is a basic feasible direction relative to the basic feasible
solution x̄ if x̄ + z̄ is feasible, for sufficiently small positive  ∈ R . That is, if

A−1 −1
β b − Aβ Aηj ≥ 0 ,

for sufficiently small positive  ∈ R .

Recall that x̄β = A−1 −1
β b , and let Āηj := Aβ Aηj . So, we need that

x̄β − Āηj ≥ 0 ,

for sufficiently small positive  ∈ R . That is,

x̄βi − āi,ηj ≥ 0 ,

for i = 1, 2, . . . , m . If āi,ηj ≤ 0 , for some i , then this imposes no restriction at all on  .

So, the only condition that we need for z̄ to be a basic feasible direction relative to the basic
feasible solution x̄ is that there exists  > 0 satisfying
x̄βi
≤ , for all i such that āi,ηj > 0 .
āi,ηj
Equivalently, we simply need that

x̄βi > 0 , for all i such that āi,ηj > 0 .

So, we have the following result:

Theorem 3.5
For a standard-form problem (P), suppose that x̄ is a basic feasible solution relative to
the basic partition β, η . Consider choosing a non-basic index ηj . Then the associated
basic direction z̄ is a feasible direction relative to x̄ if and only if

x̄βi > 0 , for all i such that āi,ηj > 0 .

30 CHAPTER 3. ALGEBRA VERSUS GEOMETRY

3.3 Basic Feasible Rays and Extreme Rays

For a non-empty convex set S ⊂ Rn , a ray of S is a ẑ 6= 0 in Rn such that x̂ + τ ẑ ∈ S ,
for all x̂ ∈ S and all positive τ ∈ R .
Focusing on the standard-from problem (P), it is easy to see that ẑ 6= 0 is a ray of
the feasible region if and only if Aẑ = 0 and ẑ ≥ 0.
Recall from Section 3.2 that for a standard-form problem (P), a basic direction z̄ ∈
R is associated with the basic partition β, η and a choice of non-basic index ηj via
n

z̄η := ej ∈ Rn−m ;
z̄β := −A−1
β Aη j ∈ Rm .

If the basic direction z̄ is a ray, then we call it a basic feasible ray. We have already
seen that Az̄ = 0. Furthermore, z̄ ≥ 0 if and only if Āηj := A−1 β Aηj ≤ 0.
Therefore, we have the following result:

Theorem 3.6
The basic direction z̄ is a ray of the feasible region of (P) if and only if Āηj ≤ 0 .

Recall, further, that z̄ is a basic feasible direction relative to the basic feasible solution x̄
if x̄ + z̄ is feasible, for sufficiently small positive  ∈ R . Therefore, if z̄ is a basic feasible
ray, relative to the basic partition β, η and x̄ is the basic feasible solution relative to the
same basic partition, then z̄ is a basic feasible direction relative to x̄ .
A ray ẑ of a convex set S is an extreme ray if we cannot write

ẑ = z 1 + z 2 , with z 1 6= µz 2 being rays of S and µ 6= 0 .

Similarly to the correspondence between basic feasible solutions and extreme points for
standard-form problems, we have the following two results.

Theorem 3.7
Every basic feasible ray of standard-form (P) is an extreme ray of its feasible region.

Theorem 3.8
Every extreme ray of the feasible region of standard-form (P) is a positive multiple of
a basic feasible ray.
3.4. EXERCISES 31

3.4 Exercises
Exercise 3.1 (Illustrate algebraic and geometric concepts)
Make a small example, say with six variables and four equations, to fully illustrate all
of the concepts in this chapter. In A.2, there are MATLAB scripts and functions that could
be very helpful here.

Exercise 3.2 (Basic feasible rays are extreme rays)

Prove Theorem 3.7.

Exercise 3.3 (Extreme rays are positive multiples of basic feasible rays)
If you are feeling very ambitious, prove Theorem 3.8.

Exercise 3.4 (Dual basic direction — do this if you will be doing Exercise 4.2)
Let β, η be a basic partition for our standard-form problem (P). As you will see on the
first page of the next chapter, we can associate with the basis β, a dual solution

ȳ 0 := c0β A−1
β

of
max y 0 b
(D)
y 0 A ≤ c0 .
It is easy to see that ȳ satisfies the constraints y 0 Aβ ≤ c0β (of (D)) with equality; that is,
the dual constraints indexed from β are “active”.
Let us assume that ȳ is feasible for (D). Now, let β` be a basic index, and let w̄ := H`·
be row ` of H := A−1 β . Consider ȳ := ȳ − λw̄ , and explain (with algebraic justification)
˜ 0

what is happening to the activity of each constraint of (D), as λ increases. HINT: Think
about the cases of (i) i = `, (ii) i ∈ β, i 6= `, and (iii) j ∈ η .
Chapter 4

The Simplex Algorithm

Our goal in this chapter is as follows:

• Develop a mathematically-complete Simplex Algorithm for optimizing standard-

form problems.

4.1 A Sufficient Optimality Criterion

Definition 4.1
The dual solution of (D) associated with basis β is

ȳ 0 := c0β A−1
β .

Lemma 4.2
If β is a basis, then the primal basic solution x̄ (feasible or not) and the dual solution
ȳ (feasible or not) associated with β have equal objective value.

Proof. The objective value of x̄ is c0 x̄ = c0β x̄β + c0η x̄η = c0β (A−1 0 −1
β b) + cη 0 = cβ Aβ b . The
0

objective value of ȳ is ȳ 0 b = (c0β A−1 0 −1

β )b = cβ Aβ b . t
u

33
34 CHAPTER 4. THE SIMPLEX ALGORITHM

Definition 4.3
The vector of reduced costs associated with basis β is

c̄0 := c0 − c0β A−1 0 0

β A = c − ȳ A .

Lemma 4.4
The dual solution of (D) associated with basis β is feasible for (D) if

c̄η ≥ 0 .

Proof. Using the definitions of ȳ and c̄ , the condition c̄η ≥ 0 is equivalent to

ȳ 0 Aη ≤ c0η .

The definition of ȳ gives us

ȳ 0 Aβ = c0β (equivalently, c̄β = 0) .

So we have
ȳ 0 [Aβ , Aη ] ≤ (c0β , c0η ) ,
or, equivalently,
ȳ 0 A ≤ c0 ,
Hence ȳ is feasible for (D). t
u

Theorem 4.5 (Weak Optimal Basis Theorem)

If β is a feasible basis and c̄η ≥ 0 , then the primal solution x̄ and the dual solution ȳ
associated with β are optimal.

Proof. We have already observed that c0 x̄ = ȳ 0 b for the pair of primal and dual solutions
associated with the basis β . If these solutions x̄ and ȳ are feasible for (P) and (D),
respectively, then by weak duality these solutions are optimal. t
u

We can also take (P) and transform it into an equivalent form that is quite revealing.
Clearly, (P) is equivalent to

min c0β xβ + c0η xη

Aβ xβ + Aη x η = b
xβ ≥ 0 , x η ≥ 0 .

Next, multiplying the equations on the left by A−1

β , we see that they are equivalent to

xβ + A−1 −1
β Aη x η = Aβ b .
4.2. THE SIMPLEX ALGORITHM WITH NO WORRIES 35

We can also see this as

xβ = A−1 −1
β b − Aβ Aη xη .

Using this equation to substitute for xβ in the objective function, we are led to the linear
objective function
 
c0β A−1
β b + min c0
η − c0 −1
A
β β A 0 −1 0
η xη = cβ Aβ b + min c̄η xη ,

which is equivalent to the original one on the set of points satisfying Ax = b . In this
equivalent form, it is now solely expressed in terms of xη . Now, if c̄η ≥ 0 , the best we
could hope for in minimizing is to set xη = 0 . But the unique solution having xη = 0
is the basic feasible solution x̄ . So that x̄ is optimal.

Example 4.6
This is a continuation of Example 3.1. In Figure 4.1, we have depicted the sufficient
optimality criterion, in the space of a particular choice of non-basic variables — not the
choice previously depicted. Specifically, we consider the equivalent problem

min c̄0η xη
Āη xη ≤ x̄β ;
xη ≥ 0 .

This plot demonstrates the optimality of β := (5, 4, 6, 3) (η := (2, 1)). The basic direc-
tions available from the basic feasible solution x̄ appear as standard unit vectors in the
space of the non-basic variables. The solution x̄ is optimal because c̄η ≥ 0 ; we can also
think of this as c̄η having a non-negative dot product with each of the standard unit
vectors, hence neither direction is improving.

4.2 The Simplex Algorithm with No Worries

Improving direction. Often it is helpful to directly refer to individual elements of the

vector c̄η ; namely,

c̄ηj = cηj − c0β A−1 0

β Aηj = cηj − cβ Āηj , for j = 1, 2, . . . , n − m .
36 CHAPTER 4. THE SIMPLEX ALGORITHM

Figure 4.1: Sufficient optimality criterion

If the sufficient optimality criterion is not satisfied, then we choose an ηj such that c̄ηj
is negative, and we consider solutions that increase the value of xηj up from x̄ηj = 0 ,
changing the values of the basic variables to insure that we still satisfy the equations
Ax = b , while holding the other non-basic variables at zero.
Operationally, we take the basic direction z̄ ∈ Rn defined by

z̄η := ej ∈ Rn−m ;
z̄β := −A−1
β Aηj = −Āηj ∈ Rm ,

and we consider solutions of the form x̄ + λz̄ , with λ > 0 . The motivation is based on
the observations that

• c0 (x̄ + λz̄) − c0 x̄ = λc0 z̄ = λc̄ηj < 0 ;

• A(x̄ + λz̄) = Ax̄ + λAz̄ = b + λ0 = b .

That is, the objective function changes at the rate of c̄ηj , and we maintain satisfaction
of the Ax = b constraints.

Maximum step — the ratio test and a sufficient unboundedness criterion. By our
choice of direction z̄, all variables that are non-basic with respect to the current choice
of basis remain non-negative (xηj increases from 0 and the others remain at 0). So the
4.2. THE SIMPLEX ALGORITHM WITH NO WORRIES 37

only thing that restricts our movement in the direction z̄ from x̄ is that we have to make
sure that the current basic variables remain non-negative. This is easy to take care of.
We just make sure that we choose λ > 0 so that

x̄β + λz̄β = x̄β − λĀηj ≥ 0 .

Notice that for i such āi,ηj ≤ 0 , there is no limit on how large λ can be. In fact, it can well
happen that Āηj ≤ 0 . In this case, x̄ + λz̄ is feasible for all λ > 0 and c0 (x̄ + λz̄) → −∞
as λ → +∞ , so the problem is unbounded.
Otherwise, to insure that x̄ + λz̄ ≥ 0 , we just enforce
x̄βi
λ≤ , for i such that āi,ηj > 0 .
āi,ηj

Finally, to get the best improvement in the direction z̄ from x̄, we let λ equal
 
x̄βi
λ̄ := min .
i : āi,ηj >0 āi,ηj

Non-degeneracy. There is a significant issue in even carrying out one iteration of this
algorithm. If x̄βi = 0 for some i such that āi,ηj > 0 , then λ̄ = 0 , and we are not able to
make any change from x̄ in the direction z̄ . Just for now, we will simply assume away
this problem, using the following hypothesis that every basic variable of every basic
feasible solution is positive.
Definition 4.7
The problem (P) satisfies the non-degeneracy hypothesis if for every feasible basis β,
we have x̄βi > 0 for i = 1, 2, . . . , m .

Under the non-degeneracy hypothesis, λ̄ > 0 .

Another basic feasible solution. By our construction, the new solution x̄ + λ̄z̄ is fea-
sible and has lesser objective value than that of x̄ . We can repeat the construction as
long as the new solution is basic. If it is basic, there is a natural guess as to what an ap-
propriate basis may be. The variable xηj , formerly non-basic at value 0 has increased
to λ̄ , so clearly it must become basic. Also, at least one variable that was basic now has
value 0 . In fact, under our non-degeneracy hypothesis, once we establish that the new
solution is basic, we observe that exactly one variable that was basic now has value 0.
Let  
∗ x̄βi
i := argmin .
i : āi,ηj >0 āi,ηj

If there is more than one i that achieves the minimum (which can happen if we do not
assume the non-degeneracy hypothesis), then we will see that the choice of i∗ can be
any of these. We can see that xβi∗ has value 0 in x̄ + λ̄z̄ . So it is natural to hope we can
replace xβi∗ as a basic variable with xηj .
Let
β̃ := (β1 , β2 , . . . , βi∗ −1 , ηj , βi∗ +1 , . . . , βm )
38 CHAPTER 4. THE SIMPLEX ALGORITHM

and
η̃ := (η1 , η2 , . . . , ηj−1 , βi∗ , ηj+1 , . . . , ηn−m ) .

Lemma 4.8
Aβ̃ is invertible.

Proof. Aβ̃ is invertible precisely when the following matrix is invertible:

 
A−1
β A β̃ = A −1
β A β 1 , A β 2 , . . . , A β i−1
, Aηj , Aβi+1
, . . . , A β m
 
= e1 , e2 , . . . , ei−1 , Āηj , ei+1 , . . . , em .

But the determinant of this matrix is precisely āi,ηj 6= 0 . t

u

Lemma 4.9
The unique solution of Ax = b having xη̃ = 0 is x̄ + λ̄z̄ .

Proof. x̄η̃ + λ̄z̄η̃ = 0 . Moreover, x̄ + λ̄z̄ is the unique solution to Ax = b having xη̃ = 0
because Aβ̃ is invertible. t
u

Putting these two lemmata together, we have the following key result.

Theorem 4.10
x̄ + λ̄z̄ is a basic solution; in fact, it is the basic solution determined by β̃ .

Passing from the partition β, η to the partition β̃, η̃ is commonly referred to as a

pivot.

Worry-Free Simplex Algorithm

Input: c ∈ Rn , b ∈ Rm , A ∈ Rm×n of full row rank m , for the standard-form problem:

min c0 x
Ax = b ; (P)
x ≥ 0,

where x is a vector of variables in Rn .

0. Start with any basic feasible partition β, η .

1. Let x̄ and ȳ be the primal and dual solutions associated with β, η .

If c̄η ≥ 0, then STOP: x̄ and ȳ are optimal.
4.2. THE SIMPLEX ALGORITHM WITH NO WORRIES 39

2. Otherwise, choose a non-basic index ηj with c̄ηj < 0 .

3. If Āηj ≤ 0 , then STOP: (P) is unbounded and (D) is infeasible.
4. Otherwise, let  
∗ x̄βi
i := argmin ,
i : āi,ηj >0 āi,ηj
replace β with  
β1 , β2 , . . . , βi∗ −1 , ηj , βi∗ +1 , . . . , βm
and η with  
η1 , η2 , . . . , ηj−1 , βi∗ , ηj+1 , . . . , ηn−m .

5. GOTO 1.

Theorem 4.11
Under the non-degeneracy hypothesis, the Worry-Free Simplex Algorithm terminates
correctly.

Proof. Under the non-degeneracy hypothesis, every time we visit Step 1, we have a pri-
mal feasible solution with a decreased objective value. This implies that we never revisit
a basic feasible partition. But there are only a finite number of basic feasible partitions,
so we must terminate, after a finite number of pivots. But there are only two places
where the algorithm terminates; either in Step 1 where we correctly identify that x̄ and
ȳ are optimal by our sufficient optimality criterion, or in Step 3 because of our sufficient
unboundedness criterion. t
u

Remark 4.12
There are two very significant issues remaining:
• How do we handle degeneracy? (see Section 4.3).
• How do we initialize the algorithm in Step 0? (see Section 4.4).
40 CHAPTER 4. THE SIMPLEX ALGORITHM

4.3 Anticycling

To handle degeneracy, we will eliminate it with an algebraic perturbation. It is

convenient to make the perturbation depend on an m × m non-singular matrix B —
eventually we will choose B in a convenient manner. We replace the problem (P) with

min c0 x
Ax = b (B) ; (P (B))
x ≥ 0 ,

where

• b (B) := b + B~ , and ~ := (, 2 , . . . , m )0 (these are exponents not superscripts);

• the scalar  is an arbitrarily small indeterminant;  is not given a numerical value; it

is simply considered to be a quantity that is positive, yet smaller than any positive
real number;

• 0 denotes a vector in which all entries are the zero polynomial (in );

• the variables xj are polynomials in  with real coefficients;

• the ordering of polynomials used to interpret the inequality ≥ is described next.

The ordering is actually quite simple, but for the sake of precision, we describe it for-
mally.
4.3. ANTICYCLING 41

An ordered ring. The set of polynomials in , with real coefficients, form what is
known in mathematics
P as an “ordered
Pm ring”. The ordering < is simple to describe.
Let p() := m j=0 jp j and q() :=
j=0 jq j . Then p() < q() if the least j for which

pj 6= qj has pj < qj . Another way to think about the ordering < is that p() < q()
if p() < q() when  is considered to be an arbitrarily small positive number. Notice
how the ordering < is in a certain sense a more refined ordering than < . That is, if
p(0) < q(0) , then p() < q() , but we can have p(0) = q(0) without having p() = q() .
Finally, we note that the zero polynomial “0 ”(all coefficients equal to 0) is the zero of
this ordered ring, so we can speak, for example about polynomials that are positive
with respect to the ordering < . Concretely, p() 6= 0 is positive if the least i for which
pi 6= 0 satisfies pi > 0 . Emphasizing that < is a more refined ordering than < , we see
that p() ≥ 0 implies that p(0) = p0 ≥ 0 .
For an arbitrary basis β, the associated basic solution x̄ has x̄β := A−1 β (b + B~) =
−1
x̄β + Aβ B~ . It is evident that x̄βi is a polynomial, of degree at most m, in  , for


each i = 1, . . . , m . Because the ordering < refines the ordering < , we have that
x̄β ≥ 0 implies that x̄β ≥ 0 . That is, any basic feasible partition for (P (B)) is a basic
feasible partition for (P). This implies that applying the Worry-Free Simplex Algorithm
to (P (B)), using the ratio test to enforce feasibility of x̄ in (P (B)) at each iteration,
implies that each associated x̄β is feasible for (P). That is, the choice of a leaving variable
dictated by the ratio test when we work with (P (B)) is valid if we instead do the ratio
test working with (P).
The objective value associated with x̄ is c0β A−1
β (b+B~ ) = ȳ 0 b+ ȳ 0 B~ , is a polynomial
(of degree at most m) in  . Therefore, we can order basic solutions for (P (B)) using
< , and that ordering refines the ordering of the objective values of the corresponding
basic solution of (P). This implies that if x̄ is optimal for (P (B)) , then the x̄ associated
with the same basis is optimal for (P).

Lemma 4.13
The -perturbed problem (P (B)) satisfies the non-degeneracy hypothesis.

Proof. For an arbitrary basis matrix Aβ , the associated basic solution x̄ has x̄β := A−1
β (b+
−1
B~) = x̄β + Aβ B~ . As we have already pointed out, x̄βi is a polynomial, of degree at


most m, in  , for each i = 1, . . . , m . x̄βi = 0 implies that the i-th row of A−1β B is all
−1
zero. But this is impossible for the invertible matrix Aβ B . t
u

Theorem 4.14
Let β 0 be a basis that is feasible for (P). Then the Worry-Free Simplex Algorithm ap-
plied to (P (Aβ 0 )), starting from the basis β 0 , correctly demonstrates that (P) is un-
bounded or finds an optimal basic partition for (P).
42 CHAPTER 4. THE SIMPLEX ALGORITHM

Proof. The first important point to notice is that we are choosing the perturbation of the
original right-hand side to depend on the choice of a basis that is feasible for (P). Then
we observe that x̄β 0 := A−1
β0
(b + Aβ 0~) = A−1
β0
b +~ . Now because x̄ is feasible for (P), we
have A−1
β0
b ≥ 0 . Then, the ordering < implies that x̄β 0 = A−1β0
b + ~ ≥ 0 . Therefore,
the basis β is feasible for (P (Aβ 0 )), and the Worry-Free Simplex Algorithm can indeed
0

be started for (P (Aβ 0 )) on β 0 .

Notice that it is only in Step 4 of the Worry-Free Simplex Algorithm that really de-
pends on whether we are considering (P (Aβ 0 )) or (P). The sufficient optimality cri-
terion and the sufficient unboundedness criterion are identical for (P (Aβ 0 )) and (P).
Because (P (Aβ 0 )) satisfies the non-degeneracy hypothesis, the Worry-Free Simplex Al-
gorithm correctly terminates for (P (Aβ 0 )). t
u

(Pivot.mp4)

Figure 4.2: With some .pdf viewers, you can click above to see or download a short
video. Or just see it on YouTube (probably with an ad) by clicking here.

4.4 Obtaining a Basic Feasible Solution

Next, we will deal with the problem of finding an initial basic feasible solution for the
standard-form problem

min c0 x
Ax = b ; (P)
x ≥ 0.
4.4. OBTAINING A BASIC FEASIBLE SOLUTION 43

4.4.1 Ignoring degeneracy

At first, we ignore the degeneracy issue — why worry about two things at once?! The
idea is rather simple. First, we choose any basic partition β̃, η̃ . If we are lucky, then
A−1
β̃
b≥0.

Otherwise, we have some work to do. We define a new non-negative variable xn+1 ,
which we temporarily adjoin as an additional non-basic variable. So our basic indices
remain as  
β̃ = β̃1 , β̃2 , . . . , β̃m ,
while our non-basic indices are extended to
 
η̃ = η̃1 , η̃2 , . . . , η̃n−m , η̃n−m+1 := n + 1 .

This variable xn+1 is termed an artificial variable. The column for the constraint
matrix associated with xn+1 is defined as An+1 := −Aβ̃ 1 . Hence Ān+1 = −1 . Finally,
we temporarily put aside the objective function from (P) and replace it with one of
minimizing the artificial variable xn+1 . That is, we consider the so-called phase-one
problem
min xn+1
Ax + An+1 xn+1 = b ; (Φ)
x , xn+1 ≥ 0 .
With this terminology, the original problem (P) is referred to as the phase-two prob-
lem.
44 CHAPTER 4. THE SIMPLEX ALGORITHM

It is evident that any feasible solution x̂ of (Φ) with x̂n+1 = 0 is feasible for (P).
Moreover, if the minimum objective value of (Φ) is greater than 0, then we can conclude
that (P) has no feasible solution. So, toward establishing whether or not (P) has a
feasible solution, we focus our attention on (Φ). We will soon see that we can easily
find a basic feasible solution of (Φ).

Finding a basic feasible solution of (Φ). Choose i∗ so that x̄β̃i∗ is most negative. Then
we exchange β̃i∗ with η̃n−m+1 = n + 1 . That is, our new basic indices are
 
β := β̃1 , β̃2 , . . . , β̃i∗ −1 , n + 1, β̃i∗ +1 , . . . , β̃m ,

and our new non-basic indices are

 
η := η̃1 , η̃2 , . . . , η̃n−m , β̃i∗ .

Lemma 4.15
The basic solution of (Φ) associated with the basic partition β, η is feasible for (Φ).

Proof. This pivot, from β̃, η̃ to β, η amounts to moving in the basic direction z̄ ∈ Rn+1
defined by
z̄η̃ := en−m+1 ∈ Rn−m+1 ;
z̄β̃ := −A−1
β̃
An+1 = 1 ∈ Rm ,

in the amount λ := −x̄β̃i∗ > 0 . That is, x̄ + λz̄ is the basic solution associated with the
basic partition β, η . Notice how when we move in the direction z̄ , all basic variables
increase at exactly the same rate that xn+1 does. So, using this direction to increase
xn+1 from 0 to −x̄β̃i∗ > 0 results in all basic variables increasing by exactly −x̄β̃i∗ > 0 .
By the choice of i∗ , this causes all basic variable to become non-negative, and xβ̃i∗ to
become 0 , whereupon it can leave the basis in exchange for xn+1 . t
u

The end game for (Φ). If (P) is feasible, then at the very last iteration of the Worry-
Free Simplex Algorithm on (Φ), the objective value will drop from a positive number to
zero. As this happens, xn+1 will be eligible to leave the basis, but so may other variables
also be eligible. That is, there could be a tie in the ratio-test of Step 4 of the Worry-Free
Simplex Algorithm. As is the case whenever there is a tie, any of the tying indices can
leave the basis — all of the associated variables are becoming zero simultaneously. For
our purposes, it is critical that if there is a tie, we choose i∗ := n + 1 ; that is, xn+1 must
be selected to become non-basic. In this way, we not only get a feasible solution to (P),
we get a basis for it that does not use the artificial variable xn+1 . Now, starting from
this basis, we can smoothly shift to minimizing the objective function of (P).
4.4. OBTAINING A BASIC FEASIBLE SOLUTION 45

4.4.2 Not ignoring degeneracy

Anticycling for (Φ). There is one lingering issue remaining. We have not discussed
anticycling for (Φ).

But this is relatively simple. We define an -perturbed version

min xn+1
Ax + An+1 xn+1 = b (B) ; (Φ )
x , xn+1 ≥ 0 ,

where b (B) := b + ~ , and ~ := (, 2 , . . . , m )0 . Then we choose i so that x̄β̃ is most

i
negative with respect to the ordering < , and exchange i and n + 1 as before. Then, as
in Lemma 4.15, the resulting basis is feasible for (Φ ).

We do need to manage the final iteration a bit carefully. There are two different
ways we can do this.

“Early arrival”. If (P) has a feasible solution, at some point the value of xn+1 will
decrease to a homogeneous polynomial in  . That is, the constant term will become 0.
At this point, although xn+1 may not be eligible to leave the basis for (Φ ), it will be
eligible to leave for (P). So, at this point we let xn+1 leave the basis, and we terminate
the solution process for (Φ ), having found a feasible basis for (P). In fact, we have just
constructively proved the following result.

Theorem 4.16
If standard form (P) has a feasible solution, then it has a basic feasible solution.
46 CHAPTER 4. THE SIMPLEX ALGORITHM

Note that because xn+1 may not have been eligible to leave the basis for (Φ ) when
we apply the “early arrival’ idea, the resulting basis may not be feasible for (P ). So we
will have to re-perturb (P) .

“Be patient”. Perhaps a more elegant way to handle the situation is to fully solve (Φ ).
In doing so, if (P) has a feasible solution, then the minimum objective value of (Φ ) will
be 0 (i.e., the zero polynomial), and xn+1 will necessarily be non-basic. That is because,
at every iteration, every basic variable in (Φ ) is positive. Because xn+1 legally left the
basis for (Φ ) at the final iteration, the resulting basis is feasible for (P ). So we do not
re-perturb (P) , and we simply revert to solving (P ) from the final basis of (Φ ) .

4.5 The Simplex Algorithm

4.6. EXERCISES 47

“This is a very complicated case,

Maude. You know, a lotta ins, a
lotta outs, a lotta what-have-yous.
And, uh, a lotta strands to keep in
my head, man. Lotta strands in
old Duder’s head.” — The Dude

Putting everything together, we get a mathematically complete algorithm for linear

optimization. That is:

1. Apply an algebraic perturbation to the phase-one problem;

2. Solve the phase-one problem using the Worry-Free Simplex Algorithm, adapted
to algebraically perturbed problems, but always giving preference to xn+1 for
leaving the basis whenever it is eligible to leave for the unperturbed problem.
Go to the next step, as soon as xn+1 leaves the basis;

3. Starting from the feasible basis obtained for the original standard-form problem,
apply an algebraic perturbation (Note that the previous step may have left us with
a basis that is feasible for the original unperturbed problem, but infeasible for the
original perturbed problem — this is why we apply a perturbation anew);

4. Solve the problem using the Worry-Free Simplex Algorithm, adapted to alge-
braically perturbed problems.

It is important to know that the Simplex Algorithm will be used, later, to prove the
celebrated Strong Duality Theorem. For that reason, it is important that our algorithm
be mathematically complete. But from a practical computational viewpoint, there is
substantial overhead in working with the -perturbed problems. Therefore, in practice,
no computer code that is routinely applied to large instances worries about the potential
for cycling associated with degeneracy.

4.6 Exercises
Exercise 4.1 (Carry out the Simplex Algorithm)
In Appendix A.2, there are MATLAB scripts and functions which implement the primitive
steps of the simplex algorithm. Using these primitives only, write a script to carry out
the simplex algorithm. Be sure to read in the data using pivot_setup.m. Do not worry
about degeneracy/anti-cycling. But I do want you to take care of algorithmically finding
an initial feasible basis as described in Section 4.4.1. Make some small examples to fully
illustrate the different possibilities for (P) (i.e., infeasible, optimal, unbounded).

Exercise 4.2 (Dual change — first do Exercise 3.4)

Let β, η be any basic partition for the standard-form problem (P). The associated dual
solution is ȳ 0 := c0β A−1
β . Now, suppose that we pivot, letting ηj enter the basis and β`
leave the basis, so that the new partition β̃, η̃ is also a basic partition (in other words,
48 CHAPTER 4. THE SIMPLEX ALGORITHM

Aβ̃ is invertible). Let ȳ˜ be the dual solution associated with the basic partition β̃, η̃ , and
let H`· be row ` of H := A−1 β . Prove that

c̄ηj 0
ȳ˜ = ȳ + H
ā`,ηj `·

HINT: Use the Sherman-Morrison formula; see Section 1.3.

Exercise 4.3 (Traditional phase one)

Instead of organizing the phase-one problem as (Φ), we could first scale rows of Ax = b
as necessary so as to achieve b ≥ 0. Then we can formulate the “traditional phase-one
problem”
Pm
min x
Pm j=1 n+j
Ax + j=1 j xn+j = b ;
e (Φ̂)
x , xn+1 , . . . , xn+m ≥ 0 .
Here we have m artificial variables: xn+1 , xn+2 , . . . , xn+m . It is easy to see that (i)
β := {n + 1, n + 2, . . . , n + m} is feasible for (Φ̂), and (ii) the optimal value of (Φ̂) is
zero if and only if (P) has a feasible solution.
It may be that the optimal value of (Φ̂) is zero, but the optimal basis discovered by
the Simplex Algorithm applied to (Φ̂) contains some of the indices {n+1, n+2, . . . , n+
m} of artificial variables. Describe how we can take such an optimal basis and pass to a
different optimal basis that uses none of {n+ 1, n+ 2, . . . , n+m} (and is thus a feasible
basis for (P)). HINT: Use Theorem 1.3.

Exercise 4.4 (Worry-Free Simplex Algorithm can cycle)

Let θ = 2π/k, with integer k ≥ 5. The idea is to use the symmetry of the geometric
circle, and complete a cycle of the Worry-Free Simplex Algorithm in 2k pivots. Choose
a constant γ satisfying 0 < γ < (sin θ)/(1 − cos θ) . Let
   
1 0
A1 := , A2 := .
0 γ

Let  
cos θ − sin θ
R := .
sin θ cos θ
4.6. EXERCISES 49

Then, for j = 3, 4, . . . , 2k , let


R(j−1)/2 A1 , for odd j ;
Aj :=
R(j−2)/2 A2 , for even j .

We can observe that for odd j , Aj is a rotation of A1 by (j − 1)π/k radians, and for even
j , Aj is a rotation of A2 by (j − 2)π/k radians.
Let cj := 1 − a1j − a2j /γ , for j = 1, 2, . . . , 2k , and let b := (0, 0)0 . Because b = 0 ,
the problem is fully degenerate; that is, x̄ = 0 for all basic solutions x̄ . Notice that
this implies that either the problem has optimal objective value of zero, or the objective
value is unbounded.
For k = 5 , check that you can choose γ = cot θ , and then check that the following
is a sequence of bases β that are legal for the Worry-Free Simplex Algorithm:

β = (1, 2) → (2, 3) → (3, 4) → . . . → (2k − 1, 2k) → (2k, 1) → (1, 2) .

You need to check that for every pivot, the incoming basic variable xηj has negative
reduced cost, and that the outgoing variable is legally selected — that is that āi,ηj > 0 .
Feel free to use MATLAB, MATHEMATICA, etc.
If you are feeling ambitious, check that for all k ≥ 5 , we get a cycle of the Worry-Free
Simplex Algorithm.
Note that it may seem hard to grasp the picture at all4 . But see Section 6.1.3 and
the following Mathematica code. Note that the animation can be activated within the
Acrobat Reader using the controls below the figure.

(* circle.m // Jon Lee *)

(* *)

t =
2 Pi/5; c = Cos[t]; s = Sin[t]; n = 69; e = .75;
m {{c, −s, 0}, {s, c, 0}, {(c − 1)/c, s/c, 1}};
=
x =
{{1}, {0}, {0}};
y =
{{0}, {Cot[t]}, {0}};
T =
Partition[
Flatten[{x, y, m.x, m.y, m.m.x, m.m.y, m.m.m.x, m.m.m.y, m.m.m.m.x,
m.m.m.m.y, x}], {3}, {3}];
ListAnimate[
Table[Graphics3D[
Table[{GrayLevel[e], Polygon[{{0, 0, 0}, T[[i]], T[[i + 1]]}]},
{i,10}], Boxed −> False, PlotRangePadding −> 2.5,
BoxRatios −> {2, 2, 2}, SphericalRegion −> True,
ViewPoint −> {Sin[2 Pi*t/n], Cos[2 Pi*t/n], 0},
Lighting −> False], {t, 0, n}]]
50 CHAPTER 4. THE SIMPLEX ALGORITHM

Figure 4.3: A picture of the cycle with k = 5

Chapter 5

Duality

Our goals in this chapter are as follows:

• Establish the Strong Duality Theorem for the standard-form problem.
• Establish the Complementary Slackness Theorem for the standard-form problem.
• See how duality and complementarity carry over to general linear-optimization
problems.
• Learn about “theorems of the alternative.”
As usual, we focus on the standard-form problem

min c0 x
Ax = b ; (P)
x ≥ 0
and its dual
max y 0 b
(D)
y 0 A ≤ c0 .

51
52 CHAPTER 5. DUALITY

5.1 The Strong Duality Theorem

We have already seen two simple duality theorems:

• Weak Duality Theorem. If x̂ is feasible in (P) and ŷ is feasible in (D), then c0 x̂ ≥

ŷ 0 b .

• Weak Optimal Basis Theorem. If β is a feasible basis and c̄η ≥ 0 , then the primal
solution x̄ and the dual solution ȳ associated with β are optimal.

The Weak Duality Theorem directly implies that if x̂ is feasible in (P) and ŷ is feasible
in (D), and c0 x̂ = ŷ 0 b , then x̂ and ŷ are optimal. Thinking about it this way, we see that
both the Weak Duality Theorem and the Weak Optimal Basis Theorem assert conditions
that are sufficient for establishing optimality.

Theorem 5.1 (Strong Optimal Basis Theorem)

If (P) has a feasible solution, and (P) is not unbounded, then there exists a basis β such
that the associated basic solution x̄ and the associated dual solution ȳ are optimal.
Moreover, c0 x̄ = ȳ 0 b .

Proof. If (P) has a feasible solution and (P) is not unbounded, then the Simplex Algo-
rithm will terminate with a basis β such that the associated basic solution x̄ and the
associated dual solution ȳ are optimal. t
u

As a direct consequence, we have a celebrated theorem.

Theorem 5.2 (Strong Duality Theorem)

If (P) has a feasible solution, and (P) is not unbounded, then there exist feasible solu-
tions x̂ for (P) and ŷ for (D) that are optimal. Moreover, c0 x̂ = ŷ 0 b .

It is important to realize that the Strong Optimal Basis Theorem and the Strong
Duality Theorem depend on the correctness of the Simplex Algorithm — this includes:
(i) the correctness of the phase-one procedure to find an initial feasible basis of (P),
and (ii) the anti-cycling methodology.
5.2. COMPLEMENTARY SLACKNESS 53

5.2 Complementary Slackness

Definition 5.3
With respect to the standard-form problem (P) and its dual (D), the solutions x̂ and ŷ
are complementary if
(cj − ŷ 0 A·j )x̂j = 0 , for j = 1, 2, . . . , n ;
ŷi (Ai· x̂ − bi ) = 0 , for i = 1, 2, . . . , m .

Theorem 5.4
If x̄ is a basic solution (feasible or not) of standard-form (P), and ȳ is the associated
dual solution, then x̄ and ȳ are complementary.

Proof. Notice that if x̄ is a basic solution then Ax̄ = b. Then we can see that comple-
mentarity of x̄ and ȳ amounts to
c̄j x̄j = 0 , for j = 1, 2, . . . , n .
It is clear then that x̄ and ȳ are complementary, because if x̄j > 0 , then j is a basic
index, and c̄j = 0 for basic indices. t
u

Theorem 5.5
If x̂ and ŷ are complementary with respect to (P) and (D), then c0 x̂ = ŷ 0 b .

Proof.
c0 x̂ − ŷ 0 b = (c0 − ŷ 0 A)x̂ + ŷ 0 (Ax̂ − b) ,
which is 0 by complementarity. t
u

Corollary 5.6 (Weak Complementary Slackness Theorem)

If x̂ and ŷ are feasible and complementary with respect to (P) and (D), then x̂ and ŷ
are optimal.
54 CHAPTER 5. DUALITY

Proof. This immediately follows from Theorem 5.5 and the Weak Duality Theorem. t
u

Theorem 5.7 (Strong Complementary Slackness Theorem)

If x̂ and ŷ are optimal for (P) and (D), then x̂ and ŷ are complementary (with respect
to (P) and (D)).

Proof. If x̂ and ŷ are optimal, then by the Strong Duality Theorem, we have c0 x̂− ŷ 0 b = 0 .
Therefore, we have
0 = (c0 − ŷ 0 A)x̂ + ŷ 0 (Ax̂ − b)
Xn m
X
0
= (cj − ŷ A·j )x̂j + ŷi (Ai· x̂ − bi ) .
j=1 i=1

Next, observing that x̂ and ŷ are feasible, we have

n
X m
X
0
(cj − ŷ A·j ) x̂j + ŷi (Ai· x̂ − bi ) .
j=1
| {z } |{z} i=1 | {z }
≥0 ≥0 =0

Clearly this expression is equal to a non-negative number. Finally, we observe that this
expression can only be equal to 0 if
(cj − ŷ 0 A·j )x̂j = 0 , for j = 1, 2, . . . , n .

t
u

5.3 Duality for General Linear-Optimization Problems

Thus far, we have focused on duality for the standard-form problem (P). But we will see
that every linear-optimization problem has a natural dual. Consider the rather general
linear minimization problem
min c0P xP + c0N xN + c0U xU
AGP xP + AGN xN + AGU xU ≥ bG ;
ALP xP + ALN xN + ALU xU ≤ bL ; (G)
AEP xP + AEN xN + AEU xU = bE ;
xP ≥ 0 , xN ≤ 0 .
We will see in the next result that a natural dual for it is
0 b
max yG + yL0 bL + 0 b
yE
G E
0 A
yG + yL0 ALP + 0 A
yE ≤ c0P ;
GP EP
0 A
yG GN + yL0 ALN + 0 A
yE EN ≥ c0N ; (H)
0 A
yG + yL0 ALU + 0 A
yE = c0U ;
GU EU
0 ≥0,
yG yL0 ≤ 0 .
5.3. DUALITY FOR GENERAL LINEAR-OPTIMIZATION PROBLEMS 55

Theorem 5.8
• Weak Duality Theorem: If (x̂P , x̂N , x̂U ) is feasible in (G) and (ŷG , ŷL , ŷE ) is fea-
sible in (H), then c0P x̂P + c0N x̂N + c0U x̂U ≥ ŷG
0 b + ŷ 0 b + ŷ 0 b .
G L L E E

• Strong Duality Theorem: If (G) has a feasible solution, and (G) is not un-
bounded, then there exist feasible solutions (x̂P , x̂N , x̂U ) for (G) and (ŷG , ŷL , ŷE )
for (H) that are optimal. Moreover, c0P x̂P +c0N x̂N +c0U x̂U = ŷG 0 b + ŷ 0 b + ŷ 0 b .
G L L E E

Proof. The Weak Duality Theorem for general problems can be demonstrated as easily
as it was for the standard-form problem and its dual. But the Strong Duality Theorem
for general problems is most easily obtained by converting our general problem (G) to
the standard-form

min c0P xP − c0N x̃N + c0U x̃U − ˜U

c0U x̃
AGP xP − AGN x̃N + AGU x̃U − AGU x̃ ˜U − sG = bG ;
ALP xP − ALN x̃N + ALU x̃U − ALU x̃U ˜ + tL = bL ;
AEP xP − AEN x̃N + AEU x̃U − AEU x̃ ˜U = bE ;
xP ≥ 0 , x̃N ≥ 0 , x̃U ≥ 0 , ˜
x̃U ≥ 0 , sG ≥ 0 , tL ≥ 0 .

˜U for xU . Taking the dual of this standard-

Above, we substituted −x̃N for xN and x̃U − x̃
form problem, we obtain
0 b
max yG + yL0 bL + 0 b
yE
G E
0 A
yG + yL0 ALP + 0 A
yE ≤ c0P ;
GP EP
0 A
− yG − yL0 ALN − 0 A
yE ≤ −c0N ;
GN EN
0 A
yG + yL0 ALU + 0 A
yE ≤ c0U ;
GU EU
0 A
− yG − yL0 ALU − 0 A
yE ≤ −c0U ;
GU EU
0
− yG ≤ 0 ;
+ yL0 ≤ 0 ,

which is clearly equivalent to (H). t

u

Definition 5.9
With respect to (G) and its dual (H), the solutions (x̂P , x̂N , x̂U ) and (ŷG , ŷL , ŷE ) are
complementary if
0


cj − ŷG AGj − ŷL0 ALj − ŷE
0
AEj x̂j = 0 , for all j ;
ŷi (AiP xP + AiN xN + AiU xU − bi ) = 0 , for all i .

Theorem 5.10
• Weak Complementary Slackness Theorem: If (x̂P , x̂N , x̂U ) and (ŷG , ŷL , ŷE ) are
feasible and complementary with respect to (G) and (H), then (x̂P , x̂N , x̂U ) and
(ŷG , ŷL , ŷE ) are optimal.
56 CHAPTER 5. DUALITY

• Strong Complementary Slackness Theorem: If (x̂P , x̂N , x̂U ) and (ŷG , ŷL , ŷE )
are optimal for (G) and (H), (x̂P , x̂N , x̂U ) and (ŷG , ŷL , ŷE ) are complementary
(with respect to (G) and (H)).

Proof. Similarly to the proof for standard-form (P) and its dual (D), we consider the
following expression.

X
0
0 = (cj − ŷG AGj − ŷL0 ALj − ŷE
0
AEj ) x̂j
j∈P
| {z } |{z}
≥0 ≥0
X
0
+ (cj − ŷG AGj − ŷL0 ALj − ŷE
0
AEj ) x̂j
j∈N
| {z } |{z}
≤0 ≤0
X
0
+ (cj − ŷG −
AGj −ŷL0 ALj 0
ŷE AEj ) x̂j
j∈U
| {z }
=0
X
+ ŷi (AiP xP + AiN xN + AiU xU − bi )
|{z} | {z }
i∈G ≥0 ≥0
X
+ ŷi (AiP xP + AiN xN + AiU xU − bi )
|{z} | {z }
i∈L ≤0 ≤0
X
+ ŷi (AiP xP + AiN xN + AiU xU − bi ) .
| {z }
i∈E =0

The results follows easily using the Weak and Strong Duality Theorems for (G) and
(H). t
u

The table below summarizes the duality relationships between the type of each pri-
mal constraint and the type of each associated dual variable. Highlighted in yellow
are the relationships for the standard-form (P) and its dual (D). It is important to note
that the columns are labeled “min” and “max”, rather than primal and dual — the table
is not correct if “min” and “max” are interchanged.

 min max 
 ≥ ≥0 
constraints ≤ ≤0 variables
 
 = unres. 
 ≥0 ≤ 
variables ≤0 ≥ constraints
 
unres. =
5.4. THEOREMS OF THE ALTERNATIVE 57

5.4 Theorems of the Alternative

In this section, we use linear-optimization duality to understand when a linear-

optimization problem has a feasible solution. This fundamental result, expounded by
Farkas5 , opened the door for studying linear inequalities and optimization.

Theorem 5.11 (Farkas Lemma)

Let A ∈ Rm×n and b ∈ Rm be given. Then exactly one of the following two systems
has a solution.
Ax = b ;
(I)
x ≥ 0.
y0b > 0 ;
(II)
y 0 A ≤ 00 .

Proof. It is easy to see that there cannot simultaneously be a solution x̂ to (I) and ŷ to
(II). Otherwise we would have

0 ≥ ŷ 0 A x̂ = ŷ 0 b > 0 ,
|{z} |{z}
≤0 ≥0

which is a clear inconsistency.

Next, suppose that (I) has no solution. Then the following problem is infeasible:

min 00 x
Ax = b ; (P)
x ≥ 0.
Its dual is
max y 0 b
(D)
y 0 A ≤ 00 .
Because (P) is infeasible, then (D) is either infeasible or unbounded. But ŷ := 0 is a
feasible solution to (D), therefore (D) must be unbounded. Therefore, there exists a
feasible solution ŷ to (D) having objective value greater than zero (or even any fixed
constant). Such a ŷ is a solution to (II). t
u
Remark 5.12
Geometrically, the Farkas Lemma asserts that exactly one of the following holds:
58 CHAPTER 5. DUALITY

(I) b is in the “cone generated by the columns of A” (i.e., b is a non-negative linear

combination of the columns of A), or
(II) there is ŷ ∈ Rm that makes an acute angle with b and a non-acute (i.e., right or
obtuse) angle with every column of A .
In the case of (II), considering the hyperplane H containing the origin having ŷ as
its normal vector, this H separates b from the cone generated by the columns of A .
So, the Farkas Lemma has the geometric interpretation as a“Separating-Hyperplane
Theorem.” See Figure 5.1 for an example with m = 2 and n = 4 . The cone is red and
the point b that we separate from the cone is blue. The green point is a solution ŷ for
(II), and the dashed green line is the separating hyperplane. Notice how the (solid)
green vector makes an acute angle with the blue vector and a non-acute angle with all
points in the cone.

Figure 5.1: Case (II) of the Farkas Lemma

In a similar fashion to the Farkas Lemma, we can develop theorems of this type for
feasible regions of other linear-optimization problems.
5.5. EXERCISES 59

Theorem 5.13 (Theorem of the Alternative for Linear Inequalities)

Let A ∈ Rm×n and b ∈ Rm be given. Then exactly one of the following two systems
has a solution.
Ax ≥ b . (I)
y0b > 0 ;
y 0 A = 00 ; (II)
y ≥ 0.

Proof. It is easy to see that there cannot simultaneously be a solution x̂ to (I) and ŷ to
(II). Otherwise we would have

0 = ŷ 0 A x̂ ≥ ŷ 0 b > 0 ,
|{z}
=0

which is a clear inconsistency.

Next, suppose that (I) has no solution. Then the following problem is infeasible:

min 00 x
(P)
Ax ≥ b .

Its dual is
max y 0 b
y 0 A = 00 ; (D)
y ≥ 0.

Because (P) is infeasible, then (D) is either infeasible or unbounded. But ŷ := 0 is

a feasible solution to (D), therefore (D) must be unbounded. Therefore, there exists a
feasible solution ŷ to (D) having objective value greater than zero (or even greater than
any fixed constant). Such a ŷ is a solution to (II). t
u

5.5 Exercises
Exercise 5.1 (Dual picture)
For the standard-form problem (P) and its dual (D), explain aspects of duality and
complementarity using this picture:
60 CHAPTER 5. DUALITY

Exercise 5.2 (Reduced costs as dual values)

In this exercise, we will see that we can regard reduced costs (corresponding to an opti-
mal basic partition) as (optimal) values of dual variables for non-negativity constraints.
Consider the ordinary standard-form problem

z := min c0 x dual variables

Ax = b ; y (P)
x ≥ 0,

and let β, η be an optimal basic partition for (P).

We can equivalently see (P) as

z := min c0 x dual variables

Ax = b ; y (P̃)
x ≥ 0, w

where in (P̃), we regard the non-negativity constraints of (P) as ordinary structural

constraints — with dual variables.
Define w̄ ∈ Rn by
w̄β := 0 ∈ Rm ;
w̄η := c̄η ∈ Rn−m .
Prove that together, ȳ := c0β A−1
β and w̄ are optimal for the dual of (P̃).

Exercise 5.3 (Duality and complementarity with AMPL)

After optimization using AMPL, it is easy to get more information regarding primal and
dual problems. If the constraints are Constraints , then the AMPL command ‘display
5.5. EXERCISES 61

Constraints.dual’ displays the associated dual variables and ‘display Constraints.slack’

displays the associated slackness. Using these commands, develop an example to verify
the concepts of duality and complementarity developed in this chapter.
Exercise 5.4 (Complementary slackness)
Construct an example where we are given x̂ and ŷ and asked to check whether x̂ is
optimal using complementary slackness. I want your example to have the property
that x̂ is optimal, x̂ and ŷ are complementary, but ŷ is not feasible.
The idea is to see an example where there is not a unique dual solution complemen-
tary to x̂ , and so x̂ is optimal, but we only verify it with another choice of ŷ.
Exercise 5.5 (Over complementarity)
With respect to the standard-form problem (P) and its dual (D), complementary solu-
tions x̂ and ŷ are overly complementary if exactly one of

cj − ŷ 0 A·j and x̂j is 0 , for j = 1, 2, . . . , n .

Prove that if (P) has an optimal solution, then there are always optimal solutions for
(P) and (D) that are overly complementary.
HINT: Let v be the optimal objective value of (P). For each j = 1, 2, . . . , n , consider

max xj
c0 x ≤ v
(Pj )
Ax = b .
x ≥ 0.
(Pj ) seeks an optimal solution of (P) that has xj positive. Using the dual of (Pj ), show
that if no optimal solution x̂ of (P) has x̂j positive, then there is an optimal solution ŷ
of (D) with cj − ŷ 0 A·j positive. Once you do this you can conclude that, for any fixed
j, there are optimal solutions x̂ and ŷ with the property that exactly one of

cj − ŷ 0 A·j and x̂j is 0 .

Take all of these n pairs of solutions x̂ and ŷ and combine them appropriately to con-
struct optimal x̂ and ŷ that are overly complementary.

Exercise 5.6 (Another proof of a Theorem of the Alternative)

Prove the Theorem of the Alternative for Linear Inequalities directly from the Farkas
Lemma, without appealing to linear-optimization duality. HINT: Transform (I) of the
Theorem of the Alternative for Linear Inequalities to a system of the form of (I) of the
Farkas Lemma.
62 CHAPTER 5. DUALITY

Exercise 5.7 (A general Theorem of the Alternative)

State and prove a “Theorem of the Alternative” for the system:

AG
P xP + AG G
N xN + AU xU ≥ bG ;
L
AP x P + L
AN xN + AL ≤ bL ;
U xU (I)
E
AP x P + AE x + A Ex = bE ;
N N U U
xP ≥ 0 , xN ≤ 0 .

Exercise 5.8 (Dual ray)

Consider the linear-optimization problem

min c0 x
Ax ≥ b ; (P)
x ≥ 0.

a) Suppose that (P) is infeasible. Then, by a ‘Theorem of the Alternative’ there is a

solution to what system?

b) Suppose, further, that the dual (D) of (P) is feasible. Take a feasible solution ŷ of
(D) and a solution ỹ to your system of part (a) and combine them appropriately
to prove that (D) is unbounded.
Chapter 6

Sensitivity Analysis

Our goal in this chapter is as follows:

• Learn how the optimal value of a linear-optimization problem behaves when the
right-hand side vector and objective vector are varied.

6.1 Right-Hand Side Changes

We define a function f : Rm → R via

f (b) := min c0 x
Ax = b ; (Pb )
x ≥ 0.

That is, (Pb ) is simply (P) with the optimal objective value viewed as a function of its
right-hand side vector b .

63
64 CHAPTER 6. SENSITIVITY ANALYSIS

6.1.1 Local analysis

Consider a fixed basis β for (Pb ). Associated with that basis is the basic solution
x̄β = A−1 0 −1
β b and the corresponding dual solution ȳ = cβ Aβ . Let us assume that ȳ is
0

feasible for the dual of (Pb ) — or, equivalently, c0η − ȳ 0 Aη ≥ 00 . Considering the set B
of b ∈ Rm such that β is an optimal basis, is it easy to see that B is just the set of b such
that x̄β := A−1 β b ≥ 0 . That is, B ⊂ R is the solution set of m linear inequalities (in
m

fact, it is a “simplicial cone” — we will return to this point in Section 6.1.3). Now, for
b ∈ B , we have f (b) = ȳ 0 b . Therefore, f is a linear function on b ∈ B . Moreover, as
∂f
long as b is in the interior of B , we have ∂b i
= ȳi . So we have that ȳ is the gradient of
f , as long as b is in the interior of B . Now what does it mean for b to be in the interior
of B ? It just means that x̄βi > 0 for i = 1, 2, . . . , m .
Let us focus our attention on changes to a single right-hand side element bi . Sup-
pose that β is an optimal basis of (P) , and consider the problem

min c0 x
Ax = b + ∆i ei ; (Pi )
x ≥ 0,

where ∆i ∈ R . The basis β is feasible (and hence still optimal) for (Pi ) if A−1
β (b +
−1
∆i ei ) ≥ 0 . Let h := Aβ ei . So
i

[h1 , h2 , . . . , hm ] = A−1
β .

Then, the condition A−1 β (b + ∆i ei ) ≥ 0 can be re-expressed as x̄β + ∆i h ≥ 0 . It is

i

straightforward to check that β is feasible (and hence still optimal) for (Pi ) as long as
∆i is in the interval [Li , Ui ] , where

Li := max −x̄βk /hik ,
k : hik >0

and

Ui := min −x̄βk /hik .
k : hik <0

It is worth noting that it can be the case that hik ≤ 0 for all k, in which case we define
Li := −∞, and it could be the case that hik ≥ 0 for all k, in which case we define
Ui := +∞,
In summary, for all ∆i satisfying Li ≤ ∆i ≤ Ui , β is an optimal basis of (P) .
It is important to emphasize that this result pertains to changing one right-hand side
element and holding all others constant. For a result on simultaneously changing all
right-hand side elements, we refer to Exercise 6.3.
6.1. RIGHT-HAND SIDE CHANGES 65

6.1.2 Global analysis

The domain of f is the set of b for which (Pb ) has an optimal solution. Assuming
that the dual of (Pb ) is feasible (note that this just means that y 0 A ≤ c0 has a solution),
then (Pb ) is never unbounded. So the domain of f is just the set of b ∈ Rm such that
(Pb ) is feasible.

Theorem 6.1
The domain of f is a convex set.

Proof. Suppose that bj is in the domain of f , for j = 1, 2 . Therefore, there exist xj that
are feasible for (Pbj ) , for j = 1, 2 . For any 0 < λ < 1 , let b̂ := λb1 + (1 − λ)b2 , and
consider x̂ := λx1 + (1 − λ)x2 . It is easy to check that x̂ is feasible for (Pb̂ ) , so we can
conclude that b̂ is in the domain of f . t
u

Before going further, we need a few definitions. We consider functions f : Rm → R.

The domain of f is the subset S of Rm on which f is defined. We assume that S is a
convex set. A function f : Rm → R is a convex function on its domain S, if

f (λu1 + (1 − λ)u2 ) ≤ λf (u1 ) + (1 − λ)f (u2 ) ,

for all u1 , u2 ∈ S and 0 < λ < 1 . That is, f is never underestimated by linear interpo-
lation.
AP function f : Rm → R is an affine function, if it has the form f (u1 , . . . , um ) =
a0 + m i=1 ai ui , for constants a0 , a1 , . . . , am ∈ R . If a0 = 0 , then we say that f is a
linear function. Affine (and hence linear) functions are easily seen to be convex.
A function f : Rm → R having a convex set as its domain is a convex piecewise-
linear function if, on its domain, it is the pointwise maximum of a finite number of affine
functions.
66 CHAPTER 6. SENSITIVITY ANALYSIS

It would be strange to refer to a function as being “convex piecewise-linear” if it were

not convex! The next result justifies the moniker.

Theorem 6.2
If fˇ is a convex piecewise-linear function, then it is a convex function.

Proof. Let
fˇ(u) := max {fi (u)} ,
1≤i≤k

for u in the domain of fˇ , where each fi is an affine function. That is, fˇ is the pointwise
maximum of a finite number (k) of affine functions.
Then, for 0 < λ < 1 and u1 , u2 ∈ Rm ,

fˇ(λu1 + (1 − λ)u2 ) = max fi (λu1 + (1 − λ)u2 )
1≤i≤k

= max λfi (u1 ) + (1 − λ)fi (u2 ) (using the definition of affine)
1≤i≤k
 
≤ max λfi (u1 ) + max (1 − λ)fi (u2 )
1≤i≤k 1≤i≤k
 
= λ max fi (u ) + (1 − λ) max fi (u2 )
1
1≤i≤k 1≤i≤k

= λf (u ) + (1 − λ)fˇ(u ) .
ˇ 1 2

t
u

Theorem 6.3
f is a convex piecewise-linear function on its domain.

Proof. We refer to the dual

f (b) := max y 0 b
(Db )
y 0 A ≤ c0 ;

of (Pb ).
6.1. RIGHT-HAND SIDE CHANGES 67

A basis β is feasible or not for (Db ), independent of b . Thinking about it this way,
we can see that
n  o
f (b) = max c0β A−1 β b : β is a dual feasible basis ,

and so f is a convex piecewise-linear function, because it is the pointwise maximum of

a finite number of affine (even linear) functions. t
u

6.1.3 A brief detour: the column geometry for the Simplex Algorithm

In this section, we will describe a geometry for visualizing the Simplex Algorithm.6
The ordinary geometry for a standard-form problem, in the space of the non-basic vari-
ables for same choice of basis, can be visualized when n − m = 2 or 3. The “column
geometry” that we will describe is in Rm+1 , so it can be visualized when m + 1 = 2 or
3. Note that the graph of the function f (b) (introduced at the start of this chapter) is
also in Rm+1 , which is why we take the present detour.
We think of the n points
 
cj
,
Aj
for j = 1, 2, . . . , n , and the additional so-called requirement line
  
z
: z∈R .
b

We think of the first component of these points and of the line as the vertical dimension;
so the requirement line is thought of as vertical. It of particular interest to think of the
cone generated by the n points. That is,
( ! )
c0 x
K := ∈ Rm+1 : x ≥ 0 .
Ax

Notice how the top coordinate of a point in the cone gives the objective value of the
associated x for (P). So the goal of solving (P) can be thought of as that of finding a
point on the intersection of the requirement line and the cone that is as low as possible.
68 CHAPTER 6. SENSITIVITY ANALYSIS

Restricting ourselves to a basis β , we have the cone

( ! )
c0β xβ
Kβ := ∈R m+1
: xβ ≥ 0 .
Aβ x β
The cone Kβ is an “m-dimensional simplicial cone.” Next, we observe that if β is a
feasible basis, then Kβ intersects the requirement line uniquely at the point
!
c0β x̄β
,
Aβ x̄β
where x̄ is the basic solution associated with β .
In a pivot of the Simplex Algorithm from basis β to basis β̃, we do so with the goal
of having Kβ̃ intersect the requirement line at a lower point than did Kβ . In Figure
6.1 (m = 2 and the coordinate axes are the red lines), we see an example depicting a
single pivot. Kβ is the yellow cone, intersecting the blue requirement line at the red
point. After the pivot (with one cone generator exchanged), we have the green cone
Kβ̃ intersecting the requirement line at the pink point.

Figure 6.1: A simplex pivot

So at each iteration of the Simplex Algorithm, we exchange a single “generator” of the

simplicial cone Kβ associated with our basis β, to descend along the requirement line,
ultimately finding a point of K that meets the requirement at its lowest point.

6.2 Objective Changes

“Here is what is needed for Occupy Wall Street to become a force for change: a clear,
and clearly expressed, objective. Or two.” — Elayne Boosler
6.2. OBJECTIVE CHANGES 69

We define a function g : Rn → R via

g(c) := min c0 x
Ax = b ; (Pc )
x ≥ 0.

That is, (Pc ) is simply (P) with the optimal objective value viewed as a function of its
objective vector c .

6.2.1 Local analysis

Consider a fixed basis β for (Pc ). Associated with that basis is the basic solution x̄β =
A−1 0 −1
β b and the corresponding dual solution ȳ = cβ Aβ . Let us assume that x̄ is feasible
0

for (Pc ) — or, equivalently, A−1β b ≥ 0 . Considering the set C of c ∈ R such that β is an
n

optimal basis, is it easy to see that this is just the set of c such that cη − c0β A−1
0
β Aη ≥ 0 .
0

That is, C ⊂ Rn is the solution set of n − m linear inequalities (in fact, it is a cone). Now,
for c ∈ C , we have g(c) = c0β x̄β . Therefore, g is a linear function on c ∈ C .

6.2.2 Global analysis

The domain of g is the set of c for which (Pc ) has an optimal solution. Assuming that
(Pc ) is feasible, then the domain of g is just the set of c ∈ Rn such that (Pc ) is not
unbounded.
Similarly to the case of variations in the right-hand side vector b, we have the fol-
lowing two results.

Theorem 6.4
The domain of g is a convex set.

A function g : Rn → R is a concave function on its domain S, if

g(λu1 + (1 − λ)u2 ) ≥ λg(u1 ) + (1 − λ)g(u2 ) ,

for all u1 , u2 ∈ S and 0 < λ < 1 . That is, f is never overestimated by linear inter-
polation. The function g is a concave piecewise-linear function if it is the pointwise
minimum of a finite number of affine functions.
70 CHAPTER 6. SENSITIVITY ANALYSIS

Theorem 6.5
g is a concave piecewise-linear function on its domain.

6.2.3 Local sensitivity analysis with a modeling language

To carry out some local sensitivity analysis with AMPL, use the following options

option solver cplex;

option cplex_options 'sensitivity';

Then use the AMPL command solve to solve the problem (in particular, to determine an
optimal basis — though AMPL will not tell you the optimal basis that it found).
AMPL has suffixes to retrieve some local sensitivity information from the solver. For a
constraint named <Constraint>, <Constraint>.current gives the current right-hand
side value, while <Constraint>.down and <Constraint>.up give lower and upper lim-
its on that right-hand side value, such that the optimal basis remains optimal. Changing
a particular right-hand side value, in this way, assumes that the right-hand side values
for the other constraints are held constant (though see Exercise 6.3 for the case of si-
multaneous
P changes). In our notation from Section 6.1.1, if <Constraint> corresponds
to nj=1 aij xj = bi (that is, the i-th constraint of (Pb ), then

bi = <Constraint>.current ,
Li = <Constraint>.down − <Constraint>.current ,
Ui = <Constraint>.up − <Constraint>.current .

In a similar manner, for a variable named <Variable>, we have that <Variable>.current,

<Variable>.down and <Variable>.up give the current value, and lower and upper lim-
its on the coefficient of <Variable> in the objective function. Additionally, <Variable>.lrc
give the reduced cost of <Variable>. As we learned in Section ??, this reduced cost of
<Variable> is the value of the optimal dual variable for the non-negativity constraint
on <Variable>. If <Variable> also has an explicit upper bound, then <Variable>.urc
is the value of the dual variable for the upper-bound constraint on <Variable>.

6.3 Exercises
Exercise 6.1 (Illustrate local sensitivity analysis)
Make an original example to illustrate the local sensitivity-analysis concepts of this
chapter. Use a combination of hand calculations and AMPL.

Exercise 6.2 (Illustrate global sensitivity analysis)

Using AMPL, make an original example, with at least three constraints, graphing the
objective value of (P ) , as a single b[i] is varied from −∞ to +∞ . As you work on this,
bear in mind Theorem 6.3.
6.3. EXERCISES 71

Exercise 6.3 (“I feel that I know the change that is needed.” — Mahatma Gandhi)
We are given 2m numbers satisfying Li ≤ 0 ≤ Ui , i = 1, 2, . . . , m . Let β be an optimal
basis for all of the m problems

min c0 x
Ax = b + ∆i ei ; (Pi )
x ≥ 0,

for all ∆i satisfying Li ≤ ∆i ≤ Ui . Let’s be clear on what this means: For each i
individually, the basis β is optimal when the ith right-hand side component is changed
from bi to bi + ∆i , as long as ∆i is in the interval [Li , Ui ] (see Section 6.1.1).
The point of this problem is to be able to say something about simultaneously chang-
ing all of the bi . Prove that we can simultaneously change bi to
 
Li
b̃i := bi + λi ,
Ui
P
where λi ≥ 0 , when m i=1 λi ≤ 1 . [Note that in the formula above, for each i we can
pick either Li (a decrease) or Ui (an increase)].

Exercise 6.4 (Domain for objective variations)

Prove Theorem 6.4.

Exercise 6.5 (Concave piecewise-linear function)

Prove Theorem 6.5.
Chapter 7

Large-Scale Linear Optimization

Our goals in this chapter are as follows:

• To see some approaches to large-scale linear-optimization problems

• In particular, to learn about decomposition, Lagrangian relaxation and column

generation.

• Also, via a study of the “cutting-stock problem,” we will have a first glimpse at
some issues associated with integer-linear optimization.

7.1 Decomposition

73
74 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

In this section we describe what is usually known as Dantzig-Wolfe Decomposi-

tion. It is an algorithm aimed at efficiently solving certain kinds of structured linear-
optimization problems. The general viewpoint is that we might have a very efficient
way to solve a certain type of structured linear-optimization problem, if it were not for
a small number of constraints that break the structure. For example, the constraint ma-
trix might have the form in Figure 7.1, where if it were not for the top constraints, the
optimization problem would separate into many small problems7 .
 
···
 
 
 
 
 . . 
 . 

Figure 7.1: Nearly separates

7.1.1 The master reformulation

Theorem 7.1 (The Representation Theorem)

Let
min c0 x
Ax = b ; (P)
x ≥ 0.
Suppose that (P) has a non-empty feasible region. Let X := {x̂j : j ∈ J } be the set
of basic-feasible solutions of (P), and let Z := {ẑ k : k ∈ K} be the set of basic-feasible
rays of (P). Then the feasible region of (P) is equal to
 
X X X 
λj x̂j + µk ẑ k : λj = 1 ; λj ≥ 0 , j ∈ J ; µk ≥ 0 , k ∈ K .
 
j∈J k∈K j∈J

Proof. Let S be the feasible region of (P) . Let

 
X X X 
S0 = λj x̂j + µk ẑ k : λj = 1 ; λj ≥ 0 , j ∈ J ; µk ≥ 0 , k ∈ K .
 
j∈J k∈K j∈J

We will demonstrate that S = S 0 . It is very easy to check that S 0 ⊂ S , and we leave

that to the reader. For the other direction, suppose that x̂ ∈ S , and consider the system
X X
λj x̂j + µk ẑ k = x̂ ;
j∈J
X k∈K
λj = 1; (I)
j∈J
λj ≥ 0 , j ∈ J ; µk ≥ 0 , k ∈ K .
7.1. DECOMPOSITION 75

Keep in mind that in (I), x̂ is fixed as well as are the x̂j and the ẑ k — the variables are
the λj and the µk . By way of establishing that S ⊂ S 0 , suppose that x̂ ∈ / S 0 — that is,
suppose that (I) has no solution. Applying the Farkas Lemma to (I) , we see that the
system
w0 x̂ + t > 0 ;
w0 x̂j + t ≤ 0 , ∀ j ∈ J ; (II)
0
w ẑ k ≤ 0, ∀k∈K
has a solution, say ŵ, t̂ . Now, consider the linear-optimization problem

min −ŵ0 x
Ax = b ; (P̂)
x ≥ 0.

(P̂) cannot be unbounded, because −ŵ0 ẑ k ≥ 0 , for all k ∈ K . In addition, every

basic feasible solution of (P̂) has objective value at least t̂ . By Theorem 5.1 (the Strong
Optimal Basis Theorem), this implies that the optimal value of (P̂) is at least t̂ . But
the objective value −ŵ0 x̂ of x̂ is less than t̂ . Therefore, x̂ cannot be feasible. That is,
/S.
x̂ ∈ t
u

Corollary 7.2 (The Decomposition Theorem)

Let
min c0 x
Ex = h ;
(Q)
Ax = b ;
x ≥ 0.
Let S := {x ∈ Rn : Ax = b , x ≥ 0} , let X := {x̂j : j ∈ J } be the set of basic-feasible
solutions S, and let Z := {ẑ k : k ∈ K} be the set of basic-feasible rays of S. Then (Q)
is equivalent to the Master Problem
X
X 
min c0 x̂j λj + c0 ẑ k µk
j∈J k∈K
X
X 
j
E x̂ λj + E ẑ k µk = h;
j∈J k∈K
(M)
X
λj = 1;
j∈J
λj ≥ 0 , j ∈ J , µk ≥ 0 , k ∈ K .

Proof. Using the Representation Theorem, we just substitute the expression

X X
λj x̂j + µk ẑ k
j∈J k∈K

for x in c0 x and in Ex = h of (Q), and it is easy to see that (M) is equivalent to (Q). t
u
76 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

Decomposition is typically applied in a way such that the constraints defining (S)
are somehow relatively “nice,” and the constraints Ex = h somehow are “complicat-
ing” the situation. For example, we may have a problem where the overall constraint
matrix has the form depicted in Figure 7.1. In such a scenario, we would let


E := ···

and  
 
 
A :=  .. 
 . 

7.1.2 Solution of the Master via the Simplex Algorithm

Next, we describe how to solve (M) using the Simplex Algorithm. Our viewpoint is
that we cannot write out (M) explicitly; there are typically far too many variables. But
we can reasonably maintain a basic solution of (M) , because the number of constraints
is just one more than the number of constraints in Ex = h .
The only part of the Simplex Algorithm that is sensitive to the total number of vari-
ables is the step in which we check whether there is a variable with a negative reduced
cost. So rather than checking this directly, we will find an indirect way to carry it out.
Toward this end, we define dual variables y and σ for (M) .
X
X 
min c0 x̂j λj + c0 ẑ k µk dual variables
j∈J k∈K
X
X 
E x̂j λj + E ẑ k µk = h; y
(M)
j∈J
X k∈K
λj = 1; σ
j∈J
λj ≥ 0 , j ∈ J , µk ≥ 0 , k ∈ K .

While σ is a scalar variable, y is a vector with a component for each row of E .

We will temporarily put aside how we calculate values for y and σ , but for now we
suppose that we have a basic solution of (M) and an associated dual solution ȳ and σ̄ .

Entering variable. With respect to such a dual solution, the reduced cost of a variable
λj is



c0 x̂j − ȳ 0 E x̂j − σ̄ = −σ̄ + c0 − ȳ 0 E x̂j .
It is noteworthy that with the dual solution fixed (at ȳ and σ̄), the reduced cost of λj is
a constant (−σ̄) plus a linear function of x̂j . A variable λj is eligible to enter the basis
if its reduced cost is negative. So we formulate the following optimization problem:

−σ̄ + min (c0 − ȳ 0 E) x

Ax = b ; (SUB)
x ≥ 0.
7.1. DECOMPOSITION 77

If the “subproblem” (SUB) has as optimal solution, then it has a basic optimal solution
— that is, an x̂j . In such a case, if the optimal objective value of (SUB) is negative, then
the λj corresponding to the optimal x̂j is eligible to enter the current basis of (M) . On
the other hand, if the optimal objective value of (SUB) is non-negative, then we have a
proof that no non-basic λj is eligible to enter the current basis of (M) .
If (SUB) is unbounded, then (SUB) has a basic feasible ray ẑ k having negative ob-
jective
value. That

is, 0(c −0 ȳ E)k ẑ < 0 . Amazingly, the reduced cost of µk is precisely
0 0 k

c ẑ − ȳ E ẑ = (c − ȳ E) ẑ , so, in fact, µk is then eligible to enter the current basis

0 k 0 k

of (M) .

Leaving variable. To determine the choice of leaving variable, let us suppose that B
is the basis matrix for (M) . Note that B consists of at least one column of the form
 
E x̂j
1

and columns of the form  

E ẑ k
.
0
With respect to the current basis, to carry out the ratio test of the Simplex Algorithm,
we simply need
 
−1 h
B
1
and either  
−1 E x̂j
B
1
if λj is entering the basis, or
 
−1 E ẑ k
B
0
if µk is entering the basis.

Calculation of basic primal and dual solutions. It is helpful to explain a bit about the
calculation of basic primal and dual solutions. As we have said, B consists of at least
one column of the form  
E x̂j
1
and columns of the form  
E ẑ k
.
0
So organizing the basic variables λj and µk into a vector ζ , with their order appropri-
ately matched with the columns of B , the vector ζ̄ of values of ζ is precisely the solution
of  
h
Bζ = .
1
78 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

That is,  
−1 h
ζ̄ = B .
1
Finally, organizing the costs c0 x̂j and c0 ẑ k of the basic variables λj and µk into a vector
ξ , with their order appropriately matched with the columns of B , the associated dual
solution (ȳ, σ̄) is precisely the solution of
(y 0 , σ)B = ξ 0 .
That is,
(ȳ 0 , σ̄) = ξ 0 B −1 .

Starting basis. It is not obvious how to construct a feasible starting basis for (M) ;
after all, we may not have at hand any basic feasible solutions and rays of (S) . Next, we
give a simple recipe. We assume that the problem is first in a slightly different form,
where the complicating constraints are inequalities:
min c̃0 x̃
Ẽ x̃ ≤ h ;
(Q̃)
Ãx̃ = b ;
x̃ ≥ 0 ,
where x̃ is a vector of n − m variables, Ẽ ∈ Rm×(n−m) , Ã ∈ Rp×(n−m) , and all of the
other data has dimensions that conform.
Introducing m slack variables for the Ẽ x̃ ≤ h constraints, we have the equivalent
problem
min c̃0 x̃
Ẽ x̃ + s = h ;
Ãx̃ = b;
x̃ , s ≥ 0 .
Now, it is convenient to define x ∈ Rn by

x̃j , for j = 1, . . . , n − m ;
xj :=
sj−(n−m) , for j = n − m + 1, . . . , n ,
and to define E ∈ Rm×n by

Ẽj , for j = 1, . . . , n − m ;
Ej :=
ej−(n−m) , for j = n − m + 1, . . . , n ,
and A ∈ Rp×n by 
Ãj , for j = 1, . . . , n − m ;
Aj :=
0 , for j = n − m + 1, . . . , n .
 
That is, x = x̃s , E = [Ẽ, Im ] , A = [Ã, 0] , so now our system has the familiar
equivalent form
min c0 x
Ex = h;
(Q)
Ax = b;
x ≥ 0.
7.1. DECOMPOSITION 79

Now, we have everything organized to specify how to get an initial basis for (M).
First, we take as x̂1 any basic feasible solution of (P) . Such a solution can be readily ob-
tained by using our usual (phase-one) methodology of the Simplex Algorithm. Next,
we observe that for k = 1, 2 . . . , m , ẑ k := en−(m+k) is a basic feasible ray of S . The
ray ẑ k corresponds to increasing the slack variable sk (arbitrarily); because, the slack
variable sk = xn−m+k does not truly enter into the Ax = b constraints (i.e., the coeffi-
cients of sk is zero in those constraints), we do not have to alter the values of any other
variables when sk is increased.
So, we have as an initial set of basic variables µ1 , . . . , µm (corresponding to ẑ 1 , . . . , ẑ m )
and λ1 (corresponding to x1 ). Notice that E ẑ k = ek , for k = 1, 2 . . . , m . Organizing
our basic variables in the order µ1 , µ2 , . . . , µm , λ1 , we have the initial basis matrix
 
Im E x̂1
B= .
00 1

It is very easy to see that this is an invertible matrix.

Finally, it is very important to realize that we have given a recipe for finding an initial
basic solution of (M) . [It happens that this basic solution is feasible precisely when x1
satisfies the Ex = h constraints]. But this solution may not be feasible. If the associated
basic solution of (M) is not feasible (i.e., has any negative components), then we would
introduce an artificial variable and do a phase-one procedure.

A demonstration implementation.

It is not completely trivial to write a small MATLAB code for the Decomposition Algo-
rithm. First of all, we solve the subproblems (SUB) using functionality of MATLAB. With
this, if a linear-optimization problem is unbounded, MATLAB does not give us access to
the basic-feasible ray on which the objective is decreasing. Because of this, our MATLAB
code quits if it encounters an unbounded subproblem.
Another point is that rather than carry out the simplex method at a detailed level
on the Master Problem (M), we just accumulate all columns of (M) that we generate,
and always solve linear-optimization problems, using functionality of MATLAB, with all
of the columns generated thus far. In this way, we do not maintain bases ourselves,
and we do not carry out the detailed pivots of the Simplex Algorithm. Note that the
linear-optimization functionality of MATLAB does give us a dual solution, so we do not
compute that ourselves.
80 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

% Decomp.m // Jon Lee

%
% Apply Decomposition to
% z = min c'x
% Ex + Is = h
% Ax = b
% x , s >= 0
%
% NOTE: WE ONLY HANDLE PROBLEMS WITH BOUNDED SUBPROBLEM LPs.
% IF WE ENCOUNTER AN UNBOUNDED SUBPROBLEM, THEN WE QUIT.

options = optimoptions('linprog','Algorithm','dual−simplex');

% generate a random example

n = 50 % number of x variables
m1 = 15 % number of 'complicating' equations = number of s variables
m2 = 10 % number of 'nice' equations

rng('default');
rng(1); % set seed
E = rand(m1,n); % random m1−by−n matrix
A = rand(m2,n); % random m2−by−n matrix

w = rand(n,1); % generate rhs's so problem is feasible

h = E*w + 0.01*rand(m1,1);
b = A*w;

c = rand(n,1); % generate random objective

% first we will calculate the optimal value z of the original LP, just to
% compare later.
disp('Optimizing the original LP without decomposition');
[x,z,exitflag,output,lambda]=linprog([c;zeros(m1,1)],[],[], ...
[horzcat(E,eye(m1)); horzcat(A,zeros(m2,m1))],[h; b], ...
zeros(n+m1,1),[],[],options);
if (exitflag < 1)
disp('fail 1: original LP without decomposition did not solve correctly');
return;
end;
disp('Optimal value of the original LP:');
z
%disp('dual solution:');
%disp(− lambda.eqlin); % Need to flip the signs due to a Matlab convention

disp('Starting Decomposition Algorithm');

MAXIT = 50 % number of iterations

% Set up the initial master basis

% ... First, we need any extreme point of the subproblem
x1 = linprog(c',[],[],A,b,zeros(n,1),[],options);
X = x1; % start accumulating the extreme points
CX = c'*x1; % and their costs
numx = 1;
7.1. DECOMPOSITION 81

%Next, we build out the initial master basis

B = [horzcat(eye(m1), E*x1) ; horzcat(zeros(1,m1), 1)];
numcol = m1+1;
% OK, now we have a basis matrix for the master.
% Let's see if this basis is feasible.
zeta = linsolve(B,[h ; 1])
% If it is not feasible, we need to adjoin an artificial column
if (min(zeta) < −0.00000001)
disp('*** Phase−one needed');
% adjoin an artificial column
numcol = numcol + 1;
artind = numcol;
B(:,artind) = −B*ones(m1+1,1);
xi = zeros(numcol,1);
xi(artind) = 1;

% OK, now we are going to solve a phase−one problem to drive the artificial
% variable to zero. Of course, we need to do this with decomposition.
k = 1; % iteration counter
subval = −Inf;
sigma = 0.0;
while −sigma + subval < −0.00000001 && k <= MAXIT
% Set the dual variables by solving the restricted master
[zeta,masterval,exitflag,output,dualsol] = linprog(xi,[],[],B, ...
[h ; 1],zeros(numcol,1),[],options);
masterval
% Need to flip the signs due to a Matlab convention
y = − dualsol.eqlin(1:m1);
sigma = − dualsol.eqlin(m1+1);
% OK, now we can set up and solve the phase−one subproblem
[x,subval,exitflag] = linprog((−y'*E)',[],[],A,b,zeros(n,1),[],options);
if (exitflag < 1)
disp('fail 2: phase−one subproblem did not solve correctly');
return;
end;
% Append [E*x; 1] to B, and set up new restricted master
numcol = numcol + 1;
numx = numx + 1;
X = [X x];
CX = [CX ; c'*x];
B = [B [E*x ; 1]];
% Oh, we should build out the cost vector as well
xi = [xi ; 0];
k = k+1;
end;

% sanity check
if (−sigma + subval < −0.00000001)
disp('fail 3: we hit the max number of iterations for phase−one');
return;
end;

% sanity check
if (zeta(artind) > 0.00000001)
disp('fail 4: we thought we finished phase−one');
disp(' but it seems the aritifical variable is still positive');
82 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

disp(zeta(artind));
return;
end;

% phase−one clean−up: peel off the last column −−− which was not improving,
% and also delete the artifical column
B(:,numcol) =[];
numcol = numcol − 1;
X(:,numx) =[];
CX(numx,:) =[];
numx = numx − 1;
B(:,artind) = [];
numcol = numcol − 1;

end;

%
% Now time for phase−two
disp('*** Starting phase−two');

results1 = zeros(MAXIT,1);
results2 = zeros(MAXIT,1);

k = 0; % iteration counter
subval = −Inf;
sigma = 0.0;
while −sigma + subval < −0.00000001 && k <= MAXIT
k = k + 1;
% Set the dual variables by solving the restricted master
obj = [zeros(m1,1) ; CX];
[zeta,masterval,exitflag,output,dualsol] = linprog(obj,[],[],B, ...
[h ; 1],zeros(numcol,1),[],options);
masterval
% Need to flip the signs due to a Matlab convention
y = − dualsol.eqlin(1:m1);
sigma = − dualsol.eqlin(m1+1);
results1(k)=k;
results2(k)=masterval;
% OK, now we can set up and solve the phase−two subproblem
[x,subval,exitflag] = linprog((c'−y'*E)',[],[],A,b,zeros(n,1),[],options);
if (exitflag < 1)
disp('fail 5: phase−two subproblem did not solve correctly');
return;
end;
% Append [E*x; 1] to B, and set up new restricted master
numcol = numcol + 1;
numx = numx + 1;
X = [X x];
CX = [CX ; c'*x];
B = [B [E*x ; 1]];
end;

% sanity check
if (−sigma + subval < −0.00000001)
disp('fail 6: we hit the max number of iterations for phase−two');
return;
7.1. DECOMPOSITION 83

end;

% Clean up from phase−two:

%
% Peel off the last column −−− which was not improving
B(:,numcol) =[];
numcol = numcol −1;
X(:,numx) =[];
CX(numx,:) =[];
numx = numx − 1;

xdw = X*zeta(m1+1:numcol) % this is the solution in the original variables x

DWz = CX'*(zeta(m1+1:numcol))

if ((abs(DWz − z) < 0.01) && (abs(c'*xdw − DWz) < 0.00001) && ...
(norm(b−A*xdw) < 0.00001) && (norm(min(h−E*xdw,0)) < 0.00001))
disp('IT WORKED!!!'); % exactly what did we check?
end

clf; % clear figure

% plot the sequence of upper bounds
plot(results1(1:k),results2(1:k),'k−−.','MarkerSize',15);
hold on;
% plot a horizontal line at height z
plot([1,k],[ z z ], 'r−','LineWidth',1.5)
axis tight;
xlabel('Iteration');
ylabel('Decomposition upper bound');
legend('Decomposition UB','z');

print −djpeg Decomp.jpeg;

In Figure 7.2, we see quite good behavior for (“phase-two” of) the Decomposition
Algorithm.

Convergence and lower bounds. Practically speaking, the convergence behavior of

the Decomposition Algorithm can suffer from a tailing-off effect. That is, while the
sequence of objective values for successive iterates is non-increasing, at some point im-
provements can become quite small. It would be helpful to know when we already
have a very good but possibly non-optimal solution. If we could rapidly get a good
lower bound on z , then we could stop the Decomposition when the its objective value
is close to such a lower bound. Lower bounds on z can be obtained from feasible solu-
tions to the dual of (Q) . But there is another way, closely related to the dual of (Q), to
rapidly get good lower bounds. We develop this in the next section.
84 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

Figure 7.2: Example of upper bounds generated by Decomposition

7.2 Lagrangian Relaxation

7.2. LAGRANGIAN RELAXATION 85

Again, we consider
z := min c0 x
Ex = h ;
(Q)
Ax = b ;
x ≥ 0,
but our focus now is on efficiently getting a good lower bound on z , with again the
view that we are able to quickly solve many linear-optimization problems having only
the constraints: Ax = b , x ≥ 0 .

7.2.1 Lagrangian bounds

For any fixed choice of ŷ, consider the following “Lagrangian” optimization problem

v(ŷ) := ŷ 0 h + min (c0 − ŷ 0 E)x

Ax = b (Lŷ )
x ≥ 0.

Note that the only variables in the minimization are x , because we consider ŷ to be
fixed.

Theorem 7.3
v(ŷ) ≤ z , for all ŷ in the domain of v.

Proof. Let x∗ be an optimal solution for (Q). Clearly x∗ is feasible for (Lŷ ). Therefore

v(ŷ) ≤ ŷ 0 h + (c0 − ŷ 0 E)x∗

= c0 x∗ + ŷ 0 (h − Ex∗ )
= z.

The last equation uses the fact that x∗ is optimal for (Q), so z = c0 x∗ and Ex∗ = h . t
u

From what we learned in studying sensitivity analysis, it can be seen that v is a

concave (piecewise-linear) function on its domain (see Theorem 6.5). Because of this
nice behavior, it is plausible that we could calculate the maximum of v as a means of
getting a good lower bound on z . Before doing that, we examine the precise relation-
ship between primal and dual solutions of (Q), minimizers of v , and primal and dual
solutions of the Lagrangian.

Theorem 7.4
Suppose that x∗ is optimal for (Q) , and suppose that ŷ and π̂ are optimal for the dual
of (Q) . Then x∗ is optimal for (Lŷ ) , π̂ is optimal for the dual of (Lŷ ) , ŷ is a maximizer
of v , and the maximum value of v is z .
86 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

Proof. x∗ is clearly feasible for (Lŷ ) . Because ŷ and π̂ are feasible for the dual of (Q) ,
we have π̂ 0 A + ŷ 0 E ≤ c0 , and so π̂ is feasible for the dual of (Lŷ ) .
Using the Strong Duality Theorem for (Q) implies that c0 x∗ = ŷ 0 h + π̂ 0 b . Using
that E x̂∗ = h (feasibility of x∗ in (Q)), we then have that (c0 − ŷ 0 E) x∗ = π̂ 0 b . Finally,
using the Weak Duality Theorem for (Lŷ ) , we have that x∗ is optimal for (Lŷ ) and π̂ is
optimal for the dual of (Lŷ ) .
Next,

z ≥ v(ŷ) (by Theorem 7.3)

= ŷ h + c − ŷ 0 E x∗
0 0
(because x∗ is optimal for (Lŷ ))
= ŷ 0 h + c0 x∗ − ŷ 0 h (because Ex∗ = h)
= c0 x∗
= z.

Therefore the first inequality is an equation, and so ŷ is a maximizer of v and the max-
imum value is z . t
u

Theorem 7.5
Suppose that ŷ is a maximizer of v , and suppose that π̂ is optimal for the dual of (Lŷ ) .
Then ŷ and π̂ are optimal for the dual of (Q) , and the optimal value of (Q) is v(ŷ) .

Proof.

v(ŷ) = max {v(y)}

y
n 
o
= max y 0 h + min c0 − y 0 E x : Ax = b , x ≥ 0
y x
n  o
= max y h + max π 0 b : π 0 A ≤ c0 − y 0 E
0
y π
 0
= max y h + π b : y 0 E + π 0 A ≤ c0
0
y,π
= z.

The third equation follows from taking the dual of the inner (minimization) problem.
The last equation follows from seeing that the final maximization (over y and π simul-
taneously) is just the dual of (Q).
So, we have established that the optimal value z of (Q) is v(ŷ) . Looking a bit more
closely, we have established that z = ŷ 0 h + π̂ 0 b , and because π̂ 0 A ≤ c0 − ŷ 0 E , we have
that ŷ and π̂ are optimal for the dual of (Q) . t
u

Note that the conclusion of Theorem 7.5 gives us an optimal ŷ and π̂ for the dual of
(Q), but not an optimal x∗ for (Q) itself.
7.2. LAGRANGIAN RELAXATION 87

7.2.2 Solving the Lagrangian Dual

Theorem 7.3 gives us a simple way to calculate a lower bound on z , by solving a poten-
tially much-easier linear-optimization problem. But the bound depends on the choice
of ŷ ∈ Rm . Can we find the best such ŷ ? This would entail solving the so-called La-
grangian Dual problem of maximizing v(y) over all y in the domain of v . It should seem
that there is hope for doing this — because v is a concave function. But v is not a smooth
function (it is piecewise linear), so we cannot rely on calculus-based techniques.

Theorem 7.6
Suppose that we fix ŷ , and solve for v(ŷ) . Let x̂ be the solution of (Lŷ ) . Let γ̂ :=
h − E x̂ . Then
v(ỹ) ≤ v(ŷ) + (ỹ − ŷ)0 γ̂ ,
for all ỹ in the domain of v .

Proof.
v(ŷ) + (ỹ − ŷ)0 γ̂ = ŷ 0 h + (c0 − ŷ 0 E)x̂ + (ỹ − ŷ)0 (h − E x̂)
= ỹ 0 h + (c0 − ỹ 0 E)x̂
≥ v(ỹ) .
The last equation follows from the fact that x̂ is feasible (but possible not optimal) for
(Lỹ ). t
u

Subgradient. What is v(ŷ)+(ỹ− ŷ)0 γ̂ ? It is a linear estimation of v(ỹ) starting from the
actual value of v at ŷ . The direction ỹ − ŷ is what we add to ŷ to move to ỹ . The choice
of γ̂ := h − E x̂ is made so that Theorem 7.6 holds. That is, γ̂ is chosen in such a way
that the linear estimation is always an upper bound on the value v(ỹ) of the function,
for all ỹ in the domain of f . The nice property of γ̂ demonstrated with Theorem 7.6
has a name: we say that γ̂ := h − E x̂ is a subgradient of (the concave function) v at ŷ
(because it satisfies the inequality of Theorem 7.6).

Subgradient Optimization. Next, we describe a simple “Subgradient Optimization

Algorithm” for solving the Lagrangian Dual. The general idea is to iteratively move in
the direction of a subgradient.
Subgradient Optimization Algorithm

0. Start with any ŷ 1 ∈ Rm . Let k := 1 .

1. Solve (Lŷk ) to get x̂k .

2. Calculate the subgradient γ̂ k := h − E x̂k .

3. Let ŷ k+1 ← ŷ k + λk γ̂ k .
4. Let k ← k + 1 , and GOTO 1.
88 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

Convergence. We have neglected, thus far, to fully specify the Subgradient Optimiza-
tion Algorithm. We can stop if, at some iteration k , we have γ̂ k = 0, because the al-
gorithm will make no further progress if this happens, and indeed we will have found
that ŷ k is a maximizer of v . But this is actually very unlikely to happen. In practice, we
may stop if k reaches some pre-specified iteration limit, or if after many iterations, v is
barely increasing.
We are interested in mathematically analyzing the convergence behavior of the al-
gorithm, letting the algorithm iterate infinitely. We will see that the method converges
(in a certain sense), if we take a sequence P of λ2 k > 0 that in P∞ some sense slowly di-
verges; Specifically, we will require that ∞ λ
k=1 k < +∞ and k=1 λk = +∞ . That is,
“square summable, but not summable.” For example, taking λk := α/(β + k) , with
α > 0 and β ≥ 0 , we get a sequence of step sizes satisfying P this property; in par-
ticular, for αP= 1 and β = 0 we havePthe harmonic series ∞ k=1 1/k which satisfies
∞ ∞
ln(k + 1) < k=1 1/k < ln(k) + 1 and k=1 1/k = π /6 . 2 2

To prove convergence of the algorithm, we must first establish a key technical lemma.

Lemma 7.7
Let y ∗ be any maximizer of v . Suppose that λk > 0 , for all k . Then

k
X k

X
ky ∗ − ŷ k+1 k2 ≤ ky ∗ − ŷ 1 k2 − 2 λi v(y ∗ ) − v(ŷ i ) + λ2i kγ̂ i k2 .
i=1 i=1

Proof. The proof is by induction on k . The inequality is trivially true for the base case
of k = 0 . So, consider general k > 0 .
   
ky ∗ − ŷ k+1 k2 = k y ∗ − ŷ k − λk γ̂ k k2
   0    
= y ∗ − ŷ k − λk γ̂ k y ∗ − ŷ k − λk γ̂ k
 0
= ky ∗ − ŷ k k2 + kλk γ̂ k k2 − 2λk y ∗ − ŷ k γ̂ k
 
≤ ky ∗ − ŷ k k2 + λ2k kγ̂ k k2 − 2λk v(y ∗ ) − v(ŷ k )
" k−1 k−1
#
X
X
∗ 1 2 ∗ i 2 i 2
≤ ky − ŷ k − 2 λi v(y ) − v(ŷ ) + λi kγ̂ k
i=1 i=1
 
+ λ2k kγ̂ k k2 ∗
− 2λk v(y ) − v(ŷ ) k
.

The penultimate inequality uses the assumption that λk > 0 and the subgradient in-
equality:
v(ỹ) ≤ v(ŷ k ) + (ỹ − ŷ k )0 γ̂ k ,
plugging in y ∗ for ỹ. The final inequality uses the inductive hypothesis. The final ex-
pression easily simplifies to the right-hand side of the inequality in the statement of the
lemma. t
u
7.2. LAGRANGIAN RELAXATION 89

Now, let 
vk∗ := maxki=1 v(ŷ i ) , for k = 1, 2, . . .
That is, vk∗ is the best value seen up through the k-th iteration.

Theorem 7.8 (“Square summable, but not summable” convergence)

Let y ∗ be any maximizer of v . Assume that we take a basic solution as the solution of
each
P∞ Lagrangian subproblem.
P∞ Suppose that λk > 0 , for all k . Suppose further that
λ 2 < +∞ and λ = +∞ . Then limk→∞ vk∗ = v(y ∗ ) .
k=1 k k=1 k

Proof. Because the left-hand side of the inequality in the statement of Lemma 7.7 is
non-negative, we have
k
X k
X
∗

∗ 1 2
2 i
λi v(y ) − v(ŷ ) ≤ ky − ŷ k + λ2i kγ̂ i k2 .
i=1 i=1

Because vk∗ ≥ v(ŷ i ) for all i ≤ k , we then have

k
! k
X X
∗
2 λi (v(y ) − vk∗ ) ∗ 1 2
≤ ky − ŷ k + λ2i kγ̂ i k2 ,
i=1 i=1

or P
∗ ky ∗ − ŷ 1 k2 + ki=1 λ2i kγ̂ i k2
v(y ) − vk∗ ≤ P .
2 ki=1 λi
Next, we observe that kγ̂ i k2 is bounded by some constant Γ , independent of i , because
our algorithm takes γ̂ = h−E x̂ , where x̂ is a basic solution of a Lagrangian subproblem.
There are only a finite number of bases. Therefore, we can take

Γ = max kh − E x̂k2 : x̂ is a basic solution of Ax = b , x ≥ 0 .

So, we have P
∗ ky ∗ − ŷ 1 k2 + Γ ki=1 λ2i
v(y ) − vk∗ ≤ P .
2 ki=1 λi
P
Now, we get our result by observing that ky ∗ − ŷ 1 k2 is a constant, ki=1 λ2i is converging
P
to a constant and ki=1 λi goes to +∞ (as k increases without limit), and so the right-
hand side of the final inequality converges to zero. The result follows. t
u

A demonstration implementation. It is very easy to write a small MATLAB code for

Subgradient Optimization.
90 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

% SubgradientOpt.m // Jon Lee

%
% Apply Subgradient Optimization to
% z = min c'x
% Ex = h
% Ax = b
% x >= 0
%
% NOTE: WE ONLY HANDLE PROBLEMS WITH BOUNDED SUBPROBLEM LPs.
% IF WE ENCOUNTER AN UNBOUNDED SUBPROBLEM, THEN WE QUIT.

options = optimoptions('linprog','Algorithm','dual−simplex');

% generate a random example

n = 50; % number of variables
m1 = 15; % number of equations to relax
m2 = 10; % number of equations to keep

rng('default');
rng(1); % set seed
E = rand(m1,n); % random m1−by−n matrix
A = rand(m2,n); % random m2−by−n matrix

w = rand(n,1); % generate rhs's so problem is feasible

h = E*w;
b = A*w;

c = rand(n,1); % generate random objective

% first we will calculate the optimal value z of the original LP, just to
% compare later
disp('Optimizing the original LP without Lagrangian relaxation');
[x,z,exitflag] = linprog(c,[],[],[E; A],[h; b],zeros(n,1),[],options);
if (exitflag < 1)
disp('fail 1: original LP (without using LR) did not solve correctly');
return;
end;

disp('Optimal value of the original LP:');

z

disp('Starting Subgradient Optimization');

MAXIT = 100; % number of iterations

results1 = zeros(MAXIT,1);
results2 = zeros(MAXIT,1);

k = 0; % iteration counter
y = zeros(m1,1); % initialize y as you like
g = zeros(m1,1); % initialize g arbitrarily −−− it will essentially be ignored
stepsize = 0; % just to have us really start at the initialized y
bestlb = −Inf;
while k < MAXIT
k = k + 1; % increment the iteration counter
7.2. LAGRANGIAN RELAXATION 91

y = y + stepsize*g; % take a step in the direction of the subgradient

% solve the Lagrangian
[x,subval,exitflag,output,dualsol] = linprog((c'−y'*E)',[],[],A,b,zeros(n,1),[],options);
v = y'*h + (c'−y'*E)*x; % calculate the value of the Lagrangian
disp(v);
bestlb = max(bestlb,v);
results1(k)=k;
results2(k)=v;
g = h − E*x; % calculate the subgradient
stepsize = 1/k; % calculate the step size
end

bestlb

clf; % clear figure

% plot the sequence of MAXIT lower bounds
plot(results1,results2,'k−−.','MarkerSize',15);
hold on;
% plot a horizontal line at height z
plot([1,MAXIT],[ z z ], 'r−','LineWidth',2.5)
axis tight;
xlabel('Iteration');
ylabel('Lagrangian lower bound');
legend('Lagrangian LB','z','Location','SouthEast');

print −djpeg SubgradientOpt.jpeg;

% Next, if y really solves the Lagrangian dual, and pi is the

% optimal solution to the Lagrangian subproblem corresponding to y,
% then y and pi should be an optimal dual solution to the given problem.
%
% Let's see if y and pi are nearly dual feasible.

pi = − dualsol.eqlin;
Total_Dual_Infeasibility = norm(min(c' − y'*E − pi'*A,zeros(1,n)))

In Figure 7.3, we see quite good behavior for Subgradient Optimization.

Practical steps. Practically speaking, in order to get a ŷ with a reasonably high value
of v(ŷ) , it can be better to choose a sequence of λk that depends on a “good guess” of
the optimal value of v(ŷ), taking bigger steps when one is far away, and smaller steps
when one is close (try to develop this idea in Exercise 7.3). Then, the method is usually
stopped after a predetermined number of iterations or after progress becomes very
slow.

Dual estimation. From Theorem 7.5, we see that the Subgradient Optimization Method
is a way to try and quickly find estimates of an optimal solution to the dual of (Q). But
note that we give something up — we do not get an x∗ that solves (Q) from a ŷ that
maximizes v and a π̂ that is optimal for the dual of (Lŷ ) . There is no guarantee that a
x̂ that is optimal for (Lŷ ) will be feasible for (Q) . Moreover, we need for the Subgra-
92 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

Figure 7.3: Example of lower bounds generated by Subgradient Optimization

dient Optimization Method to have nearly converged to certify that ŷ and π̂ are nearly
optimal for the dual of (Q).

7.3 The Cutting-Stock Problem

The cutting-stock problem is a nice concrete topic at this point. We will develop a
technique for it, using column generation, but the context is different than for decom-
7.3. THE CUTTING-STOCK PROBLEM 93

position. Moreover, the topic is a nice segue into integer linear optimization — the topic
of the next chapter.
The story is as follows. We have stock rolls of some type of paper of (integer) width
W . But we encounter (integer) demand di for rolls of (integer) width wi < W , for
i = 1, 2, . . . , m . The cutting-stock problem is to find a plan for satisfying demand,
using as few stock rolls as possible.8

7.3.1 Formulation via cutting patterns

There are several different ways to formulate the cutting-stock problem mathematically.
A particularly useful way is based on a consideration of the problem from the point of
view of the worker who has to adjust the cutting machine. What she dearly hopes for
is that a plan can be formulated that does not require that the machine be adjusted for
(different cutting patterns) too many times. That is, she hopes that there are a relatively
small number of ways that will be utilized for cutting a stock roll, and that these good
ways can each be repeated many times.
With this idea in mind, we define a cutting pattern to be a solution of
Pm
i=1 wi ai ≤ W ;
ai ≥ 0 integer, i = 1, . . . , m ,
where ai is the number of pieces of width wi that the pattern yields.
Conceptually, we could form a matrix A with m rows, and an enormous number
of columns, where each column is a distinct pattern. Then, letting xj be the number of
times that we use pattern Aj , we can conceptually formulate the cutting-stock problem
as P
z := min x
P j j
j Aj x j ≥ d ; (CSP)
xj ≥ 0 integer, ∀ j .

7.3.2 Solution via continuous relaxation

Our approach to getting a good solution to (CSP) is to solve its continuous relaxation
and then round. Toward this end, we subtract surplus variables and consider the linear-
optimization problem
P
z := min x
P j j
j Aj xj − t = d ; (CSP)
xj ≥ 0 , ∀ j ;
t ≥0.

We endeavor to compute a basic optimum (x̄, t̄) . Because of the nature of the formula-
tion, we can see that dx̄e is feasible for (CSP). Moreover, we have produced a solution
using 10 dx̄e stock rolls, and we can give an a priori bound on its quality. Specifically,
as we will see in the next theorem, the solution that we obtain wastes at most m − 1
stock rolls, in comparison with an optimal solution. Moreover, we have a practically-
computable bound on the number of wasted rolls, which is no worse than the worst-
case bound of m − 1. That is, our waste is at worst 10 dx̄e − dze .
94 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

Theorem 7.9

dze ≤ z ≤ 10 dx̄e ≤ dze + (m − 1) .

Proof. Because (CSP) is a relaxation of (CSP) and because z is an integer, we have

dze ≤ zP. Because dx̄e is a feasible solution of (CSP), Pm we have z ≤0 1 dx̄e
0
P . Now,
m
1 dx̄e = P i=1 (x̄βi + fi ) , with each fi < 1 . P
0 But i=1 (x̄βi + fi ) = 1 x̄ + m i=1 fi ≤
d10 x̄e + m f
i=1 i . Therefore, 1 0 dx̄e ≤ d10 x̄e + m
f
i=1 i . Now the left-hand side of this
last inequality is an integer, so we may round down the right-hand side. So we can
conclude that 10 dx̄e ≤ dze + (m − 1) . t
u

7.3.3 The knapsack subproblem

Toward describing how we can solve (CSP) by the Simplex Algorithm, we introduce a
vector y ∈ Rm of dual variables.
P
z := min j xj dual variables
P
A x
j j j − t = d ; y
(CSP)
xj ≥ 0 , ∀ j ;
t ≥0.

We suppose that we have a feasible basis of (CSP) and that we have, at hand, the as-
sociated dual solution ȳ . For each i , 1 ≤ i ≤ m , the reduced cost of ti is simply ȳi .
Therefore, if ȳi < 0 , then ti is eligible to enter the basis.
So, moving forward, we may assume that ȳi ≥ 0 for all i . We now want to examine
the reduced cost of an xj variable. The reduced cost is simply

m
X
1 − ȳ 0 Aj = 1 − ȳi aij .
i=1

P
The variable xj is eligible to enter the basis then if 1 − m
i=1 ȳi aij < 0 . Therefore, to
check whether there is some column xj with negative reduced cost, we can solve the
so-called knapsack problem
Pm
max i=1 ȳi ai
Pm
i=1 wi ai ≤ W;
ai ≥ 0 integer, i = 1, . . . , m ,

and check whether the optimal value is greater than one. If it is, then the new variable
that we associate with this solution pattern (i.e., column of the constraint matrix) is
eligible to enter the basis.
7.3. THE CUTTING-STOCK PROBLEM 95

Our algorithmic approach for the knapsack problem is via recursive optimization
(known popularly as dynamic programming9 ). We will solve this problem for all positive
integers up through W . That is, we will solve
P
f (s) := max P m i=1 ȳi ai
m
i=1 wi ai ≤ s ;
ai ≥ 0 integer, i = 1, . . . , m ,

starting with f (s) = 0 , for 0 ≤ s < minm

i=1 {wi } , and proceeding from s = mini=1 {wi }−
m

1 by incrementing the argument of f by 1 at each step. Then, we have the recursion

f (s) = max {ȳi + f (s − wi )} , for s ≥ minm

i=1 {wi } .
i : wi ≤s

It is important to note that we can always calculate f (s) provided that we have already
calculated f (s0 ) for all s0 < s . Why does this work? It follows from a very simple
observation: If we have optimally filled a knapsack of capacity s and we remove any
item i, then what remains optimally fills a knapsack of capacity s − wi . If there were a
better way to fill the knapsack of capacity s−wi , then we could take such a way, replace
the item i , and we would have found a better way to fill a knapsack of capacity s . Of
course, we do not know even a single item that we can be sure is in an optimally filled
knapsack of capacity s , and this is why in the recursion, we maximize over all items
that can fit in (i.e., i : wi ≤ s).
The recursion appears to calculate the value of f (s) , but it is not immediate how to
recover optimal values of the ai . Actually, this is rather easy.

Recover the Solution of a Knapsack Problem

0. Let s := W , and let ai := 0 , for i = 1, . . . , m .

1. While (s > 0)

(a) Find ı̂ : f (s) = ȳı̂ + f (s − wı̂ ).

(b) Let aı̂ := aı̂ + 1 .
(c) Let s := s − wı̂ .

2. Return ai , for i = 1, . . . , m .

Note that in Step 1.a, there must be such an ı̂ , by virtue of the recursive formula for
calculating f (s) . In fact, if we like, we can save an appropriate ı̂ associated with each s
at the time that we calculate f (s) .
96 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

7.3.4 Applying the Simplex Algorithm

An initial feasible basis. It is easy to get an initial feasible basis. We just consider the
m patterns Ai := bW/wi cei , for i = 1, 2, . . . , m . The values of the m basic variables
associated with the basis of these patterns are x̄i = di /bW/wi c , which are clearly non-
negative.

Basic solutions: dual and primal. At any iteration, the basis matrix B has some columns
corresponding to patterns and possibly other columns for ti variables. The column cor-
responding to ti is −ei .
Organizing the basic variables xj and ti into a vector ζ , with their order appropri-
ately matched with the columns of B , the vector ζ̄ of values of ζ is precisely the solution
of
Bζ = d .
That is,
ζ̄ = B −1 d .
The cost of an xj is 1, while the cost of a ti is 0. Organizing the costs of the basic
variables into a vector ξ , with their order appropriately matched with the columns of
B , the associated dual solution ȳ is precisely the solution of

y0B = ξ0 .

That is,
ȳ 0 = ξ 0 B −1 .

7.3.5 A demonstration implementation

We can use AMPL, in a somewhat sophisticated manner, to implement our algorithm
for the cutting-stock problem. As we did for the Decomposition Algorithm, rather
than carry out the simplex method at a detailed level on (CSP), we just accumulate
all columns of (CSP) that we generate, and always solve linear-optimization problems,
using functionality of AMPL, with all of the columns generated thus far. In this way,
we do not maintain bases ourselves, and we do not carry out the detailed pivots of the
Simplex Algorithm. Note that the linear-optimization functionality of AMPL does give
us a dual solution, so we do not compute that ourselves.
First, we specify our csp.mod :

# csp.mod // Jon Lee

#

param W > 0; # width of stock rolls

param m >0; # number of widths
set WIDTHS:={1..m}; # set of widths to be cut
param d{WIDTHS} > 0; # number of each width to be cut
param w {WIDTHS} > 0; # actual widths
7.3. THE CUTTING-STOCK PROBLEM 97

param nPAT integer >= 0; # number of patterns

set PATTERNS := 1..nPAT; # set of patterns

param A {WIDTHS,PATTERNS} integer >= 0;

var X {PATTERNS} integer >= 0;

# rolls cut using each pattern
minimize RollsUsed: # minimize rolls used
sum {j in PATTERNS} X[j];

subj to FillDemand {i in WIDTHS}:

sum {j in PATTERNS} A[i,j] * X[j]>= d[i];

# −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−
# KNAPSACK SUBPROBLEM FOR CUTTING STOCK
# −−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−−

param ybar {WIDTHS} default 0.0;

var a {WIDTHS} integer >= 0;

minimize Reduced_Cost:
1 − sum {i in WIDTHS} ybar[i] * a[i];

subj to Width_Limit:
sum {i in WIDTHS} w[i] * a[i] <= W;

For a sample data set, we can have csp.dat :

110
5
70 205
40 2321
55 143
25 1089
35 117

Finally, we can have csp.run :

# csp.run // Jon Lee

#

reset;

option solver cplex;

option solution_round 6;
option solver_msg 0;

model csp.mod;
#data csp.dat;
98 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

read W,m,{i in 1..m} (w[i],d[i]) < csp.dat;

problem Cutting_Opt: RollsUsed, FillDemand, X;

option relax_integrality 1;
option presolve 0;

problem Pattern_Gen: Reduced_Cost, Width_Limit, a;

option relax_integrality 0;
option presolve 1;

let nPAT := 0;

for {i in WIDTHS} {
let nPAT := nPAT + 1;
let A[i,nPAT] := floor (W/w[i]);
let {i2 in WIDTHS: i2 <> i} A[i2,nPAT] := 0;
};

repeat {
solve Cutting_Opt > crap;
let {i in WIDTHS} ybar[i] := FillDemand[i].dual;
solve Pattern_Gen > crap;
if Reduced_Cost < −0.00001 then {
printf "\nImproving column generated. Reduced cost = %11.2e ", Reduced_Cost;
let nPAT := nPAT + 1;
let {i in WIDTHS} A[i,nPAT] := a[i];
}
else break;
};

print " "; print "LP solved! Solution: "; print " ";
display X;
display RollsUsed;
printf "\nLower bound = %11i (LP obj value rounded up) \n\n", ceil(RollsUsed);

let {j in PATTERNS} X[j] := ceil(X[j]); # round up

printf "\n\n\nSolution rounded up to integers: %3i rolls\n\n", sum {j in PATTERNS} X[j];

for {j in PATTERNS : X[j]>0} {

printf "CUT %4i OF:\n", X[j];
for {i in WIDTHS} {
printf "%3i ", i;
printf "%4i", A[i,j];
printf "\n";
}
printf "\n\n";
};

# Try to use integer−linear optimization to get a better solution

option Cutting_Opt.relax_integrality 0;
option Cutting_Opt.presolve 10;
solve Cutting_Opt;
7.3. THE CUTTING-STOCK PROBLEM 99

printf "\n\nBest integer solution considering all patterns generated:\n";

display RollsUsed; print " ";

for {j in PATTERNS : X[j]>0} {

printf "CUT %4i OF:\n", X[j];
for {i in WIDTHS} {
printf "%3i ", i;
printf "%4i", A[i,j];
printf "\n";
}
printf "\n\n";
};

On the data set provided, we get the following output:

Improving column generated. Reduced cost = -5.00e-01

Improving column generated. Reduced cost = -2.50e-01
Improving column generated. Reduced cost = -8.33e-02
Improving column generated. Reduced cost = -5.56e-02
Improving column generated. Reduced cost = 5.55e-17
LP solved! Solution:

X [*] :=
1 0
2 0
3 71.5
4 0
5 0
6 205
7 1033.38
8 18.5385
9 49.2308
;

RollsUsed = 1377.65

Lower bound = 1378 (LP obj value rounded up)

Solution rounded up to integers: 1380 rolls

CUT 72 OF:
1 0
2 0
3 2
4 0
5 0
100 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

CUT 205 OF:

1 1
2 1
3 0
4 0
5 0

CUT 1034 OF:

1 0
2 2
3 0
4 1
5 0

CUT 19 OF:
1 0
2 0
3 0
4 3
5 1

CUT 50 OF:
1 0
2 1
3 0
4 0
5 2

CPLEX 12.2.0.0:

Best integer solution considering all patterns generated:

RollsUsed = 1379

CUT 72 OF:
1 0
2 0
3 2
4 0
5 0

CUT 205 OF:

1 1
2 1
3 0
4 0
5 0
7.4. EXERCISES 101

CUT 1034 OF:

1 0
2 2
3 0
4 1
5 0

CUT 19 OF:
1 0
2 0
3 0
4 3
5 1

CUT 49 OF:
1 0
2 1
3 0
4 0
5 2

Our algorithm gives a lower bound of 1378 on the minimum number of stock rolls
needed to cover demand, and it gives us an upper bound (feasible solution) of 1380.
By solving a further integer-linear optimization problem to determine the best way to
cover demand using all patterns generated in the course of our algorithm, we improve
the upper bound to 1379. It remains unknown as to whether the optimal solution to
this instance is 1378 or 1379.

7.4 Exercises
Exercise 7.1 (Dual solutions)
Refer to (Q) and (M) defined in the Decomposition Theorem (i.e., Corollary 7.2) What
is the relationship between optimal dual solutions of (Q) and (M) ?

Exercise 7.2 (Lagrangian value function)

Using Theorem 6.5, prove that v (from Section 7.2.1) is a concave piecewise-linear func-
tion on its domain.
Exercise 7.3 (Play with subgradient optimization)
Play with the MATLAB code for Subgradient Optimization. Try bigger examples. Try
different ideas for the step size, with the goal of gaining faster convergence — be a real
engineer and think ‘outside of the box’ (you can use any information you like: e.g., the
current subgradient γ̂ k , the current function value v(γ̂ k ), an estimate of the maximum
value of v, etc.).
102 CHAPTER 7. LARGE-SCALE LINEAR OPTIMIZATION

Also, Theorem 7.5, together with the fact that Subgradient Optimization converges
in the limit to a a solution of the Lagrangian Dual together, tells us that the ŷ and π̂
produced should be optimal for the dual of (Q). Check this out: see to what extent ŷ
and π̂ are optimal for the dual of (Q) (i.e., feasible and objective value near z).

Exercise 7.4 (Cutting it closer to reality)

Real cutting machines may have a limited number, say K , of blades. PThis means that
m
we
Pm can cut at most K + 1 pieces for patterns that leave no scrap (i.e., Pm i ai = W ⇒
i=1 w
P ai ≤ K +1) and at most K pieces for patterns that leave scrap (i.e., i=1 wi ai < W
i=1
⇒ m i=1 ai ≤ K). Describe how to modify our algorithm for the cutting-stock problem
to account for this. Modify the AMPL code that I provided to try this out.

Exercise 7.5 (Another kind of question)

Print is dying, right? Why should we care about the cutting-stock problem?
Chapter 8

Integer-Linear Optimization

Our goals in this chapter are as follows:

• to develop some elementary facility with modeling using integer variables;

• to learn how to recognize when we can expect solutions of linear-optimization

problems to be integer automatically;

• to learn the fundamentals of the ideas that most solvers employ to handle integer
variables;

• to learn something about solver-aware modeling in the context of integer variables.

8.1 Integrality for Free

8.1.1 Some structured models
Network-flow problem. A finite network G is described by a finite set of nodes N
and a finite set A of arcs. Each arc e has two key attributes, namely its tail t(e) ∈ N and
its head h(e) ∈ N , both nodes. We think of a (single) commodity as being allowed to
“flow” along each arc, from its tail to its head. Indeed, we have “flow” variables

xe := amount of flow on arc e ,

for e ∈ A . Formally, a flow x̂ on G is simply an assignment of any real numbers x̂e to

the variables xe , for all e ∈ A . We assume that the flow on arc e should be non-negative
and should not exceed

ue := the flow upper bound on arc e ,

103
104 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

for e ∈ A . Associated with each arc e is a cost

ce := cost per-unit-flow on arc e ,

for e ∈ A . The (total) cost of the flow x̂ is defined to be

X
ce x̂e .
e∈A

We assume that we have further data for the nodes. Namely,

bv := the net supply at node v ,

for v ∈ N . A flow is conservative if the net flow out of node v , minus the net flow into
node v , is equal to the net supply at node v , for all nodes v ∈ N .
The (single-commodity min-cost) network-flow problem is to find a minimum-
cost conservative flow that is non-negative and respects the flow upper bounds on the
arcs. We can formulate this as follows:
X
min ce xe
e∈A
X X
xe − xe = bv , ∀ v ∈ N ;
e∈A : e∈A :
t(e)=v h(e)=v

0 ≤ xe ≤ ue , ∀e∈A.

As we have stated this, it is just a structured linear-optimization problem. But there are
many situations where the given net supplies at the nodes and the given flow capacities
on the arcs are integer, and we wish to constrain the flow variables to be integers.
We will see that it is useful to think of the network-flow problem in matrix-vector
language. We define the network matrix of G to be a matrix A having rows indexed
from N , columns indexed from A , and entries

 1 , if v = t(e) ;
ave := −1 , if v = h(e);

0 , if v ∈
/ {t(e), h(e)},

for v ∈ N , e ∈ A . With this notation, and organizing the bv in a column-vector indexed

accordingly with the rows of A , and organizing the ce , xe and ue as three column-
vectors indexed accordingly with the columns of A , we can rewrite the network-flow
formulation as
min c0 x
Ax = b ;
x ≤ u;
x ≥ 0.
8.1. INTEGRALITY FOR FREE 105

Assignment problem on a graph.

A finite bipartite graph G is described by two finite sets of vertices V1 and V2 , and
a set E of ordered pairs of edges, each one of which is of the form (i, j) with i ∈ V1 and
j ∈ V2 . A perfect matching M of G is a subset of E such that each vertex of the graph
meets exactly one edge in M . We assume that there are given edge weights

cij := for (i, j) ∈ E ,

and our goal is to find a perfect matching that has minimum (total) weight.
We can define

xij := indicator variable for choosing edge (i, j) to be in M ,

for all (i, j) ∈ E . Then we can model the problem of finding a perfect matching of G
having minimum weight via the formulation:
X
min cij xij
(i,j)∈E
X
xij = 1, ∀ i ∈ V1 ;
j∈V2 :
(i,j)∈E
X
xij = 1, ∀ j ∈ V2 ;
i∈V1 :
(i,j)∈E

xij ∈ {0, 1} , ∀ (i, j) ∈ E .

It will be useful to think of this assignment-problem formulation in matrix-vector

language. We define the vertex-edge incidence matrix of the bipartite graph G to be
a matrix A having rows indexed from V1 ∪ V2 , columns indexed from E , and entries

1 , if v = i or v = j ;
av,(i,j) :=
0 , otherwise,

for v ∈ V1 ∪V2 , (i, j) ∈ E . With this notation, and organizing the cij , xij and as column-
vectors indexed accordingly with the columns of A , we can rewrite the assignment-
problem formulation as
min c0 x
Ax = 1 ;
x ∈ {0, 1}E ,
106 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Staffing problem. In this problem, we have discrete time periods numbered 1, 2, . . . , m ,

and we are given
bi := the minimum number of workers required at time period i ,
for each i = 1, 2, . . . , m . Additionally, there is an allowable set of “shifts.” An allowable
shift is simply a given collection of time periods that a worker is allowed to staff. It
may well be that not all subsets of {1, 2, . . . , m} are allowable; e.g., we may not want
to allow too many or too few time periods, and we may not want to allow idle time
to be interspersed between non-idle times. We suppose that the allowable shifts are
numbered 1, 2, . . . , n , and we have
cj := the per worker cost to staff shift j ,
for each j = 1, 2, . . . , n . It is convenient to encode the shifts as a 0, 1-valued matrix A ,
where 
1 , if shift j contains time period i ;
aij :=
0 , otherwise,
for i = 1, 2, . . . , m , j = 1, 2, . . . , n . Letting x be an n-vector of variables, with xj
representing the number of workers assigned to shift j , we can formulate the staffing
problem as
min c0 x
Ax ≥ b ;
x ≥ 0 integer.

As we have stated it, this staffing problem is really a very general type of integer-
linear-optimization problem because we have not restricted the form of A beyond it
being 0, 1-valued. In some situations, however, it may may be reasonable to assume
that shifts must consist of a consecutive set of time periods. In this case, the 1’s in each
column of A occur consecutively, so we call A a consecutive-ones matrix.

8.1.2 Unimodular basis matrices and total unimodularity

In this section we explore the essential properties of a constraint matrix so that basic
solutions are guaranteed to be integer. This has important implications for the network-
flow, assignment, and staffing problems that we introduced.
8.1. INTEGRALITY FOR FREE 107

Definition 8.1
Let A be an m × n real matrix. A basis matrix Aβ is unimodular if det(Aβ ) = ±1 .
Checking whether a large unstructured matrix has all of its basis matrices unimod-
ular is not a simple matter. Nonetheless, we will see that this property is very useful
for guaranteeing integer optimal of linear-optimization problems, and certain structured
constraint matrices have this property.

Theorem 8.2
If A is an integer matrix, all basis matrices of A are unimodular, and b is an integer
vector, then every basic solution x̄ of

Ax = b ;
x ≥ 0

is an integer vector.

Proof. Of course x̄ηj = 0, an integer, for j = 1, 2, . . . , n − m, so we concentrate now on

the basic variables. By Cramer’s rule, the basic variables take on the values
det(Aβ (i))
x̄βi = , for i = 1, 2, . . . , m ,
det(Aβ )
where Aβ (i) is defined to be the matrix Aβ with its i-th column, Aβi , replaced by b .
Because we assume that A and b are all integer, the numerator above is the determinant
of an integer matrix, which is an integer. Next, the fact that A has unimodular basis
matrices tells us that the determinant of the invertible matrix Aβ is ±1 . That is, the
denominator above is ±1 . So, we have an integer divided by ±1 , which results in an
integer value for x̄βi . t
u
We note that Theorem 8.2 asserts that all basic solutions are integer, whether or not
they are feasible. There is a converse to this theorem.

Theorem 8.3
Let A be an integer matrix in Rm×n . If the system

Ax = b ;
x ≥ 0

has integer basic feasible solutions for every integer vector b ∈ Rm , then all basis
matrices of A are unimodular.

It is important to note that the hypothesis of Theorem 8.3 is weaker than the con-
clusion of Theorem 8.2. For Theorem 8.3, we only require integrality for basic feasible
solutions.
108 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Proof. (Theorem 8.3). Let β be an arbitrary basis, choose an arbitrary i (1 ≤ i ≤ m),

and consider the associated basic solution when b := ei + ∆Aβ 1 . The basic solution
x̄ has x̄β equal to the i-th column of A−1
β plus ∆1 . Note that if we choose ∆ to be an
integer, then b is integer. Furthermore, if we choose ∆ to be sufficiently large, then x̄β
is non-negative. Therefore, we can chose ∆ so that b is integer and x̄ is a basic feasible
solution. Therefore, by our hypothesis, x̄ is integer. So the i-th column of A−1
β plus ∆1
−1
is an integer vector . But this clearly implies that the i-th column of Aβ is an integer
vector . Now, because i was arbitrary, we conclude that A−1 β is an integer matrix . Of
course Aβ is an integer matrix as well. Now, it is a trivial observation that an integer
matrix has an integer determinant. Furthermore, the determinants of Aβ and A−1 β are
reciprocals. Of course the only integers with integer reciprocal are 1 and −1 . Therefore,
the determinant of Aβ is 1 or −1 . t
u

Before turning to specific structured linear-optimization problems, we introduce a

stronger property than unimodularity of basis matrices. The main reason for intro-
ducing it is that for the structured linear-optimization problems that we will look at,
the constraint matrices satisfy this stronger property, and the inductive proofs that we
would deploy for proving the weaker property naturally prove the stronger property
as well.
Definition 8.4
Let A be an m × n real matrix. A is totally unimodular (TU) if every square non-
singular submatrix B of A has det(B) = ±1 .

Obviously every entry of a TU matrix must be 0 , ±1 , because the determinant of

a 1 × 1 submatrix is just its single entry. It is quite easy to make an example of even a
2 × 2 non-TU matrix with all entries 0 , ±1:
 
1 −1
.
1 1

It is trivial to see that if A is TU, then every basis matrix of A is unimodular. But
note that even for integer A , every basis matrix of A could be unimodular, but A need
not be TU. For example,  
2 1
1 1
has only itself as a basis matrix, and its determinant is 1, but there is a 1 × 1 submatrix
with determinant 2, so A is not TU. Still, as the next result indicates, there is a way to
get the TU property from unimodularity of basis matrices.

Theorem 8.5
If every basis matrix of [A, Im ] is unimodular, then A is TU.

Proof. Let B be an r × r invertible submatrix of A , with r < m . It is an easy matter to

choose a (m × m) basis matrix H of [A, Im ] that includes all r columns of A that include
8.1. INTEGRALITY FOR FREE 109

columns of B , and then the m − r identity columns that have their ones in rows other
than those used by B . If we permute the rows of A so that B is within the first r rows,
then we can put the identity columns to the right, in their natural order, and the basis
we construct is !
B 0
H= .
× Im−r
Clearly B and H have the same determinant. Therefore, the fact that every basis matrix
has determinant 1 or −1 implies that B does as well. t
u

Next, we point out some simple transformations that preserve the TU property.

Theorem 8.6
If A is TU, then all of the following leave A TU.

(i) multiplying any rows or columns of A by −1 ;

(ii) duplicating any rows or columns of A ;

(iii) appending standard-unit columns (that is, all entries equal to 0 except a single
entry of 1) ;

(iv) taking the transpose of A .

We leave the simple proof to the reader.

Remark 8.7
Relationship with transformations of linear-optimization problems. The significance
of Theorem 8.6 for linear-optimization problems can be understood via the following
observations:

• (i) allows for reversing the sense of an inequality (i.e., switching between “≤”
and “≥”) or variable (i.e., switching between non-negative and non-positive) in
a linear-optimization problem with constraint matrix A .

• (ii) together with (i) allows for replacing an equation with a pair of oppositely
senses inequalities and for replacing a sign-unrestricted variable with the differ-
ence of a pair of non-negative variables.

• (iii) allows for adding a non-negative slack variable for a “≤” inequality, to trans-
form it into an equation. Combining (iii) with (i) , we can similarly subtract a
non-negative surplus variable for a “≥” inequality, to transform it into an equa-
tion.

• (iv) allows for taking the dual of a linear-optimization problem with constraint
matrix A .
110 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

8.1.3 Consequences of total unimodularity

Network flow.

Theorem 8.8
If A is a network matrix, then A is TU.

Proof. A network matrix is simply a 0 , ±1-valued matrix with exactly one +1 and one
−1 in each column.
Let B be an r × r invertible submatrix of the network matrix A . We will demon-
strate that det(B) = ±1 , by induction on r . For the base case, r = 1 , the invertible
submatrices have a single entry which is ±1 , which of course has determinant ±1 .
Now suppose that r > 1 , and we inductively assume that all (r − 1) × (r − 1) invertible
submatrices of A have determinant ±1 .
Because we assume that B is invertible, it cannot have a column that is a zero-vector.
Moreover, it cannot be that every column of B has exactly one +1 and one −1 . Be-
cause, by simply adding up all the rows of B , we have a non-trivial linear combination
of the rows of B which yields the zero vector. Therefore, B is not invertible in this case.
So, we only need to consider the situation in which B has a column with a single
non-zero ±1 . By expanding the determinant along such a column, we see that, up to a
sign, the determinant of B is the same as the determinant of an (r−1)×(r−1) invertible
submatrix of A . By the inductive hypothesis, this is ±1 . t
u

Corollary 8.9
The (single-commodity min-cost) network-flow formulation
X
min ce xe
e∈A
X X
xe − xe = bv , ∀ v ∈ N ;
e∈A : e∈A :
t(e)=v h(e)=v

0 ≤ xe ≤ ue , ∀e∈A.

has an integer optimal solution if: (i) it has an optimal solution, (ii) each bv is an
integer, and (iii) each ue is an integer or is infinite.

Proof. Recall that we can rewrite the network-flow formulation as

min c0 x
Ax = b ;
x ≤ u;
x ≥ 0,
8.1. INTEGRALITY FOR FREE 111

where A is a network matrix. For the purpose of proving the theorem, we may as well
assume that the linear-optimization problem has an optimal solution. Next, we trans-
form the formulation into standard form:
min c0 x
Ax = b;
x + s = u;
x , s ≥ 0.
 
A 0
The constraint matrix has the form . This matrix is TU, by virtue of the fact
I I
that A is TU, and that it arises from A using operations that preserve the TU property.
Finally, we delete any redundant equations from this system of equations, and we delete
any rows that have infinite right-hand side ue . The resulting constraint matrix is TU,
and the right-hand side is integer, so an optimal basic solution exists and will be integer.
t
u

Assignments.

Theorem 8.10
If A is the vertex-edge incidence matrix of a bipartite graph, then A is TU.

Proof. The constraint matrix A for the formulation has its rows indexed by the vertices
of G . With each edge having exactly one vertex in V1 and exactly one vertex in V2 , the
constraint matrix has the property that for each column, the only non-zeros are a single
1 in a row indexed from V1 and a single 1 in a row indexed from V2 .
Certainly multiplying any rows (or columns) of a matrix does not bear upon whether
or not it is TU. It is easy to see that by multiplying the rows of A indexed from V1 , we
obtain a network matrix, thus by Theorem 8.8, the result follows. t
u

Corollary 8.11
The continuous relaxation of the following formulation for finding a minimum-weight
perfect matching of the bipartite graph G has an 0, 1-valued solution whenever it is
feasible. X
min cij xij
(i,j)∈E
X
xij = 1 , ∀ i ∈ V1 ;
j∈V2 :
(i,j)∈E
X
xij = 1 , ∀ j ∈ V2 ;
i∈V1 :
(i,j)∈E

xij ≥ 0 , ∀ (i, j) ∈ E .
112 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Proof. After deleting any redundant equations, the resulting formulation as a TU con-
straint matrix and integer right-hand side. Therefore, its basic solutions are all integer.
The constraints imply that no variable can be greater than 1, therefore the optimal value
is not unbounded, and the only integer solutions have all xij ∈ {0, 1} . The result fol-
lows. t
u

A matching M of G is a subset of E such that each vertex of the graph is met by

no more that one edge in M . An interesting variation on the problem of finding a perfect
matching of G having minimum weight, is to find a maximum-cardinality matching of
G . This problem is always feasible, because M := ∅ is always a matching.
How big can a matching of a finite graph G be? A vertex cover of G is a set W of
vertices that touches all of the edges of G . Notice that if M is a matching and W is a
vertex cover, then |M | ≤ |W | , because each element of W touches at most one element
of M . Can we always find a matching M and a vertex cover W so that |M | = |W | ? The
next result, due to the mathematician König10 , tells us that the answer is ’yes’ when G
is bipartite.

Corollary 8.12 (König’s Theorem)

If G is a bipartite graph, then the maximum cardinality of a matching of G is equal to
the minimum cardinality of a vertex cover of G .

Proof. We can formulate the problem of finding the maximum cardinality of a matching
of G as follows:
X
max xij
(i,j)∈E
X
xij ≤ 1 , ∀ i ∈ V1 ;
j∈V2 :
(i,j)∈E
X
xij ≤ 1 , ∀ j ∈ V2 ;
i∈V1 :
(i,j)∈E

xij ≥ 0 integer, ∀ (i, j) ∈ E .

It is easy to see that we can relax integrality, and the optimal value will be unchanged,
because A is TU, and the constraint matrix will remain TU after introducing slack vari-
ables. The dual of the resulting linear-optimization problem is
X
min yv
v∈V

yi + yj ≥ 1 , ∀ (i, j) ∈ E ;
yv ≥ 0 , ∀ v ∈ V .
It is easy to see that after putting this into standard form via the subtraction of sur-
plus variables, the constraint matrix has the form [A0 , −I] , where A is the vertex-edge
incidence matrix of G . This matrix is TU, therefore an optimal integer solution exists.
8.1. INTEGRALITY FOR FREE 113

Next, we observe that because of the minimization objective and the form of the
constraints, an optimal integer solution will be 0, 1-valued; just observe that if ȳ is an
integer feasible solution and ȳv > 1 , for some v ∈ V , then decreasing ȳv to 1 (holding
the other components of ȳ constant, produces another integer feasible solution with a
lesser objective value. This implies that every integer feasible solution ȳ with any ȳv > 1
is not optimal.
Next, let ŷ be an optimal 0, 1-valued solution. Let

W := {v ∈ V : ŷv = 1} .
P
It is easy to see that W is a vertex cover of G and that |W | = v∈V ŷv . The result now
follows from the strong duality theorem. t
u

Staffing.

Theorem 8.13
If A is a consecutive-ones matrix, then A is TU.

Proof. Let B be an r × r invertible submatrix of a consecutive-ones matrix A . We will

demonstrate that det(B) = ±1 , by induction on r . We take care that we preserve the
ordering of the rows of A in B . In this way, B is also a consecutive-ones matrix. Note
that only the sign of the determinant of B depends on the ordering of its rows (and
columns).
For the base case, r = 1 , the invertible submatrix B has a single entry which is
1 , which of course has determinant 1 . Now suppose that r > 1 , and we inductively
assume that all (r − 1) × (r − 1) invertible submatrices of all consecutive-ones matrices
have determinant ±1 . (We will see that the ’all’ in the inductive hypothesis will be
needed — it will not be enough to consider just (r − 1) × (r − 1) invertible submatrices
of our given matrix A).
Next, we will reorder the columns of B so that all columns with a 1 in the first row
come before all columns with a 0 in the first row. Note that there must be a column
with a 1 in the first row, otherwise B would not be invertible. Next, we further reorder
the columns, so that among all columns with a 1 in the first row, a column of that type
with the fewest number of 1s is first.
114 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Our matrix B now has this form

 
1 1···1 0···0
 1 1···1 
 
 .. .. .. 
 . . . 
 
 1 1···1 ,
 



0
..
G 



0
. F 

where F and G are the submatrices indicated. Note that F and G are each consecutive-
ones matrices.

Next, we subtract the top row from all other rows that have a 1 in the first column.
Such row operations do not change the determinant of B , and we get a matrix of the
form  
1 1···1 0···0
 0 0···0 
 
 .. .. .. 
 . . . 
 
 0 0···0 .
 
 0

 .
G 


 .
0
.
F 

Note that this resulting matrix need not be a consecutive-ones matrix — but that is
not needed. By expanding the determinant of this latter matrix along the first column,
we see that the determinant of this matrix is the same as that of the matrix obtained by
striking out its first row and column,
 
0···0
 .. .. 
 . . 
 
 0···0 
 .

 G 


F 

But this matrix is an (r − 1) × (r − 1) invertible consecutive-ones matrix (note that it is

not necessarily a submatrix of A). So, by our inductive hypothesis, its determinant is
±1 .
t
u
8.2. MODELING TECHNIQUES 115

Corollary 8.14
Let A be a shift matrix such that each shift is a contiguous set of time periods, let c be
a vector of non-negative costs, and let b be a vector of non-negative integer demands
for workers in the time periods. Then there is an optimal solution x̄ of the continuous
relaxation
min c0 x
Ax ≥ b ;
x ≥ 0
of the staffing formulation that has x̄ integer, whenever the relaxation is feasible.

Proof. A is a consecutive-ones matrix when each shift is a contiguous set of time periods.
Therefore A is TU. After subtracting surplus variables to put the problem into standard
form, the constraint matrix takes the form [A, −I] , which is also T U . The result follows.
t
u

8.2 Modeling Techniques

116 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

8.2.1 Disjunctions

Example 8.15
Suppose that we have a single variable x ∈ R , and we want to model the disjunction
−12 ≤ x ≤ 2 or 5 ≤ x ≤ 20 .
By introducing a binary variable y ∈ {0, 1} , we can model the disjunction as
x ≤ 2 + M1 y ,
x + M2 (1 − y) ≥ 5 ,
where the constant scalars M1 and M2 (so-called big M’s) are chosen to be appropri-
ately large. A little analysis tell us how large. Considering our assumption that x could
be as large as 20 , we see that M1 should be at least 18 . Considering our assumption
that x could be as small as −12 , we see that M2 should be at least 17 . In fact, we should
choose these constants to be as small as possible so as make the feasible region with
y ∈ {0, 1} relaxed to 0 ≤ y ≤ 1 as small as possible. So, the best model for us is:
x ≤ 2 + 18y ,
x + 17(1 − y) ≥ 5 .
It is interesting to see a two-dimensional graph of this in x − y space; see Figures 8.1
and 8.2.

8.2.2 Forcing constraints

The uncapacitated facility-location problem involves n customers, numbered 1, 2, . . . , n
and m facilities, numbered 1, 2 . . . , m . Associated with each facility, we have
fi := fixed cost for operating facility i ,
8.2. MODELING TECHNIQUES 117

Figure 8.1: Optimal choice of “big M’s”

Figure 8.2: Comparing optimal vs non-optimal “big M’s”

for i = 1, . . . , m . Associated with each customer/facility pair, we have

cij := cost for satisfying all of customer j’s demand from facility i ,

for i = 1, . . . , m , j = 1, . . . , n . The goal is to determine a set of facilities to operate and

an allocation of each customers demand across operation facilities, so as to minimize
118 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

the total cost. The problem is “uncapacitated” in the sense that each facility has no limit
on its ability to satisfy demand from even all customers.
We formulate this optimization problem with

yi := indicator variable for operating facility i ,

for i = 1, . . . , m , and

xij := fraction of customer j demand satisfied by facility i ,

for i = 1, . . . , m , j = 1, . . . , n .
Our formulation is as follows:
Pm Pm Pn
min i=1 fi yi + i=1 j=1 cij xij
Pm
i=1 xij = 1, for j = 1, . . . , n ;

for i = 1, . . . , m ,
−yi + xij ≤ 0,
j = 1, . . . , n ;

yi ∈ {0, 1}, for i = 1, . . . , m ;

for i = 1, . . . , m ,
xij ≥ 0 ,
j = 1, . . . , n .

All of these constraints are self-explanatory except for the mn constraints:

−yi + xij ≤ 0 for i = 1, . . . , m , j = 1, . . . , n . (S)

These constraints simply enforce that for any feasible solution x̂, ŷ , we have that ŷi = 1
whenever x̂ij > 0 . It is an interesting point that this could also be enforced via the m
constraints:
Xn
−nyi + xij ≤ 0 , for i = 1, . . . , m . (W)
j=1

We can view the coefficient −n of yi as a “big M”, rendering the constraint vacuous
when yi = 1 .
Despite the apparent parsimony of the latter formulation, it turns out that the orig-
inal formulation is preferred.

8.2.3 Piecewise-linear univariate functions

Of course many useful functions are non-linear. Integer-linear optimization affords a
good way to approximate well-behaved univariate non-linear functions. Suppose that
f : R → R has domain the interval [l, u] , with l < u . For some n ≥ 2 , we choose n
breakpoints l = ξ 1 < ξ 2 < · · · < ξ n−1 < ξ n = u . Then, we approximate f linearly
between adjacent pairs of breakpoints. That is, we approximate f by
n
X
fˆ(x) := λj f (ξ j ) ,
j=1
8.2. MODELING TECHNIQUES 119

where we require that

n
X
λj = 1;
j=1
λj ≥ 0 , for j = 1, . . . , n ,

and the adjacency condition:

λj and λj+1 may be positive for only one value of j .

This adjacency condition means that we “activate” the interval [ξ j , ξ j+1 ] for approxi-
mating f (x) . That is, we will approximate f (x) by

λj f (ξ j ) + λj+1 f (ξ j+1 ) ,

with

λj + λj+1 = 1 ;
λj , λj+1 ≥ 0 .

We can enforce the adjacency condition using 0, 1-variables. Let


1, if the interval [ξ j , ξ j+1 ] is activated;
yj :=
0, otherwise,

for j = 1, 2, . . . , n − 1 .
The situation is depicted in Figure 8.3, where the red curve graphs the non-linear
function f .
We only want to allow one of the n − 1 intervals to be activated, so we use the
constraint
n−1
X
yj = 1 .
j=1

We only want to allow λ1 > 0 if the first interval [ξ 1 , ξ 2 ] is activated. For an internal
breakpoint ξ j , 1 < j < n , we only want to allow λj > 0 if either [ξ j−1 , ξ j ] or [ξ j , ξ j+1 ]
is activated. We only want to allow λn > 0 if the last interval [ξ n−1 , ξ n ] is activated. We
can accomplish these restrictions with the constraints

λ1 ≤ y1 ;
λj ≤ yj−1 + yj , for j = 2, . . . , n − 1 ;
λn ≤ yn−1 .

Notice how if yk is 1 , for some k (1 ≤ k ≤ n), and necessarily all of the other yj are
0 (j 6= k), then only λk and λk+1 can be positive.
How do we actually use this? If we have a model involving such
P a non-linear f (x),
then wherever we have f (x) in the model, we simply substitute nj=1 λj f (ξ j ) , and we
120 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Figure 8.3: Piecewise-linear approximation

incorporate the further constraints:

n
X
λj = 1;
j=1
n−1
X
yj = 1;
j=1
λ 1 ≤ y1 ;
λj ≤ yj−1 + yj , for j = 2, . . . , n − 1 ;
λn ≤ yn−1 ;
λj ≥ 0 , for j = 1, . . . , n ;
yj ∈ {0, 1} , for j = 1, . . . , n − 1 .
Pn
Of course a very non-linear f (x) will demand an fˆ(x) := j=1 λj f (ξ ) with a high
j

value for n , so as to get an accurate approximation. And higher values for n imply
more binary variables yj , which come at a high computational cost.
8.3. A PRELUDE TO ALGORITHMS 121

8.3 A Prelude to Algorithms

For reasons that will become apparent, for the purpose of developing algorithms for
linear-optimization problems in which some variable are required to be integer, it is
convenient to assume that our problem has the form
z := max y 0 b
y 0 A ≤ c0 ;
(DI )
y ∈ Rm ;
yi integer, for i ∈ I .
The set I ⊂ {1, 2, . . . , m} allows for a given subset of the variables to be constrained to
be integer. This linear-optimization problem has a non-standard form, but it is conve-
nient that the dual of the continuous relaxation has the standard form
min c0 x
Ax = b ; (P)
x ≥ 0.
To prove that an algorithm for (DI ) is finite, it is helpful to assume that the feasible
region of the continuous relaxation (D) of (DI ) is non-empty and bounded.
We saw in Section 7.3.2 that there are situations in which rounding the solution of a
continuous relaxation can yield a good solution to an optimization problem involving
integer variables. But generally, this is not the case.
Example 8.16
Consider the problem
max y2
2ky1 + y2 ≤ 2k ;
− 2ky1 + y2 ≤ 0 ;
− y2 ≤ 0 ;
y1 , y 2 integer,
where k ≥ 1 is a possibly large positive integer. It is easy to check that (y1 , y2 ) = (0, 0)
and (y1 , y2 ) = (1, 0) are both optimal solutions of this problem, but the optimal solution
of the continuous relaxation is (y1 , y2 ) = ( 21 , k). If we consider rounding y1 up or down
in the continuous solution, we do not get a feasible solution, and moreover we are quite
far from the optimal solutions.
Example 8.17
Consider the problem
Pm
max i=1 yi
yi + y` ≤ 1, for all 1 ≤ i < ` ≤ m;
−yi ≤ 0, for all 1 ≤ i ≤ m;
yi integer, for all 1 ≤ i ≤ m,
where m ≥ 3 is a possibly large positive integer. It is easy to check that each integer
optimal solution sets any single variable to one and the rest to zero, achieving objective
value 1. While the (unique) continuous optimal solution sets all variables to 12 , achiev-
ing objective value m
2 . We can see that the continuous solution is not closely related to
the integer solutions.
122 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

8.4 Branch-and-Bound

Next, we look at a rudimentary framework called branch-and-bound, which aims at

finding an optimal solution of (DI ), a linear-optimization problem having some integer
variables. We assume that (P), the dual of the continuous relaxation of (DI ), has a
feasible solution. Hence, even the continuous relaxation (D) of (DI ) is not unbounded.
Our algorithm maintains a list L of optimization problems that all have the gen-
eral form of (DI ). Keep in mind that problems on the list have integer variables. We
maintain a lower bound LB, satisfying LB ≤ z . Put simply, LB is the objective value of
the best (objective maximizing) feasible solution ỹLB of (DI ) that we have seen so far.
Initially, we set LB = −∞ , and we update it in an increasing fashion.
The algorithm maintains the key invariant for branch-and-bound:
Every feasible solution of the original problem (DI ) with greater objective value than LB is
feasible for a problem on the list.
We stop when the list is empty, and because of the property that we maintain, we
correctly conclude that the optimal value of (DI ) is LB when we do stop.
At a general step of the algorithm, we select and remove a problem (D̃I ) on the
list, and we solve its continuous relaxation (D̃). If this continuous relaxation is infeasi-
ble, then we do nothing further with this problem. Otherwise, we let ȳ be its optimal
solution, and we proceed as follows.

• If ȳ 0 b ≤ LB , then no feasible solution to the selected problem can have objective

value greater than LB, so we are done processing this selected problem.

• If ȳi is integer for all i ∈ I , then we have solved the selected problem. In this case,
if ȳ 0 b > LB , then we

– reset LB to ȳ 0 b ;
– reset ȳLB to ȳ .

• Finally, if ȳ 0 b > LB and ȳi is not integer for all i ∈ I , then (it is possible that this
selected problem has a feasible solution that is better than ȳLB , so) we

– select some i ∈ I such that ȳi is not integer;

8.4. BRANCH-AND-BOUND 123

– place two new “child” problems on the list, one with the constraint yi ≤ bȳi c
appended (the so-called down branch), and the other with the constraint
yi ≥ dȳi e appended (the so-called up branch).
(observe that every feasible solution to a parent is feasible for one of its chil-
dren, if it has children.)

Because the key invariant for branch-and-bound is maintained by the processing

rules, the following result is evident.

Theorem 8.18
Suppose that the original (P) is feasible. Then at termination of branch-and-bound,
we have LB= −∞ if (DI ) is infeasible or with ȳLB being an optimal solution of (DI ).

Finite termination. If the feasible region of the continuous relaxation (D) of (DI ) is
a bounded set, then we can guarantee finite termination. If we do not want to make
such an assumption, then if we assume that the data for the formulation is rational, it is
possible to bound the region that needs to be searched, and we can again assure finite
termination.

Solving continuous relaxations. Some remarks are in order regarding the solution
of continuous relaxations. Conceptually, we apply the Simplex Algorithm to the dual
(P̃) of the continuous relaxation (D̃) of a problem (D̃I ) selected and removed from
the list. At the outset, for an optimal basis β of (P̃), the optimal dual solution is given
by ȳ 0 := c0β A−1
β . If i ∈ I is chosen, such that ȳi is not an integer, then we replace the
selected problem (D̃I ) with one child having the additional constraint yi ≤ bȳi c (the
down branch) and another with the constraint yi ≥ dȳi e appended (the up branch).
Adding a constraint to (D̃) adds a variable to the standard-form problem (P̃). So, a
basis for (P̃) remains feasible after we introduce such a variable.

• The down branch: The constraint yi ≤ bȳi c , dualizes to a new variable xdown in
(P̃). The variable xdown has a new column Adown := ei and a cost coefficient of
cdown := bȳi c . Notice that the fact that ȳi is not an integer (and hence ȳ violates
yi ≤ bȳi c) translates into the fact that the reduced cost c̄down of xdown is c̄down =
cdown − ȳ 0 Adown = bȳi c − ȳi < 0 , so xdown is eligible to enter the basis.

• The up branch: Similarly, the constraint yi ≥ dȳi e , or equivalently −yi ≤ −dȳi e ,

dualizes to a new variable xup in (P̃). The variable xup has a new column Aup :=
−ei and a cost coefficient of cup := −dȳi e . Notice that the fact that ȳi is not an
integer (and hence ȳ violates yi ≥ dȳi e) translates into the fact that the reduced
cost c̄up of xup is c̄up = cup − ȳ 0 Aup = −dȳi e + ȳi < 0 , so xup is eligible to enter the
basis.
124 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

In either case, provided that we have kept the optimal basis for the (P̃) associated
with a problem (D̃ ), the Simplex Algorithm picks up on the (P̃) ˜ associated with a
I
˜ ) of that problem , with the new variable of the child’s (P̃)
child (D̃ ˜ entering the basis.
I
Notice that the (P̃) associated with a problem (D̃I ) on the list could be unbounded.
But this just implies that the problem (D̃I ) is infeasible.

Partially solving continuous relaxations. Notice that as the Simplex Algorithm is ap-
plied to the (P̃) associated with any problem (D̃I ) from the list, we generate a sequence
of non-increasing objective values, each one of which is an upperbound on the optimal
objective value of (D̃I ). That is, for any such (P̃), we start with the upperbound value
of its parent, and then we gradually decrease it, step-by-step of the Simplex Algorithm.
At any point in this process, if the objective value of the Simplex Algorithm falls at or
below the current LB, we can immediately terminate the Simplex Algorithm on such a
(P̃) — its optimal objective value will be no greater than LB — and conclude that the
optimal objective value of (D̃I ) is no greater than LB.

A global upper bound. As the algorithm progresses, if we let UBbetter be the maximum,
over all problems on the list, of the objective value of the continuous relaxations, then
any feasible solution ŷ with objective value greater than that LB satisfies ŷ 0 b ≤ UBbetter .
Of course, it may be that no optimal solution is feasible to any problem on the list —
for example if it happens that LB = z . But we can see that

z ≤ UB := max {UBbetter , LB} .

It may be useful to have UB at hand, because we can always stop the computation
early, say when UB − LB < τ , returning the feasible solution ȳLB , with the knowledge
0 b ≤ τ . But notice that we do not readily have the objective value of the con-
that z − ȳLB
tinuous relaxation for problems on the list — we only solve the continuous relaxation
for such a problem after it is selected (for processing). But, for every problem on the
list, we can simply keep track of the optimal objective value of its parent’s continuous
relaxation, and use that instead. Alternatively, we can re-organize our computations a
bit, solving continuous relaxations of subproblems before we put them on the list.

Selecting a subproblem from the list. Which subproblem from the list should we
process next?
• A strategy of last-in/first-out, known as diving, often results in good increases in
LB. To completely specify such a strategy, one would have to decide which of the
two children of a subproblem is put on the list last (i.e., the down branch or the
up branch). A good choice can affect the performance of this rule, and such a
good choice depends on the type of model being solved.

• A strategy of first-in/first-out is very bad. It can easily result in an explosion in the

size of the list of subproblems.

• A strategy of choosing a subproblem to branch on having objective value for its

continuous relaxation equal to UB, known as best bound, is a sound strategy for
8.4. BRANCH-AND-BOUND 125

seeking a decrease in UB. If such a rule is desired, then it is best to solve continuous
relaxations of subproblems before we put them on the list.

A hybrid strategy, doing mostly diving at the start (to get a reasonable value of LB)
and shifting more and more to best bound (to work on proving that LB is at or near the
optimal value) has rather robust performance.

Selecting a branching variable. Probably very many times, we will need to choose
an i ∈ I for which ȳi is fractional, in order to branch and create the child subproblems.
Which such i should we choose? Naïve rules such as choosing randomly or the so-called
most fractional rule of choosing an i that maximizes min{ȳi − bȳi c , dȳi e − ȳi } seem to
have rather poor performance. Better rules are based on estimates of how the objective
value of the children will change relative to the parent.

Using dual variables to bound the “other side” of an inequality. Our constraint
system y 0 A ≤ c0 can be viewed as y 0 Aj ≤ cj , for j = 1, 2, . . . , n ; that is, cj is an upper
bound on y 0 Aj . We may wonder if we can also derive lower bounds on y 0 Aj .

Theorem 8.19
Let LB be the objective value of any feasible solution of (DI ). Let x̄ be an optimal
solution of (P), and assume that x̄j > 0 for some j . Then

LB − c0 x̄
cj + ≤ y 0 Aj
x̄j

is satisfied by every optimal solution of (DI ).

Proof. We consider a parametric version of (DI ). For ∆j ∈ R, consider

z(∆j ) := max y 0 b
y 0 A ≤ c0 + ∆j e0j ;
(DI (∆j ))
y ∈ Rm ;
yi integer, for i ∈ I .

Let zR (∆j ) be defined the same way as z(∆j ), but with integrality relaxed. Using ideas
from Chapters 6 and 7, we can see that zR is a concave (piecewise-linear) function on
its domain, and x̄j is a subgradient of zR at ∆j = 0 . It follows that

z(∆j ) ≤ zR (∆j ) ≤ zR (0) + ∆j x̄j = c0 x̄ + ∆j x̄j .

So, we can observe that for

LB − c0 x̄
∆j < ,
x̄j
we will have z(∆j ) < LB. Therefore, every ŷ that is feasible for (DI (∆j )) with ∆j <
(LB − c0 x̄)/x̄j will have ŷ 0 b < LB . So such a ŷ cannot be optimal for (DI ). t
u
126 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

It is interesting to consider two special cases of Theorem 8.19:

Corollary 8.20 (Variable fixing)

Let LB be the objective value of any feasible solution of (DI ). Let x̄ be an optimal
solution of (P). Assume that x̄j > 0 is the optimal dual variable for a constraint of the
form: yk ≤ 1 (or −yk ≤ 0) . If c0 x̄ − LB < x̄j , then yk = 1 (respectively, yk = 0) is
satisfied by every optimal solution of (DI ).

Because of Exercise 5.2, this is known as reduced-cost fixing.

8.5 Cutting Planes

This section is adapted from material in [2] and [4]. In fact, those papers were devel-
oped to achieve versions of Gomory cutting-plane algorithms (with finiteness proofs)
that would mesh with our column-generation treatment of many topics in this book
(i.e., cutting stock, decomposition, and branch-and-bound).

8.5.1 Pure
In this section, we assume that all yi variables are constrained to be integer. That is,
I = {1, 2, . . . , m}
We can choose any non-negative w ∈ Rn , and we see that

w ≥ 0 and y 0 A ≤ c0 =⇒ y 0 (Aw) ≤ c0 w .

Note that this inequality is valid for all solutions of y 0 A ≤ c0 , integer or not. Next, if Aw
is integer, we can exploit the integrality of y . We see that

Aw ∈ Zm , y ∈ Zm =⇒ y 0 (Aw) ≤ bc0 wc ,

for all integer solutions of y 0 A ≤ c0 .

The inequality y 0 (Aw) ≤ bc0 wc is called a Chvátal-Gomory cut. The condition Aw ∈
Z may seem a little awkward, but usually we have that A is integer, so we can get
m

Aw ∈ Zm by then just choosing w ∈ Zn . In fact, for the remained of this section, we will
assume that A and c are integer.
Of course, it is by no means clear how to choose appropriate w, and this is critical
for getting useful inequalities. We should also bear in mind that there are examples
for which Chvátal-Gomory are rather ineffectual. Trying to apply such cuts to Example
8.16 reveals that infeasible integer points can “guard” Chvátal-Gomory cuts from getting
close to any feasible integer points.
We would like to develop a concrete algorithmic scheme for generating Chvátal-
Gomory cuts. We will do this via basic solutions. Let β be any basis for P. The asso-
ciated dual basic solution (for the continuous relaxation (D)) is ȳ 0 := c0β A−1 β . Suppose
that ȳi is not an integer. Our goal is to derive a valid cut for (DI ) that is violated by ȳ.
8.5. CUTTING PLANES 127

Let
b̃ := ei + Aβ r,

where r ∈ Zm , and, as usual, ei denotes the i-th standard unit vector in Rm . Note that
by construction, b̃ ∈ Zm .

Theorem 8.21
ȳ 0 b̃ is not an integer, and so y 0 b̃ ≤ bȳ 0 b̃c cuts off ȳ.

Proof. ȳ 0 b̃ = ȳ 0 (ei + Aβ r) = ȳi + (c0β A−1 ȳi + c0β r ∈

β )Aβ r = |{z} / Z. t
u
|{z}
∈Z
/ ∈Z

At this point, we have an inequality y 0 b̃ ≤ bȳ 0 b̃c which cuts off ȳ, but we have not
established its validity for (DI ).
Let H·i := A−1 −1
β e , the i-th column of Aβ . Now let
i

w := H·i + r.

Clearly we can choose r ∈ Zm so that w ≥ 0; we simply choose r ∈ Zm so that

rk ≥ −bhki c, for k = 1, . . . , m. (∗≥ )

Theorem 8.22
Choosing r ∈ Zm satisfying (∗≥ ), we have that y 0 b̃ ≤ bȳ 0 b̃c is valid for (DI ).

Proof. Because w ≥ 0 and y 0 A ≤ c0 , we have the validity of

y 0 Aβ (A−1 i 0 −1 i
β e + r) ≤ cβ (Aβ e + r),

even for the continuous relaxation (D) of (DI ). Simplifying this, we have

y 0 (ei + Aβ r) ≤ ȳi + c0β r.

The left-hand side is clearly y 0 b̃, and the right-hand side is

ȳi + c0β r = ȳi + ȳ 0 Aβ r = ȳ 0 (ei + Aβ r) = ȳ 0 b̃.

So we have that y 0 b̃ ≤ ȳ 0 b̃ is valid even for (D). Finally, observing that b̃ ∈ Zm and y is
constrained to be in Zm for (DI ), we can round down the right-hand side and get the
result. t
u
128 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

So, given any non-integer basic dual solution ȳ, we have a way to produce a valid
inequality for (DI ) that cuts it off. This cut for (DI ) is used as a column for (P): the
column is b̃ with objective coefficient bȳ 0 b̃c. Taking β to be an optimal basis for (P), the
new variable corresponding to this column is the unique variable eligible to enter the
basis in the context of the primal simplex algorithm applied to (P) — the reduced cost
is precisely
bȳ 0 b̃c − ȳ 0 b̃ < 0.
The new column for A is b̃ which is integer. The new objective coefficient for c is bȳ 0 b̃c
which is an integer. So the original assumption that A and c are integer is maintained,
and we can repeat. In this way, we get a legitimate cutting-plane framework for (DI ) —
though we emphasize that we do our computations as column generation with respect
to (P).
There is clearly a lot of flexibility in how r can be chosen. Next, we demonstrate that
in a very concrete sense, it is always best to choose a minimal r ∈ Zm satisfying (∗≥ ).

Theorem 8.23
Let r ∈ Zm be defined by

rk = −bhki c, for k = 1, . . . , m, (∗= )

and suppose that r̂ ∈ Zm satisfies (∗≥ ) and r ≤ r̂. Then the cut determined by r
dominates the cut determined by r̂.

Proof. It is easy to check that our cut can be re-expressed as

yi ≤ bȳi c + c0β − y 0 Aβ r.

Noting that c0β − y 0 Aβ ≥ 0 for all y that are feasible for (D), we see that the strongest
inequality is obtained by choosing r ∈ Zm to be minimal. t
u

Example 8.24
Let
   
7 8 −1 1 3 26
A= , b=
5 6 −1 2 1 19


and c0 = 126 141 −10 5 67 .
So, the integer program (DI ) which we seek to solve is defined by five inequalities in
the two variables y1 and y2 . For the basis of (P), β = (1, 2), we have
   
7 8 −1 3 −4
Aβ = , and hence Aβ = .
5 6 −5/2 7/2

It is easy to check that for this choice of basis, we have

 
2
x̄β = ,
3/2
8.5. CUTTING PLANES 129

and for the non-basis η = {3, 4, 5, 6}, we have c̄0η = 5 1/2 1 , which are both non-
negative, and so this basis is optimal for (P). The associated dual basic solution is


ȳ 0 = 51/2 −21/2 , and the objective value is z = 463 1/2.

Because both ȳ1 and ȳ2 are not integer, we can derive a cut for (DI ) from either.
Recalling the procedure, for any fraction ȳi , we start with the i-th column H·i of H :=
A−1
β , and we get a new A·j := e + Aβ r. Throughout we will choose r via (∗= ). So we
i

have,
          
3 −3 1 7 8 −3 4
H·1 = ⇒r= ⇒ b̃ = + = =: A·6
−5/2 3 0 5 6 3 3

          
−4 4 0 7 8 4 4
H·2 = ⇒r= ⇒ b̃ = + = .
7/2 −3 1 5 6 −3 3

In fact, for this iteration of this example, we get the same cut for either choice of i. To
calculate the right-hand side of the cut, we have
 
0

4
ȳ b̃ = 51/2 −21/2 = 70 1/2,
3

so the cut for (DI ) is

4y1 + 3y2 ≤ 70.
Now, we do our simplex-method calculations with respect to (P). The new column
for (P) is A·6 (above) with objective coefficient c6 := 70.
Following the usual ratio test, when index 6 enters the basis, index 2 leaves the basis,
and so the new basis is β = (1, 6), with
 
7 4
Aβ = ,
5 3

with objective value 462, a decrease. At this point, index 5 has a negative reduced cost,
and index 1 leaves the basis. So we now have β = (5, 6), which turns out to be optimal
for the current (P). We have


ȳ 0 = 131/5 −58/5 , and the objective value is z = 460 4/5.

We observe that the objective function has decreased, but unfortunately both ȳ1 and ȳ1
are not integers. So we must continue. We have
   
3 4 −1 3/5 −4/5
Aβ = , and hence Aβ = .
1 3 −1/5 3/5

We observe that the objective function has decreased, but because both ȳ1 and ȳ2
are not integers, we can again derive a cut for (DI ) from either. We calculate
          
3/5 0 1 3 4 0 5
H·1 = ⇒r= ⇒ b̃ = + = =: A·7
−1/5 1 0 1 3 1 3
130 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

          
−4/5 1 0 3 4 1 3
H·2 = ⇒r= ⇒ b̃ = + = =: A·8 .
3/5 0 1 1 3 0 2

Correspondingly, we have ȳ 0 A·7 = 96 1/5 and ȳ 0 A·8 = 55 2/5, giving us c7 := 96 and

c8 := 55. So, we have two possible cuts for (DI ):

5y1 + 3y2 ≤ 96 and 3y1 + 2y2 ≤ 55.

Choosing to incorporate both as columns for (P), and letting index 8 enter the basis,
index 5 leaves (according to the ratio test), and it turns out that we reach an optimal
basis β = (8, 6) after this single pivot. At this point, we have


ȳ 0 = 25 −10 , and the objective value is z = 460.

Not only has the objective decreased, but now all of the ȳi are integers, so we have an
optimal solution for (DI ).

8.5.2 Mixed
In this section, we no longer assume that all yi variables are constrained to be integer.
That is, we only assume that non-empty I ⊂ {1, 2, . . . , m}. The cuts from the previous
section cannot be guaranteed to be valid, so we start anew.
Let β be any basis partition for(P), and let ȳ be the associated dual basic solution.
Suppose that ȳi ∈/ Z, for some i ∈ I. We aim to find a cut, valid for (DI ) and violated
by ȳ.
Let
b̃1 := ei + Aβ r,

and r ∈ Rm will be determined later. We will accumulate the conditions we need to

impose on r, as we go.
Let w1 be the basic solution associated with the basis β and the “right-hand side” b̃1 .
So wβ1 = h·i + r, where h·i is defined as the i-th column of A−1
β , and wη = 0. Choosing
1

r ≥ −h·i , we can make w1 ≥ 0. Moreover, c0 w1 = c0β (h·i + r) = c0β h·i + c0β r = ȳi + c0β r,
so because we assume that ȳi ∈ / Z, we can choose r ∈ Zm , and we have that c0 w1 ∈ / Z.
Next, let
b̃2 := Aβ r.

Let w2 be the basic solution associated with the basis β and the “right-hand side” b̃2 .
So, now further choosing r ≥ 0, we have wβ2 = r ≥ 0, wη2 = 0, and c0 w2 = c0β r.
So, we choose r ∈ Zm so that:

rk ≥ max{−bhki c, 0}, for k = 1, . . . , m, (∗≥+ )

8.5. CUTTING PLANES 131

Because we have chosen w1 and w2 to be non-negative, forming (y 0 A)wl ≤ c0 wl , for

l = 1, 2, we get a pair of valid inequalities for D. They have the form y 0 b̃l ≤ c0 wl , for
l = 1, 2. Let αj0 denote the j-th row of Aβ . Then our inequalities have the form:
X
(1 + αi0 r)yi + (αj0 r)yj ≤ ȳi + ȳ 0 Aβ r, (I1)
j:j6=i
X
(αi0 r)yi + (αj0 r)yj ≤ ȳ 0 Aβ r. (I2)
j:j6=i
P
Now, defining z := j:j6=i (αj r)yj ,
0 we have the following inequalities in the two vari-
ables yi and z:

slope
(1 + αi0 r)yi + z ≤ ȳi + ȳ 0 Aβ r −1/(1 + αi0 r) (B1)
(αi0 r)yi 0
+ z ≤ ȳ Aβ r −1/αi0 r (B2)

Note that the intersection point (yi∗ , z ∗ ) of the lines associated with
P these inequalities
(subtract the second equation from the first) has yi∗ = ȳi and z ∗ = j:j6=i (αj0 r)ȳj . Also,
the “slopes” indicated regard yi as the ordinate and z as the abscissa.
Bearing in mind that we choose r ∈ Zm and that A is assumed to be integer, we have
that αi0 r ∈ Z. There are now two cases to consider:

• αi0 r ≥ 0, in which case the first line has negative slope and the second line has
more negative slope (or infinite αi0 r = 0);

• αi0 r ≤ −1, in which case the second line has positive slope and the first line has
more positive slope (or infinite αi0 r = −1).

See Figures 8.4 and 8.5.

In both cases, we are interested in the point (z 1 , yi1 ) where the first line intersects
the line yi = bȳi c + 1 and the point (z 2 , yi2 ) where the second line intersects the line
yi = bȳi c.
We can check that

z 1 = ȳi + ȳ 0 Aβ r − (1 + αi0 r) (bȳi c + 1) ,

z 2 = ȳ 0 Aβ r − (αi0 r)bȳi c.

Subtracting, we have
z1 − z2 = (ȳi − bȳi c) −(1 + αi0 r ),
| {z } |{z}
∈(0,1) ∈Z

so we see that: z 1 < z 2 precisely when αi0 r ≥ 0; z 2 < z 1 precisely when αi0 r ≤ −1.
Moreover, the slope of the line through the pair of points (z 1 , yi1 ) and (z 2 , yi2 ) is just

1 1
= .
z1 −z 2 (ȳi − bȳi c) − (1 + αi0 r)
132 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Figure 8.4: (F-BMI) cut when αi0 r ≥ 0

yi (F-BMI)

(B2)

(B1)

Figure 8.5: (F-BMI) cut when αi0 r ≤ −1

z

(B2)

(B1)
yi
(F-BMI)

We now define the inequality

(ȳi − bȳi c) − (1 + αi0 r) (yi − bȳi c) ≥ z − ȳ 0 Aβ r + (αi0 r)bȳi c,

which has the more convenient form

(1 + αi0 r) − (ȳi − bȳi c) yi + z ≤ ȳ 0 Aβ r − (ȳi − bȳi c − 1) bȳi c. (F-BMI)

By construction, we have the following two results.

Lemma 8.25
(F-BMI) is satisfied at equality by both of the points (z 1 , yi1 ) and (z 2 , yi2 ).
8.5. CUTTING PLANES 133

Lemma 8.26
(F-BMI) is valid for
 
(yi , z) ∈ R2 : (B1), yi ≥ dȳi e ∪ (yi , z) ∈ R2 : (B2), yi ≤ bȳi c .

Lemma 8.27
(F-BMI) is violated by the point (yi∗ , z ∗ ).

Proof. Plugging (yi∗ , z ∗ ) into (F-BMI), and making some if-and-only-if manipulations,
we obtain
(ȳi − bȳi c − 1) (ȳi − bȳi c) ≥ 0,
which is not satisfied. t
u

Finally, translating (F-BMI) back to the original variables y ∈ Rm , we get

X
(1 + αi0 r) − (ȳi − bȳi c) yi + (αj0 r)yj ≤ ȳ 0 Aβ r − (ȳi − bȳi c − 1) bȳi c,
j:j6=i

or,
− (ȳi − bȳi c − 1) yi + y 0 Aβ r ≤ ȳ 0 Aβ r − (ȳi − bȳi c − 1) bȳi c,
which, finally has the convenient form

y 0 (Aβ r − (ȳi − bȳi c − 1) ei ) ≤ c0β r − (ȳi − bȳi c − 1) bȳi c. (F-GMI)

We immediately have:

Theorem 8.28
(F-GMI) is violated by the point ȳ.

Finally, we have:

Theorem 8.29
(F-GMI) is valid for the following relaxation of the feasible region of (D):

{y ∈ Rm : y 0 Aβ ≤ c0β , yi ≥ dȳi e} ∪ {y ∈ Rm : y 0 Aβ ≤ c0β , yi ≤ bȳi c}

134 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Proof. The proof, maybe obvious, is by a simple disjunctive argument. We will argue
that (F-BMI) is valid for both S1 := {y ∈ Rm : y 0 Aβ ≤ c0β , −yi ≤ −bȳi c − 1} and
S2 := {y ∈ Rm : y 0 Aβ ≤ c0β , yi ≤ bȳi c}.
The inequality (F-BMI) is simply the sum of (B1) and the scalar ȳi − bȳi c times
−yi ≤ −bȳi c − 1. It follows than that taking (I1) plus ȳi − bȳi c times −yi ≤ −bȳi c − 1,
we get an inequality equivalent to (F-GMI).
Similarly, it is easy to check that the inequality (F-BMI) is simply (B2) plus 1 − (ȳi −
bȳi c) times yi ≤ bȳi c. It follows than that taking (I2) plus 1 − (ȳi − bȳi c) times yi ≤ bȳi c,
we also get an inequality equivalent to (F-GMI). t
u

In our algorithm, we append columns to (P), rather than cuts to (D). The column
for (P) corresponding to (F-GMI) is
Aβ r − (ȳi − bȳi c − 1) ei ,
and the associated cost coefficient is
c0β r − (ȳi − bȳi c − 1) bȳi c.

So A−1
β times the column is
r − (ȳi − bȳi c − 1) h·i .
Agreeing with what we calculated in Proposition 8.27, we have the following result.

Proposition 8.30
The reduced cost of the column for (P) corresponding to (F-GMI) is

(ȳi − bȳi c − 1) (ȳi − bȳi c) < 0.

Proof.
c0β r − (ȳi − bȳi c − 1) bȳi c − c0β (r − (ȳi − bȳi c − 1) h·i )


= (ȳi − bȳi c − 1) c0β h·i − bȳi c
= (ȳi − bȳi c − 1) (ȳi − bȳi c) .
t
u

Next, we come to the choice of r.

Theorem 8.31
Let r ∈ Zm be defined by

rk = max{0, −bhki c}, for k = 1, 2, . . . , m, (∗=+ )

and suppose that r̂ ∈ Zm satisfies (∗≥+ ) and r ≤ r̂. Then the cut determined by r
dominates the cut determined by r̂.
8.6. EXERCISES 135

Proof. We simply rewrite (F-GMI) as

(c0β − y 0 Aβ )r ≥ (ȳi − bȳi c − 1) (bȳi c − yi ).

Observing that c0β − y 0 Aβ ≥ 0 for y that are feasible for (D), we see that the tightest
inequality of this type, satisfying (∗≥+ ), arises by choosing a minimal r. The result
follows. t
u

8.5.3 Finite termination

Making a version of our Gomory cutting-plane scheme that we can prove is finitely
terminating is rather technical. Though it can be done in essentially the same manner
for both pure and mixed cases. We need to treat the objective-function value as an
additional variable (numbered first), employ the Simplex Algorithm adapted to the -
perturbed problem, always choose the least-index i ∈ I having ȳi ∈
/ Z and choose r via
(∗= ) or (∗=+ ) as appropriate to generate the Gomory cuts. Details can be found in [2]
and [4].

8.5.4 Branch-and-Cut
State-of-the-art algorithms for (mixed-)integer linear optimization combine cuts with
branch-and-bound. There are a lot of software design and tuning issues that make this
work successfully.

8.6 Exercises
Exercise 8.1 (Task scheduling, continued)
Consider again the “task scheduling” Exercise 2.5. Take the dual of the linear-optimization
problem that you formulated. Explain how this dual can be interpreted as a kind of net-
work problem. Using AMPL, solve the dual of the example that you created for Exercise
2.5 and interpret the solution.

Exercise 8.2 (Pivoting and total unimodularity)

A pivot in an m × n matrix A means choosing a row i and column j with aij 6= 0 ,

a
subtracting akj ij
times row i from all other rows k(6= i) , and then dividing row i by aij .
Note that after the pivot, column j becomes the i-th standard-unit column. Prove that
if A is TU, then it is TU after a pivot.
136 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Exercise 8.3 (Comparing formulations for a toy problem)

Consider the systems:

S1 : 2x1 + 2x2 + x3 + x4 ≤ 2 ;
xj ≤ 1;
−xj ≤ 0.

S2 : x1 + x2 + x3 ≤ 1 ;
x1 + x2 + x4 ≤ 1 ;
−xj ≤ 0.

S3 : x1 + x2 ≤ 1 ;
x1 + x3 ≤ 1 ;
x1 + x4 ≤ 1 ;
x2 + x3 ≤ 1 ;
x2 + x4 ≤ 1 ;
−xj ≤ 0.

Notice that each system has precisely the same set of integer solutions. In fact, each
system chooses, via its feasible integer (0/1) solutions, the “vertex packings” of the
following graph.

A vertex packing of a graph is a set of vertices with no edges between them. For this
particular graph we can see that the packings are: ∅, {1}, {2}, {3}, {4}, {3, 4}.
Compare the feasible regions Si of the continuous relaxations, for each pair of these
systems. Specifically, for each choice of pair i 6= j , demonstrate whether or not the
solution set of Si is contained in the solution set of Sj . HINT: To prove that the solution
set of Si is contained in the solution set of Sj , it suffices to demonstrate that every
inequality of Sj is a non-negative linear combination of the inequalities of Si . To prove
that the solution set of Si is not contained in the solution set of Sj , it suffices to give a
solution of Si that is not a solution of Sj .

Exercise 8.4 (Comparing facility-location formulations)

We have seen two formulations of the forcing constraints for the uncapacitated facility-
location problem. We have a choice of the mn constraints: P −yi + xij ≤ 0 , for i =
1, . . . , m and j = 1, . . . , n , or the m constraints: −nyi + nj=1 xij ≤ 0 , for i = 1, . . . , m .
8.6. EXERCISES 137

Which formulation is stronger? That is, compare (both computationally and analyti-
cally) the strength of the two associated continuous relaxations (i.e., when we relax
yi ∈ {0, 1} to 0 ≤ yi ≤ 1 , for i = 1, . . . , m). In Appendix A.4, there is AMPL code for
trying computational experiments.
Exercise 8.5 (Comparing piecewise-linear formulations)
We have seen that the adjacency condition for piecewise-linear univariate functions can
be modeled by
λ1 ≤ y1 ;
λj ≤ yj−1 + yj , for j = 2, . . . , n − 1 ;
λn ≤ yn−1 .
An alternative formulation is
j
X j+1
X
yi ≤ λi , for j = 1, . . . , n − 2 ;
i=1 i=1
n−1
X Xn
yi ≤ λi , for j = 2, . . . , n − 1 .
i=j i=j

Explain why this alternative formulation is valid, and compare its strength to the orig-
inal formulation, when we relax yi ∈ {0, 1} to 0 ≤ yi ≤ 1 , for i = 1, P. n. . , n − 1 . (Note
that
Pn−1 for both formulations, we require λi ≥ 0, for i = 1, . . . , n , i=1 λi = 1 , and
i=1 i y = 1).
Exercise 8.6 (Variable fixing)
Prove Corollary 8.20.
Exercise 8.7 (Gomory cuts)
Prove that we need at least k Chvátal-Gomory cuts to solve Example 8.16.
Exercise 8.8 (Solve pure integer problems using Gomory cuts)
Extend what you did for Exercise 4.1 to now solve integer problems using Gomory cuts.
In Appendix A.3, there is a MATLAB script pure_gomory.m for generating Gomory cuts.
Using this additional primitives only, extend your script from Exercise 4.1 to solve pure
integer problems using Gomory cuts. Be sure to read in the data using pivot_setup.m.
As before, do not worry about degeneracy/anti-cycling. Make some small examples to
fully illustrate your code.
Exercise 8.9 (Make amends)
Find an interesting applied problem, model it as a pure- or mixed- integer linear-optimization
problem, and test your model with AMPL.
138 CHAPTER 8. INTEGER-LINEAR OPTIMIZATION

Credit will be given for deft modeling, sophisticated use of AMPL, testing on meaningfully-
large instances, and insightful analysis. Try to play with CPLEX integer solver options
(they can be set through AMPL) to get better behavior of the solver.
Your grade on this problem will replace your grades on up to 6 homework problems
(i.e., up to 6 homework problems on which you have lower grades than you get on this
one). I will not consider any re-grades on this one! If you already have all or mostly A’s (or
not), do a good job on this one because you want to impress me, and because you are
ambitious, and because this problem is what we have been working towards all during
the course, and because you should always finish strong.
8.6. EXERCISES 139

Take rest
Appendices

141
 143

A.1 LATEX template

 LATEX TEMPLATE

Your actual name (youremail@umich.edu), Another actual name (anotheremail@umich.edu)

This template can serve as a starting point for learning LATEX. You may download MiKTeX from
miktex.org to get started. Look at the source file for this document (in Section 5) to see how to get
all of the effects demonstrated.

1. This is the first section where we make some lists

It is easy to make enumerated lists:
(1) This is the first item
(2) Here is the second
And even enumerated sublists:
(1) This is the first item
(2) Here is the second with a sublist
(a) first sublist item
(b) and here is the second

2. Here is a second section where we typeset some math

Pn
You can typeset math inline, like j=1 aij xj , by just enclosing the math in dollar signs.
But if you want to display the math, then you do it like this:
n
X
aij xj ∀ i = 1, . . . , m.
j=1
And here is a matrix:  
1 π 2 21 ν
 6.2 r 2 4 5 
|y 0 | R R r R̂
Here is an equation array, with the equal signs nicely aligned:
n
X
(2.1) xj = 5
j=1
Xn
(2.2) yj = 7
j=1
X
(2.3) xj = 29
j∈S

The equations are automatically numbered, like x.y, where x is the section number and y is the y-th
equation in section x. By tagging the equations with labels, we can refer to them later, like (2.3) and (2.1).
Theorem 2.1. This is my favorite theorem.
Proof. Unfortunately, the space here does not allow for including my ingenious proof of Theorem 2.1. 

3. Here is how I typset a standard-form linear-optimization problem

min c0 x
(P) Ax = b;
x ≥ 0.
Notice that in this example, there are 4 columns separated by 3 &’s. The ‘rrcl’ organizes justification
within a column. Of course, one can make more columns.

Date: December 20, 2016.

1
2 LATEX TEMPLATE

4. Graphics
This is how to include and refer to Figure 1 with pdfLaTeX.

Figure 1. Another duality

5. The LATEX commands to produce this document

Look at the LATEX commands in this section to see how each of the elements of this document was produced.
Also, this section serves to show how text files (e.g., programs) can be included in a LATEX document verbatim.
% LaTeX_Template.tex // J. Lee
%
% ----------------------------------------------------------------
% AMS-LaTeX ************************************************
% **** -----------------------------------------------------------
\documentclass{amsart}
\usepackage{graphicx,amsmath,amsthm}
\usepackage{hyperref}
\usepackage{verbatim}
\usepackage[a4paper,text={16.5cm,25.2cm},centering]{geometry}
% ----------------------------------------------------------------
\vfuzz2pt % Don’t report over-full v-boxes if over-edge is small
\hfuzz2pt % Don’t report over-full h-boxes if over-edge is small
% THEOREMS -------------------------------------------------------
\newtheorem{thm}{Theorem}[section]
\newtheorem{cor}[thm]{Corollary}
\newtheorem{lem}[thm]{Lemma}
\newtheorem{prop}[thm]{Proposition}
\theoremstyle{definition}
\newtheorem{defn}[thm]{Definition}
\theoremstyle{remark}
\newtheorem{rem}[thm]{Remark}
\numberwithin{equation}{section}
% MATH -----------------------------------------------------------
\newcommand{\Real}{\mathbb R}
\newcommand{\eps}{\varepsilon}
\newcommand{\To}{\longrightarrow}
\newcommand{\BX}{\mathbf{B}(X)}
\newcommand{\A}{\mathcal{A}}
% ----------------------------------------------------------------
\begin{document}
 LATEX TEMPLATE 3

\title{\LaTeX~ Template}

\date{\today}

\maketitle

\href{mailto:youremail@umich.edu,anotheremail@umich.edu}
{Your actual name (youremail@umich.edu),
Another actual name (anotheremail@umich.edu)}

%
%\medskip
%
%(this identifies your work and it \emph{greatly} help’s me in returning homework to you by email
%---- just plug in the appropriate replacements in the \LaTeX~ source; then when I click on the
%hyperlink above, my email program opens up starting a message to you)

\bigskip

% ----------------------------------------------------------------

This template can serve as a starting point for learning \LaTeX. You may download MiKTeX from
{\tt miktex.org}
to get started. Look at the source file for this
document (in Section \ref{sec:appendix})
to see how to get all of the effects demonstrated.

\section{This is the first section where we make some lists}

It is easy to make enumerated lists:

\begin{enumerate}
\item This is the first item
\item Here is the second
\end{enumerate}

And even enumerated sublists:

\begin{enumerate}
\item This is the first item
\item Here is the second with a sublist
\begin{enumerate}
\item first sublist item
\item and here is the second
\end{enumerate}
\end{enumerate}

\section{Here is a second section where we typeset some math}

You can typeset math inline, like $\sum_{j=1}^n a_{ij} x_j$, by just enclosing the math in dollar signs.

But if you want to \emph{display} the math, then you do it like this:

\[
\sum_{j=1}^n a_{ij} x_j~ \forall~ i=1,\ldots,m.
\]

And here is a matrix:

\[
\left(
\begin{array}{ccccc}
1 & \pi & 2& \frac{1}{2} & \nu \\
6.2 & r & 2 & 4 & 5 \\
|y’| & \mathcal{R} & \mathbb{R} & \underbar{r} & \hat{R} \\
\end{array}
\right)
\]
4 LATEX TEMPLATE

Here is an equation array, with the equal signs nicely aligned:

\begin{eqnarray}
\sum_{j=1}^n x_j &=& 5 \label{E1} \\
\sum_{j=1}^n y_j &=& 7 \label{E7} \\
\sum_{j\in S} x_j &=& 29 \label{E4}
\end{eqnarray}

The equations are automatically numbered, like $x.y$, where

$x$ is the section number and $y$ is the $y$-th equation in section $x$.
By tagging the equations
with labels, we can refer to them later, like (\ref{E4}) and (\ref{E1}).

\begin{thm}\label{Favorite}
This is my favorite theorem.
\end{thm}
\begin{proof}
Unfortunately, the space here does not allow for including my ingenious proof
of Theorem \ref{Favorite}.
\end{proof}

\section{Here is how I typset a standard-form linear-optimization problem}

\[
\tag{P}
\begin{array}{rrcl}
\min & c’x & & \\
& Ax & = & b~; \\
& x & \geq & \mathbf{0}~.
\end{array}
\]

Notice that in this example, there are 4 columns separated by 3 \&’s.

The ‘rrcl’ organizes justification within a column.
Of course, one can make more columns.

\section{Graphics}

This is how to include and refer to Figure \ref{nameoffigure} with pdfLaTeX.

\begin{figure}[h!!]
\includegraphics[width=0.5\textwidth]{yinyang.jpg}
\caption{Another duality}\label{nameoffigure}
\end{figure}

\section{The \LaTeX~ commands to produce this document}

\label{sec:appendix}

Look at the \LaTeX~ commands in this section to see how each of the elements
of this document was produced. Also, this section serves to show
how text files (e.g., programs) can be included in a \LaTeX~ document verbatim.

\bigskip

\hrule

\small
\verbatiminput{LaTeX_Template.tex}
\normalsize

\hrule

\bigskip

% ----------------------------------------------------------------
 LATEX TEMPLATE 5

\end{document}
% ----------------------------------------------------------------
 149

A.2 MATLAB for deconstructing the Simplex Algorithm

• pivot_setup.m : Script sets up all of the data structures and executes the script
pivot_input.m (or a user-created .m file — file name input at MATLAB command
prompt) to read in the input. Must execute pivot_algebra.m after setup.

• pivot_input.m : Script holds the default input which (or alternative user input
file) is executed by the script pivot_setup.m . Do not run pivot_input.m from a
MATLAB command prompt.
• pivot_algebra.m : Script to carry out the linear algebra after setup and after a
swap.

• pivot_direction.m : Function pivot_direction(j) displays the basic direction

associated with the current basis and increasing the j-th non-basic variable (up
from zero). Should only be used after the script pivot_algebra.m is executed.

• pivot_ratios.m : Used to determine a leaving variable to maintain feasibility

when the j-th non-basic variable is increased (from zero). Function pivot_ratios(j)
calculates the ratios associated with the current basis and increasing the j-th non-
basic variable (up from zero). Should only be used after the script pivot_algebra.m
is executed.

• pivot_swap.m : Function pivot_swap(j,i) exchanges the j-th non-basic variable

with the i-th basic variable. Must execute the script pivot_algebra.m after a
swap.
150

• pivot_plot.m : If n − m = 2 , script produces a plot in the space of the (two)

non-basic variables. Feasible basic solutions map to green points and infeasible
ones map to red points. If the feasible region is bounded and there are at least
three extreme points, then the feasible region is shaded with cyan. If there are at
least three extreme points and there is no basic feasible ray relative to the current
basis, then the convex hull of the basic feasible solutions is still shaded with cyan.

% pivot_setup.m // Jon Lee

% set up a pivoting example

clear all;

global A b c m n beta eta zbar ratios

try
reply = input('NAME of NAME.m file holding input? [pivot_input]: ', 's');
if isempty(reply)
reply = 'pivot_input';
end
catch
reply = 'pivot_input'; % need catch for execution on MathWorks Cloud
end
eval(reply);

if (size(b) ~= m)
display('size(b) does not match number of rows of A')
return
end
if (size(c) ~= n)
display('size(c) does not match number of columns of A')
return
end
if(size(setdiff(beta,1:n)) > 0)
display('beta has elements not in 1,2,...,n')
return
end
if(size(setdiff(eta,1:n)) > 0)
display('eta has elements not in 1,2,...,n')
return
end
if (size(beta) ~= m)
display('size(beta) does not match number of rows of A')
return
end
if (size(eta) ~= n−m)
display('size(eta) does not match number of cols minus number of rows of A')
return
end

zbar = sym(zeros(n,1));
ratios = sym(zeros(m,1));
 151

display('Available quantites: A, b, c, m, n, beta, eta');

display('Seems like a good time to run pivot_algebra');

% pivot_input.m // Jon Lee

% data for pivoting example

A = [1 2 1 0 0 0; 3 1 0 1 0 0; 3/2 3/2 0 0 1 0; 0 1 0 0 0 1];

A= sym(A);
c = [6 7 −2 0 4 9/2]';
c=sym(c);
b = [7 9 6 33/10]';
b=sym(b);
beta = [1,2,4,6];
[m,n] = size(A);
eta = setdiff(1:n,beta); % lazy eta initialization

% pivot_input2.m // Jon Lee

% data for pivoting example
%
% this is really the same problem as pivot_input.m, but
% − the rhs is symbolically perturbed, so we can easily do lex pivots.
%
syms t real;
A = [1 2 1 0 0 0; 3 1 0 1 0 0; 3/2 3/2 0 0 1 0; 0 1 0 0 0 1];
A= sym(A);
c = [6 7 −2 0 4 9/2]';
c=sym(c);
b = [7 9 6 33/10]';
b=sym(b);
beta = [1,2,4,6];
[m,n] = size(A);
eta = setdiff(1:n,beta); % lazy eta initialization
b = b + A(:,beta)*[t t^2 t^3 t^4]'; % lex feasible if beta is feasible

% pivot_input3.m // Jon Lee

% data for pivoting example

syms theta gamma pi;

syms t real;

theta= 2*pi/5;
gamma = cot(theta);

R= [ cos(theta) −sin(theta); sin(theta) cos(theta)];

A1 = [1 0]';
A2 = [0 gamma]';
A = [A1 A2 R*A1 R*A2 R^2*A1 R^2*A2 R^3*A1 R^3*A2 R^4*A1 R^4*A2];
A= sym(A);
c = ( ones(1,10) − [1 1/gamma]*A )';
c=sym(c);
152

b = [0 0]';
b=sym(b);
beta = [1,2];
[m,n] = size(A);
eta = setdiff(1:n,beta); % lazy eta initialization
%
% uncomment next line to lexically perturb the rhs
b = b + A(:,beta)*[t t^2]'; % lex feasible if beta is feasible

% pivot_algebra.m // Jon Lee

% do the linear algebra

global A_beta Abar_eta c_beta ybar xbar_beta xbar c_eta cbar_eta

A_beta = A(:,beta);
% if (rank(A_beta,0.0001) < size(A_beta,1))
% display('error: A_beta not invertible')
% return;
% end;
A_eta = A(:,eta);
c_beta = c(beta);
c_eta = c(eta);
ybar = linsolve(A_beta',c_beta);
xbar = sym(zeros(n,1));
xbar_beta = linsolve(A_beta,b);
xbar(beta) = xbar_beta;
xbar_eta = sym(zeros(n−m,1));
xbar(eta) = xbar_eta;

Abar_eta = linsolve(A_beta,A_eta);

% if (norm(A * xbar − A_beta * xbar_beta + A_eta * xbar_eta) > 0.00001)

% display('error: something is very wrong')
% return
% end

cbar_eta = (c(eta)' − ybar'*A(:,eta));

obj = c'*xbar;

display('Available display items:');

display(' A, b, c, m, n, beta, eta,');
display(' A_beta, A_eta, c_beta, c_eta,');
display(' xbar_beta, xbar, obj, ybar, Abar_eta, cbar_eta,');
display(' pivot_ratios(j), pivot_direction(j), pivot_plot');
display(' ');
display(' To see XXX in floating point, type: double(XXX),')
display(' or if XXX is symbolic or you want better control')
display(' of precision, type: vpa(XXX,d)')
display(' to get the same and with d digits of precision')
display(' ');
display('Available functionality:');
display(' pivot_swap(j,i)');
 153

% pivot_direction.m // Jon Lee

% view the direction zbar corresponding to increasing x_eta(j).
% syntax is pivot_direction(j)

function pivot_direction(j)
global beta eta m n Abar_eta zbar
if (j > n−m)
display('error: not so many directions')
return
end
zbar = sym(zeros(n,1));
zbar(beta) = −Abar_eta(:,j);
zbar(eta(j))=1;
zbar
end

% pivot_ratios.m // Jon Lee

% calculate ratios for determing a variable to leave the basis.
% syntax is ratios(j), where we are letting eta_j enter the basis

function pivot_ratios(j)
global xbar_beta Abar_eta m n ratios;
if (j>n−m)
display('error: j out of range.')
else
ratios = xbar_beta ./ Abar_eta(:,j)
end

display(' To see the ratios in floating point, type: double(ratios),')

display(' or if they are symbolic or you want better control')
display(' of precision, type: vpa(ratios,d)')
display(' to get the same and with d digits of precision')

% pivot_swap.m // Jon Lee

% swap eta_j with beta_i
% syntax is pivot_swap(j,i)

function swap(j,i)
global beta eta m n;
if ((i>m) | (j>n−m))
display('error: i or j out of range. swap not accepted');
else
savebetai = beta(i);
beta(i) = eta(j);
eta(j) = savebetai;
display('swap accepted −−− new partition:')
display(beta);
display(eta);
display('*** MUST APPLY pivot_algebra! ***')
end
154

% pivot_plot.m // Jon Lee

% Make a 2−D plot in the space of the nonbasic variables (if n−m=2)

if (n−m ~= 2)
display('error: cannot plot unless n−m = 2')
return
end

clf;
hold on

syms x_eta1 x_eta2;

% a bit annoying but apparently necessary to symbolically perturb (below)

% in case a line is parallel to an axis.
x_eta1_axis = ezplot('0*x_eta1+x_eta2');
x_eta2_axis = ezplot('x_eta1+0*x_eta2');

set(x_eta1_axis,'LineStyle','−','Color',[0 0 0],'LineWidth',2);
set(x_eta2_axis,'LineStyle','−','Color',[0 0 0],'LineWidth',2);

h = Abar_eta*[x_eta1 ; x_eta2] − xbar_beta;

h = [h ; 'x_eta1' ; 'x_eta2'];

jcmap = colormap(lines);
for i = 1:m
% a bit annoying but apparently necessary to symbolically perturb (below)
% in case a line is parallel to an axis.
p(i) = ezplot(strcat(char(h(i)),'+0*x_eta1+0*x_eta2'));
set(p(i),'Color',jcmap(i,:),'LineWidth',2);
name{i} = ['x_{\beta_{',num2str(i),'}} = x_{' int2str(beta(i)),'}'];
end

title('Graph in the space of the nonbasic variables ($\bar{c}_\eta$ is magenta)',...

'interpreter','Latex','FontSize',14);
legend(p, name,'AutoUpdate','off');
xlabel(strcat('x_{\eta_1}= x_{',int2str(eta(1)),'}'));
ylabel(strcat('x_{\eta_2}= x_{',int2str(eta(2)),'}'));

if (xbar_beta >= 0)
plot([0],[0],'Marker','o','MarkerFaceColor','green','MarkerEdgeColor',...
'black','MarkerSize',5);
polygonlist=[0,0];
else
plot([0],[0],'Marker','o','MarkerFaceColor','red','MarkerEdgeColor',...
'black','MarkerSize',5);
polygonlist=[ , ];
end

for i = 1:m+1
for j = i+1:m+2
[M, d] = equationsToMatrix(h(i) == 0, h(j) == 0, [x_eta1, x_eta2]);
if (rank(M) == 2)
intpoint=linsolve(M,d);
[warnmsg, msgid] = lastwarn;
if ~strcmp(msgid,'symbolic:sym:mldivide:warnmsg2')
 155

if ( [linsolve(A_beta, b−A_eta*intpoint) ; intpoint] >=0 )

plot([intpoint(1)],[intpoint(2)],'Marker','o', ...
'MarkerFaceColor','green','MarkerEdgeColor','black', ...
'MarkerSize',5);
polygonlist =[polygonlist; intpoint'];
else
plot([intpoint(1)],[intpoint(2)],'Marker','o', ...
'MarkerFaceColor','red','MarkerEdgeColor','black', ...
'MarkerSize',5);
end
end
end
end
end

pivot_directions = zeros(n,2);
pivot_direction(1);
pivot_directions(:,1)=zbar;
pivot_direction(2);
pivot_directions(:,2)=zbar;
pivot_directions;

if ( size(polygonlist,1) > 2 && any(pivot_directions(:,1) < 0) && ...

any(pivot_directions(:,2) < 0) )
polyx=polygonlist(:,1);
polyy=polygonlist(:,2);
K = convhull(double(polyx),double(polyy));
hull = fill(polyx(K),polyy(K),'c');
uistack(hull, 'bottom');
end

quiver([0],[0],cbar_eta(1),cbar_eta(2),'LineWidth',2,'MarkerSize',2, ...
'Color','magenta','LineStyle','−');
156

Sample sequence of commands:

>> pivot_setup

>> pivot_algebra

>> pivot_plot

>> pivot_ratios(2)

>> pivot_swap(2,4)

>> pivot_algebra

>> pivot_ratios(1)

>> pivot_swap(1,1)

>> pivot_algebra

>> pivot_ratios(2)

>> pivot_swap(2,2)

>> pivot_algebra
 157

Note that the MATLAB GUI allows to pan beyond the initial display range:
 159

A.3 MATLAB Gomory cuts

% gomory_input.m // Jon Lee

% data for gomory example

A = [7 8 −1 1 3; 5 6 −1 2 1];
A= sym(A);
c = [126 141 −10 5 67]';
c=sym(c);
b = [26 19]';
b=sym(b);
beta = [1,2];
[m,n] = size(A);
eta = setdiff(1:n,beta); % lazy eta initialization

% gomory_input2.m // Jon Lee

% data for gomory example
%
% this is really the same problem as gomory_input.m, but
% − the objective is put in as a constraint, so we can us it for cuts;
% − the rhs is symbolically perturbed, so we can easily do lex pivots.
%
syms t real;
A = [1 0 0 0 0 0; −26 7 8 −1 1 3; −19 5 6 −1 2 1];
A= sym(A);
c = [0 126 141 −10 5 67]';
c=sym(c);
b = [1 t t^2]';
b=sym(b);
beta = [1,2,3];
[m,n] = size(A);
eta = setdiff(1:n,beta); % lazy eta initialization

% pure_gomory.m // Jon Lee

% append a primal column corresponding to the pure gomory cut
% syntax is pure_gomory(i), where we choose i so that ybar(i) is fractional

function pure_gomory(i)
global A c A_beta ybar m n beta eta;

ei = zeros(m,1); ei(i)=1; % ei is the i−th standard unit column

hi = linsolve(A_beta,ei); % i−th column of basis inverse
r = −floor(hi); % best choice of r
btilde = ei + A_beta*r; % new column for P
A = [A btilde];
c = [c; floor(ybar'*btilde)];

eta = setdiff(1:n+1, beta);

n = n+1;
display('*** PROBABLY WANT TO APPLY pivot_algebra! ***')
160

% mixed_gomory.m // Jon Lee

% append a primal column corresponding to the mixed gomory cut
% syntax is mixed_gomory(i), where we choose i (for a y(i)
% that is required to be integer) so that ybar(i) is fractional

function mixed_gomory(i)
global A c A_beta c_beta ybar m n beta eta;

ei = zeros(m,1); ei(i)=1; % ei is the i−th standard unit column

hi = linsolve(A_beta,ei); % i−th column of basis inverse
r = subplus(−floor(hi)); % best choice of r
btilde = A_beta*r −(ybar(i)−floor(ybar(i))−1)*ei; % new column for P
A = [A btilde];
c = [c; c_beta'*r − (ybar(i)−floor(ybar(i))−1)*floor(ybar(i))];

eta = setdiff(1:n+1, beta);

n = n+1;
display('*** PROBABLY WANT TO APPLY pivot_algebra! ***')
 161

A.4 AMPL for uncapacitated facility-location problem

AMPL code for computationally comparing formulations of the uncapacitated facility-
location problem:

# uflstrong.mod // Jon Lee

#

param M integer > 1; # NUMBER OF FACILITIES

param N integer > 1; # NUMBER OF CUSTOMERS

set FACILITIES := {1..M};

set CUSTOMERS := {1..N};

set LINKS := FACILITIES cross CUSTOMERS;

param facility_cost {FACILITIES} := round(Uniform(1,100));

param shipping_cost {LINKS} := round(Uniform(1,100));

var X {(i,j) in LINKS} >= 0;

var Y {i in FACILITIES} binary;

minimize Total_Cost:
sum {(i,j) in LINKS} shipping_cost[i,j] * X[i,j]
+ sum {i in FACILITIES} facility_cost[i] * Y[i];

subject to Only_Ship_If_Open {(i,j) in LINKS}:

X[i,j] <= Y[i];

subject to Satisfy_Demand {j in CUSTOMERS}:

sum {(i,j) in LINKS} X[i,j] = 1;

# uflstrong.run // Jon Lee

#

reset;
option randseed 412837;
option display_1col 0;
option display_transpose −20;
option show_stats 1;
option solver cplex;
option cplex_options 'presolve=0';
model uflstrong.mod;
data uflstrong.dat;
option relax_integrality 1;
solve;
#display X;
display Y;
display Total_Cost;
162

# uflstrong.dat // Jon Lee

#

param M := 30;

param N := 100;

# uflweak.mod // Jon Lee

#

param M integer > 1; # NUMBER OF FACILITIES

param N integer > 1; # NUMBER OF CUSTOMERS

set FACILITIES := {1..M};

set CUSTOMERS := {1..N};

set LINKS := FACILITIES cross CUSTOMERS;

param facility_cost {FACILITIES} := round(Uniform(1,100));

param shipping_cost {LINKS} := round(Uniform(1,100));

var X {(i,j) in LINKS} >= 0;

var Y {i in FACILITIES} binary;

minimize Total_Cost:
sum {(i,j) in LINKS} shipping_cost[i,j] * X[i,j]
+ sum {i in FACILITIES} facility_cost[i] * Y[i];

subject to Only_Ship_If_Open {i in FACILITIES}:

sum {(i,j) in LINKS} X[i,j] <= N * Y[i];

subject to Satisfy_Demand {j in CUSTOMERS}:

sum {(i,j) in LINKS} X[i,j] = 1;

# uflweak.run // Jon Lee

#

reset;
option randseed 412837;
option display_1col 0;
option display_transpose −20;
option show_stats 3;
option solver cplex;
option cplex_options 'presolve=0';
model uflweak.mod;
data uflweak.dat;
option relax_integrality 1;
solve;
#display X;
display Y;
display Total_Cost;
 163

# uflweak.dat // Jon Lee

#

param M := 30; # NUMBER OF FACILITIES

param N := 100; # NUMBER OF CUSTOMERS

END NOTES 165

End Notes

1
“The reader will find no figures in this work. The methods which I set forth do not require either
constructions or geometrical or mechanical reasonings: but only algebraic operations, subject to a regular
and uniform rule of procedure.” — Joseph-Louis Lagrange, Preface to “Mécanique Analytique,” 1815.

2
“Il est facile de voir que...”, “il est facile de conclure que...”, etc. — Pierre-Simon Laplace, frequently
in “Traité de Mécanique Céleste.”

3
“One would be able to draw thence well some corollaries that I omit for fear of boring you.” — Gabriel
Cramer, Letter to Nicolas Bernoulli, 21 May 1728. Translated from “Die Werke von Jakob Bernoulli,” by
R.J. Pulskamp.
166 END NOTES

4
“Two months after I made up the example, I lost the mental picture which produced it. I really regret
this, because a lot of people have asked me your question, and I can’t answer.” — Alan J. Hoffman, private
communication with J. Lee, August, 1994.

5
“Fourier hat sich selbst vielfach um Ungleichungen bemüht, aber ohne erheblichen Erfolg.” — Gyula
Farkas, “Über die Theorie der Einfachen Ungleichungen,” Journal für die Reine und Angewandte Mathematik,
vol. 124:1–27.

6
“The particular geometry used in my thesis was in the dimension of the columns instead of the rows.
This column geometry gave me the insight that made me believe the Simplex Method would be a very
efficient solution technique for solving linear programs. This I proposed in the summer of 1947 and by
good luck it worked!” — George B. Dantzig, “Reminiscences about the origins of linear programming,”
Operations Research Letters vol. 1 (1981/82), no. 2, 43–48.
END NOTES 167

7
“George would often call me in and talk about something on his mind. One day in around 1959,
he told me about a couple of problem areas: something that Ray Fulkerson worked on, something else
whose details I forget. In both cases, he was using a linear programming model and the simplex method
on a problem that had a tremendous amount of data. Dantzig in one case, Fulkerson in another, had
devised an ad hoc method of creating the data at the moment it was needed to fit into the problem. I
reflected on this problem for quite awhile. And then it suddenly occurred to me that they were all doing
the same thing! They were essentially solving a linear programming problem whose data - whose columns
- being an important part of the data, were too many to write down. But you could devise a procedure
for creating one when you needed it, and creating one that the simplex method would choose to work
with at that moment. Call it the column-generation method. The immediate, lovely looking application
was to the linear programming problem, in which you have a number of linear programming problems
connected only by a small number of constraints. That fit in beautifully with the pattern. It was a way
of decomposing such a problem. So we referred to it as the decomposition algorithm. And that rapidly
became very famous.” — Philip Wolfe, interviewed by Irv Lustig ∼2003.

8
“So they have this assortment of widths and quantities, which they are somehow supposed to make
out of all these ten-foot rolls. So that was called the cutting stock problem in the case of paper. So Paul
[Gilmore] and I got interested in that. We struck out (failed) first on some sort of a steel cutting problem,
but we seemed to have some grip on the paper thing, and we used to visit the paper mills to see what
they actually did. And I can tell you, paper mills are so impressive. I mean they throw a lot of junk in at
one end, like tree trunks or something that’s wood, and out the other end comes – swissssssh – paper! It’s
one damn long machine, like a hundred yards long. They smell a lot, too. We were quite successful. They
didn’t have computers; believe me, no computer in the place. So we helped the salesman to sell them the
first computer.” — Ralph E. Gomory, interviewed by William Thomas, New York City, July 19, 2010.
168 END NOTES

9
“I spent the Fall quarter (of 1950) at RAND. My first task was to find a name for multistage decision
processes. An interesting question is, Where did the name, dynamic programming, come from? The
1950s were not good years for mathematical research. We had a very interesting gentleman in Washington
named Wilson. He was Secretary of Defense, and he actually had a pathological fear and hatred of the
word research. I’m not using the term lightly; I’m using it precisely. His face would suffuse, he would
turn red, and he would get violent if people used the term research in his presence. You can imagine
how he felt, then, about the term mathematical. The RAND Corporation was employed by the Air Force,
and the Air Force had Wilson as its boss, essentially. Hence, I felt I had to do something to shield Wilson
and the Air Force from the fact that I was really doing mathematics inside the RAND Corporation. What
title, what name, could I choose? In the first place I was interested in planning, in decision making, in
thinking. But planning, is not a good word for various reasons. I decided therefore to use the word
‘programming’. I wanted to get across the idea that this was dynamic, this was multistage, this was time-
varying I thought, lets kill two birds with one stone. Lets take a word that has an absolutely precise
meaning, namely dynamic, in the classical physical sense. It also has a very interesting property as an
adjective, and that is its impossible to use the word dynamic in a pejorative sense. Try thinking of some
combination that will possibly give it a pejorative meaning. It’s impossible. Thus, I thought dynamic
programming was a good name. It was something not even a Congressman could object to. So I used it as
an umbrella for my activities.” — Richard E. Bellman, “Eye of the Hurricane: An Autobiography,” 1984.

10
“Vielleicht noch mehr als der Berührung der Menschheit mit der Natur verdankt die Graphentheorie
der Berührung der Menschen untereinander.” — Dénes König, “Theorie Der Endlichen Und Unendlichen
Graphen,” 1936.
The Afterward

169
Bibliography

[1] Stephen P. Bradley, Arnoldo C. Hax, and Thomas L. Magnanti. Applied mathematical
programming. Addison Wesley, 1977. Available at http://web.mit.edu/15.053/
www/.
[2] Qi He and Jon Lee. Another pedagogy for pure-integer gomory. RAIRO-Operations
Research, 51(1):189–197, 2017.

[3] Jon Lee. A first course in combinatorial optimization. Cambridge Texts in Applied
Mathematics. Cambridge University Press, Cambridge, 2004.

[4] Jon Lee and Angelika Wiegele. Another pedagogy for mixed-integer gomory.
EURO Journal on Computational Optimization, 2017 (to appear).

[5] Katta G. Murty. Linear and combinatorial programming. John Wiley & Sons Inc., New
York, 1976.

171
Index of definitions

1-norm, 13 cost, 14, 104

∞-norm, 13 Cramer’s rule, 7
(matrix) product, 4 cutting pattern, 93
(single-commodity min-cost) cutting-stock problem, 93
network-flow problem, 14, 104
concave piecewise-linear function, 69 Dantzig-Wolfe Decomposition, 74
convex piecewise-linear function, 65 demand nodes, 19
determinant, 6
affine function, 65 dimension, 4
algebraic perturbation, 40 diving, 124
arcs, 14, 103 dot product, 4
artificial variable, 43 down branch, 123

basic direction, 28 edge weights, 105

basic feasible direction relative to the edges, 105
basic feasible solution, 29 elementary row operations, 5
basic feasible ray, 30 extreme point, 25
basic feasible solution, 24 extreme ray, 30
basic partition, 23
basic solution, 24 feasible, 2
basis, 4, 23 feasible direction relative to the feasible
basis matrix, 24 solution, 28
best bound, 124 feasible region, 2
big M, 116 flow, 14, 103
bipartite graph, 105 full column rank, 5
branch-and-bound, 122 full row rank, 5
breakpoints, 118
Gauss-Jordan elimination, 6
Chvátal-Gomory cut, 126
head, 14, 103
column space, 5
complementary, 53, 55 identity matrix, 6
concave function, 69 inverse, 6
consecutive-ones matrix, 106 invertible, 6
conservative, 14, 104
convex function, 65 key invariant for branch-and-bound,
convex set, 25 122

173
174 INDEX OF DEFINITIONS

knapsack problem, 94 ratio test, 36

ray, 30
Lagrangian Dual, 87 recursive optimization, 95
Laplace expansion, 6 reduced-cost fixing, 126
linear combination, 4 row space, 5
linear constraints, 1
linear function, 65 scalar product, 4
Linear optimization, 1 Sherman-Morrison formula, 6
linearly independent, 4 slack variable, 2
lower bound, 122 solution, 2
span, 4
Master Problem, 75
standard form, 2
matching, 112
subgradient, 87
max-norm, 13
sufficient unboundedness criterion, 36
most fractional, 125
supply nodes, 19
network, 14, 103 surplus variable, 2
network matrix, 104
tail, 14, 103
nodes, 14, 103
the adjacency condition, 119
non-basis, 23
totally unimodular (TU), 108
non-degeneracy hypothesis, 37
transportation problem, 19
null space, 5
trivial, 4
objective function, 1
uncapacitated facility-location
optimal, 2
problem, 116
overly complementary, 61
unimodular, 107
perfect matching, 105 up branch, 123
phase-one problem, 43
phase-two problem, 43 vertex cover, 112
pivot, 38, 135 vertex packing, 136
polyhedron, 1 vertex-edge incidence matrix of the
bipartite graph, 105
rank, 5 vertices, 105