Duality

Optimality Conditions
Constrained Optimization Algorithms

Constrained Optimization

Constrained Optimization

Cenk Toker ELE 704 Optimization 1 / 117

 Duality
Optimality Conditions
Constrained Optimization Algorithms

Outline

- Duality
- Optimality Conditions
- Constrained Optimization Algorithms

Cenk Toker ELE 704 Optimization 2 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Duality

- Consider the standard minimization problem (will be referred as the

primal problem)
min f (x)
s.t. g(x) ≤ 0
h(x) = 0
where g(x) and h(x)
 T
g(x) = g1 (x) g2 (x) ··· gi (x) ··· gL (x)
 T
h(x) = h1 (x) h2 (x) ··· hj (x) ··· hM (x)
represent the L inequality and M equality constraints, respectively.
- The domain of the optimization problem D is defined by
L
\ M
\
D = dom f (x) ∩ dom gi (x) ∩ dom hj (x)
i=1 j=1

- Any point x̃ ∈ D satisfying the constraints is called a feasible point, i.e.,

g(x̃) ≤ 0 and h(x̃)= 0.
Cenk Toker ELE 704 Optimization 3 / 117
 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Lagrange Dual Function

- Define the Lagrangian L : RN × RL × RM → R as

L(x, λ, ν) = f (x) + λT g(x) + ν T h(x)
L
X M
X
= f (x) + λi gi (x) + νj hj (x)
i=1 j=1

where λi ≥ 0 and νj are called the Lagrange multipliers, and λ and ν

 T
λ = λ1 λ2 · · · λi · · · λL
 T
ν = ν1 ν2 · · · νj · · · νM
are called the Lagrange multiplier vectors.

- On the feasible set F where

F = {x̃ | x̃ ∈ D ∧ g(x̃) ≤ 0 ∧ h(x̃) = 0}
Lagragian has a value less than or equal to the cost function, i.e,
L(x̃, λ, ν) ≤ f (x̃), ∀x̃ ∈ F, ∀λi ≥ 0
.
Cenk Toker ELE 704 Optimization 4 / 117
 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- Then the Lagrange dual function, ℓ(λ, ν), is defined as

ℓ(λ, ν) = inf L(x, λ, ν)

x∈D
 
= inf f (x) + λT g(x) + ν T h(x)
x∈D
L M
!
X X
= inf f (x) + λi gi (x) + νj hj (x)
x∈D
i=1 j=1

- Dual function ℓ(λ, ν) is the pointwise infimum of a set of affine functions

of λ and ν, hence it is concave even if f (x), gi (x) and hj (x) are not
convex.

Cenk Toker ELE 704 Optimization 5 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- Proposition: The dual function constitutes a lower bound on p∗ = f (x∗ ),

i.e.,
ℓ(λ, ν) ≤ p∗ ∀λ ≥ 0

- Proof: Let x̃ be a feasible point, that is x̃ ∈ F (i.e., gi (x̃) ≤ 0 and

hi (x̃) = 0), and λ > 0, then

λT g(x̃) + ν T h(x̃) ≤ 0.

- Then, the Lagrangian is

L(x̃, λ, ν) = f (x̃) + λT g(x̃) + ν T h(x̃) ≤ f (x̃)

| {z }
≤0

- So,
ℓ(λ, ν) = inf L(x, λ, ν) ≤ L(x̃, λ, ν) ≤ f (x̃) ∀x̃ □
x∈D

- The pair (λ, ν) is called dual feasible when λ ≥ 0.

Cenk Toker ELE 704 Optimization 6 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Cenk Toker ELE 704 Optimization 7 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Cenk Toker ELE 704 Optimization 8 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Examples:
- Least Squares (LS) solution of linear equations

min xT x
s.t. Ax = b

- The Lagrangian is

L(x, ν) = xT x + ν T (Ax − b)

- then the Lagrange dual function is

ℓ(ν) = inf L(x, ν)

x

- Since L(x, ν) is quadratic in x, it is convex

1
∇L(x, ν) = 2x + AT ν = 0 ⇒ x∗ = − AT ν
2
- Hence Lagrange dual function is given by
1 1
ℓ(ν) = L(− AT ν, ν) = − ν T AT Aν − bT ν
2 4
- which is obviously concave p∗ ≥ ℓ(ν) = − 14 ν T AT Aν − bT ν
Cenk Toker ELE 704 Optimization 9 / 117
 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- Linear Programming (LP)

min cT x
s.t. Ax = b
−x≤0

- The Lagrangian is

L(x, λ, ν) = cT x − λT x + ν T (Ax − b)

- then the Lagrange dual function is

 T 
ℓ(λ, ν) = −bT ν + inf c + AT ν − λ x
x

- In order ℓ(ν) to be bounded, we must have c + AT ν − λ = 0, i.e.,

(
−bT ν, c + AT ν − λ = 0
ℓ(λ, ν) =
−∞, otherwise

- Affine, i.e., concave when c + AT ν − λ = 0 with λ ≥ 0

- So, p∗ ≥ −bT ν when c + AT ν − λ = 0 with λ ≥ 0.
Cenk Toker ELE 704 Optimization 10 / 117
 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- Two-way partitioning (a non-convex problem)

min xT Wx
s.t. x2j = 1, j = 1, . . . , N

- This is a discrete-problem since xj ∈ {∓1}, and very difficult to

solve for large N .
- The Lagrange dual function is
( N
)
T
X
νj x2j


ℓ(ν) = inf x Wx + −1
x
j=1
n o
= inf xT (W + diag ν)x − 1T ν
x
(
−1T ν, W + diag ν ⪰ 0
=
−∞, else

- We may take ν = −λmin (W)1T which yields

p∗ ≥ −1T ν = N λmin (W)

where λmin (W) is the minimum eigen value of W.

Cenk Toker ELE 704 Optimization 11 / 117
 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

N
x2j = 1, then
P
- - If we relax the constraint to be ∥x∥2 = 1, i.e.,
j=1
problem becomes easy to solve

min xT Wx
s.t. ∥x∥2 = 1

- with an exact solution of

p∗ = N λmin (W)

where λmin (W) is the minimum eigen value of W.

Cenk Toker ELE 704 Optimization 12 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

The Lagrange Dual Problem

max ℓ(λ, ν)
s.t. λ ≥ 0

gives the best lower bound for p∗ , i.e., p∗ ≥ ℓ(λ, ν).

- The pairs ℓ(λ, ν) with λ ≥ 0, ν ∈ RM and ℓ(λ, ν) > −∞ are dual
feasible.

- Solution of the above problem for a dual feasible set is called the dual
optimal point (λ∗ , ν ∗ ) (i.e., optimal Lagrange multipliers)

p̂∗ = ℓ(λ∗ , ν ∗ ) ≤ p∗

- Some hidden (implicit) constraints can be made explicit in the dual

problem, e.g. consider the LP problems on the next slides.

Cenk Toker ELE 704 Optimization 13 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- Linear problem (LP)

min cT x
s.t. Ax = b
x≥0

has the dual function

(
−bT ν, c + AT ν − λ = 0
ℓ(λ, ν) =
−∞, otherwise

- So, the dual problem can be given by

max − bT ν
s.t. AT ν − λ + c = 0
λ≥0

- The dual problem can be further simplified to the following problem

max − bT ν
s.t. AT ν + c ≥ 0

Cenk Toker ELE 704 Optimization 14 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- Linear problem (LP) with inequality

min cT x
s.t. Ax ≤ b

has the dual function

(
−bT λ, AT λ + c = 0
ℓ(λ) =
−∞, otherwise

- So, the dual problem can be given by

max − bT λ
s.t. AT λ + c = 0
λ≥0

Cenk Toker ELE 704 Optimization 15 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Weak Duality

- We know that

sup ℓ(λ, ν) = p̂∗ ≤ p∗ = inf f (x)

λ∈R+ ,ν x∈D

- The inequality means weak duality.

- Here (p∗ − p̂∗ ) is called the duality gap, which is always nonnegative.
- Weak duality always holds even when the primal problem is non-convex.

Cenk Toker ELE 704 Optimization 16 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Strong Duality

- Strong duality refers to the case where the duality gap is zero, i.e.,

p̂∗ = p∗

- In general it does not hold.

- It may hold if the primal problem is convex.
- The conditions under which strong duality hold are called constraint
qualifications, where one of them is the Slater’s condition.

Cenk Toker ELE 704 Optimization 17 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Slater’s Condition

- Strong duality holds for a convex problem

min f (x)
s.t. g(x) ≤ 0
Ax = b

- if it is strongly feasible, i.e., ∃x ∈ interior D

g(x) < 0
Ax = b

- i.e., inequality constraints hold with strict inequality.

Cenk Toker ELE 704 Optimization 18 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

Saddle-point Interpretation

- First consider the following problem,

 
sup L(x, λ) = sup f (x) + λT g(x)
λ≥0 λ≥0
L
!
X
= sup f (x) + λi gi (x)
λ≥0 i=1
(
f (x), gi (x) ≤ 0, ∀i
=
∞, otherwise

with no equality constraint.

- If gi (x) ≤ 0, then optimum choice is λi = 0.

Cenk Toker ELE 704 Optimization 19 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- Hence from duality gap, we have

p̂∗ ≤ p∗

- with weak duality as the inequality

sup inf L(x, λ) ≤ inf sup L(x, λ)

λ≥0 x x λ≥0

- and strong duality as the equality

sup inf L(x, λ) = inf sup L(x, λ)

λ≥0 x x λ≥0

- i.e. with strong duality we can switch inf and sup for λ ≥ 0.

Cenk Toker ELE 704 Optimization 20 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- In general, for any f (w, z) : RN × RL → R

sup inf f (w, z) ≤ inf sup f (w, z)

z∈Z w∈W w∈W z∈Z

with W ⊆ RN , Z ⊆ RL , and is called the max-min inequality.

- f (w, z) satisfy the strong max-min property or the saddle-point property

if the above inequality holds with equality.

- A point {(w̃, z̃)|w̃ ∈ W, z̃ ∈ Z} is called as the saddle-point for f (w, z) if

f (w̃, z) ≤ f (w̃, z̃) ≤ f (w, z̃)

∀w ∈ W, ∀z ∈ Z

- i.e.,

f (w̃, z̃) = inf f (w, z̃)

w∈W

= sup f (w̃, z)
z∈Z

the strong max-min property holds with the value f (w̃, z̃).

Cenk Toker ELE 704 Optimization 21 / 117

 Duality Lagrange Dual Function
Optimality Conditions Dual Problem
Constrained Optimization Algorithms Weak and Strong Duality

- For Lagrange duality, if x∗ and λ∗ are optimal points for the primal and
dual problems with strong duality (zero duality gap), then they form a
saddle-point for the Lagrangian, or vice-versa (i.e., the converse is also
true).

Cenk Toker ELE 704 Optimization 22 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Certificate of Suboptimality

- We know that a dual feasible point satisfies

ℓ(λ, ν) ≤ p∗

i.e., the point (λ, ν) is proof or certificate of this condition.

- Then,
f (x) − p∗ ≤ f (x) − ℓ(λ, ν)

- for primal feasible point x and dual feasible point (λ, ν) whereas the
duality gap associated with these points is

f (x) − ℓ(λ, ν)

- in other words

p∗ ∈ [ℓ(λ, ν), f (x)] and p̂∗ ∈ [ℓ(λ, ν), f (x)]

- If the duality gap is zero, i.e., f (x) = ℓ(λ, ν) then x∗ = x and

(λ∗ , ν ∗ ) = (λ, ν) are the primal and dual optimal points.

Cenk Toker ELE 704 Optimization 23 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Stopping Criterion

- If an algorithm produces the sequences x(k) and (λ(k) , ν (k) ) check for
?
f (x(k) ) − ℓ(λ(k) , ν (k) ) < ϵ

to guarantee ϵ-suboptimality. ϵ can approach to zero, i.e., ϵ → 0, for

strong duality.

Cenk Toker ELE 704 Optimization 24 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Complementary Slackness

- Assume that x∗ and (λ∗ , ν ∗ ) satisfy strong duality

f (x∗ ) = ℓ(λ∗ , ν ∗ )
 
= inf f (x) + λT g(x) + ν T h(x)
x
!
X X
= inf f (x) + λi gi (x) + νj hj (x)
x
i j
X X
≤ f (x∗ ) + λ∗i gi (x∗ ) + νj∗ hj (x∗ )
i j
| {z } | {z }
≤0 =0

≤ f (x∗ )

Cenk Toker ELE 704 Optimization 25 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

- Observations: (due to strong duality)

- Inequality in the third line always holds with equality, i.e., x∗ minimizes
L(x, λ∗ , ν ∗ )
P ∗
- From the forth line we have λi gi (x∗ ) = 0 and since λi gi (x) ≤ 0
i

λ∗i gi (x∗ ) =0 ∀i

which is known as complementary slackness.

- In other words

λ∗i > 0 if gi (x∗ ) = 0

λ∗i = 0 if gi (x∗ ) < 0

- i.e., the ith optimal lagrange multiplier is

- positive if gi (x∗ ) is active at x∗ .
- zero if gi (x∗ ) is not active at x∗ .

Cenk Toker ELE 704 Optimization 26 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

KKT Optimality Conditions

- x∗ minimizes L(x, λ∗ , ν ∗ ), thus ∇L(x, λ∗ , ν ∗ ) = 0, i.e.,

X ∗ X ∗
∇f (x∗ ) + λi ∇gi (x∗ ) + νj ∇hj (x∗ ) = 0
i j

- Then the Karush-Kuhn-Tucker (KKT) conditions for x∗ and (λ∗ , ν ∗ )

being primal and dual optimal points with zero duality gap (i.e., with
strong duality) are

gi (x∗ ) ≤ 0 (constraint)
hj (x∗ ) = 0 (constraint)
λ∗i ≥0 (constraint)
λ∗i gi (x ) = 0
∗
(complementary slackness)
∗
X X
∇f (x ) + λ∗i ∇gi (x∗ ) + νj∗ ∇hj (x∗ ) =0
i j

Cenk Toker ELE 704 Optimization 27 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

- For any optimization problem with differentiable objective (cost) and

constraint functions for which strong duality holds, any pair of primal and
optimal points satisfy KKT conditions.

- For convex problems:

- If f (x), gi (x) and hj (x) are convex and x̃, λ̃ and ν̃ satisfy KKT
conditions, then they are optimal points, i.e.,
- from complementary slackness f (x̃) = L(x̃, λ̃, ν̃)
 
- from last condition ℓ(λ̃, ν̃) = L(x̃, λ̃, ν̃) = inf L(x, λ̃, ν̃) . Note
x
that L(x, λ̃, ν̃) is convex in x.

- Thus,
f (x̃) = ℓ(λ̃, ν̃).

Cenk Toker ELE 704 Optimization 28 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Example 18:

1 T
min x Qx + cT x + r (Q : SPD)
2
s.t. Ax = b

Soln: From KKT conditions

Ax∗ = b
)
AT x∗
    
Q −c
= ∗ =
Qx∗ + c + AT ν ∗ = 0 A 0 ν b

Cenk Toker ELE 704 Optimization 29 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Example 19:

Solution:

Cenk Toker ELE 704 Optimization 30 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Cenk Toker ELE 704 Optimization 31 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Cenk Toker ELE 704 Optimization 32 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Solving The Primal Problem via The Dual

- If strong duality holds and if dual optimal solution (λ∗ , ν ∗ ) exists, then
we can compute a primal optimal solution from the dual solutions.

- When strong duality holds and (λ∗ , ν ∗ ) is given, the minimizer of

L(x, λ∗ , ν ∗ ), i.e.,
X ∗ X ∗
min f (x) + λi gi (x) + νj hj (x)
i j

is unique and if it is primal feasible, then it is also primal optimal.

- If the dual problem is easier to solve (e.g. has less dimensions or has an
analytical solution), then solving the dual problem and finding the
optimal dual parameters (λ∗ , ν ∗ ) first and then solving

x∗ = argmin L(x, λ∗ , ν ∗ )

will be an acceptable method to solve constrained minimization problems.

Cenk Toker ELE 704 Optimization 33 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

- Example 20: Consider the following problem

N
X
min f (x) = fi (xi )
i=1

s.t. aT x = b

where each fi (x) is strictly convex and differentiable, a ∈ RN and b ∈ R.

Assume that the problem has a unique solution (non-empty and
non-infinity) and it is dual feasible.

- f (x) is separable because fi (xi ) is a function of xi only. Now the

Lagrangian will be given as
N
X  
L(x, ν) = fi (xi ) + ν aT x − b
i=1
N
X
= −bν + (fi (xi ) + νai xi )
i=1

Cenk Toker ELE 704 Optimization 34 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

- Then the dual function is given by

ℓ(ν) = inf L(x, ν)
x
N
X
= −bν + inf (fi (xi ) + νai xi )
x
i=1
N
X
= −bν + inf (fi (xi ) + νai xi )
xi
i=1 | {z }
fi∗ (−νai )

N
X
= −bν − fi∗ (−νai )
i=1

where fi∗ (y) is the conjugate function of fi (x).

- NOTE: The conjugate function f ∗ (y) is the maximum gap between the
linear function yT x and f (x) (see Boyd Section 3.3). If f (x) is
differentiable, this occurs at a point x where ∇f (x) = y. Note that,
f ∗ (y) is a convex function.
 
f ∗ (y) = sup yT x − f (x)
x∈dom f (x)

Cenk Toker ELE 704 Optimization 35 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

- Then the dual problem is a function of a scalar ν ∈ R

N
!
X
max −bν − fi∗ (−νai )
ν
i=1

- Once we find ν ∗ , we know that L(x, ν ∗ ) is strictly convex as each fi (x) is

strictly convex. So, we can find x∗ by solving

∇L(x, ν ∗ ) = 0
∂fi (xi )
= −ν ∗ ai .
∂xi

Cenk Toker ELE 704 Optimization 36 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

Perturbation and Sensitivity Analysis

- Original Problem
primal dual

min f (x) max ℓ(λ, ν)

s.t. g(x) ≤ 0 s.t. λ ≥ 0
h(x) = 0

- Perturbed problem
primal dual

min f (x) max ℓ(λ, ν) − λT u − ν T v

s.t. g(x) ≤ u s.t. λ ≥ 0
h(x) = v

Cenk Toker ELE 704 Optimization 37 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

- Here u = [ui ]L×1 and v = [vj ]M ×1 are called the perturbations. When
ui = 0 and vj = 0, the problem becomes the original problem. If ui > 0,
it means we have relaxed the i-th inequality constraint, and if ui < 0, it
means we have tightened the i-th inequality constraint.

- Let us use the notation p∗ (u, v) to denote the optimal solution of the
perturbed problem. Thus, the optimal solution of the original problem is
p∗ = p∗ (0, 0) = ℓ(λ∗ , ν ∗ ).

- Assume that strong duality holds and optimal dual solution exists, i.e.,
p∗ = ℓ(λ∗ , ν ∗ ). Then, we can show that

p∗ (u, v) ≥ p∗ (0, 0) − λ∗T u − ν ∗T v

Cenk Toker ELE 704 Optimization 38 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

- If λi is large and ui < 0, then p∗ (u, v) is guaranteed to increase greatly.

- If λi is small and ui > 0, then p∗ (u, v) will not decrease too much.

- If |νj | is large
- If νj > 0 and vi < 0, then p∗ (u, v) is guaranteed to increase greatly.
- If νj < 0 and vi > 0, then p∗ (u, v) is guaranteed to increase greatly.

- If |νj | is small
- If νj > 0 and vi > 0, then p∗ (u, v) will not decrease too much.
- If νj < 0 and vi < 0, then p∗ (u, v) will not decrease too much.

Cenk Toker ELE 704 Optimization 39 / 117

 Certificate of Suboptimality and Stopping Criterion
Duality Complementary Slackness
Optimality Conditions KKT Optimality Conditions
Constrained Optimization Algorithms Solving The Primal Problem via The Dual
Perturbation and Sensitivity Analysis

- The perturbation inequality, p∗ (u, v) ≥ p∗ (0, 0) − λ∗T u − ν ∗T v, give a

lower bound on the perturbed optimal value, but no upper bound. For
this reason the results are not symmetric respect to loosening or
tightening a constraint, For example, if λi is large and we loosen the i-th
constraint a bit (i.e., take ui small and positive, 0 < ui < ϵ), then
perturbation inequality is not useful, it does not imply that optimal value
will decrease considerably.

-
Cenk Toker ELE 704 Optimization 40 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Introduction

- The general constrained optimization problem

min f (x)
s.t. g(x) ≤ 0
h(x) = 0

- Defn (Active Constraint): Given a feasible point x(k) , if gi (x) ≤ 0 is

satisfied with equality, i.e., gi (x) = 0 , then constraint i is said to be
active at x(k) . Otherwise it is inactive at x(k) .

Cenk Toker ELE 704 Optimization 41 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Cenk Toker ELE 704 Optimization 42 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Primal Methods
See Luenberger Chapter 12.
- A primal method is a search method that works on the original problem
directly by searching through the feasible region for the optimal solution.
Each point in the process is feasible and the value of the objective
function constantly decreases.
- For a problem with N variables and M equality constraints, primal
methods work in the feasible space of dimension N − M .
- Advantages:
- x(k) are all composed of feasible points.
- If x(k) is a convergent sequence, it converges at least to a local
minimum.
- Do not rely on special problem structure, e.g. convexity, in other
words, primal methods are applicable to general non-linear problems.
- Disadvantages:
- must start from a feasible initial point.
- may fail to converge for inequality constraints if precaution is not
taken.
Cenk Toker ELE 704 Optimization 43 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Feasible Direction Methods

- Update equation is
x(k+1) = x(k) + α(k) d(k)

- d(k) must be a descent direction and x(k) + α(k) d(k) must be contained
in the feasible region, i.e., d(k) must be a feasible direction for some
α(k) > 0.

- Very similar to unconstrained descent methods but now line search is

constrained so that x(k+1) is also a feasible point.

Cenk Toker ELE 704 Optimization 44 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Example 21: Consider the following problem

min f (x)
s.t. aTi x ≤ bi , i = 1, . . . , M

- Let A(k) be the set of indices representing active constraints, i.e.,

aTi x(k+1) = bi , i ∈ A(k) at x(k) , then the direction vector d(k) is
calculated by
min ∇T f (x(k) )d
d

s.t. aTi d ≤ 0, i ∈ A(k)

N
X
|di | = 1
i=1

- Last line ensures a bounded solution. The other constraints assure that
vectors of the form x(k) + α(k) d(k) will be feasible for sufficiently small
α(k) > 0, and subject to these conditions, d(k) is chosen to line up as
closely as possible with the negative gradient of f (x(k) ). In some sense
this will result in the locally best direction in which to proceed. The
overall procedure progresses by generating feasible directions in this
manner, and moving along them to decrease the objective.
Cenk Toker ELE 704 Optimization 45 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

There are two major shortcomings of feasible direction methods that require
that they be modified in most cases.
- The first shortcoming is that for general problems there may not exist any
feasible directions. If, for example, a problem had nonlinear equality
constraints, we might find ourselves in the situation where no straight line
from x(k) has a feasible segment. For such problems it is necessary either
to relax our requirement of feasibility by allowing points to deviate
slightly from the constraint surface or to introduce the concept of moving
along curves rather than straight lines.

- A second shortcoming is that in simplest form most feasible direction

methods are not globally convergent. They are subject to jamming
(sometimes referred to as zigzagging) where the sequence of points
generated by the process converges to a point that is not even a
constrained local minimum point.

Cenk Toker ELE 704 Optimization 46 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Active Set Methods

- The idea underlying active set methods is to partition inequality

constraints into two groups: those that are to be treated as active and
those that are to be treated as inactive. The constraints treated as
inactive are essentially ignored.

- Consider the following problem

min f (x)
s.t. g(x) ≤ 0
For simplicity, no equality constraints, the inclusion of equality constraints
will be straightforward.

- Necessary conditions for optimum x∗ are

 
∇f (x∗ ) + ∇ λT g(x∗ ) = 0
g(x∗ ) ≤ 0
λT g(x∗ ) = 0
λ≥0
Cenk Toker ELE 704 Optimization 47 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Let A be the set of indices of active constraints (i.e., gi (x∗ ) = 0, i ∈ A).

Then
X
∇f (x∗ ) + λi ∇gi (x∗ ) = 0
i∈A

gi (x∗ ) = 0, i∈A
∗
gi (x ) < 0, i∈
/A
λi ≥ 0, i∈A
λi = 0, i∈
/A

Inactive constraints are inhibited (i.e., λi = 0).

- Just taking active constraints in A, problem is converted to an

equality-constrained-only problem.

Cenk Toker ELE 704 Optimization 48 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Active set method:

- at each step find the working set W and treat it as the active set
(can be a subset of the actual active set, i.e., W ⊆ A ).
- move to a lower point on the surface of the working set
- find a new working set W for this point
- repeat

- The surface defined by the working set W will be called the working
surface.

Cenk Toker ELE 704 Optimization 49 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Given any working set W ⊆ A, Assume that xW is a solution to the

problem PW

min f (x)
s.t. gi (x) = 0, i∈W

also satisfying gi (xW ) < 0, i ∈

/ A. If xW cannot be found change the
working set W until a solution xW is obtained.

- Once xW is found, then solve

X
∇f (xW ) + λi ∇gi (xW ) = 0
i∈W

to find λi .

- If λi ≥ 0, ∀i ∈ W, then xW is a local optimal solution of the original

problem.

- If ∃i ∈ W such that λi < 0, then remove the i-th constraint from the
working set.

Cenk Toker ELE 704 Optimization 50 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- The surface defined by a working set is called a working surface. By

dropping i from W and moving on the new working surface (toward the
interior of the feasible region F, we move to an improved solution. )

- Monitoring the movement to avoid infeasibility until one or more

constraints become active, then add them to the working set W. Now,
solve the changed problem PW again to find a new xW and repeat the
previous steps.

Cenk Toker ELE 704 Optimization 51 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- If we can assure that the objective function (cost function) value is

monotonically decreasing, then any working set will not appear twice in
the process. Hence the active set method terminates in a finite number of
iterations.)

Cenk Toker ELE 704 Optimization 52 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Active Sets Algorithm: For a given working set W

- repeat
- repeat
- minimize f (x) over the working set W using
x(k+1) = x(k) + α(k) d(k)
- check whether a new constraint becomes active.
- if so, add the new constraint to the working set W.
- until some stopping criterion is satisfied
- check the Lagrange multipliers λi
- drop constraints with λi < 0 from the working set W
- until some stopping criterion is satisfied

- Note that f (x) strictly decreases at each step.

Cenk Toker ELE 704 Optimization 53 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Disadvantage: The inner loop must terminate at a global optimum in

order to determine correct λi , and the same working set is not
encountered in the following iterations.

- Discuss: How can we integarte equality constraints to the original

problem?

Cenk Toker ELE 704 Optimization 54 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Gradient Projection Method

- Gradient Projection Method is an extension of the Gradient (or Steepest)

Descent Method (GD or SD) to the constrained case.
- Let us first consider linear constraints
min f (x)
s.t. aTi x ≤ bi , i ∈ I1
aTi x = bi , i ∈ I2

Let us take the active constraints,i.e., aTi x = bi , as the working set W

and seek a feasible descent direction
find ∇T f (x)d < 0
while satisfying aTi d = 0, i∈W

i.e., d must lie in the tangent plane defined by aTi d = 0, i ∈ W.

- Hence, the above problem is the projection of −∇f (x) on to this tangent
plane.
Cenk Toker ELE 704 Optimization 55 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Another perspective is to let the equality Ax = b to represent all the

active constraints aTi x = bi , i ∈ W. Thus

min f (x)
s.t. Ax = b

- If we use first order Taylor approximation around point x(k) , such that
f (x) = f (x(k) + d) ∼= f (x(k) ) + ∇T f (x(k) )d for small enough d , the we
will have

min f (x(k) ) + ∇T f (x(k) )d

 
s.t. A x(k) + d = b

dT Id ≤ 1

- As Ax(k) = b and Ax(k+1) = b, so this implies that Ad = 0.

Cenk Toker ELE 704 Optimization 56 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Thus, the problem simplifies to

min ∇T f (x(k) )d
s.t. Ad = 0
dT Id ≤ 1

- Ad = 0 defines the tangent plane M ⊆ RN and ensures that x(k+1) is

still feasible.

- dT Id ≤ 1 is the Euclidean unit ball (∥d∥22 ≤ 1), thus this is the projected
GD algorithm.

- We may also use the constraint dT Qd ≤ 1, where Q is a SPD matrix, to

obtain the projected SD algorithm.

Cenk Toker ELE 704 Optimization 57 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Projected Steepest Descent Algorithm (PSDA):

- For a feasible initial point x(0) (i.e., x(0) ∈ F)

- Solve the Direction Finding Problem (DFP)

d(k) = argmin ∇T f (x(k) )d

s.t. Ad = 0
dT Qd ≤ 1, (Q : SPD)

- If ∇T f (x(k) )d(k) = 0, stop. x(k) is a KKT point.

- Solve α(k) = argmin f (x(k) + αd(k) ) using a line search algorithm, e.g.
exact or backtracking line search.

- x(k+1) = x(k) + α(k) d(k)

- Goto Step 1 with x(0) = x(k+1) .

Cenk Toker ELE 704 Optimization 58 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Projection:
- DFP problem is another constrained optimization problem which should
satisfy its KKT conditions, i.e.,

Ad(k) = 0
T
d(k) Qd(k) = 0
∇f (x(k) ) + 2βk Qd(k) + AT λk = 0
βk ≥ 0
T
 
βk 1 − d(k) Qd(k) = 0

- Here, Ad(k) = 0 defines the tangent plane M(k) ⊆ RN .

- If we set d̃(k) = 2βk d(k) , From the third condition we find that

d̃(k) = −Q−1 ∇f (x(k) ) − Q−1 AT λk

- Now if put this value into the first condition we obtain

 −1
λk = − AQ−1 AT AQ−1 ∇f (x(k) )

Cenk Toker ELE 704 Optimization 59 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Thus, d̃(k) is obtained as

  −1 
d̃(k) = − Q−1 − Q−1 AT AQ−1 AT AQ−1 ∇f (x(k) )

= −P(k) ∇f (x(k) )

- where  −1
P(k) = Q−1 − Q−1 AT AQ−1 AT AQ−1
is called the projection matrix.

- Note that if Q = I, then

 −1
P(k) = I − AT AAT A

- As 2βk is just a scaling factor, we can safely write that the descent
direction d(k) is given by

d(k) = −P(k) ∇f (x(k) )

Cenk Toker ELE 704 Optimization 60 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

PSDA with DFP Algorithm: Given a feasible point x(k) (i.e., x(k) ∈ F)
- Find the active constraint set W(k) and form A (actually A(k) ) matrix
- Calculate
 −1
P(k) = Q−1 − Q−1 AT AQ−1 AT AQ−1

d(k) = −P(k) ∇f (x(k) )

- If d(k) ̸= 0
- Find α
n o
α1 = max α : x(k) + αd(k) is feasible
n o
α2 = argmin f (x + αd(k) ) : 0 ≤ α ≤ α1

- x(k+1) = x(k) + α2 d(k)

- Goto Step 1 (with x(k) = x(k+1) )

Cenk Toker ELE 704 Optimization 61 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- If d(k) = 0
- Find λ  −1
λ = − AQ−1 AT AQ−1 ∇f (x(k) )

- If λi ≥ 0, ∀i, stop. x(k) satisfies KKT.

- If ∃λi < 0, delete the row from A corresponding to the inequality
with the most negative component of λi and drop its index from W.
Goto Step 2.

Cenk Toker ELE 704 Optimization 62 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Example 22: Consider the following problem

min x21 + x22 + x23 + x24 − 2x1 − 3x4
s.t. 2x1 + x2 + x3 + 4x4 = 7
x1 + x2 + 2x3 + x4 = 6
xi ≥ 0, i = 1, 2, 3, 4
T
Given the initial point x(0) = 2 2 1 0 , find the initial direction


d(0) for the projected gradient descent algorithm (PGDA).

- The active constraints are the two equalities and the inequality x4 ≥ 0,
thus  
2 1 1 4
A = 1 1 2 1 
0 0 0 1

- Also, ∇f (x(0) ) is given by

 
2
4
∇f (x(0) ) = 
2


−3

Cenk Toker ELE 704 Optimization 63 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- So,
 
22 9 4
AAT =  9 7 1
4 1 1
 
−1 6 −5 −19

T 1 
AA = −5 6 14 
11
−19 14 73

- Hence, the projection matrix P(0) is given by

 
1 −3 1 0
 −1 1  −3 9 −3 0
P(0) T
= I − A AA T
A=  
11  1 −3 1 0
0 0 0 0

- Finally, the driection d(0) is given by

 
−8
1 
 24  .
d(0) (0) (0)

= −P ∇f (x ) =
11 −8
0

Cenk Toker ELE 704 Optimization 64 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Nonlinear constraints:
- Consider the problem which defines only the active constraints

min f (x)
s.t. h(x) = 0

- In linear constraints x(k) lie on the tangent plane M. However in

nonlinear constraints, the surface defined by the constraints and the
tangent plane M touch in a single point

-
Cenk Toker ELE 704 Optimization 65 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- In this case updated point must be projected onto the constrained surface.
For projected gradient descent method, the projection matrix is given by
 −1
P(k) = I − JTh (x(k) ) Jh (x(k) )JTh (x(k) ) Jh (x(k) )

where Jh (x) is the Jacobian of h(x), i.e.,

 T
Jh (x) = ∇hT (x) .

Cenk Toker ELE 704 Optimization 66 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Equality Constrained Optimization

- See Boyd Chapter 10.

min f (x)
s.t. Ax = b

where f (x) : RN → R, convex, twice differentiable, A ∈ RM ×N , M < N .

- Minimum occurs at

p∗ = inf {f (x) | Ax = b} = f (x∗ )

- where x∗ satisfies KKT conditions

Ax∗ = b
∇f (x∗ ) + AT ν ∗ = 0

Cenk Toker ELE 704 Optimization 67 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Quadratic Minimization

1 T
min x Qx + cT x + r
2
s.t. Ax = b

where Q ∈ RN ×N is a symmetric positive semidefine matrix (SPSD),

A ∈ RM ×N and b ∈ RM .

- Using KKT conditions

Ax∗ = b
)
A T x∗
    
Q −c
= =
Qx∗ + c + AT ν ∗ = 0 A 0 ν∗ b
| {z }
KKT matrix

- Above equations defines a KKT system for an equality constrained

quadratic optimization problem with (N + M ) linear equations in the
(N + M ) variables (x∗ , ν ∗ ).

Cenk Toker ELE 704 Optimization 68 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Eliminating Equality Constraints

- One general approach to solving the equality constrained problem is to

eliminate the equality constraints and then solve the resulting
unconstrained problem using methods for unconstrained minimization.
- We first find a matrix F ∈ RN ×(N −M ) and vector x̂ that parametrize the
(affine) feasible set:
n o
{x | Ax = b} = Fz + x̂ | z ∈ RN −M .
Here x̂ can be chosen as any particular solution of Ax = b, and
F ∈ RN ×(N −M ) is any matrix whose range (columnspace) is the
nullspace of A, i.e., R(F) = N (A).
- We then form the reduced or eliminated optimization problem
min f˜(z) = f (Fz + x̂)
which is an unconstrained problem with variable z ∈ RN −M . From its
solution z∗ , we can find the solution of the equality constrained problem
as
x∗ = Fz∗ + x̂
.
Cenk Toker ELE 704 Optimization 69 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- There are, of course, many possible choices for the elimination matrix F,
which can be chosen as any matrix in RN ×(N −M ) with the range
(columnspace) of the matrix F is equals to the nullspace of the matrix A,
i.e., R(F) = N (A). If F is one such matrix, and T ∈ R(M −N )×(M −N ) is
nonsingular, then F̃ = FT is also a suitable elimination matrix, since
R(F̃) = R(F) = N (A).

- Conversely, if F and F̃ are any two suitable elimination matrices, then

there is some nonsingular T such that F̃ = FT. If we eliminate the
equality constraints using F, we solve the unconstrained problem
min f (Fz + x̂) while if F̃ is used, we solve the unconstrained problem
min f (F̃z̃ + x̂) = f (F(Tz̃) + x̂) This problem is equivalent to the one
above, and is simply obtained by the change of coordinates z = Tz̃. In
other words, changing the elimination matrix can be thought of as
changing variables in the reduced problem.

Cenk Toker ELE 704 Optimization 70 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Example 23:

Cenk Toker ELE 704 Optimization 71 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Newton’s Method with Equality Constraints

- It is the same as unconstrained Newton’s Method except

- initial point is feasible, x(0) ∈ F, i.e., Ax(0) = b.
- Newton step ∆xnt is a feasible direction, i.e., A∆xnt = 0.

- In order to use Newton’s Method on the problem

min f (x)
s.t. Ax = b

we can use second-order Taylor approximation around x (actually x(k) ) to

obtain a quadratic minimization problem
1
min f (x + ∆x) = f (x) + ∇T f (x)∆x + ∆xT H(x)∆x
2
s.t. A (x + ∆x) = b

- This problem is convex if H(x) ⪰ 0

Cenk Toker ELE 704 Optimization 72 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Using the results of quadratic minimization with equality constraints

H(x) AT ∆xnt
    
−∇f (x)
∗ = .
A 0 ν 0
| {z }
KKT matrix

Solution exists when the KKT matrix is non-singular.

- The same solution can also be obtained by setting x∗ = x + ∆xnt in the

optimality condition equations of the original problem

Ax∗ = b
∇f (x∗ ) + AT ν ∗ = 0

as ∆xnt and ν ∗ should satisfy the optimality conditions.

Cenk Toker ELE 704 Optimization 73 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Newton decrement λ(x) is the same as the one used for the
unconstrained problem, i.e.,
 1/2
λ(x) = ∆xTnt H(x)∆xnt = ∥∆xnt ∥H(x)

being the norm of the Newton step in the norm defined by H(x).
λ2 (x) λ2 (x)
- 2
, being a good estimate of f (x) − p∗ (i.e., f (x) − p∗ ≈ 2
), can
2
λ (x)
be used as the stopping criterion 2
≤ ϵ.

- Appears in the line search

d f (x + α∆xnt )
= ∇T f (x)∆xnt = −λ2 (x)
dα α=0

- In order to Newton step ∆xnt be a feasible descent direction the

following two conditions need to be satisfied
∇T f (x)∆xnt = −λ2 (x)
A∆xnt = 0

- The solution for equality constrained Newton’s Method is invariant to

affine transformations.
Cenk Toker ELE 704 Optimization 74 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Newton’s Method with Equality Constraint Elimination

min f (x)
≡ min f˜(z) = f (Fz + x̂)
s.t. Ax = b
with R(F) = N (A) and Ax̂ = b.

- The gradient and Hessian of f˜(z) are

∇f˜(z) = FT ∇f (Fz + x̂)

H̃(z) = FT H(Fz + x̂)F

- Then, the KKT matrix is invertible, iff H̃(z) is invertible.

- The Newton step of the reduced problem is

 −1
∆znt = −H̃−1 (z)∇f˜(z) = − FT H(x)F FT ∇f (x)

where x = Fz + x̂.

Cenk Toker ELE 704 Optimization 75 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- It can be shown that

∆xnt = F∆znt
where ∆xnt is the Newton step of the original problem with equality
constraints.

- The Newton decrement λ̃(z) is the same as the Newton decrement of the
original problem

λ̃2 (z) = ∆zTnt H̃(z)∆znt

= ∆zTnt FT H(x)F∆znt
= ∆xTnt H(x)∆xnt
= λ2 (x)

Cenk Toker ELE 704 Optimization 76 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Penalty and Barrier Methods

See Luenberger Chapter 13.

- They approximate constrained problems by unconstrained problems

- The approximation is obtained by adding

- a term with a high cost for violating the constraints (penalty
methods)
- a term that favors points interior to the feasible points over those
near the boundary (barrier methods)
to the objective function (cost function).

- The penalty and barrier methods work directly in original the

N -dimensional space RN rather than the (N − M )-dimensional space
RN −M as in the case of the primal methods.

Cenk Toker ELE 704 Optimization 77 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Penalty Methods

min f (x)
s.t. x ∈ F
- The idea is to replace the this problem by the following penalty problem
min f (x) + cP (x)
where c ∈ R++ is a constant and P (x) : RN → R is the penalty function.
- Here
- P (x) is continuous
- P (x) ≥ 0 ∀x ∈ RN
- P (x) = 0 iff x ∈ F
- In general, the following quadratic penalty function is used
L M
1X 1X
P (x) = (max {0, gi (x)})2 + (hj (x))2
2 i=1 2 j=1

where gi (x) ≤ 0 are the inequality constraints and hj (x) = 0 are the
equality constraints.
Cenk Toker ELE 704 Optimization 78 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

2
1
(max {0, gi (x)})2 with g1 (x) = x − b
P
- For example, consider P (x) = 2
i=1
and g2 (x) = a − x

- For large c, the minimum point of the penalty problem will be in a region
where P (x) is small.

- For increasing c, the solution will approach the feasible region F and as
c → ∞ the penalty problem will converge to a solution of the constrained
problem.
Cenk Toker ELE 704 Optimization 79 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Penalty Method:
n o
- Let c(k) , k = 1, 2, . . . be a sequence tending to ∞ such that ∀k
c(k) ≥ 0 and c(k+1) > c(k) .

- Let
q(c, x) = f (x) + cP (x)
and for each k solve the penalty problem

min q(c(k) , x)

obtaining a solution point x(k) .

Cenk Toker ELE 704 Optimization 80 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Lemma: As k → ∞, c(k+1) > c(k) (e.g., start from an exterior point)

- q(c(k) , x(k) ) ≤ q(c(k+1) , x(k+1) )
- P (x(k) ) ≥ P (x(k+1) )
- f (x(k) ) ≤ f (x(k+1) )
- f (x∗ ) ≥ q(c(k) , x) ≥ f (x(k) )
n o
- Theorem: Let x(k) , k = 1, 2, . . . be a sequence generated by the
penalty method. Then, any limit of the sequence is a solution of the
original constrained problem.

Cenk Toker ELE 704 Optimization 81 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Barrier Methods

- Also known as interior methods.

min f (x)
s.t. x ∈ F
- The idea is to replace the this problem by the following barrier problem
1
min f (x) + B(x)
c
s.t. x ∈ interior of F
where c ∈ R++ is a constant and B(x) : RN → R is the barrier function.
- Here, barrier function B(x) is defined on the interior of F such that
- B(x) is continuous
- B(x) ≥ 0
- B(x) → ∞ as x approaches the boundary of F
- Ideally, (
0, x ∈ interior F
B(x) =
∞, x∈/ interior F
Cenk Toker ELE 704 Optimization 82 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- There are several approximations. Two common barrier functions for the
inequality constraints are given below
L
X
Log barrier: B(x) = − log (−gi (x))
i=1
L
X 1
Inverse barrier: B(x) = −
i=1
gi (x)

Cenk Toker ELE 704 Optimization 83 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- For example, consider B(x) = −1/g1 (x) − 1/g2 (x), g1 (x) = x − b and
g2 (x) = a − x.

Cenk Toker ELE 704 Optimization 84 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Barrier Method:
n o
- Let c(k) , k = 1, 2, . . . be a sequence tending to ∞ such that ∀k
c(k) ≥ 0 and c(k+1) > c(k) .

- Let
1
r(c, x) = f (x) + B(x)
c
and for each k solve the barrier problem

min r(c(k) , x)
s.t. x ∈ interior of F

obtaining a solution point x(k) .

Cenk Toker ELE 704 Optimization 85 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Lemma: As k → ∞, c(k+1) > c(k) (start from an interior point)

- r(c(k) , x(k) ) ≥ r(c(k+1) , x(k+1) )
- B(x(k) ) ≤ B(x(k+1) )
- f (x(k) ) ≥ f (x(k+1) )
- f (x∗ ) ≤ f (x(k) ) ≤ r(c(k) , x)
n o
- Theorem: Any limit point of a sequence x(k) , k = 1, 2, . . . generated
by the barrier method is a solution of the original constrained problem.

Cenk Toker ELE 704 Optimization 86 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Properties of the Penalty & Barrier Methods

- The penalty method solves the unconstrained problem

min f (x) + cP (x)

 
- Let pi (x) = max {0, gi (x)} and p(x) = pi L×1

- Let the penalty function γ(x) where P (x) = γ(p(x)) to be the Euclidean
norm function
L
X
γ(y) = yT y ⇒ P (x) = pT (x)p(x) = (pi (x))2
i=1

- or more generally to be the quadratic norm function

γ(y) = yT Γ y ⇒ P (x) = pT (x) Γ p(x)

- The Hessian of the above problem becomes more and more ill-conditioned
as c → ∞

Cenk Toker ELE 704 Optimization 87 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Defining
q(c, x) = f (x) + c γ(p(x))

- The Hessian Q(c, x) is given by

Q(c, x) = F(x) + c ∇T γ(p(x)) G(x) + c JTp (x) Γ(p(x)) Jp (x)

where F(x), G(x) and Γ(x) are Hessians of f (x), g(x) and γ(x)
respectively, and Jp (x) is the Jacobian of p(x).

- If at x∗ there are r active constraints, then the Hessian matrices

Q(c(k) , x(k) ) have r eigenvalues tending to ∞ as c(k) → ∞ and (n − r)
eigenvalues tending to some finite value. In other words, condition
number goes to infinity (κ → ∞) as c(k) → ∞.

- Gradient Descent may not be directly applied, instead Newton’s Method

is preferred!

- Same observation also applies to the barrier method.

Cenk Toker ELE 704 Optimization 88 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Interior-Point Methods

See Boyd Chapter 11.

L
!
1 X
min f (x) + − log (−gi (x))
c i=1

s.t. Ax = b

- Minimum occurs at
p∗ = inf {f (x) | Ax = b} = f (x∗ )
Cenk Toker ELE 704 Optimization 89 / 117
 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- The function
L
X
ϕ(x) = − log (−gi (x))
i=1

with dom ϕ(x) = x ∈ RN | gi (x) < 0, ∀i is called the logarithmic



barrier function.

- We will modify the Newton’s algorithm to solve the above problem. So,
we will need
L
X 1
∇ϕ(x) = − ∇gi (x)
i=1
gi (x)
L L
X 1 X 1
Hϕ (x) = ∇ϕ(x)∇T ϕ(x) − Hg (x)
i=1
2
gi (x) i=1
gi (x) i

where Hϕ (x) = ∇ ϕ(x) and Hgi (x) = ∇2 gi (x) are the Hessians of ϕ(x)
2

and gi (x) respectively.

Cenk Toker ELE 704 Optimization 90 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Central Path

- Consider the equivalent problem (c > 0)

min cf (x) + ϕ(x)

s.t. Ax = b

- The point x∗ (c) is the solution. The trajectory of x∗ (c) as a function of c

is called the central path. Points on the central path satisfy

Ax∗ (c) = b



gi (x∗ (c)) < 0, ∀i Centrality conditions

c ∇f (x∗ (c)) + ∇ϕ(x∗ (c)) + AT ν̂ = 0


for some ν̂ ∈ RM .

- Last line could be rewritten as

L
X 1
c ∇f (x∗ (c)) − ∇gi (x∗ (c)) + AT ν̂ = 0
i=1
gi (x∗ (c))

Cenk Toker ELE 704 Optimization 91 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Example 24: Inequality form linear programming. The logarithmic barrier

function for an LP in inequality form

min eT x
s.t. Ax ≤ b

- (where e ∈ RN , A ∈ RN ×L and b ∈ RL are constants) is given by

L
X  
ϕ(x) = − log bi − aTi x , dom ϕ(x) = {x | Ax < b}
i=1

where aT1 , . . . , aTL are the rows of A.

Cenk Toker ELE 704 Optimization 92 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- The gradient and Hessian of the barrier function are

L
X 1
∇ϕ(x) = ai
i=1
bi − aTi x
= AT d
L
X 1 T
Hϕ (x) = 2 ai ai
i=1 (bi − aTi x)
= AT (diag d)2 A

1
where di = .
bi − aTi x
- Since x is strictly feasible, we have d > 0, so the Hessian of ϕ(x), Hϕ (x)
is nonsingular if and only if A has rank N , i.e. full-rank.

Cenk Toker ELE 704 Optimization 93 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- The centrality condition is c e + AT d = 0

- We can give a simple geometric interpretation of the centrality condition.
At a point x∗ (c) on the central path the gradient ∇ϕ(x∗ (c)), which is
normal to the level set of ϕ(x) through x∗ (c), must be parallel to −e. In
other words, the hyperplane eT x = eT x∗ (c) is tangent to the level set of
ϕ(x) through x∗ (c). Figure below shows an example with L = 6 and
N = 2.

Cenk Toker ELE 704 Optimization 94 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- The dashed curves in the previous figure show three contour lines of the
logarithmic barrier function ϕ(x). The central path converges to the
optimal point x∗ as c → ∞. Also shown is the point on the central path
with c = 10. The optimality condition at this point can be verified
geometrically: The line eT x = eT x∗ (10) is tangent to the contour line of
ϕ(x) through x∗ (10).

Cenk Toker ELE 704 Optimization 95 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Dual Points from Central Path

- From the centrality condition, let

1
λ∗i (c) = − , ∀i
c gi (x∗ (c))
ν̂
ν ∗ (c) =
c
- then
L
X
∇f (x∗ (c)) + λ∗i (c)∇gi (x∗ (c)) + AT ν ∗ (c) = 0
i=1

- Hence, from KKT, x∗ (c) minimizes

L(x, λ, ν) = f (x) + λT g(x) + ν T (Ax − b)

for λ = λ∗ (c) and ν = ν ∗ (c) for a particular c, which means that

(λ∗ (c), ν ∗ (c)) is a dual feasible pair.

Cenk Toker ELE 704 Optimization 96 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- So, the dual function optimal value p̂∗ (c) = ℓ(λ∗ (c), ν ∗ (c)) is finite and
given as
L
X
ℓ(λ∗ (c), ν ∗ (c)) = f (x∗ (c)) + λ∗i (c)gi (x∗ (c)) + ν ∗T (c) (Ax∗ (c) − b)
i=1
| {z } | {z }
=1
c
=0

L
= f (x∗ (c)) −
c

L
- Duality gap is and
c
L
f (x∗ (c)) − p∗ ≤
c
goes to zero, as c → ∞.

Cenk Toker ELE 704 Optimization 97 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

KKT Interpretation

- Central path (centrality) conditions can be seen as deformed KKT

conditions, i.e.,

Ax = b 

gi (x∗ ) ≤ 0, ∀i 




∗

λi ≥ 0, ∀i 


satisfied by

1
∗ ∗
−λi gi (x ) = , ∀i x∗ (c), λ∗ (c) and ν ∗ (c)
c 



X L 

∗ ∗ ∗ T ∗

∇f (x ) + λi ∇gi (x ) + A ν = 0




i=1

1
- Complementary slackness, λi gi (x) = 0 → −λ∗i gi (x∗ ) =
c
- As c → ∞, x∗ (c), λ∗ (c) and ν ∗ (c) almost satisfy the KKT optimality
conditions.

Cenk Toker ELE 704 Optimization 98 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Newton Step for Modified KKT equations

1
- Let λi = − , then
c gi (x)
L
X 1
∇f (x) − ∇gi (x) + AT ν = 0
i=1
c gi (x)
Ax = b

- To solve this set of (N + M ) linear independent equations (of N + M

variables x and ν) consider the nonlinear part of the first set of equations
L L
1 X 1
∇gi (x + d) ∼
X
∇f (x + d) − = ∇f (x) − ∇gi (x)
i=1
c gi (x + d) i=1
c gi (x)
| {z }
=ĝ

L L
X 1 X 1
+ H(x)d − Hgi (x)d + 2
∇gi (x)∇T gi (x)d
i=1
c gi (x) i=1
c g i (x)
| {z }
=Ĥd

Cenk Toker ELE 704 Optimization 99 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Substituting back, we obtain

Ĥd + AT ν = −ĝ
Ad = 0

where
L L
X 1 X 1
Ĥ = H(x) − Hg (x) + ∇gi (x)∇T gi (x)
i=1
c gi (x) i i=1
c g 2
i (x)

L
X 1
ĝ = ∇f (x) − ∇gi (x)
i=1
c gi (x)

- Using the derivations of ∇ϕ(x) and Hϕ (x)

1
Ĥ = H(x) + Hϕ (x)
c
1
ĝ = ∇f (x) + ∇ϕ(x)
c

Cenk Toker ELE 704 Optimization 100 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Let us represent the previous modified KKT equations in matrix form

Ĥ AT d
    
−ĝ
=
A 0 ν 0
∗
- whose solution would give the modified Newton step ∆xnt and νnt

c H(x) + Hϕ (x) AT ∆xnt

    
−c ∇f (x) − ∇ϕ(x)
∗ =
A 0 νnt 0
∗
where νnt = c ν∗.

- Using this Newton step, the Interior-Point Method (i.e., Barrier Method)
can be constructed.

Cenk Toker ELE 704 Optimization 101 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

The Interior-Point Method (Barrier Method):

Given a strictly feasible x ∈ F, c = c(0) (> 0), µ > 1 and tolerance ϵ > 0
- repeat
- Centering step:
- Compute x∗ (c) by minimizing the modified barrier problem

x∗ (c) = argmin (c f (x) + ϕ(x))

s.t. Ax = b

starting at x using the modified Newton’s Method.

- Update: x = x∗ (c)
L
- Stopping criterion: quit if <ϵ
c
- Increase c:, c = µ c

Cenk Toker ELE 704 Optimization 102 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Accuracy of centering:
- Computing x∗ (c) exactly is not necessary since the central path has
no significance beyond the fact that it leads to a solution of the
original problem as c → ∞; inexact centering will still yield a
sequence of points x(k) that converges to an optimal point. Inexact
centering, however, means that the points λ∗ (c) and ν ∗ (c),
computed from the first two equation given in the section titled
"Dual points from central path", are not exactly dual feasible. This
can be corrected by adding a correction term to these formulae,
which yields a dual feasible point provided the computed x is near
the central path, i.e., x∗ (c).

- On the other hand, the cost of computing an extremely accurate

minimizer of c f (x) + ϕ(x), as compared to the cost of computing a
good minimizer of c f (x) + ϕ(x), is only marginally more, i.e., a few
Newton steps at most. For this reason it is not unreasonable to
assume exact centering.

Cenk Toker ELE 704 Optimization 103 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Choice of µ:
- µ provides a trade-off in the number of iterations for the inner and
outer loops.

- small µ: at each step inner loop starts from a very good point, few
inner loop iterations are required, but too many outer loop
iterations may be required.

- large µ: at each step c increases a large amount, the current

starting point may not be a good point for the inner loop. Too
many inner loop iterations may be required, but few outer loop
iterations are required.

- In practice, small values of µ (i.e., near one) result in many outer

iterations, with just a few Newton steps for each outer iteration.
For µ in a fairly large range, from around 3 to 100 or so, the two
effects nearly cancel, so the total number of Newton steps remains
approximately constant. This means that the choice of µ is not
particularly critical; values from around 10 to 20 or so seem to work
well.

Cenk Toker ELE 704 Optimization 104 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Choice of c(0) :
- large c(0) : first run of inner loop may require too many iterations.

- small c(0) : more outer-loop iterations are required.

- One reasonable choice is to choose c(0) so that c(0)

L
is
(0) ∗
approximately of the same order as f (x ) − p , or µ times this
amount. For example, if a dual feasible point (λ, ν) is known, with
duality gap η = f (x(0) ) − ℓ(λ, ν), then we can take c(0) = Lη
.
Thus, in the first outer iteration we simply compute a pair with the
same duality gap as the initial primal and dual feasible points.

- Another possibility is to find c(0) which minimizes

inf ∥c ∇f (x(0) ) + ∇ϕ(x(0) ) + AT ν∥2

ν

Cenk Toker ELE 704 Optimization 105 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Example 25:

Solution:

Cenk Toker ELE 704 Optimization 106 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Cenk Toker ELE 704 Optimization 107 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Cenk Toker ELE 704 Optimization 108 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Example 26:

Solution:

Cenk Toker ELE 704 Optimization 109 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Cenk Toker ELE 704 Optimization 110 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

How to start from a feasible point?

- The interior-point method requires a strictly feasible starting point x(0)

- If such a point is not knows, a preliminary stage, Phase I is run first

Cenk Toker ELE 704 Optimization 111 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Basic Phase I Method:

- Consider gi (x) ≤ 0, i = 1, 2, . . . , L and Ax = b.
L
- We always assume that we are given a point x(0) ∈
T
dom gi (x)
i=1
(0)
satisfying Ax = b.
- Then, we form the following optimization problem

min s
s.t. gi (x) ≤ s, i = 1, 2, . . . , L
Ax = b

in the variables x ∈ RN and s ∈ R.

- s is a bound on the maximum infeasibility of the inequalities and it is to
be driven below zero.
- The problem is always feasible when select s(0) ≥ max gi (x(0) ) together
i
L
(0) T (0)
with the given x ∈ dom gi (x) with Ax = b.
i=1

Cenk Toker ELE 704 Optimization 112 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

- Then, apply the interior-point method to solve the above problem. There
are three cases depending on the optimal value p̄∗
- If p̄∗ < 0, then a strictly feasible solution is reached. Moreover if
(x, s) is feasible with s < 0, then x satisfies gi (x) < 0. This means
we do not need to solve the optimization problem with high
accuracy; we can terminate when s < 0.
- If p̄∗ > 0, then there is no feasible solution.
- If p̄∗ = 0, and the minimum is attained at x∗ and s∗ = 0, then the
set of inequalities is feasible, but not strictly feasible. However, if
p̄∗ = 0 and the minimum is not attained, then the inequalities are
infeasible.

Cenk Toker ELE 704 Optimization 113 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Example 27:

Solution:

Cenk Toker ELE 704 Optimization 114 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Cenk Toker ELE 704 Optimization 115 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Cenk Toker ELE 704 Optimization 116 / 117

 Introduction
Duality Primal Methods
Optimality Conditions Equality Constrained Optimization
Constrained Optimization Algorithms Penalty and Barrier Methods
Interior-Point Methods

Cenk Toker ELE 704 Optimization 117 / 117