ELE8311: Advanced Engineering Mathematics –

Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Hassan A. Bashir

ELE8301: Advanced Engineering Mathematics

Module 4–Optimization: Introduction, Formulation, Linear Programming and
Graphical Methods

4.Introduction

Definitions:

- Literally, Optimization refers to seeking of an alternative having the most cost effective or
highest achievable performance under the given constraints, by maximizing desired factors
and minimizing undesired ones. In other words, it is an art, process, or methodology of
making something (such as a design, system, or decision) as fully perfect, functional, or
effective as possible.

- Mathematically, optimization is a mathematical technique for finding a maximum or

minimum value of a function of several variables subject to a set of constraints, as linear
programming or systems analysis.

In general, optimization principles are of basic importance in modern engineering design and
system operation in various areas. The recent development has been influenced by
computers capable of solving large-scale problems and by the creation of corresponding new
optimization techniques transforming the field to a large subject area of its own.

4.1 Basic Concepts – Constrained and Unconstrained Optimization

i. Objective Function
In an optimization problem, the objective is to optimize (maximize or minimize) some
function 𝑓. This function is called the objective function or cost function.

Maximization: For example, an objective function 𝑓 to be maximized may be the revenue

in a production process, the yield per minute in a chemical process, the mileage per gallon of
a certain type of automobile, the hourly number of customers serviced in a bank, the
hardness of steel alloy, the tensile strength of a rope, or the throughput of a communication
channel.

Minimization: One may wish to minimize 𝑓 if 𝑓 is the cost per unit of producing certain
type of camera, the operating cost of some power plant, the daily loss of heat in a heating
system, the idling time of some lathe, or the delay time of a communication channel.
ii. Variables:
In most optimization problems, the object function 𝑓 depends on several variables

1
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 .
These variables are called control or decision variables since one can control or decide
their values. The number of the control variables determines the dimension of the
optimization problem.

For example, the yield of a chemical process may depend on temperature 𝑥1 , air pressure 𝑥2 ,
moisture content 𝑥3 , cross-sectional area of outlet 𝑥4 , etc.

iii. Constraints
In many problems, the choice of the control variables 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 is not entirely free but is
subject to some additional restrictions arising from the nature of the problem and the
variables. Such restrictions are called constraints.

For example, if 𝑥1 is a production cost, then 𝑥1 ≥ 0, and there are many other variables (time,
weight, distance travelled by a sales person, etc.) that can take only nonnegative values.
Constraints can also have the form of equations (instead of inequalities).

iv. Optimization theory

This develops methods for making optimal choices of the control variables 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 which
maximize (or minimize) the objective function 𝑓, that is, methods for finding optimal values
of 𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 .

v. Unconstrained Optimization
This is concerned with optimization problems where the objective function 𝑓(𝑥1 , 𝑥2 , ⋯ , 𝑥𝑛 ) or
𝑓(𝐱) is not subject to any constraints.

Minimum: an objective function 𝑓 has a minimum at a point 𝐱 = 𝐗 𝟎 in a region 𝑅 where 𝑓 is

defined if:
𝑓(𝐱) ≥ 𝑓(𝐗 𝟎 ) ∶ ∀ 𝑥 ∈ 𝑅

Maximum: Similarly, an objective function 𝑓 has a maximum at a point 𝐱 = 𝐗 𝟎 in a region 𝑅

where 𝑓 is defined if:
𝑓(𝐱) ≤ 𝑓(𝐗 𝟎 ) ∶ ∀ 𝑥 ∈ 𝑅

Minima and Maxima together are called Extrema.

vi. Local Minimum or Maximum

An objective function 𝑓 is said to have a local minimum at point 𝐗 𝟎 if

𝑓(𝐱) ≥ 𝑓(𝐗 𝟎 )

For all 𝐱 in a neighborhood of 𝐗 𝟎 such that

|𝐱 − 𝐗 𝟎 | < 𝑟,
where 𝐗 𝟎 = (𝑋1 , ⋯ , 𝑋𝑛 ) and 𝑟 > 0 is sufficiently small.

2
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
Similarly, 𝑓 has a local maximum at 𝑋0 if 𝑓(𝐱) ≥ 𝑓(𝐗 𝟎 ) for all x satisfying |𝐱 − 𝐗 𝟎 | < 𝑟.

vii. Differentiable Systems

If 𝑓 is differentiable and has an extremum at a point 𝐗 𝟎 in the interior of a region 𝑅 (not on
the boundary), then the partial derivatives 𝜕𝑓/𝜕𝑥1 , ⋯ 𝜕𝑓/𝜕𝑥𝑛 must be zero at 𝐗 𝟎 . These are
the components of a vector that is called the gradient of 𝑓 and denoted by the ∇𝑓. Thus,

∇𝑓(𝐗 𝟎 ) = 0. (4.1)

The point 𝑋0 at which (4.1) holds is called a stationary point of 𝑓.

Note that the above condition (4.1) is necessary for an extremum of 𝑓 at 𝐗 𝟎 in the interior of
𝑅, but is not sufficient.

Examples:
a. If 𝑛 = 1, then for 𝑦 = 𝑓(𝑥), condition (4.1) is 𝑦 ′ = 𝑓 ′ (𝑋0 ) = 0; and, for instance, 𝑦 =
𝑥 3 satisfies 𝑦 ′ = 3𝑥 2 = 0 at 𝑥 = 𝑋0 = 0 where 𝑓 has no extremum but a point of
inflection.
b. Similarly, for 𝑓(𝑥) = 𝑥1 𝑥2 we have ∇𝑓(𝟎) = 𝟎, and 𝑓 does not have an extremum but a
saddle point at 𝟎.

Hence, after solving (4.1), one must still find out whether one has obtained an extremum or
not.

For systems with 𝑛 = 1:

- The conditions 𝑦 ′ (𝑋0 ) = 0, 𝑦 ′′ (𝑋0 ) > 0 guarantee a local minimum at 𝑋0 .

- However, the conditions 𝑦 ′ (𝑋0 ) = 0, 𝑦 ′′ (𝑋0 ) < 0 guarantee a local maximum at 𝑋0 .

For systems with 𝑛 > 1 there exist similar criteria.

Examples – Classical Optimization Exercises:

1. Given a sheet of card board paper of dimensions 20 by 30 inches. Find the value of edge
cuts 𝑥 that maximizes the volume of an open-top box.

3
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Solution

Box Length = 30 − 2𝑥
Box Width = 20 − 2𝑥
Box Height = 𝑥

Box Volume 𝑉(𝑥) = 𝑥(20 − 2𝑥)(30 − 2𝑥) = 4𝑥 3 − 100𝑥 2 + 600𝑥

0 ≤ 𝑥 ≤ 10

To figure out the critical point, we evaluate the first derivative and set it to zero.
𝑑𝑉(𝑥)
The first derivative = 0 and set it to zero to find the turning point.
𝑑𝑥
Check for maxima or minima by evaluating the second derivative 𝑑 2 𝑉(𝑥)/𝑑𝑥 2 .

2. Profit Optimization – Suppose production of pairs of shoes 𝑥 (thousand of pairs) is sold at

a revenue 𝑟(𝑥) = $10 ⋅ 𝑥. If the cost of production is 𝐶(𝑥) = 𝑥 3 − 6𝑥 2 + 15𝑥. Find the number
of pairs 𝑥 that must be produced to maximize the profit.

Solution:
The profit function is 𝑃(𝑥) = 𝑟(𝑥) − 𝐶(𝑥) = 10𝑥 − 𝑥 3 − 6𝑥 2 − 15𝑥 = −𝑥 3 + 6𝑥 2 − 5𝑥.

The critical points are:

𝑃′ (𝑥) = −3𝑥 2 + 12𝑥 − 5 = 0

Therefore,
𝑥1 = 3.528; 𝑥2 = 0.4725

To check for max or min

𝑃′′ (𝑥) = −6𝑥 + 12

Thus,
At 𝑥 = 𝑥1 , 𝑃′′ (𝑥) = −13.128 < 0 (max) and at 𝑥 = 𝑥2 , 𝑃′′ (𝑥) > 0 (min)

Thus, production of 3,528 pairs of shoes will yield a maximum profit of $13,128.

Exercise:
Optimization of Cost of Materials – A rectangular container with an open top needs to have a
volume of 10m3. The length of its base is twice the width. Material for the base costs $10 per
m2. Material for the sides costs $6 per m2. Find the cost of the material for the cheapest
container.

Hint: 𝑉 = 𝑥 ⋅ 2𝑥 ⋅ ℎ = 10; the cost function is 𝐶(𝑥) = 20𝑥 2 + 12𝑥ℎ + 24𝑥ℎ = 20𝑥 2 + 180/𝑥 and
the optimal cost is $163 at 𝑥 = 1.65.

4
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Remarks
Evaluating the solution to systems (4.1) is often difficult. For this reason one generally
prefers solution by iteration. Iterative procedure is a search process that starts at some
point and moves stepwise to points at which 𝑓 gets smaller (in the case of a minimum) or
larger (in the case of a maximum).

4.2 The Method of Steepest Descent or Gradient Method

This is an iterative method in which the basic idea is to find a minimum of 𝑓(𝑥) by repeatedly
computing minima of a function 𝑔(𝑡) of a single variable 𝑡 as follows.

Numerical Methods: The 3-step rule

Given any starting point 𝑥0
i. Determine a search direction 𝑆𝑖 = −∇𝑓(𝑥𝑖 ) (direction of steepest descent)

ii. Calculate the step length 𝜆𝑖 and calculate the next point as:

𝑥𝑖+1 = 𝑥𝑖 + 𝜆𝑖 𝑆𝑖 = 𝑥𝑖 − 𝜆𝑖 ∇𝑓(𝑥𝑖 )

iii. Termination: If ∇𝑓(𝑥𝑖+1 ) ≈ 0 or the steps 𝜆𝑖 ≈ 0, then 𝑥𝑖+1 is the optimum

point. Else, take 𝑥𝑖+1 as the next point and repeat from step i.

Remark: Numerical methods vary based on the method used to evaluate the step length 𝜆𝑖 .

Steepest Descent Procedure

Suppose that 𝑓 has a minimum at 𝐗 𝟎 and we start at a point 𝐱. Then we look for a minimum
of 𝑓 closest to 𝐱 along the straight line in the direction −∇𝑓(𝐱) of maximum decrease of 𝑓 at
𝐱 called the steepest descent.

Thus, we determine the value of a step length 𝑡 using a parameterized equation 𝑧(𝑡):

𝐳(𝑡) = 𝐱 − 𝑡∇𝑓(𝐱),
such that, the point 𝑧(𝑡) corresponds to a point at which the function
𝑔(𝑡) = 𝑓(𝐳(𝑡))
has a minimum. Then 𝐳(𝑡) is taken as the next approximation to 𝐗 𝟎 .

Example 4.1:
Use the steepest descent method to determine the minimum of

𝑓(𝐱) = 𝑥12 + 3𝑥22 ,

starting from 𝐱𝟎 = (6, 3) = 6𝐢 + 3𝐣.
5
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
Solution
By inspection, one could see that 𝑓(𝐱) has a minimum at 𝟎. This helps us understand how
the method works.
We obtain the search direction 𝑆𝑖 = −∇𝑓(𝑥𝑖 ):

2𝑥1
∇𝑓(𝐱) = [ ] = 2𝑥1 𝐢 + 𝟔𝑥2 𝐣
6𝑥2

Evaluate the step length 𝑡 as follows:

𝑥1 2𝑥 (1 − 2𝑡)𝑥1
𝐳(𝑡) = 𝐱 − 𝑡∇𝑓(𝐱) = [𝑥 ] − 𝑡 [ 1 ] = [ ] = (1 − 2𝑡)𝑥1 𝐢 + (1 − 6𝑡)𝑥2 𝐣
2 6𝑥2 (1 − 6𝑡)𝑥2

𝑔(𝑡) = 𝑓(𝐳(𝑡)) = (1 − 2𝑡)2 𝑥12 + 3(1 − 6𝑡)2 𝑥22 .

We then calculate the derivative of 𝑔(𝑡) wrt 𝑡.

𝑔′ (𝑡) = 2(1 − 2𝑡)𝑥12 (−2) + 6(1 − 6𝑡)𝑥22 (−6),

Setting 𝑔′ (𝑡) = 0 and solving for 𝑡 gives

𝑥12 + 9𝑥22
𝑡= .
2𝑥12 + 54𝑥22

Starting from 𝑥0 = (6, 3), we compute the next values in Table 4.1 which are shown in Figure
4.1. as follows:
𝑥𝑖+1 = 𝑥𝑖 + 𝜆𝑖 𝑆𝑖 = 𝑥𝑖 − 𝑡𝑖 ∇𝑓(𝑥𝑖 )

Figure 4.1: The Method of Steepest Descent on Example 4.1

6
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Table4.1: The Method of Steepest Descent, Computations in Example 4.1

Exercises:
1. Do three (3) steepest descent steps when:

2. Apply Steepest descent on the following general cases

3.
Verify that in Example 4.1, successive gradients are orthogonal. Why?

What happens if you apply steepest descent on 𝑓(𝑥) = 𝑥12 + 𝑥22 ?

7
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
4.3 Constrained Optimization - Linear Programming (LP)
Linear programming (linear optimization) consists of methods for solving optimization
problems with constraints, that is, methods for finding a maximum (or a minimum) 𝑥 =
[𝑥1 , ⋯ , 𝑥𝑛 ] of a linear objective function

𝑧 = 𝑓(𝑥) = 𝑎1 𝑥1 + 𝑎2 𝑥2 + ⋯ + 𝑎𝑛 𝑥𝑛

Satisfying the constraints which are in form of linear inequalities, such as 3𝑥1 + 4𝑥2 ≤ 36 or
𝑥1 ≥ 0, etc.

Problems of this kind arise frequently in production, inventory management, bond trading,
operation of power plants, routing delivery vehicles, airplane scheduling, and so on. Progress
in Computer Technology has made it possible to solve programming problems involving
hundreds or thousands or more variables.

Example 4.2
Production Plan – Energy Savers, Inc., produces heaters of types 𝑆 and 𝐿. The wholesale
price is $40 per heater for S and $88 for 𝐿. Two time constraints results from the use of two
machines 𝑀1 and 𝑀2 . On 𝑀1 one needs 2min for an 𝑆 heater and 8min for an 𝐿 heater. On
𝑀2 one needs 5 min for an S heater and 2 min for an L heater. Determine production figures
𝑥1 and 𝑥2 for 𝑆 and 𝐿 respectively (number of heaters produced per hour) so that the hourly
revenue
𝑧 = 𝑓(𝑥) = 40𝑥1 + 88𝑥2
is maximum.

Solution
Production figures 𝑥1 and 𝑥2 must be nonnegative. Hence the objective function to be
maximized and the four constraints are:

8
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Figure 4.2: Linear Programming on Example 4.2

The graph in Figure 4.2 shows the plots for systems (0 to 4) with the constancy lines
𝑧 = 𝑐𝑜𝑛𝑠𝑡
marked (0). The constancy lines are lines of constant revenue obtained from the objective
function. Their slope is −40/88 = −5/11. To increase the revenue z, one must move the line
upward (parallel to itself) as shown in Figure 4.2.

The shaded area (quadrangle) in Figure 4.2 is called the feasible region. It is the set of all
feasible solutions, meaning solutions that satisfy all four constraints. Figure 4.2 also lists
the revenue at points 𝑂, 𝐴, 𝐵, 𝐶 . The optimal solution is obtained by moving the line of
constant revenue up as much as possible without leaving the feasible region completely.

It could be seen that the optimum is reached when the line passes through 𝐵 , the
intersection (10, 5) of (1) and (2), at that point, the optimal revenue

𝑧𝑚𝑎𝑥 = 40 ⋅ 10 + 88 ⋅ 5 = $840

is obtained by producing twice as many 𝑆 heaters as 𝐿 heaters.

Remarks:
- The graphical method demonstrated in Example 4.2 is only possible because the
problem is limited to only two variables 𝑥1 and 𝑥2 . However, most practical
problems involve much more than two variables.
- Problems of example 4.2 cannot be solved by setting certain partial derivatives
equal to zero, because crucial to such problems is the feasible region in which the
control variables are allowed to vary. Hence more suitable solution methods are
necessary.

9
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

4.3.1 Normal form of Linear Programming Problem

In general, constraints in linear programming can be written more uniformly. Consider the
system described in Example 4.2, the constraints in (1),

2𝑥1 + 8𝑥2 ≤ 60.

can be written as: 60 − 2𝑥1 − 8𝑥2 ≥ 0, that is, the quantity𝑥3

𝑥3 = 60 − 2𝑥1 − 8𝑥2
is nonnegative.

Hence, the original inequality can now be written as an equation

2𝑥1 + 8𝑥2 + 𝑥3 = 60,

where
𝑥3 ≥ 0.

𝑥3 is a nonnegative auxiliary variable introduced for converting inequalities to equations.

Such a variable is called a slack variable, because it takes up the slack (difference)
between the two sides of the inequality.

Example 4.3
Conversion of inequalities in Example 4.2 by the use of slack variables – with the aid of two
slack variables 𝑥3 , 𝑥4 one can write the linear programming problem in Example 4.2 in the
following standard form.

Maximize 𝑓 = 40𝑥1 + 88𝑥2

Subject to the constraints

We now have 𝑛 = 4 variables and 𝑚 = 2 (linearly independent) equations, so that two of the
four variables, for example, 𝑥1 , 𝑥2 , determine the others.

10
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
Notice that each of the four sides of the quadrangle in Figure 4.2 now has an equation of the
form 𝑥𝑖 = 0:

Hence, at any vertex of the quadrangle, 𝑛 − 𝑚 = 4 − 2 = 2 of the variables are zero and the
others are nonnegative. Thus, at vertex 𝐴, we have 𝑥2 = 0, 𝑥4 = 0, and so on.

The Normal Form

The above example suggests that a general linear optimization problem can be brought to a
normal form as follows:

Maximize 𝑓 = 𝑐1 𝑥1 + 𝑐2 𝑥2 + ⋯ + 𝑐𝑛 𝑥𝑛 (4.2a)

Subject to the constraints: (4.2b)

With all 𝑏𝑗 nonnegative. (if a 𝑏𝑗 < 0, multiply the equation by -1). Here 𝑥1 , ⋯ , 𝑥𝑛 include the
slack variables (for which the 𝑐𝑗 ’s in 𝑓 are zero). We assume that the equations in (4.2b) are
linearly independent. Then if we chose values for 𝑛 − 𝑚 of the variables, the system uniquely
determines the others. Of course, since 𝑥1 ≥ 0, ⋯ , 𝑥𝑛 ≥ 0, the choice is not entirely free.

The general form above includes minimization of the objective function 𝑓 since that
corresponds to maximizing – 𝑓.

Definitions:

i. Feasible solution
An n-tuple (𝑥1 , ⋯ , 𝑥𝑛 ) that satisfies all the constraints in (4.2b) is called a feasible point
or feasible solution.

11
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
ii. Optimal Solution
A feasible solution is called an optimal solution if for it the objective function 𝑓 becomes
maximum (or minimum) compared with the values of 𝑓 at all feasible solutions.

iii. Basic feasible solution

This is a feasible solution for which at least 𝑛 − 𝑚 of the variables 𝑥1 , ⋯ , 𝑥𝑛 are zero. For
instance, in Example 4.2, we have 𝑛 = 4 variables and 𝑚 = 2 equations and the basic
feasible solutions are the four vertices of the quadrangle 𝑂, 𝐴, 𝐵, 𝐶 in Figure 4.2. Here the
basic feasible solution 𝐵 is an optimal solution (and is the only optimal solution in this
case).

Theorem – Optimal Solution

Some optimal solution of a linear programming problem (4.2) is also a basic feasible solution
of (4.2).

A problem can have many optimal solutions and not all of them may be basic feasible
solutions; but the theorem above guarantees that one can find an optimal solution by
searching through the basic feasible solutions only – a great simplification!

However, searching through the basic feasible solutions still involve a lot of work (there are
𝑛 𝑛
( ) = ( ) different ways of equating 𝑛 − 𝑚 of the 𝑛 variables to zero) even when 𝑛 and
𝑛−𝑚 𝑚
𝑚 are relatively small. Hence a more systematic approach is needed.

Exercises:
1. Why are slack variables always nonnegative? How many of them do we need?
2. Could we find a profit 𝑓(𝐱) = 𝑎1 𝑥1 + 𝑎2 𝑥2 whose maximum is at the interior of the
quadrangle in Figure 4.2? Give reason for your answer.

3. Describe and graph the region in the first quadrant of the 𝑥1 𝑥2 −plane determined by the
following inequalities:

12
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

4. Maximize the given objective function 𝑓 subject to the given constraints:

4.4 The Simplex Method

In 1948, G. B. Dantzig published an iterative method, called the simplex, for finding an
optimal solution of a linear programming problem when written in its normal form. To find
the optimal solution in such a case we only need to consider the basic feasible solutions.

In the Simplex method, one proceeds stepwise from one basic feasible solution to another in
such a way that the objective function 𝑓 always increases its value.

13
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Example 4.4:
Consider the LP in Example 4.2, in its original form the problem concerned the maximization
of the objective function:

Converting the first two inequalities to equations by introducing two slack variables 𝑥3 , 𝑥4 we
obtained the normal form of the problem.

Together with the objective function written as an equation 𝑧 − 40𝑥1 − 88𝑥2 = 0, we have:

where 𝑥1 ≥ 0, ⋯ , 𝑥4 ≥ 0.

Equation (4.4) is a linear system of equations. To find the optimal solution, one may consider
its augmented matrix.

14
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

The augmented matrix in (4.5) is called the simplex tableau or simplex table (the initial
simplex table 𝐓𝟎 ).

Basic and Nonbasic Variables

Every simplex table contains of two kinds of variables 𝑥𝑗 , basic and nonbasic variables.
The basic variables are those whose columns have only one nonzero entry. Thus, from
(4.5) 𝑥3 , 𝑥4 are basic and 𝑥1 , 𝑥2 are nonbasic.

Every simplex table gives a basic feasible solution. It is obtained by setting the nonbasic
variables to zero. Thus, simplex table 𝐓𝟎 gives the basic feasible solution

Iteration
The optimal solution (its location and value) is now obtained stepwise by pivoting, designed
to take us to basic feasible solutions with higher and higher values of 𝑧 until the maximum of
𝑧 is reached.

The choice of the pivot equation and the pivot point are different from that in the Gauss
elimination. This is because the decision variables 𝑥1 , 𝑥2 , 𝑥3 , 𝑥4 are restricted to nonnegative
values.

Iteration 1.
- Operation 𝑶𝟏 : Selection of the pivot column – Choose the first column with a
negative entry in Row 1. In simplex table 𝐓𝟎 (4.5) this is Column 2 (because of the -
40).
- Operation 𝑶𝟐 : Selection of the row of the pivot – Divide the right sides of the simplex
table in (4.5) by the corresponding entries of the column just selected (60/2 = 30,
60/5 = 12)and take as the pivot the equation that gives the smallest quotient. Thus,
the pivot is 5 because 60/5 is the smallest.
- Operation 𝑶𝟑 : Elimination by Row Operations – Eliminates the entries above and
below the pivot (as in Gauss-Jordan operation).

Applying the three operations above give the next simplex table 𝐓𝟏 (4.6).

15
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

From the simplex table 𝐓𝟏 (4.6) it can be seen that the basic variables are now 𝑥1 , 𝑥3 and
nonbasic variables are 𝑥2 , 𝑥4 . Setting the latter to zero gives the next basic feasible solution
given by 𝐓𝟏 ,

This corresponds to vertex 𝐴 in Figure 4.2. Thus, the first iteration moved us from vertex
𝑂: (0, 0) with 𝑧 = 0 to vertex 𝐴: (12, 0) with the greater 𝑧 = 480. The increase is a result of
elimination of a term (−40𝑥1 ) with a negative coefficient. Hence elimination is applied only to
negative entries in Row 1 but to no others – Justifying the selection of the pivot column.

Suppose the second row of 𝐓𝟎 (4.5) was used as the pivot (thus 2 instead of 5), then we
would have obtained 𝑧 = 1200 (Check!), but this line of constant revenue 𝑧 = 1200 lies
entirely outside the feasible region in Figure 4.2. This is the reason for choosing 5 as the
pivot since it gave the smallest quotient (60/5 = 12).

Iteration 2.
- Operation 𝑶𝟏 : Select Column 3 of 𝐓𝟏 (4.6) as the column of the pivot (i.e.,−72 < 0).
- Operation 𝑶𝟐 : We have 36/7.2 = 5 and 60/2 = 30. Select 7.2 as the pivot since 5 <
30.
- Operation 𝑶𝟑 : Elimination by Row Operations gives the next simplex table 𝐓𝟐 (4.7).

In 𝐓𝟐 , 𝑥1 , 𝑥2 are basic and 𝑥3 , 𝑥4 are nonbasic. Setting the nonbasic to zero yields a new basic
feasible solution with higher revenue.

16
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
This corresponds to vertex 𝐵 in Figure 4.2. in this step 𝑧 has increased from 480 to 840 due
to the elimination of -72 in 𝐓𝟏 .

Since the simplex table 𝐓𝟐 contains no more negative entries in Row 1, it can be concluded
that
𝑧 = 𝑓(10, 5) = 40 ⋅ 10 + 88 ⋅ 5 = 840

is the maximum possible revenue.

Thus, the maximum revenue is obtained if the company produces twice as many 𝑆 heaters as
𝐿 heaters.

Minimization
To minimize𝑧 = 𝑓(𝐱), take as the columns of the pivots those whose entry in Row 1 is
positive (instead of negative). Then in such a column consider only positive entries and take
as pivot a row entry that yield the smallest quotient (as before).

Exercises:
1. Write the following problems in normal form and optimize by the simplex method
assuming all 𝑥𝑗 to be nonnegative.

4.4.1 Difficulties in Simplex Method

Recall that in simplex method progress is made in the search for optimal solution by
iteratively stepping from one basic feasible solution to another. Occasionally, two kinds of
difficulties may occur.

i. Degeneracy
A degenerate feasible solution is a feasible solution at which more than the usual number,𝑛 −
𝑚, of the variables are zero. Here 𝑛 is the number of variables (including slack variables) and
𝑚 is the number of constraints (not including the bounds 𝑥𝑗 ≥ 0).

In the last example (Example 4.2), 𝑛 = 4 and 𝑚 = 2 , and the occurring basic feasible
solutions were nondegenerate; 𝑛 − 𝑚 = 2 were zero in each such solution.

17
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
Remedy: In the case of a degenerate feasible solution we do an extra elimination step in
which a basic variable that is zero for that solution becomes nonbasic (and a nonbasic
variable becomes basic instead). See the following example.

Example 4.5:
Simplex method with Degenerate feasible solution
AB Steel, Inc., produces two kinds of iron 𝐼1 , 𝐼2 by using three kinds of raw material 𝑅1 , 𝑅2 , 𝑅3
(scrap and two kinds of ore) as shown below. Maximize the daily profit.

Solution
Let 𝑥1 and 𝑥2 denote the amount (in tons) of iron 𝐼1 and 𝐼2 respectively, produced per day.
Then our problem is as follows.

By introducing slack variables 𝑥3 , 𝑥4 , 𝑥5 we obtain the normal form of the constraint

18
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

From the normal form, we create the initial simplex table 𝐓𝟎 as follows:

From 𝐓𝟎 we see that 𝑥1 , 𝑥2 are nonbasic variables and 𝑥3 , 𝑥4 , 𝑥5 are basic. With 𝑥1 = 0, 𝑥2 = 0
we obtain a basic feasible solution

This is vertex 𝑂: (0,0) in Figure 4.3. We have 𝑛 = 5 variables, 𝑚 = 3 constraints and 𝑛 − 𝑚 =

2 variables equal to zero in our solution, thus vertex 𝑂 is nondegenerate.

Figure 4.3: Linear Programming with Degenerate Vertex (A) - Example 4.5

19
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
Iteration 1.
- Operation 𝑶𝟏 : Select Column 2 of 𝐓𝟎 as the column of the pivot (i.e.,−150 < 0).
- Operation 𝑶𝟐 : Row selection of pivot. We have 16/2 = 8 , 8/1 = 8 and 3.5/0 not
possible. Hence Row 2 or 3 could be chosen. If we choose Row 2, the pivot is 2.
- Operation 𝑶𝟑 : Elimination by Row Operations gives the next simplex table 𝐓𝟏 .

Basic variables are: 𝑥1 , 𝑥4 , 𝑥5 and the nonbasic are 𝑥2 , 𝑥3 . Setting the nonbasic to zero, we
obtain from 𝐓𝟏 the basic feasible solution

This is 𝐴: (8, 0) in Figure 4.3. This solution is degenerate because 𝑥4 = 0 (in addition to 𝑥2 =
0, 𝑥3 = 0); geometrically, the straight line 𝑥4 = 0 also passes through 𝐴. This requires the
next step, in which 𝑥4 will become nonbasic.

Iteration 2.
- Operation 𝑶𝟏 : Select Column 3 of 𝐓𝟏 as the column of the pivot (i.e.,−225 < 0).
- Operation 𝑶𝟐 : Row selection of pivot. We have 16/1 = 16, 0/0.5 = 0. Hence ½ must
serve as the pivot.
- Operation 𝑶𝟑 : Elimination by Row Operations gives the next simplex table 𝐓𝟐 .

Now the basic variables are 𝑥1 , 𝑥2 , 𝑥5 and the nonbasic are 𝑥3 , 𝑥4 . Hence 𝑥4 has become
nonbasic as intended. By equating the nonbasic variables to zero we obtain:

This is still 𝐴: (8, 0) in Figure 4.3 and 𝑧 has not increased. But this opens the way to the
maximum, which we reach in the next step.

20
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Iteration 3.
- Operation 𝑶𝟏 : Select Column 4 of 𝐓𝟐 as the column of the pivot (i.e.,−150 < 0).
- Operation 𝑶𝟐 : Row selection of pivot. We have 16/2 = 8, 0/−0.5 = 0, 3.5/1 = 3.5.
Hence we can take 1 as the pivot (with -0.5 as the pivot we would not leave A, check
it!).
- Operation 𝑶𝟑 : Elimination by Row Operations gives the next simplex table 𝐓𝟑 .

The basic variables are 𝑥1 , 𝑥2 , 𝑥3 and nonbasic are 𝑥4 , 𝑥5 . The basic feasible solution is
obtained by equating the nonbasic variables to zero as follows:

That is vertex 𝐵: (4.5, 3.5) in Figure 4.3. Since Row 1 has no more negative entries, we have
reached the maximum daily profit

𝑧𝑚𝑎𝑥 = 𝑓(4.5, 3.5) = 150 ⋅ 4.5 + 300 ⋅ 3.5 = $1725.

This is obtained by using 4.5 tons of iron 𝐼1 and 3.5 tons of iron 𝐼2 .

ii. Difficulty in Starting

Occasionally, it may hard to find a basic feasible solution to start from. In such a case
the idea of artificial variable (or several such variables) is helpful. See the following
example.

Example 4.6.

Simplex Method with Starting Difficulty – Artificial Variable

21
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Solution
By introducing slack variables, we achieve the following normal form

Notice that the first slack variable is negative (or zero). This makes 𝑥3 nonnegative within the
feasibility region (and negative outside).

Figure 4.4: Linear Programming with Difficult Start - Example 4.6

From the normal form of the problem we obtain the simplex table 𝐓𝟎 as follows:

22
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Now, 𝑥1 , 𝑥2 are nonbasic and we would like to take 𝑥3 , 𝑥4 , 𝑥5 as basic variables. By our usual
process of equating the nonbasic variables to zero we obtain from this table:

Notice that 𝑥3 < 0 indicates the vertex (0, 0) lies outside the feasible region. Thus, we cannot
proceed immediately.

Now, instead of searching for other basic variables, we use the following idea. We solve the
second equation from the normal form of the constraints for 𝑥3 , we have

1
𝑥3 = −1 + 𝑥1 − 𝑥2 .
2
To this we now add a variable 𝑥6 on the right.

1
𝑥3 = −1 + 𝑥1 − 𝑥2 + 𝑥6 .
2

The added variable 𝑥6 is called an artificial variable and is subject to the constraint 𝑥6 ≥ 0.

One must take care that the artificial variable 𝑥6 (which is not part of the original problem)
will disappear eventually. This can be accomplished by adding a term – 𝑀𝑥6 with very large
𝑀 to the objective function. Because of modified constraints in the equations above, we have
a modified objective function for the extended problem.

1
𝑧̂ = 𝑧 − 𝑀𝑥6 = 2𝑥1 + 𝑥2 − 𝑀𝑥6 = (2 + 𝑀)𝑥1 + (1 − 𝑀) 𝑥2 − 𝑀𝑥3 − 𝑀.
2

Now, the simplex table corresponding to the new or extended problem is as follows:

The last row of the simplex table results from the above modified equation of 𝑥3 with the
artificial variable 𝑥6 written as:

1
𝑥1 − 𝑥2 − 𝑥3 + 𝑥6 = 1.
2

23
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

From 𝐓𝟎 we can now start taking 𝑥4 , 𝑥5 , 𝑥6 as the basic variables and 𝑥1 , 𝑥2 , 𝑥3 as the nonbasic
variables. Column 2 has a negative first entry. We can take the second entry (1 in Row 2) as
the pivot. This gives:

The simplex table 𝐓𝟏 corresponds 𝑥1 = 1, 𝑥2 = 0 , i.e. vertex 𝐴 in Figure 4.4, 𝑥3 = 0, 𝑥4 =

1, 𝑥5 = 3, 𝑥6 = 0. We can now drop Row 5 and Column 7, thus getting rid of the artificial
variable 𝑥6 as desired. We now have:

In Column 3 we choose 3/2 as the next pivot. This gives the following simplex table 𝐓𝟑 .

24
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

The simplex table 𝐓𝟑 corresponds to 𝑥1 = 2, 𝑥2 = 2 (vertex B in Figure 4.4), 𝑥3 = 0, 𝑥4 =

2, 𝑥5 = 0.

In column 4 we choose 4/3 as the pivot by the usual principle. This gives

3
This corresponds to 𝑥1 = 3, 𝑥2 = 1 (vertex C in Figure 4.4), 𝑥3 = , 𝑥4 = 0, 𝑥5 = 0. This is the
2
maximum 𝑓𝑚𝑎𝑥 = 𝑓(3, 1) = 7.

Matlab: On digital computers, the optimization of LP problems become very simple. Try out: linprog()
In Matlab, the standard form of the LP is as follows:

The constraints are re-written as: −4𝑥1 + 6𝑥2 ≤ 9; 𝑥1 + 𝑥2 ≤ 4. The problem can then be solved by:

>> f = [1; -2]; % Defining the Objective function

>> A = [-4, 6; 1, 1]; b = [9; 4]; % LHS and RHS inequalities
>>Aeq = []; beq = []; % No equality constraints
>>lb = [0; 0]; ub = []; % Lower bound, no upper bound
>> x0 = [1; 1]; % Initial guess – starting point
>> x = linprog(f, A, b, Aeq, beq, lb, up, x0)

25
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

> In linprog at 235

Optimization terminated.
x=
1.5000
2.5000

Exercises:

Remarks
Following two conditions need to be met before applying the simplex method:

1. The right-hand side of each constraint inequality should be non-negative. In case, any
linear programming problem has a negative resource value, then it should be
converted into positive value by multiplying both the sides of constraint inequality by
“-1”.
2. The decision variables in the linear programming problem should be non-negative.

Thus, the simplex algorithm is efficient since it considers few feasible solutions, by following
the vertices of the polyhedron (polytope), to determine the optimal solution to the linear
programming problem.

26
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
4.5 Duality in Linear Programming

Definition: The Duality in Linear Programming states that every linear programming
problem has another linear programming problem related to it and thus can be derived from
it. The original linear programming problem is called “Primal,” while the derived
linear problem is called “Dual.”

Before solving for the duality, the original linear programming problem must to be formulated
in its standard form. Standard form means, all the variables in the problem should be non-
negative and “≥”, ”≤” sign is used in the minimization case and the maximization case
respectively.

Primal-Dual LP – The Intuition

Example 4.7: Let’s look at the following primal LP problem,
Maximize: 𝑍 = 3𝑥1 + 4𝑥2

1
Subject to: 𝑥 + 2𝑥2 ≤ 30
2 1
3𝑥1 + 𝑥2 ≤ 25
𝑥1 , 𝑥2 ≥ 0
Suppose we multiply the first constraint by a constant, say 6,
1
6 ( 𝑥1 + 2𝑥2 ) ≤ 6(30) ⇒ 3𝑥1 + 12𝑥2 ≤ 180
2
Notice that each one of the coefficients of the new constraint ≥ the corresponding one in the
objective function.
Therefore, since this constraint is ≤ 180, then 180 is also an upper bound for the
objective function:
∴ 𝑍 = 3𝑥1 + 4𝑥2 ≤ 3𝑥1 + 12𝑥2 ≤ 180 ⇒ 𝑍 ≤ 180
Similarly,
If the 2nd constraint is multiplied by a constant, say 4, we have:
4(3𝑥1 + 𝑥2 ) ≤ 4(25) ⇒ 12𝑥1 + 4𝑥2 ≤ 100

Hence, 100 is another upper bound to our objective function 𝑀𝑖𝑛 𝑍

∴ 𝑍 = 3𝑥1 + 4𝑥2 ≤ 12𝑥1 + 4𝑥2 ≤ 100 ⇒ 𝑍 ≤ 100

In the same way, linearly combining the constraints we have:

A tighter upper bound of 85:
1
2 ⋅ [ 𝑥1 + 2𝑥2 ≤ 30] + 1 ⋅ [3𝑥1 + 𝑥2 ≤ 25] ⇒ 4𝑥1 + 5𝑥2 ≤ 85
2
∴ 𝑍 = 3𝑥1 + 4𝑥2 ≤ 4𝑥1 + 5𝑥2 ≤ 85 ⇒ 𝑍 ≤ 85
Hence, to find an upper bound for the LP problem, we only need to find a linear combination
of the primal problem constraints with a suitable set of multipliers, say 𝑦1 , 𝑦2 , …
1
𝑦1 ⋅ [ 𝑥1 + 2𝑥2 ≤ 30] + 𝑦2 ⋅ [3𝑥1 + 𝑥2 ≤ 25]
2

27
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
1
= ( 𝑦1 + 3𝑦2 ) 𝑥1 + (2𝑦1 + 𝑦2 )𝑥2 ≤ 30𝑦1 + 25𝑦2
2
Hence, in general,
1
∴ 𝑍 = 3𝑥1 + 4𝑥2 ≤ ( 𝑦1 + 3𝑦2 ) 𝑥1 + (2𝑦1 + 𝑦2 )𝑥2 ≤ 30𝑦1 + 25𝑦2 ⇒ 𝑍 ≤ 30𝑦1 + 25𝑦2
2
Therefore, we just end up with a new problem that says: to maximize the original primal
problem, we need to minimize a new problem subject to new set of constraints as follows:

Minimize: 𝑍 = 30𝑦1 + 25𝑦2

1
Subject to: 𝑦 + 3𝑦2 ≥ 3
2 1
2𝑦1 + 𝑦2 ≥ 4
𝑦1 , 𝑦2 ≥ 0
This is called the Dual of the original Primal LP.

Example 4.8:
Consider the following Linear Program:

Apply the Simplex algorithm to show that the solution to the LP, which is the maximum value
of the objective function, is 8. Check!

Note that if we take linear combinations of the three equations judiciously, we can get an
upper-bound on the objective function. For instance:

This gives an upper bound on the objective function which we sought to maximize. In fact,
this is a tight upper bound. Is it a coincidence? Duality explores these issues.

Let us associate a nonnegative variable with each of the three equations, say 𝑦1 , 𝑦2 , 𝑦3 .

Since 𝑦𝑖 ′s are nonnegative we can get an upper bound on the objective function whenever:

28
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

and the upper bound is precisely:

So we get a new linear program:

As we will see, the minimum value of this linear program will give us an optimum upper
bound on the optimum value of the original linear program.

Example 4.9: Primal-Dual conversion by Inspection

Maximize: 𝑍 = 50𝑥1 + 30𝑥2

Subject to: 4𝑥1 + 3𝑥2 ≤ 100

3𝑥1 + 5𝑥2 ≤ 150
𝑥1 , 𝑥2 ≥ 0
The concept of duality can be applied to the above original linear programming problem as:

Minimize: 𝐺 = 100𝑦1 + 150𝑦2

Subject to: 4𝑦1 + 3𝑦2 ≥ 50

3𝑦1 + 5𝑦2 ≥ 30
𝑦1 , 𝑦2 ≥ 0
The following observations were made while forming the dual linear programming problem:

1. The primal or original linear programming problem is of the maximization type while
the dual problem is of minimization type.
2. The constraint values 100 and 150 of the primal problem have become the coefficient
of dual variables 𝑦1 and 𝑦2 in the objective function of a dual problem and while the
coefficient of the variables in the objective function of a primal problem has become
the constraint value in the dual problem.
3. The first column in the constraint inequality of primal problem has become the first
row in a dual problem and similarly the second column of constraint has become the
second row in the dual problem.
4. The directions of inequalities have also changed, i.e. in the dual problem, the sign is
the reverse of a primal problem. Such that in the primal problem, the inequality sign
was “≤” but in the dual problem, the sign of inequality becomes “≥”.

Note: The dual of a dual problem is the primal problem. Check!

29
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
Note: The dual of the linear program (𝑨, 𝑏, 𝑐), is (−𝑨𝑇 , −𝑐, −𝑏). i.e.,

Exercise 1:
Find the dual of the primal problem
Maximize: 𝑍 = 3𝑥2 + 5𝑥3
Subject to: 𝑥1 + 𝑥2 ≤ 2
2𝑥1 + 𝑥2 + 6𝑥3 ≥ 6
𝑥1 − 𝑥2 + 3𝑥3 = 4
𝑥1 , 𝑥2 , 𝑥3 ≥ 0

Duality Properties
Weak Duality property: In a maximization LP, the value of the primal LP for any feasible
solution is bounded from above by any feasible solution to its dual. In other words, for every
feasible solution 𝑥 of the primal and 𝑦 of the dual:

𝑐 𝑇 𝑥 ≤ 𝑏𝑇 𝑦

Unboundedness property: If primal (dual) problem has an unbounded solution, then the dual
(primal) is infeasible.

Strong Duality property: If the primal (dual) problem has a finite optimal solution, then so
does the dual (primal) problem and the values of primal and dual are equal at their optimal
solutions. The equality holds:
𝑐 𝑇 𝑥 = 𝑏 𝑇 𝑦.

Thus, Strong Duality Theorem suggests that – If both primal and dual are bounded and
feasible then,

Graphically:

30
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
Remarks:
An important application of the duality theory is in the definition of prices of resources.
This is the case in the Location Marginal Price (LMP) algorithm used in electricity markets.
As a matter of fact, this algorithm consists in minimizing the total cost of production of
electricity with constraints on the demand at each geographical location. The demand can be
viewed as resource. If the demand at a location increases, then it would cause an
incremented cost of production of electricity.

In the electricity market, this incremented cost is considered as the price of electricity at that
point. It is a great model that takes into account electricity losses for instance. If a
demand is at a location far from generators that induce plenty of losses, an increase of the
demand at that location will induce an important incremental production, hence a higher
incremental cost of production. Therefore, the dual variable associated to the demand, which
is the price, will be higher.
Economic Interpretation:
If we interpret our primal LP problem as a classical "resource allocation" problem, its
dual can be interpreted as a "resource valuation" problem.

Resource Allocation and Valuation

In the context of linear programming, the primal problem typically represents a
resource allocation problem. This means that we are trying to allocate limited resources
to various activities in order to maximize or minimize a certain objective function.

The dual problem, on the other hand, can be interpreted as a resource valuation
problem. It determines the optimal values (or shadow prices) of the resources, which
represent the maximum amount we would be willing to pay for an additional unit of each
resource.

Example 4.10: The Smart Robber

A smart robber wants to steal as much gold and dollar bills as he can. He is limited by the
volume of his backpack and the maximal weight he can carry.

Now, let’s notice that we can write the problem as follows.

This primal problem represents allocating two resources (𝑣𝑜𝑙𝑢𝑚𝑒𝑔𝑜𝑙𝑑 and 𝑣𝑜𝑙𝑢𝑚𝑒𝑏𝑖𝑙𝑙𝑠 )
to maximize a certain objective function.

31
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
The problem formulation above is what is called the primal linear program. The dual LP
will totally change our understanding of the problem.

In the primal LP, constraints had constant numbers on their right. These constant numbers
are our resources. They say what we are able to do concerning each constraint.

The dual problem, however, consists in evaluating how much our combined resources are
worth. If the offer meets the demand, our resources will be just as much as their potentials,
which is the worth of the robbery.

In the dual problem we will attribute values to the resources (as in “how much they are
worth”). These values are called the dual variables.

In our case, we have two constraints, so we will have 2 dual variables.

- The first dual variable, let’s call it 𝑣𝑎𝑙𝑢𝑒𝑣𝑜𝑙𝑢𝑚𝑒 refers to the value of one unit of
volume.
- The second dual variable refers to the value of one unit of weight. 𝑣𝑎𝑙𝑢𝑒𝑤𝑒𝑖𝑔ℎ𝑡 .

Now, we can write the value of the robbery with these two new variables:

Now, how are the values per volume and per weight determined?

“If I wanted to sell my resources, potential buyers are going to minimize the value of my
resources. So their valuations are the minimum of the total value. But as a seller, I will argue
that each of my resource is worth a lot, because it enables the robbery of more gold and
more bills.”

32
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Obviously, the values of resources depend on the actual values of gold and bills per volume.

For the value of gold, if the constraints enabled us to steal one more volume of gold, then
incremental value of the robbery would be at least the value of this one volume of gold. It
could be more, if we use the new constraints to steal something else than gold that is worth
more. In other words, if the total volume enabled us to steal one more unit of volume of gold,
and if we could carry one more unit of weight of one volume of gold, then the value of this
incremental steal would be at least the value of one more volume of gold.

Thus,

Similarly, for the dollar bill,

In fact, we can do this reasoning with any variable of the primal problem and ask ourselves
the question:
“If we add one unit of the variable, how will it affect the total valuation?”

From that we deduce a constraint on dual variables. Thus, any variable in the primal
problem is associated with a constraint in the dual problem and vice-versa.

Finally, notice the fact that if we increase the total volume, then we have more possibilities
for the primal variables, which are the volumes of stolen gold and bills. Therefore, the value
of a unit of volume cannot be negative. This adds two more constraints on the sign of
dual variables.

The Dual LP is therefore:

In the dual problem, the variables 𝑣𝑎𝑙𝑢𝑒𝑣𝑜𝑙𝑢𝑚𝑒 and 𝑣𝑎𝑙𝑢𝑒𝑤𝑒𝑖𝑔ℎ𝑡 represent the shadow
prices of the two resources. Solving the dual problem will give us the optimal values of
these shadow prices, which can be interpreted as the maximum amount we would be willing
to pay for an additional unit of each resource.

The most important result is the strong duality property: optimal values of the primal
and dual problems are the same. We can solve the primal problem simply by solving the
dual problem! And sometimes, the dual problem can be much simpler than the primal one.

33
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

The Feasible Regions for the Primal and Dual Problems

Exercises:
1. Suppose an automobile company wishes determine the optimal numbers of cars and
trucks to build in light of capacity constraints. There are two decision variables: the number
of cars 𝑥1 in thousands and the number of trucks 𝑥2 in thousands. The linear program is
given by:

i. Show that the optimal solution is given approximately by 𝑥1 = 20.4 and 𝑥2 = 6.5,
generating a profit of about $77.3 million.
ii. Sketch the constraints, feasible region and identify the optimal solution
geometrically.
iii. Write the equivalent dual problem and check whether the min of dual equals the
optimal in (i) above.

2. A firm producing three types of automobile trailers. Let the decision variables be:

𝑥1 = number of flat-bed trailers produced per month,

𝑥2 = number of economy trailers produced per month,
𝑥3 = number of luxury trailers produced per month.

34
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
The constraining resources of the production operation are the metalworking and
woodworking capacities measured in days per month. The linear program to maximize
contribution to the firm’s overhead (in hundreds of dollars) is:

Find the optimal solution and compare it against that of the dual LP.

35
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
4.6 Combinatorial Optimization

This section focuses on important classes of optimization problems that concern graphs and
digraphs as they arise in real-life applications. For example, in transportation of minimum
cost or time, best assignment of workers to jobs, most efficient use of communication
networks, and many others.

Consider the following graph

Check:
Cycle is a closed path; path has each vertex traced ones; trail has each edge traced once.

4.6.1 Shortest Path Problems

To define the concept of a shortest path, we assume that 𝐺 = (𝑉, 𝐸) is a weighted graph,
that is, each edge (𝑣𝑖 , 𝑣𝑗 ) in 𝐺 has a given weight or length 𝑙𝑖𝑗 > 0. Then a shortest path
𝑣1 → 𝑣𝑘 (with fixed 𝑣1 and 𝑣𝑘 ) is a path such that the sum of the lengths of its edges

𝑙12 + 𝑙23 + 𝑙34 + ⋯ + 𝑙𝑘−1,𝑘

(𝑙12 = length of 𝑣1 , 𝑣2 , etc.) is minimum (as small as possible among all paths from 𝑣1 to 𝑣𝑘 ).

In a similar way, a longest path 𝑣1 → 𝑣𝑘 is the one for which that sum is maximum.

Note that the “length” 𝑙𝑖𝑗 (often called the cost or weight) can be actual length measured in
miles or travel time or gasoline expenses, but it may also be something entirely different.

For instance, a travelling salesman problem may require determination of a shortest

Hamiltonian cycle in a graph, that is, a cycle that contains all the vertices of the graph.

As another example, by choosing the most profitable route 𝑣1 → 𝑣𝑘 , a salesman may want to
maximize ∑𝑙𝑖𝑗 where 𝑙𝑖𝑗 is his expected commission minus his travel expenses for going from
town 𝑖 to town 𝑗.

36
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods
In an investment problem, 𝑖 may be the day an investment is made, 𝑗 the day it matures,
and 𝑙𝑖𝑗 the resulting profit, and one gets a graph by considering the various possibilities of
investing and reinvesting over a given period of time.

All Edges Have length 𝒍 = 𝟏

In this case, a shortest path 𝑣1 → 𝑣𝑘 is the one that has the smallest number of edges among
all paths 𝑣1 → 𝑣𝑘 in a given graph 𝐺. For this case, we employ a Breadth First Search
(BFS) algorithm. In BFS algorithm each step visits all neighboring (all adjacent) vertices of a
vertex as opposed to a Depth First Search (DFS) algorithm which makes a long trail (as in
a maze).

Algorithm 4.1 presents the Moore’s BFS algorithm for shortest path in graph 𝐺 from a vertex
𝑠 (start) to a vertex 𝑡 (terminal). To guarantee that there is a path from 𝑠 to 𝑡, we assume
that 𝐺 is connected, that is, for any two vertices 𝑣 and 𝑤 there is a path 𝑣 → 𝑤 in 𝐺.

Example – Application of the BFS Algorithm

Find a shortest path 𝑠 → 𝑡 in the graph shown in Figure below.

37
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Solution
The highlighted edges form a shortest path from 𝑠 to 𝑡. However, there is another shortest
path 𝑠 → 𝑡. Check it!

In the algorithm (program) we must introduce a rule that makes backtracking unique
because otherwise the computer would not know what to do next if at some step there is a
choice (for instance, in the graph above, when it got back to the vertex labeled 2). The
following rules is natural.

Backtracking rule: Using numbering of the vertices from 1 to 𝑛 (not labeling!), at each step,
if a vertex labeled 𝑖 is reached, take as the next vertex that with the smallest number (not
label!) among all the vertices labeled 𝑖 − 1.

Exercise:
1. What is the complexity of the Moore’s BFS algorithm?
Hint: Each edge is scanned twice 𝑐(2𝑚) → 𝑂(𝑚).
2. Find a shortest path 𝑃: 𝑠 → 𝑡 and its length by Moore’s BFS algorithm; sketch the graph
with the labels and highlight 𝑃 using thick lines.

3. The postman problem is the problem of finding a closed walk 𝑊: 𝑠 → 𝑠 (𝑠 is the post
office) in a graph 𝐺 with edges (𝑖, 𝑗) of length 𝑙𝑖𝑗 > 0 such that every edge of 𝐺 is traversed
at least once and the length of 𝑊 is minimum. Find a solution for the graph below by
inspection. (The postman problem is also called the Chinese postman problem).

38
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

4.6.3 Bellman’s Principle

In most practical applications, graph edges may have any lengths 𝑙𝑖𝑗 > 0. Thus, by relaxing
the assumption that each edge has length 𝑙 = 1, we have a general case which is of higher
practical significance.

Suppose in a problem of finding shortest paths from a given vertex denoted by 1 (the origin)
to all other vertices 2, 3, … , 𝑛 of 𝐺. Let the length of the shortest path be 𝐿𝑗 for 𝑃𝑗 : 1 → 𝑗 ∈ 𝐺.

Then, Bellman’s optimality principle states that:

If 𝑃𝑗 : 1 → 𝑗 ∈ 𝐺 is a shortest path from the origin 1 to the last edge 𝑗, then 𝑃𝑗 : 1 → 𝑖 (obtained
by dropping (𝑖, 𝑗) from 𝑃𝑗 ) is a shortest path 1 → 𝑖.

Bellman equations
For fixed 𝑗, one may obtain various paths 1 → 𝑗 by taking shortest paths 𝑃𝑖 for various 𝑖 for
which there is in 𝐺 an edge (𝑖, 𝑗) to the corresponding 𝑃𝑖 . These paths have lengths 𝐿𝑖 + 𝑙𝑖𝑗
(where 𝐿𝑖 = length of the path 𝑃𝑖 ). To get the minimum over 𝑖, one can pick an 𝑖 for which
𝐿𝑖 + 𝑙𝑖𝑗 is smallest.
By Bellman principle, this gives a shortest path 1 → 𝑗 with length

𝐿1 = 0
𝐿𝑗 = min(𝐿𝑖 + 𝑙𝑖𝑗 ) ∶ 𝑗 = 2, ⋯ , 𝑛
𝑖≠j
The Bellman equations suggest the idea of one of the best-known algorithms for the
shortest path problem, as follows.

Dijkstra’s Algorithm for Shortest Paths

Dijkstra’s algorithm involve a connected graph 𝐺 in which for any two vertices 𝑣 and 𝑤 in 𝐺,
there is a path 𝑣 → 𝑤 . The algorithm is a labeling procedure. At each stage of the
computation, each vertex 𝑣 is labelled either:

39
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

(PL) a permanent label = length 𝑙𝑣 of a shortest path 1 → 𝑣

Or
(TL) a temporary label = upper bound 𝐿̃𝑣 for the length of a shortest path 1 → 𝑣.

The set of all permanent and temporary labels are denoted by 𝒫ℒ and 𝒯ℒ respectively.

Examples – Application of Dijkstra’s Algorithm

Apply the Dijkstra’s algorithm to the following graph to find the shortest path from vertex 1
to vertices 2, 3, 4.

40
 ELE8311: Advanced Engineering Mathematics –
Module 4–Optimization: Intro., Formulation, Linear Prog. and Graphical Methods

Solution:
The following steps enlist the computations based on the Dijkstra’s algorithm

Thus, the length of the resulting shortest path is 𝐿2 = 6, 𝐿3 = 5, 𝐿4 = 7.

Complexity: The complexity of the Dijkstra’s algorithm is 𝑂(𝑛2 ).

Exercise:
The net of roads in the following graph connecting four villages is to be reduced to minimum
length, but so that one can still reach every village from every other village. Which of the
roads should be retained? Find the solution (a) by inspection, (b) by Dijkstra’s algorithm.

41