Design and Analysis of Algorithm

Course Code : 5009

Lecture 1

Dr. Huma Qayyum

Department of Software Engineering
[email protected]
 Course Objectives
• This course introduces students to the analysis and
design of computer algorithms. Upon completion of
this course, students will be able to do the following:
– Analyze the asymptotic performance of algorithms.
– Show a understanding with major algorithms and data
structures.
– Apply important algorithmic design examples and methods
of analysis.
Lecture # Lecture Contents Task
1. Introduction and Asymptotic Notations
2. Mathematical Tools and Application
3. Amortized analysis
4. Inductive tools and algorithm correctness Quiz 01, Assignment 01
5. Minimum Cost Spanning trees
6. Divide and Conquer
7. Greedy method Quiz 02
8. Decrease-and-Conquer

MIDTERM
9. Dynamic Programming I
10. Dynamic Programming II
11. Backtracking I
12. Backtracking II Quiz 03, Assignment 02
13. Branch And Bound Techniques I
14. Branch And Bound Techniques II
15. NP Complete Problems I
16. NP Complete Problems II Quiz 04
FINAL EXAMS
 Course Information

• Office hours
– 9:00-11:00 Wednesday and Friday
• Grading policy
– Assign and Quiz. 20%, Midterm: 30%, Final: 50%
 More Information
• Textbook
– Anany Levitin-Introduction to the Design and Analysis of Algorithms -
Addison Wesley (Pearson Education Inc.) (2011).pdf

• Others
– Introduction to Design & Analysis Computer Algorithm 3rd, Sara
Baase, Allen Van Gelder, Adison-Wesley, 2000.
– Algorithms, Richard Johnsonbaugh, Marcus Schaefer, Prentice Hall,
2004.
– Introduction to The Design and Analysis of Algorithms 2nd Edition,
Anany Levitin, Adison-Wesley, 2007.
 An Algorithm
• An Algorithm is a finite sequence of instructions, each of which has a
clear meaning and can be performed with a finite amount of effort in a
finite length of time. No matter what the input values may be, an
algorithm terminates after executing a finite number of instructions.

In addition every algorithm must satisfy the following criteria:

• Input: there are zero or more quantities, which are externally supplied;
Output: at least one quantity is produced;
• Definiteness: each instruction must be clear and unambiguous;

• Finiteness: if we trace out the instructions of an algorithm, then for all

cases the algorithm will terminate after a finite number of steps;

• Effectiveness: every instruction must be sufficiently basic that it can in

principle be carried out by a person using only pencil and paper.
 Algorithmic

It is the science that lets designers study and

evaluateALGORITHMICS
the effect of algorithms based on
various factors so that the best algorithm is
selected to meet a particular task in given
circumstances. It is also the science that tells
how to design a new algorithm for a particular
job.
 Types of Algorithms
PROBABILISITIC ALGORITHM

•In this algorithm, chosen values are used

in such a way that the probability of chosen
each value is known and
controlled

For example Random Quick Sort

 HEURISTIC ALGORITHM

• This type of algorithm is based largely on optimism and

often with minimal theoretical support. Here error can
not be controlled but may be estimated how
large it is.

9
 APPROXIMATE ALGORITHM

In this algorithm, answer is obtained that is

as précised as required in decimal
notation. In other words it specifies the
error we are willing to accept.

For example, two figures accuracy or 8

figures or whatever is required

10
 What is a program?
• A program is the expression of an algorithm in
a programming language
• a set of instructions which the computer will
follow to solve a problem
Let us write down the algorithm for a
problem that is familiar to us

Converting a decimal number into binary

Convert 75 to Binary
2 75 remainder
2 37 1
2 18 1
2 9 0
2 4 1
2 2 0
2 1 0
0 1

1001011
 Algorithm for Decimal-to-Binary Conversion

1. Write the decimal number

2. Divide by 2; write quotient and remainder

3. Repeat step 2 on the quotient; keep on repeating

until the quotient becomes zero

4. Write all remainder digits in the reverse order (last

remainder first) to form the final result
 Algorithm (Better Definition)
1st Definition:
Sequence of steps that can be taken
to solve a problem

Better Definition:
A precise sequence of a limited
number of unambiguous, executable
steps that terminates in the form of a
solution
 Analysis of Algorithms
• Analysis in the context of algorithms is concerned with
predicting the resources that are requires:
– Computational time
– Memory
• However, Time – generally measured in terms of the
number of steps required to execute an algorithm - is the
resource of most interest

• By analyzing several candidate algorithms, the most

efficient one(s) can be identified
 Time taken by an algorithm?

• Performance measurement or Apostoriori Analysis

• Implementing the algorithm in a machine and
then calculating the time taken by the system to
execute the program successfully.
• Performance Evaluation or Apriori Analysis.
Before implementing the algorithm in a
system. This is done as follows
 Selecting Among Algorithms

When choosing among competing, successful

solutions to a problem, choose the one which is the
least complex

This principle is called the “Ockham’s Razor,” after

William of Ockham - famous 13th century English
philosopher
• 1. How long the algorithm takes :-will be
represented as a function of the size of
• the input f(n)→how long it takes if ‘n’ is the size
of input.
• 2. How fast the function that characterizes the
running time grows with the input
• size. “Rate of growth of running time”.
• The algorithm with less rate of growth of running
time is considered better.
Looking for Efficiency
• Investment on computing equipment
• Investment on algorithmics
Investment on computing equipment
• Computer solves a particular problem in 10-4 x 2n sec
where n is size of instances
- If n = 10, 10-4 x 210 = 0.1024 sec
- If n = 20, 10-4 x 220 = 104.85 sec
- If n = 30, 10-4 x 230 = 107374 sec = 29.82 hours
- If n = 38, 10-4 x 238 = 7635.49 hours = 254 days
• Purchase of new machine which solves the problem
in 10-6 x 2n sec
- If n = 38, 10-6 x 238 = 763.54 hours = 31 days
- If n = 45, 10-6 x 245 = 11 years
• No big gain on investment on new machine
Looking for Efficiency
Investment on algorithmics
• A cubic algorithm can solve the problem at very high
speed. An old machine can compute the result in 10 -2
x n3 sec.
- If n = 10, 10-2 x 103 = 10 sec
- If n = 20, 10-2 x 203 = 80 sec
- If n = 30, 10-2 x 303 = 270 sec = 4.5 minutes
- If n > 200, 10-2 x 2003 = 1 day
• New algorithm offers a much greater improvement
than the purchase of new computing equipment
• New algorithm + new computing machine can provide
results 100 times faster and enables to solve
instances 4 or 5 times bigger than with the new
algorithm alone in the same length of time.
 What kinds of problems are solved by
algorithms?
• Searching (Linear and Non-Linear)
• Sorting ( Elementary and Advanced)
• String Processing ( Pattern Matching, Parsing,
Compression, Cryptography)
• Optimization Problem ( Shortest routs, Minimum costs)
• Image Processing( Compression, Conversion, Matching)
• Data Mining Algorithms ( Clustering, Cleansing, Rules
Mining)
• Mathematical Algorithms ( Random Number generator,
Matrix Operation etc. )
 Types of Algorithmic techniques
• Divide and Conquer
• Greedy Algorithms
• Dynamic Programming
• Heuristics
• Brute Force Technique
• Backtracking
 Algorithm Representation

• Generally, SW developers represent them in one of

three forms:
– Pseudo code
– Flowcharts
– Actual code
Classification of Algorithms
 Complexity of Algorithms
The complexity of an algorithm M is the function f(n) which gives the running time and/or
storage space requirement of the algorithm in terms of the size ‘n’ of the input data.
Mostly, the storage space required by an algorithm is simply a multiple of the data size ‘n’.
Complexity shall refer to the running time of the algorithm.
The function f(n), gives the running time of an algorithm, depends not only on the size ‘n’
of the input data but also on the particular data. The complexity function f(n) for certain
cases are:

1. Best Case : The minimum possible value of f(n) is called the best case.

2. Average Case : The expected value of f(n).

3. Worst Case : The maximum value of f(n) for any key possible input.
The field of computer science, which studies efficiency of algorithms, is known as analysis
of algorithms.

Algorithms can be evaluated by a variety of criteria. Most often we shall be interested in

the rate of growth of the time or space required to solve larger and larger instances of a
problem. We will associate with the problem an integer, called the size of the problem,
which is a measure of the quantity of input data.
 Rate of Growth:
The following notations are commonly use notations in
performance analysis and used to characterize the complexity of an
algorithm:

1. Big–OH (O) 1,

2. Big–OMEGA (Ω),

3. Big–THETA () and

4. Little–OH (o)
 Big–OH O (Upper Bound)
f(n) = O(g(n)), (pronounced order of or big oh), says that the growth rate of f(n) is less
than or equal (<) that of g(n).
Little–OH (o)
 Growth Rate of Running Time

• Changing the hardware/ software

environment
– Affects T(n) by a constant factor, but
– Does not alter the growth rate of T(n)
• The linear growth rate of the running time
T(n) is a basic property of algorithm
arrayMax

Wednesday, August
Analysis 21, 2019
of Algorithm 34
 Example
• Associate a "cost" with each statement.
• Find the "total cost“ by finding the total number of times
each statement is executed.
Algorithm 1 Algorithm 2

Cost Cost
arr[0] = 0; c1 for(i=0; i<=N; i++) c2
arr[1] = 0; c1 arr[i] = 0; c1
arr[2] = 0; c1
... ...
arr[N-1] = 0; c1
----------- -------------
c1+c1+...+c1 = c1 x N (N+1) x c2 + N x c1 =
(c2 + c1) x N + c2

35
 Another Example
• Algorithm 3 Cost
sum = 0; c1
for(i=0; i<=N; i++) c2
for(j=0; j<=N; j++) c2
sum += arr[i][j]; c3
------------
c1 + c2 x (N+1) + c2 x N x (N+1) + c3 x N2

36
 Rate of Growth
• Consider the example of buying elephants and
goldfish:
Cost: cost_of_elephants + cost_of_goldfish
Cost ~ cost_of_elephants (approximation)
• The low order terms in a function are relatively
insignificant for large n
n4 + 100n2 + 10n + 50 ~ n4

i.e., we say that n4 + 100n2 + 10n + 50 and n4 have the

same rate of growth
37
 Big-Oh Notation
• To simplify the running time estimation,
for a function f(n), we ignore the constants
and lower order terms.

Example: 10n3+4n2-4n+5 is O(n3).

Wednesday,Analysis of Algorithm
August 21, 2019 38
 A Simple Example – Linear Search
INPUT: a sequence of n numbers, key to search for.
OUTPUT: true if key occurs in the sequence, false otherwise.
LinearSearch(A, key) cost times
1 i1 c1 
n
i 2
1 1
2 while i ≤ n and A[i] != key c2 n+1
3 do i++ c3 n
if i  n c4 1
4 then return true c5 1
5 else return false c6 1
x ranges between 1 and n+1.
So, the running time ranges between
c1+ c2+ c4 + c5 – best case
and
c1+ c2(n+1)+ c3n + c4 + c6 – worst case

Comp 122,
 A Simple Example – Linear Search
INPUT: a sequence of n numbers, key to search for.
OUTPUT: true if key occurs in the sequence, false otherwise.
LinearSearch(A, key) cost times
1 i1 1 n
i 2
1 1
2 while i ≤ n and A[i] != key 1 n+1
3 do i++ 1 n
4 if i  n 1 1
5 then return true 1 1
6 else return false 1 1
Assign a cost of 1 to all statement executions.
Now, the running time ranges between
1+ 1+ 1 + 1 = 4 – best case
and
1+ (n+1)+ n + 1 + 1 = 2n+4 – worst case

Comp 122,
 A Simple Example – Linear Search
INPUT: a sequence of n numbers, key to search for.
OUTPUT: true if key occurs in the sequence, false otherwise.
LinearSearch(A, key) cost times
1 i1 1 n
i 2
1 1
2 while i ≤ n and A[i] != key 1 n
3 do i++ 1 n-1
4 if i  n 1 1
5 then return true 1 1
6 else return false 1 1
If we assume that we search for a random item in the list,
on an average, Statements 2 and 3 will be executed n/2 times.
Running times of other statements are independent of input.
Hence, average-case complexity is
1+ n/2+ n/2 + 1 + 1 = n+3

Comp 122,
 Order of growth
• Principal interest is to determine
– how running time grows with input size – Order of growth.
– the running time for large inputs – Asymptotic complexity.
• In determining the above,
– Lower-order terms and coefficient of the highest-order term are
insignificant.
– Ex: In 7n5+6n3+n+10, which term dominates the running time for very
large n?
• Complexity of an algorithm is denoted by the highest-order
term in the expression for running time.
– Ex: Ο(n), Θ(1), Ω(n2), etc.
– Constant complexity when running time is independent of the input size
– denoted Ο(1).
– Linear Search: Best case Θ(1), Worst and Average cases: Θ(n).
• More on Ο, Θ, and Ω in next class. Use Θ for the present.

Comp 122,
Important Summations

43
Important Summations that should be
Committed to Memory.

44
Analysis: A Harder Example

45
 Solution

• How do we analyze the running time of an

algorithm that has complex nested loop?
• The answer write out the loops as
summations and then solve the summations.
• To convert loops into summations, we
work from inside-out.

46
 Analysis: A Harder Example

It is executed for k = j, j − 1, j − 2, . . . , 0. Time spent inside the

while loop is constant. Let I() be the time spent in the while loop

47
Analysis: A Harder Example

48
Analysis: A Harder Example

49
 Order of Magnitude Analysis
Measure speed with respect to the part of the sum
that grows quickest

50 N 2  31N 3  24 N  15
3N 2  N  21  4  3N
Ordering:

1  log 2 N  N  N log 2 N  N  N  2  3
2 3 N N