CSC2204 ANALYSIS OF ALGORITHMS

2022/2023

Course Lectures
Dr. Muhammad Yusuf
Baffa Sani
Farouk Yushau
Bilal

1
Contents
1 Introduction to Design and analysis of algorithms 3

2 Growth of Functions ( Asymptotic notations) 7

2.1 How to analyse an Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
2.2 Asymptotic Notations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

3 Recurrences, Solution of Recurrences by substitution, Recursion Tree and Master

Method 16

4 Recursion tree method 20

4.1 Master Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
4.2 Worst case analysis of merge sort, quick sort and binary search . . . . . . . . . . . . . 22

2
1 Introduction to Design and analysis of algorithms
The Basic objective of solving problem with multiple constraints such as problem size performance
and cost in terms of space and time. The goal is to design fast, efficient and effective solution to a
problem domain. Some problems are easy to solve and some are hard. Quite cleverness is required
to design solution with fast and better approach. Designing new system need a new technology and
background of the new technology is the enhancement of existing algorithm. The study of algorithm
is to design efficient algorithm not only limited in reducing cost and time but to enhance scalability,
reliability and availability.

What is an algorithm?
■ Algorithm is a set of steps to complete a task.

Example of algorithm

Task: ATM withdrawal system.

Algorithm:
1. Insert the card

2. Enter the PIN

3. Enter the amount

4. Withdraw amount

5. Check balance

6. Cancel/clear
Task: To make a cup of tea..

Algorithm:
1. Add water to the kettle

2. Boil it

3. Add tea leaves

4. Add Milk

5. Add sugar

3
 What is Computer algorithm?
”a set of steps to accomplish or complete a task that is described precisely enough that a
computer can run it”.

Described precisely: very difficult for a machine to know how much water, milk to be added etc.
in the above tea making algorithm.

These algorithms run on computers or computational devices.For example, GPS in our smartphones,
Google hangouts.

GPS uses shortest path algorithm. Online shopping uses cryptography which uses RSA algorithm.

Characteristics of an algorithm

ˆ Must take an input.

ˆ Must give some output (true/false, value etc.)

ˆ Definiteness - each instruction is clear and unambiguous.

ˆ Finiteness - algorithm terminates after a finite number of steps.

ˆ Effectiveness - every instruction must be basic i.e. simple instruction.

Expectation from an algorithm

ˆ Correctness:-

■ Correct: Algorithms must produce correct result.

■ Produces an incorrect answer:Even if it fails to give correct results all the time still there
is a control on how often it gives wrong result. Eg. Rabin-Miller Primality Test (Used in
RSA algorithm): It doesn’t give correct answer all the time. 1 out of 250 times it gives
incorrect result.
■ Approximation algorithm: Exact solution is not found, but near optimal solutions can be
obtained. (Applied to optimization problem.)

ˆ Less resource usage:

■ Algorithms should use less resources (time and space).

4
Resource usage:
Here, the time is considered to be the primary measure of efficiency. We are also concerned with
how much the respective algorithm involves the computer memory. But mostly time is the resource
that is dealt with. And the actual running time depends on a variety of backgrounds: like the speed
of a Computer, the language in which the algorithm is implemented, the compiler/interpreter, skill
of the programmers etc.

So, mainly the resource usage can be divided into:

1. Memory (space)

2. Time

Time taken by an algorithm

performance measurement or Apostoriori Analysis: Implementing the algorithm in a machine and
then calculating the time taken by the system to execute the program successfully.

Performance Evaluation or Apriori Analysis. Before implementing the algorithm in a system. This
is done as follows

1. How long the algorithm takes :- will be represented as a function of the size of the input.
f (n) → how long it takes if n is the size of input.

2. How fast the function that characterizes the running time grows with the input size.
”Running time growth rate”.
The algorithm with less running time growth rate is considered better.

How algorithm is a technology?

Algorithms are just like a technology. We all use latest and greatest processors but we need to
run implementations of good algorithms on that computer in order to properly take benefits of our
money that we spent to have the latest processor. Let’s make this example more concrete by pitting
a faster computer (computer A) running a sorting algorithm whose running time on n values grows
like n2 against a slower computer (computer B) running a sorting algorithm whose running time
grows like n log n. They each must sort an array of 10 million numbers. Suppose that computer A
executes 10 billion instructions per second (faster than any single sequential computer at the time
of this writing) and computer B executes only 10 million instructions per second, so that computer
A is 1000 times faster than computer B in raw computing power. To make the difference even more

5
dramatic, suppose that the world’s craftiest programmer codes in machine language for computer
A, and the resulting code requires 2n2 instructions to sort n numbers. Suppose further that just an
average programmer writes for computer B, using a high level language with an inefficient compiler,
with the resulting code taking 50n log n instructions.

Computer A (Faster) Computer B(Slower)

Running time grows like n2 . Grows in n log n.
10 billion instructions per sec. 10 million instruction per sec
2n2 instruction. 50 n log n instruction.

2 × (107 )2
Computer A takes = = 20,000 seconds (more than 5:5 hours)
1010
50 × log 107
Computer B takes = ≈ 1163 seconds (under 20 minutes)
107

So choosing a good algorithm (algorithm with slower rate of growth) as used by computer B affects
a lot.

6
2 Growth of Functions ( Asymptotic notations)
Before going for growth of functions and asymptotic notation let us see how to analyse an algorithm.

2.1 How to analyse an Algorithm

Insertion sort

A good algorithm for sorting a small number of elements.

It works the way you might sort a hand of playing cards:

ˆ Start with an empty left hand and the cards face down on the table.

ˆ Then remove one card at a time from the table, and insert it into the correct position in the
left hand.

ˆ To find the correct position for a card, compare it with each of the cards already in the hand,
from right to left.

ˆ At all times, the cards held in the left hand are sorted, and these cards were originally the top
cards of the pile on the table.

Let us form an algorithm for Insertion sort (which sort a sequence of numbers).

7
Example

Analyzing algorithms

We want to predict the resources that the algorithm requires. Usually, running time.

Why analyze?

Why not just code up the algorithm, run the code, and time it?
Because that would tell you how long the code takes to run

ˆ on your particular computer,

ˆ on that particular input,

ˆ with your particular implementation,

ˆ using your particular compiler or interpreter,

ˆ with the particular libraries linked in,

ˆ with the particular background tasks running at the time.

You wouldn’t be able to predict how long the code would take on a different computer, with a
different input, if implemented in a different programming language, etc.
Instead, devise a formula that characterizes the running time.

Random-access machine (RAM) model

In order to predict resource requirements, we need a computational model.

ˆ Instructions are executed one after another. No concurrent operations.

ˆ It’s too tedious to define each of the instructions and their associated time costs.

ˆ Instead, we recognize that we’ll use instructions commonly found in real computers:

8
 – Arithmetic: add, subtract, multiply, divide, remainder, floor, ceiling). Also, shift left/shift
right/shift.
– Data movement: load, store, copy.
– Control: conditional/unconditional branch, subroutine call and return.

Each of these instructions takes a constant amount of time. Ignore memory hierarchy (cache and
virtual memory).

How do we analyze an algorithm’s running time?

The time taken by an algorithm depends on the input.

ˆ Sorting 1000 numbers takes longer than sorting 3 numbers.

ˆ A given sorting algorithm may even take differing amounts of time on two inputs of the same
size.

ˆ For example, we’ll see that insertion sort takes less time to sort n elements when they are
already sorted than when they are in reverse sorted order.

Input size

Depends on the problem being studied.

ˆ Usually, the number of items in the input. Like the size n of the array being sorted.

ˆ But could be something else. If multiplying two integers, could be the total number of bits in
the two integers.

ˆ Could be described by more than one number. For example, graph algorithm running times
are usually expressed in terms of the number of vertices and the number of edges in the input
graph.

Running time

On a particular input, it is the number of primitive operations (steps) executed.

ˆ Want to define steps to be machine-independent.

ˆ Figure that each line of pseudocode requires a constant amount of time.

ˆ One line may take a different amount of time than another, but each execution of line k takes
the same amount of time Ck .

9
 ˆ This is assuming that the line consists only of primitive operations.

– If the line is a subroutine call, then the actual call takes constant time, but the execution
of the subroutine being called might not.
– If the line specifies operations other than primitive ones, then it might take more than
constant time. Example: “sort the points by x-coordinate.”

Analysis of insertion sort

ˆ Assume that the kth line takes time ck , which is a constant. (Since the third line is a comment,
it takes no time.)

ˆ For i = 1, 2, . . . n, let ti be the number of times that the while loop test is executed for that
value of i.

ˆ Note that when a for or while loop exits in the usual way - due to the test in the loop header
- the test is executed one time more than the loop body.

The running time of the algorithm is

X
(cost of statement).(number of times statement is executed)
all statements

Let T (n) = running time of INSERTION-SORT.

n
P n
P n
P
T (n) = c1 n + c2 (n − 1) + c4 (n − 1) + c5 ti + c6 (ti − 1) + c7 (ti − 1) + c8 (n − 1)
i=2 i=2 i=2

The running time depends on the values of ti . These vary according to the input.

Best case

The array is already sorted. All ti are 1.

T (n) = c1 n + c2 (n − 1) + c4 (n − 1) + c5 (n − 1) + c8 (n − 1)
= (c1 + c2 + c4 + c5 + c8 )n + (c2 + c4 + c5 + c8 )
= an + b for constants a and b (that depend on the statement costs ck )

Worst case

The array is in reverse sorted order. We have to compare key with all elements to the left of the ith
position =⇒ compare with i − 1 elements. Since the while loop exits because i reaches 0, there’s

10
one additional test after the i − 1 tests =⇒ tj = i. Running time is
     
n(n + 1) n(n − 1) n(n − 1)
T (n) = c1 n + c2 (n − 1) + c4 (n − 1) + c5 − 1 + c6 + c7 + c8 (n − 1)
2 2 2
c c6 c7  2  c5 c6 c7 
5
= + + n + c1 + c2 + c4 + − − + c8 n − (c2 + c4 + c5 + c8 )
2 2 2 2 2 2
2
= an + bn + c for constants a, b, c (that again depend on statement costs ck )

We usually concentrate on finding the worst-case running time: the longest running time for any
input of size n.

Why we concentrate on worst-case running time?

ˆ The worst-case running time gives a guaranteed upper bound on the running time for any
input.

ˆ For some algorithms, the worst case occurs often. For example, when searching, the worst case
often occurs when the item being searched for is not present, and searches for absent items
may be frequent.

ˆ Why not analyze the average case? Because it’s often about as bad as the worst case.

Order of growth
Another abstraction to ease analysis and focus on the important features.
Look only at the leading term of the formula for running time.

ˆ Drop lower-order terms.

ˆ Ignore the constant coefficient in the leading term.

Example: For insertion sort, we already abstracted away the actual statement costs to conclude
that the worst-case running time is an2 bn + c.
Drop lower-order terms =⇒ an2 .
Ignore constant coefficient =⇒ n2 .
But we cannot say that the worst-case running time T (n) equals n2 .
It grows like n2 . But it doesn’t equal n2 .
We say that the running time is Θn2 to capture the notion that the order of growth is n2 .

We usually consider one algorithm to be more efficient than another if its worst-case
running time has a smaller order of growth.

11
2.2 Asymptotic Notations
ˆ It is a way to describe the characteristics of a function in the limit.

ˆ It describes the rate of growth of functions.

ˆ Focus on what’s important by abstracting away low-order terms and constant factors.

ˆ It is a way to compare “sizes” of functions:

1. O ≈≤
2. Ω ≈≥
3. Θ ≈=
4. o ≈<
5. ω ≈>

O-notation
O-notation characterizes an upper bound on the asymptotic behavior of a function: it says that a
function grows no faster than a certain rate. This rate is based on the highest order term.

O(g(n)) = {f (n) : there exist positive constants c and n0 such that 0 ≤ f (n) ≤ cg(n) for all n ≥
n0 }

g(n) is an asymptotic upper bound for f (n).

If f (n) ∈ Og(g(n)), we write f (n) = O(g(n))

Example
2n2 = O(n3 ), with c = 1 and n0 = 2.

Example
f (n) = 7n3 + 100n2 − 20n + 6 is O(n3 ), since the highest order term is 7n3 , and therefore the function

12
grows no faster than n3 .

Ω-notation
Ω-notation characterizes a lower bound on the asymptotic behavior of a function: it says that a
function grows at least as fast as a certain rate. This rate is again based on the highest-order term.

Ω(g(n)) = {f (n) : there exist positive constants c and n0 such that 0 ≤ cg(n) ≤ f (n) for all n ≥
n0 }

g(n) is an asymptotic lower bound for f (n)

Example
n2 + n = Ω(n2 )

Example
√
n = Ω(log n), with c = 1 and n0 = 16.

Example
f (n) = 7n3 + 100n2 − 20n + 6 is Ω(n3 ), since the highest order term is 7n3 , and therefore the function
grows at least as fast as n3 .

Θ-notation
Θ-notation characterizes a tight bound on the asymptotic behavior of a function: it says that a
function grows precisely at a certain rate, again based on the highest order term.

If a function is both O(f (n)) and Ω(f (n)), then a function is Θ(f (n)).

Θ(g(n)) = {f (n) : there exist positive constants c1 , c2 and n0 such that 0 ≤ c1 g(n) ≤ f (n) ≤ c2 g(n)
for all n ≥ n0 }

13
 g(n) is an asymptotically tight bound for f (n)

Example
n2 1 1
− 2n = Θ(n2 ), with c1 = , c2 = , and n0 = 8.
2 4 2

Theorem 2.1. f (n) = Θ(g(n)) if and only if f = O(g(n)) and f = Ω(g(n))

ø-notation
o(g(n)) = {f (n) : for all constants c > 0, there exists a constant n0>0 such that 0 ≤ f (n) < cg(n)
for all n ≥ n0 }.

f (n)
Another view, probably easier to use: lim = 0.
n↣∞ g(n)

Example
n1.9999 = o(n2 )

n2
= o(n2 )
log n

n2 ̸= o(n2 )

n2
̸= o(n2 )
1000

ω-notation
ω = {f (n) : for all constants c > 0, there exists a constant n0 > n such that 0 ≤ cg(n) < f (n) for
all n ≥ n0 }

f (n)
A much easier definition to use: lim = ∞.
n→∞ g(n)

14
Example
n2.0001 = ω(n2 )

n2 log n = ω(n2 )

n2 ̸= ω(n2 )

Exercise 2.1. .

1. Suppose that for inputs of size n on a particular computer, insertion sort runs in 8n2 steps and
merge sort runs in 64n log n steps. For which values of n does insertion sort beat merge sort?

2. What is the smallest value of n such that an algorithm whose running time is 100n2 runs faster
than an algorithm whose running time is 2n on the same machine?

15
3 Recurrences, Solution of Recurrences by substitution, Re-
cursion Tree and Master Method
Recursion is a particularly powerful kind of reduction, which can be described loosely as follows:

ˆ If the given instance of the problem is small or simple enough, just solve it.

ˆ Otherwise, reduce the problem to one or more simpler instances of the same problem.

Recursion is generally expressed in terms of recurrences. In other words, when an algorithm calls to
itself, we can often describe its running time by a recurrence equation which describes the overall
running time of a problem of size n in terms of the running time on smaller inputs.
For example, the worst case running time T (n) of the merge sort procedure by recurrence can be
expressed as

Θ(1); if n = 1
T (n) =
2T ( n ) + Θ(n); if n > 1
2

whose solution can be found as T (n) = Θ(n log n)

There are various techniques to solve recurrences.

1. SUBSTITUTION METHOD:
The substitution method comprises of 3 steps

i. Guess the form of the solution

ii. Verify by induction
iii. Solve for constants

We substitute the guessed solution for the function when applying the inductive hypothesis
to smaller values. Hence the name ”substitution method”. This method is powerful, but we
must be able to guess the form of the answer in order to apply it.
Example n
Recurrence: T (n) = 4T + n and T
2

16
Step 1: Guess the solution
n
T (n) = 4T +n
n2
T (n) = 4T
2
=⇒ T (2n) = 4T (n)
=⇒ T (n) = n2

So, T (n) is order of n2

We guess that the solution is T (n) = O(n3 )

Step 2: So we must prove that T (n) ≤ cn3 for some constant c.
Assume T (k) ≤ ck 3

n
T (n) = 4T +n
2
 n 3
≤ 4c +n
2
n3
≤c +n
2 
cn3

3
≤ cn − −n
2
 3 
3 cn
T (n) ≤ cn as − n is always positive
2

Therefore, we have shown that T (n) ≤ cn3 , so T (n) = O(n3 ).

Step 3: To solve for constants

cn3
−n≥0
2
=⇒ n ≥ 1
=⇒ c ≥ 2

Now suppose we guess that T (n) = O(n2 ) which is tight upper bound

Assume, T (k) ≤ ck 2

17
 so, we should prove that T (n) ≤ cn2
n
T (n) = 4T +n
2
 n 2
= 4c +n
2
= cn2 + n

So, T (n) will never be less than cn2 . But if we will take the assumption of T (k) = c1 k 2 − c2 k,
then we can find that T (n) = O(n2 )

2. BY ITERATIVE METHOD:
Example n
Recurrence: T (n) = 2T +n
2

h n ni
T (n) = 2 2T + +n
n 4 2
= 22 T +n+n
4
h n ni
= 22 2T + + 2n
 n 8 4
= 23 T 3 + 3n
2
.
.
.
n
T (n) = 2k T + kn After k iterations.
2k
n
Sub problem size is 1 after k = 1 =⇒ k = log n
2
So, after k = log n iterations, the sub-problem size will be 1.
Substituting k = log n in the equation above, we obtained
T (n) = nT (1) + n log n
=⇒ = nc + n log n where c = T (1)
=⇒ = O(n log n)

3. BY RECURSION TREE METHOD:

In a recursion tree, each node represents the cost of a single sub-problem somewhere in the
set of recursive problems invocations we sum the cost within each level of the tree to obtain a

18
set of per level cost, and then we sum all the per level cost to determine the total cost of all
levels of recursion.

Example n
Constructing a recursion tree for the recurrence T (n) = 3T + cn2
4

19
4 Recursion tree method

20
4.1 Master Method

21
4.2 Worst case analysis of merge sort, quick sort and binary search

22