CHAPTER: ONE

BASICS OF ALGORITHM ANALYSIS

Contents:

1. Introduction
2. Characteristics of Algorithm

3. Algorithm Complexity
4. Algorithm Analysis

5. Asymptotic Analysis

6. Asymptotic Notations
Introduction

Algorithm is a step-by-step procedure, which defines a set of instructions to be

executed in a certain order to get the desired output. Algorithms are generally created
independent of underlying languages, i.e. an algorithm can be implemented in more
than one programming language.

It is an explicit, precise, unambiguous, sequence of instructions intended to

accomplish a specific purpose.

How to write an algorithm?

How to write an algorithm?

There are no well-defined standards for writing algorithms.

Rather, it is problem and resource dependent. Algorithms are never written to support
a particular programming code.

As we know that all programming languages share basic code constructs like loops
(do, for, while), flow-control (if-else), etc.
These common constructs can be used to write an algorithm.

Example:
Let's try to learn algorithm-writing by using an example.

Problem − Design an algorithm to add two numbers and display the result.
Solution:

Page 1 of 55
 step 1 − START

step 2 − declare three integers a, b & c

step 3 − define values of a & b

step 4 − add values of a & b

step 5 − store output of step 4 to c

step 6 − print c

step 7 – STOP
There are so many ways of writing algorithm for the above problem. It can also be
written as –

step 1 − START ADD

step 2 − get values of a & b

step 3 − c ← a + b
step 4 − display c

step 5 – STOP

N.B: We design an algorithm to get a solution of a given problem. A problem can be

solved in more than one ways.

Characteristics of Algorithm

Page 2 of 55
Not all procedures can be called an algorithm. An algorithm should have the following
characteristics:

✓ Unambiguous
✓ Input

✓ Output
✓ Correct

✓ Complete
✓ Definiteness

✓ Sequence

✓ Language independence

Algorithm Complexity
Suppose X is an algorithm and n is the size of input data, the time and space used by
the algorithm X are the two main factors, which decide the efficiency of X.

Time Factor – Time is measured by counting the number of key operations such as
comparisons in the sorting algorithm.
Space Factor − Space is measured by counting the maximum memory space required
by the algorithm.

The complexity of an algorithm f(n) gives the running time and/or the storage space
required by the algorithm in terms of n as the size of input data.
Space Complexity

Space complexity of an algorithm represents the amount of memory space required by

the algorithm in its life cycle.

It is a measure of the amount of working storage an algorithm needs.

That means how much memory, in the worst case, is needed at any point in the
algorithm.

The space required by an algorithm is equal to the sum of the following two
components:
A fixed part that is a space required to store certain data and variables, that are
independent of the size of the problem.

Page 3 of 55
A variable part is a space required by variables, whose size depends on the size of the
problem.

Space complexity S(P) of any algorithm P is :

S(P) = C + S, where C is the fixed part and S is the variable part of the algorithm,
which depends on instance characteristic I.

Following is a simple example that tries to explain the concept:

Algorithm: SUM(A, B)

Step 1 - START
Step 2 - C ← A + B + 10

Step 3 - Stop

Here we have three variables A, B, and C and one constant.

Hence S(P) = 1+3.

So the space complexity is O(1).

Now, space depends on data types of given variables and constant types and it will be
multiplied accordingly.

Example: 1 Find the space complexity of the following algorithm

int sum(int x, int y, int z) {
int total = x + y + z;

return total;
}

The above algorithm requires 3 units of space for the parameters and 1 for the local
variable, and this never changes, so this is O(1).
Example: 2 Find the space complexity of the following algorithm

int sum(int a[], int n) {

int total = 0;

for (int i = 0; i < n; i++) {

total=total+ a[i];
}

return total;

Page 4 of 55
 }

The space required by the above algorithm can be computed as:

✓ N unit of space for a

✓ 1 unit of space for n

✓ 1 unit of space for i

✓ 1 unit of space for total

Total space required is N+3, which is O(N)

Algorithm Analysis

First, it is better to understand why we always give attention to performance analysis

when we talk about algorithm analysis.
Why performance analysis?
There are many important things that should be taken care of, like user friendliness,
clarity, security, maintainability, etc.

Why to worry about performance?

The answer to this is simple, we can have all the above things only if we have
performance. So performance is like currency through which we can buy all the above
things.

Another reason for studying performance is – speed is fun!

To summarize, performance == scale.

It is a process of predicting the resource requirement of algorithms in a given

environment.

In order to solve a problem, there are many possible algorithms.

One has to be able to choose the best algorithm for the problem at hand using some
scientific method.
To classify some data structures and algorithms as good, we need precise ways of
analyzing them in terms of resource requirement.

The main resources are:

✓ Running Time

✓ Memory Usage
Running time is usually treated as the most important since computational time is the
most precious resource in most problem domains.

Page 5 of 55
There are two approaches to measure the efficiency of algorithms:

✓ Theoretical (Priori Analysis):

✓ Empirical (Posterior Analysis):

[Link] (Priori Analysis)

It is determining the quantity of resources required mathematically (Execution time,
memory space, etc.) needed by each algorithm.

2. Empirical (Posterior Analysis)

It is all about programming competing algorithms and trying them on different
instances.

However, it is difficult to use actual clock-time as a consistent measure of an

algorithm’s efficiency, because clock-time can vary based on many things. Some of
them are:
✓ Specific processor speed

✓ Current processor load

✓ Specific data for a particular run of the program

✓ Input Size

✓ Input Properties
✓ Operating Environment

Asymptotic Analysis

Asymptotic analysis refers to computing the running time of any operation in

mathematical units of computation.
For example, the running time of one operation is computed as f(n) and may be for
another operation it is computed as g(𝒏𝟐 ).

This means the first operation running time will increase linearly with the increase
in n and the running time of the second operation will increase exponentially
when n increases.
Similarly, the running time of both operations will be nearly the same if n is
significantly small.

Page 6 of 55
Usually, the time required by an algorithm falls under three types −

✓ Best Case − Minimum time required for program execution.

✓ Average Case − Average time required for program execution.

✓ Worst Case − Maximum time required for program execution.

How do we compare algorithms?

Reminder! You can only compare apples to apples.

You can't compare an algorithm to do arithmetic operation to an algorithm that sorts a

list of integers.

But a comparison of two algorithms to do arithmetic operations (one multiplication,

one addition) will tell you something meaningful.

To do so, we need to define a number of objective measures.

✓ Compare execution times?

✓ Not good: times are specific to a particular computer!!

✓ Count the number of statements executed?

✓ Not good: number of statements vary with the programming language as

well as the style of the individual programmer.
Ideal Solution
Express running time as a function of the input size n (i.e., f(n)).

Compare different functions corresponding to running times.

Such an analysis is independent of machine time, programming style, etc.

Goals of Comparing Algorithms

Many measures for comparing algorithms

✓ Security

✓ Clarity
✓ Performance

When comparing performance

Use large inputs because probably any algorithm is “plenty good” for small inputs (n
< 10 always fast)

Page 7 of 55
Answer should be independent of CPU speed, programming language, coding tricks,
etc.

Asymptotic Notations
Given two algorithms for a task, how do we find out which one is better?

One naive way of doing this is – implement both the algorithms and run the two
programs on your computer for different inputs and see which one takes less time.
There are many problems with this approach for analysis of algorithms.

✓ It might be possible that for some inputs, first algorithm performs better than
the second. And for some inputs second performs better.
✓ It might also be possible that for some inputs, first algorithm performs better on
one machine and the second works better on other machine for some other
inputs.

Asymptotic Analysis is the big idea that handles above issues in analysing algorithms.
Asymptotic analysis refers to computing the running time of any operation in
mathematical units of computation.

In Asymptotic Analysis, we evaluate the performance of an algorithm in terms of

input size (we don’t measure the actual running time).
We calculate, how does the time (or space) taken by an algorithm increases with the
input size.

Running Time Calculations

General Rules

1. For loop

• The number of iterations times the time of the inside statements.

2. Nested loops

• The product of the number of iterations times the time of the inside
statements.
3. Consecutive Statements

• The sum of running time of each segment.

4. If/Else

• The testing time plus the larger running time of the cases.

Page 8 of 55
Analysis of iterative programs with simple examples

1. O(1): Time complexity of a function (or set of statements) is considered as O(1) if

it doesn’t contain loop, recursion and call to any other non-constant time function.
// set of non-recursive and non-loop statements

For example swap() function has O(1) time complexity.

A loop or recursion that runs a constant number of times is also considered as O(1).
For example the following loop is O(1).

// Here c is a constant
for (int i = 1; i <= c; i++) {

// some O(1) expressions

}
2. O(n): Time Complexity of a loop is considered as O(n) if the loop variables is
incremented / decremented by a constant amount.

For example, the following function has O(n) time complexity.

for (int i = 1; i <= n; i ++) {

// some O(1) expressions//cout<<i

}

3. O(𝒏𝒄 ): Time complexity of nested loops is equal to the number of times the
innermost statement is executed.

For example the following sample loops have O(𝒏𝟐 ) time complexity

for (int i = 1; i <=n; i += c) {

for (int j = 1; j <=n; j += c) {

// some O(1) expressions

}

}
Page 9 of 55
4. O(Logn): Time Complexity of a loop is considered as O(Logn) if the loop variables
is divided / multiplied by a constant amount.

for (int i = 1; i <=n; i *= c) {

// some O(1) expressions

}
for (int i = n; i > 0; i /= c) {

// some O(1) expressions

}

Types of asymptotic notations:

✓ Big Oh notations(O)
✓ Big Omega notations(Ω)

✓ Big Theta notations()

1. Big oh Notations
The notation Ο(n) is the formal way to express the upper bound of an algorithm's
running time.

It measures the worst case time complexity or the longest amount of time an
algorithm can possibly take to complete.
Definition:

f (n)= O (g (n)) if there exist c, k ∊ ℛ+ such that for all n≥ k, f (n) ≤ c.g (n).

Example 1: Given f(n)=5n+3 and g(n)=n. Prove that running time f(n)=O(g(n)).

Proof: By the Big oh definition, f(n) is O(n) if and only if f(n)<=c.g(n) for all n>=k.
f(n)<=c.g(n), 5n+3<=c.n

For c=5+3=8, the above statement is valid in any case

5n+3<=8n for all n>=1

So, f(n)=O(g(n)) for (c,k)=(8,1)

Example 2: Given f(n)=𝒏𝟑 +20n+1 and g(n)= 𝒏𝟑 . Prove that running time
f(n)=O(g(n)).
Proof: By the Big oh definition, f(n) is O(n) if and only if f(n)<=c.g(n) for all n>=k.

f(n)<=c.g(n)

Page 10 of 55
𝒏𝟑 +20n+1 <=c. 𝒏𝟑

For c=1+20+1=22, the above statement is valid in any case

𝒏𝟑 +20n+1 <=22. 𝒏𝟑 for all n>=1

So, f(n)=O(g(n)) for (c,k)=(22,1)

Example 3: Given f(n)=𝒏𝟑 +5n+3 and g(n)= 𝒏𝟐 . Prove that running time f(n) is not
O(g(n)).

Proof: By the Big oh definition, f(n) is O(n) if and only if f(n)<=c.g(n) for all n>=k.
f(n)<=c.g(n)

𝒏𝟑 +5n+3 <=c. 𝒏𝟐
For (c,k)=(9,1), the above statement is valid only once.

𝒏𝟑 +5n+3 <=9. 𝒏𝟐 for n=1 only.

Therefore, the Big-Oh condition can not hold true for the above statement. (the left
side of the latter inequality is growing infinitely, so that there is no such constant
factor c).

Big-Oh and Growth Rate

The big-Oh notation gives an upper bound on the growth rate of a function.
The statement “f(n) is O(g(n))” means that the growth rate of f(n) is no more than the
growth rate of g(n).

If f(n) is a polynomial of degree d, then f(n) is O(nd ), i.e.,

✓ Drop lower-order terms

✓ Drop constant factors

Use the smallest possible class of functions

Say “2n is O(n)” instead of “2n is O(n2)”

Use the simplest expression of the class

Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

Example:
1. For f(n)=4n & g(n)=n 2, prove f(n)=O(g(n))

A valid proof is to find valid c and k

We can then chose k = 4, and c=1

Page 11 of 55
When n=4, f=16 and g=16, so this is the crossing over point

We also have infinitely many others choices for c and k, such as

k = 78, and c=42

K=1, and c=4

Some comment on Notation

We say (3n 2+17) is in O(n 2)

We may also say/write it as

(3n2 +17) is O(n2)

(3n2 +17) = O(n2)

(3n2 +17) ∈ O(n2)

But it’s not ‘=‘ as in ‘equality’:

We would never say O(n2) = (3n2 +17)

2. Big Omega Notation (Ω)

The notation Ω(n) is the formal way to express the lower bound of an algorithm's
running time.

It measures the best case time complexity or the best amount of time an algorithm can
possibly take to complete.
Ω Notation can be useful when we have lower bound on time complexity of an
algorithm.

Definition:
Ω(g(n)) = {f(n): there exist positive constants c and k such

that 0 <= c*g(n) <= f(n) for all n >= k}

Example: If f(n) =n 2, then f(n)= ( n)

In simple terms, f(n)= ( g (n)) means that the growth rate of f(n) is greater than or
equal to g(n).
3. Big Theta Notation ()

The theta notation bounds a functions from above and below, so it defines exact
asymptotic behaviour.
A simple way to get Theta notation of an expression is to drop low order terms and
ignore leading constants.

Page 12 of 55
Big-Theta is a tight bound; both upper and lower bound. Tight bound is more precise,
but also more difficult to compute.

Definition:
Θ(g(n)) = {f(n): there exist positive constants c1, c2 and k such

that 0 <= c1*g(n) <= f(n) <= c2*g(n) for all n >= k}
The above definition means, if f(n) is theta of g(n), then the value f(n) is always between
c1*g(n) and c2*g(n) for large values of n (n >= k).

The definition of theta also requires that f(n) must be non -negative for values of n
greater than k.
If the algorithm for the input n takes 2n2 + 2n + 1 operations to finish, we say that its
time complexity is O(n2 ), but is also O(n3 ) , O(n10 ) , O(n30 ) and O(n100 ) .

However, it is Ө(n2 ) and it is not Ө(n3 ), Ө(n4 ) etc.

Warning!
Algorithm that is Ө(f(n)) is also O(f(n)), but not vice versa!

Example:

Let f(n)=4n2 + 3n + 1 , and g(n)=n2 .

Prove that f(n)= (n2 ).

From the definition of Big-theta notation above, we need to find constants c1,c2 and k
that satisfies the statement below.
c1*g(n) <= f(n) <= c2*g(n) for all n >= k

Let f(n)=4n2 + 3n + 1 , and g(n)=n2 .

Let c1=1 c2=8 and k=1.

The equation becomes

1*g(n) <= f(n) <= 8*g(n) for all n >= 1

n2 <=4n2 + 3n + 1 <=8n2 for k=1

So, the above statement will always holds true for n>=1.
Less common notation:

There are two more notations called

✓ little o and

Page 13 of 55
 ✓ little omega.

Little o provides strict upper bound (equality condition is removed from Big O) and
little omega provides strict lower bound (equality condition removed from big omega).
Little o notation (o)

Definition:
Let f(n) and g(n) be functions that map positive integers to positive real numbers.

We say that f(n) is ο(g(n)) (or f(n) Ε ο(g(n))) if for any real constant c > 0, there exists
an integer constant n0 ≥ 1 such that 0 ≤ f(n) < c*g(n).
In mathematical relation,f(n)=o(g(n))means
𝑓(𝑛)
𝐿𝑖𝑚 =0
n→∞ 𝑔(𝑛)

Example: Prove that: 7n + 8 ∈ o(n 2)

In order for that to be true, for any c, we have to be able to find an n0 that makes f(n)
< c * g(n) asymptotically true.
Then lets check the limits,
(7𝑛+8)
lim f(n)/g(n) =𝐿𝑖𝑚
n→∞ 𝑛2

(7)
= 𝐿𝑖𝑚
n→∞ 𝑛

= 0 (true)
Hence 7n + 8 ∈ o(n 2)

Little ω notation

Definition:
Let f(n) and g(n) be functions that map positive integers to positive real numbers.

We say that f(n) is ω(g(n)) (or f(n) ∈ ω(g(n))) if for any real constant c > 0, there exists
an integer constant n0 ≥ 1 such that f(n) > c * g(n) ≥ 0 for every integer n ≥ n0.

f(n) has a higher growth rate than g(n) so main difference between Big Omega (Ω) and
little omega (ω) lies in their definitions.

In the case of Big Omega f(n)=Ω(g(n)) and the bound is 0<=cg(n)<=f(n), but in case
of little omega, it is true for 0<=c*g(n)<f(n).
We use ω notation to denote a lower bound that is not asymptotically tight.
and, f(n) ∈ ω(g(n)) if and only if g(n) ∈ ο((f(n)).

In mathematical relation,if f(n) ∈ ω(g(n)) then,

Page 14 of 55
 𝑓(𝑛)
𝐿𝑖𝑚 =∞
n→∞ 𝑔(𝑛)

Example: Prove that 4n + 6 ∈ ω(1).

The little omega(ο) running time can be proven by applying limit formula given
below.
𝑓(𝑛)
if 𝐿𝑖𝑚 =∞, then functions f(n) is ω(g(n))
n→∞ 𝑔(𝑛)
Here, we have functions f(n)=4n+6 and g(n)=1
4𝑛+6
𝐿𝑖𝑚 =∞
n→∞ 1
and, also for any c we can get n0 for this inequality 0 <= c*g(n) < f(n), 0 <= c*1 <
4n+6
Hence proved.

Do Not Be Confused!

✓ Best-Case does not always imply (f(n))

✓ Average-Case does not always imply Θ(f(n))

✓ Worst-Case does not always imply O(f(n))

Example: f(n)=𝑛2 +4n and g(n)=𝑛2

So we can say that f(n)=O(𝑛2 ) and also

f(n)= (n2 ) for some constants.

Best, Average, and Worst are specific to the algorithm!

Page 15 of 55
 CHAPTER: TWO
RECURENCE
Contents

1. Introduction to recurrence

2. Substitution method
2. Iteration method

3. Recurrence tree method

4. Master Method

Introduction to recurrence

When an algorithm contains a recursive call to itself, its running time can often be
described by a recurrence.

A recurrence is an equation or inequality that describes a function in terms of its value

on smaller inputs. It is commonly used to describe the runtime of recursive algorithms.

Many algorithms are recursive in nature. When we analyse them, we get a recurrence
relation for time complexity. We get running time on an input of size n as a function of
n. For example, in Merge Sort, to sort a given array, we divide it in two halves and
recursively repeat the process for the two halves. Finally, we merge the results.

The recurrence of Merge Sort can be written as:

T(n) = 2T(n/2) + cn.

There are many other algorithms like Binary Search, and others.

There are mainly four ways for solving recurrences.

✓ Substitution Method

✓ Iteration Method
✓ Recurrence tree method, and

✓ Master Method

1. Substitution Method
The substitution method for solving recurrences is famously described using two steps:

✓ Guess the form of the solution.

✓ Use induction to show that the guess is valid.

Page 16 of 55
N.B: It is powerful method, but applied only in cases when it is easy to guess the form
of the solution. It can be used to establish either an upper or lower bounds on a
recurrence.
General Rules for substitution method

✓ There is no general way to guess the correct solutions to recurrences.

✓ Guessing needs experience and occasionally creativity.

Example:

For example, consider the recurrence T(n) = 2T(n/2) + n

We guess the solution as T(n) = O(nLogn). Now we use induction to prove our guess.

We need to prove that T(n) <= cnLogn.

We can assume that it is true for values smaller than n.

Therefore, T (n/2) <=c(n/2)log(n/2),

T(n) = 2T(n/2) + n
= 2(c(n/2)Log(n/2)) + n

= cnlog(n/2)+n
= cnlog 𝑛 - cnlog 2 + n, since log(2)=1

= cnlog 𝑛 - cn + n

= cnlog 𝑛 which is true for (c>=1 and n>=2), because for n=1,

the equation becomes meaningless. (i.e: log 2 1=0).

Exercise
1. Apply the substitution method to show that the solution

Of T(n)=T(n−1)+n is O(𝑛2 ).

2. Consider the recurrence defined by T(n)=T( 𝑛⁄2 )+1. Show that the solution to this
recurrence is O(log 2 𝑛).

2. Iteration method
✓ No need to guess the solution.

✓ Needs more algebra than the substitution method.

✓ Simply iterate (expand) the recurrence and express it as a summation of terms
dependent only on n and the initial conditions.

Page 17 of 55
Example 1: T(n)= T(n-1)+n for n>1, T(0)= 0, T(1)= 1.

T(n)= T(n-1)+n
= [T(n-2)+n-1]+n= T(n-2)+(n-1)+n

= [T(n-3)+n-2]+(n-1)+n= T(n-3)+(n-2)+(n-1)+n
after i Substitutions----

= T(n-i)+(n-i+1)+(n-i+2)+------+n

at n th iteration
= T(0)+1+2+3+-------+n= n(n+1)/2
1
= (𝑛2 + 𝑛). So, the time complexity becomes T(n)=O(𝒏𝟐 )
2

Example 2: T(n)= 2T(n/2)+4n and T(1)= 4. Solve the recurrence using the iteration
method.

3. Recurrence Tree Method:

In this method, we draw a recurrence tree and calculate the time taken by every level of
tree.

1. Recursion Tree Method is a pictorial representation of an iteration method which is

in the form of a tree where at each level nodes are expanded.
2. In general, we consider the second term in recurrence as root.
3. It is useful when the divide & Conquer algorithm is used.
4. It is sometimes difficult to come up with a good guess. In Recursion tree, each root
and child represents the cost of a single subproblem.
5. We sum the costs within each of the levels of the tree to obtain a set of pre-level costs and
then sum all pre-level costs to determine the total cost of all levels of the recursion.
6. A Recursion Tree is best used to generate a good guess, which can be verified by the
Substitution Method.
Finally, we sum the work done at all levels.

To draw the recurrence tree, we start from the given recurrence and keep drawing till
we find a pattern among levels.
The pattern is typically an arithmetic or geometric series.

𝑛
Example 1 Consider T (n) = 2T( )+ n 2
2

We have to obtain the asymptotic bound using recursion tree method.

Solution: The Recursion tree for the above recurrence is

Page 18 of 55
Example 2: Consider the following recurrence

𝑛
T (n) = 4T( ) +n Obtain the asymptotic bound using recursion tree method.
2

Solution: The recursion trees for the above recurrence

Page 19 of 55
Example 3: Consider the following recurrence

Obtain the asymptotic bound using recursion tree method.

Solution: The given Recurrence has the following recursion tree

When we add the values across the levels of the recursion trees, we get a value of n for every
level. The longest path from the root to leaf is

Page 20 of 55
 Exercise 1. Given a recurrence T(n) =𝑇(𝑛⁄4 )+𝑇( 𝑛⁄2)+ 𝑛2 . Solve using recurrence
tree method.

2. Given a recurrence T(n) = 2𝑇 (𝑛⁄2 )+4n and T(1)=4 . Solve using recurrence tree
method.
4. Master Method:
✓ It is another method for solving recurrences of the form T(n)= aT(n/b)+f(n),
where a>=1 and b>1 are constants and f(n) is asymptotically positive
function.
✓ Requires memorization of three cases.
✓ Can solve many recurrences quite easily.

T(n)= aT(n/b)+f(n) means:

✓ The algorithm divides a problem of size n into a sub problems.

✓ Each sub problem has size of n/b, a and b are positive constants.

✓ The a sub problems are solved recursively each in time T(n/b).

✓ f(n) describes the cost of dividing the problem and combining the results
of the sub problems.

Master Theorem: For Master method:

Let a>=1 and b>1 be constants, let f(n) be a function, and let T(n) be defined on the
non-negative integers by the recurrence T(n)= aT(n/b)+f(n), then T(n) can be bounded
asymptotically as follows:

1. If f(n)= O(n logb a-€) for some constant €>0, then T(n)= (n logb a).

2. If f(n)= (n logba), then T(n)= (n logba log n).

3. If f(n)= (n logb a+€ ) for some constant €>0, then T(n)= (f(n)).

Example 1: Given T(n)= 9T(n/3) +n. Solve the recurrence using the master method.

Soln: We have a=9, b=3, f(n)= n, thus we need to check that

Page 21 of 55
 Is f(n)= nlog ba?

n= n logba= n log3 9= n 2, which is not true!

Since f(n)= O (n log39-€ ), where €=1, we can apply case 1.

So, T(n)= (nlog ba)=  (n2)

Example 2: Given T(n)= 2T(n/2)+n. Solve the recurrence using the master method.
Soln: We have a=2, b=2, f(n)= n, thus we need to check that

Is f(n)= nlog ba ?

n= n logba= n log2 2= n ,which is true!.

Since f(n)= (n logb a ), so we can apply case 2.

So, T(n)= (nlog balog 𝑛)=  (nlog 𝑛)

Example 3: Given T(n)= 4T(n/2)+𝑛3 . Solve the recurrence using the master method.

Soln: We have a=4, b=2, f(n)=𝑛3 , thus we need to check that

Is f(n)= nlog ba ?
Is 𝑛3 = n logba

= n log24

= 𝑛2 ,which is not true!.

Since f(n)= (n logba+€) ,where €=1. so we can apply case 3.

So, T(n)= (f(n))=  (𝑛3 )

Another way of solving recurrences using Master method

Let be expressed as T(n)= aT(n/b)+f(n) and let d be the degree of a function f(n).

𝑎 > 𝑏𝑑 , 𝑡ℎ𝑒𝑛 𝑐𝑎𝑠𝑒 1 𝑎𝑝𝑝𝑙𝑖𝑒𝑠

Then, if { 𝑎 = 𝑏𝑑 , 𝑡ℎ𝑒𝑛 𝑐𝑎𝑠𝑒 2 𝑎𝑝𝑝𝑙𝑖𝑒𝑠
𝑎 < 𝑏𝑑 , 𝑡ℎ𝑒𝑛 𝑐𝑎𝑠𝑒 3 𝑎𝑝𝑝𝑙𝑖𝑒𝑠
Let’s check out the previous examples

1. Given T(n)= 9T(n/3)+n. Solve the recurrence using the master method.
Here we have a=9, b=3 and d=1.

So 9 > 31 which implies that 𝑎 > 𝑏𝑑 . Then we can apply case 2.

Page 22 of 55
 T(n) = (nlogba)

= (nlog39)

= (𝑛2 )

2. Given T(n)= 8T(n/2)+1000𝑛2 . Solve the recurrence using the master method.

Here we have a=8, b=2 and d=2. Then we need to compare a and 𝑏𝑑 .

So 8 > 22 which implies that 𝑎 > 𝑏𝑑 . Then we can apply case 1.

T(n) = (nlogba)

= (nlog28)

= (𝑛3 )

3. Given T(n)= 2T(n/2)+10n. Solve the recurrence using the master method.

Here we have a=2, b=2 and d=[Link] we need to compare a and 𝑏𝑑 .

So 2 = 21 which implies that 𝑎 = 𝑏𝑑 . Then we can apply case 2.

T(n)= (nlogba log n).

= (n log22 log n).

= (nlog n)

Page 23 of 55
 CHAPTER: THREE
SEARCHING AND SORTING ALGORITHMS
1. Linear search

Linear search is a very simple search algorithm. In this type of search, a sequential
search is made over all items one by one. Every item is checked and if a match is found
then that particular item is returned, otherwise the search continues till the end of the
data collection.
Algorithm

Linear Search ( Array A, Value x)

Step 1: Set i to 1
Step 2: if i > n then go to step 7

Step 3: if A[i] = x then go to step 6

Step 4: Set i to i + 1

Step 5: Go to Step 2
Step 6: Print Element x Found at index i and go to step 8

Step 7: Print element not found

Step 8: Exit
Pseudocode

function linear_search (list, value)

for each item in the list

if match item == value

return the item's location

end if , end for

end function
2. Binary search

Binary search is a fast search algorithm with run-time complexity of Ο(log n). This
search algorithm works on the principle of divide and conquer. For this algorithm to
work properly, the data collection should be in the sorted form.

Page 24 of 55
Binary search looks for a particular item by comparing the middle most item of the
collection. If a match occurs, then the index of item is returned. If the middle item is
greater than the item, then the item is searched in the sub-array to the right of the middle
item. Otherwise, the item is searched for in the sub-array to the left of the middle item.
This process continues on the sub-array as well until the size of the subarray reduces to
zero.

Pseudo code
Procedure binary_search

A ← sorted array

n ← size of array
x ← value ot be searched

Set lowerBound = 0
Set upperBound = n-1

while x not found

if upperBound < lowerBound

EXIT: x does not exists.

set midPoint = (lowerBound + upperBound) / 2

if A[midPoint] < x

set lowerBound = midPoint + 1

if A[midPoint] > x
set upperBound = midPoint - 1

if A[midPoint] = x
EXIT: x found at location midPoint

end while
end procedure

Page 25 of 55
 ADVANCED SEARCHING ALGORITHMS
3. Interpolation search

Interpolation search is an improved variant of binary search. This search algorithm

works on the probing position of the required value. For this algorithm to work properly,
the data collection should be in a sorted form and equally distributed.
Positioning in Binary Search

In binary search, if the desired data is not found then the rest of the list is divided in two
parts, lower and higher. The search is carried out in either of them.
Even when the data is sorted, binary search does not take advantage to probe the
position of the desired data.

Position Probing in Interpolation Search

Interpolation search finds a particular item by computing the probe position. Initially,
the probe position is the position of the middle most item of the collection.

If a match occurs, then the index of the item is returned. To split the list into two parts,
we use the following method:
pos = low + [ (x-list[low])*(high-low) / (list[high]-list[low]) ]

list[] ==> List (initial search space)

x ==> Element to be searched

low ==> Starting index in arr[]

high ==> Ending index in arr[]

N.B: The idea of the formula is to return higher value of pos when element to be
searched is closer to arr[hi]. And smaller value when closer to arr[lo]
If the middle item is greater than the item, then the probe position is again calculated in
the sub-array to the right of the middle item. Otherwise, the item is searched in the sub-
array to the left of the middle item. This process continues on the sub -array as well until
the size of subarray reduces to zero.

Runtime complexity of interpolation search algorithm is Ο(log (log n)).

Algorithm:

Step 1 − Start searching data from middle of the list.

Page 26 of 55
 Step 2 − If it is a match, return the index of the item, and exit.

Step 3 − If it is not a match, probe position.

Step 4 − Divide the list using probing formula and find the new middle.

Step 5 − If data is greater than middle, search in higher sub -list.

Step 6 − If data is smaller than middle, search in lower sub -list.

Step 7 − Repeat until match.

Pseudocode:
A → Array list

N → Size of A
X → Target Value

Procedure Interpolation_Search()
Set Lo → 0

Set Mid → -1

Set Hi → N-1
While X does not match

if Lo equals to Hi OR A[Lo] equals to A[Hi]

EXIT: Failure, Target not found

end if

Set Mid = low + [ (x-A[low])*(high-low) / (A[high]-A[low]) ]

if A[Mid] = X

EXIT: Success, Target found at Mid

else

if A[Mid] < X

Set Lo to Mid+1
else if A[Mid] > X

Set Hi to Mid-1
end if

end if

Page 27 of 55
 End While

End Procedure
4. Jump search

Jump search is a searching algorithm for sorted arrays. The basic idea is to check
fewer elements by jumping ahead by fixed steps or skipping some elements in place of
searching all elements.
Algorithm for jump search

The steps involved in this algorithm are:

(Block size: B and list size: N; list is sorted in ascending order)

Step 1: Start from first index

Step 2: Jump head by B elements. Current position = Current position + B. If position

is out of element list, set current position to last position.

Step 3: If element at current position < target element, then do Linear Search on element
from position current position -B to current position else go to step 2. If current position
is last position, go to step 4.
Step 4: Exit. Element not found.

Working principle (Implementation) of jump search

int jump search(int arr[], int n, int x){

// Finding block size to be jumped

int step = sqrt(n);

// Finding the block where element is present (if it is present)

int prev = 0;

while (arr[min(step, n)-1] < x) {

prev = step;

step += sqrt(n);
if (prev >= n)

return -1; }

// Doing a linear search for x in block beginning with prev.

while (arr[prev] < x) {

Page 28 of 55
 prev++;

// If we reached next block or end of array, element is not present.

if (prev == min(step, n))

return -1; }
// If element is found

if (arr[prev] == x)

return prev;
return -1;

For example, suppose we have an array arr[] of size n and block (to be jumped) size m.
Then we search at the indexes arr[0], arr[m], arr[2m]…..arr[km] and so on.
Once we find the interval (arr[km] < x < arr[(k+1)m]), we perform a linear search
operation from the index km to find the element x.

How to find a perfect block size?

In the worst case, we have to do N/B jumps and if the element is not present, we perform
B-1 comparisons.

Therefore, the total number of comparisons in the worst case will be ((N/B) + B-1). The
value of the function ((N/B) + B-1) will be minimum when B = √N.
Therefore, the best block size is B = √N.

Time complexity of jump search

Worst case time complexity: O(√N)

Average case time complexity: O(√N)

Best case time complexity: O(1)

Important points in jump search

✓ Works only sorted arrays.

✓ The optimal size of a block to be jumped is (√ n). This makes the time complexity
of Jump Search O(√ n).

✓ The time complexity of Jump Search is between Linear Search ( ( O(n) ) and
Binary Search ( O (Log n) ).
✓ Binary Search is better than Jump Search, but Jump search has an advantage that
we traverse back only.

Page 29 of 55
Example (solved in lecture class):

Find the element 22 from the below array using jump search algorithm:

4 6 8 10 13 14 20 22 25 30

5. Exponential search

Exponential search algorithm is a search algorithm for searching sorted,

unbounded/infinite lists.
Exponential search involves two basic steps:

✓ Find range where element is present

✓ Execute Binary Search algorithm in above found range.

How to find the range where element may be present?

The idea is to start with sub-list of size 1. Compare the last element of the list with the
target element, then try size 2, then 4 and so on until last element of the list is not greater.
Once we find a location loc (after repeated doubling of list size), we know that the
element must be present between loc/2 and loc.

Working principle(implementation) of exponential search

int exponential_search(int arr[],int n,int k){

if(arr[0]==k)
return 0;

int i=1;

while(i<=n&&arr[min(i,(n-1))]<k){
i=i*2;}

return binary_search(arr,i/2,min(i,(n-1)),k); }
//Perform binary search

int binary_search(int arr[],int l,int r,int k){

while(l<=r){
int mid=(l+r)/2;

Page 30 of 55
 if(arr[mid]==k)

return mid;
else if(arr[mid]>k)

r=mid-1;
else

l=mid+1; }

return -1; }
Time complexity of exponential search

✓ Worst case time complexity: O(log n)

✓ Average case time complexity: O(log n)

✓ Best case time complexity: O(1)

Applications

Exponential Binary Search is useful for unbounded searches where size of array is
infinite.

It works better than Binary Search for bounded arrays when the element to be searched
is closer to the beginning of the array.

Example (solved in lecture class):

Find the element 80 from the below array using exponential search algorithm:

8 10 13 20 36 37 40 45 60 80

Page 31 of 55
 SORTING ALGORITHMS
Sorting is one of the most important operations performed by computers. It is a process
of reordering a list of items in either increasing or decreasing order.

Following are some examples of essentiality of sorting in real-life scenarios:

✓ Telephone Directory – The telephone directory stores the telephone numbers of
people sorted by their names, so that the names can be searched easily.

✓ Dictionary – The dictionary stores words in an alphabetical order so that

searching of any word becomes easy.

Sorting can be:

✓ In - place sorting Ex: insertion sort, bubble sort and

✓ Not - in - place sorting Ex: Merge sort

Some important terms in sorting techniques

Increasing Order

A sequence of values is said to be in increasing order, if the successive element is greater

than the previous one. For example, 1, 3, 4, 6, 8, 9 are in increasing order, as every next
element is greater than the previous element.

Decreasing Order
A sequence of values is said to be in decreasing order, if the successive element is less

than the current one. For example, 9, 8, 6, 4, 3, 1 are in decreasing order, as every next
element is less than the previous element.

Non-Increasing Order
A sequence of values is said to be in non-increasing order, if the successive element is

less than or equal to its previous element in the sequence. This order occurs when the

sequence contains duplicate values. For example, 9, 8, 6, 3, 3, 1 are in non -increasing

order, as every next element is less than or equal to (in case of 3) but not greater than

any previous element.

Non-Decreasing Order

A sequence of values is said to be in non-decreasing order, if the successive element is

Page 32 of 55
greater than or equal to its previous element in the sequence. This order occurs when
the sequence contains duplicate values. For example, 1, 3, 3, 6, 8, 9 are in non -
decreasing order, as every next element is greater than or equal to (in case of 3) but not
less than the previous one.
The following are simple sorting algorithms used to sort small-sized lists.

✓ Insertion Sort

✓ Selection Sort
✓ Bubble Sort

1. Insertion sort
This is an in-place comparison-based sorting algorithm. Here, a sub-list is maintained
which is always sorted. For example, the lower part of an array is maintained to be
sorted.

An element which is to be 'inserted’ in this sorted sub-list, has to find its appropriate
place and then it has to be inserted there. Hence the name, insertion sort.
The array is searched sequentially and unsorted items are moved and inserted in to the
sorted sub-list (in the same array). This algorithm is not suitable for large data sets as
its average and worst case complexity are of Ο(𝒏𝟐 ), where n is the number of items.

How insertion sort works?

In Insertion sort, the list is divided into two parts: the sorted part followed by the
unsorted part.

In each step, pick the first element from the unsorted part and insert in the correct
position in the sorted part.
Initially, the sorted part is empty and finally, the unsorted part is empty.

Working principle (Implementation) of insertion sort

void insertionSort(int arr[], int n) {

int key, j;

for (int i = 1; i < n; i++) {

key = arr[i];

j = i - 1;
while (j >= 0 && arr[j] > key) {

arr[j + 1] = arr[j];

Page 33 of 55
 j = j - 1; }

arr[j + 1] = key; } }
Example: Sort the elements [12,11,13,5,6] using insertion sort

12, 11, 13, 5, 6

Let us loop for i = 1 (second element of the array) to 4 (last element of the array)

i = 1. Since 11 is smaller than 12, move 12 and insert 11 before 12

11, 12, 13, 5, 6
i = 2. 13 will remain at its position as all elements in A[0..I-1] are smaller than 13
11, 12, 13, 5, 6

i = 3. 5 will move to the beginning and all other elements from 11 to 13 will move one
position ahead of their current position.
5, 11, 12, 13, 6
i = 4. 6 will move to position after 5, and elements from 11 to 13 will move one
position ahead of their current position.
5, 6, 11, 12, 13

Time complexity of insertion sort

✓ If we have n values in our array, Insertion Sort has a time complexity of O(n²) in
the worst case.

✓ In the best case, we already have a sorted array but we need to go through the
array at least once to be sure! Therefore, in the best case, Insertion Sort
takes O(n) time complexity.
✓ Since it is in-place sorting, space complexity is O(1).

2. Selection sort

Selection sort is a simple sorting algorithm. This sorting algorithm is an in -place

comparison-based algorithm in which the list is divided into two parts, the sorted part
at the left end and the unsorted part at the right end. Initially, the sorted part is empty
and the unsorted part is the entire list.

The smallest element is selected from the unsorted array and swapped with the leftmost
element, and that element becomes a part of the sorted array. This process continues
moving unsorted array boundary by one element to the right.
This algorithm is not suitable for large data sets as its average and worst case
complexities are of O(n²), where n is the number of items.

Page 34 of 55
Selection sort works as follows:

1. Find the minimum value in the list

2. Swap it with the value in the first position

3. Repeat the steps above for remainder of the list (starting at the second position)
Working principle(Implementation) of selection sort

void selectionSort(int arr[], int n) {

for(int i = 0; i < (n-1); i++) {

int min_indx=i;

for(int j=i+1;j<n;j++) {
if(arr[j]<arr[min_indx]){

min_indx=j;
}}

swap(arr,i,min_indx); } }

//swap the elements

void swap(int arr[], int firstIndex, int min_indx)

{
int temp;

temp = arr[firstIndex];

arr[firstIndex] = arr[min_indx];
arr[min_indx] = temp;

}
Example: Sort arr[] = [64, 25, 12, 22, 11] using selection sort

Step 1: Find the minimum element in arr[0...4] and place it at beginning

11 25 12 22 64
Step 2: Find the minimum element in arr[1...4] and place it at beginning of arr[1...4]

11 12 25 22 64
Step 3: Find the minimum element in arr[2...4] and place it at beginning of arr[2...4]

11 12 22 25 64

Page 35 of 55
Step 4: Find the minimum element in arr[3...4] and place it at beginning of arr[3...4]

11 12 22 25 64
Time and Space Complexity of selection sort

✓ To sort an array with Selection Sort, you must iterate through the array once for
every value you have in the array.
✓ If we have n values in our array, Selection Sort has a time complexity of O(n²) in
the worst case.

✓ In the best case, we already have a sorted array but we need to go through the
array O(n²) times to be sure!
✓ Therefore, Selection Sort’s best and worst case time complexity are the same.

✓ Space complexity is O(1), since it is in-place sorting

3. Bubble sort

It is also known as exchange sort. It is a simple sorting algorithm.

It works by repeatedly stepping through the list to be sorted, comparing two items at a
time and swapping them if they are in the wrong order.
The pass through the list is repeated until no swaps are needed, which means the list is
sorted.

Algorithm of bubble sort

Step 1: Compare two adjacent elements and swap them if they are not in the correct
order

Step 2: Do this for all elements in the list

Step 3: If even one element has been swapped in the above run of N elements, go to
Step 1 and repeat the process. If no element has been swapped in the above run for N
elements, stop. The list is sorted.

Working principle(implementation) of bubble sort

// An optimized version of Bubble Sort

void bubbleSort(int arr[], int n) {

int i, j;
bool swapped;

for (i = 0; i < n-1; i++) {

Page 36 of 55
 swapped = false;

for (j = 0; j < n-i-1; j++) {

if (arr[j] > arr[j+1]) {

swap(arr,j, (j+1));
swapped = true; } }

// IF no two elements were swapped by inner loop, then break

if (swapped == false)
break; } }

Example: Sort the array [ 5, 1, 4, 2, 8 ]

First Pass (i=0): ( 5 1 4 2 8 ) –> ( 1 5 4 2 8 ),

Here, algorithm compares the first two elements, and swaps since 5 > 1.

( 1 5 4 2 8 ) –> ( 1 4 5 2 8 ), Swap since 5 > 4.

( 1 4 5 2 8 ) –> ( 1 4 2 5 8 ), Swap since 5 > 2.

( 1 4 2 5 8 ) –> ( 1 4 2 5 8 ),

Now, since these elements are already in order (8 > 5), algorithm does not swap
them.
Second Pass (i=1): ( 1 4 2 5 8 ) –> ( 1 4 2 5 8 )

( 1 4 2 5 8 ) –> ( 1 2 4 5 8 ), Swap since 4 > 2

( 1 2 4 5 8 ) –> ( 1 2 4 5 8 )
( 1 2 4 5 8 ) –> ( 1 2 4 5 8 )

Now, the array is already sorted, but our algorithm does not know if it is completed.
The algorithm needs another pass without any swap to know it is sorted.
Third Pass (i=2) : ( 1 2 4 5 8 ) –> ( 1 2 4 5 8 )

(1 2 4 5 8) –> (1 2 4 5 8)
(1 2 4 5 8) –> (1 2 4 5 8)

(1 2 4 5 8) –> (1 2 4 5 8)

Finally, the algorithm terminates.

Page 37 of 55
 Time and space complexity of bubble sort

✓ In Bubble Sort, n-1 comparisons will be done in the 1st pass, n-2 in 2nd pass, n-
3 in 3rd pass and so on. So the total number of comparisons will be:
(n-1) + (n-2) + (n-3) + ..... + 3 + 2 + 1

Sum = n(n-1+1)/2

T(n)= O(𝒏𝟐 ).

✓ Worst and Average Case Time Complexity: O(𝑛2 ). Worst case occurs when
array is reverse sorted.

✓ Best Case Time Complexity: O(n). Best case occurs when array is already sorted.
✓ Space Complexity is: O(1), since it is in-place sorting

Working principle(implementation) of bubble sort

// An optimized version of Bubble Sort

void bubbleSort(int arr[], int n) {

int i, j;
bool swapped;

for (i = 0; i < n-1; i++) {

swapped = false;

for (j = 0; j < n-i-1; j++) {

if (arr[j] > arr[j+1]) {

swap(arr,j, (j+1));

swapped = true; } }
// IF no two elements were swapped by inner loop, then break

if (swapped == false)

break; } }

Example: Sort the array [ 5, 1, 4, 2, 8 ]

First Pass (i=0): ( 5 1 4 2 8 ) –> ( 1 5 4 2 8 ),

Here, algorithm compares the first two elements, and swaps since 5 > 1.

Page 38 of 55
( 1 5 4 2 8 ) –> ( 1 4 5 2 8 ), Swap since 5 > 4.

( 1 4 5 2 8 ) –> ( 1 4 2 5 8 ), Swap since 5 > 2.

( 1 4 2 5 8 ) –> ( 1 4 2 5 8 ),

Now, since these elements are already in order (8 > 5), algorithm does not swap
them.
Second Pass (i=1): ( 1 4 2 5 8 ) –> ( 1 4 2 5 8 )

( 1 4 2 5 8 ) –> ( 1 2 4 5 8 ), Swap since 4 > 2

( 1 2 4 5 8 ) –> ( 1 2 4 5 8 )

( 1 2 4 5 8 ) –> ( 1 2 4 5 8 )

Now, the array is already sorted, but our algorithm does not know if it is completed.
The algorithm needs another pass without any swap to know it is sorted.
Third Pass (i=2) : ( 1 2 4 5 8 ) –> ( 1 2 4 5 8 )

(1 2 4 5 8) –> (1 2 4 5 8)
(1 2 4 5 8) –> (1 2 4 5 8)

(1 2 4 5 8) –> (1 2 4 5 8)
Finally, the algorithm terminates.

Time and space complexity of bubble sort

✓ In Bubble Sort, n-1 comparisons will be done in the 1st pass, n-2 in 2nd pass, n-
3 in 3rd pass and so on. So the total number of comparisons will be:
(n-1) + (n-2) + (n-3) + ..... + 3 + 2 + 1

Sum = n(n-1+1)/2

T(n)= O(𝒏𝟐 ).

✓ Worst and Average Case Time Complexity: O(𝑛2 ). Worst case occurs when
array is reverse sorted.

✓ Best Case Time Complexity: O(n). Best case occurs when array is already sorted.
✓ Space Complexity is: O(1), since it is in-place sorting

Page 39 of 55
 ADVANCED SORTING ALGORITHMS

1. Quick Sort Algorithm

General used sites for document preparation
1. [Link]
2. [Link]

3. [Link]
4. [Link] (Very important site)
Quick Sort is also based on the concept of Divide and Conquer, just like merge sort. But in
quick sort all the heavy lifting (major work) is done while dividing the array into subarrays,
while in case of merge sort, all the real work happens during merging the subarrays. In case of
quick sort, the combine step does absolutely nothing.
It is also called partition-exchange sort. This algorithm divides the list into three main parts:

1. Elements less than the Pivot element

2. Pivot element (Central element)
3. Elements greater than the pivot element

Pivot element can be any element from the array, it can be the first element, the last element
or any random element. In this tutorial, we will take the rightmost element or the last element
as pivot.
The general algorithm for Quicksort is given below.
quicksort (A, low, high)

begin

Declare array A[N] to be sorted

low = 1st element; high = last element; pivot

if(low < high)

begin

pivot = partition (A, low, high);

quicksort (A, low, pivot-1)

quicksort (A, pivot+1,high)

Page 40 of 55
 End

end

For example: In the array {52, 37, 63, 14, 17, 8, 6, 25}, we take 25 as pivot. So after the first
pass, the list will be changed like this.

{6 8 17 14 25 63 37 52}
Hence after the first pass, pivot will be set at its position, with all the elements smaller to it
on its left and all the elements larger than to its right. Now 6 8 17 14 and 63 37 52 are
considered as two separate sub arrays, and same recursive logic will be applied on them, and
we will keep doing this until the complete array is sorted.

How Quick Sorting Works?

Following are the steps involved in quick sort algorithm:

1. After selecting an element as pivot, which is the last index of the array in our case, we
divide the array for the first time.
2. In quick sort, we call this partitioning. It is not simple breaking down of array into 2
subarrays, but in case of partitioning, the array elements are positioned so that all the
elements smaller than the pivot will be on the left side of the pivot and all the
elements greater than the pivot will be on the right side of it.
3. And the pivot element will be at its final sorted position.
4. The elements to the left and right, may not be sorted.
5. Then we pick subarrays, elements on the left of pivot and elements on the right
of pivot, and we perform partitioning on them by choosing a pivot in the subarrays.

Complexity Analysis of Quick Sort

For an array, in which partitioning leads to unbalanced subarrays, to an extent where on the
left side there are no elements, with all the elements greater than the pivot, hence on the right
side.
And if keep on getting unbalanced subarrays, then the running time is the worst case, which
is O(n2 ).

Whereas if partitioning leads to almost equal subarrays, then the running time is the best,
with time complexity as O (n*log n).

Worst Case Time Complexity: O(n2 )

Best Case Time Complexity: O (n*log n)

Page 41 of 55
 2. Merge Sort Algorithm
Merge Sort follows the rule of Divide and Conquer to sort a given set of numbers/elements,
recursively, hence consuming less time. Merge sort, runs in O (n*log n) time in all the cases.
Before jumping on to, how merge sort works and its implementation, first let’s understand what
is the rule of Divide and Conquer?

Divide and Conquer

If we can break a single big problem into smaller sub-problems, solve the smaller sub-
problems and combine their solutions to find the solution for the original big problem, it
becomes easier to solve the whole problem.
Let's take an example, Divide and Rule.

When Britishers came to India, they saw a country with different religions living in harmony,
hardworking but naive citizens, unity in diversity, and found it difficult to establish their
empire. So, they adopted the policy of Divide and Rule. Where the population of India was
collectively a one big problem for them, they divided the problem into smaller problems, by
instigating rivalries between local kings, making them stand against each other, and this
worked very well for them.

Well, that was history, and a socio-political policy (Divide and Rule), but the idea here is, if
we can somehow divide a problem into smaller sub-problems, it becomes easier to eventually
solve the whole problem.
In Merge Sort, the given unsorted array with n elements, is divided into n subarrays, each
having one element, because a single element is always sorted in itself. Then, it repeatedly
merges these subarrays, to produce new sorted subarrays, and in the end, one complete sorted
array is produced.

The concept of Divide and Conquer involves three steps:

1. Divide the problem into multiple small problems.

2. Conquer the sub problems by solving them. The idea is to break down the problem
into atomic sub problems, where they are actually solved.
3. Combine the solutions of the sub problems to find the solution of the actual problem.

Page 42 of 55
How Merge Sort Works?
As we have already discussed that merge sort utilizes divide-and-conquer rule to break the
problem into sub-problems, the problem in this case being, sorting a given array.
In merge sort, we break the given array midway, for example if the original array had 6
elements, then merge sort will break it down into two subarrays with 3 elements each. But
breaking the original array into 2 smaller subarrays is not helping us in sorting the array. So,
we will break these subarrays into even smaller subarrays, until we have multiple subarrays
with single element in them. Now, the idea here is that an array with a single element is already
sorted, so once we break the original array into subarrays which has only a single element, we
have successfully broken down our problem into base problems. And then we have to merge
all these sorted subarrays, step by step to form one single sorted array.

General Algorithm
The general pseudo-code for the merge sort technique is given below.
Declare an array Arr of length N
If N=1, Arr is already sorted
If N>1,
Left = 0, right = N-1
Find middle = (left + right)/2
Call merge_sort(Arr,left,middle) =>sort first half recursively
Call merge_sort(Arr,middle+1,right) => sort second half recursively
Call merge(Arr, left, middle, right) to merge sorted arrays in above steps.
Exit

Let's consider an array with values {14, 7, 3, 12, 9, 11, 6, 12}

Below, we have a pictorial representation of how merge sort will sort the given array.

Page 43 of 55
In merge sort we follow the following steps:

1. We take a variable p and store the starting index of our array in this. And we take
another variable r and store the last index of array in it.
2. Then we find the middle of the array using the formula (p + r)/2 and mark the middle
index as q, and break the array into two subarrays, from p to q and from q +
1 to r index.
3. Then we divide these 2 subarrays again, just like we divided our main array and this
continues.
4. Once we have divided the main array into subarrays with single elements, then we
start merging the subarrays.

Page 44 of 55
Complexity Analysis of Merge Sort
Merge Sort is quite fast, and has a time complexity of O (n*log n). It is also a stable sort, which
means the "equal" elements are ordered in the same order in the sorted list. In this section we
will understand why the running time for merge sort is O (n*log n).

[Link] Sort Algorithm

Heap Sort is one of the best sorting methods being in-place and with no quadratic worst-case
running time. Heap sort involves building a Heap data structure from the given array and
then utilizing the Heap to sort the array.
You must be wondering, how converting an array of numbers into a heap data structure will
help in sorting the array. To understand this, let's start by understanding what is a Heap.

What is a Heap?
Heap is a special tree-based data structure, that satisfies the following special heap properties:

1. Shape Property: Heap data structure is always a Complete Binary Tree, which
means all levels of the tree are fully filled.

2. Heap Property: All nodes are either greater than or equal to or less than or equal to
each of its children. If the parent nodes are greater than their child nodes, heap is
called a Max-Heap, and if the parent nodes are smaller than their child nodes, heap is
called Min-Heap.

Page 45 of 55
How Heap Sort Works?
Heap sort algorithm is divided into two basic parts:

Creating a Heap of the unsorted list/array.

Then a sorted array is created by repeatedly removing the largest/smallest element from the
heap, and inserting it into the array. The heap is reconstructed after each removal.
Initially on receiving an unsorted list, the first step in heap sort is to create a Heap data structure
(Max-Heap or Min-Heap). Once heap is built, the first element of the Heap is either largest or
smallest (depending upon Max-Heap or Min-Heap), so we put the first element of the heap in
our array. Then we again make heap using the remaining elements, to again pick the first
element of the heap and put it into the array. We keep on doing the same repeatedly untill we
have the complete sorted list in our array.
In the below algorithm, initially heapsort() function is called, which calls heapify() to build the
heap.
[Link] Sort algorithm

Shell sort is often termed as an improvement over insertion sort. In insertion sort, we take
increments by 1 to compare elements and put them in their proper position.
In shell sort, the list is sorted by breaking it down into a number of smaller sub lists. It’s not
necessary that the lists need to be with contiguous elements. Instead, shell sort technique uses
increment i, which is also called “gap” and uses it to create a list of elements that are “i”
elements apart.

Page 46 of 55
Time Complexity: Time complexity of shell sort is O(n2). The gap is reducing by half in every
iteration.

The Algorithm for shell sort is given below.

shell_sort (A, N)

where A – list to be sorted; N – gap_size

set gap_size = N, flag = 1

while gap_size > 1 or flag = 1, repeat

begin

set flag = 0

set gap_size = (gap_size + 1)/2

end
for i = 0 to i< (N-gap_size) repeat

begin

if A[i + gap_size] > A[i]

swap A[i + gap_size], A[i]

set flag = 0

end
end

Page 47 of 55
 CHAPTER 4

ALGORITHM DESIGN TECHNIQUES

No matter which programming language you use for programming, it is important to
learn algorithm design techniques in data structures in order to be able to build scalable
systems.

Selecting a proper design technique for algorithms is a complex but important task.
Following are some of the main algorithm design techniques:
1. Brute-force or exhaustive search
2. Divide and Conquer
3. Greedy Algorithms
4. Dynamic Programming
5. Branch and Bound Algorithm
6. Randomized Algorithm
7. Backtracking
8. Amortized analysis
A given problem can be solved in various different approaches and some approaches
deliver much more efficient results than others. Algorithm analysis is a technique used
to measure the effectiveness and performance of the algorithms. It helps to determine
the quality of an algorithm based on several parameters such as user -friendliness,
maintainability, security, space usage and usage of other resources.

Selecting a proper designing technique for a parallel algorithm is the most difficult and
important task. Most of the parallel programming problems may have more than one
solution. In this chapter, we will discuss the following designing techniques for par allel
algorithms −

Divide and conquer

Greedy Method
Dynamic Programming

Backtracking
Branch & Bound

Linear Programming

Divide and Conquer Method

Page 48 of 55
In the divide and conquer approach, the problem is divided into several small sub-
problems. Then the sub-problems are solved recursively and combined to get the
solution of the original problem.
The divide and conquer approach involves the following steps at each level −

Divide − The original problem is divided into sub-problems.

Conquer − The sub-problems are solved recursively.

Combine − The solutions of the sub-problems are combined together to get the solution
of the original problem.

Greedy Method
In greedy algorithm of optimizing solution, the best solution is chosen at any moment.
A greedy algorithm is very easy to apply to complex problems. It decides which step
will provide the most accurate solution in the next step.

This algorithm is a called greedy because when the optimal solution to the smaller
instance is provided, the algorithm does not consider the total program as a whole. Once
a solution is considered, the greedy algorithm never considers the same solution aga in.
A greedy algorithm works recursively creating a group of objects from the smallest
possible component parts. Recursion is a procedure to solve a problem in which the
solution to a specific problem is dependent on the solution of the smaller instance of
that problem.

Dynamic Programming
Dynamic programming is an optimization technique, which divides the problem into
smaller sub-problems and after solving each sub-problem, dynamic programming
combines all the solutions to get ultimate solution. Unlike divide and conquer method,
dynamic programming reuses the solution to the sub-problems many times. Recursive
algorithm for Fibonacci Series is an example of dynamic programming.
Backtracking Algorithm

Backtracking is an optimization technique to solve combinational problems. It is

applied to both programmatic and real-life problems. Eight queen problem, Sudoku
puzzle and going through a maze are popular examples where backtracking algorithm
is used.
In backtracking, we start with a possible solution, which satisfies all the required
conditions. Then we move to the next level and if that level does not produce a
satisfactory solution, we return one level back and start with a new option.

Branch and Bound

Page 49 of 55
A branch and bound algorithm is an optimization technique to get an optimal solution
to the problem. It looks for the best solution for a given problem in the entire space of
the solution. The bounds in the function to be optimized are merged with the valu e of
the latest best solution. It allows the algorithm to find parts of the solution space
completely.
The purpose of a branch and bound search is to maintain the lowest -cost path to a target.
Once a solution is found, it can keep improving the solution. Branch and bound search
is implemented in depth-bounded search and depth–first search.

Linear Programming
Linear programming describes a wide class of optimization job where both the
optimization criterion and the constraints are linear functions. It is a technique to get
the best outcome like maximum profit, shortest path, or lowest cost.

In this programming, we have a set of variables and we have to assign absolute values
to them to satisfy a set of linear equations and to maximize or minimize a given linear
objective function.
Dynamic Programming

Dynamic Programming is mainly an optimization over plain recursion. Wherever we

see a recursive solution that has repeated calls for same inputs, we can optimize it using
Dynamic Programming. The idea is to simply store the results of sub -problems, so that
we do not have to re-compute them when needed later. This simple optimization reduces
time complexities from exponential to polynomial. For example, if we write simple
recursive solution for Fibonacci Numbers, we get exponential time complexity and if
we optimize it by storing solutions of sub-problems, time complexity reduces to linear.

Page 50 of 55
Backtracking Algorithms

Backtracking is an algorithmic-technique for solving problems recursively by trying to

build a solution incrementally, one piece at a time, removing those solutions that fail to
satisfy the constraints of the problem at any point of time (by time, here, is referred to
the time elapsed till reaching any level of the search tree).

For example, consider the SudoKo solving Problem, we try filling digits one by one.
Whenever we find that current digit cannot lead to a solution, we remove it (backtrack)
and try next digit. This is better than naive approach (generating all possible
combinations of digits and then trying every combination one by one) as it drops a set
of permutations whenever it backtracks.
Branch and Bound Algorithm

Branch and bound is an algorithm design paradigm which is generally used for solving
combinatorial optimization problems. These problems are typically exponential in
terms of time complexity and may require exploring all possible permutations in worst
case. The Branch and Bound Algorithm technique solves these problems relatively
quickly.

Let us consider the 0/1 Knapsack problem to understand Branch and Bound.

There are many algorithms by which the knapsack problem can be solved:
================================

Brute-force algorithm
It is a type of algorithm that tries a large number of patterns to solve a problem. A
common example of a brute force algorithm is a security threat that attempts to guess a
password using known common passwords.

=====================
Dynamic programming

Dynamic programming, like the divide-and-conquer method, solves problems by

combining the solutions to sub-problems.

A dynamic-programming algorithm solves each sub-problem just once and then saves
its answer in a table, thereby avoiding the work of re-computing the answer every time
it solves each sub-problem.

We typically apply dynamic programming to optimization problems. Such problems

can have many possible solutions. Each solution has a value, and we wish to find a
solution with the optimal (minimum or maximum) value. We call such a solution an

Page 51 of 55
optimal solution to the problem, as opposed to the optimal solution, since there may be
several solutions that achieve the optimal value.

When developing a dynamic-programming algorithm, we follow a sequence of

four steps:
1. Characterize the structure of an optimal solution.

2. Recursively define the value of an optimal solution.

3. Compute the value of an optimal solution, typically in a bottom-up fashion.

4. Construct an optimal solution from computed information.

Amortize Analysis
This analysis is used when the occasional operation is very slow, but most of the
operations which are executing very frequently are faster. Data structures we need
amortized analysis for Hash Tables, Disjoint Sets etc. In the Hash-table, the most of
the time the searching time complexity is O (1), but sometimes it executes O(n)
operations. When we want to search or insert an element in a hash table for most of
the cases it is constant time taking the task, but when a collision occurs, it needs
O(n) times operations for collision resolution.

Page 52 of 55
 Chapter Five
Algorithm for graph problems
An algorithm is a mathematical process to solve a problem using a well-defined or optimal
number of steps. It is simply the basic technique used to get a specific job done. A graph is
an abstract notation used to represent the connection between all pairs of objects. Graphs
are widely-used mathematical structures visualized by two basic components: nodes and
edges. Graph algorithms are used to solve the problems of representing graphs as networks
like airline flights, how the Internet is connected, or social network connectivity on
Facebook. They are also popular in NLP and machine learning to form networks. Some of
the top graph algorithms include:

1. Depth first search

2. Connected Components
3. Topological sort
4. Shortest path
5. Sets of strings
6. Search trees, string sorting, binary search
7. Exact string matching
8. Finding a pattern (string) in a text (string)
9. Approximate string matching
10. Finding in the text something that is
11. similar to the pattern

Page 53 of 55
 Chapter Six
Problem Complexity
Problem complexity refers to the resources (primarily time and space) required by an
algorithm to solve a problem as the input size grows. It's a way to characterize the inherent
difficulty of a problem, independent of specific implementations or hardware. We express
complexity using Big O notation (and related notations).
Analyzing Algorithm Complexity:
1. Identify the Basic Operations: Determine the operations that contribute most to the
running time (e.g., comparisons, assignments, arithmetic operations).
2. Count the Operations: Count how many times these basic operations are executed as a
function of the input size n.
3. Determine the Dominant Term: Identify the term that grows the fastest as n increases.
This is the dominant term.
4. Express in Big O Notation: Drop the constant factors and lower-order terms. Express
the complexity using Big O notation.

Why Complexity Matters

Understanding problem complexity is crucial for:
• Choosing the right algorithm: For a given problem, you might have multiple algorithms
with different complexities. You want to choose the most efficient one for the expected
input size.
• Predicting performance: Knowing the complexity of an algorithm allows you to
estimate how it will perform on large inputs.
• Identifying bottlenecks: If you have a slow program, analyzing the complexity of
different parts can help you identify the performance bottlenecks.
• Comparing algorithms: Complexity provides a way to compare the inherent efficiency
of different algorithms.
P, NP, and NP-Completeness:
• P: The class of problems that can be solved by a deterministic algorithm in polynomial
time.

Page 54 of 55
• NP: The class of problems whose solutions can be verified in polynomial time. It's not
known whether all problems in NP can also be solved in polynomial time (the P vs. NP
problem).
• NP-Complete: The hardest problems in NP. If you can solve any NP-complete problem
in polynomial time, you can solve all problems in NP in polynomial time.
• NP-hard: Problems that are at least as hard as the hardest problems in NP, but they may
not be in NP themselves.

1. Algorithm Vs Problem Complexity

2. The upper and lower bounds
3. Open and Closed problems
4. tractable and intractable problems

Page 55 of 55