Design and Analysis of Algorithms

Introduction to Algorithm and Analysis

What is Algorithm?

ØProgram = Algorithm + Data

ØAlgorithm: Effective procedure for solving a problem in a finite number of steps
Ø It must terminate (end)
Ø It must be Effective, i.e. answer to the problem is found
ØA typical real-life example might be cooking a new recipe
ØDifferent ways of stating an algorithm:
ØStep representation: Statements to follow a step-by-step representation
ØPseudocode Representation: English-like format with restricted vocabulary
ØFlowchart Representation: Graphically-oriented representation with
sequence, decision, repetition actions
History of Algorithm
• The word algorithm was related to to Khwãrezm, an oasis region in
Central Asia along the Amu Darya
• An influential scholar from that region was the 9th-century Persian
mathematician Muḥammad ibn Mūsā al-Khwārizmī
• He was a mathematician, astronomer, geographer, and scholar in the
House of Wisdom in Baghdad!
• He was famous especially for his contributions to the study of algebra
• He wrote a book in Arabic about Hindu-Arabic numerals; the Latin
translation of the book title was Algoritmi de numero Indorum
• The Latin algoritmi became algorithm in English after a few
intermediate transformations
Characteristics of Algorithm
Basic Constructs in Pseudocode
Find largest: Graphical Representation
Find largest: Pseudocode Representation
Input: A list of positive integers
1. Set Largest to 0
2. while (more integers)
2.1 if (the integer is greater than Largest)
then
2.1.1 Set largest to the value of the integer
End if
End while
3. Return Largest
End
Subroutine in An Algorithm: Pseudocode Form
Segregate Even and Odd numbers
• Given an array A[], we are to write a function that segregates even
and odd numbers
• The functions should put all even numbers first, and then odd
numbers
• Example:
Input = {12, 34, 45, 9, 8, 90, 3}
Output = {12, 34, 8, 90, 45, 9, 3}
• Note: No restriction imposed on the order of the elements in each
segments
Segregate Even and Odd numbers: Algorithm
Algorithm: segregateEvenOdd
Input: Array A[size]
Output: Segregated Array
1) Initialize two index variables left and right:
left = 0, right = size -1
2) Keep incrementing left index until we see an odd number
3) Keep decrementing right index until we see an even number
4) If left < right then swap A[left] and A[right]
What is the Course About?
• The theoretical study of design and analysis of computer algorithms
• Analysis: predict the cost of an algorithm in terms of resources and
performance
• Design: design algorithms which minimize the cost
• Objective: Improve the performance of algorithm
• Running Time
• Used Space
• Energy Consumed, etc.
• Note: Performance often draws the line between what is possible and
what is impossible
Efficiency of Algorithms
• Computer Scientists do not mere write programs!!!
• They also analyze them
• Question: How efficient is a program?
ØHow much time does a program take to complete?
ØHow much memory does a program use during execution?
ØHow do these change as the amount of data changes?
ØWhat is the difference between the best case, the average case and worst
case efficiency, if any?
ØWhich one of the above is preferred measure and when?
Measuring Efficiency
• How many computations will a program (method, algorithm) perform
to get the desired answer?
• A Simplified Notion to measure performance of an algorithm:
• View the algorithms as a program
• D ete r m i n e by a n a l ys i s t h e to ta l n u m b e r exe c u ta b l e state m e nt s
(computations) in program or method as a function of the amount of data
• Focus on the dominant term in the function
• To a n a l y z e a n a l g o r i t h m , w e m u s t h a v e a m o d e l o f t h e
implementation technology
• We assume a simplified machine model: Random Access Machine
(RAM)
Random Access Machine (RAM) Model
• A generic one processor machine model of computation
• Instructions are executed sequentially (and not concurrently)
• Restrictions in RAM model:
• We do not go too deep (into the machine instructions, say)
• We do not abuse the notion by assuming unrealistic instruction in instruction set (say
by assuming that there exists a single instruction for sorting a list)
• The model has instructions commonly found in real computers:
• Arithmetic (add, subtract, multiply, divide, remainder, floor, ceiling)
• Data movement (load, store, copy)
• Control (conditional and unconditional branch, subroutine call, return)
• Each such instruction takes a constant time to execute
RAM Model: Data Types & Storage
• Two allowed data types: integer and floating point
• Notion of precision is ignored
• There exists a limit on the size of each word of data
• We assume that an integer input of size n is represented by c*log2n bits for
some constant c ≥ 1
• A word is large enough to hold the value of n
• A word cannot be arbitrarily large to hold a huge amount of data (unrealistic
otherwise)
• All memory access takes a constant time step
• Note: Often, the constant is taken as one time step
RAM Model: Some Other Points
• No memory hierarchy: No cache memory or Paging
• Useful for real computers; but too complex to deal in theoretical analyses

• Few gray areas in the model:

• Is exponentiation a linear time operation? NO
• Is computation of 2n a linear time operation??? (Shift Left Operation!!!)

• We require knowledge in logic, inductive reasoning, combinatorics,

probability theory, algebra, and above all observation and intuition!!!
• Despite restrictions, RAM-model analyses are often excellent
predictors of performance on actual machines!!!
Measuring Efficiency: Example
public int sum (int[] Values) {
int result = 0;
for (int i = 0; i < Values.length; i++)
result += Values[i];
return result;
}
• How many statements are executed by method sum in terms of a function
of the length of the input array, Values
• Let N is the size of the input array Values
• Note: N commonly denotes a variable that represents the amount of data
Measuring Efficiency: Counting Up Statements
Øint result = 0; 1
Øint i = 0; 1
Øi < values.length; N+1
Øi++ N
Øresult += values[i]; N
Øreturn result; 1
• T(N) = 3N + 4
• T(N) is the number of executable statements in method sum as a
function of Values.length, which is N, the size of the input to sum
Representing Efficiency
• When determining complexity of an algorithm we want to simplify
things
• We hide some details to make comparisons easier
• So, in our measure of efficiency:
• We focus only on the dominant term in the functional expression
• Ignore other non-dominant terms
• Ignore the coefficient of the dominant term, as well

• Result: The Growth of Function and The Order Notation

Complexity of an Algorithm
• Definition: Complexity of an algorithm is the number of steps that
takes to transform the input data into the desired output
• Each simple operation and each memory access (refer RAM model)
corresponds to a step
• Complexity of an algorithm is a function of the size of the input (or
size of the instance)
• Let us refer the complexity of an algorithm A as CA(n), where n is the
size of the input to the algorithm
Complexity: Various Notions
• Worst-Case Complexity of an algorithm:
• Maximum number of computational steps required for the execution of an algorithm over all
possible inputs of same size
• Provides an upper bound on the complexity of an algorithm
• The complexity that is observed for the most difficult input instances
• The most Pessimistic view

• Best-Case Complexity of an algorithm:

• Minimum number of computational steps required for the execution of an algorithm over all
possible inputs of same size
• Provides a lower bound on the complexity of an algorithm
• The complexity that is observed when one of the easiest possible input instances is picked as
input
• The most Optimistic view
Complexity: Various Notions
• Average Case Complexity of an algorithm:
• The average amount of resources the algorithm consumes assuming some
plausible frequency of occurrences of each input instance
• Possibly the most meaningful one among the complexity measures
• However, Figuring out the average cost is much more difficult than figuring
out the worst case or the best case costs!
• We have to assume a probability distribution for the types of input
instances to the algorithm
• Difficulty: What is the distribution of real-world instances!!!
Complexity: Various Notions

In general this
is the notion
that we use to
characterize the
complexity of
algorithms

We perform
upper bound
analysis on
algorithms
Comparing Algorithms w.r.t. Complexity
• Consider the scenario where algorithm A1 and A2 has complexities:
CA1(n) = 0.5n2, CA2(n) = 5n
• Which one has the larger complexity?

CA2(n) = 5n ≥ CA1(n) = 0.5n2 for n ≤ 10

But,
CA1(n) = 0.5n2 > CA2(n) = 5n for n >10
Comparing Algorithms w.r.t. Complexity
• W h e n we l o o k at t h e c o m p l ex i t y o f a l g o r i t h m s , w e t h i n k
asymptotically

• In other words, we compare two algorithms as the problem sizes

tend to infinity!

• The notion is called the Asymptotic Complexity

Growth Rates
• In general, we only worry about growth rates of algorithms!!!
• Reason 1: Our main objective is to analyze the cost performance of
algorithms asymptotically
• Reasonable in part, because computers get faster and faster every year
• Reason 2: Deriving the exact cost of algorithms often are not feasible,
as they are quite complicated to analyze
• Reason 3: When analyzing an algorithm we are not that interested in
the exact time the algorithm takes to run
• Often we only want to compare two algorithms for the same problem
• The property that makes one algorithm preferred over another is its growth
rate relative to the other algorithm’s growth rate
Big-O Notation
• The most common method and notation for discussing the execution time
of algorithms is Big-O, also spoken Order
• Big-O represents an upper bound
• Definition: For functions f and g from the set of integers to the set of real
numbers, we say f(n) is O(g(n)) to denote the following:
∃C ∈ ℝ+ , k∈ ℕ such that ∀ n > k, |f(n)| ≤ C |g(n)|
• We say “f(n) is big O of g(n)”, or write f(n) = O(g(n)) to refer the above
• Recipe for proving f(n) = O(g(n)):
• Find a real constant C and an integer constant k (called witnesses to the fact that f(x)
is O(g(x))) so that the above inequality holds
Big-O Notation: Graphical View

f(x) is O(g(x))

k
Examples:
• Is 17 n2 – 5 = O(n2)?
 17n 2 5  17n 2 n  1
(c 17, n0  1)
 17n 2 5  O(n 2 )
• Is 35 n3 + 100 = O(n3)?
 35n 3  100  36n 3 n  5
(c  36, n0  5)
 35n 3  100  O(n 3 )
• Is 6 ∙ 2n + n2 = O(2n)?

 6  2n  n2  7  2n n  5
(c  7, n0  5)
 6  2 n  n 2  O(2 n )
Theorem:
Let f ( x)  an x n  an 1 x n 1    a1 x  a0
where a0 , a1 ,, an 1 , an are real numbers.
Then
n
f ( x) is O( x )
Proof of the Theorem

Assume the triangle inequality that states: |x| + |y|  |x + y|

| f ( x) |  | an x n  an 1 x n 1    a1 x  a0 |
 | an | x n  | an 1 | x n 1    | a1 | x  | a0 |
 x n (| an |  | an 1 | / x    | a1 | / x n 1  | a0 | / x n )
 x n (| an |  | an 1 |    | a1 |  | a0 |)
Therefore
| f ( x) | Cx n
When
C  | an |  | an 1 |    | a1 |  | a0 |; x  1
Therefore
f ( x) is O( x n ) C  | an |  | an 1 |    | a1 |  | a0 |; k  1
Estimating Function Growth: Example

Estimate the sum of the first n positive integers

1  2    n  n  n  n  n2
So
2
(1  2    n ) is O(n ) C  1; k  1
Estimating Function Growth: Example

Estimate f(n) = n! and log n!

n! 1 2  3   n
 n  n  n  n
 nn
So
n! is O(n n ), C  1; K  1.
Taking the log
log n! log n n  n log n
So
log n! is O(n log n), C  1; K  1.
Growth Rates: Popular Complexity Classes
n! 2n n3
n2

n log n

n (linear time)

log n

n
Figure 2. Growth rates of some important complexity classes
Growth Rates: Illustration
Assume that we have a machine that can execute 1,000,000 required
operations per second
Algorithm 1 Algorithm 2 Algorithm 3 Algorithm 4 Algorithm 5

Frequency 33n 6nlogn 13n2 3.4n3 2n

count
n=10 < 1 sec < 1 sec < 1 sec < 1 sec < 1 sec
n=10,000 < 1 sec 6 sec 22 min 39 days many many
centuries

Execution time for algorithms with the given time complexities

Big-O: Some Useful Rules
1. If f(n) = O(n2), then f(n) = O(n3), f(n) = O(n4), …

2. If f1(n) = O(g1(n)) and f2(n) = O(g2(n)), then

(f1 + f2)(n) = O(max(g1(n), g2(n)))

3. If f1(n) = O(g(n)) and f2(n) = O(g(n)), then (f1 + f2)(n) = O(g(n))

4. If f1(n) = O(g1(n)) and f2(n) is O(g2(n)), then

(f1 . f2)(n) is O(g1(n) . g2(n))
Big-Ω Notation
• Big- Ω represents a lower bound
• Definition: For functions f and g from the set of integers to the set of
real numbers, we say f(n) is Ω(g(n)) to denote the following:
∃C ∈ℝ +, k∈ℕ such that ∀ n > k, |f(n)| ≥ C |g(n)|
• We say “f(n) is big omega of g(n)”, or write f(n) = Ω(g(n)) to refer the
above

• Note: If f(n) = O(g(n)) then g(n) = Ω(f(n))

Big-Ω Notation

| f ( n) |

c | g ( n ) |

f(n) = (g(n))
k
Figure: Illustrating Ω
Examples: •Is 3n  2  ( n) ?
 3n  2  3  n n  1
(c  3, n0  1)
 3n  2  ( n)
•Is 3n  2  (1) ?
 3n  2  1  1 n  1
(c  1, n0  1)
 3n  2  (1)
value of n0
•Is 6  2 n  n 2  (n100 )?
 6  2 n  n 2  ? n100 n  ?
When n is bigger, 2n will grow faster than n100. ( Yes, you can find n0 )
 6  2 n  n 2  (n100 )
Big-Θ Notation
Definition: f ( n)  ( g ( n)) if and only if  three positive constants, c1, c2, n0 ,

such that c1  | g (n) |  | f (n) |  c 2  | g (n) | n  n0 .

c 2  | g ( n) |

| f ( n) |
f(n) = Θ(g(n))
c1  | g (n) |

n0 Figure: Illustrating Θ
Examples:
• Is 3n + 2 = (n)?
 3n  3n  2  4 n n  2
(c1  3, c 2  4, n 0  2)
 3n  2   ( n )

• Is 3n + 2 = (n2)?
2
 3n  2  (n )
2
 3n  2  (n )
Examples:

• Is 6  2 n  n 2   ( 2 n ) ?
 6  2n  6  2n  n2  7  2n n  4
(c1  6, c 2  7, n 0  4)
 6  2 n  n 2  (2 n )

• Is 4n 3  3n 2  (n 2 ) ?
3 2 2
 4n  3n  O(n )
 4n 3  3n 2  (n 2 )
Some Useful Rules
1) Transitive Property:
If f (n)  O ( g (n)) and g(n)  O(h(n)) then f ( n )  O ( h ( n ))

If f (n)  ( g (n)) and g(n)  (h(n)) then f ( n )   ( h ( n ))

If f (n)  ( g (n)) and g(n)  (h(n)) then f ( n)  ( h( n))

2) Reflexive Property:

f (n)  O ( f (n))
f (n)  ( f (n))
f (n)  ( f (n))
Some Useful Rules

3) Symmetric:

f (n)  ( g (n)) if and only if g (n)  ( f (n))

f (n)  O ( g (n)) if and only if g (n)  ( f (n))

Little-o Notation
• Definition: For functions f and g from the set of integers to the set of
real numbers, we say f(n) = o(g(n)) if
∀c>0, k∈ℕ so that ∀n>k, f(n) ≤ c·g(n)

• Little-o is a kind of strict upper bound

• Example: n2 = o(n2log n)
Little-ω Notation
• Definition: For functions f and g from the set of integers to the set of
real numbers, we say f(n) = ω(g(n)) if
∀c>0, k∈ℕ so that ∀n>k, f(n) ≥ c·g(n)

• Little-ω is a kind of strict lower bound

• Example: n2 = ω(nlogn)

• Note: For functions f and g, f(n) = o(g(n)) if and only if g(n) = ω(f(n))
Sorting Problem
• Input: A sequence of n numbers: a1, a2, a3, …, an
• Output: A permutation (a' 1 , a' 2 , a' 3 , …, a' n ) of the input number
sequence a1, a2, a3, …, an such that a'1 ≤ a'2 ≤ a'3 ≤ … ≤ a'n
Insertion Sort
• Efficient for sorting small numbers (We shall see shortly)

• In-place sort: Takes an array A[0..n-1] (sequence of n elements) and

arranges them in place, so that it is sorted

• Loop Invariant: At the start of each iteration of the algorithm, the subarray a[0,
1, ..., j-1] contains the elements originally in a[0, 1, …, j-1], but in sorted order
Insertion Sort: Pseudo Code
Insertion-Sort(A)
1. For j ← 1 to (length(A) – 1) Do
2. key ← A[j]
//Insert A[j] into the sorted sequence A[0 ... j-1]
3. i←j–1
4. While i > 0 and A[i] > key Do
5. A[i + 1] ← A[i]
6. i←i–1
7. End of While Loop
8. A[i + 1] ← key //as A[i] ≤ key, so we place key on the right side of A[i]
9. End of For Loop
Insertion Sort: Correctness
• Initialization:
• Before the first loop starts, j=1.
• So, A[0] is an array of single element and so is trivially sorted
• Maintenance:
• The outer for loop has its index moving like j=1, 2, …, n-1 (if A has n elements)
• At the beginning of the for loop assume that the array is sorted from A[0, ..., j-1]
• The inner while loop of the jth iteration places A[j] at its correct position
• Thus at the end of the jth iteration, the array is sorted from A[0, ..., j]
• Thus, the invariance is maintained
• Then j becomes j+1.
• Also, using the same inductive reasoning the elements are also the same as in the
original array in the locations A[0, …, j].
Insertion Sort: Correctness
• Termination:
• The for loop terminates when j = n
• Thus, by the previous observations the array is sorted from A[0, …, n-1] and
the elements are also the same as in the original array.

• Thus, the algorithm indeed sorts and is thus correct!

Insertion Sort: Pseudo Code
Insertion-Sort(A)
1. For j ← 1 to (length(A) – 1) Do How many
2. key ← A[j] iterations
//Insert A[j] into the sorted sequence A[0 ... j-1] for each of these
3. i←j–1 loops???
4. While i > 0 and A[i] > key Do
5. A[i + 1] ← A[i]
6. i←i–1
7. End of While Loop
8. A[i + 1] ← key //as A[i] ≤ key, so we place key on the right side of A[i]
9. End of For Loop
Insertion Sort: Run Time Estimation
Insertion-Sort(A) Cost Step Count
1. For j ← 1 to (length(A) – 1) Do c1 n
2. key ← A[j] c2 n-1
3. i←j–1 c3 n-1
4. While i > 0 and A[i] > key Do c4 �=�
�−�
��

5. A[i + 1] ← A[i] c5 �−�

�=�
(�� − �)

6. i←i–1 c6
�−�
(�� − �)
�=�

7. End of While Loop

8. A[i + 1] ← key c7 n-1
9. End of For Loop
n-1 n-1 n-1
T(n)=c1n  c2 ( n  1)  c3 ( n  1)  c4  t j  c5  (t j  1)  c6  (t j  1)  c7 (n-1)
j=1 j=1 j=1
Insertion Sort: Best Case Run Time
• Consider the case when the input array A[0, …, n-1] is already sorted
• For every value of j = 0 to n – 1:
• tj = ???
• While loop at step 4 would find A[i] ≤ key at the very first iteration
• The loop terminate after only the first step of comparison
• So, clearly tj = 1 for all j = 0 to n – 1
n-1 n-1 n-1
T(n)=c1n  c2 ( n  1)  c3 ( n  1)  c4  1  c5  (1  1)  c6  (1  1)  c7 (n-1)
j=1 j=1 j=1

=(c1  c2  c3  c4  c7 ) n  ( c2  c3  c4  c7 )

The best case run time of Insertion Sort is thus a linear function of n
 Insertion Sort: Worst Case Run Time
• Consider the case when the input array A[0, …, n-1] is reverse-sorted
• For every value of j = 0 to n – 1:
• tj = ???
• While loop at step 4 would find A[i] ≥ key at every iteration
• The loop terminate after comparing key with all the elements of A[0, …, j-1]
• So, clearly tj = j for all j = 0 to n – 1
n-1 n-1 n-1
T(n)=c1n  c2 (n  1)  c3 (n  1)  c4  j  c5  (j  1)  c6  (j  1)  c7 (n-1)
j=1 j=1 j=1

c 4 c5 c6 2 c 3c 3c
=(   )n  (c1  c2  c3  4  5  6 )n  (c5  c6  c2  c3  c7 )
2 2 2 2 2 2

The worst case run time of Insertion Sort is thus a quadratic function of n
 Insertion Sort: Average Case Run Time
• Instead of an input of a particular type, all the inputs of the given
size are equally probable here
• For every value of j = 0 to n – 1:
• tj = ???
• We may assume that elements in the array A[0, ..., j-1] are randomly chosen
• Then, on an average, half the elements are greater than key while half are less
• Hence, on an average, tj = j/2 for all j = 0 to n – 1
�−1 �−1 �−1
� � �
�(�) = �1 � + �2 (� − 1) + �3 (� − 1) + �4 2 + �5 2 − 1 + �6 2 − 1 + �7 (� − 1)
�=1 �=1 �=1
�4 �5 �6 2 �4 3�5 3�6
= + + � + �1 + �2 + �3 − − − � + (�5 + �6 − �2 − �3 − �7 )
4 4 4 4 4 4

The average case run time of Insertion Sort is thus a quadratic function of n
Insertion Sort: In a Nutshell
Worst case: Input reverse sorted.
n
T ( n)   ( j )  n 2  [arithmetic series]
j 2
Average case: All permutations equally likely.
n
T ( n)    ( j / 2)  n 2
j 2
Is insertion sort a fast sorting algorithm?
• Moderately so, for small n.
• Not at all, for large n.
Recursion
• Themes
• Recursion
• Recurrence Relations
• Recursive Definitions
• Induction (prove properties of recursive programs and objects defined
recursively)
• Examples
• Tower of Hanoi
• Merge Sort
• GCD
Recurrence Relations
• A Recurrence Relation for the sequence {a n } is an equation that
expresses a n is terms of one or more of the previous terms of the
s e q u e n c e , n a m e l y, a 0 , a 1 , … , a n - 1 , fo r a l l i n t e g e rs n w i t h
n ≥ n0, where n0 is a nonnegative integer
• A sequence {a n } is called a solution of a recurrence relation if its
terms satisfy the recurrence relation
• Note: A recurrence relation is like a recursively-defined sequence, but
without specifying any initial values (Initial Conditions)
• The same recurrence relation can have (and usually has) multiple solutions
based on its choice of initial condition(s)
• If both the initial conditions and the recurrence relation are specified, then
the sequence is uniquely determined
Tower of Hanoi
• Problem Statement:
• There are three towers
• 64 gold disks, with decreasing sizes, placed on the first tower
• You need to move all of the disks from the first tower to the second tower
• Larger disks can not be placed on top of smaller disks
• The third tower can be used to temporarily hold disks
• Challenge: Assume one disk can be moved in 1 second. Is this
possible to move all 64 disks in a week?
• To create an algorithm to solve this problem, it is convenient to
generalize it to an “N-disk” problem, where in our case N = 64
Tower of Hanoi: Recursive Solution

A B C
Tower of Hanoi: Recursive Solution

A B C
Tower of Hanoi: Recursive Solution

A B C
Tower of Hanoi: Recursive Solution

A B C
Tower of Hanoi: Recursive Algorithm
void Hanoi(int n, string a, string b, string c)
{
if (n == 1) /* base case */
Move(a, b);
else { /* recursion */
Hanoi(n-1, a, c, b);
Move(a, b);
Hanoi(n-1, c, b, a);
}
}
Tower of Hanoi: Recursive Solution for N=3

A B C
Tower of Hanoi: Recursive Solution for N=3

A B C
Tower of Hanoi: Recursive Solution for N=3

A B C
Tower of Hanoi: Recursive Solution for N=3

A B C
Tower of Hanoi: Recursive Solution for N=3

A B C
Tower of Hanoi: Recursive Solution for N=3

A B C
Tower of Hanoi: Recursive Solution for N=3

A B C
Tower of Hanoi: Recursive Solution for N=3

A B C
Tower of Hanoi: Correctness

Correctness of Recursive algorithm follows by Inductive Reasoning

Tower of Hanoi: Recurrence Relation
• M(n): The number of moves required by the algorithm to solve the n-
disk problem
• Then,
M(n) = M(n-1) + 1 + M(n-1)

i.e., M(n) = 2M(n-1) + 1, for n ≥ 2

and M(1) = 1
Solving Recurrence Relation
• Four methods for solving Recurrence Relations:

• Iteration Method

• Substitution Method

• Recursion Tree Method

• Master Method
Iteration Method
• Expand the recurrence iteratively

• Express it as a summation of terms of n and the initial condition(s)

• Challenge: Intermediate Expressions might become complicated

Iteration Method: Example 1
• Solve: T(n) = 2T(n-1) + 1, T(1) = 1 [Refer Tower of Hanoi Problem]
T(n) = 2T(n-1) + 1
=2.(2T(n-2) + 1) + 1
= 22 T(n-2) + (2 + 1)
= 22 (2.T(n-3) + 1) + (2 + 1)
= 23 T(n-3) + (22 + 2 + 1)
=…
= 2n-1 T(1) + (2n-2 + … + 2 + 1) = 2n-1 + 2n-2 + … + 2 + 1 = O(2n)
Iteration Method: Example 2
• Solve: T(n) = T(n-1) + 1, T(1) = 1
T(n) = T(n-1) + 1
=(T(n-2) + 1) + 1
= T(n-2) + (1 + 1)
= (T(n-3) + 1) + (1 + 1)
= T(n-3) + (1 + 1 + 1)
=…
= T(1) + (1 + … + 1 + 1) = 1 + 1 + … + 1 + 1 = O(n)
Iteration Method: Example 3
• Solve: T(n) = 4T(n/2) + n2/log n, T(1) = Θ(1)
T(n) = n2/log n + 4T(n/2)
= n2/log n + 4[(n/2)2/log(n/2) + 4T(n/22)]
= n2/log n + n2/log(n/2) + 42T(n/22)
= ...
�−1 �2
= �=0 � + 4 T(n/2 ) [Continuing upto k-th iteration, k ≥ 1]
k k
log � 2

We next choose a value of k which causes our recurrence to reach a

known base case; we know that n/2k = 1 if k = log n
log � −1 �2 2 log � − 1 1
T(n) = �=0 � +4
log n Θ(1) = � �=0 + nlog 4 Θ(1)
log � 2 log � − �
2 ��� � 1
=� �=1 �
+ Θ(n2) = n2Θ(log log n) + Θ(n2) = Θ(n2 log log n)
Substitution Method
• Consists of two main steps:

• Guess the Solution

• Use the mathematical induction to find the boundary condition and show
that the guess is correct

• Challenge: How to get a good guess!!!

Substitution Method: Example 1
• Solve: T(n) = T(n/2) + 1
• Let us guess: T(n) = O(log n)
• i.e., we want to prove that T(n) ≤ c log n for some constant c
• We assume that the above is true for values smaller than n

T(n) = T(n/2) + 1 ≤ c log(n/2) + 1 = c (log n – log 2) + 1

= c log n – c + 1 ≤ c log n, provided c ≥ 1
Hence, by the rule strong induction, the guess is validated
Substitution Method: Example 2
• Solve: T(n) = 2 T(n/2) + n
• Let us guess: T(n) = O(n log n)
• i.e., we want to prove that T(n) ≤ c n log n for some constant c
• We assume that the above is true for values smaller than n

T(n) = 2T(n/2) + n ≤ 2 (c(n/2) log(n/2)) + n = cn (log n – log 2) + n

= cn log n – cn + n = cn log n – (c – 1)n ≤ cn log n, provided c ≥ 1
Hence, by the rule strong induction, the guess is validated
Recursion Tree Method
• Recursion Tree Method is a pictorial representation of an iteration
method which is in the form of a tree where at each level nodes are
expanded
• In general, we consider the second term in recurrence as root
• It is useful when the Divide-n-Conquer algorithm is used
• In a Recursion tree, each intermediate root and child represents the cost of
a single subproblem of the original problem
• We sum the costs within each of the levels of the tree to obtain a set of
pre-level costs (subproblem cost), and then sum all pre-level costs to
determine the total cost of all levels of the recursion (final cost)
• A Recursion Tree is best used to generate a Good Guess for a solution to a
recurrence relation, which can be verified by the Substitution Method at a
later stage
Recursion Tree Method: Example 1

• Solve: T(n) = 2 T(n/2) + n2

• The Recursion tree for the
above recurrence is:

T(n) = n2 + n2/2 + n2/4 + n2/8

+ … log n times
= n2(1 + ½ + ¼ + … )
≤ 2n2
Therefore, T(n) = O(n2)
Recursion Tree Method: Example 2

• Solve: T(n) = 4 T(n/2) + n

• The Recursion tree for the
above recurrence is:

T(n) = n + 2n + 4n + 8n + …
(log2 n times)
= n(1 + 2 + 4 + … log2 n times)
≤ n (2log n – 1)/(2 – 1) = n (n – 1)
Therefore, T(n) = O(n2)
Solving Recurrence Relation: Master Method
• The Master Method is based on Master Theorem stated as follows.
Master Method
• Master theorem applies only when the recurrence relation has the following
form:
T(n) = aT(n/b) + f(n) where a ≥ 1 and b > 1

• Loosely speaking, there are following three cases in Master Method:

• If f(n) = O(nc) where c < logba, then T(n) = Θ(nlogba)

• If f(n) = Θ(nc) where c = logba, then T(n) = Θ(nc logbn)

• If f(n) = Ω(nc) where c > logba plus regularity condition holds, then T(n) = Θ(f(n))
Master Method
The master method is
mainly derived from
the recurrence tree
method

Recurrence tree for

T(n) = aT(n/b) + f(n)
is shown beside:
Master Method: Interpretation
• In the recurrence tree method, we calculate the total work done

• If the work done at leaves is polynomially more, then leaves are the
dominant part, and our result becomes the work done at the leaf level
(Case 1)

• If work done at leaves and root is asymptotically the same, then our result
becomes height multiplied by work done at any level (Case 2)

• If work done at the root is asymptotically more, then our result becomes
work done at the root (Case 3).
Master Method: Examples
• Solve the following Recurrences using Master Method:

a) T(n) = 8T(n/2) + 1000n2

b) T(n) = 9T(n/3) + n
c) T(n) = 2T(n/2) + 10n
d) T(n) = T(n/2) + Θ(1)
e) T(n) = 2T(n/2) + n2
f) T(n) = 3T(n/4) + n log n
g) T(n) = T(√n) + 1
h) T(n) = 2T(n/2) + n log n
i) T(n) = 2T(n/2) + n / log n
Master Method: Examples
• Example 1: T(n) = 8T(n/2) + 1000n2
• Comparing above with T(n) = aT(n/b) + f(n) where a ≥ 1 and b > 1
a = 8, b = 2, f(n) = 1000n2, logba = log28 = 3
Also, f(n) = 1000n2 = O(n3-ε), if we choose ε = 1
• So case 1 of Master theorem is satisfied here
• Hence, according to case 1 of Master theorem,
T(n) = Θ(nlogba) = Θ(n3)
Master Method: Examples
• Example 2: T(n) = 9T(n/3) + n
• Comparing above with T(n) = aT(n/b) + f(n) where a ≥ 1 and b > 1
a = 9, b = 3, f(n) = n, logba = log39 = 2
Also, f(n) = n = O(n2-ε), if we choose ε = 1
• So case 1 of Master theorem is satisfied here
• Hence, according to case 1 of Master theorem,
T(n) = Θ(nlogba) = Θ(n2)
Master Method: Examples
• Example 3: T(n) = 2T(n/2) + 10n
• Comparing above with T(n) = aT(n/b) + f(n) where a ≥ 1 and b > 1
a = 2, b = 2, f(n) = 10n, logba = log22 = 1
Further, f(n) = 10n = Θ(n1) = Θ(nlog22)
• So, It falls on the case 2 of Master theorem, applying which we get:
T(n) = Θ(n log n)
Master Method: Examples
• Example 4: T(n) = T(n/2) + Θ(1)
• Comparing above with T(n) = aT(n/b) + f(n) where a ≥ 1 and b > 1
a = 1, b = 2, f(n) = 10n, logba = log21 = 0
Further, f(n) = Θ(1) = Θ(n0) = Θ(nlog21)
• So, It falls on the case 2 of Master theorem, applying which we get:
T(n) = Θ(log n)
Master Method: Examples
• Example 5: Solve: T(n) = 2T(n/2) + n2
• Comparing above with T(n) = aT(n/b) + f(n) where a ≥ 1 and b > 1
a = 2, b = 2, f(n) = n2, logba = log22 = 1
Also, f(n) = n2 = Ω (n1+ε), if we choose ε = 1
• Also, if we choose c = ½ < 1, then we find
a f(n/b) = 2.(n/2)2 ≤ ½ n2 = c f(n), for all n ≥ 1 [Regularity condition]
• So both conditions of case 3 of Master theorem is satisfied here
• Hence, according to case 3 of Master theorem,
T(n) = Θ(f(n)) = Θ(n2)
Master Method: Examples
• Example 6: Solve: T(n) = 3T(n/4) + n log n
• Comparing above with T(n) = aT(n/b) + f(n) where a ≥ 1 and b > 1
a = 3, b = 4, f(n) = n log n, logba = log43 = 0.793 ≈ 0.8
Also, f(n) = n log n = Ω (n0.8+ε), if we choose ε = 0.2
• Further, a f(n/b) = 3 f(n/4) = 3 (n/4) log (n/4) = ¾ n log n – 3/2 n ≤ ¾ n log n
• From above, a f(n/b) ≤ c f(n) for all n ≥ 1, if we choose c = ¾ < 1
• So both conditions of case 3 of Master theorem are satisfied here
• Hence, according to case 3 of Master theorem,
T(n) = Θ(f(n)) = Θ(n log n)
Master Method: Examples
• Example 7: T(n) = T(√n) + 1
• Note: Sometimes, recurrence relations can’t be directly solved using
standard techniques
• we need to convert the recurrence relation into appropriate form before solving
• Let us substitute n = 2m, when we get T(2m) = T(√2m) + 1 = T(2m/2) + 1
• Letting T(2m) = S(m), we have S(m) = S(m/2) + 1
• Try Master Theorem on Revised Recurrence relation:
a = 1, b = 2, log21 = 0, f(m) = 1 = Θ(m0) = Θ(mlog21)
• So, following case 2, S(m) = Θ(log m)
• Hence, T(n) = Θ(log log n)
Master Method: Limitations
• It is worth noting that the three cases of Master Theorem do not
cover all the possibilities for f(n)
• There is a gap between cases 1 and 2 when f(n) is smaller than nlogba but not
polynomially smaller
• Similarly, there is a gap between cases 2 and 3 when f(n) is larger than nlogba ,
but not polynomially larger
• If the function f(n) falls into one of the above-mentioned gaps, or if
the regularity condition in case 3 fails to hold, we cannot apply the
Master method to solve the recurrence
• However, any other method may be applied for solving the same
Master Method: Examples
• Example 8: T(n) = 2T(n/2) + n log n
• Comparing above with T(n) = aT(n/b) + f(n) where a ≥ 1 and b > 1
a = 2, b = 2, f(n) = n log n, logba = log22 = 1
It appears that we are close to case 3
• Can we choose ε > 0, such that f(n) = n log n = Ω (n1+ε) ?
• NO, since the function nlogn is asymptotically larger, but not
polynomially larger, as f(n)/nlogba = nlogn/n = log n is asymptotically
less than nε for any ε > 0
• So, it falls between gap of case 1 and case 2, and hence, Master
Theorem does not apply for this recurrence relation
Master Theorem: A Limited Extension
• There is a limited 4-th condition of the Master Theorem that allows us
to consider polylogarithmic functions