Introduction to Design and Analysis of Algorithms

BACKTRACKING

Nguyễn Ngọc Thảo

[email protected]
ALLPPT.com _ Free PowerPoint Templates, Diagrams and Charts

ALLPPT.com _ Free PowerPoint Templates, Diagrams and Charts

This slide is adapted from the Lecture notes of the course
CSC14111 – Introduction to the Design and Analysis of
Algorithms taught by Dr. Nguyen Thanh Phuong (2023).

2
Introduction to
Backtracking
Introduction to Backtracking
• Backtracking is a more intelligent variation of the exhaustive
search technique.
• It can solve large instances of difficult combinatorial problems.
• However, we still face the same curse of exponential
explosion in the worst case, as in exhaustive search.

Principle idea
• Construct solutions one component at a time
• If no potential values of the remaining components can lead
to a solution, these components are not generated at all.
4
The state-space tree
• Backtracking builds a state-space tree with nodes reflecting
specific choices for solution components.
• It then terminates a node when no solution can be obtained
from its descendants. 2 ways to color A

B A A

A C B B B B

C C C C
D

The graph coloring problem D D D D

with two colors.
5
The state-space tree
• The root shows an initial state before the search for a
solution begins.
• The nodes of the first level in the tree represent the choices
made for the first component of a solution.
• The nodes of the second level represent the choices for the
second component, and so on.
• A node in a state-space tree is promising if it may lead to a
complete solution; otherwise, it is nonpromising.
• Leaves show either dead ends or complete solutions found.

6
 Example: The m-coloring problem

Given an undirected graph and a number m, the task is to color the given
graph with at most m colors such that no two adjacent vertices of the
graph are colored with the same color.

Image credit: GeeksforGeeks 7

Image credit: GeeksforGeeks 8
 Backtracking(u) {
if (promising(u))
Backtracking versions if (there is a solution at u)
Output the solution;
else
for (each child v of u)
Backtracking(v);
}

Backtracking(u) {
for (each child v of u)
if (promising(v))
if (there is a solution at v)
Output the solution;
else
Backtracking(v);
}
9
 Example: The n-queens problem

In this example, we consider the 4-queens variant, and for each step, put a
queen onto a single row.

Image credit: GeeksforGeeks 10

promising(i) {
j = 1; flag = true;
while (j < i && flag) {
if (col[i] == col[j] || abs(col[i] - col[j]) == i - j)
flag = false;
j++;
} n_Queens(i) {
return flag; if (promising(i))
} if (i == n)
print(col[1 .. n]);
else
for (j = 1; j ≤ n; j++) {
col[i + 1] = j;
n_Queens(i + 1);
}
}
n_Queens(0); 11
promising(i) {
j = 1; flag = true;
while (j < i && flag) {
if (col[i] == col[j] || abs(col[i] - col[j]) == i - j)
flag = false;
j++;
}
return flag; n_Queens(i) {
} for (j = 1; j ≤ n; j++) {
col[i] = j;
if (promising(i))
if (i == n)
print(col[1 .. n]);
else
n_Queens(i + 1);
}
}
n_Queens(1); 12
 Example: The Knight tour

In this example, a knight is placed on the first cell 𝑟0 , 𝑐0 of an empty

board of the size 𝑛 × 𝑛 and, and it must visit each cell exactly once.

13
KnightTour(i, r, c) {
for (k = 1; k ≤ 8; k++) {
u = r + row[k];
v = c + col[k];
if ((1  u,v  n) && (cb[u][v] == 0)) {
cb[u][v] = i;
if (i == n2)
print(h);
else 4 3
KnightTour(i + 1, u, v);
5 2
cb[u][v] = 0;
}
}
6 1
}
cb[r0][c0] = 1; 7 8
KnightTour(2, r0, c0);
14
 Example: The maze navigation problem

A robot starts at a certain position (the starting position) in the given

maze, and tries to reach another position (the goal position).
Positions in the maze will either be open or blocked with an obstacle. Of
course, the robot can only move to positions without obstacles and
must stay within the maze.
At any given moment, the robot can only move one step in one of four
directions: North, East, South, and West.
The robot should search for a path from the starting position to the goal
position (a solution path) until it finds one or until it exhausts all
possibilities. In addition, it should mark the path found (if any).

15
 Example: The maze problem

The original map A partial search that A solution path

leads to a dead end.

16
bool Find_Path(r, c) {
if ((r, c) ∉ Maze)
return false;
if (Maze[r][c] == ‘G’) return true;
if (Maze[r][c] == ‘’) return false;
if (Maze[r][c] == ‘#’) return false;
Maze[r][c] = ‘’;
if (Find_Path(r - 1, c) == true) return true;
if (Find_Path(r, c + 1) == true) return true;
if (Find_Path(r + 1, c) == true) return true;
if (Find_Path(r, c - 1) == true) return true;
Maze[r][c] = ‘.’;
return false;
}
Find_Path(r0, c0);

17
 Example: Hamiltonian Circuit Problem

bool promising(int pos, int v) {

if (pos == n && G[v][path[1]] == 0)
return false;
else
if (G[path[pos - 1]][v] == 0)
return false;
else
for (int i = 1; i < pos; i++)
if (path[i] == v)
return false;
return true;
}

18
Hamiltonian(bool G[1..n][1..n], int path[1..n], int pos) {
if (pos == n + 1)
print(path);
else
for (v = 1; v  n; v++)
if (promising(pos, v)) {
path[pos] = v;
Hamiltonian(G, path, pos + 1);
}
}
path[1 .. n] = -1;
path[1] = 1;
Hamiltonian(G, path, 2);

19
 Example: Sum of Subsets Problem

Find a subset of a given set 𝑊 = 𝑤1 . 𝑤2 , … , 𝑤𝑛 of 𝑛 positive integers

whose sum is equal to a given positive integer 𝑡.

2 3 7 1 4 5 sum = 7

{1, 2, 4} {2, 5} {3, 4} {7}

We can assume that the elements are sorted in increasing order.

𝑤1 < 𝑤2 < ⋯ < 𝑤𝑛 .

20
 The first approach

The solution 𝑆 is a vector of the size 𝑛: 𝑠1 , 𝑠2 , … , 𝑤𝑛 where 𝑠𝑖 ∈ {0,1}.

For each 𝑖 ∈ {1, 2, … , 𝑛}, the value of 𝑠𝑖 indicates whether 𝑤𝑖 is in the subset.

21
bool s[1 .. n] = {false};
total = σ𝒏𝒊=𝟏 𝒘[𝒊];
sort(w);
if (w[1]  t  total)
SoS(1, 0, total, w, s);
SoS(k, sum, total, w[1 .. n], s[1 .. n]) {
if (sum == t)
print(s);
else
if ((sum + total  t) && (sum + w[k]  t)) {
s[k] = true;
SoS(k + 1, sum + w[k], total - w[k], w, s);
s[k] = false;
SoS(k + 1, sum, total - w[k], w, s);
}
}
22
 The second approach

The solution 𝑆 is the set of selected items.

Assume that initially the given set has 4 items: W = 𝑤1 , 𝑤2 , … , 𝑤𝑛 .
The state-space tree will be constructed as shown.
23
SoS(s[1 .. n], size, sum, start) {
if (sum == t)
print(s, size);
else
for (i = start; i  n; i++) {
s[size] = w[i];
SoS(s, size + 1, sum + w[i], i + 1);
}
}
s[1 .. n] = {0};
total = σ𝒏𝒊=𝟏 𝒘[𝒊];
if (min(w)  t && t  total)
SoS(s, 1, 0, 1);

24
 SoS(s[1 .. n], size, sum, start, total) {
if (sum == t)
print(s, size);
else {
lost = 0;
for (i = start; i ≤ n; i++) {
if ((sum + total – lost ≥ t) && (sum + w[i] ≤ t)){
s[size] = w[i];
SoS(s, size+1, sum+w[i], i+1, total–lost -w[i]);
}
lost += w[i];
}
}
} The upgraded version
s[1 .. n] = {0};
total = σ𝒏𝒊=𝟏 𝒘[𝒊];
if (w[1]  t  total)
SoS(s, 1, 0, 1); 25
Backtracking vs. Branch and Bound
• Branch and bound breaks the problem into smaller sub-
problems, and then eliminating certain branches based on
bounds on the optimal solution.
• It is commonly use for optimization problems.
• E.g., traveling salesman problem, knapsack problem, etc.
• Branch and bound can use various strategies, not only DFS.

26
 Example: The Knapsack problem

The following figure demonstrates using Backtracking to solve the problem.

Image credit: GeeksforGeeks 27

 Example: The Knapsack problem

Branch and bound does the work better by ignoring nodes due to either
bounds or infeasibility.

Image credit: GeeksforGeeks 28

29
 Introduction to Design and Analysis of Algorithms

DIVIDE AND CONQUER

Nguyễn Ngọc Thảo

[email protected]
ALLPPT.com _ Free PowerPoint Templates, Diagrams and Charts

ALLPPT.com _ Free PowerPoint Templates, Diagrams and Charts

This slide is adapted from the Lecture notes of the course
CSC14111 – Introduction to the Design and Analysis of
Algorithms taught by Dr. Nguyen Thanh Phuong (2023).

2
Introduction to
Divide and Conquer
Divide and Conquer
• Divide and Conquer (DAC) is probably the best known
general algorithm design technique.

4
Divide and Conquer

Step 1. Divide the problem into several subproblems of the

same type, ideally of about equal size.
Step 2. Solve the subproblems, typically recursively.
We may use a different algorithm, especially when the subproblems
become small enough.
Step 3. Combine the solutions to the subproblems to get a
solution to the original problem.

5
DAC: The general recurrence
• The most typical case is to divide a problem instance of size
𝑛 into 𝒂 𝑎 > 1 instances of size 𝒏/𝒃 𝑏 > 1 .
• For simplicity, assume that size 𝑛 is a power of 𝑏.
• The general divide-and-conquer recurrence for running time
is
𝒏
𝑻 𝒏 =𝒂𝑻 + 𝒇(𝒏)
𝒃
• 𝑓(𝑛) is a function for the time spent on dividing an instance of size 𝑛
into those of size 𝑛/𝑏 and combining their solutions.

6
Master theorem
• Given the divide-and-conquer recurrence
𝑻 𝒏 = 𝒂 𝑻 𝒏/𝒃 + 𝒇(𝒏)
• If 𝑓(𝑛) ∈ Θ 𝑛𝑑 , where 𝑑 ≥ 0, then

𝚯 𝒏𝒅 𝒂 < 𝒃𝒅
𝑻 𝒏 ∈ 𝚯 𝒏𝒅 𝐥𝐨𝐠 𝒏 𝒂 = 𝒃𝒅
𝚯 𝒏𝐥𝐨𝐠𝒃 𝒂 𝒂 > 𝒃𝒅

• Analogous results hold for the Ο and Ω notations, too.

7
 Example: Find the maximum value from an array

Finding the maximum value from an array of 𝑛 numbers (for simplicity, 𝑛 = 2𝑘 )

findMax(a, l, r) {
if (l == r) return a[l];
m = (l + r) / 2;
return max(findMax(a, l, m), findMax(a, m + 1, r));
}

𝒏 𝒏>𝟏
The divide-and-conquer recurrence is: 𝑻 𝒏 = ቐ𝟐𝑻 + 𝚯(𝟏)
𝟐
𝟎 𝒏=𝟏

Thus, 𝑻(𝒏) ∈ 𝜣 𝒏 since 𝑇(𝑛) ∈ Θ 𝑛log𝑏 𝑎 with 𝑑 = 0, 𝑎 = 2, 𝑏 = 2.

8
 Example: Find both the maximum and minimum values

Finding simultaneously the maximum and minimum values from an array

of 𝑛 numbers (for simplicity, 𝑛 = 2𝑘 )

MinMax(a[1..n]) { MinMax(a[1..n]) {
min = max = a[1]; min = max = a[1];
for (i = 2; i ≤ n; i++) { for (i = 2; i ≤ n; i++)
if (a[i] > max) if (a[i] > max)
max = a[i]; max = a[i];
if (a[i] < min) else
min = a[i]; if (a[i] < min)
} min = a[i];
return <min, max> return <min, max>
} }
9
MinMax(l, r, & min, & max) {
if (l ≥ r - 1)
if (a[l] < a[r]) {
min = a[l];
max = a[r];
}
else {
min = a[r];
max = a[l];
}
else {
…
m = (l + r) / 2;
MinMax(l, m, minL, maxL);
MinMax(m + 1, r, minR, maxR);
min = (minL < minR) ? minL : minR;
max = (maxL < maxR) ? maxR : maxL;
}
} 10
 Example: Find both the maximum and minimum values

Finding simultaneously the maximum and minimum values from an array

of 𝑛 numbers (for simplicity, 𝑛 = 2𝑘 )

The general divide-and-conquer recurrence is:

𝒏 𝒏 𝒏>𝟐
𝑻 𝒏 = ቐ𝑻 𝟐
+𝑻
𝟐
+𝟐
𝟎 𝒏≤𝟐

Thus, 𝑻(𝒏) ∈ 𝚯 𝒏 since 𝑇(𝑛) ∈ Θ 𝑛log𝑏 𝑎 with 𝑑 = 0, 𝑎 = 2, 𝑏 = 2.

11
 Example: Merge sort

This approach sorts a given array 𝑎1 , 𝑎2 , … , 𝑎𝑛 by dividing it into two halves.

𝑎1 , 𝑎2 , … , 𝑎 𝑛 and 𝑎 𝑛 +1 , 𝑎 𝑛 +2 , … , 𝑎𝑛
2 2 2

sorting each of them recursively, and then merging the two smaller sorted
arrays into a single sorted one.

12
 merge(a[1 .. n], low, mid, high) {
i = low; j = mid + 1;
k = low;
while (i  mid) && (j  high)
if (a[i]  a[j]) buf[k++] = a[i++];
else buf[k++] = a[j++];
if (i > mid) buf[k .. high] = a[j .. high];
else buf[k .. high] = a[i .. mid];
a[low .. high] = buf[low .. high];
}
mergeSort(a[1 .. n], low, high) {
if (low < high) {
mid =  (low + high) / 2;
mergeSort(a, low, mid);
mergeSort(a, mid + 1, high);
merge(a, low, mid, high);
}
}
mergeSort(a, 1, n); 13
 Example: Merge sort

Assuming that the key comparison is the basic operation.

𝒏 𝒏 𝒏 𝑛>1
• Best case: 𝑻 𝒏 = ቐ𝑻 𝟐
+𝑻
𝟐
+
𝟐 ∈ 𝜣 𝒏 𝐥𝐨𝐠 𝒏
𝟎 𝑛≤1

𝒏 𝒏
𝑻 +𝑻 +𝒏−𝟏 𝑛 > 1
• Worst case: 𝑻 𝒏 = ቐ 𝟐 𝟐 ∈ 𝜣 𝒏 𝐥𝐨𝐠 𝒏
𝟎 𝑛≤1
Assuming that the assignment statement is the basic operation.
𝒏 𝒏 𝑛>1
𝑻 𝒏 = ቐ𝑻 𝟐
+𝑻
𝟐
+𝒏 ∈ 𝜣 𝒏 𝐥𝐨𝐠 𝒏
𝟎 𝑛≤1
14
 Example: Merge sort

This approach sorts a given array 𝑎1 , 𝑎2 , … , 𝑎𝑛 by partitioning it into two

subregions following the pivot 𝑎𝑠 , such that
𝑎1 , 𝑎2 , … , 𝑎𝑠−1 ≤ 𝑎𝑠 ≤ 𝑎𝑠+1 , … , 𝑎𝑛

≤ 𝑎𝑠 ≤
𝑎1 , 𝑎2 , … , 𝑎𝑠−1 𝑎𝑠+1 , … , 𝑎𝑛

15
 Partition(a[left .. right]) {
p = a[left];
i = left; j = right + 1;
do {
do i++; while (a[i] < p);
do j--; while (a[j] > p); Does this design work?
swap(a[i], a[j]);
} while (i < j);
swap(a[i], a[j]);
swap(a[left], a[j]);
return j;
} Quicksort(a[left .. right]) {
if (left < right){
s = Partition(a[left .. right]);
Quicksort(a[left .. s – 1]);
Quicksort(a[s + 1 .. right]);
}
} 16
 Example: Quick sort

For simplicity, assume that 𝑎1 , 𝑎2 , … , 𝑎𝑛 contains no duplicate values and

the size 𝑛 = 2𝑘 .
Two comparisons in loops are the basic operations.

• Best case: 𝑩(𝒏) ∈ 𝚯(𝒏 𝐥𝐨𝐠 𝒏)

• Worst case: 𝑾(𝒏) ∈ 𝚯(𝒏𝟐 )

• Average case: 𝑨(𝒏) ≈ 𝚯(𝟏, 𝟑𝟗 𝒏 𝐥𝐨𝐠 𝒏)

17
18
 1

Chapter 2: Fundamentals of the Analysis of Algorithm Efficiency

There are two kinds of efficiency: time efficiency and space efficiency. Time
efficiency, also called time complexity, indicates how fast an algorithm in question runs.
Space efficiency, also called space complexity, refers to the amount of memory units
required by the algorithm in addition to the space needed for its input and output.

Measuring an input’s size

Obviously, almost all algorithms run longer on larger inputs. Therefore, it is logical
to investigate an algorithm’s efficiency as a function of some parameter 𝑛 indicating the
algorithm’s input size.

Units for Measuring Running Time

We can simply use some standard unit of time measurement, such as second, or
millisecond, and so on to measure the running time of a program implementing the
algorithm. However, there are some drawbacks to such an approach.

One possible approach is to count the number of times the algorithm’s basic
operation is executed.
Note: The basic operation of an algorithm is the most important one. It contributes the most
to the total running time.

Let 𝑓(𝑛) be the polynomial that represents the number of times the algorithm’s basic
operation is executed on inputs of size 𝑛, and let 𝑡 be the execution time of the basic
operation on a particular computer. Then we can estimate the running time 𝑇(𝑛) of a
program implementing this algorithm on that computer by the formula

𝑇(𝑛) ≈ 𝑡 × 𝑓(𝑛)

The count 𝑓(𝑛) does not contain any information about operations that are not basic.
As a result, the count itself is often computed only approximately. Further, the constant 𝑡
is also an approximation whose reliability is not always easy to assess. However, the
formula can give a reasonable estimate of the algorithm’s running time.
 2

Orders/Rates of Growth

Given a polynomial 𝑓(𝑛) that represents the number of times the algorithm’s basic
operation is executed on inputs of size 𝑛. For large values of 𝑛, constants as well as all
terms except the one of largest degree will be eliminated.

Example: Consider two functions 0.1𝑛2 + 𝑛 + 100 and 0.1𝑛2.

𝒏 𝟎. 𝟏𝒏𝟐 𝟎. 𝟏𝒏𝟐 + 𝒏 + 𝟏𝟎𝟎
10 10 120
100 1000 1200
1000 100000 101100

Example: Given 𝑓(𝑛) = 12𝑛(𝑛 − 1). If the value of 𝑛 is large enough then

1 1
𝑓(𝑛) = 2𝑛2 − 2𝑛 ≈ 𝑛2

By the way, we may wonder how much longer will the algorithm run if we double
its input size? The answer is about four times longer. Why?

𝑇(2𝑛) 𝑓(2𝑛) (2𝑛)2

= = =4
𝑇(𝑛) 𝑓(𝑛) 𝑛2

The following table illustrates values (some approximate) of several standard

functions important for analysis of algorithms

𝑛 log 2 𝑛 𝑛 𝑛 log 2 𝑛 𝑛2 𝑛3 2𝑛 𝑛!
101 3.3 101 3.3×101 102 103 103 3.6×106
102 6.6 102 6.6×102 104 106 1.3×1030 9.3×10157
103 10 103 10×103 106 109
104 13 104 13×104 108 1012
105 17 105 17×105 1010 1015
106 20 106 20×106 1012 1018

The exponential function 2𝑛 and the factorial function 𝑛! grow so fast that their
values become astronomically large even for rather small values of 𝑛. Both of them are
often referred to as “exponential-growth functions” (or simply “exponential”).
PRINTED BY: Phuong Nguyen <[email protected]>. Printing is for personal, private use only. No part of this book may be reproduced or transmitted without
publisher's prior permission. Violators will be prosecuted.

“The greatest singer in the world cannot save a bad song!”

Example: Binary search

Example: Computing the 𝑛𝑡ℎ Fibonacci number in recursive manner.

Fibonacci(n) {
if (n ≤ 1)
return n;
return Fibonacci(n - 1) + Fibonacci(n - 2);
}

If this algorithm is programmed on a computer that makes one billion operations

per second then

𝑛 40 60 80 100 120 160 200

Time

Worst-Case, Best-Case, and Average-Case Efficiencies

For many algorithms the running time depends not only on an input size but also on
the specifics of a particular input.
 4

Example: Sequential search – the algorithm that searches for a given search key 𝑘
in a list of 𝑛 elements.
search(a[1 .. n], k) {
for (i = 1; i ≤ n; i++)
if (a[i] == k)
return 1;
return 0;
}

The best-case efficiency of an algorithm is its efficiency for the best-case input of
size 𝑛, which is an input of size 𝑛 for which the algorithm runs the fastest among all
possible inputs of that size.

The worst-case efficiency of an algorithm is its efficiency for the worst-case input
of size 𝑛, which is an input of size 𝑛 for which the algorithm runs the longest among all
possible inputs of that size.

The average-case efficiency of an algorithm indicates the algorithm’s behavior on

a “typical” or “random” input. To analyze the algorithm’s average-case efficiency, we must
make some assumptions about possible inputs of size 𝑛.

Asymptotic Notations

In the following discussion, 𝑓(𝑛) and 𝑔(𝑛) can be any nonnegative functions
defined on the set of natural numbers. In the context we are interested in, 𝑓(𝑛) will be an
algorithm’s running time, and 𝑔(𝑛) will be some simple function to compare with.
Informally:
– O(𝑔(𝑛)) is the set of all functions with a lower or same order of growth as 𝑔(𝑛).
– Ω(𝑔(𝑛)) is the set of all functions with a higher or same order of growth as 𝑔(𝑛).
– Θ(𝑔(𝑛)) is the set of all functions that have the same order of growth as 𝑔(𝑛).
 5

Big 𝑂 notation
O(𝑔(𝑛)) = {𝑓(𝑛): ∃𝑐 ∈ ℝ+ ∧ 𝑛0 ∈ ℕ, 0 ≤ 𝑓(𝑛) ≤ 𝑐𝑔(𝑛), ∀𝑛 ≥ 𝑛0 }

Explanation: A function 𝑓(𝑛) is said to be in O(𝑔(𝑛)) if 𝑓(𝑛) is bounded above by some

positive constant multiple of 𝑔(𝑛) for all large 𝑛
Example: If 𝑓(𝑛) = 2𝑛 + 1, 𝑔(𝑛) = 𝑛2 then 𝑓(𝑛) ∈ O(𝑛2 )

Big 𝛺 notation
Ω(𝑔(𝑛)) = {𝑓(𝑛): ∃𝑐 ∈ ℝ+ ∧ 𝑛0 ∈ ℕ, 0 ≤ 𝑐𝑔(𝑛) ≤ 𝑓(𝑛), ∀𝑛 ≥ 𝑛0 }

Explanation: A function 𝑓(𝑛) is said to be in Ω(𝑔(𝑛)) if 𝑓(𝑛) is bounded below by some

positive constant multiple of 𝑔(𝑛) for all large 𝑛
Example: If 𝑓(𝑛) = 𝑛3 + 2𝑛2 + 3, 𝑔(𝑛) = 𝑛2 then 𝑓(𝑛) ∈ Ω(𝑛2 ).

Big 𝛩 notation
Θ(𝑔(𝑛)) = {𝑓(𝑛): ∃𝑐1 , 𝑐2 ∈ ℝ+ ∧ 𝑛0 ∈ ℕ, 0 ≤ 𝑐1 𝑔(𝑛) ≤ 𝑓(𝑛) ≤ 𝑐2 𝑔(𝑛), ∀𝑛 ≥ 𝑛0 }

Explanation: A function 𝑓(𝑛) is said to be in Θ(𝑔(𝑛)) if 𝑓(𝑛) is bounded both above and
below by some positive constant multiples of 𝑔(𝑛) for all large 𝑛
1
Example: If 𝑓(𝑛) = 2 𝑛2 − 3𝑛, 𝑔(𝑛) = 𝑛2 then 𝑓(𝑛) ∈ Θ(𝑛2 )
 6

Some related theorems

Theorem 1: Given 𝑓(𝑛) ∈ ℝ+ and 𝑔(𝑛) ∈ ℝ+ :

𝑓(𝑛) ∈ Θ(𝑔(𝑛)) ⇔ 𝑓(𝑛) ∈ O(𝑔(𝑛)) ∧ 𝑓(𝑛) ∈ Ω(𝑔(𝑛))

Theorem 2: Given 𝑓(𝑛) = ∑𝑑𝑖=0 𝑎𝑖 𝑛𝑖 , 𝑎𝑑 > 0:

𝑓(𝑛) ∈ O(𝑛𝑑 )
where 𝑐 = ∑𝑑𝑖=0|𝑎𝑖 | , ∀𝑛 > 1.

Theorem 3: If 𝑓1 (𝑛) ∈ O(𝑔1 (𝑛)) and 𝑓2 (𝑛) ∈ O(𝑔2 (𝑛)):

𝑓1 (𝑛) + 𝑓2 (𝑛) ∈ O (𝑚𝑎𝑥(𝑔1 (𝑛), 𝑔2 (𝑛)))
(The analogous assertions are true for the Θ and Ω notations as well.)
 7

Mathematical Analysis of Nonrecursive Algorithms

General plan for analyzing the time efficiency of nonrecursive algorithms is as

follows:

1. Decide on a parameter (or parameters) indicating an input’s size.

2. Identify the algorithm’s basic operation.
3. Check whether the number of times the basic operation is executed depends only on
the size of an input. If it also depends on some additional property, the worst-case,
average-case, and best-case efficiencies have to be investigated separately.
4. Set up a sum expressing the number of times the algorithm’s basic operation is
executed.
5. Using standard formulas and rules of sum manipulation, either find a closed-form
formula for the count or, at the very least, establish its order of growth.

Example: Finding the value of the largest element in a list of 𝑛 numbers

MaxElement(a[1 .. n]) {
max = a[1];
for (i = 2; i ≤ n; i++)
if (a[i] > max)
max = a[i];
return max;
}

Example: Multiplying two matrices

MatrixMultiplication(a[1 .. n, 1 .. n], b[1 .. n, 1 .. n])

{
for (i = 1; i ≤ n; i++)
for (j = 1; j ≤ n; j++) {
c[i, j] = 0;
for (k = 1; k ≤ n; k++)
c[i, j] = c[i, j] + a[i, k] * b[k, j];
}
}
 8

Example: Bubble sort

BubbleSort(a[1 .. n]) {
for (i = 2; i < n; i++)
for (j = n; j  i; j--)
if (a[j - 1] > a[j])
a[j - 1]  a[j];
}

Let’s denote 𝐶(𝑛) the number of times the comparison is executed. Now, we find a
formula expressing it as a function of size 𝑛.
𝑛(𝑛 − 1)
𝐶(𝑛) = ∈ Θ(𝑛2 )
2

A minor improvement of Bubblesort

BubbleSort(a[1 .. n]) {
flag = true;
m = 1;
while (flag) {
flag = false;
m++;
for (j = n; j  m; j--)
if (a[j - 1] > a[j]) {
a[j - 1]  a[j];
flag = true;
}
}
}

The best-case efficiency: 𝐵(𝑛) = (𝑛 − 1)

The worst-case efficiency:

𝑛( 𝑛 − 1)
𝑊(𝑛) = ∈ Θ(𝑛2 )
2
 9

The average-case efficiency:

The standard assumption is that the probability the program stops after each iteration
1
of the while loop is 𝑛−1.
Let’s denote 𝐶(𝑖) the number of times the comparison is executed after the 𝑖𝑡ℎ
iteration of the while loop. Then, we can find the average number of comparisons 𝐴(𝑛) as
follows:
𝑛−1
1
𝐴(𝑛) = ∑ 𝐶(𝑖) ∈ Θ(𝑛2 )
𝑛−1
𝑖=1

Example: Insertion sort

Original version Insertion sort with sentinel

InsertionSort(a[1 .. n]) { InsWithSentinel(a[1 .. n]) {
for (i = 2; i  n; i++) { for (i = 2; i ≤ n; i++) {
v = a[i]; a[0] = v = a[i];
j = i – 1; j = i – 1;
while (j  1) && (a[j] while (a[j] > v) {
> v) { a[j + 1] = a[j];
a[j + 1] = a[j]; j--;
j--; }
} a[j + 1] = v;
a[j + 1] = v; }
} }
}

The best-case efficiency: 𝐵(𝑛) = 𝑛 − 1 ∈ Θ(𝑛)

The worst-case efficiency:

𝑛
𝑛 ( 𝑛 − 1)
𝑊(𝑛) = ∑(𝑖 − 1) = ∈ Θ(𝑛2 )
2
𝑖=2

The average-case efficiency: Let’s denote 𝐶(𝑖) the average number of times the
comparison is executed when the algorithm inserts the 𝑖𝑡ℎ element into the left sorted
subarray.
 10

𝑖−1
1 1
𝐶(𝑖) = × (𝑖 − 1) + ∑ × 𝑗
𝑖 𝑖
𝑗=1
The average number of comparisons 𝐴(𝑛) is as follows:
𝑛
𝑛2 − 𝑛 𝑛2
𝐴(𝑛) = ∑ 𝐶(𝑖) ≈ + 𝑛 − ln 𝑛 − 𝛾 ≈ ∈ Θ(𝑛2 )
4 4
𝑖=2
Hint:

𝑛
1 1 1 1
∑ = 1 + + + ⋯ + ≈ ln 𝑛 + 𝛾
𝑖 2 3 𝑛
𝑖=1
where 𝛾 = 0.5772 … is Euler constant.

Example: Finding the number of binary digits in the binary representation of a

positive decimal integer.

BitCount(n) {
count = 1;
while (n > 1) {
count++;
n = n / 2;
}
return count;
}

The exact formula for the number of times the comparison will be executed is
actually ⌊log 2 𝑛⌋ + 1 ∈ Θ(log 𝑛). This formula also indicates the number of bits in the
binary representation of 𝑛.
 11

Recurrence relations

Example: How many binary strings of length 𝑛 with no two adjacent 0’s.

Example: Rabbits and the Fibonacci Numbers (Fibonacci, 1202)

A young pair of rabbits (one of each sex) is placed on a desert island. A pair of
rabbits does not breed until they are 2 months old. After they are 2 months old, each pair
of rabbits produces another pair each month. Find a recurrence relation for the number of
pairs of rabbits on the island after 𝑛 months, assuming that no rabbits ever die.
Let’s denote 𝐹𝑛 the number of pairs of rabbits after 𝑛 months. Firstly, there is not
any pair of rabbits on this island so 𝐹0 = 0.

Reproducing pairs Young pairs Month Total pairs

 1 1
 2 1
  3 2
   4 3
     5 5
   
6 8
   

To find the number of pairs after 𝑛 months, add the number on the island the
previous month, 𝐹𝑛−1 , and the number of newborn pairs, which equals 𝐹𝑛−2 , because each
newborn pair comes from a pair at least 2 months old.
Consequently, the Fibonacci sequence is defined by the initial conditions 𝐹0 =
0, 𝐹1 = 1, and the recurrence relation:
𝐹𝑛 = 𝐹𝑛−1 + 𝐹𝑛−2 for 𝑛 = 2,3,4, …

Solving recurrence relations

We say that we have solved the recurrence relation together with the initial
conditions when we find an explicit formula, called a closed formula, for the terms of the
sequence.
 12

Example: Solve the recurrence relation 𝑥𝑛 = 2𝑥𝑛−1 + 1 with the initial condition
𝑥1 = 1.

First approach: Forward substitution

𝑥1 = 1 = 21 − 1
𝑥2 = 2𝑥1 + 1 = 3 = 22 − 1
𝑥3 = 2𝑥2 + 1 = 7 = 23 − 1
𝑥4 = 2𝑥3 + 1 = 15 = 24 − 1
…
𝑥𝑛 = 2𝑥𝑛−1 + 1 = 2𝑛 − 1

In this approach, we find successive terms beginning with the initial condition and
ending with 𝑥𝑛 .

Second approach: Backward substitution

𝑥𝑛 = 2𝑥𝑛−1 + 1 = 2(2𝑥𝑛−2 + 1) + 1 = 22 𝑥𝑛−2 + 21 + 20 = 22 (2𝑥𝑛−3 + 1) + 21 + 20
= 23 𝑥𝑛−3 + 22 + 21 + 20 = ⋯ = 2𝑖 𝑥𝑛−𝑖 + 2𝑖−1 + ⋯ + 22 + 21 + 20
Based on the initial condition 𝑥1 = 1, let’s set 𝑛 − 𝑖 = 1 and we have 𝑖 = 𝑛 − 1.
Therefore, we see that:
𝑥𝑛 = 2𝑖 𝑥𝑛−𝑖 + 2𝑖−1 + ⋯ + 22 + 21 + 20 = 2𝑛−1 𝑥𝑛−(𝑛−1) + 2𝑛−1−1 + ⋯ + 22 + 21 +
20 = 2𝑛−1 𝑥1 + 2𝑛−2 + ⋯ + 22 + 21 + 20 = 2𝑛 − 1

This approach is called backward substitution, because we began with 𝑥𝑛 and

iterated to express it in terms of falling terms of the sequence until we found it in terms of
𝑥1 – the initial condition.

Note: When we use forward/backward substitution, we essentialy guess a formula for the
terms of the sequence. We need to use mathematical induction to prove that our guess is
correct.

Solving Linear Recurrence Relations

A wide variety of recurrence relations occur in models. Some of these recurrence
relations can be solved using iteration (forward/backward substitution) or some other ad
hoc technique. However, one important class of recurrence relations can be explicitly
solved in a systematic way. These are recurrence relations that express the terms of a
sequence as linear combinations of previous terms.
 13

Definition: A linear homogeneous recurrence relation of degree 𝑘 with constant

coefficients is a recurrence relation of the form
𝑥𝑛 = 𝑐1 𝑥𝑛−1 + 𝑐2 𝑥𝑛−2 + ⋯ + 𝑐𝑘 𝑥𝑛−𝑘
or
𝑓(𝑛) = 𝑥𝑛 − 𝑐1 𝑥𝑛−1 − 𝑐2 𝑥𝑛−2 − ⋯ − 𝑐𝑘 𝑥𝑛−𝑘 = 0
where 𝑐1 , 𝑐2 , … , 𝑐𝑘 ∈ ℝ, 𝑐𝑘 ≠ 0 and the k initial conditions:
𝑥0 = 𝐶0 , 𝑥1 = 𝐶1 , … , 𝑥𝑘−1 = 𝐶𝑘−1

2
Note: The recurrence relation 𝑥𝑛 = 𝑥𝑛−1 + 𝑥𝑛−2 is not linear. The recurrence relation 𝑥𝑛 =
2𝑥𝑛−1 + 3 is not homogeneous. The recurrence relation 𝑥𝑛 = 𝑛𝑥𝑛−1 does not have
constant coefficients.

Solving linear homogeneous recurrence relations with constant coefficients

We can observe that 𝑥𝑛 = 𝑟 𝑛 is a solution of the recurrence relation:

𝑥𝑛 = 𝑐1 𝑥𝑛−1 + 𝑐2 𝑥𝑛−2 + ⋯ + 𝑐𝑘 𝑥𝑛−𝑘
if and only if
𝑟 𝑛 = 𝑐1 𝑟 𝑛−1 + 𝑐2 𝑟 𝑛−2 + ⋯ + 𝑐𝑘 𝑟 𝑛−𝑘
When both sides of this equation are divided by 𝑟 𝑛−𝑘 (when 𝑟 ≠ 0) and the right-
hand side is subtracted from the left, we obtain the equation
𝑟 𝑘 − 𝑐1 𝑟 𝑘−1 − 𝑐2 𝑟 𝑘−2 − ⋯ − 𝑐𝑘 𝑟 0 = 0
We call this the characteristic equation of the recurrence relation. The solutions of
this equation are called the characteristic roots of the recurrence relation. These
characteristic roots can be used to give an explicit formula for all the solutions of the
recurrence relation.

For simplicity, let’s consider linear homogeneous recurrence relations of degree

two. The characteristic equation in this case has the following form:
𝑎𝑟 2 + 𝑏𝑟 + 𝑐 = 0

Case 1: Suppose that the equation has two distinct roots 𝑟1 ∈ ℝ and 𝑟2 ∈ ℝ. Then the
solution is:
𝑥𝑛 = 𝛼𝑟1 𝑛 + 𝛽𝑟2 𝑛 , ∀𝛼, 𝛽 ∈ ℝ
Case 2: Suppose that the equation has only one root 𝑟 ∈ ℝ. Then the solution is:
𝑥𝑛 = 𝛼𝑟 𝑛 + 𝛽𝑛𝑟 𝑛 , ∀𝛼, 𝛽 ∈ ℝ
 14

Example: What is the solution of the recurrence relation

𝑥𝑛 = 𝑥𝑛−1 + 2𝑥𝑛−2
with initial conditions 𝑥0 = 2, 𝑥1 = 7.
Hint: The solution is 𝑥𝑛 = 3 × 2𝑛 − (−1)𝑛

Example: Find an explicit formula for the following recurrence relation

𝑥𝑛 = 6𝑥𝑛−1 − 9𝑥𝑛−2
with initial conditions 𝑥0 = 0, 𝑥1 = 3.
Hint: The solution is 𝑥𝑛 = 𝑛3𝑛
 15

Mathematical Analysis of Recursive Algorithms

General plan for analyzing the time efficiency of recursive algorithms is as follows:

1. Decide on a parameter (or parameters) indicating an input’s size.

2. Identify the algorithm’s basic operation.
3. Check whether the number of times the basic operation is executed can vary on
different inputs of the same size. If it can, the worst-case, average-case, and best-
case efficiencies have to be investigated separately.
4. Set up a recurrence relation, with an appropriate initial condition, for the number of
times the basic operation is executed.
5. Solve the recurrence or, at least, ascertain the order of growth of its solution.

Example: Finding the factorial of 𝑛: 𝑛!

Factorial(n) {
if (n == 0)
return 1;
return Factorial(n – 1) * n;
}

Let’s denote 𝑀(𝑛) the number of times the basic operation is executed. The
recurrence relation is as follows:
𝑀(𝑛) = 𝑀(𝑛 − 1) + 1
with the initial condition 𝑀(0) = 0.
Hint: 𝑀(𝑛) ∈ Θ(𝑛)

Example: Tower of Hanoi puzzle with 𝑛 disks

HNTower(n, left, middle, right) {

if (n) {
HNTower(n – 1, left, right, middle);
Movedisk(1, left, right);
HNTower(n – 1, middle, left, right);
}
}
Let’s denote 𝑀(𝑛) the number of moves. The recurrence relation is as follows:
 16

𝑀(𝑛) = 𝑀(𝑛 − 1) + 1 + 𝑀(𝑛 − 1) = 2𝑀(𝑛 − 1) + 1

with the initial condition 𝑀(1) = 1.
Hint: 𝑀(𝑛) = 2𝑛 − 1 ∈ Θ(2𝑛 )

Example: Finding the number of binary digits in the binary representation of a

positive decimal integer.

BitCount(n) {
if (n == 1) return 1;
return BitCount(n / 2) + 1;
}

Let’s denote 𝐴(𝑛) the number of times the basic operation is executed. Then, the
𝑛
number of additions made in computing BitCount(n / 2) is 𝐴 (⌊2 ⌋). The recurrence
relation is as follows:
𝑛
𝐴(𝑛) = 𝐴 (⌊2 ⌋) + 1
with the initial condition 𝐴(1) = 0

Definition:
Let 𝑔(𝑛) be a nonnegative function defined on the set of natural numbers. 𝑔(𝑛) is
called smooth if it is eventually nondecreasing and
𝑔(2𝑛) ∈ Θ(𝑔(𝑛))

Theorem “Smoothness rule”:

Let 𝑓(𝑛) be an eventually nondecreasing function and 𝑔(𝑛) be a smooth function.
If 𝑓(𝑛) ∈ Θ(𝑔(𝑛)) for values of 𝑛 that are powers of 𝑏 where 𝑏 ≥ 2, then
𝑓(𝑛) ∈ Θ(𝑔(𝑛))
(The analogous results hold for the cases of 𝑂 and Ω as well.)

The standard approach to solving such a recurrence is to solve it only for 𝑛 = 2𝑘

and then take advantage of the above theorem, which claims that the order of growth
observed for 𝑛 = 2𝑘 gives a correct answer about the order of growth for all values of 𝑛.
𝑛
Let’s assume that 𝑛 = 2𝑘 . The recurrence relation 𝐴(𝑛) = 𝐴 (⌊ ⌋) + 1 takes the
2
form:
𝐴(2𝑘 ) = 𝐴(2𝑘−1 ) + 1
 17

𝐴(20 ) = 0
Hint: 𝐴(𝑛) = log 2 𝑛 ∈ Θ(log 𝑛)

Example: Computing the 𝑛𝑡ℎ Fibonacci number

The recurrence relation is rewritten as follows:

𝐹𝑛 − 𝐹𝑛−1 − 𝐹𝑛−2 = 0
with two initial conditions 𝐹0 = 0, 𝐹1 = 1. Then, the characteristic equation of the
recurrence relation is:
𝑟2 − 𝑟 − 1 = 0
and
1 ± √(−1)2 − 4(1)(−1) 1 ± √5
𝑟1,2 = =
2 2
are two distinct roots of this equation.
Hence,
𝑛 𝑛
1 + √5 1 − √5
𝐹𝑛 = 𝛼 ( ) +𝛽( )
2 2

How to determine the values of 𝛼 and 𝛽? Based on two initial conditions, we may
construct a system of equations:
0 0
1 + √5 1 − √5
𝐹0 = 𝛼 ( ) +𝛽( ) =0
2 2
1 1
1 + √5 1 − √5
𝐹1 = 𝛼 ( ) +𝛽( ) =1
2 2
Solving this system of equations gives us 𝛼 = 1⁄√5 and 𝛽 = − 1⁄√5. Therefore,
𝑛 𝑛
1 1 + √5 1 1 − √5
𝐹𝑛 = ( ) − ( )
√5 2 √5 2
1+√5 1−√5
Let’s denote 𝜙 = 2 ≈ 1.61803, 𝜙̂ = 2 = − 1⁄𝜙 ≈ −0.61803:
1
𝐹𝑛 = (𝜙 𝑛 − 𝜙̂ 𝑛 )
√5
 18

Some algorithms for computing the 𝑛𝑡ℎ Fibonacci number

Recursive approach

Fibonacci(n) {
if (n ≤ 1)
return n;
return Fibonacci(n - 1) + Fibonacci(n - 2);
}

Let’s denote 𝐴(𝑛) the number of times the basic operation is executed to compute
𝐹𝑛 . We get the following recurrence equation for it:
𝐴(𝑛) = 𝐴(𝑛 − 1) + 𝐴(𝑛 − 2) + 1 với 𝑛 > 1
and two initial conditions 𝐴(0) = 0, 𝐴(1) = 0. After solving this recurrence equation, we
get:
1
𝐴(𝑛) = (𝜙 𝑛+1 − 𝜙̂ 𝑛+1 ) − 1 ∈ Θ(𝜙 𝑛 )
√5

Nonrecursive approach
1
It’s easy to construct a linear algorithm using the formula 𝐹𝑛 = (𝜙 𝑛 − 𝜙̂ 𝑛 ).
√5
1
Note: In practice we may use the formula 𝐹𝑛 = 𝑟𝑜𝑢𝑛𝑑 ( 𝜙 𝑛 ) when 𝑛 → ∞.
√5

Dynamic programming (Θ(𝑛))

Fibonacci(n) { Fibonacci(n) {
if (n ≤ 1) if (n ≤ 1)
return n; return n;
f0 = 0; f1 = 1; f0 = 0;
for (i = 2; i ≤ n; i++) { f1 = 1;
fn = f0 + f1; for (i = 2; i ≤ n; i++) {
f0 = f1; f1 = f1 + f0;
f1 = fn; f0 = f1 – f0;
} }
return fn; return f1;
} }
 19

Matrix approach
It’s easy to prove the correctness of the following equation using mathematical
induction:
𝐹(𝑛 + 1) 𝐹(𝑛) 1 1𝑛
[ ]=[ ] với 𝑛 ≥ 1
𝐹(𝑛) 𝐹(𝑛 − 1) 1 0
1 1𝑛
The question is how to efficiently compute [ ] ? The following formula is our
1 0
answer:
2
1 1 𝑛/2
([ ] ) 𝑛 𝑖𝑠 𝑒𝑣𝑒𝑛
1 1𝑛 1 0
[ ] = 2
1 0 1 1 ⌊𝑛/2⌋ 1 1
([ ] ) ×[ ] 𝑛 𝑖𝑠 𝑜𝑑𝑑
{ 1 0 1 0
Obviously, the running time of the “matrix” approach is Θ(log 𝑛).
Recursive version
int fib(int n) { void multiply(F[2][2],
F[2][2] = {{1, 1},{1, 0}}; T[2][2]) {
if (n == 0) return 0; t1 = F[0][0]*T[0][0] +
power(F, n - 1); F[0][1]*T[1][0];
return F[0][0]; t2 = F[0][0]*T[0][1] +
} F[0][1]*T[1][1];
t3 = F[1][0]*T[0][0] +
void power(int F[2][2], int F[1][1]*T[1][0];
n) { t4 = F[1][0]*T[0][1] +
if (n ≤ 1) F[1][1]*T[1][1];
return; F[0][0] = t1;
T[2][2] = {{1, 1},{1, 0}}; F[0][1] = t2;
power(F, n / 2); F[1][0] = t3;
multiply(F, F); F[1][1] = t4;
if (n % 2 != 0) }
multiply(F,T); void main() {
} cout << fib(5);
}

The weakness of the above code is recursive calls. Using a “loop” approach is
always better.
 20

While loop version

int Fibonacci(int n) {
i = 1; j = 0;
k = 0; h = 1;
while (n) {
if (n % 2) {
t = j * h;
j = i * h + j * k + t;
i = i * k + t;
}
t = h * h;
h = 2 * k * h + t;
k = k * k + t;
n = n / 2;
}
return j;
}
 1

Chapter 3: Brute Force and Exhaustive Search

Introduction

The subject of this chapter is brute force and its important special case, exhaustive
search. Brute force can be described as follows:
Brute force is a straightforward approach to solving a problem, usually directly
based on the problem statement and definitions of the concepts involved.

Example: Computing 𝑥 𝑛 .
By the definition of exponentiation,
𝑥𝑛 = ⏟ 𝑥 × …× 𝑥
𝑛 𝑡𝑖𝑚𝑒𝑠
𝑛
This suggests simply computing 𝑥 by multiplying 1 by 𝑥 𝑛 times.

The role of brute force

1. Brute-force is applicable to a very wide variety of problems.
2. A brute-force algorithm can still be useful for solving small-size instances of a
problem.
3. The expense of designing a more efficient algorithm may be unjustifiable if only a
few instances of a problem need to be solved and a brute-force algorithm can solve
those instances with acceptable speed.
4. A brute-force algorithm can serve an important theoretical or educational purpose
as a yardstick with which to judge more efficient alternatives for solving a problem.
5. A first application of the brute-force approach often results in an algorithm that can
be improved with a modest amount of effort.

Example: Bubble sort - Θ(𝑛2 )

Bubblesort(a[1 .. n]) {
for (i = 2; i  n; i++)
for (j = n; j  i; j--)
if (a[j - 1] > a[j])
a[j - 1]  a[j];
}
Note: We can improve the above algorithm by exploiting some observations and an
upgrade version of Bubble sort is Shake sort.
 2

Example: Brute force string matching

SequentialStringSearch(T[1 .. n], P[1 .. m]) {
count = 0;
for (i = 1; i ≤ n – m + 1; i++) {
j = 0;
while (j < m) && (P[1 + j] == T[i + j])
j++;
if (j == m)
count++;
}
return count;
}

The best-case efficiency: 𝐵(𝑛) =

The worst-case efficiency: 𝑊(𝑛) =
The average-case efficiency: 𝐴(𝑛) =

Find the subsequence with largest sum of elements in an array

Given an array of 𝑛 integers 𝑎1 , 𝑎2 , … , 𝑎𝑛 . The task is to find indices 𝑖 and 𝑗 with

𝑗
1 ≤ 𝑖 ≤ 𝑗 ≤ 𝑛, such that the sum ∑𝑘=𝑖 𝑎𝑘 is as large as possible. If the array contains all
non-positive numbers, then the largest sum is 0.

Brute force version with running time Θ(𝑛3 )

MaxContSubSum(a[1 .. n]) {
maxSum = 0;
for (i = 1; i ≤ n; i++)
for (j = i; j ≤ n; j++) {
curSum = 0;
for (k = i; k ≤ j; k++)
curSum += a[k];
if (curSum > maxSum)
maxSum = curSum;
}
return maxSum;
}
 3

Upgrade version with running time Θ(𝑛2 )

MaxContSubSum(a[1 .. n]) {
maxSum = 0;
for (i = 1; i ≤ n; i++) {
curSum = 0;
for (j = i; j ≤ n; j++) {
curSum += a[j];
if (curSum > maxSum)
maxSum = curSum;
}
}
return maxSum;
}

Dynamic programming version with running time Θ(𝑛)

MaxContSubSum(a[1 .. n]) {
maxSum = curSum = 0;
for (j = 1; j ≤ n; j++) {
curSum += a[j];
if (curSum > maxSum)
maxSum = curSum;
else
if (curSum < 0)
curSum = 0;
}
return maxSum;
}

The change-making problem

Given 𝑘 denominations: 𝑑1 < 𝑑2 < ⋯ < 𝑑𝑘 where 𝑑1 = 1. Find the minimum

number of coins (of certain denominations) that add up to a given amount of money 𝑛.
 4

Example: Suppose that there are 4 denominations: 𝑑1 = 1, 𝑑2 = 5, 𝑑3 = 10, 𝑑4 =

25 and the amount of money 𝑛 = 72. The minimum number of coins is 6 including two
pennies, two dimes, and two quarters.

Idea: Find all 𝑘-tuples 〈𝑐1 , 𝑐2 , … , 𝑐𝑘 〉 such that:

𝑐1 × 𝑑1 + 𝑐2 × 𝑑2 + ⋯ + 𝑐𝑘 × 𝑑𝑘 = 𝑛
where 𝑐𝑖 is the number of coins of denomination 𝑑𝑖 . Among these 𝑘-tuples we choose the
one with the smallest sum ∑𝑘𝑖=1 𝑐𝑖 .

The efficiency of this algorithm is as follows:

𝑛 𝑛 𝑛 𝑛𝑘
𝑇(𝑛) = × ×…× = ∈ Θ(𝑛𝑘 )
𝑑1 𝑑2 𝑑𝑘 𝑑1 × 𝑑2 × … × 𝑑𝑘
because 𝑑1 , 𝑑2 , … , 𝑑𝑘 are constants.

Closest-Pair Problem

This problem calls for finding the two closest points in a set of 𝑛 points in the plan.

Algorithm with running time Θ(𝑛2 )

BruteForceClosestPoints(P[1 .. n]) {
dmin = ;
for (i = 1; i ≤ n - 1; i++)
for (j = i + 1; j ≤ n; j++) {
d = √(𝑃[𝑖]. 𝑥 − 𝑃[𝑗]. 𝑥)2 + (𝑃[𝑖]. 𝑦 − 𝑃[𝑗]. 𝑦)2 ;
if (d < dmin) {
dmin = d;
point1 = P[i];
point2 = P[j];
}
}
return <point1, point2>;
}
 5

Convex-hull problem

Intuitively, the convex-hull of a set of 𝑛 points in the plane is the smallest convex
polygon that contains all of them either inside or on its boundary.
Note: Mathematicians call the vertices of such a polygon “extreme points.” Finding these
points is the way to construct the convex hull for a given set of 𝑛 points.

A brute force approach is based on the following observation:

“A line segment 𝑝𝑞̅̅̅ connecting two points 𝑝 and 𝑞 of a set of 𝑛 points is a part of
the convex hull’s boundary if and only if all the other points of the set lie on the same side
of the straight line through these two points.”

It’s known that the straight line through two points (𝑥1 , 𝑦1 ) and (𝑥2 , 𝑦2 ) in the
coordinate plane can be defined by the equation
𝑎𝑥 + 𝑏𝑦 = 𝑐
where 𝑎 = 𝑦2 − 𝑦1 , 𝑏 = 𝑥1 − 𝑥2 , 𝑐 = 𝑥1 𝑦2 − 𝑥2 𝑦1 .
Second, such a line divides the plane into two half-planes: for all the points in one
of them, 𝑎𝑥 + 𝑏𝑦 > 𝑐, while for all the points in the other, 𝑎𝑥 + 𝑏𝑦 < 𝑐. For the points on
the line itself, 𝑎𝑥 + 𝑏𝑦 = 𝑐.

Brute force algorithm with running time Θ(𝑛3 )

for (each point pi  S: i = 1 → n - 1)
for (each point pj  S: j = i + 1 → n) {
Construct the line 𝑝̅̅̅̅̅;
𝑖 𝑝𝑗
if (all the other points in S lie on the same side of
𝑝
̅̅̅̅̅)
𝑖 𝑝𝑗
Store the pair of two points <pi, qj>;
}
 6

Now, let’s consider another brute force approach whose output is a list of the
extreme points in a counterclockwise order.

Initially, the point with the lowest y-value must be the first extreme point in the
result list. If there are two points with the same lowest y-value then the leftmost one is the
first extreme point.

Assume that 𝑝 is the current extreme point, how to find out the next one?

Let’s denote 𝑐𝑢𝑟𝐴𝑛𝑔𝑙𝑒 the current angle (initially, 𝑐𝑢𝑟𝐴𝑛𝑔𝑙𝑒 = 0). The next
extreme point 𝑞 must satisfy the following condition:

̂
𝑐𝑢𝑟𝐴𝑛𝑔𝑙𝑒 ≤ (𝑝𝑞
̅̅̅, ̂
0𝑥) < (𝑝𝑟
̅̅̅, 0𝑥), ∀𝑟 ∈ 𝑆: 𝑟 ≠ 𝑝, 𝑟 ≠ 𝑞

Note: The value of 𝑐𝑢𝑟𝐴𝑛𝑔𝑙𝑒 is increased from 0 to 2𝜋.

 

  

Algorithm

computeAngle(point from, point to) {

angle = atan2(to.y - from.y, to.x - from.x);
if (angle < 0)
angle += 2 * ;
return angle;
}
 7

findNextExtremePoint(S, cur, curAngle) {

minAngle = 2 * ;
S \= cur;
for (each point p in S) {
angle = computeAngle(cur, p);
if (angle < minAngle && angle ≥ curAngle) {
next = p;
minAngle = angle;
}
}
S = cur;
return [next, minAngle];
}

computeConvexHull(S) {
convexHull = ;
// Let “first” be the first extreme point
convexHull = first;
curAngle = 0;
point cur = first;
while (true) {
[next, curAngle] = findNextExtremePoint(S, cur,
curAngle);
if (first == next)
break;
convexHull = next;
cur = next;
}
return convexHull;
}

What is the efficiency of this algorithm? In the worst case, the running time is
2 ).
Θ(𝑛 Otherwise?
 8

Exhaustive Search

Exhaustive search is simply a brute-force approach to combinatorial problems.

Algorithm for finding all subsets of a set of size 𝑛

for (k = 0; k < 2n; k++)
Output the binary string of length n representing k;

Algorithm for generating all permutations of a set

Permutation(pivot, a[1 .. n]) {
if (pivot == n)
Output(a);
else
for (i = pivot; i  n; i++) {
a[pivot]  a[i];
Permutation(pivot + 1, a);
a[pivot]  a[i];
}
}
Permutation(1, a);

Example: Let’s consider all permutations of four elements 𝑎1 , 𝑎2 , 𝑎3 , 𝑎4

𝑎1 , 𝑎2 , 𝑎3 , 𝑎4 𝑎2 , 𝑎1 , 𝑎3 , 𝑎4 𝑎3 , 𝑎2 , 𝑎1 , 𝑎4 𝑎4 , 𝑎2 , 𝑎3 , 𝑎1
𝑎1 , 𝑎2 , 𝑎4 , 𝑎3 𝑎2 , 𝑎1 , 𝑎4 , 𝑎3 𝑎3 , 𝑎2 , 𝑎4 , 𝑎1 𝑎4 , 𝑎2 , 𝑎1 , 𝑎3
𝑎1 , 𝑎3 , 𝑎2 , 𝑎4 𝑎2 , 𝑎3 , 𝑎1 , 𝑎4 𝑎3 , 𝑎1 , 𝑎2 , 𝑎4 𝑎4 , 𝑎3 , 𝑎2 , 𝑎1
𝑎1 , 𝑎3 , 𝑎4 , 𝑎2 𝑎2 , 𝑎3 , 𝑎4 , 𝑎1 𝑎3 , 𝑎1 , 𝑎4 , 𝑎2 𝑎4 , 𝑎3 , 𝑎1 , 𝑎2
𝑎1 , 𝑎4 , 𝑎3 , 𝑎2 𝑎2 , 𝑎4 , 𝑎3 , 𝑎1 𝑎3 , 𝑎4 , 𝑎1 , 𝑎2 𝑎4 , 𝑎1 , 𝑎3 , 𝑎2
𝑎1 , 𝑎4 , 𝑎2 , 𝑎3 𝑎2 , 𝑎4 , 𝑎1 , 𝑎3 𝑎3 , 𝑎4 , 𝑎2 , 𝑎1 𝑎4 , 𝑎1 , 𝑎2 , 𝑎3

Let’s denote 𝑇(𝑛) the number of times the basic operation must be executed to
generate all permutations of a set of 𝑛 elements. The recurrence relation is as follows:
𝑇(𝑛) = 𝑛𝑇(𝑛 − 1) + Θ(𝑛)
with the initial condition 𝑇(1) = 0.
Hint: 𝑇(𝑛) ∈ Ω(𝑛!)
 9

Traveling Salesman Problem

The problem asks to find the shortest tour through a given set of 𝑛 cities that visits
each city exactly once before returning to the city where it started.

Note: The problem can also be stated as the problem of finding the shortest Hamiltonian
circuit of a weighted graph.

Hamiltonian circuit: Given a graph of 𝑛 vertices. A Hamiltonian circuit is defined

as a sequence of 𝑛 + 1 vertices: 𝑣𝑖0 , 𝑣𝑖1 , …, 𝑣𝑖𝑛−1 , 𝑣𝑖𝑛 such that:
1. 𝑣𝑖0 ≡ 𝑣𝑖𝑛
2. ∀𝑗 ∈ [0, 𝑛 − 1]: 𝑣𝑖𝑗 is adjacent to 𝑣𝑖𝑗+1
3. ∀𝑗, 𝑘 ∈ [0, 𝑛 − 1]: 𝑣𝑖𝑗 ≠ 𝑣𝑖𝑘 iff 𝑗 ≠ 𝑘

In words, a Hamiltonian circuit is defined as a cycle that passes through all the
vertices of the graph exactly once.

Idea: With no loss of generality, we can assume that all circuits start and end at one
particular vertex 𝑣𝑖0 . Thus, we can get (𝑛 − 1)! potential tours by generating all the
permutations of 𝑛 − 1 intermediate cities and attach 𝑣𝑖0 to the beginning and the end of
each permutation. Then, we verify if each potential tours is a Hamiltonian circuit? If it’s
true then we compute the tour lengths, and find the shortest among them.

The time efficiency of this algoritm is Θ(𝑛!).

Sum of subsets problem

The sum of subsets problem consists of finding all subsets of a given set A =
{𝑎1 , 𝑎2 , … , 𝑎𝑛 } of 𝑛 distinct positive integers that sum to a positive integer 𝑘.

Example: If A = {3,5,6,7,8,9,10} and 𝑘 = 15, there will be more than one subset
whose sum is 15. The subsets are {3,5,7}, {7,8}, {6,9}, …

A brute force approach is to generate all subsets of the given set A and compute the
sum of each subset.
The time efficiency of this algoritm is Θ(2𝑛 ).
 10

Knapsach problem

Given 𝑛 items of known weights 𝑤1 , 𝑤2 , … , 𝑤𝑛 and values 𝑣1 , 𝑣2 , … , 𝑣𝑛 and a

knapsack of capacity 𝐶, find the most valuable subset of the items that fit into the knapsack.

The knapsack problem can also be formally stated as follows:

𝑛

𝑚𝑎𝑥𝑖𝑚𝑖𝑧𝑒 ∑ 𝑣𝑖 𝑥𝑖
𝑖=1
𝑛

𝑠𝑢𝑏𝑗𝑒𝑐𝑡 𝑡𝑜 ∑ 𝑤𝑖 𝑥𝑖 ≤ 𝐶 𝑤ℎ𝑒𝑟𝑒 𝑥𝑖 ∈ {0, 1}, 𝑖 = 1, . . , 𝑛

𝑖=1

A simplistic approach to solving this problem would be to enumerate all subsets of

the 𝑛 items, and select the one that satisfies the constraints and maximizes the profits.
The obvious problem with this strategy is the running time which is at least Θ(2𝑛 )
corresponding to the power set of 𝑛 items.

Assignment problem

There are 𝑛 people who need to be assigned to execute 𝑛 jobs, one person per job.
The cost if the 𝑖 𝑡ℎ person is assigned to the 𝑗 𝑡ℎ job is a known quantity 𝐶𝑖,𝑗 for each pair
𝑖, 𝑗 = 1,2, … , 𝑛. The problem is to find an assignment with the minimum total cost.

Example:
𝐶 Job 1 Job 2 Job 3 Job 4
Person 1 9 2 7 8
Person 2 6 4 3 7
Person 3 5 8 1 8
Person 4 7 6 9 4

We can describe feasible solutions to the assignment problem as 𝑛-tuples

〈𝑗1 , 𝑗2 , … , 𝑗𝑛 〉: ∀𝑖 ∈ [1, 𝑛], 𝑗𝑖 ∈ [1, 𝑛] in which the 𝑖𝑡ℎ component (𝑗𝑖 ) indicates the column
of the element selected in the 𝑖𝑡ℎ row.

Example: 〈2,4,1,3〉 indicates the assignment of person 1 to job 2, person 2 to job 4,

person 3 to job 1, and person 4 to job 3.
 11

The exhaustive-search approach to this problem would require generating all the
permutations of integers 1,2, … , 𝑛, computing the total cost of each assignment by
summing up the corresponding elements of the cost matrix, and finally selecting the one
with the smallest sum.

Of course, the running time of this approach is Θ(𝑛!).

 1

Chapter 4: Backtracking

Introduction

Backtracking is a more intelligent variation of the exhaustive-search technique. This

approach makes it possible to solve some large instances of difficult combinatorial
problems, though, in the worst case, we still face the same curse of exponential explosion
encountered in exhaustive search.
The principal idea of backtracking is to construct solutions one component at a time
and if no potential values of the remaining components can lead to a solution, the remaining
components are not generated at all.

The state-space tree

Backtracking is based on the construction of a state-space tree whose nodes reflect

specific choices made for a solution’s components.
The root of this tree represents an initial state before the search for a solution begins.
The nodes of the first level in the tree represent the choices made for the first component
of a solution, the nodes of the second level represent the choices for the second component,
and so on. A node in a state-space tree is said to be promising if it corresponds to a partially
constructed solution that may still lead to a complete solution; otherwise, it is called
nonpromising. Leaves represent either nonpromising dead ends or complete solutions
found by the algorithm.
Backtracking technique terminates a node as soon as it can be guaranteed that no
solution to the problem can be obtained by considering choices that correspond to the
node’s descendants.

Backtracking algorithm – the first version

Backtracking(u) {
if (promising(u))
if (there is a solution at u)
Output the solution;
else
for (each child v of u)
Backtracking(v);
}
 2

Backtracking algorithm – the second version

Backtracking(u) {
for (each child v of u)
if (promising(v))
if (there is a solution at v)
Output the solution;
else
Backtracking(v);
}

The 𝒏-Queens problem

The 𝑛-Queens is the problem of placing 𝑛 chess queens on an 𝑛 × 𝑛 chessboard so

that no two queens attack each other.

(a) (h)
1,1 1,2

(b) (d) (i)

2,3 2,4 2,4

(e) (j)
3,2 3,1
(c) (g)

(k)
4,3
(f)
 3

Algorithm
promising(i) {
j = 1; flag = true;
while (j < i && flag) {
if (col[i] == col[j] || abs(col[i] - col[j]) == i - j)
flag = false;
j++;
}
return flag;
}

Version 1
n_Queens(i) {
if (promising(i))
if (i == n)
print(col[1 .. n]);
else
for (j = 1; j ≤ n; j++) {
col[i + 1] = j;
n_Queens(i + 1);
}
}
n_Queens(0);

Version 2
n_Queens(i) {
for (j = 1; j ≤ n; j++) {
col[i] = j;
if (promising(i))
if (i == n)
print(col[1 .. n]);
else
n_Queens(i + 1);
}
}
n_Queens(1);
 4

The Knight’s tour problem

A knight is placed on the first cell 〈𝑟0 , 𝑐0 〉 of an empty board of the size 𝑛 × 𝑛 and,
moving according to the rules of chess, must visit each cell exactly once.

Note: Numbers in cells indicate move number of knight.

 4  3  -2
5 2 -1
  0
6 1 1
 7 0  2
-2 -1 0 1 2

Algorithm
KnightTour(i, r, c) {
for (k = 1; k ≤ 8; k++) {
u = r + row[k];
v = c + col[k];

if ((1  u, v  n) && (cb[u][v] == 0)) {

cb[u][v] = i;

if (i == n2)
print(h);
else
KnightTour(i + 1, u, v);

cb[u][v] = 0;
}
}
}

cb[r0][c0] = 1;
KnightTour(2, r0, c0);
 5

Maze problem

A robot is asked to navigate a maze. It is placed at a certain position (the starting

position) in the maze and is asked to try to reach another position (the goal position).
Positions in the maze will either be open or blocked with an obstacle. Of course, the robot
can only move to positions without obstacles and must stay within the maze.
At any given moment, the robot can only move 1 step in one of 4 directions: North,
East, South, and West.
The robot should search for a path from the starting position to the goal position
(a solution path) until it finds one or until it exhausts all possibilities. In addition, it should
mark the path it finds (if any) in the maze.

# S # # . # North


# . . . . #
. # . # . #
. . . . # # West East
. . # # . G
# . . . . #
# # # . # # South
(a)
# S # # . # # S # # . #
#     # #   . . #
. # . # . # . #  # . #
. . . . # # .   . # #
. . # # . G .  # #  G
# . . . . # #     #
# # # . # # # # # . # #
(b) (c)

1 2 3 4 5 6
#### #### #### #### #### ####
##..# ##..# ##..# ##..# ##..# ##..#
##..# ##..# ##..# ##..# ##..# ##..#
##.# ##.# ##.# ##.# ##.# #.#.#
###... ###... ###... ###... ###... ###...
G...## G...## G...## G...## G...## G...##
 6

Algorithm
bool Find_Path(r, c) {
if ((r, c)  Maze)
return false;
if (Maze[r][c] == ‘G’)
return true;
if (Maze[r][c] == ‘’)
return false;
if (Maze[r][c] == ‘#’)
return false;

Maze[r][c] = ‘’;

if (Find_Path(r - 1, c) == true)
return true;
if (Find_Path(r, c + 1) == true)
return true;
if (Find_Path(r + 1, c) == true)
return true;
if (Find_Path(r, c - 1) == true)
return true;

Maze[r][c] = ‘.’;
return false;
}
Find_Path(r0, c0);
 7

Hamiltonian Circuit Problem

Algorithm
bool promising(int pos, int v) {
if (pos == n && G[v][path[1]] == 0) // (3)
return false;
else
if (G[path[pos - 1]][v] == 0) // (2)
return false;
else
for (int i = 1; i < pos; i++) // (4)
if (path[i] == v)
return false;
return true;
}
Hamiltonian(bool G[1..n][1..n], int path[1..n], int pos) {
if (pos == n + 1)
print(path);
else
for (v = 1; v  n; v++)
if (promising(pos, v)) {
path[pos] = v;
Hamiltonian(G, path, pos + 1);
}
}
path[1 .. n] = -1;
path[1] = 1;
Hamiltonian(G, path, 2);
 8

Sum of Subsets Problem

Find a subset of a given set W = {𝑤1 , 𝑤2 , … , 𝑤𝑛 } of 𝑛 positive integers whose sum

is equal to a given positive integer 𝑡.

Note: It is convenient to sort the set’s elements in increasing order. So, we will assume that
𝑤1 < 𝑤2 < ⋯ < 𝑤𝑛

The first approach

The solution 𝑆 is a vector of the size 𝑛: {𝑠1 , 𝑠2 , … , 𝑠𝑛 } where 𝑠𝑖 ∈ {0,1}. For each
𝑖 ∈ {1,2, … , 𝑛}, the value of 𝑠𝑖 indicates whether 𝑤𝑖 is in the subset or not.

Example: W = {3,5,6,7} và 𝑡 = 15.

Level 0 0
+3 +0

Level 1 3 0

+5 +0 +5 +0

Level 2 8 3 5 0

+6 +0 +6 +0 +6 +0 

Level 3 14 8 9 3 11 5
 +7
   
+0

Level 4 15 8
✓ 

Algorithm
bool s[1 .. n] = {false};
total = ∑𝑛𝑖=1 𝑤[𝑖];
sort(w);
if (w[1]  t  total)
SoS(1, 0, total, w, s);
…
 9

SoS(k, sum, total, w[1 .. n], s[1 .. n]) {

if (sum == t)
print(s);
else
if ((sum + total  t) && (sum + w[k]  t)) {
s[k] = true;
SoS(k + 1, sum + w[k], total - w[k], w, s);
s[k] = false;
SoS(k + 1, sum, total - w[k], w, s);
}
}

The second approach

The solution 𝑆 is the set of selected items.

Assume that initially the given set has 4 items W = {𝑤1 , 𝑤2 , 𝑤3 , 𝑤4 }. The state-space
tree will be constructed as follows:

0
+ w1 + w4
+ w2 + w3

1 2 3 4
+ w2 + w4 + w3 + w4 + w4
+ w3

5 6 7 8 9 10

+ w3 + w4 + w4 + w4

11 12 13 14

+ w4

15
 10

Algorithm
SoS(s[1 .. n], size, sum, start) {
if (sum == t)
print(s, size);
else
for (i = start; i  n; i++) {
s[size] = w[i];
SoS(s, size + 1, sum + w[i], i + 1);
}
}
s[1 .. n] = {0};
total = ∑𝑛𝑖=1 𝑤[𝑖];
if (min(w)  t && t  total)
SoS(s, 1, 0, 1);

Algorithm (upgraded version)

SoS(s[1 .. n], size, sum, start, total) {
if (sum == t)
print(s, size);
else {
lost = 0;
for (i = start; i  n; i++) {
if ((sum + total – lost ≥ t) && (sum + w[i] ≤ t)) {
s[size] = w[i];
SoS(s, size+1, sum+w[i], i+1, total–lost - w[i]);
}
lost += w[i];
}
}
}
s[1 .. n] = {0};
total = ∑𝑛𝑖=1 𝑤[𝑖];
sort(w);
if (w[1]  t  total)
SoS(s, 1, 0, 1, total);
 1

Chapter 5: Divide-and-Conquer

Introduction

Divide-and-conquer is probably the best known general algorithm design technique.

Divide-and-conquer algorithms work according to the following general plan:
Step 1. A problem is divided into several subproblems of the same type, ideally of
about equal size.
Step 2. The subproblems are solved (typically recursively, though sometimes a
different algorithm is employed, especially when subproblems become small
enough).
Step 3. The solutions to the subproblems are combined to get a solution to the original
problem.

Example: Finding the maximum value from an array of 𝑛 numbers (for simplicity,
𝑛 is a power of 2).

The general divide-and-conquer recurrence

In the most typical case of divide-and-conquer, a problem’s instance of size 𝑛 is

divided into 𝑎(> 1) instances of size 𝑛⁄𝑏 where 𝑏 > 1. For simplicity, assuming that size
𝑛 is a power of 𝑏; we get the following recurrence for the running time 𝑇(𝑛):

𝑇(𝑛) = 𝑎𝑇(𝑛⁄𝑏) + 𝑓(𝑛)

where 𝑓(𝑛) is a function that accounts for the time spent on dividing an instance of size 𝑛
into instances of size 𝑛⁄𝑏 and combining their solutions. This recurrence is called the
general divide-and-conquer recurrence.

The efficiency analysis of many divide-and-conquer algorithms is greatly simplified

by the following theorem:
Master theorem: Given the divide-and-conquer recurrence 𝑇(𝑛) = 𝑎𝑇(𝑛⁄𝑏) + 𝑓(𝑛). If
𝑓(𝑛) ∈ Θ(𝑛𝑑 ) where 𝑑 ≥ 0 then:
Θ(𝑛𝑑 ) 𝑎 < 𝑏𝑑
𝑇(𝑛) ∈ {Θ(𝑛𝑑 log 𝑛) 𝑎 = 𝑏 𝑑
Θ(𝑛log𝑏 𝑎 ) 𝑎 > 𝑏 𝑑
Analogous results hold for the Ο and  notations, too.
 2

Example: Finding the maximum value from an array of 𝑛 numbers (for simplicity,
𝑘
𝑛 = 2 ).
findMax(a, l, r) {
if (l == r) return a[l];
m = (l + r) / 2;
return max(findMax(a, l, m), findMax(a, m + 1, r));
}
The divide-and-conquer recurrence is as follows:
𝑛
2𝑇 ( ) + Θ(1) 𝑛 > 1
𝑇(𝑛) = { 2
0 𝑛=1

Example: Finding simultaneously the maximum and minimum values from an array
of 𝑛 numbers.

Algorithm
MinMax(l, r, & min, & max) {
if (l ≥ r - 1)
if (a[l] < a[r]) {
min = a[l];
max = a[r];
}
else {
min = a[r];
max = a[l];
}
else {
m = (l + r) / 2;
MinMax(l, m, minL, maxL);
MinMax(m + 1, r, minR, maxR);
min = (minL < minR) ? minL : minR;
max = (maxL < maxR) ? maxR : maxL;
}
}

The divide-and-conquer recurrence is as follows:

𝑛 𝑛
𝐶 (⌊ ⌋) + 𝐶 (𝑛 − ⌊ ⌋) + 2 𝑛>2
𝐶(𝑛) = { 2 2
1 𝑛≤2
 3

Mergesort

This approach sorts a given array 𝑎1 , 𝑎2 , … , 𝑎𝑛 by dividing it into two halves:

𝑎1 , 𝑎2 , … , 𝑎⌊𝑛⌋
2
𝑎⌊𝑛⌋+1 , 𝑎⌊𝑛⌋+2 , … , 𝑎𝑛
2 2
sorting each of them recursively, and then merging the two smaller sorted arrays into a
single sorted one.

32749168

3274 9168
Tách

32 74 91 68

3 2 7 4 9 1 6 8

23 47 19 68
Trộn

2347 1689

12346789

Algorithm
mergeSort(a[1 .. n], low, high) {
if (low < high) {
mid = (low + high) / 2;
mergeSort(a, low, mid);
mergeSort(a, mid + 1, high);
merge(a, low, mid, high);
}
}
 4

merge(a[1 .. n], low, mid, high) {

i = low;
j = mid + 1;
k = low;
while (i  mid) && (j  high)
if (a[i]  a[j])
buf[k ++] = a[i ++];
else
buf[k ++] = a[j ++];

if (i > mid)
buf[k .. high] = a[j .. high];
else
buf[k .. high] = a[i .. mid];

a[low .. high] = buf[low .. high];

}
mergeSort(a, 1, n);

How efficient is mergesort?

• Assuming that the key comparison is the basic operation:

In the best case:
𝑛 𝑛 𝑛
𝑇 (⌊ ⌋) + 𝑇 (⌈ ⌉) + ⌊ ⌋ 𝑛>1
𝑇(𝑛) = { 2 2 2
0 𝑛=1
Hint: 𝑇(𝑛) ∈ Θ(𝑛 log 𝑛)

In the worst case:

𝑛 𝑛
𝑇 (𝑛 ) = {
𝑇 (⌊ ⌋) + 𝑇 (⌈ ⌉) + (𝑛 − 1) 𝑛 > 1
2 2
0 𝑛=1
Hint: 𝑇(𝑛) ∈ Θ(𝑛 log 𝑛)
• Assuming that the assignment statement is the basic operation:
𝑛 𝑛
𝑀 (⌊ ⌋) + 𝑀 (⌈ ⌉) + 𝑛 𝑛 > 1
𝑀(𝑛) = { 2 2
0 𝑛=1
Hint: 𝑀(𝑛) ∈ Θ(𝑛 log 𝑛)
 5

Quicksort

Unlike mergesort, which divides its input elements according to their position in the
array, quicksort divides them according to their value. This process is called partition.
A partition is an arrangement of the array’s elements so that all the elements to the
left of some element 𝑎𝑠 are less than or equal to 𝑎𝑠 , and all the elements to the right of 𝑎𝑠
are greater than or equal to it:
{𝑎1 … 𝑎𝑠−1 } ≤ 𝑎𝑠 ≤ {𝑎𝑠+1 … 𝑎𝑛 }
After a partition is achieved, 𝑎𝑠 will be in its final position in the sorted array, and
we can continue sorting the two subarrays to the left and to the right of 𝑎𝑠 independently
by the same method.

Algorithm
Quicksort(a[left .. right]) {
if (left < right){
s = Partition(a[left .. right]);
Quicksort(a[left .. s – 1]);
Quicksort(a[s + 1 .. right]);
}
}
Partition(a[left .. right]) {
p = a[left];
i = left;
j = right + 1;
do {
do i++; while (a[i] < p);
do j--; while (a[j] > p);
swap(a[i], a[j]);
} while (i < j);

swap(a[i], a[j]);
swap(a[left], a[j]);

return j;
}

Does this design work?

 6

Analysis of Quicksort

For simplicity, assuming that the sequence 𝑎1 , 𝑎2 , … , 𝑎𝑛 contains no duplicate

values and the size 𝑛 is a power of 2: 𝑛 = 2𝑘 . Two comparisons in loops are the basic
operation.

In the best case:

𝐶𝑏 (𝑛) ∈ Θ(𝑛 log 𝑛)

In the worst case:

𝐶𝑤 (𝑛) ∈ Θ(𝑛2 )

In the average case:

𝐶(𝑛) ≈ 1.39𝑛 log 2 𝑛
 7

Multiplication of Large Integers

A simple quadratic-time algorithm for multiplying large integers is one that mimics
the standard way learned in school. We will develop one that is better than quadratic time.

The basic idea: Observing the multiplication of two complex numbers

(𝑎 + 𝑏𝑖)(𝑐 + 𝑑𝑖) = (𝑎𝑐 − 𝑏𝑑) + (𝑏𝑐 + 𝑎𝑑)𝑖
K. F. Gauss perceived that:
𝑏𝑐 + 𝑎𝑑 = (𝑎 + 𝑏)(𝑐 + 𝑑) − (𝑎𝑐 + 𝑏𝑑)

We assume that the data type large_integer representing a large integer was
constructed. It is not difficult to write linear-time algorithms for three operations:
mul 10m, div 10m, and mod 10m.
Let’s consider the algorithm that implements the multiplication of two large
integers: 𝑢 × 𝑣
Algorithm
large_integer MUL(large_integer u, v) {
large_integer x, y, w, z;

n = max(number of digits in u, number of digits in v);

if (u == 0 || v == 0)
return 0;
else
if (n  )
return u × v; // built-in operator
else {
m = n / 2;
x = u div 10m; y = u mod 10m;
w = v div 10m; z = v mod 10m;
return MUL(x, w) mul 102m +
(MUL(x, z) + MUL(y, w)) mul 10m +
MUL(y, z);
}
}
 8

Analysis of the algorithm

The divide-and-conquer recurrence is as follows:
𝑛
4𝑇 ( ) + Θ(𝑛) 𝑛 > 𝛼
𝑇(𝑛) = { 2
1 𝑛≤𝛼
The Master theorem implies that 𝑇(𝑛) ∈ Θ(𝑛2 ).

Algorithm (upgraded version)

large_integer MUL(large_integer u, v, n) {
n = max(number of digits in u, number of digits in v);

if (u == 0 || v == 0)
return 0;
else
if (n  )
return u × v;
else {
m = n / 2;
x = u div 10m; y = u mod 10m;
w = v div 10m; z = v mod 10m;
r = MUL(x + y, w + z);
p = MUL(x, w);
q = MUL(y, z);

return p mul 102m + (r – p – q) mul 10m + q;

}
}
In this case, the divide-and-conquer recurrence is as follows:
𝑛
3𝑇 ( ) + Θ(𝑛) 𝑛 > 𝛼
𝑇(𝑛) = { 2
1 𝑛≤𝛼

Since 𝑑 = 1, 𝑎 = 3, 𝑏 = 2, the Master theorem implies that 𝑇(𝑛) ∈ Θ(𝑛log2 3 ) ≈

Θ(𝑛1.585 ).
 9

Extension: Multiplication of two positive integers of 𝑛 bits. Assuming that 𝑛 is the power
of 2.

Let’s 𝑥 and 𝑦 be two positive integers of 𝑛 bits. Obviously:

𝑥 = 2𝑛/2 𝑥𝐿 + 𝑥𝑅
𝑦 = 2𝑛/2 𝑦𝐿 + 𝑦𝑅
where 𝑥𝐿 , 𝑥𝑅 are two positive integers represented by 𝑛⁄2 leftmost bits and 𝑛⁄2 rightmost
bits of 𝑥 , respectively; similarly, 𝑦𝐿 , 𝑦𝑅 are two positive integers represented by 𝑛⁄2
leftmost bits and 𝑛⁄2 rightmost bits of 𝑦, respectively.

Example: Given 𝑥 = 13510 = 100001112. Then,

𝑥𝐿 = 810 (= 10002 )
𝑥𝑅 = 710 (= 01112 )
𝑥 = 2𝑛/2 𝑥𝐿 + 𝑥𝑅 = 28/2 × 810 + 710

Now, we get:
𝑥 × 𝑦 = (2𝑛/2 𝑥𝐿 + 𝑥𝑅 ) × (2𝑛/2 𝑦𝐿 + 𝑦𝑅 ) = 2𝑛 × 𝑥𝐿 𝑦𝐿 + 2𝑛/2 × (𝑥𝐿 𝑦𝑅 + 𝑥𝑅 𝑦𝐿 ) + 𝑥𝑅 𝑦𝑅
Algorithm
int multiply(x, y) {
n = max(|x| , |y| );
bit bit

if (n  ) return x × y;
xL = n / 2 leftmost bits of x;
xR = n / 2 rightmost bits of x;
yL = n / 2 leftmost bits of y;
yR = n / 2 rightmost bits of y;

r = multiply(xL + xR, yL + yR);

p = multiply(xL, yL);
q = multiply(xR, yR);

return p × 2n + (r - p - q) × 2n/2 + q;
}
 10

Strassen’s Matrix Multiplication

Suppose we want the product 𝐶 of two 2 × 2 matrices, 𝐴 and 𝐵. That is,

𝑐11 𝑐12 𝑎11 𝑎12 𝑏11 𝑏12
[𝑐
21 𝑐22 ] = [𝑎21 𝑎22 ] × [𝑏21 𝑏22 ]
𝑎 × 𝑏11 + 𝑎12 × 𝑏21 𝑎11 × 𝑏12 + 𝑎12 × 𝑏22
= [ 11 ]
𝑎21 × 𝑏11 + 𝑎22 × 𝑏21 𝑎21 × 𝑏12 + 𝑎22 × 𝑏22
Of course, the time complexity of this straightforward method is 𝑇(𝑛) = 𝑛3, where
𝑛 is the number of rows and columns in the matrices. To be specific, the above matrix
multiplication requires eight multiplications and four additions.
However, Strassen determined that if we let
𝑚1 = (𝑎11 + 𝑎22 ) × (𝑏11 + 𝑏22 )
𝑚2 = (𝑎21 + 𝑎22 ) × 𝑏11
𝑚3 = 𝑎11 × (𝑏12 − 𝑏22 )
𝑚4 = 𝑎22 × (𝑏21 − 𝑏11 )
𝑚5 = (𝑎11 + 𝑎12 ) × 𝑏22
𝑚6 = (𝑎21 − 𝑎11 ) × (𝑏11 + 𝑏12 )
𝑚7 = (𝑎12 − 𝑎22 ) × (𝑏21 + 𝑏22 )
the product 𝐶 is given by
𝑐11 𝑐12 𝑚1 + 𝑚4 − 𝑚5 + 𝑚7 𝑚3 + 𝑚5
[𝑐
21 𝑐22 ] = [ 𝑚2 + 𝑚4 𝑚1 + 𝑚3 − 𝑚2 + 𝑚6 ]
Strassen’s method requires seven multiplications and 18 additions/subtractions.
Thus, we have saved ourselves one multiplication at the expense of doing 14 additional
additions or subtractions.
Let 𝐴 and 𝐵 be matrices of size 𝑛 × 𝑛, where 𝑛 = 2𝑘 . Let 𝐶 be the product of 𝐴 and
𝐵. Each of these matrices is divided into four submatrices as follows:
𝐶 𝐶12 𝐴 𝐴12 𝐵 𝐵12
[ 11 ] = [ 11 ] × [ 11 ]
𝐶21 𝐶22 𝐴21 𝐴22 𝐵21 𝐵22
where
𝑐1,1 ⋯ 𝑐1,𝑛 𝑐1,𝑛+1 ⋯ 𝑐1,𝑛
2 2
𝐶11 =[ ⋮ ⋱ ⋮ ] 𝐶12 = [ ⋮ ⋱ ⋮ ]
𝑐𝑛,1 ⋯ 𝑐𝑛,𝑛 𝑐𝑛,𝑛+1 ⋯ 𝑐𝑛,𝑛
2 22 22 2
𝑐𝑛+1,1 ⋯ 𝑐𝑛+1,𝑛 𝑐𝑛+1,𝑛+1 ⋯ 𝑐𝑛+1,𝑛
2 2 2 2 2 2
𝐶21 =[ ⋮ ⋱ ⋮ ] 𝐶22 = [ ⋮ ⋱ ⋮ ]
𝑐𝑛,1 ⋯ 𝑐𝑛,𝑛 𝑐𝑛,𝑛+1 ⋯ 𝑐𝑛,𝑛
2 2
Using Strassen’s method, first we compute:
 11

𝑀1 = (𝐴11 + 𝐴22 ) × (𝐵11 + 𝐵22 )

where our operations are now matrix addition and multiplication. In the same way, we
compute 𝑀2 through 𝑀7 . Next we compute
𝐶11 = 𝑀1 + 𝑀4 − 𝑀5 + 𝑀7
and 𝐶12 , 𝐶21 , 𝐶22 . Finally, the product 𝐶 of 𝐴 and 𝐵 is obtained by combining the four
submatrices 𝐶𝑖𝑗 .
Algorithm
Strassen(n, A[1..n][1..n], B[1..n][1..n], C[1..n][1..n]) {
if (n  )
C = A × B;
else {
"Partition A into 4 submatrices A11, A12, A21, A22";
"Partition B into 4 submatrices B11, B12, B21, B22";

Strassen(n/2, A11 + A22, B11 + B22, M1);

…
Strassen(n/2, A12 – A22, B21 + B22, M7);

C11 = M1 + M4 – M5 + M7;
C12 = M3 + M5;
C21 = M2 + M4;
C22 = M1 + M3 – M2 + M6;

Combine C11, C12, C21, C22 into C;

}
}

Analysis of the algorithm

The divide-and-conquer recurrence is as follows:
𝑛 𝑛 2
𝑇(𝑛) = {7𝑇 ( ) + 18 ( ) 𝑛>𝛼
2 2
1 𝑛≤𝛼
The Master theorem implies that: 𝑇(𝑛) ∈ Θ(𝑛log2 7 ) ≈ Θ(𝑛2.81 )
 12

Find the substring with largest sum of elements in an array

 
Algorithm

sumMax(a[1..n], l, r) {
if (l == r) return max(a[l], 0);

c = (l + r) / 2;
maxLS = sumMax(a, l, c);
maxRS = sumMax(a, c + 1, r);

tmp = maxLpartS = 0;
for (i = c; i  l; i--) {
tmp += a[i];
if (tmp > maxLpartS) maxLpartS = tmp;
}
tmp = maxRpartS = 0;
for (i = c + 1; i  r; i++) {
tmp += a[i];
if (tmp > maxRpartS) maxRpartS = tmp;
}
tmp = maxLpartS + maxRpartS;
return max(tmp, maxLS, maxRS);
}
max = sumMax(a, 1, n);

Analysis of the algorithm

The divide-and-conquer recurrence is as follows:
𝑛 𝑛
𝑇 (⌊ ⌋) + 𝑇 (⌈ ⌉) + Θ(𝑛) 𝑛 > 1
𝑇(𝑛) = { 2 2
0 𝑛=1
Hint: 𝑇(𝑛) ∈ Θ(𝑛 log 𝑛)
 13

Closest-Pair Problem
Let 𝑃 be a list of 𝑛 > 1 points in the Cartesian plane: 𝑃 = {𝑝1 , 𝑝2 , … , 𝑝𝑛 }. Find
a pair of points with the smallest distance between them.
For the sake of simplicity and without loss of generality, we can assume that the
points in 𝑃 are ordered in nondecreasing order of their 𝑥 coordinate. In addition, let 𝑄 be a
list of all and only points in 𝑃 sorted in nondecreasing order of the 𝑦 coordinate.
If 2 ≤ 𝑛 ≤ 3, the problem can be solved by the obvious brute-force algorithm.
Besides, 𝑛 = 2,3 is also the stopping condition of the recursive process.
𝑛 𝑛
If 𝑛 > 3, we can divide the points into two subsets 𝑃𝐿 and 𝑃𝑅 of ⌈2 ⌉ and ⌊2 ⌋ points,
respectively, by drawing a vertical line through the median of their 𝑥 coordinates so that
𝑛 𝑛
⌈2 ⌉ points lie to the left of or on the line itself, and ⌊2 ⌋ points lie to the right of or on the
line . Then we can solve the closest-pair problem recursively for subsets 𝑃𝐿 and 𝑃𝑅 . Let
𝛿𝐿 and 𝛿𝑅 be the smallest distances between pairs of points in 𝑃𝐿 and 𝑃𝑅 , respectively, and
let 𝛿 = min{𝛿𝐿 , 𝛿𝑅 }.

S 

R

L

 

Note that 𝛿 is not necessarily the smallest distance between all the point pairs
because points of a closer pair can lie on the opposite sides of the separating line .
Therefore, we need to examine such points. Obviously, we can limit our attention to the
points inside the symmetric vertical strip of width 2𝛿 around the separating line , since
the distance between any other pair of points is at least 𝛿.
Algorithm
 14

ClosestPair(Point P[1..n], Point Q[1..n]) {

if (|P|  3)
return the minimal distance found by the brute-force
algorithm;

= P[n/2].x;

Copy the first n/2 points of P to PL;

Copy the same n/2 points from Q to QL;
Copy the remaining n/2 points of P to PR;
Copy the same n/2 points from Q to QR;

L = ClosestPair(PL, QL);
R = ClosestPair(PR, QR);
 = min(L, R);

Copy all the points p of Q for which |p.x - | <  into

S[1..k];
min = ;
for (i = 1; i < k; i++) {
j = i + 1;
while (j  k) && (|S[i].y – S[j].y| < min) {
min = min(√(𝑆[𝑖]. 𝑥 − 𝑆[𝑗]. 𝑥)2 + (𝑆[𝑖]. 𝑦 − 𝑆[𝑗]. 𝑦)2, min)
j++;
}
}

return min;
}

Analysis of the algorithm

The divide-and-conquer recurrence is as follows:
𝑛
𝑇(𝑛) = 2𝑇 ( ) + Θ(𝑛) ∈ Θ(𝑛 log 𝑛)
2
 15

The change-making problem

Given 𝑘 denominations: 𝑑1 < 𝑑2 < ⋯ < 𝑑𝑘 where 𝑑1 = 1 . Find the minimum

number of coins (of certain denominations) that add up to a given amount of money 𝑛.

Algorithm
moneyChange(d[1..k], money) {
for (i = 1; i  k; i++)
if (d[i] == money)
return 1;

minCoins = money;
for (i = 1; i  money / 2; i++) {
tmpSum = moneyChange(d, i) + moneyChange(d, money - i);
if (tmpSum < minCoins)
minCoins = tmpSum;
}
return minCoins;
}

Analysis of the algorithm

The divide-and-conquer recurrence is as follows:
⌊𝑛⁄2⌋
𝑛
∑ (𝑇(𝑖) + 𝑇(𝑛 − 𝑖)) + Θ ( ) 𝑛 > 2
𝑇(𝑛) = 2
𝑖=1
1 𝑛=2
{ 0 𝑛=1
Hint: 𝑇(𝑛) ∈ Ω(2𝑛 )
 16

Algorithm (upgraded version)

moneyChange(d[1..k], money) {
for (i = 1; i  k; i++)
if (d[i] == money)
return 1;

minCoins = money;
for (i = 1; i  k; i++)
if (money > d[i]) {
tmpSum = 1 + moneyChange(d, money - d[i]);
if (tmpSum < minCoins)
minCoins = tmpSum;
}
return minCoins;
}
 Introduction to Design and Analysis of Algorithms

Algorithm Efficiency

Nguyễn Ngọc Thảo

[email protected]
ALLPPT.com _ Free PowerPoint Templates, Diagrams and Charts

ALLPPT.com _ Free PowerPoint Templates, Diagrams and Charts

This slide is adapted from the Lecture notes of the course
CSC14111 – Introduction to the Design and Analysis of
Algorithms taught by Dr. Nguyen Thanh Phuong (2023).

2
The analysis of
Algorithm efficiency
Algorithm efficiencies
• There are two kinds of algorithm efficiency (or complexity):
time efficiency and space efficiency.

Memory Running
space time

• Time complexity indicates how fast an algorithm runs.

• Space efficiency refers to the amount of memory required
by the algorithm.
• (in addition to the space needed for its input and output)

4
Measure the algorithm efficiencies
• Let 𝒜 be an algorithm and 𝒫 be the program implementing 𝒜.
• The time complexity of algorithm 𝒜 can be measured based on
the running time of program 𝒫.
• We simply use some standard time measurement units (e.g. seconds).
• Similarly, the space complexity of algorithm 𝒜 corresponds to the
amount of memory occupied by that program 𝒫.
• The common space measurement units are from a hierarchy of bytes
(B), e.g., kilobytes (KB), megabytes (MB), or gigabytes (GB).

• The above metrics are device-dependent, introducing biases to

the comparisons of algorithms.

5
Measure the time complexity
• The time efficiency of an algorithm is measured by counting
the number of times the basic operation(s) is executed.

Basic operation
The basic operation of an algorithm is the most important
one, contributing the most to the total running time.
• It is usually in the algorithm’s innermost loop.
• It usually relates to the data that needs to be processed.

6
Measure the time complexity
• The efficiency of an algorithm is represented as a function of
some parameter 𝑛 indicating the input size.
• Given the following terms.
• 𝒇(𝒏): the polynomial that represents the number of times the basic
operation is executed on inputs of size 𝑛.
• 𝑡: the execution time of the basic operation on a particular computer.
• Then, the running time 𝑇(𝑛) of a program implementing the
algorithm in consideration on that computer is
𝑻(𝒏) ≈ 𝒕 × 𝒇(𝒏)

7
The running time 𝑇(𝑛): Comments
• The formula 𝑻(𝒏) can only give a reasonable estimate of the
algorithm’s running time.
• The count 𝑓(𝑛) says nothing about non-basic operations.
• The reliability of the constant 𝑡 is not always easy to assess.

8
The orders of growth
• Consider a polynomial 𝑓(𝑛) that represents the number of
times the basic operation is executed on inputs of size 𝑛.
• For large values of 𝑛, constants as well as all terms except
the one of largest degree will be eliminated.

9
 Example: Why ignore low-order terms?

Assume that there are two different algorithms which are designed to
solve the same problem.
Let 𝑓1 𝑛 = 0.1 × 𝑛2 + 10 × 𝑛 + 100 and 𝑓2 𝑛 = 0.1 × 𝑛2 be the number
of times the basic operation is executed by the first and second algorithm,
respectively.

𝒏 𝒇𝟏 (𝒏) 𝒇𝟐 (𝒏) 𝒇𝟏 (𝒏) / 𝒇𝟐 (𝒏)

101 210 10 21

102 2,100 1,000 2.1

103 110,100 100,000 1.101

104 100,010,000,100 100,000,000,000 1.000100001

10
 Example: What happen when doubling the input size?

1
Consider the function 𝑓 𝑛 = 𝑛(𝑛 − 1).
2
1 1
If the value of 𝑛 is large enough then 𝑓 𝑛 = 𝑛2 − 𝑛 ≈ 𝑛2 .
2 2
How much longer will the algorithm run if we double its input size?
𝑇(2𝑛) 𝑓(2𝑛) (2𝑛)2
= = =4
𝑇(𝑛) 𝑓(𝑛) 𝑛2

11
Common growth-rate functions

12
Common growth-rate functions

The exponential function 2𝑛 and the factorial function 𝑛! grow so fast that their
values become astronomically large even for rather small values of 𝑛.

13
Example: Compute the running time for Fibonacci problem

Consider the following pseudo-code for computing the 𝑛𝑡ℎ Fibonacci

number in recursive manner.
Fibonacci(n) {
if (n ≤ 1)
return n;
return Fibonacci(n - 1) + Fibonacci(n - 2);
}

Assume that this algorithm is programmed on a computer that makes one

billion operations per second.
What is the running time of the above pseudo-code for each input size?
𝑛 40 60 80 100 120 160 200
Time 14
Worst-case, Best-case,
and Average-case
Algorithm efficiency: Analysis cases
• The running time of a problem may depend on the specifics
of a particular input, beside the input size.

SequentialSearch(a[1 .. n], k) {
for (i = 1; i ≤ n; i++)
Whether the search procedure
if (a[i] == k)
stops depends on the position
of the key 𝑘. return 1;
return 0;
}

16
Algorithm efficiency: Analysis cases
• The best-case efficiency of an algorithm is the algorithm’s
efficiency for the best-case input of size 𝑛.
• The algorithm runs the fastest among all possible inputs of that size.
• The worst-case efficiency of an algorithm is the algorithm’s
efficiency for the worst-case input of size 𝑛.
• The algorithm runs the longest among all possible inputs of that size.
• The average-case efficiency of an algorithm indicates the
algorithm’s behavior on a “typical” or “random” input.
• We must make some assumptions about possible inputs of size 𝑛.

17
Asymptotic notations
• Let 𝑓(𝑛) and 𝑔(𝑛) can be any nonnegative functions defined
on the set of natural numbers.
• 𝑓(𝑛) will be an algorithm’s running time, and 𝑔(𝑛) will be
some simple function to compare with.

18
Asymptotic notations: Big-O
• A function 𝑓(𝑛) is said to be in 𝑂(𝑔(𝑛)) if it is bounded above
by some positive constant multiple of 𝑔(𝑛) for all large 𝑛.
𝑂(𝑔(𝑛)) = {𝑓(𝑛): ∃𝑐 ∈ ℝ+ ∧ 𝑛0 ∈ ℕ, 0 ≤ 𝑓(𝑛) ≤ 𝑐𝑔(𝑛), ∀𝑛 ≥ 𝑛0 }

• Example: If 𝑓 𝑛 = 2𝑛 + 1, 𝑔(𝑛) = 𝑛2 then 𝑓(𝑛) ∈ 𝑂(𝑛2 ).

• 𝑂(𝑔(𝑛)) is the set of all functions with a lower or same order

of growth as 𝑔(𝑛).

19
Asymptotic notations: Big- 
• A function 𝑓(𝑛) is said to be in Ω(𝑔(𝑛)) if it is bounded below
by some positive constant multiple of 𝑔(𝑛) for all large 𝑛.
Ω(𝑔(𝑛)) = {𝑓(𝑛): ∃𝑐 ∈ ℝ+ ∧ 𝑛0 ∈ ℕ, 0 ≤ 𝑐𝑔(𝑛) ≤ 𝑓(𝑛), ∀𝑛 ≥ 𝑛0 }

• Example: 𝑓 𝑛 = 𝑛3 + 2𝑛2 + 3, 𝑔(𝑛) = 𝑛2 → 𝑓(𝑛) ∈ Ω(𝑛2 ).

• Ω(𝑔(𝑛)) is the set of all functions with a higher or same

order of growth as 𝑔(𝑛).

20
Asymptotic notations: Big-
• A function 𝑓(𝑛) is said to be in Θ(𝑔(𝑛)) if it is bounded below
by some positive constant multiple of 𝑔(𝑛) for all large 𝑛.
Θ(𝑔(𝑛)) = {𝑓(𝑛): ∃𝑐1 , 𝑐2 ∈ ℝ+ ∧ 𝑛0 ∈ ℕ,
0 ≤ 𝑐1 𝑔(𝑛) ≤ 𝑓(𝑛) ≤ 𝑐2 𝑔(𝑛), ∀𝑛 ≥ 𝑛0 }

1 2
• Example: 𝑓 𝑛 = 𝑛 − 3𝑛, 𝑔(𝑛) = 𝑛2 → 𝑓(𝑛) ∈ Θ(𝑛2 ).
2

• Θ(𝑔(𝑛)) is the set of all functions with the same order of

growth as 𝑔(𝑛).

21
Big-O notation Big- notation Big- notation

3 log 𝑛 + 8 4𝑛2 4𝑛3 + 3𝑛2

𝑶 𝒏𝟐 5𝑛 + 7 6𝑛2 + 9 6𝑛6 + 4𝑛4 𝛀 𝒏𝟐
2 log 𝑛 5𝑛2 + 2𝑛 2𝑛 + 4𝑛

𝚯 𝒏𝟐
22
Big-O and related theorems
• Theorem 1: Given 𝑓(𝑛) ∈ ℝ+ and 𝑔 𝑛 ∈ ℝ+ .
𝑓(𝑛) ∈ Θ(𝑔(𝑛)) ⇔ 𝑓(𝑛) ∈ 𝑂(𝑔(𝑛)) ∧ 𝑓(𝑛) ∈ Ω(𝑔(𝑛))

• Theorem 2: Given 𝑓(𝑛) = σ𝑑𝑖=0 𝑎𝑖 𝑛𝑖 , 𝑎𝑑 > 0.

𝑓(𝑛) ∈ 𝑂(𝑛𝑑 )
where 𝑐 = σ𝑑𝑖=0 𝑎𝑖 , ∀𝑛 > 1

• Theorem 3: If 𝑓1 𝑛 ∈ 𝑂 𝑔1 𝑛 and 𝑓2 𝑛 ∈ 𝑂 𝑔2 𝑛 then

𝑓1 (𝑛) + 𝑓2 (𝑛) ∈ 𝑂 max(𝑔1 𝑛 , 𝑔2 𝑛

• The analogous assertions are also true for Θ and Ω notations.

23
Mathematical analysis of
Nonrecursive Algorithms
Time efficiency analysis: A general plan
1. Decide on the parameter(s) that indicates the input size.
2. Identify the basic operation of the algorithm.
3. Check whether the number of times the basic operation is
executed depends on some additional property. If it does,
the efficiency cases must be investigated separately.
4. Set up a sum expressing the number of times the basic
operation is executed.
5. Using standard formulas and rules of sum manipulation,
either find a closed-form formula for the count or, at the
very least, establish its order of growth.

25
 Example: Analyze the time efficiency for an algorithm

Consider the following pseudo-code for finding the value of the largest
element in a list of 𝑛 numbers.
MaxElement(a[1 .. n]) {
max = a[1];
for (i = 2; i ≤ n; i++)
if (a[i] > max)
max = a[i];
return max;
}

The expression of time efficiency for the above code is

𝑇 𝑛 =𝑛−1

26
 Example: Analyze the time efficiency for an algorithm

Consider the following pseudo-code for multiplying two matrices.

MatrixMultiplication(a[1 .. n, 1 .. n], b[1 .. n, 1 .. n]){

for (i = 1; i ≤ n; i++)
for (j = 1; j ≤ n; j++) {
c[i, j] = 0;
for (k = 1; k ≤ n; k++)
c[i, j] = c[i, j] + a[i, k] * b[k, j];
}
}

The expression of time efficiency for the above code is

𝑇 𝑛 = 𝑛 × 𝑛 × 𝑛 = 𝑛3

27
 Example: Bubblesort and its improvement

Consider the following pseudo-code for the original Bubblesort.

BubbleSort(a[1 .. n]) {
for (i = 2; i < n; i++)
for (j = n; j  i; j--)
if (a[j - 1] > a[j])
a[j - 1] ⇆ a[j];
}

Let 𝐶(𝑖) be the number of comparisons made on a data set of size 𝑛.

𝑛(𝑛 − 1)
Then, 𝐶 𝑖 = ∈ Θ 𝑛2 for all distributions of the input array.
2

28
 Example: Bubblesort and its improvement

Consider an improved version of the previous Bubblesort.

BubbleSortImproved(a[1 .. n]) {
• Best case:
flag = true;
i = 1; 𝐵 𝑛 =𝑛−1∈Θ 𝑛
while (flag) { • Worst case:
flag = false; 𝑛(𝑛 − 1)
i++; 𝑊 𝑛 = ∈ Θ 𝑛2
2
for (j = n; j  i; j--)
• Average case
if (a[j - 1] > a[j]) {
𝑛−1
a[j - 1] ⇄ a[j]; 1
flag = true; 𝐴 𝑛 = ෍ 𝐶(𝑖) ∈ Θ 𝑛2
𝑛−1
} 𝑖=1
number of comparisons
}
made after iteration 𝑖 𝑡ℎ
}
29
 Example: Insertionsort and its improvement

Consider Insertionsort and its improvement.

Original Insertionsort Improved Insertionsort

InsertionSort(a[1 .. n]) { InsWithSentinel(a[1 .. n]) {

for (i = 2; i ≤ n; i++) { for (i = 2; i ≤ n; i++) {
v = a[i]; a[0] = v = a[i];
j = i – 1; j = i – 1;
while (j  1 && a[j] > v) { while (a[j] > v) {
a[j + 1] = a[j]; a[j + 1] = a[j];
j--; j--;
} }
a[j + 1] = v; a[j + 1] = v;
} }
} }

30
 Example: Insertionsort and its improvement

The best-case efficiency:

𝐵 𝑛 =𝑛−1∈Θ 𝑛
The worst-case efficiency:
𝑛
𝑛(𝑛 − 1)
𝑊 𝑛 = ෍(𝑖 − 1) = ∈ Θ 𝑛2
2
𝑖=1
The average-case efficiency:
𝑛
𝑛2 − 𝑛 𝑛2
𝐴 𝑛 = ෍ 𝐶(𝑖) ≈ + 𝑛 − ln 𝑛 − 𝛾 ≈ ∈ Θ 𝑛2
4 4
𝑖=2
1 1
where 𝐶 𝑖 = × 𝑖 − 1 + σ𝑖−1
𝑗=1 × 𝑗 — the average number of times the
𝑖 𝑖
comparison is executed when the algorithm inserts the 𝑖 𝑡ℎ element into
the left sorted subarray.
31
 Example: Insertionsort and its improvement

𝑛
1 1 1 1
෍ = 1 + + + ⋯ + ≈ ln 𝑛 + 𝛾
𝑖 2 3 𝑛
𝑖=1

𝛾 = 0.5722 … is the Euler constant

Image credit: Math Stack Exchange 32

 Example: Analyze the time efficiency for an algorithm

Consider the pseudo-code for finding the BitCount(n) {

number of binary digits in the binary count = 1;
representation of a positive decimal integer. while (n > 1) {
count++;
n = n / 2;
}
return count;
}

The number of times the comparison will be executed is

𝑇 𝑛 = log 2 𝑛 + 1 ∈ Θ log 2 𝑛
It is also the number of bits in the binary representation of 𝑛.

33
Recurrence relations

Rabbits and the Fibonacci Numbers (Fibonacci, 1202)

A young pair of rabbits (one of each sex) is placed on a desert
island. A pair of rabbits does not breed until they are 2 months
old. After they are 2 months old, each pair of rabbits produces
another pair each month.
Find a recurrence relation for the number of pairs of rabbits on
the island after 𝑛 months, assuming that no rabbits ever die.

34
Recurrence relations
• Let 𝐹𝑛 denote the number of pairs of rabbits after 𝑛 months.
• Firstly, there is not any pair of rabbits on this island, 𝐹𝑛 = 0.
Reproducing pairs Young pairs Month Total pairs
1 1 1
2 2 1
3 1 3 2
4 2 1 4 3
5 3 2 1 1 5 5
6 4 2 2
6 8
3 1 1 1
35
Recurrence relations
• The problem resembles the Fibonacci sequence, in which
the recurrence relation is defined as follows.
𝐹0 = 0
𝐹1 = 1
𝐹𝑛 = 𝐹𝑛−1 + 𝐹𝑛−2 𝑓𝑜𝑟 𝑛 ≥ 2

• To find the number of rabbit pairs after 𝑛 months, add the

number on the island the previous month, 𝐹𝑛−1 , and the
number of newborn pairs, 𝐹𝑛−2 .
• Each newborn pair comes from a pair at least 2 months old.

36
Solve recurrence relations
• The solution is an explicit formula, called a closed formula,
for the terms of the sequence.
• Example: Solving the recurrence relation
𝑥1 = 1
𝑥𝑛 = 2𝑥𝑛−1 + 1
gives us
𝑥𝑛 = 2𝑛 − 1

• We need to use mathematical induction to prove that our

guess is correct.

37
Approaches: Forward substitution
• We find successive terms beginning with the initial condition
about 𝑥1 and ending with 𝑥𝑛 .
• 𝑥1 = 1 = 21 − 1
• 𝑥2 = 2𝑥1 + 1 = 3 = 22 − 1
• 𝑥3 = 2𝑥2 + 1 = 7 = 23 − 1
• 𝑥4 = 2𝑥3 + 1 = 15 = 24 − 1
• …
• 𝑥𝑛 = 2𝑥𝑛−1 + 1 = 2𝑛 − 1

38
Approaches: Backward substitution
• From 𝑥𝑛 , we iterate to express it in terms of falling terms of the sequence
until we found it in terms of 𝑥1 .
• 𝑥𝑛 = 2𝑥𝑛−1 + 1
= 2 2𝑥𝑛−2 + 1 + 1 = 22 𝑥𝑛−2 + 21 + 20
= 22 2𝑥𝑛−3 + 1 + 21 + 20 = 23 𝑥𝑛−3 + 22 + 21 + 20
= 2𝑖 𝑥𝑛−𝑖 + 2𝑖−1 + ⋯ + 22 + 21 + 20
• Based on the initial condition 𝑥1 = 1, let’s set 𝑛 − 𝑖 = 1 to have 𝑖 = 𝑛 − 1.
• Therefore, 𝑥𝑛 = 2𝑛−1 𝑥𝑛−(𝑛−1) + 2(𝑛−1)−1 + ⋯ + 22 + 21 + 20
= 2𝑛−1 𝑥1 + 2𝑛−2 + ⋯ + 22 + 21 + 20
= 2𝑛 − 1

39
Linear recurrence relations
• A wide variety of recurrence relations occur in models.
• We solve them by using either iteration (forward/backward
substitution) or some other adhoc technique.

• Linear recurrence relations express the terms of a sequence

as linear combinations of previous terms.
• It is an important class of recurrence relation that can be
solved in a systematic way.

40
Linear homogeneous recurrence relations
• A linear homogeneous recurrence relation of degree 𝑘 with
constant coefficients is a recurrence relation of the form
𝑥𝑛 = 𝑐1 𝑥𝑛−1 + 𝑐2 𝑥𝑛−2 + ⋯ + 𝑐𝑘 𝑥𝑛−𝑘
or
𝑓 𝑛 = 𝑥𝑛 − 𝑐1 𝑥𝑛−1 + 𝑐2 𝑥𝑛−2 + ⋯ + 𝑐𝑘 𝑥𝑛−𝑘 = 0

where 𝑐1 , 𝑐2 , … , 𝑐𝑘 ∈ ℝ, 𝑐𝑘 ≠ 0 and the 𝑘 initial conditions are

𝑥0 = 𝐶0 , 𝑥1 = 𝐶1 , … , 𝑥𝑘−1 = 𝐶𝑘−1
• There are many solutions, depending on the value of the
initial condition.

41
Solving linear recurrence relations
• For example, the recurrence relations shown below are not
linear homogeneous with constant coefficients.
2
• 𝑥𝑛 = 𝑥𝑛−1 + 𝑥𝑛−1 is not linear.
• 𝑥𝑛 = 2𝑥𝑛−1 + 3 is not homogeneous.
• 𝑥𝑛 = 𝑛𝑥𝑛−1 does not have constant coefficients.

42
Linear homogeneous recurrence relations
• 𝑥𝑛 = 𝑟 𝑛 is a solution of the recurrence relation
𝑥𝑛 = 𝑐1 𝑥𝑛−1 + 𝑐2 𝑥𝑛−2 + ⋯ + 𝑐𝑘 𝑥𝑛−𝑘
if and only if
𝑟 𝑛 = 𝑐1 𝑟 𝑛−1 + 𝑐2 𝑟 𝑛−2 + ⋯ + 𝑐𝑘 𝑟 𝑛−𝑘
• Divide both sides of this equation by 𝑟 𝑛−𝑘 (when 𝑟 ≠ 0) and
substract the terms on the right.
𝑟 𝑘 − 𝑐1 𝑟 𝑘−1 − 𝑐2 𝑟 𝑘−2 − ⋯ − 𝑐𝑘 𝑟 0 = 0
• That is the characteristic equation of the recurrence relation.

43
Linear homogeneous recurrence relations
• The solutions of the characteristic equation of the recurrence
relation are the characteristic roots of the recurrence.
• We use these characteristic roots to give an explicit formula
for all the solutions of the recurrence relation.

• For simplicity, let’s consider linear homogeneous recurrence

relations of degree two.
• The characteristic equation in this case has the below form.
𝑎𝑟 2 + 𝑏𝑟 + 𝑐 = 0
• 𝑥𝑛 can be solved in one of the three cases.

44
Linear homogeneous recurrence relations
• Case 1: Suppose that the equation has two distinct roots
𝑟1 ∈ ℝ and 𝑟2 ∈ ℝ. The solution is:
𝑥 𝑛 = 𝛼𝑟1𝑛 + 𝛽𝑟2𝑛 ∀𝛼, 𝛽 ∈ ℝ
• Case 2: Suppose that the equation has only one root 𝑟 ∈ ℝ.
The solution is:
𝑥 𝑛 = 𝛼𝑟 𝑛 + 𝛽𝑛𝑟 𝑛 ∀𝛼, 𝛽 ∈ ℝ
• Case 3: If 𝑟1,2 = 𝑢 ± 𝑖𝑣, then the solution is
𝑥 𝑛 = 𝛾 𝑛 𝛼 cos 𝑛𝜃 + 𝛽 sin 𝑛𝜃
∀𝛼, 𝛽 ∈ ℝ, 𝛾 = 𝑢2 + 𝑣 2 , 𝜃 = arctan 𝑣/𝑢

45
 Example: Find the solution of the recurrence relation

Find the solution of the followingrecurrence relation

𝑥𝑛 = 𝑥𝑛−1 + 2𝑥𝑛−2

with initial conditions 𝑥0 = 2 and 𝑥1 = 7.

The solution is 𝑥𝑛 = 3 × 2𝑛 − −1 𝑛

46
 Example: Find the solution of the recurrence relation

Find an explicit formula of the followingrecurrence relation

𝑥𝑛 = 6𝑥𝑛−1 − 9𝑥𝑛−2

with initial conditions 𝑥0 = 0 and 𝑥1 = 3.

The solution is 𝑥𝑛 = 𝑛3𝑛

47
Mathematical analysis of
Recursive Algorithms
Time efficiency analysis: A general plan
1. Decide on the parameter(s) that indicates the input size.
2. Identify the basic operation of the algorithm.
3. Check whether the number of times the basic operation is
executed depends on some additional property. If it does,
the efficiency cases must be investigated separately.
4. Set up a recurrence relation, with an appropriate initial
condition, for the number of times the basic operation is
executed.
5. Solve the recurrence or, at least, ascertain the order of
growth of its solution.

49
 Example: Find the factorial of 𝑛: 𝑛!

Consider the pseudo-code for finding the factorial of 𝑛: 𝑛!.

Factorial(n) {
if (n == 0)
return 1;
return Factorial(n – 1) * n;
}

Let 𝑀(𝑛) denote the number of times the basic operation is executed.
The recurrence relation is as follows:
𝑀(𝑛) = 𝑀(𝑛 − 1) + 1
𝑀(0) = 0
Thus, 𝑀(𝑛) ∈ Θ(𝑛).
50
 Example: Solve the Tower of Hanoi puzzle with 𝑛 disks

Consider the pseudo-code for solving the Tower of Hanoi puzzle of 𝑛 disks.

HNTower(n, left, middle, right) {

if (n) {
HNTower(n – 1, left, right, middle);
Movedisk(1, left, right);
HNTower(n – 1, middle, left, right);
}
}

Let 𝑀(𝑛) denote the number of times the basic operation is executed.
Thus, 𝑀 𝑛 = 2𝑛 − 1 ∈ Θ(2𝑛 ), with the following recurrence relation.
𝑀 𝑛 = 𝑀 𝑛 − 1 + 1 + 𝑀 𝑛 − 1 = 2𝑀 𝑛 − 1 + 1
𝑀(1) = 1
51
 Example: Analyze the time efficiency for an algorithm

Consider the pseudo-code for finding the number of binary digits in the
binary representation of a positive decimal integer.

BitCount(n) {
if (n == 1) return 1;
return BitCount( n / 2) + 1;
}

Let 𝐴(𝑛) denote the number of times the basic operation is executed.
𝑛
The number of additions made in computing BitCount( n / 2) is 𝐴 (⌊ ⌋).
2
The recurrence relation is as follows:
𝑛
𝐴(𝑛) = 𝐴 (⌊ ⌋) + 1
2
𝐴(1) = 0
52
“Smoothness rule” theorem
• Let 𝑔(𝑛) be a nonnegative function defined on the set of natural numbers.
• 𝑔(𝑛) is called smooth if it is finally nondecreasing and 𝑔 2𝑛 ∈ Θ 𝑔 𝑛 ) .

“Smoothness rule” theorem

Let 𝑓(𝑛) be an eventually nondecreasing function and 𝑔(𝑛) be
a smooth function.
If 𝑓 𝑛 ∈ Θ 𝑔 𝑛 ) for values of 𝑛 that are powers of 𝑏 where
𝑏 ≥ 2, then
𝒇(𝒏) ∈ 𝚯 𝒈(𝒏 )

(The analogous results hold for the cases of 𝑂 and Ω as well.)

53
Solve a recurrence relation: Approach
• Solve the given recurrence relation only for 𝑛 = 2𝑘 .
• Apply “Smoothness rule” theorem to give a correct answer
about the order of growth for all values of 𝑛.
• This rule claims that the order of growth observed for 𝑛 = 2𝑘 .

54
 Example: Analyze the time efficiency for an algorithm

Consider the pseudo-code for finding the number of binary digits in the
binary representation of a positive decimal integer.

BitCount(n) {
if (n == 1) return 1;
return BitCount( n / 2) + 1;
}

Assume that 𝑛 = 2𝑘 .
𝑛
The recurrence relation 𝐴(𝑛) = 𝐴 (⌊ ⌋) + 1 takes the form:
2
𝐴(2𝑘 ) = 𝐴 (2𝑘−1 ) + 1
𝐴(20 ) = 0
Thus, 𝐴(𝑛) = log 2 𝑛 ∈ Θ log 2 𝑛 .
55
 Example: Compute the 𝒏𝒕𝒉 Fibonacci number

The recurrence relation is 𝐹𝑛 = 𝐹𝑛−1 + 𝐹𝑛−2 with 𝐹0 = 0 and 𝐹1 = 1.

The characteristic equation of the recurrence relation is 𝑟 2 − 𝑟 − 1 = 0,

1± −1 2− 4(1)(−1) 1 ± 5
which has two distinct roots: 𝑟1,2 = =
2 2
𝑛 𝑛
1+ 5 1− 5
Hence, 𝐹𝑛 = 𝛼 +𝛽
2 2
0 1
1+ 5 1− 5
Solve the system of equations 𝐹0 = 𝛼 +𝛽 =0
2 2
1 1
1+ 5 1− 5
𝐹1 = 𝛼 +𝛽 =1
2 2
to obtain 𝛼 = 1Τ 5 and 𝛽 = −1Τ 5.
56
 Example: Compute the 𝒏𝒕𝒉 Fibonacci number

Therefore, the closed-form solution for the 𝑛𝑡ℎ Fibonacci number, known
as Binet's formula, is
𝑛 𝑛
1
1+ 5 1 1− 5
𝐹𝑛 = −
5 2 5 2

1+ 5 1− 5
Let 𝜙 = ≈ 1.61803 (golden ratio) and 𝜙෠ = = 1 − 𝜙 ≈ −0.61803.
2 2

1
Then, 𝐹𝑛 = 𝜙 𝑛 − 𝜙෠ 𝑛
5

57
Some example algorithms
𝑛𝑡ℎ Fibonacci number: Recursive approach
Fibonacci(n) {
if (n ≤ 1)
return n;
return Fibonacci(n - 1) + Fibonacci(n - 2);
}

• Let 𝐴(𝑛) be the number of times the basic operation to compute 𝐹𝑛 .

• The recurrence equation for this approach is
𝐴 𝑛 = 𝐴 𝑛 − 1 + 𝐴 𝑛 − 2 + 1 𝑤𝑖𝑡ℎ 𝑛 > 1
𝐴 0 = 0, 𝐴 1 = 0
• Solving this recurrence equation gives us
1
𝐴 𝑛 = 𝜙 𝑛+1 − 𝜙෠ 𝑛+1 − 1 ∈ Θ 𝜙 𝑛
5
59
𝑛𝑡ℎ Fibonacci number: Non-recursive
• It’s easy to construct a linear algorithm using the formula
1
𝐹𝑛 = 𝜙 𝑛 − 𝜙෠ 𝑛
5

• In practice, we may use the below formula when 𝑛 → ∞.

1
𝐹𝑛 = 𝑟𝑜𝑢𝑛𝑑 𝜙𝑛
5

60
𝑛𝑡ℎ Fibonacci number: DP
• Dynamic programming
Fibonacci(n) { Fibonacci(n) {
if (n ≤ 1) if (n ≤ 1)
return n; return n;
f0 = 0; f0 = 0;
f1 = 1; f1 = 1;
for (i = 2; i ≤ n; i++) { for (i = 2; i ≤ n; i++) {
fn = f0 + f1; f1 = f1 + f0;
f0 = f1; f0 = f1 – f0;
f1 = fn; }
} return f1;
return fn; }
}
61
𝑛𝑡ℎ Fibonacci number: Matrix approach
• Consider the following equation
𝑛
𝐹(𝑛 + 1) 𝐹(𝑛) 1 1
= 𝑤𝑖𝑡ℎ 𝑛 ≥ 1
𝐹(𝑛) 𝐹(𝑛 − 1) 1 0
• It is easy to prove its correctness using induction.

• The following formula efficiently computes the right term.

𝑛/2 2
1 1
𝑛
𝑛 𝑖𝑠 𝑒𝑣𝑒𝑛
1 1 1 0
= 2
1 0 𝑛/2
1 1 1 1
× 𝑛 𝑖𝑠 𝑜𝑑𝑑
1 0 1 0

• The running time of this approach is Θ log 2 𝑛

62
𝑛𝑡ℎ Fibonacci number: Matrix approach
int fib(int n) {
F[2][2] = {{1, 1},{1, 0}};
if (n == 0) return 0;
power(F, n - 1);
return F[0][0];
} void power(int F[2][2], int n) {
if (n ≤ 1) return;
T[2][2] = {{1, 1},{1, 0}};
power(F, n / 2);
multiply(F, F);
if (n % 2 != 0)
multiply(F,T);
}

63
𝑛𝑡ℎ Fibonacci number: Matrix approach
void multiply(F[2][2], T[2][2]) {
t1 = F[0][0]*T[0][0] + F[0][1]*T[1][0];
t2 = F[0][0]*T[0][1] + F[0][1]*T[1][1];
t3 = F[1][0]*T[0][0] + F[1][1]*T[1][0];
t4 = F[1][0]*T[0][1] + F[1][1]*T[1][1];
F[0][0] = t1; F[0][1] = t2;
F[1][0] = t3; F[1][1] = t4;
}
void main() {
cout << fib(5);
}

• The weakness of the above code is recursive calls.

• Using a “loop” approach is always better.
64
𝑛𝑡ℎ Fibonacci number: Matrix approach

int Fibonacci(int n) {
i = 1; j = 0; k = 0; h = 1;
while (n) {
if (n % 2) {
t = j * h;
j = i * h + j * k + t;
i = i * k + t;
}
t = h * h; h = 2 * k * h + t; k = k * k + t;
n = n / 2;
}
return j;
}
65
66