Module 1:

Analyzing the Efficiency of

Algorithms

Dr. Natarajan Meghanathan

Associate Professor of Computer Science
Jackson State University
Jackson, MS 39217
E-mail: natarajan.meghanathan@jsums.edu

Based on Chapter 2 in the Textbook

 What is an Algorithm?
• An algorithm is a sequence of unambiguous instructions for solving a
problem, i.e., for obtaining a required output for any legitimate input in
a finite amount of time. Problem

Algorithm

Input Computer Output

• Important Points about Algorithms

– The non-ambiguity requirement for each step of an algorithm
cannot be compromised
– The range of inputs for which an algorithm works has to be
specified carefully.
– The same algorithm can be represented in several different ways
– There may exist several algorithms for solving the same problem.
• Can be based on very different ideas and can solve the problem with
dramatically different speeds
 The Analysis Framework
• Time efficiency (time complexity): indicates how fast an algorithm
runs
• Space efficiency (space complexity): refers to the amount of
memory units required by the algorithm in addition to the space
needed for its input and output
• Algorithms that have non-appreciable space complexity are said to
be in-place.
• The time efficiency of an algorithm is typically as a function of the
input size (one or more input parameters)
– Algorithms that input a collection of values:
• The time efficiency of sorting a list of integers is represented in terms of the
number of integers (n) in the list
• For matrix multiplication, the input size is typically referred as n*n.
• For graphs, the input size is the set of Vertices (V) and edges (E).
– Algorithms that input only one value:
• The time efficiency depends on the magnitude of the integer. In such cases,
the algorithm efficiency is represented as the number of bits 1+ log 2 n 
needed to represent the integer n
 Units for Measuring Running Time
• The running time of an algorithm is to be measured with a unit that is
independent of the extraneous factors like the processor speed,
quality of implementation, compiler and etc.
– At the same time, it is not practical as well as not needed to count the
number of times, each operation of an algorithm is performed.
• Basic Operation: The operation contributing the most to the total
running time of an algorithm.
– It is typically the most time consuming operation in the algorithm’s
innermost loop.
• Examples: Key comparison operation; arithmetic operation (division being
the most time-consuming, followed by multiplication)
– We will count the number of times the algorithm’s basic operation is
executed on inputs of size n.
 Examples for
Input Size and Basic Operations
Problem Input size measure Basic operation

Searching for key in a Number of list’s items,

Key comparison
list of n items i.e. n

Multiplication of two Matrix dimensions or Multiplication of two

matrices total number of elements numbers

Checking primality of n’size = number of digits

Division
a given integer n (in binary representation)

Visiting a vertex or
Typical graph problem #vertices and/or edges
traversing an edge
 Orders of Growth
• We are more interested in the order of growth on the number of times
the basic operation is executed on the input size of an algorithm.
• Because, for smaller inputs, it is difficult to distinguish efficient
algorithms vs. inefficient ones.
• For example, if the number of basic operations of two algorithms to
solve a particular problem are n and n2 respectively, then
– if n = 3, then we may say there is not much difference between requiring
3 basic operations and 9 basic operations and the two algorithms have
about the same running time.
– On the other hand, if n = 10000, then it does makes a difference whether
the number of times the basic operation is executed is n or n2.
Exponential-growth
functions

Source: Table 2.1

From Levitin, 3rd ed.
Best-case, Average-case, Worst-case
• For many algorithms, the actual running time may not only
depend on the input size; but, also on the specifics of a
particular input.
– For example, sorting algorithms (like insertion sort) may run
faster on an input sequence that is almost-sorted rather than on a
randomly generated input sequence.

• Worst case: Cworst(n) – maximum number of times the basic

operation is executed over inputs of size n
• Best case: Cbest(n) – minimum # times over inputs of size n
• Average case: Cavg(n) – “average” over inputs of size n
– Number of times the basic operation will be executed on typical
input
– NOT the average of worst and best case
– Expected number of basic operations considered as a random
variable under some assumption about the probability distribution
of all possible inputs
 Example for Worst and Best-Case
Analysis: Sequential Search

/* Assume the second condition will not

be executed if the first condition evaluates to
false */

• Worst-Case: Cworst(n) = n
• Best-Case: Cbest(n) = 1
 Probability-based Average-Case
Analysis of Sequential Search
• If p is the probability of finding an element in the list, then (1-p) is the
probability of not finding an element in the list.
• Also, on an n-element list, the probability of finding the search key as
the ith element in the list is p/n for all values of 1 ≤i ≤ n

• If p = 1 (the element that we will search for always exists in the list),
then Cavg(n) = (n+1)/2. That is, on average, we visit half of the entries
in the list to search for any element in the list.
• If p = 0 (all the time, the element that we will search never exists),
then Cavg(n) = n. That is, we visit all the elements in the list.
YouTube Link: https://www.youtube.com/watch?v=8V-bHrPykrE
Asymptotic Notations: Intro
2n ≤ 0.05 n2
for n ≥ 40
2n = O(n2)

0.05n2 ≥ 2n
for n ≥ 40
0.05n2 = Ω(n)
Asymptotic Notations: Intro
2n ≤ 5n
5n for n ≥ 1
2n = O(n)

2n ≥ n
for n ≥ 1
2n 2n = Ω(n)
n As 2n = O(n)
and 2n = Ω(n),
we say
2n = Θ(n)

n
 Asymptotic Notations: Formal Intro

t(n) = O(g(n)) t(n) = Ω(g(n))

t(n) ≤ c*g(n) for all n ≥ n0 t(n) ≥ c*g(n) for all n ≥ n0
c is a positive constant (> 0) c is a positive constant (> 0)
and n0 is a non-negative integer and n0 is a non-negative integer

Note: If t(n) = O(g(n))
g(n) = Ω(t(n)); also, if t(n) = Ω(g(n))
g(n) = O(t(n))
Asymptotic Notations: Formal Intro

t(n) = Θ(g(n))
c2*g(n) ≤ t(n) ≤ c1*g(n) for all n ≥ n0
c1 and c2 are positive constants (> 0)
and n0 is a non-negative integer
 Asymptotic Notations: Examples
• Let t(n) and g(n) be any non-negative functions defined on a set of all
real numbers.
• We say t(n) = O(g(n)) for all functions t(n) that have a lower or the
same order of growth as g(n), within a constant multiple as n  ∞.
– Examples:

• We say t(n) = Ω(g(n)) for all functions t(n) that have a higher or the
same order of growth as g(n), within a constant multiple as n  ∞.
– Examples:

• We say t(n) = Θ(g(n)) for all functions t(n) that have the same order of
growth as g(n), within a constant multiple as n  ∞.
– Examples: an2 + bn + c = Θ(n2);
n2 + logn = Θ(n2)
 Useful Property of Big-Oh Notation
• If t1(n) ∈ O(g1(n)) and t2(n) ∈ O(g2(n)) , then
t1(n) + t2(n) ∈ O(max{g1(n), g2(n)})

Note: The above property implies that

if an algorithm comprises of two or more
consecutively executed parts, the overall
efficiency of the algorithm is determined by
the part with the highest order of growth.
 Extending the Property
• The property that we proved in the previous slide can be
extended directly for the Θ notation:
• If t1(n) ∈ Θ(g1(n)) and t2(n) ∈ Θ(g2(n)) , then
t1(n) + t2(n) ∈ Θ(max{g1(n), g2(n)})

• The property can be applied for the Ω notation with a

slight change: Replace the Max with the Min.
• If t1(n) ∈ Ω(g1(n)) and t2(n) ∈ Ω(g2(n)) , then
t1(n) + t2(n) ∈ Ω(min{g1(n), g2(n)})
Using Limits to Compare Order of Growth

The first case means t(n) = O(g(n)

if the second case is true, then t(n) = Θ(g(n))
The last case means t(n) = Ω(g(n))

L’Hopital’s Rule

Note: t’(n) and g’(n) are first-order derivatives of t(n) and g(n)

Stirling’s Formula
Example 1: To Determine the
Order of Growth
Example 1: To Determine the
Order of Growth (continued…)
Example 2: To Determine the
Order of Growth
Example 2: To Determine the
Order of Growth (continued…)
Examples to Compare the Order of Growth

Example 3: Compare the order of growth of log2n and logn2.

 Some More Examples: Order of Growth

• a) (n2+1)10 : Informally, = (n2+1)10 ≈ n20.

Formally,

b)

c)
d)
 Some More Examples: Order of Growth

The listing of the functions in the increasing

Order of growth is as follows:

log2n 2logn 6 logn 6 18

Lim ----------- = Lim ---------------- = ----------- = -------------- = Lim -------- = 0
n  ∞ n1/3 n  ∞ n*(1/3)n(-2/3) n(1/3) n*(1/3)n(-2/3) n  ∞ n(1/3)
Hence, log2n = O(n1/3)
 Time Efficiency of Non-recursive
Algorithms: General Plan for Analysis
• Decide on parameter n indicating input size
• Identify algorithm’s basic operation
• Determine worst, average, and best cases for input of size
n, if the number of times the basic operation gets executed
varies with specific instances (inputs)
• Set up a sum for the number of times the basic operation is
executed
• Simplify the sum using standard formulas and rules
Useful Summation Formulas and Rules
Σl≤i≤u1 = 1+1+…+1 = u - l + 1
In particular, Σl≤i≤n1 = n - 1 + 1 = n ∈ Θ(n)

Σ1≤i≤n i = 1+2+…+n = n(n+1)/2 ≈ n2/2 ∈ Θ(n2)

Σ1≤i≤n i2 = 12+22+…+n2 = n(n+1)(2n+1)/6 ≈ n3/3 ∈ Θ(n3)

Σ0≤i≤n ai = 1 + a +…+ an = (an+1 - 1)/(a - 1) for any a ≠ 1

In particular, Σ0≤i≤n 2i = 20 + 21 +…+ 2n = 2n+1 - 1 ∈ Θ(2n )

Σ(ai ± bi ) = Σai ± Σbi Σcai = cΣai Σl≤i≤uai = Σl≤i≤mai + Σm+1≤i≤uai

u

∑1 = (u − l + 1)
i =l
 Examples on Summation
• 1 + 3 + 5 + 7 + …. + 999

• 2 + 4 + 8 + 16 + … + 1024
Example 1: Finding Max. Element

• The basic operation is the comparison executed on each repetition of

the loop.
• In this algorithm, the number of comparisons is the same for all arrays
of size n.
• The algorithm makes one comparison on each execution of the loop,
which is repeated for each value of the loop’s variable i within the
bounds 1 and n-1 (inclusively). Hence,
Example 2: Sequential Key Search

/* Assume the second condition will not

be executed if the first condition evaluates to
false */
Depending on the value of the Key K,
the number of comparisons performed
can range from 1 to n.

• Worst-Case: Cworst(n) = n In such cases, it would be more appropriate

• Best-Case: Cbest(n) = 1 to denote the overall time complexity for the
basic operation as O(n), rather than Θ(n).
 Example 3: Finding Prime

The number of division operations could range anywhere from 2 to n-1.

For example, if n = 9, the number of divisions is only 2 (for i = 2 and 3)
If n = 541, a prime the number of divisions is 539 (for i = 2, 3, …, 540)

Thus, for a given n, the number of divisions could be as small as 1,

and as large as n – 2. In such cases, it would be more appropriate to denote
the overall time complexity for the basic operation as O(n), rather than Θ(n).
 Rule of Thumb
• We will use the notation Θ(f(n)), when the number of times the basic
operation will be executed is tightly bound to f(n).
– Like in Example 1 – To find the maximum element in an array
– In general: When the best case / worst case ratio tends to a constant
(which was 1 for Example 1), as n grows to ∞.
• We will use the notation O(f(n)), when the number of times the basic
operation will be executed is bounded at most (i.e., the upper
bound) to f(n) and the lower bound could be anything arbitrary (like
1).
– Like in Example 3 – To find whether an integer is prime or not.
– In general: When the best case / worst case ratio tends to 0, as n grows
to ∞. For Example 3, the best case / worst case ratio is 1/(n-2), and this
ratio evaluates to 0, as n grows to ∞.

• Comparison of O and Θ: Which is better?

• If two algorithms A and B for a particular problem incur overall time
complexities O(f(n)) and Θ(f(n)), then, it would be better to choose
algorithm A with complexity O(f(n)) as the algorithm incurs at most
f(n) complexity for certain inputs; where as, algorithm B incurs f(n)
complexity on all inputs.
 Revisiting Examples 1 and 3
• Example 1: To find the max. element
– Best-case: n-1 comparisons
– Worst-case: n-1 comparisons

After discarding the constants and lower order terms, the overall time
complexity is Θ(worst-case comparisons) = Θ(n).

• Example 3: To find whether an integer is prime or not

– Best-case: 1 division
– Worst-case: n-2 divisions

After discarding the constants and lower order terms, the overall time
complexity is O(worst-case comparisons) = O(n).
Example 4: Element Uniqueness Problem

Best-case situation:
If the two first elements of the array are the same, then we can exit
after one comparison. Best case = 1 comparison.
Worst-case situation:
• The basic operation is the comparison in the inner loop. The worst-
case happens for two-kinds of inputs:
– Arrays with no equal elements
– Arrays in which only the last two elements are the pair of equal
elements
Example 4: Element Uniqueness Problem
• For these kinds of inputs, one comparison is made for each repetition
of the innermost loop, i.e., for each value of the loop’s variable j
between its limits i+1 and n-1; and this is repeated for each value of the
outer loop i.e., for each value of the loop’s variable i between its limits 0
and n-2. Accordingly, we get,

Best − case 1 2 Hence, overall time

lim = lim 2 = lim 2 = 0 Complexity is O(n2)
n − > ∞ Worst − case n − >∞ n / 2 n − >∞ n
After discarding the constants
and lower order terms, if any
Example 5: Matrix Multiplication
 Example 5: Matrix Multiplication
• The innermost loop has two arithmetic operations – addition and
multiplication – executed once, each time the loop is entered.
• Basic operation - the multiplication operation in the innermost loop.
• Since the number of basic operations depends only on the size of the
input matrices and not on the value of the elements, the best-case,
average-case and worst-case scenarios are one and the same.

• The number of multiplications made for every pair of specific values of

variables i and j is:

• The total number of multiplications is expressed as follows:

We do the same number of multiplications all the time.

Hence, overall time complexity is Θ(n3).
 Time Efficiency of Recursive
Algorithms: General Plan for Analysis
• Decide on a parameter indicating an input’s size.

• Identify the algorithm’s basic operation.

• Check whether the number of times the basic op. is executed may vary
on different inputs of the same size. (If it may, the worst, average, and
best cases must be investigated separately.)

• Set up a recurrence relation with an appropriate initial condition

expressing the number of times the basic op. is executed.

• Solve the recurrence (or, at the very least, establish its solution’s order
of growth) by backward substitutions or another method.
 Recursive Evaluation of n!
Definition: n ! = 1 ∗ 2 ∗ … ∗(n-1) ∗ n for n ≥ 1 and 0! = 1
• Recursive definition of n!: F(n) = F(n-1) ∗ n for n ≥ 1 and
F(0) = 1

M(n-1) = M(n-2) + 1; M(n-2) = M(n-3)+1

M(n) = [M(n-2)+1] + 1 = M(n-2) + 2 = [M(n-3)+1+2] = M(n-3) + 3

= M(n-n) + n = n
Overall time Complexity: Θ(n)
YouTube Link: https://www.youtube.com/watch?v=K25MWuKKYAY
 Counting the # Bits of an Integer

# Divisions

Since the recursive calls end when n is equal to 1 and there are no divisions
made, the initial condition is: A(1) = 0.

Solution Approach: If we use the backward substitution

method (as we did in the previous two examples, we will
get stuck for values of n that are not powers of 2).
We proceed by setting n = 2k for k ≥ 0.
New recurrence
relation to solve:
Counting the # Bits of an Integer
 Examples for Solving Recurrence
1) Relations

=Θ(2n)
2)

3)
4)
5)
6)