1.

Introduction
What is an Algorithm?
An algorithm is a finite set of instructions to solve a particular problem. In addition, all
algorithms must satisfy the following criteria:
a. Input. Zero or more quantities are externally supplied.
b. Output. At least one quantity is produced.
c. Definiteness. Each instruction is clear and unambiguous. It must be perfectly clear
what should be done.
d. Finiteness. If we trace out the instruction of an algorithm, then for all cases, the
algorithm terminates after a finite number of steps.
e. Effectiveness. Every instruction must be very basic so that it can be carried out, in
principle, by a person using only pencil and paper. It is not enough that each
operation be definite as in criterion c; it also must be feasible.

Algorithm design and analysis process - We now briefly discuss a sequence of steps one
typically goes through in designing and analyzing an algorithm

• Understanding the Problem - From a practical perspective, the first thing you need to
do before designing an algorithm is to understand completely the problem given. An
input to an algorithm specifies an instance of the problem the algorithm solves. It is very
important to specify exactly the set of instances the algorithm needs to handle.
• Ascertaining the Capabilities of the Computational Device - Once you completely
understand a problem, you need to ascertain the capabilities of the computational device
the algorithm is intended for. Select appropriate model from sequential or parallel
programming model.

Page|1.2
• Choosing between Exact and Approximate Problem Solving - The next principal
decision is to choose between solving the problem exactly and solving it approximately.
Because, there are important problems that simply cannot be solved exactly for most of
their instances and some of the available algorithms for solving a problem exactly can be
unacceptably slow because of the problem’s intrinsic complexity.
• Algorithm Design Techniques - An algorithm design technique (or “strategy” or
“paradigm”) is a general approach to solving problems algorithmically that is applicable
to a variety of problems from different areas of computing. They provide guidance for
designing algorithms for new problems, i.e., problems for which there is no known
satisfactory algorithm.
• Designing an Algorithm and Data Structures - One should pay close attention to
choosing data structures appropriate for the operations performed by the algorithm. For
example, the sieve of Eratosthenes would run longer if we used a linked list instead of an
array in its implementation. Algorithms + Data Structures = Programs
• Methods of Specifying an Algorithm- Once you have designed an algorithm; you need
to specify it in some fashion. These are the two options that are most widely used
nowadays for specifying algorithms. Using a natural language has an obvious appeal;
however, the inherent ambiguity of any natural language makes a concise and clear
description of algorithms surprisingly difficult. Pseudocode is a mixture of a natural
language and programming language like constructs. Pseudocode is usually more precise
than natural language, and its usage often yields more succinct algorithm descriptions.
• Proving an Algorithm’s Correctness - Once an algorithm has been specified, you have
to prove its correctness. That is, you have to prove that the algorithm yields a required
result for every legitimate input in a finite amount of time. For some algorithms, a proof
of correctness is quite easy; for others, it can be quite complex. A common technique for
proving correctness is to use mathematical induction because an algorithm’s iterations
provide a natural sequence of steps needed for such proofs.
• Analyzing an Algorithm - After correctness, by far the most important is efficiency. In
fact, there are two kinds of algorithm efficiency: time efficiency, indicating how fast the
algorithm runs, and space efficiency, indicating how much extra memory it uses. Another
desirable characteristic of an algorithm is simplicity. Unlike efficiency, which can be
precisely defined and investigated with mathematical rigor, simplicity, like beauty, is to a
considerable degree in the eye of the beholder.
• Coding an Algorithm - Most algorithms are destined to be ultimately implemented as
computer programs. Implementing an algorithm correctly is necessary but not sufficient:
you would not like to diminish your algorithm’s power by an inefficient implementation.
Modern compilers do provide a certain safety net in this regard, especially when they are
used in their code optimization mode.

Page|1.3
Page|1.4
Analysis Framework
General framework for analyzing the efficiency of algorithms is discussed here. There are
two kinds of efficiency: time efficiency and space efficiency. Time efficiency indicates how
fast an algorithm in question runs; space efficiency deals with the extra space the algorithm
requires.
In the early days of electronic computing, both resources time and space were at a premium.
Now the amount of extra space required by an algorithm is typically not of as much concern,
In addition, the research experience has shown that for most problems, we can achieve much
more spectacular progress in speed than in space. Therefore, following a well-established
tradition of algorithm textbooks, we primarily concentrate on time efficiency.

Measuring an Input’s Size

It is observed that almost all algorithms run longer on larger inputs. For example, it takes
longer to sort larger arrays, multiply larger matrices, and so on. Therefore, it is logical to
investigate an algorithm's efficiency as a function of some parameter n indicating the
algorithm's input size.
There are situations, where the choice of a parameter indicating an input size does matter.
The choice of an appropriate size metric can be influenced by operations of the algorithm in
question. For example, how should we measure an input's size for a spell-checking
algorithm? If the algorithm examines individual characters of its input, then we should
measure the size by the number of characters; if it works by processing words, we should
count their number in the input.
We should make a special note about measuring the size of inputs for algorithms involving
properties of numbers (e.g., checking whether a given integer n is prime). For such
algorithms, computer scientists prefer measuring size by the number b of bits in the n's binary
representation: b = ⎝log2 n ] + 1. This metric usually gives a better idea about the efficiency
of algorithms in question.

Units for Measuring Running lime

To measure an algorithm's efficiency, we would like to have a metric that does not depend
on these extraneous factors. One possible approach is to count the number of times each of
the algorithm's operations is executed. This approach is both excessively difficult and, as we
shall see, usually unnecessary. The thing to do is to identify the most important operation of
the algorithm, called the basic operation, the operation contributing the most to the total
running time, and compute the number of times the basic operation is executed.
For example, most sorting algorithms work by comparing elements (keys) of a list being
sorted with each other; for such algorithms, the basic operation is a key comparison.
As another example, algorithms for matrix multiplication and polynomial evaluation
require two arithmetic operations: multiplication and addition.

Page|1.5
Let cop be the execution time of an algorithm's basic operation on a particular computer, and
let C(n) be the number of times this operation needs to be executed for this algorithm. Then
we can estimate the running time T(n) of a program implementing this algorithm on that
computer by the formula:
T(n) = copC(n)

unless n is extremely large or very small, the formula can give a reasonable estimate of the
algorithm's running time.
It is for these reasons that the efficiency analysis framework ignores multiplicative constants
and concentrates on the count's order of growth to within a constant multiple for large-size
inputs.

Orders of Growth
Why this emphasis on the count's order of growth for large input sizes? Because for large
values of n, it is the function's order of growth that counts: just look at table which contains
values of a few functions particularly important for analysis of algorithms.
Table: Values
of several
functions
important for
analysis of
algorithms

Algorithms that require an exponential number of operations are practical for solving only
problems of very small sizes.

Worst-Case, Best-Case, and Average-Case Efficiencies

Definition: The worst-case efficiency of an algorithm is its efficiency for the worst-case
input of size n, for which the algorithm runs the longest among all possible inputs of that size.
Consider the algorithm for sequential search.

Page|1.6
The running time of above algorithm can be quite different for the same list size n. In the
worst case, when there are no matching elements or the first matching element happens to
be the last one on the list, the algorithm makes the largest number of key comparisons
among all possible inputs of size n: Cworst(n) = n.
In general, we analyze the algorithm to see what kind of inputs yield the largest value of the
basic operation's count C(n) among all possible inputs of size n and then compute this worst-
case value Cworst (n). The worst-case analysis provides algorithm's efficiency by bounding its
running time from above. Thus it guarantees that for any instance of size n, the running time
will not exceed Cworst (n), its running time on the worst-case inputs.
Definition: The best-case efficiency of an algorithm is its efficiency for the best-case input
of size n, for which the algorithm runs the fastest among all possible inputs of that size.
We determine the kind of inputs for which the count C(n) will be the smallest among all
possible inputs of size n. For example, for sequential search, best-case inputs are lists of size
n with their first elements equal to a search key; Cbest(n) = 1.
The analysis of the best-case efficiency is not nearly as important as that of the worst-case
efficiency. Also, neither the worst-case analysis nor its best-case counterpart yields the
necessary information about an algorithm's behavior on a "typical" or "random" input. This
information is provided by average-case efficiency.
Definition: the average-case complexity of an algorithm is the amount of time used by the
algorithm, averaged over all possible inputs.
Let us consider again sequential search. The standard assumptions are that (a) the probability
of a successful search is equal top (0 ≤ p ≤ 1) and (b) the probability of the first match
occurring in the ith position of the list is the same for every i. We can find the average number
of key comparisons Cavg (n) as follows.
In the case of a successful search, the probability of the first match occurring in the ith
position of the list is p/n for every i, and the number of comparisons made by the algorithm
in such a situation is obviously i. In the case of an unsuccessful search, the number of
comparisons is n with the probability of such a search being (1- p). Therefore,

Investigation of the average-case efficiency is considerably more difficult than investigation

of the worst-case and best-case efficiencies. But there are many important algorithms for
which the average case efficiency is much better than the overly pessimistic worst-case
efficiency would lead us to believe. Note that average-case efficiency cannot be obtained by
taking the average of the worst-case and the best-case efficiencies.

Page|1.7
Summary of analysis framework
• Both time and space efficiencies are measured as functions of the algorithm's input size.
• Time efficiency is measured by counting the number of times the algorithm's basic
operation is executed. Space efficiency is measured by counting the number of extra
memory units consumed by the algorithm.
• The efficiencies of some algorithms may differ significantly for inputs of the same size.
For such algorithms, we need to distinguish between the worst-case, average-case, and
best-case efficiencies.
• The framework's primary interest lies in the order of growth of the algorithm's running
time (or extra memory units consumed) as its input size goes to infinity.

2. Performance Analysis

Space complexity
Total amount of computer memory required by an algorithm to complete its execution is
called as space complexity of that algorithm. The Space required by an algorithm is the sum
of following components
• A fixed part that is independent of the input and output. This includes memory space
for codes, variables, constants and so on.
• A variable part that depends on the input, output and recursion stack. ( We call these
parameters as instance characteristics)
Space requirement S(P) of an algorithm P, S(P) = c + Sp where c is a constant depends
on the fixed part, Sp is the instance characteristics
Example-1: Consider following algorithm abc()

Here fixed component depends on the size of a, b and c. Also instance characteristics Sp=0
Example-2: Let us consider the algorithm to find sum of array.
For the algorithm given here the problem instances are characterized by n, the number of
elements to be summed. The space needed by a[ ] depends on n. So the space complexity can
be written as; Ssum(n) ≥ (n+3) n for a[ ], One each for n, i and s.

Page|1.8
 Time complexity
Usually, the execution time or run-time of the program is refereed as its time complexity
denoted by tp (instance characteristics). This is the sum of the time taken to execute all
instructions in the program.
Exact estimation runtime is a complex task, as the number of instruction executed is
dependent on the input data. Also different instructions will take different time to execute. So
for the estimation of the time complexity we count only the number of p ogram steps.
A program step is loosely defined as syntactically or semantically meaning segment of the
program that has and execution time that is independent of instance characteristics. For
example comment has zero steps; assignment statement has one step and so on.
We can determine the steps eeded by a program to solve a particular problem instance in
two ways.
In the first method we introduce a new variable count to the program which is initialized to
zero. We also introduce statements to increment count by an appropriate amount into the
program. So when each time original program executes, the count also incremented by the
step count.
Example-1: Consider the algorithm sum( ). After the introduction of the count the program
will be as follows.

From the above we can estimate that invocation of sum( ) executes total number of 2n+3
steps.
The second method to determine the step count of an algorithm is to build a table in which
we list the total number of steps contributed by each statement. An example is shown below.

Page|1.9
Example-2: matrix addition

The above thod is both excessively difficult and, usually unnecessary. The thing to do is to
identify the most important operation of the algorithm, called the basic operation, the
operation contributing the m st to the total running time, and compute the number of times
the basic operation is executed.
Trade-off
There is often a time-space-tradeoff involved in a problem, that is, it cannot be solved with
few computing time and low memory consumption. One has to make a compromise and to
exchange computing time for memory consumption or vice versa, depending on which
algorithm one chooses and how one parameterizes it.

3. Asymptotic Notations
The efficiency analysis framework concentrates on the order of growth of an algorithm’s
basic operation count as the principal indicator of the algorithm’s efficiency. To compare and
rank such orders of growth, computer scientists use three notations: O(big oh), Ω(big
omega), Θ (big theta) and o(little oh)

Big-Oh notation
Definition: A function t(n) is said to be in O(g(n)), denoted t(n)∈O(g(n)), if t (n) is bounded
above by some constant multiple of g(n) for all large n, i.e., if there exist some positive
constant c and some nonnegative integer n0 such that
t(n) ≤ c g(n) for all n ≥ n0.

Page|1.10
Informally, O(g(n)) is the set of all functions with a lower or same order of growth as g(n)
Examples:

As another example, let us formally prove 100n + 5 ∈ O(n2)

100n + 5 ≤ 100n + n (for all n ≥ 5) = 101n ≤ 101n2. (c=101, n0=5)
Note that the definition gives us a lot of freedom in choosing specific values for constants c
and n0.

Example: To prove n2 + n = O(n3)

Strategies for Big-O Sometimes the easiest way to prove that f(n) = O(g(n)) is to take c to
be the sum of the positive coefficients of f(n). We can usually ignore the negative
coefficients.

Page|1.11
 Omega notation
Definition: A function t(n) is said to be in Ω(g(n)), denoted t(n)∈Ω(g(n)), if t(n) is bounded
below by some positive constant multiple of g(n) for all large n, i.e., if there exist some
positive constant c and some nonnegative integer n0 such that
t(n) ≥ c g(n) for all n ≥ n0.

Here is an example of the formal proof that n3 ∈Ω(n2):

n3 ≥ n2 for all n ≥ 0,
i.e., we can select c = 1 and n0 = 0.

Example:

Example: To prove n3 + 4n2 = Ω(n2)

Theta notation
A function t(n) is said to be in Θ(g(n)), denoted t(n) ∈ Θ(g(n)), if t (n) is bounded both above
and below by some positive constant multiples of g(n) for all large n, i.e., if there exist some
positive constants c1 and c2 and some nonnegative integer n0 such that
c2 g(n) S t(n) S c1g(n) for all n S n0.

Page|1.12
Example: n2 + 5n + 7 = Θ(n2)

Strategies for Ω and Θ

• Proving that a f(n) = Ω(g(n)) often requires more thought.
– Quite often, we have to pick c < 1.
– A good strategy is to pick a value of c which you think will work, and determine
which value of n0 is needed.
– Being able to do a little algebra helps.
– We can sometimes simplify by ignoring terms of f(n) with the positive
coefficients.
• The following theorem shows us that proving f(n) = Θ(g(n)) is nothing new:
Theorem: f(n) = Θ(g(n)) if and only if f(n) = O(g(n)) and f(n) = Ω(g(n)).
Thus, we just apply the previous two strategies.

Page|1.13
Page|1.14
3.4. Little Oh The function f(n) = o(g(n)) [ i.e f of n is a little oh of g of n ] if and only if
ƒ(n)
lim =0
n→œ g(n)

Example:

For comparing the order of growth limit is used

If the case-1 holds good in the above limit, we represent it by little-oh.

Page|1.15
3.5. Basic asymptotic Efficiency Classes
Class Name Comments

Page|1.16
 Mathematical Analysis of Non-recursive & Recursive Algorithms
Analysis of Non-recursive Algorithms
General Plan for Analyzing the Time Efficiency of Nonrecursive Algorithms
1. Decide on a parameter (or parameters) indicating an input’s size.
2. Identify the algorithm’s basic operation. (As a rule, it is located in innermost loop.)
3. Check whether the number of times the basic operation is executed depends only on
the size of an input. If it also depends on some additional property, the worst-case,
average-case, and, if necessary, best-case efficiencies have to be investigated
separately.
4. Set up a sum expressing the number of times the algorithm’s basic operation is
executed.
5. Using standard formulas and rules of sum manipulation, either find a closedform
formula for the count or, at the very least, establish its order of growth.
Example-1: To find maximum element in the given array
Algorithm

Here comparison is the basic operation.

Note that number of comparisions will be same for all arrays of size n. Therefore, no need to
distinguish worst, best and average cases.
Total number of basic operations (comparison) are,

Example-2: To check whether all the elements in the given array are distinct

Algorithm

Page|1.17
Here basic operation is comparison. The maximum no. of comparisons happen in the worst
case. (i.e. all the elements in the array are distinct and algorithms return true).
Total number of basic operations (comparison) in the worst case are,

1
Other than the worst case, the total comparisons are less than 2n2. ( For example if the first
two elements of the array are equal, only one comparison is computed). So in general C(n)
=O(n2)
Example-3: To perform matrix multiplication
Algorithm

Number of basic operations

(multiplications) is

Total running time:

Suppose if we take into account of addition; Algoritham also have same number of additions
A(n) = n3
Total running time:

Page|1.18
Example-4: To count the bits in the binary representation
Algorithm

The basic operation is count=count + 1 repeats no. of times

Analysis of Recursive Algorithms
General plan for analyzing the time efficiency of recursive algorithms
1. Decide on a parameter (or parameters) indicating an input’s size.
2. Identify the algorithm’s basic operation.
3. Check whether the number of times the basic operation is executed can vary on
different inputs of the same size; if it can, the worst-case, average-case, and best-case
efficiencies must be investigated separately. Set up a recurrence relation, with an
appropriate initial condition, for the number of times the basic operation is executed.
4. Solve the recurrence or, at least, ascertain the order of growth of its solution.

Example-1

Algorithm

Since the function F(n) is computed according to the formula

The number of multiplications M(n) needed to compute it must satisfy the equality

Page|1.19
Such equations are called recurrence Relations
Condition that makes the algorithm stop if n = 0 return 1. Thus recurrence relation and
initial condition for the algorithm’s number of multiplications M(n) can be stated as

We can use backward substitutions method to solve this

….

Example-2: Tower of Hanoi puzzle. In this puzzle, There are n disks of different sizes that
can slide onto any of three pegs. Initially, all the disks are on the first peg in order of size, the
largest on the bottom and the smallest on top. The goal is to move all the disks to the third
peg, using the second one as an auxiliary, if necessary. We can move only one disk at a time,
and it is forbidden to place a larger disk on top of a smaller one.
The problem has an elegant recursive solution, which is illustrated in Figure.
• To move n>1 disks from peg 1 to peg 3 (with peg 2 as auxiliary),
o we first move recursively n-1 disks from peg 1 to peg 2 (with peg 3 as auxiliary),
o then move the largest disk directly from peg 1 to peg 3, and,
o finally, move recursively n-1 disks from peg 2 to peg 3 (using peg 1 as auxiliary).
• If n = 1, we move the single disk directly from the source peg to the destination peg.

Figure: Recursive solution to the Tower of Hanoi puzzle

The number of moves M(n) depends only on n. The recurrence equation is

We have the following recurrence relation for the number of moves M(n):

Page|1.20
We solve this recurrence by t e same method of backward substitutions:

The pattern of the first three sums on the left suggests that the next one will be
24 M(n − 4) + 23 + 22 + 2 + 1, and generally, after i substitutions, we get

Since the initial condition is specified for n = 1, which is achieved for i = n - 1, we get the
following formula for the solution to recurrence,

Alternatively, by counting the number of nodes in the tree obtained by recursive calls, we
can get the total number of calls made by the Tower of Hanoi algorithm:

Figure: Tree of recursive calls made by the recursive algorithm for the Tower of Hanoi
puzzle.
Example-3

The recurrence relation can be written as

.
Also note that A(1) = 0.

Page|1.21
The standard approach to solving such a recurrence is to solve it only for n = 2k and then
take advantage of the theorem called the smoothness rule which claims that under very
broad assumptions the order of growth observed for n = 2k gives a correct answer about the
order of growth for all values of n.

Page|1.22
*****

Page|1.23