CS8451 – DESIGN AND ANALYSIS OF ALGORITHMS

IV SEM B.E CSE

ANNA UNIVERSITY CHENNAI
REGULATION: 2017

UNIT – 1
INTRODUCTION

Dr.P.GaneshKumar
Dept. of Information Technology
Anna University Regional Campus, Coimbatore
 Notion of an Algorithm
• An algorithm is a sequence of unambiguous
instructions for solving a problem.
– i.e., for obtaining a required output for any
legitimate input in a finite amount of time.
 Key points about an Algorithm
• The non-ambiguity requirement for each step of an
algorithm cannot be compromised.
• The range of inputs for which an algorithm works has to
be specified carefully.
• The same algorithm can be represented in several
different ways.
• There may exist several algorithms for solving the same
problem.
• Algorithms for the same problem can be based on very
different ideas and can solve the problem with
dramatically different speeds.
 Computing the Greatest
Common Divisor of Two Integers
• The greatest common divisor of two nonnegative,
not-both-zero integers m and n, denoted gcd(m,
n), is defined as the largest integer that divides
both m and n evenly, i.e., with a remainder of zero.
– Euclid’s algorithm
– Consecutive integer checking algorithm
– Middle-school procedure
– Incorporation of sieve of Eratosthenes into the middle-
school procedure
Fundamentals of Algorithmic Problem Solving
• Understanding the Problem
• Ascertaining the Capabilities of the Computational
Device
• Choosing between Exact and Approximate Problem
Solving
• Algorithm Design Techniques
• Designing an Algorithm and Data Structures
• Methods of Specifying an Algorithm
• Proving an Algorithm’s Correctness
• Analyzing an Algorithm
• Coding an Algorithm
Algorithm design and analysis process
 Understanding the Problem
• Read the problem’s description carefully and ask questions if you
have any doubts about the problem.
• Do a few small examples by hand, think about special cases, and ask
questions again if needed.
• An input to an algorithm specifies an instance of the problem the
algorithm solves. It is very important to specify exactly the set of
instances the algorithm needs to handle.
• If you fail to do this, your algorithm may work correctly for a
majority of inputs but crash on some “boundary” value. Remember
that a correct algorithm is not one that works most of the time, but
one that works correctly for all legitimate inputs.
 Ascertaining the Capabilities of the
Computational Device
• The vast majority of algorithms in use today are still destined to be
programmed for a computer closely resembling the von Neumann
machine.
• The essence of this architecture is captured by the so-called random-
access machine (RAM).Its central assumption is that instructions are
executed one after another, one operation at a time. Accordingly,
algorithms designed to be executed on such machines are called
sequential algorithms.
• The central assumption of the RAM model does not hold for some newer
computers that can execute operations concurrently, i.e., in parallel.
Algorithms that take advantage of this capability are called parallel
algorithms.
• There are important problems, however, that are very complex by their
nature, or have to process huge volumes of data, or deal with applications
where the time is critical. In such situations, it is imperative to be aware
of the speed and memory available on a particular computer system.
 Choosing between Exact and Approximate
Problem Solving
• Principal decision while designing an algorithm is to choose between
solving the problem exactly or solving it approximately. In the former case,
an algorithm is called an exact algorithm; in the latter case, an algorithm is
called an approximation algorithm.
• First, there are important problems that simply cannot be solved exactly
for most of their instances; examples include extracting square roots,
solving nonlinear equations, and evaluating definite integrals.
• Second, available algorithms for solving a problem exactly can be
unacceptably slow because of the problem’s intrinsic complexity. This
happens, in particular, for many problems involving a very large number of
choices; you will see examples of such difficult problems
• Third, an approximation algorithm can be a part of a more sophisticated
algorithm that solves a problem exactly.
 Algorithm Design Techniques
• An algorithm design technique (or “strategy” or “paradigm”) is a general
approach to solving problems algorithmically that is applicable to a variety of
problems from different areas of computing.
– Brute force
– Exhaustive search
– Decrease and Conquer
– Divide and Conquer
– Transform and Conquer
– Hashing
– Backtracking
– Dynamic Programming
– Greedy Technique
– Iterative Improvement
– Branch and Bound
• Algorithm design techniques make it possible to classify algorithms according to
an underlying design idea; therefore, they can serve as a natural way to both
categorize and study algorithms.
 Designing an Algorithm and Data Structures
• While the algorithm design techniques do provide a powerful set
of general approaches to algorithmic problem solving, designing
an algorithm for a particular problem may still be a challenging
task.
• Some design techniques can be simply inapplicable to the
problem in question.
• Sometimes, several techniques need to be combined, and there
are algorithms that are hard to pinpoint as applications of the
known design techniques.
• Even when a particular design technique is applicable, getting an
algorithm often requires a nontrivial ingenuity on the part of the
algorithm designer.
• With practice, both tasks—choosing among the general
techniques and applying them—get easier, but they are rarely
easy.
 Designing an Algorithm and Data Structures

• One should pay close attention to choosing data structures

appropriate for the operations performed by the algorithm.
• Some of the algorithm design techniques depend intimately on
structuring or restructuring data specifying a problem’s instance.
• Many years ago, an influential textbook proclaimed the
fundamental importance of both algorithms and data structures for
computer programming by its very title: Algorithms + Data
Structures = Programs.
• In the new world of object-oriented programming, data structures
remain crucially important for both design and analysis of
algorithms.
 Methods of Specifying an Algorithm
• Once you have designed an algorithm, you need to specify it in some fashion.

• Describing algorithms in words (natural language) and in pseudo code (mixture

of a natural language and programming language like constructs) are the two
options that are most widely used nowadays for specifying algorithms.

• In the earlier days of computing, the dominant vehicle for specifying algorithms
was a flowchart, a method of expressing an algorithm by a collection of
connected geometric shapes containing descriptions of the algorithm’s steps.

• This representation technique has proved to be inconvenient for all but very
simple algorithms; nowadays, it can be found only in old algorithm books.

• The state of the art of computing has not yet reached a point where an
algorithm’s description—be it in a natural language or pseudocode—can be fed
into an electronic computer directly. Instead, it needs to be converted into a
computer program written in a particular computer language.
 Proving an Algorithm’s Correctness
• Once an algorithm has been specified, you have to
prove its correctness, i.e., prove that the algorithm
yields a required result for every legitimate input in
a finite amount of time.
• For some algorithms, a proof of correctness is
quite easy; for others, it can be quite complex.
• A common technique for proving correctness is to
use mathematical induction because an
algorithm’s iterations provide a natural sequence
of steps needed for such proofs.
 Analyzing an Algorithm
• There are two kinds of algorithm efficiency: time efficiency,
indicating how fast the algorithm runs, and space efficiency,
indicating how much extra memory it uses.
• Another desirable characteristic of an algorithm is simplicity.
Unlike efficiency, which can be precisely defined and investigated
with mathematical rigor, simplicity, like beauty, is to a
considerable degree in the eye of the beholder.
• Simpler algorithms are easier to understand and easier to
program; consequently, the resulting programs usually contain
fewer bugs.
• Yet another desirable characteristic of an algorithm is generality.
There are, in fact, two issues here: generality of the problem the
algorithm solves and the set of inputs it accepts.
• If you are not satisfied with the algorithm’s efficiency, simplicity,
or generality, you must return to the drawing board and redesign
the algorithm.
 Coding an Algorithm
• Most algorithms are destined to be ultimately implemented
as computer programs. Programming an algorithm presents
both a peril and an opportunity. The peril lies in the
possibility of making the transition from an algorithm to a
program either incorrectly or very inefficiently.

• Some influential computer scientists strongly believe that

unless the correctness of a computer program is proven
with full mathematical rigor, the program cannot be
considered correct.

• They have developed special techniques for doing such

proofs, but the power of these techniques of formal
verification is limited so far to very small programs.
• Sorting Important Problem Types
– The sorting problem is to rearrange the items of a given list in non-decreasing order.
• Searching
– The searching problem deals with finding a given value, called a search key, in a given set
• String processing
– A string is a sequence of characters from an alphabet. Strings of particular interest are text strings, which
comprise letters, numbers, and special characters; bit strings, which comprise zeros and ones; and gene
sequences, which can be modeled by strings of characters from the four-character alphabet {A,C, G, T}.
• Graph problems
– A graph can be thought of as a collection of points called vertices, some of which are connected by line segments
called edges. Route planning, circuit board and VLSI chip fabrication, X-ray crystallography, and genetic
engineering.
• Combinatorial problems
– Problems that ask, explicitly or implicitly, to find a combinatorial object—such as a permutation, a combination,
or a subset—that satisfies certain constraints. Most of the graph problems may come under combinatorial
problems. The traveling salesman problem (TSP) is the problem of finding the shortest tour through n cities that
visits every city exactly once. The graph-coloring problem seeks to assign the smallest number of colors to the
vertices of a graph so that no two adjacent vertices are the same color.
• Geometric problems
– Geometric algorithms deal with geometric objects such as points, lines, and polygons The closest-pair problem is
self-explanatory: given n points in the plane, find the closest pair among them. The convex-hull problem asks to
find the smallest convex polygon that would include all the points of a given set.
• Numerical problem
– Numerical problems, another large special area of applications, are problems that involve mathematical objects
of continuous nature: solving equations and systems of equations, computing definite integrals, evaluating
functions, and so on.
 Fundamentals of Analysis of
Algorithm Efficiency
• Investigation of an algorithm’s efficiency with
respect to two resources
– Running time
– Memory space
• Time efficiency or Time Complexity
– Indicates how fast an algorithm in question runs.
• Space efficiency or Space Complexity
– Refers to the amount of memory units required by the
algorithm in addition to the space needed for its input
and output
 Analysis Framework

• Measuring an Input’s size.

• Units for measuring running time.

• Orders of growth.
 Measuring an Input’s Size
• Any algorithm run longer on larger inputs.
– e.g. Sort larger arrays, Multiply larger
matrices and so on.
• Investigate an algorithm’s efficiency as a
function of some parameter ‘n’ indicating
the algorithm’s input size.
• Straightforward to select such parameter.
– Size of the list
• Sorting, Searching, finding smallest element
 Measuring an Input’s Size
• Polynomial evaluation
p( x)  an x n  ...  a0
– Polynomial degree or number of its coefficients.
• Matrix Operation
– Order of matrix ‘n’
– Total number of elements in the matrix ‘N’
• Graph Operation
– Requires more than one parameter to indicate the
size of their inputs.
• Number of vertices, number of edges
 Measuring an Input’s Size
• Size metric
– Influenced by the operation of the algorithm
– e.g. Spell-check Algorithm
• Algorithm checking
– Individual character
– Individual word

• Algorithm involving properties of numbers.

– e.g. checking whether a given integer ‘n’ is prime
– Measuring size
• Number of b bits in the n’s binary representation
b  log 2 n   1
Units for Measuring Running Time
• Use standard unit of time measurement
– Second or millisecond
• Drawbacks
– Dependence on the
• Speed of a particular computer
• Quality of a program implementing the algorithm
• Quality of the compiler used in generating the
machine code
– Difficulty of clocking the actual running time of
the program.
 Units for Measuring Running Time
• A metric that does not depend on these
extraneous factors.
• One possible way
– Count the number of times each of the
algorithm’s operation is executed.
– Excessively difficulty and usually unnecessary.
• Alternative way
– Identify the most important operation of the
algorithm
• Basic operation
 Units for Measuring Running Time
• Basic operation
– Operation contributing the most to the total
running time.
• Compute the number of times the basic
operation is executed.
• Identification of basic operation
– Most time consuming operation
– Sorting algorithm
• Key comparison
– Matrix Multiplication and Polynomial Evaluation
• Multiplication
 Units for Measuring Running Time
• Frame work
– Count the number of times the algorithm’s basic
operation is executed on inputs of size ‘n’.
• Let
– Cop time of execution of an algorithms basic
operation on a particular computer.
– C (n)  number of times the operation needs to
be executed for the algorithm.
– T (n)  running time of a program implementing
the algorithm.
Units for Measuring Running Time
• Frame work

– ignores multiplicative constants and

concentrates on the count’s order of growth to
within a constant multiple for large-size inputs
 Orders of Growth
• Small sized inputs are not considered to
distinguish the efficiency of algorithm.
• For large values of n, it is the function’s
order of growth that counts.
 Orders of Growth
• Slowest growing function
– Logarithmic function
• Fastest growing function
– Exponential function
– Factorial function
• Qualitative difference among the orders of growth of the
functions
– a twofold increase in the value of n
• log2 n increases in value by just 1
• linear function increases twofold,
• linearithmic function n log2 n increases slightly more than twofold;
• quadratic function n2 increases fourfold
• cubic function n3 increases and eightfold
• 2n gets squared
• n! increases much more than that
 Worst-Case, Best-Case, and
Average-Case Efficiencies

• Many algorithms for which running time

depends not only on an input size but also
on the specifics of a particular input.
 Worst-Case Efficiency

• Efficiency for the worst-case input of size n

for which the algorithm runs the longest
among all possible inputs of that size.

C worst (n)  n
 Best-Case Efficiency

• Efficiency for the best-case input of size n

for which the algorithm runs the fastest
among all possible inputs of that size.

Cbest (n)  1
 Average-Case Efficiency
• neither the worst-case analysis nor its best-
case counterpart yields the necessary
information about an algorithm’s behavior on
a “typical” or “random” input.
• Standard assumptions
– the probability of a successful search is equal to p
(0 ≤ p ≤ 1).
– the probability of the first match occurring in the ith
position of the list is the same for every i.
 Average-Case Efficiency
• Successful Search
– the probability of the first match occurring in
the ith position of the list is p/n for every i.
– the number of comparisons made by the
algorithm in such a situation is obviously i
• Unsuccessful Search
– the number of comparisons will be n with the
probability of such a search being (1 − p)
 Average-Case Efficiency

• Successful search
– if p = 1 the average number of key comparisons made by
sequential search is (n + 1)/2.
• Unsuccessful search
– If p = 0 the average number of key comparisons will be n because
the algorithm will inspect all n elements on all such inputs
 Amortized Efficiency
• American computer scientist Robert Tarjan.

• applies not to a single run of an algorithm but rather to a

sequence of operations performed on the same data
structure.

• In some situations a single operation can be expensive, but

the total time for an entire sequence of n such operations is
always significantly better than the worst-case efficiency of
that single operation multiplied by n.

• “amortize” the high cost of such a worst-case occurrence over

the entire sequence in a manner similar to the way a business
would amortize the cost of an expensive item over the years
of the item’s productive life.
 Recapitulation of the Analysis
Framework
• Both time and space efficiencies are measured as functions of the
algorithm’s input size.

• Time efficiency is measured by counting the number of times the

algorithm’s basic operation is executed. Space efficiency is
measured by counting the number of extra memory units consumed
by the algorithm.

• The efficiencies of some algorithms may differ significantly for

inputs of the same size. For such algorithms, we need to distinguish
between the worst-case, average-case, and best-case efficiencies.

• The framework’s primary interest lies in the order of growth of the

algorithm’s running time (extra memory units consumed) as its input
size goes to infinity
 Asymptotic Notations
• Principal indicator of the algorithm’s efficiency.
– Order of growth of an algorithm’s basic operation
count.
• To compare and rank such orders of growth
– O(big oh),
– Ω(big omega),
– ϴ(big theta).
 Asymptotic Notations
• t(n) and g(n) can be any nonnegative
functions defined on the set of natural
numbers.
• t (n) will be an algorithm’s running time.
– indicated by its basic operation count C(n)
• g(n) will be some simple function to
compare the count with.
 Informal Introduction
• O(g(n))  (big oh)
– set of all functions with a lower or same order
of growth as g(n).

Linear  Lower Quadratic Same

Fourth Order Polynomial  High

Cubic Higher
 Informal Introduction
• Ω(g(n))  (big omega)
– set of all functions with a higher or same order
of growth as g(n).

• ϴ((g(n))  (big theta)

– set of all functions with the same order of growth as
g(n).
– quadratic function
 O-notation
• A function t (n) is said to be in O(g(n))
– t (n) ∈ O(g(n))
• if t (n) is bounded above by some constant
multiple of g(n) for all large n
– if there exist some positive constant c and
some nonnegative integer n0 such that
• t (n) ≤ cg(n) for all n ≥ n0
O-notation
O-notation
 Ω-notation
• A function t (n) is said to be in (g(n)),
– t (n) ∈ (g(n))
• if t (n) is bounded below by some positive
constant multiple of g(n) for all large n
– if there exist some positive constant c and
some nonnegative integer n0 such that
• t (n) ≥ cg(n) for all n ≥ n0
Ω-notation
 ϴ-notation
• A function t (n) is said to be in (g(n)),
– t (n) ∈ (g(n))
• if t (n) is bounded both above and below
by some positive constant multiples of g(n)
for all large n
– if there exist some positive constants c1 and
c2 and some nonnegative integer n0 such that
• c2g(n) ≤ t (n) ≤ c1g(n) for all n ≥ n0
ϴ-notation
ϴ-notation
 Mathematical Analysis of
Nonrecursive Algorithms
• Problem of finding the value of the largest
element in a list of n numbers.
 Problem of finding the value of the
largest element in a list of n numbers
• Measure of an input’s size
– number of elements in the array  n
• Operation most executed
– Two operation inside the for loop.
• the comparison A[i]> maxval
• the assignment maxval←A[i]
– Basic Operation  the comparison
• number of comparisons  same for all arrays of size n
• no need  worst, average, and best cases
 Problem of finding the value of the
largest element in a list of n numbers
 General Plan for Analyzing the Time
Efficiency of Nonrecursive Algorithms
• Decide on a parameter (or parameters) indicating an input’s size.
• Identify the algorithm’s basic operation. (As a rule, it is located in
the innermost loop.)
• Check whether the number of times the basic operation is
executed depends only on the size of an input. If it also depends on
some additional property, the worst-case, average-case, and, if
necessary, best-case efficiencies have to be investigated separately.
• Set up a sum expressing the number of times the algorithm’s basic
operation is executed
• Using standard formulas and rules of sum manipulation, either find
a closed form formula for the count or, at the very least, establish
its order of growth
 General Plan for Analyzing the Time
Efficiency of Non-recursive Algorithms
Element Uniqueness Problem
 Element Uniqueness Problem
• Measure of an input’s size
– number of elements in the array  n
• Basic Operation
– Innermost loop  single operation
• Comparison.
• the number of element comparisons depends
not only on n but also on whether there are
equal elements in the array and, if there are,
which array positions they occupy
 Element Uniqueness Problem
• Investigation on the worst case
– the worst case input is an array for which the
number of element comparisons C (n) is the
worst

largest among all arrays of size n.

– two kinds of worst-case inputs
• Inputs for which the algorithm does not exit the loop
prematurely.
• arrays with no equal elements and arrays in which the
last two elements are the only pair of equal elements
Element Uniqueness Problem
Matrix Multiplication
Matrix Multiplication
 Matrix Multiplication
• Input’s size  matrix order n
• operations in the innermost loop
– multiplication and addition
• well-established tradition
– multiplication as the basic operation.
• Sum for the total number of multiplications
M(n) executed by the algorithm
Matrix Multiplication
Matrix Multiplication
Mathematical Analysis of Recursive
Algorithms
 General Plan for Analyzing the Time
Efficiency of Recursive Algorithms
• Decide on a parameter (or parameters) indicating an input’s
size.
• Identify the algorithm’s basic operation.
• Check whether the number of times the basic operation is
executed can vary on different inputs of the same size; if it
can, the worst-case, average-case, and best-case
efficiencies must be investigated separately.
• Set up a recurrence relation, with an appropriate initial
condition, for the number of times the basic operation is
executed.
• Solve the recurrence or, at least, ascertain the order of
growth of its solution.
 Empirical Analysis of Algorithms
• 1. Understand the experiment’s purpose.
• 2. Decide on the efficiency metric M to be measured
and the measurement unit (an operation count vs. a
time unit).
• 3. Decide on characteristics of the input sample (its
range, size, and so on).
• 4. Prepare a program implementing the algorithm (or
algorithms) for the experimentation.
• 5. Generate a sample of inputs.
• 6. Run the algorithm (or algorithms) on the sample’s
inputs and record the data observed.
• 7. Analyze the data obtained.
 Algorithm visualization
• In addition to the mathematical and empirical analyses of algorithms, there is
yet a third way to study algorithms. It is called algorithm visualization and can
be defined as the use of images to convey some useful information about
algorithms.
• That information can be a visual illustration of an algorithm’s operation, of its
performance on different kinds of inputs, or of its execution speed versus that
of other algorithms for the same problem.
• To accomplish this goal, an algorithm visualization uses graphic elements—
points, line segments, two- or three-dimensional bars, and so on—to represent
some “interesting events” in the algorithm’s operation.
• There are two principal variations of algorithm visualization:
– Static algorithm visualization
– Dynamic algorithm visualization, also called algorithm animation
• Static algorithm visualization shows an algorithm’s progress through a series
• of still images.
• Algorithm animation, on the other hand, shows a continuous, movie-like
presentation of an algorithm’s operations. Animation is an arguably more
sophisticated option, which, of course, is much more difficult to implement.
 Reference
• All text, figures, tables, equation , images and so on are
taken from the below mentioned book that is prescribed
as one of the text book by Anna University Chennai for IV
Sem B.E CSE in the course CS8451 – Design And Analysis
Of Algorithms in Regulation: 2017.

• Anany Levitin, ―Introduction to the Design and Analysis

of Algorithms‖, Third Edition, Pearson Education, 2012.