Analysis of Algorithms

Introduction
 Informally, an algorithm is any well-defined computational procedure that takes some value, or set
of values, as input and produces some value, or set of values, as output. An algorithm is thus a
sequence of computational steps that transform the input into the output.
 We can also view an algorithm as a tool for solving a well-specified computational problem. The
statement of the problem specifies in general terms the desired input/output relationship. The
algorithm describes a specific computational procedure for achieving that input/output relationship.
 For example, one might need to sort a sequence of numbers into nondecreasing order. This problem
arises frequently in practice and provides fertile ground for introducing many standard design
techniques and analysis tools. Here is how we formally define the sorting problem:
o Input: A sequence of n numbers <a1, a2... an>.
o Output: A permutation (reordering) of the input sequence such that a1<a2<... <an.
 For example, given the input sequence <31, 41, 59, 26, 41, 58>, a sorting algorithm returns as output
the sequence <26, 31, 41, 41, 58, 59>. Such an input sequence is called an instance of the sorting
problem. In general, an instance of a problem consists of the input (satisfying whatever constraints
are imposed in the problem statement) needed to compute a solution to the problem.
 Sorting is a fundamental operation in computer science (many programs use it as an intermediate
step), and as a result a large number of good sorting algorithms have been developed. Which
algorithm is best for a given application depends on-among other factors the number of items to be
sorted, the extent to which the items are already somewhat sorted, possible restrictions on the item
values, and the kind of storage device to be used: main memory, disks, or tapes.
 An algorithm can be specified in English, as a computer program, as a pseudo code or even as a
hardware design. The only requirement is that the specification must provide a precise description of
the computational procedure to be followed.

Characteristics of an Algorithm
 All algorithms must satisfy the following criteria
o Input. Zero or more quantities are externally supplied.
o Output. At least one quantity is produced.
o Definiteness. Each instruction is clear and unambiguous.
o Finiteness. If we trace out the instructions of an algorithm, then for all cases, the
algorithm terminates after a finite number of steps.
o Effectiveness. Every instruction must be very basic so that it can be carried out,
in principle, by a person using only pencil and paper. It is not enough that each
operation be definite as in criterion 3; it also must be feasible.
 Criteria 1 and 2 require that an algorithm produce one or more outputs and have zero or
more inputs that are externally supplied. According to criterion 3, each operation must
be definite, meaning that it must be perfectly clear what should be done. Directions
such as "add 6 or 7 to x" or "compute5/0" are not permitted because it is not clear
which of the two possibilities should be done or what the result is.
 Criteria 4 for algorithms are that they terminate after a finite number of operations. A
related consideration is that the time for termination should be reasonably short. For
example, an algorithm could be devised that decides whether any given position in the
game of chess is a winning position. The algorithm works by examining all possible
moves and countermoves that could be made from the starting position. The difficulty
with this algorithm is that even using the most modern computers, it may take billions
of years to make the decision. We must be very concerned with analyzing the efficiency
of each of our algorithms.
 Criterion 5 requires that each operation be effective; each step must be such that it
can, at least in principle, be done by a person using pencil and paper in a finite
amount of time. Performing arithmetic on integers is an example of an effective
operation, but arithmetic with real numbers is not, since some values may be
expressible only by infinitely long decimal expansion. Adding two such numbers would
violate the effectiveness property.

1|Page
Algorithms as a technology
 Suppose computers were infinitely fast and computer memory was free. Would you have any reason
to study algorithms? The answer is yes, if for no other reason than that you would still like to
demonstrate that your solution method terminates and does so with the correct answer.
 If computers were infinitely fast, any correct method for solving a problem would do. You would
probably want your implementation to be within the bounds of good software engineering practice
(i.e., well designed and documented), but you would most often use whichever method was the
easiest to implement.
 Of course, computers may be fast, but they are not infinitely fast. And memory may be cheap, but it
is not free. Computing time is therefore a bounded resource, and so is space in memory. These
resources should be used wisely, and algorithms that are efficient in terms of time or space will help
you do so.

Algorithms and other technologies

 The example shows that algorithms, like computer hardware, are a technology. Total system
performance depends on choosing efficient algorithms as much as on choosing fast hardware. Just as
rapid advances are being made in other computer technologies, they are being made in algorithms as
well.
 You might wonder whether algorithms are truly that important on contemporary computers in light
of other advanced technologies, such as
o hardware with high clock rates, pipelining, and superscalar architectures,
o easy-to-use, intuitive graphical user interfaces (GUIs),
o object-oriented systems, and
o local-area and wide-area networking.
 The answer is yes. Although there are some applications that do not explicitly require algorithmic
content at the application level (e.g., some simple web-based applications), most also require a
degree of algorithmic content on their own. For example, consider a web-based service that
determines how to travel from one location to another. Its implementation would rely on fast
hardware, a graphical user interface, wide-area networking, and also possibly on object orientation.
 However, it would also require algorithms for certain operations, such as finding routes (probably
using a shortest-path algorithm), rendering maps, and interpolating addresses.
 Moreover, even an application that does not require algorithmic content at the application level relies
heavily upon algorithms. Does the application rely on fast hardware? The hardware design used
algorithms. Does the application rely on graphical user interfaces? The design of any GUI relies on
algorithms. Does the application rely on networking? Routing in networks relies heavily on
algorithms. Was the application written in a language other than machine code? Then it was
processed by a compiler, interpreter, or assembler, all of which make extensive use of algorithms.
Algorithms are at the core of most technologies used in contemporary computers.
 Furthermore, with the ever-increasing capacities of computers, we use them to solve larger problems
than ever before. As we saw in the above comparison between insertion sort and merge sort, it is at
larger problem sizes that the differences in efficiencies between algorithms become particularly
prominent.
 Having a solid base of algorithmic knowledge and technique is one characteristic that separates the
truly skilled programmers from the novices. With modern computing technology, you can
accomplish some tasks without knowing much about algorithms, but with a good background in
algorithms, you can do much, much more.
2|Page
Algorithm Analysis
 Why need algorithm analysis?
o writing a working program is not good enough
o The program may be inefficient!
o If the program is run on a large data set, then the running time becomes an issue
 We only analyze correct algorithms, An algorithm is said to be correct if, for every input instance, it
halts with the correct output. We say that a correct algorithm solves the given computational
problem. An incorrect algorithm might not halt at all on some input instances, or it might halt with an
answer other than the desired one. Contrary to what one might expect, incorrect algorithms can
sometimes be useful, if their error rate can be controlled.
 Analyzing an algorithm means predicting the resources that the algorithm requires
o Resources include
 Memory
 Communication bandwidth
 Computational time (usually most important)
 Factors affecting the running time
o Computer
o Compiler
o Algorithm used
o Input to the algorithm
 The content of the input affects the running time
 typically, the input size (number of items in the input) is the main consideration
 E.g. sorting problem -> the number of items to be sorted
 E.g. multiply two matrices together -> the total number of elements in the two
matrices
 Machine model assumed
o Instructions are executed one after another, with no concurrent operations -> Not parallel
computers

Example: Selection Problem

 Given a list of N numbers, determine the kth largest, where k <= N.
 Algorithm 1:
o Read N numbers into an array
o Sort the array in decreasing order by some simple algorithm
o Return the element in position k
 Algorithm 2:
o Read the first k elements into an array and sort them in decreasing order
o Each remaining element is read one by one
 If smaller than the kth element, then it is ignored
 Otherwise, it is placed in its correct spot in the array, bumping one element out of the
array.
o The element in the kth position is returned as the answer.
 Which algorithm is better when
o N =100 and k = 100? Or N =100 and k = 1?
 What happens when N = 1,000,000 and k = 500,000?
 There exist better algorithms
3|Page
Efficiency
 Algorithms devised to solve the same problem often differ dramatically in their efficiency. These
differences can be much more significant than differences due to hardware and software.
 As an example, we will see two algorithms for sorting. The first, known as insertion sort, takes time
roughly equal to c1n2 to sort n items, where c1 is a constant that does not depend on n. That is, it
takes time roughly proportional to n2. The second, merge sort, takes time roughly equal to c2n lg n,
where lg n stands for log2 n and c2 is another constant that also does not depend on n. Insertion sort
usually has a smaller constant factor than merge sort, so that c1 < c2. We shall see that the constant
factors can be far less significant in the running time than the dependence on the input size n. Where
merge sort has a factor of lg n in its running time, insertion sort has a factor of n, which is much
larger. Although insertion sort is usually faster than merge sort for small input sizes, once the input
size n becomes large enough, merge sort's advantage of lg n vs. n will more than compensate for the
difference in constant factors. No matter how much smaller c1 is than c2, there will always be a
crossover point beyond which merge sort is faster.
 For a concrete example, let us pit a faster computer (computer A) running insertion sort against a
slower computer (computer B) running merge sort. They each must sort an array of one million
numbers. Suppose that computer A executes one billion instructions per second and computer B
executes only ten million instructions per second, so that computer A is 100 times faster than
computer B in raw computing power.
 To make the difference even more dramatic, suppose that the world's craftiest programmer codes
insertion sort in machine language for computer A, and the resulting code requires 2n2 instructions to
sort n numbers. (Here, c1 = 2.) Merge sort, on the other hand, is programmed for computer B by an
average programmer using a high-level language with an inefficient compiler, with the resulting code
taking 50n lg n instructions (so that c2 = 50).
 To sort one million numbers, computer A takes

while computer B takes

 By using an algorithm whose running time grows more slowly, even with a poor compiler, computer
B runs 20 times faster than computer A! The advantage of merge sort is even more pronounced when
we sort ten million numbers: where insertion sort takes approximately 2.3 days, merge sort takes less
than 20 minutes. In general, as the problem size increases, so does the relative advantage of merge
sort.

Pseudocode v/s Real Code

 Here we shall typically describe algorithms as programs written in a pseudocode that is similar in
many respects to C, Pascal, or Java.
 What separates pseudocode from "real" code is that in pseudocode, we employ whatever expressive
method is most clear and concise to specify a given algorithm. Sometimes, the clearest method is
English, so do not be surprised if you come across an English phrase or sentence embedded within a
section of "real" code.
 Another difference between pseudocode and real code is that pseudocode is not typically concerned
with issues of software engineering. Issues of data abstraction, modularity, and error handling are
often ignored in order to convey the essence of the algorithm more concisely.
4|Page
What kinds of problems are solved by algorithms?
 Sorting is by no means the only computational problem for which algorithms have been developed.
Practical applications of algorithms are ubiquitous and include the following examples:
o The Human Genome Project has the goals of identifying all the 100,000 genes in human
DNA, determining the sequences of the 3 billion chemical base pairs that make up human
DNA, storing this information in databases, and developing tools for data analysis. Each of
these steps requires sophisticated algorithms. While the solutions to the various problems
involved are beyond the scope of this book, ideas from many of the chapters in this book are
used in the solution of these biological problems, thereby enabling scientists to accomplish
tasks while using resources efficiently. The savings are in time, both human and machine,
and in money, as more information can be extracted from laboratory techniques.
o The Internet enables people all around the world to quickly access and retrieve large amounts
of information. In order to do so, clever algorithms are employed to manage and manipulate
this large volume of data. Examples of problems which must be solved include finding good
routes on which the data will travel, and using a search engine to quickly find pages on which
particular information resides.
o Electronic commerce enables goods and services to be negotiated and exchanged
electronically. The ability to keep information such as credit card numbers, passwords, and
bank statements private is essential if electronic commerce is to be used widely. Public-key
cryptography and digital signatures are among the core technologies used and are based on
numerical algorithms and number theory.
o In manufacturing and other commercial settings, it is often important to allocate scarce
resources in the most beneficial way. An oil company may wish to know where to place its
wells in order to maximize its expected profit. A candidate for the presidency of the United
States may want to determine where to spend money buying campaign advertising in order to
maximize the chances of winning an election. An airline may wish to assign crews to flights
in the least expensive way possible, making sure that each flight is covered and that
government regulations regarding crew scheduling are met. An Internet service provider may
wish to determine where to place additional resources in order to serve its customers more
effectively. All of these are examples of problems that can be solved using linear
programming.

5|Page