Algorithm Analysis and Design

Algorithm Analysis and Design (R 606)
Department of Computer Science & Engineering
SJCET, Palai
Algorithm Analysis and Design (R 606) ALGORITHM ANALYSIS AND DESIGN R606 3+1+0
Module 1
Introduction and Complexity
What is an algorithm Properties of an Algorithm, Difference between Algorithm, Computational Procedure and Program, Study of Algorithms; Pseudo-code Conventions; Recursive Algorithms Space and Time Complexity Asymptotic Notations Oh, Omega, Theta, Common Complexity Functions; Recurrence Relations and Recurrence Trees for Complexity Calculations; Profiling. Deterministic and non - deterministic algorithms.
Module 2
Divide and Conquer
Control Abstraction, Finding Maximum and Minimum, Binary Search, Divide and Conquer Matrix Multiplication, Stressens Matrix Multiplication, Merge Sort, Quick Sort.
Module 3
Greedy Strategy
Control Abstraction, General Knapsack Problem, Optimal Storage on Tapes, Minimum Cost Spanning Trees Prims Algorithm, Kruskals Algorithm Job sequencing with deadlines. Module 4 Dynamic Programming
Principle of Optimality, Multi-stage Graph, All-Pairs Shortest Paths, Travelling Salesman Problem. Lower Bound Theory - Comparison Trees for Searching and Sorting, Oracles and Adversary Arguments Merging, Insertion & Selection Sort; Selection of kth Smallest Element.
Module 5
Backtracking
Control Abstraction - Bounding Functions, Control Abstraction, N-Queens Problem, Sum of Subsets, Knapsack problem. Branch and Bound Techniques FIFO, LIFO, and LC Control Abstractions, 15puzzle, Travelling Salesman Problem.
SJCET, Palai

Text Book
1.
Fundamentals of Computer Algorithms - Horowitz and Sahni, Galgotia
References
1.
Computer Algorithms Introduction to Design and Analysis - Sara Baase & Allen Van Gelder, Pearson Education Data Structures algorithms and applications - Sahni, Tata McGrHill Foundations of Algorithms - Richard Neapolitan, Kumarss N., DC Hearth & Company Introduction to algorithm- Thomas Coremen, Charles, Ronald Rivest -PHI
2. 3. 4.
SJCET, Palai
TABLE OF CONTENTS
Module 1
What is an algorithm Properties of an Algorithm ................ .. Difference between Algorithm Computational Procedure and Program ...................... Study of Algorithms........................................................................ Pseudocode convention.. Recursive Algorithms ........................ Space Complexity .................. Time Complexity............................................................................. Asymptotic Notations ....... Oh...................................... Omega........................................................................................... ......... Theta............................................................................................ .......... Common Complexity Functions.... Recurrence Relations....................................................................... Recurrence Trees for Complexity Calculations................................ Profiling............................... Deterministic and non - deterministic algorithms
7 7
8 10 10 12 19 20 25 26 28 30 33 35 39 41 41
SJCET, Palai
Module 2 Control Abstraction . Finding Maximum and Minimum ... Binary Search ........ Divide and Conquer Matrix Multiplication ................. Stressens Matrix Multiplication .. Merge Sort ... Quick Sort ....
44 45 51 54 56 60 69
Module 3 Greedy Strategy Control Abstraction General Knapsack Problem Optimal Storage on Tapes Minimum Cost Spanning Trees ... Prims Algorithm .............. Kruskals Algorithm ... 87 88 90 91 93 99
Job sequencing with deadlines.
100
Module 4
Dynamic Programming Principle of Optimality Multi-stage Graph .......
105 106
SJCET, Palai

All-Pairs Shortest Paths .. 118
Lower Bound Theory
Comparison Trees for Searching and Sorting .................................. Oracles and Adversary Arguments.................................................. Merging............................................................... Insertion................................................................... Selection Sort............................................................... Selection of kth Smallest Element.......................
127 134 135 136 142 144
Module 5 Backtracking Control Abstraction .... Bounding Functions ....... Control Abstraction .. N-Queens Problem.......................................................................... Sum of Subsets ....................................................................... Knapsack problem......................................................... Branch and Bound Techniques FIFO ........................................................................................... LIFO........................................................................................... ............ LC Control Abstractions..................................................................... 15-puzzle.................................................................................... .......... Travelling Salesman Problem........................................................ 172 177 181 182 184
149 153 149 153 158 164
SJCET, Palai
MODULE 1
SJCET, Palai
WHAT IS AN ALGORITHM
Definition: An algorithm is a finite set of instructions that, if followed, accomplishes a particular task.
PROPERTIES OF AN ALGORITHM
All algorithms must satisfy the following criteria: 1. Input. Zero or more quantities are externally supplied. 2. Output. At least one quantity is produced. 3. Definiteness. Each instruction is clear and unambiguous. 4. Finiteness. If we trace out the instructions of an algorithm, then for all cases, the algorithm terminates after a finite number of steps. 5. Effectiveness. Every instruction must be very basic so that it can be carried out, in principle, by a person using only pencil and paper. It is not enough that each operation be definite as in criterion 3; it also must be feasible. An algorithm is composed of a finite set of steps, each of which may require one or more operations. The possibility of a computer carrying out these operations necessitates that certain constraints be placed on the type of operations an algorithm can include. Criteria 1 and 2 require that an algorithm produce one or more outputs and have zero or more inputs that are externally supplied. According to criteria 3, each operation must be definite, meaning that it must be perfectly clear what should be done. The fourth criterion for algorithms is that they terminate after a finite number of operations. A related consideration is that the time for termination should be reasonably short. Criteria 5 requires that each operation be effective; each step must be such that it can, at least in principle, be done by a person using pencil and paper in a finite amount of time. Performing arithmetic on integers is an example of an effective operation, but arithmetic with real numbers is not, since some values may be expressible only by infinitely long decimal expansion.
SJCET, Palai
DIFFERENCE BETWEEN ALGORITHM, COMPUTATIONAL PROCEDURE AND PROGRAM

COMPUTATIONAL PROCEDURE Algorithms that are definite and effective are also called computational procedures. One important example of computational procedures is the operating system of a digital computer. This procedure is designed to control the execution of jobs, in such a way that when no jobs are available, it does not terminate but continues in a waiting state until a new job is entered. PROGRAM To help us achieve the criterion of definiteness, algorithms are written in a programming language. Such languages are designed so that each legitimate sentence has a unique meaning. A program is the expression of an algorithm in a programming language. Sometimes words such as procedure, function and subroutine are used synonymously for program. The study of algorithms includes many important and active areas of research. There are four distinct areas of study: 1. How to devise algorithms: - Creating an algorithm is an art which may never be fully automated. There are several techniques with which you can devise new and useful algorithms. Dynamic programming is one such technique. Some of the techniques are especially useful in fields other than computer science such as operations research and electrical engineering. 2. How to validate algorithms: - Once an algorithm is devised, it is necessary to show that it computes the correct answer for all possible legal inputs. This process is referred to as algorithm validation. It is sufficient to state the algorithm in any precise way and need not be expressed as a program. The purpose of validation is to assure us that this algorithm will work correctly independently of the issues concerning the programming language it will eventually be written in. once the validity of the method has been shown, a program can be written and a second phase begins. This phase is referred to as program proving or program verification. A proof of correctness requires that Department of Computer Science & Engineering SJCET, Palai
10
the solution be stated in two forms. One form is usually as a program which is annotated by a set of assertions about the input and output variables of the program. These assertions are often expressed in predicate calculus. The second form is called a specification, and this may also be expressed in the predicate calculus. A proof consists of showing that these two forms are equivalent in that for every given legal input, they describe the same output. A complete proof of program correctness requires that each statement of the programming language be precisely defined and all basic operations be proved correct. 3. How to analyze algorithms: - this field of study is called analysis of algorithms. As an algorithm is executed, it uses the computers central processing unit (CPU) to perform operations and its memory (both immediate and auxiliary) to hold the program and data. Analysis of algorithms or performance analysis refers to the task of determining how much computing time and storage an algorithm requires. An important result of this study is that it allows you to make quantitative judgments about the value of one algorithm over another. Another result is that it allows you to predict whether the software will meet any efficiency constraints that exist. Questions such as how well an algorithm performs in the best case, in the worst case, or on the average are typical. 4. How to test a program: - testing a program consists of two phases: debugging and profiling (or performance measurement). Debugging is the process of executing programs on sample data sets to determine whether faulty results occur and, if so to correct them. In cases in which we cannot verify the correctness of output on sample data, the following strategy can be employed: let more than one programmer develop programs for the same problem, and compare the outputs produced by these programs. If the outputs match, then there is a good chance that they are correct. A proof of correctness is much more valuable than a thousand tests, since it guarantees that the program will work correctly for all possible inputs. Profiling or performance measurement is the process of executing a correct program on data sets and measuring the time and space it takes to compute the results. These timing figures are useful in that they may confirm a previously done analysis and point out logical places to perform useful optimization. Department of Computer Science & Engineering SJCET, Palai
11
STUDY OF ALGORITHM
An algorithm is a finite set of instructions that , if followed , accomplishes a particular task. The study of algorithms includes many important and active areas of research. There are four distinct areas of study 1. How to devise algorithms: Creating an algorithm is an art which may never be
fully automated.To study various design techniques that have proven to be useful in that they have often yielded good algorithms.Dynamic programming is one such technique.some important design techniques are linear,nonlinear and integer programming . 2. How to validate algorithms: Once an algorithm is devised , it is necessary to show
that it computes the correct answer for all possible legal inputs.This process is algorithm validation.The purpose of the validation is to assure us this algorithm will work correctly independent of the issues concerning the programming language it will eventually be written in. 3. How to analyze algorithms: This field of study is called analysis of algorithms. As an algorithm is executed , it uses the computers central processing unit to hold the program and data.Analysis of algorithms refers to the task of determining how much computing time and storage an algorithm requires.It allows you to predict whether the software will meet any efficiency constraints that exits. 4. How to test a program: Testing a program consists of two phases, debugging and
profiling.Debugging is the process of executing programs on sample data sets to determine whether faulty results occur and , if so, to correct them. Profiling is the process of executing a correct program on data sets and measuring the time and space it takes to compute the results.
PSEUDOCODE CONVENTIONS
We can describe an algorithm in many ways.We can use a natural language like English, although I we select this option,we must make sure that the resulting instructions are definite. We can present most of our algorithms using a pseudocode that resembles c 1. Comments begin with // and continueuntill the end of line
Eg: count :=count+1;//count is global ;It is initially zero. Department of Computer Science & Engineering SJCET, Palai
12
2. Blocks are indicated with matching braces: { and } .A compound statement can be represent as a block.The body of a procedure also forms a block.Statements are delimited by ; Eg: for j:= 1 to n do { Count:=count+1; C[i,j]:=a[i,j]+b[i,j]; Count:=count +1; }
3. An identifier begins with a letter. The data types of variables are not explicitly declared. The types will be clear from the context .Whether a variable is global or local to a procedure will also be evident from the context. Compound data types can be formed with records. Eg: node=record { datatype_1 data_1; : datatype_n data_n; node *link; } 4. Assignment of values to variables is done using the assignment statement <variable> := <expression>; Eg: count:= count+1; 5. There are two Boolean values true and false. In order to produce these values, the logical operators and , or , and not and the relational relational operators <,<=,=,!=,>= and > are provided. Eg: if (j>1) then k:=i-1; else k:=n-1; 6. Elements of multidimentional arrays are accessed using [ and ]. For eg: if A is a two dimentional array , the (I,j) th element of the array is denoted as A[I,j]. Array indicates start at zero. 7. The following looping statements are employed: for,while and repeat until. The while loop takes the following form. While (condition) do { <statement 1> Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) : : <statement n> } 8. A conditional statement has the following forms:
13
If < condition > then <statement> If<condition> then <statement 1> else <statement 2> Here < condition > is a Boolean expression and <statement>,<statement 1>, and < statement 2> are arbitrary statements. 9. Input and output are done using the instructions read and write. No format is used to specify the size of input or output quantities. Eg: write (n is even); 10. There is only one type of procedure: Algorithm. An algorithm consists of a heading and a body. The heading takes the form Algorithm Nmae (<parameter list>)
RECURSIVE ALGORITHMS
A function that calls itself repeatedly, satisfying some condition is called a Recursive Function. The algorithm that does this is called a recursive algorithm. Using recursion, we split a complex problem into its single simplest case. The recursive function only knows how to solve that simplest case. TYPES OF RECURSION:
Linear Recursion
A linear recursive function is a function that only makes a single call to itself each time the function runs (as opposed to one that would call itself multiple times during its execution). The factorial function is a good example of linear recursion. Another example of a linear recursive function would be one to compute the square root of a number using Newton's method (assume EPSILON to be a very small number close to 0):
double my_sqrt(double x, double a) Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) { double difference = a*x-x; if (difference < 0.0) difference = -difference; if (difference < EPSILON) return(a); else return(my_sqrt(x,(a+x/a)/2.0)); }
14
Tail recursive
Tail recursion is a form of linear recursion. In tail recursion, the recursive call is the last thing the function does. Often, the value of the recursive call is returned. As such, tail recursive functions can often be easily implemented in an iterative manner; by taking out the recursive call and replacing it with a loop, the same effect can generally be achieved. In fact, a good compiler can recognize tail recursion and convert it to iteration in order to optimize the performance of the code. A good example of a tail recursive function is a function to compute the GCD, or Greatest Common Denominator, of two numbers:
int gcd(int m, int n) { int r;
if (m < n) return gcd(n,m);
r = m%n; if (r == 0) return(n); else return(gcd(n,r)); }
Binary Recursive
Some recursive functions don't just have one call to themself, they have two (or more). Functions with two recursive calls are referred to as binary recursive functions. The mathematical combinations operation is a good example of a function that can quickly be Department of Computer Science & Engineering SJCET, Palai
15
implemented as a binary recursive function. The number of combinations, often represented as nCk where we are choosing n elements out of a set of k elements, can be implemented as follows: int choose(int n, int k) { if (k == 0 || n == k) return(1); else return(choose(n-1,k) + choose(n-1,k-1)); }
Exponential recursion
An exponential recursive function is one that, if you were to draw out a representation of all the function calls, would have an exponential number of calls in relation to the size of the data set (exponential meaning if there were n elements, there would be O(an) function calls where a is a positive number). A good example an exponentially recursive function is a function to compute all the permutations of a data set. Let's write a function to take an array of n integers and print out every permutation of it. void print_array(int arr[], int n) { int i; for(i=0; i<n; i) printf("%d ", arr[i]); printf("\n"); } void print_permutations(int arr[], int n, int i) { int j, swap; print_array(arr, n); for(j=i+1; j<n; j) { swap = arr[i]; arr[i] = arr[j]; arr[j] = swap; print_permutations(arr, n, i+1); swap = arr[i]; arr[i] = arr[j]; arr[j] = swap; } }
SJCET, Palai
16
To run this function on an array arr of length n, we'd do print_permutations(arr, n, 0) where the 0 tells it to start at the beginning of the array.
Nested Recursion
In nested recursion, one of the arguments to the recursive function is the recursive function itself! These functions tend to grow extremely fast. A good example is the classic mathematical function, "Ackerman's function. It grows very quickly (even for small values of x and y, Ackermann(x,y) is extremely large) and it cannot be computed with only definite iteration (a completely defined for() loop for example); it requires indefinite iteration (recursion, for example). Ackerman's function int ackerman(int m, int n) { if (m == 0) return(n+1); else if (n == 0) return(ackerman(m-1,1)); else return(ackerman(m-1,ackerman(m,n-1))); }
Mutual Recursion
A recursive function doesn't necessarily need to call itself. Some recursive functions work in pairs or even larger groups. For example, function A calls function B which calls function C which in turn calls function A. A simple example of mutual recursion is a set of function to determine whether an integer is even or odd. int is_even(unsigned int n) { if (n==0) return 1; else return(is_odd(n-1)); } int is_odd(unsigned int n) { return (!iseven(n)); }
SJCET, Palai
17
EXAMPLES OF RECURSIVE ALGORITHMS:
The Towers of Hanoi

The Towers of Hanoi puzzle (TOH) was first posed by a French professor, Edouard Lucas, in 1883. Although commonly sold today as a childrens toy, it is often discussed in discrete mathematics or computer science books because it provides a simple example of recursion. In addition, its analysis is straightforward and it has many variations of varying difficulty. The object of the Towers of Hanoi problem is to specify the steps required to move the disks or, as we will sometimes call them, rings) from pole r (r = 1, 2, or 3) to pole s (s = 1, 2, or 3; s _= r), observing the following rules: i) Only one disk at a time may be moved. ii) At no time may a larger disk be on top of a smaller one. The most common form of the problem has r = 1 and s = 3. FIGURE .
The Towers of Hanoi problem
Solution The algorithm to solve this problem exemplifies the recursive paradigm. We imagine that we know a solution for n 1 disks (reduce to a previous case), and then we use this solution to solve the problem for n disks. Thus to move n disks from pole 1 to pole 3, we would: 1. Move n 1 disks (the imagined known solution) from pole 1 to pole 2. However we do this, the nth disk on pole 1 will never be in our way because any valid sequence of moves with only n 1 disks will still be valid if there is an nth (larger) disk always sitting at the bottom of pole 1 (why?). 2. Move disk n from pole 1 to pole 3. 3. Use the same method as in Step 1 to move the n 1 disks now on pole 2 to Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) pole 3.
18
Algorithm:
Input num [Number of disks] Pinit [Initial pole; 1 Pinit 3] Pfin [Final pole; 1 Pfin 3, Pinit _= Pfin] Output The sequence of commands to the robot to move the disks from pole Pinit to pole Pfin Algorithm Hanoi procedure H(in n, r, s) [Move n disks from pole r to pole s] if n = 1 then robot(r s) else H(n1, r, 6rs) robot(r s) H(n1, 6rs, s) endif endpro H(num, Pinit, Pfin) [Main algorithm]
ALGORITHM DESIGN TECHNIQUES

For a given problem, there are many ways to solve them. The different methods are listed below. 1. Divide and Conquer. 2. Greedy Algorithm. 3. Dynamic Programming. 4. Branch and Bound. 5. Backtracking Algorithms. 6. Randomized Algorithm. Now let us discuss each method briefly. 1. Divide and Conquer Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) Divide and conquer method consists of three steps. a. Divide the original problem into a set of sub-problems. b. Solve every sub-problem individually, recursively.
19
c. Combine the solutions of the sub-problems into a solution of the whole original problem. 2. Greedy Approach Greedy algorithms seek to optimize a function by making choices which are the best locally but do not look at the global problem. The result is a good solution but not necessarily the best one. Greedy Algorithm does not always guarantee the optimal solution however it generally produces solutions that are very close in value to the optimal solution.
3.Dynamic Programming. Dynamic Programming is a technique for efficient solution. It is a method of solving problems exhibiting the properties of overlapping sub problems and optimal sub-structure that takes much less time than other methods.
4. Branch and Bound Algorithm. In Branch and Bound Algorithm a given Algorithm which cannot be bounded has to be divided into at least two new restricted sub-problems. Branch and Bound Algorithm can be slow, however in the worst case they require efforts that grows exponentially with problem size. But in some cases the methods converge with much less effort. Branch and Bound Algorithms are methods for global optimization in non-convex problems. 5.Backtracking Algorithm. They try each possibility until they find the right one. It is a depth first search of a set of possible solution. During the search if an alternative doesnt work, the search backtrack to the choice point, the place which presented different alternatives and tries the next alternative. If there are no more choice points the search fails.
6.Randomized Algorithm. A Randomized Algorithm is defined as an algorithm that is allowed to access a source of independent, unbiased, random bits, and it is then allowed to use these random bits to influence its computation. Department of Computer Science & Engineering SJCET, Palai
20
ALGORITHMIC COMPLEXITY
The time complexity of an algorithm is given by the number of steps taken by the algorithm to compute the function it was written for. The number of steps is itself a function of the instance characteristics. Althogh any specific instance may have several characteristics, the number of steps computed as a function of some subset of these. Usually, we might wish to know how the computing time increases as the number of inputs increase. In this case the number of steps will be computed as a function of the number of inputs alone. For a different algorithm , we might be interested in determining how the computing time increases as the magnitude of one of the input increases. In this case the number of steps will be computed as a function of the magnitude of this input alone. Thus before the step count of an algorithm can be determined, we need to know which characteristics of the problem instance are to be used. These define the variables in the expression for the step count. In the case of sum, we chose to measure the time complexity as a function of the number n of elements being added. For algorithm Add, the choice of characteristics was the number m of rows and the number n of columns in the matrices being added.
SPACE COMPLEXITY
Algorithm abc computes a+b+b*c+(a+b-c)/(a+b)+4.0; Algorithm Sum computes Ei to n = 1 a[i] iteratively, where the a[i]s are real numbers; and RSum is a recursive algorithm that computes Ei n =1 a [i] The space needed by each of these algorithms is seen to be the sum of the following components: 1. A fixed part that is independent of the characteristics of the inputs and ouputs . This part typically includes the instruction space, space for simple variables and fixed-size component variables, space for constants. 2. A variable part that consists of the space needed by component variables whose size is dependent on the particular problem instance being solved, the space needed by reference variables and he recursion stack space. The space requirement s(p) of any algorithm p may thereore be written as s(p) =c+Sp, Where c is a constant. Department of Computer Science & Engineering SJCET, Palai
21
TIME COMPLEXITY
COMPLEXITY OF SIMPLE ALGORITHMS Time complexity of an algorithm is given by the number of steps taken by the algorithm to compute the function it was written for. The time T(P) taken by a program P is the sum of compile time and run time. The compile time does not depends the instance characteristics. A compiled program will run several times without recompilation. So we have to concern with the run time of the program which is denoted by tp (instance characteristics). If we know the characteristics of the compiler to be used ,we could proceed to determine the number of additions, subtractions, multiplications, divisions, compares, loads, stores and so on,that would be made the code for P. So we could obtain an expression for tp (n) of the form tp(n)=ca ADD(n) + cs SUB(n) + cm MUL(n) + cd DIV(n)+. Where n denotes the instance characteristics, and ca, cs , cm, cd, and so on, respectively, denote the time needed for an addition, subtraction, multiplication, division ,and so on, and ADD,SUB,MUL, DIV, and so on, are functions whose values are the numbers of additions, subtractions, multiplications, divisions, and so on ,that are performed when the code for P is used on an instance with characteristic n. The value of tp(n) for any n can be obtained only experimentally.The program is typed,compiled and run on a particular machine.The execution time is physically blocked,and tp(n) obtained.In a multi user system ,execution time depends on factors such as system load and number of other programs running on the computer at the time P is running,the characteristics of these other programs and so on. A propgram step is loosely defined as a syntactically or semantically meaningful segment of a program that has an execution time independent of the instance characteristics. For
example,consider the entire stataement return a+b+b*c+(a+b-c)/(a+b)+4.0 of the program given below
SJCET, Palai
Algorithm Analysis and Design (R 606) Algorihm abc(a,b,c) { return a+b+b*c+(a+b-c)/(a+b)+4.0 }
22
The above line could be considered as a step since its execution time is independent of the the instance characteristics.The number of steps any program statement is assigned depends on the kind of statement.For example comments count as zero steps;an assignment statement which does not involve any calls to other algorithms is counted as one step;in an iterative statement such as the for ,while,and repeat until statements ,we consider the step count only for the control part of the the statement.The control parts for for and while statements have the following forms: for i=(expr) to (expr1) do while(expr) do Each execution of the control part of a while statement is given by a step count equal to the number of step counts assignable to(expr).The step count for each execution of the
controlpart of a for statement is one,unless the counts attributable to (expr) and (expr1) are the functions of the control part the instance characteristics. In the latter case,the first execution of the for has a step count equal to the sum of counts for (expr)
and (expr1).Remaining executions of the for statement have a step count of one;and o on. We can determine the number of steps needed by a program to solve a particular program instance in one of the two ways.In the first method we introduce a new variable,count into the problem.This is a global variable with initial value equals zero.Statement to increment count by appropriate amount are introduced into the program.This is done so that each time a statement in the original program is executed,count is incremented by the step count of that statement.
SJCET, Palai
23
EXAMPLE FOR TIIME COMPLEXITY CALCULATION Fibonacci series of numbers starts as 0,1,1,2,3,5,8,13,21,34,55.. Each new term is obtained by taking the sum of two previous terms.If we call the first term of the sequence f0,then f0=0,f1=1,and in general fn=fn-1+fn-2, n>=2 Algorithm Fibonacci(n) // Compute the nth Fibonacci number { if (n<=1) then Write(n); else { fnm2:=0;fnm=1; For I := 1 to n do { fn = fnm1+fnm2; fnm2 =fnm1;fnm1=fn; } Write(fn) } } To analyze the complexity of this algorithm we need to consider two cases (1) n=0 or 1 and (2) n>1.When n=0 or 1 lines 4and 5 get executed once each.Since each line has an s/e of 1,total step count for this case is 2.When n>1,lines 4, 8 and 14 are each executed once.Line 9 gets executed n times ,and lines 11 and 12 get executed n-1 times each.Line 8 has an s/e of 2,line 12 has an s/e of 2 and line 13 has s/e of 0.The remaining lines that get executed have s/es of 1 The total steps for the case n>1 is therefore 4n+1.
SJCET, Palai
Algorithm Analysis and Design (R 606) Example 1 Sum of n numbers Algorithm with count statements added Algorithm sum(a,n) { S:=0.0;
24
Count := count+1;//count is global; it is initially zero. For i:=1 to n do { Count :=count + 1;//For for S:= s+a[i];count:=count+1;//For assignment } Count :=count+1;//For last time of for Count := count+1;//For the return Return s; } Simplified version of algorithm Algorithm Sum(a,n) { For i:=1 to n do count:= count+2; Count:= count+3;} Complexity calculation tRSum(n)=2+tRSum(n-1) =2(2)+tRSum(n-2) : =n(2)+tRSum(0) =2n+2, n>=0 =2+2+tRSum(n-2)
Example 2 :Complexity of Fibonacci series Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) Algorithm Fibonacci(n) //compute the nth Fibonacci number { If(n<=1) then Write (n); else { Fnm2:=0; fnm1:= 1; for i:=2 to n do { fn:= fnm1+fnm2; fnm2:=fnm1;fnm1:=fn; } Write (fn); } }
25
To analyse the time complexity of this algorithm, we need to consider the two cases (1) n=0 or 1 and(2) n>1.When n=0 or 1 , lines 4 and 5 get executed once each. Since each line has an s/e of 1, the total step count for this case is 2.When n>1, lines 4,8 and 14 are each executed once. Line 9 gets executed n times and lines 11 and 12 get executed n-1 times each.Line 8 has an s/e of 2, line 12 has an s/e of 2,and line 13 has an s/e of 0. The remaning lines that get executed have s/e of 1. The total steps for the case n>1 is therefore 4n+1.
SJCET, Palai
26
ASYMPTOTIC NOTATION Introduction

A problem may have numerous algorithmic solutions. In order to choose the best algorithm for a particular task, you need to be able to judge how long a particular solution will take to run. Or, more accurately, you need to be able to judge how long two solutions will take to run, and choose the better of the two. You don't need to know how many minutes and seconds they will take, but you do need some way to compare algorithms against one another. Asymptotic complexity is a way of expressing the main component of the cost of an algorithm, using idealized units of computational work. Consider, for example, the algorithm for sorting a deck of cards, which proceeds by repeatedly searching through the deck for the lowest card. The asymptotic complexity of this algorithm is the square of the number of cards in the deck. This quadratic behavior is the main term in the complexity formula, it says, e.g., if you double the size of the deck, then the work is roughly quadrupled. The exact formula for the cost is more complex, and contains more details than are needed to understand the essential complexity of the algorithm. With our deck of cards, in the worst case, the deck would start out reverse-sorted, so our scans would have to go all the way to the end. The first scan would involve scanning 52 cards, the next would take 51, etc. So the cost formula is 52 + 51 + ... + 1. generally, letting N be the number of cards, the formula is 1 + 2 + ... + N, which equals (N + 1) * (N / 2) = (N2 + N) / 2 = (1 / 2)N2 + N / 2. But the N^2 term dominates the expression, and this is what is key for comparing algorithm costs. (This is in fact an expensive algorithm; the best sorting algorithms run in sub-quadratic time.) Asymptotically speaking, in the limit as N tends towards infinity, 1 + 2 + ... + N gets closer and closer to the pure quadratic function (1/2) N^2. And what difference does the constant factor of 1/2 make, at this level of abstraction. So the behavior is said to be O(n2). Now let us consider how we would go about comparing the complexity of two algorithms. Let f(n) be the cost, in the worst case, of one algorithm, expressed as a function of the input size n, and g(n) be the cost function for the other algorithm. E.g., for sorting algorithms, f(10) and g(10) would be the maximum number of steps that the algorithms would take on a list of 10 items. If, for all values of n >= 0, f(n) is less than or equal to g(n), then the algorithm with Department of Computer Science & Engineering SJCET, Palai
27
complexity function f is strictly faster. But, generally speaking, our concern for computational cost is for the cases with large inputs; so the comparison of f(n) and g(n) for small values of n is less significant than the "long term" comparison of f(n) and g(n), for n larger than some threshold. Note that we have been speaking about bounds on the performance of algorithms, rather than giving exact speeds. The actual number of steps required to sort our deck of cards (with our naive quadratic algorithm) will depend upon the order in which the cards begin. The actual time to perform each of our steps will depend upon our processor speed, the condition of our processor cache, etc., etc. It's all very complicated in the concrete details, and moreover not relevant to the essence of the algorithm.
BIG-O NOTATION Definition

Big-O is the formal method of expressing the upper bound of an algorithm's running time. It's a measure of the longest amount of time it could possibly take for the algorithm to complete. More formally, for non-negative functions, f(n) and g(n), if there exists an integer n0 and a constant c > 0 such that for all integers n > n0, f(n) cg(n), then f(n) is Big O of g(n). This is denoted as "f(n) = O(g(n))". If graphed, g(n) serves as an upper bound to the curve you are analyzing, f(n). O-Notation (Upper Bound) This notation gives an upper bound for a function to within a constant factor. We write f(n) = O(g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or below cg(n).
SJCET, Palai
28
Theory Examples
So, let's take an example of Big-O. Say that f(n) = 2n + 8, and g(n) = n2. Can we find a constant c, so that 2n + 8 <= n2? The number 4 works here, giving us 16 <= 16. For any number c greater than 4, this will still work. Since we're trying to generalize this for large values of n, and small values (1, 2, 3) aren't that important, we can say that f(n) is generally faster than g(n); that is, f(n) is bound by g(n), and will always be less than it. It could then be said that f(n) runs in O(n2) time: "f-of-n runs in Big-O of n-squared time". To find the upper bound - the Big-O time - assuming we know that f(n) is equal to (exactly) 2n + 8, we can take a few shortcuts. For example, we can remove all constants from the runtime; eventually, at some value of c, they become irrelevant. This makes f(n) = 2n. Also, for convenience of comparison, we remove constant multipliers; in this case, the 2. This makes f(n) = n. It could also be said that f(n) runs in O(n) time; that lets us put a tighter (closer) upper bound onto the estimate.
Practical Examples
O(n): printing a list of n items to the screen, looking at each item once. O(ln n): also "log n", taking a list of items, cutting it in half repeatedly until there's only one item left. O(n2): taking a list of n items, and comparing every item to every other item.
SJCET, Palai
29
BIG-OMEGA NOTATION
For non-negative functions, f(n) and g(n), if there exists an integer n0 and a constant c > 0 such that for all integers n > n0, f(n) cg(n), then f(n) is omega of g(n). This is denoted as "f(n) = (g(n))". This is almost the same definition as Big Oh, except that "f(n) cg(n)", this makes g(n) a lower bound function, instead of an upper bound function. It describes the best that can happen for a given data size. -Notation (Lower Bound) This notation gives a lower bound for a function to within a constant factor. We write f(n) = (g(n)) if there are positive constants n0 and c such that to the right of n0, the value of f(n) always lies on or above cg(n).
How asymptotic notation relates to analyzing complexity

Temporal comparison is not the only issue in algorithms. There are space issues as well. Generally, a trade off between time and space is noticed in algorithms. Asymptotic notation empowers you to make that trade off. If you think of the amount of time and space your algorithm uses as a function of your data over time or space (time and space are usually analyzed separately), you can analyze how the time and space is handled when you introduce more data to your program.
SJCET, Palai
30
This is important in data structures because you want a structure that behaves efficiently as you increase the amount of data it handles. Keep in mind though that algorithms that are efficient with large amounts of data are not always simple and efficient for small amounts of data. So if you know you are working with only a small amount of data and you have concerns for speed and code space, a trade off can be made for a function that does not behave well for large amounts of data.
A few examples of asymptotic notation

Generally, we use asymptotic notation as a convenient way to examine what can happen in a function in the worst case or in the best case. For example, if you want to write a function that searches through an array of numbers and returns the smallest one:
function find-min(array a[1..n]) let j := for i := 1 to n: j := min(j, a[i]) repeat return j end
Regardless of how big or small the array is, every time we run find-min, we have to initialize the i and j integer variables and return j at the end. Therefore, we can just think of those parts of the function as constant and ignore them. So, how can we use asymptotic notation to discuss the find-min function? If we search through an array with 87 elements, then the for loop iterates 87 times, even if the very first element we hit turns out to be the minimum. Likewise, for n elements, the for loop iterates n times. Therefore we say the function runs in time O(n). What about this function:
function find-min-plus-max(array a[1..n]) // First, find the smallest element in the array let j := ;
for i := 1 to n: j := min(j, a[i]) repeat
SJCET, Palai

let minim := j
31
// Now, find the biggest element, add it to the smallest and j := ;
for i := 1 to n: j := max(j, a[i]) repeat let maxim := j
// return the sum of the two return minim + maxim; end
What's the running time for find-min-plus-max? There are two for loops, that each iterate n times, so the running time is clearly O(2n). Because 2 is a constant, we throw it away and write the running time as O(n). Why can you do this? If you recall the definition of Big-O notation, the function whose bound you're testing can be multiplied by some constant. If f(x) = 2x, we can see that if g(x) = x, then the Big-O condition holds. Thus O(2n) = O(n). This rule is general for the various asymptotic notations.
THETA
Definition: The function f(n)=theta(g(n))(read as f of n is theta of g of n) iff there exist positive constants c1,c2, and n0 such that c1g(n) <=f(n)<=c2g(n) for all n, n>=n0. Example: The function 3n+2=theta(n) as 3n+2>=3n for all n>=2 and 3n+2<=4n for all n>=2, so c1=3,c2=4 and n0=2. 3n+3= theta(n), 10n2+4n+2=theta(n2),6*2n+n2=theta(2n), and 10* log n+4= theta (log n). 3n+2 !=theta(1), 3n+3 != theta(n2), 10n2+4n+2 !=theta(n), 10n2+4n+2 !=theta(1), 6*2n+n2 !=theta(n2), !=theta(1). The theta notation is more precise than both the big oh and big omega notations. The function f(n)=theta(g(n)) iff g(n) is both lower and upper bound of f(n). 6*2n +n2 !=theta(n100), and 6 * 2n +n2
SJCET, Palai
32
Little oh
Definition: The function f(n)=0(g(n)) (read as f of n is little oh of g of n) iff Lim n->infinity Example: The function 3n+2=o(n2) since lim n->infinity (3n+2)/n2=0. 3n+2=o(n log n). 3n+2= o(n log log n). 6*2n+n2=o(3n). 6*2n+n2=o(2n log n). 3n+2 !=o(n). 6*2n+n2 !=o(2n). f(n)/g(n)=0
SJCET, Palai
33
Little omega
Definition: The function f(n)=w(g(n)) (read as f of n is little omega of g of n) iff Lim n->infinity Example: g(n)/f(n)=0
Algorithm Sum(a,n)
{ S:=0.0; for i=1 to n do s:=s+a[i]; return s; }
Alg 1. Iterative function for sum For this algorithm Sum 2n+3. So, tSum(n)=theta(n) wec determined that tSum(n)=
Asymptotic Notation Properties

Let f(n) and g(n) be asymptotically positive functions. Prove or disprove each of the following conjectures. Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) a. f(n)=O(g(n)) implies g(n)=O(f(n)). b. f(n)+g(n)= (min(f(n),g(n))).
34
c. f(n)=O(g(n)) implies lg(f(n))=O(lg(g(n))),where lg(g(n))>=1 and f(n)>=1 for all sufficiently large n. d. f(n)=O(g(n)) implies 2^f(n)=O(2^g(n)). e. f(n)=O((f(n))^2). f. f(n)=O(g(n)) implies g(n) and (f(n)). g. f(n)= (f(n/2)).
h. f(n)+o(f(n))= (f(n)). COMMON COMPLEXITY FUNCTIONS Comparison of functions

Many of the relational properties of real numbers apply to asymptotic comparisons as well. For the following, assume that f(n) and g(n) are asymptotically positive. Transitivity: f(n)= (g(n)) and g(n)= (h(n)) imply f(n)= (h(n)), f(n)=O(g(n)) and g(n)=O(h(n)) imply f(n)=O(h(n)), f(n)=(g(n)) and g(n)=(h(n)) imply f(n)=(h(n)), f(n)=o(g(n)) and g(n)=o(h(n)) imply f(n)=o(h(n)), f(n)=(g(n)) and g(n)=(h(n)) imply f(n)=(h(n)). Reflexivity: f(n)= (f(n)), f(n)=O(f(n)), f(n)=(f(n)). Symmetry: f(n)= (g(n)) if and only if g(n)= (f(n)).
SJCET, Palai
Algorithm Analysis and Design (R 606) Transpose symmetry f(n)=O(g(n)) if and only if g(n)=(f(n)), f(n)=O(g(n)) if and only if g(n)=(f(n)),
35
Because these properties hold for asymptotic notations, one can draw an analogy between the asymptotic comparison of two functions f and g and the comparison of two real numbers a and b: f(n)=O(g(n)) similar to a<=b, f(n)=(g(n)) similar to a>=b, f(n)= (g(n)) similar to a=b, f(n)=o(g(n)) similar to a<b, f(n)=(g(n)) similar to a>b.
We say that f(n) is asymptotically smaller than g(n) if f(n)=o(g(n)), and f(n) is asymptotically larger than g(n) if f(n)=(g(n)). One property of real numbers, however, does not carry over to asymptotic notations: Trichotomy: For any two real numbers a and b, exactly one of the following must hold: a<b, a=b, or a>b. Although any two real numbers can be compared, not all functions are asymptotically comparable. That is, for two functions f(n) and g(n), it may be the case that neither f(n)=O(g(n)) nor f(n)=(g(n)) holds. For example, the functions n and n^(1+sin n) cannot be compared using asymptotic notation, since the value of the exponent in n^(1+sin n) oscillates between 0 and 2, taking on all values in between.
SJCET, Palai
36
RECURRENCE RELATIONS
The recurrence relation is an equation or inequality that describes a function in terms of its values of smaller inputs. The main tool for analyzing the time efficiency of a recurrence algorithm is to setup a sum expressing the number executions of its basic operation and ascertain the solutions order of growth To solve a recurrence relation means ,to obtain a function defined on natural numbers that satisfies the recurrence. Examble: tn = 2tn-1 T(n)= 3T(n/4)+(n) Different methods for solving recurrence relation are:Substitution method Itration method Changing variables method Recurrence tree Characteristic equation method n^ method Master theorem
Solving recurrences by Substitution method

Idea: Make a guess for the form of the solution and prove by induction. Can be used to prove both upper bounds O() and lower bounds (). Lets solve T(n) = 2T(n/2) + n using substitution Guess T(n) cn log n for some constant c (that is, T(n) = O(n log n)) Proof: Base case: we need to show that our guess holds for some base case (not necessarily n = 1, some small n is ok). Ok, since function constant for small constant n. Assume holds for n/2: T(n/2) c n2 log n2 Prove that holds for n: T(n) cn log n
T(n)= 2T(n/2)+n Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) 2(c n/2 logn/2) + n = cn log n/2 +n =cn log n-cn log 2+ n =cn log n-cn + n So ok if c 1 Similarly it can be shown that T(n) = (n log n) Similarly it can be shown that T(n) = T([n/2]) + T([n/2]) + n is (n lg n). The hard part of the substitution method is often to make a good guess.
37
Solving Recurrences with the Iteration

In the iteration method we iteratively unfold the recurrence until we see the pattern. The iteration method does not require making a good guess like the substitution method (but it is often more involved than using induction). Example: Solve T(n) = 8T(n/2) + n (T(1) = 1) T(n) = n + 8T(n/2) = n + 8(8T( n/2 ) + (n/2)) = n + 8T( n/2 ) + 8(n/4)) = n+ 2n + 8T( n/2 ) = n + 2n + 8(8T( n/2 ) + ( n/2 )) = n + 2n + 8T( n/2 ) + 8(n/4 )) = n + 2n + 2n+ 8T( n/2 ) =... = n + 2n + 2n+ 2n + . . .
Masters theorem
The master method is used for solving the following type of recurrence T(n)=aT(n/b)+f(n),a 1 and b>1.In above recurrence the problem is divided in to a subproblems each of size atmost n/b.The subproblems are solved recursively each in T(n/b) Department of Computer Science & Engineering SJCET, Palai
38
time.The cost of split the problem or combine the solutions of subproblems is given by function f(n).It should be note that the number of leaves in the recursion tree is E= where
THEOREM
Let T(n) be defined on the non-negative integers by the recurrence T(n)=aT(n/b)+f(n) where a 1,b>1 are constants and f(n) be a function.Then T(n) can be asymtotically as,
CASE 1:
If f(n) O( for some e>0,then T(n) ( )
CASE 2:
If f(n) ( ) then T(n) (f(n).log n)
CASE 3:
If f(n) ( for some e>0 and f(n) O( for some e then T(n) (f(n))
SJCET, Palai
39
RECURRENCE PROBLEMS Case 1 problem: 1. T(n)=4T(n/2)+n

E=log 4/log2=2 O( O( T(n) =n ( )
Case 2 problem: 2.T(n)=4T(n/2)+

a=4, b=2 f(n)= E=log 4/log 2=2 ( ) ( .log n)
T(n)
Case 3 problem: 3.T(n)=4T(n/2)+

a=4, b=2 f(n)= E=log 4/log 2=2 ( = ( = ( O( = O( = O( ) ) ) ), T(n) ( ) SJCET, Palai ) )
40
RECURSION TREES FOR COMPLEXITY CALCULATIONS

A different way to look at the iteration method: is the recursion-tree we draw out the recursion tree with cost of single call in each noderunning time is sum of costs in all nodes if you are careful drawing the recursion tree and summing up the costs, the recursion tree is a direct proof for the solution of the recurrence, just like iteration and substitution Example: T(n) = 8T(n/2) + n (T(1) = 1)
T(n)=n+ 2n+ 2n+ 2n+ .....+2^(log n-1) n+8^(log n)
SJCET, Palai
41
Changing variables
Sometimes reucurrences can be reduced to simpler ones by changing variables Example: Solve T(n) = 2T(n) + log n Let m = log 2^m = n n = 2^m/2 T(2^m) = 2T(2^m/2) + m
T(n) = 2T(n) + log n )
Let S(m) = T(2^m) T(2^m) = 2T(2^m/2) + m ) S(m) = O(mlogm)) T(n) = T(2^m) = S(m) = O(mlogm) = O(log n log log n) S(m) = 2S(m/2) + m)
Other recurrences
Some important/typical bounds on recurrences not covered by master method: Logarithmic: (log n) Recurrence: T(n) = 1 + T(n/2) Typical example: Recurse on half the input (and throw half away) Variations: T(n) = 1 + T(99n/100) Linear: (N) Recurrence: T(n) = 1 + T(n 1) Typical example: Single loop Variations: T(n) = 1+2T(n/2), T(n) = n+T(n/2), T(n) = T(n/5)+T(7n/10+6)+n Quadratic: (n2) Recurrence: T(n) = n + T(n 1) Typical example: Nested loops Exponential: (2^n) Recurrence: T(n) = 2T(n 1)
SJCET, Palai
42
PROFILING
Profiling or performance measurement is the process of executing a correct program on data set and measuring the time and space it takes to compute the result.
NONDETERMINISTIC ALGORITHMS
The notation of algorithm that we have been using has the property that the result of every operation is uniquely defined.Algoithm with this property are termed deterministic algorithms. Such algorithms agree with the programs are executed on computer. In a theoretical framework we can remove this rustication on the outcomes of every operation. We can allow algorithms to be containing operations whose outcomes of every operation. We can allow algorithm to contain operations whose outcomes are not uniquely defined but are limited to specified set of possibilities. The machine executing such operations is allowed to choose any one of these outcomes subject to be a defined later. This leads to the concept of nondeterministic algorithms, we introduce three functions.
1. Choice (S) arbitrarily chooses one of the elements of element of set S.
2. Failure () signals an unsuccessful completion.
3. Success () signals a successful completion.

The assignment statement x=Choice (1, n) could result in x being assigned any one of the integers in the range [1,n].there is the rule of specifying how this choice is to be made. The Failure () and Success () signals are used to define a computation of the algorithm. These statements cannot be used to effect a return. Whenever there is a set of choices that leads to a successful completion, then one such set of choices is always made and the algorithm terminates successfully. A nondeterministic algorithm terminates unsuccessfully if and only if there exists no set choices leading to a success signal. The computing times for. Choice, Failure, and Success are taken to be O(1).A machine capable of executing a non Department of Computer Science & Engineering SJCET, Palai
43
nondeterministic algorithm in this way is called a nondeterministic machine. Although nondeterministic machines do not exist in practice, we see that they provide strong intuitive reasons to conclude that certain problems cannot be solved by fast deterministic algorithms. Example Consider the problem of searching for an element x in a given set of elements A[1:n],n>1.We are required to determine an index j such that A[j]=x or j=0 if x is not in A.
1. J:=Choice(1.n); 2. If A[j]=x then {write (j);Success();} 3. Write (0); Failure();
From the way a nondeterministic computation is defined, it follows that no is 0 can be the output if and only if there is no j such that A[j]=x. Complexity of nondeterministic search algorithms =O(1). Since A is not ordered, every deterministic algorithm is of complexity (n).
DETERMINISTIC ALGOITHM for n>p

The deterministic algorithm for a selection whose run time is O(n/p log log p+ log
n)The basic idea of this algorithm is same to sequential algorithm. The sequential algorithm partitions the input into groups(of size,says,5),finds the median of each group, and output recursively the median(call it M)of these group meadians.Then the rank rM of M in the input is computed, and as a result, all element from the input that are either M or >Mare dropped, depending on whether Respectively. Finally recursively. We showed that run time of this algorithm was O(n). i >rM or i rM,
an appropriate selection is performed from the remaining keys
SJCET, Palai
44
MODULE 2
SJCET, Palai
45
DIVIDE AND CONQUER

Given a function to compute on n inputs the divide and conquer strategy suggests splitting the inputs into k distinct subsets,1<k<n,yielding k subproblems. These subproblems must be solved,and then a method must be found to combine subsolutions into a solution of whole. If the subproblems are still relatively large,then the divide and conquer strategy can possibly reapplied.The principle is natrually expressed using recursive algorithm.that is smaller and smaller subproblems of the same kind are generated untill eventually subproblems that are small enough to be solved without splitting are produced. CONTROL ABSTRACTION Control abstraction means a procedures whose flowof control is clear but whose primary operations are specified by other procedures whose precise meanings are left defined.Below algorithmDAndC is initially invoked as DandC(p),where p is the problem to be solved. Small(p) is a boolean-valued function that determines whether the input size is small enough that the answer can be computed without splitting.If this is so,the function S is invoked.Otherwise the problem p is divided into smaller subproblems.These subproblems P1,P2,----,Pkare solved by recursive applications of DandC.Combine is a function that determines the solutions to P using the solutions to the k subproblems. ALGORITHM Algorithm DAndC(p) { if small(p) then return s(p); else { divide p into smaller instances p1,p2,p3-----p(k),k>=1; apply DandC to each of these subproblems; return combine(DandC(p1),DandC(p2),-----,DandC(pk)); } } Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) FINDING THE MAXIMUM AND MINIMUM
46
This is a simple problem that can be solved by the divide and-conquer technique. The problem is to find the maximum and minimum items in a set of n elements.
ALGORITHM-1 Algorithm StraightMaxMin (a, n, max, min) //Set max to the maximum and min to the minimum of a[1:n]; { max:=min:=a[1]; for i:=2 to n do { if (a[i]>max) then max:= a[i]; if a[i]<min) then min:= a[i]; } } This is a straight forward algorithm to accomplish the above problem. In analyzing the time complexity of this algorithm, we concentrate on the number of element comparisons. The frequency count for other operations in this algorithm is of the same order as that for element comparisons. StraightMaxMin requires 2(n-1) element comparisons in the best, average, and worst cases. The best case occurs when the elements are in increasing order. The number of element comparisons is (n-1). The worst case occurs when the elements are in the decreasing order. In this case, the number of element comparisons is 2(n-1). The average number of element comparisons is less than 2(n-1). On the average, a[i] is greater than max, half the time, and so the average number of comparisons is 3n/2-1. The maximum and minimum are a[i] if n=1. If n=2, the problem can be solved by making one comparison.
SJCET, Palai
Algorithm Analysis and Design (R 606) ALGORITHM-2 Algorithm MaxMin ( i, j, max, min )
47
//a[1:n] is a global array. Parameters I and j are integers, 1<=i<=j<=n. The effect is to set //max and min to the largest and smallest values in a [i : j] ,respectively. { if (i=j) then max :=min :=a[i]; // Small(p) else if (i=j-1) then //Another case of Small(p) { if( a[i] < a[j]) then { max:=a[j]; min:=a[i]; } else { max:=a[i]; min:=a[j]; } } else { // If P is not small, divide P into subproblems. Find where to split // mid := [(i+j)/2]; // Solve the subproblems. MaxMin ( i, mid, max, min ) MaxMin ( mid+1, j, max, min) if (max<max1)then max:=max1; if (min>min1)then min:=min1; } } MaxMin is a recursive algorithm that finds the maximum and minimum of the set of elements {a(i),a(i+1),.a(j)}.The situation of set sizes one (i=j) and two (i=j-1) are handled separately. For sets containing more than two elements, the midpoint is determined (just as in binary search) and two new subproblems are generated. When the maxima and minima of these subproblems are generated, the two maxima are compared and the two minima are compared to achieve the solution for the entire set.
SJCET, Palai
Algorithm Analysis and Design (R 606) Eg:- Suppose we simulate MaxMin on the following nine elements: a: [1] [2] [3] [4] [5] [6] [7] [8] [9] 22 13 -5 -8 15 60 17 31 47
48
A good way of keeping track of recursive calls is to build a tree by adding a node each time anew call is made. For this algorithm each node has four items of
information: i, j, max, and min. On the array a[] above, the tree is produced. Trees of recursive calls of MaxMin
1, 9,60,-8
1, 5, 22, -8
6,9,60,17
1, 3, 22, -5
4, 5, 15,-8
6,7,60,17
8, 9, 47, 31
1,2,22,13
3,3,-5,-5
We see that the root node contains 1 and 9 as the values of i and j corresponding to the initial call to MaxMin. This execution produces new call to MaxMin where i and j have the values 1 ,5 and 6,9, respectively, and thus split the set into two subsets of approximately the same size. From the tree, we can immediately see that the maximum depth of recursion is four(including the first call). The order in which max and min are assigned values are follows: [1,2,22,13] [3,3,-5,-5] [1,3,22,-5] [4,5,15,-8] [1,5,22,-8] [6,7,60,17] [8,9,47,31] [6,9,60,17] [1,9,60,-8]. Number of element comparisons needed for MaxMin: If T(n) represents the number of element comparisons needed, then the resulting recurrence relation is:
T ([n/2]) + T ([n/2]) +2 T (n) = 1 0
n>2 n=2 n=1 SJCET, Palai
Algorithm Analysis and Design (R 606) When n is a power of 2, n=2k we can solve this equation by successive substitutions: T(n) = 2T(n/2)+2 = 2(2T(n/4)+2) +2 = 4T(n/4)+ 4 + 2 . . = 2k-1T(2) + 1< = i < = = 2k-1 + 2k -2 = 3n/2 - 2
k-1 2 i
49
When n is a power of 2, the number of comparisons in the best , average and worst case is 3n/2-2. COMPLEXITY ANALYSIS OF FINDING MAXIMUM&MINIMUM Consider the following nine elements to simulate MaxMin 22 13 -5 -8 15 60 17 31 47
Fig below shows the tree constructed for the algorithm. Each node has four items of information: i, j, max and min. 9
1, 9, 60,-8
5
1, 5, 22,-8
8
6, 9, 60,17
1, 3, 22,-5
4, 5, 15,-8 4
6, 7, 60,17 6
1, 9, 60,-8 7
1, 2, 22, 13
3, 3, -5,-5
Consider the total number of element comparison needed for MaxMin? If T(n) represents this number, then the resulting recurrence relation is T ( n/2 ) + T( n/2 ) + 2 T (n)= 1 0 n>2 n=2 n=1
SJCET, Palai
Algorithm Analysis and Design (R 606) When n is a power of 2, n=2k for some positive integer k, then T(n) = 2T( n/2 ) + 2 = 2(2T (n/4) +2) +2 = 4T (n/4) +4+2 : : = 2k-1T (2) +1<=i<=k-1 2i = 2k-1+2k-2 =3n/2-2
50
Therefore 3n/2 is the best, average and worst case number of comparisons when n is a power of two.
Consider the count when element comparisons have the same cost as comparisons between i and j. Let C (n) be this number. Assuming n=2k for some positive integer k, we get
C(n) = 2C(n/2) + 3 n>2 2 n=2
C (n) = 2C (n/2) + 3 =4 C (n/4) + 6 + 3 : : =2k-1C (2)+30k-2 2i =2k+3*2k-1-3 =5n/2-3
if comparisons between array elements are costlier than comparisons of integer variable, then divide and conquer method is more efficient In both cases mentioned above the best case, average case and worst case complexity is (n)
SJCET, Palai
51
RECURSION TREE
DEFINITION: A recursion tree is a tree that depicts the entire recursion process. Consider the following recurrence relation. T (n) =3T (n/4)+ (n2) = 3T (n/4) + cn2 cn2 c(n/4)2 c(n/4)2 c(n/4)2 : T(n/16) T(n/16) T(n/16)
2 2 2
T(n/16) T(n/16) T(n/16) : : : : : : T(1) : : : : : T(1)
: : : : : : T(1)
c(n/16)
c(n/16) :
c(n/16) : : :
: : : : T(1)
T(n/16) T(n/16) T(n/16) : T(1) : T(1) :
: :
T(1) T(1)
T(1)
Sub problem size at depth i=n/4i Boundary condition (n=1) :=(n/4i)=1 ie i=log4n Depth of the tree=log4n+1 (0, 1, 2,logn)
Cost of each node at level i=c(n/4)i No: of node at level log4n=3log4n Cost of each node=3log4n.T(1)=(3log4n)=(nlog43) Total cost=cn2+3/16cn2+ (3/16)2cn2+ (3/16)log4n-1cn2 = (3/16)I cn2+ (nlog43)+ (nlog43) Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) = (3/16)I cn2+ (nlog43) = 1/(1-3/16)cn2+ (nlog43) = 16/13cn2+ (nlog43) = O (n2)
52
BINARY SEARCH ALGORITHMS

The Binary search requires an ordered list. Iterative Algorithm int find (const list, int target) // pre: list is sorted in ascending order //post: ITERATIVE binary search will return the index of the target element, else -1 { int mid; int first = 0; int last = list.length( ) -1; while ( first <= last ) { mid = (first + last) / 2; if ( list[mid] == target ) return mid; if ( list[mid] > target ) last = mid - 1; else } return -1; } Recursive Algorithm int find (const list, int target, int first, int last) // pre: list is sorted in ascending order //post: RECURSIVE binary search will return the index of the target element, else -1 first = mid + 1;
SJCET, Palai
Algorithm Analysis and Design (R 606) { if (first > last) return -1;
53
int mid = (first + last) / 2; if (list[mid] == target) return mid; if (list[mid] < target) return find(list, target, mid+1, last);
return find(list, target, first, mid-1);
Complexity Analysis of Binary Search

For binary search, storage is required for the n elements of the array plus the variables low, high, mid and x. A binary search tree can be created to explain the binary search where the mid value of each node is the value of the node. For example, if n=14, the resultant tree will be like the one shown below.
7
11
13
10
12
14
SJCET, Palai
54
The first comparison is s with a[7].if x<a[7], then the next comparison is with a[3]; similarly, if x>a[7], then the next comparison is with a[11]. If x is present, then the algorithm will end at one of the circular nodes tat lists the index into the array where x was found. If x is not present the algorithm will terminate at one of the square nodes.
Theorem: If n is in the range [2k-1,2k), then binary search makes at most k element comparison for a successful search and either k-1 or k comparison for an unsuccessful search. In other words the time for a successful search is O (log n) and for an unsuccessful search is (log n). E=I+2n Where E=External path length (sum of the distance of all external nodes from the root) I=Internal path length (sum of the distance of all internal nodes from the root). Let As (n) is the average number of comparison in a successful search and Au (n) is the average number of comparison in an unsuccessful search. The number of comparison to find an element represented by an internal node is one more than the distance of this node from the root. As (n) =1+I/n The number of comparisons on any path from the root to an external node is equal to the distance between the root and the external node. Since every binary tree with n internal nodes has n+1 external nodes Au (n) =E/ (n+1) As (n) =I/n+1 = (E-2n/n) + 1 =E/n-1 = (Au (n). (n+1))/n -1 = (n. Au (n) + Au (n))/n -1 = Au (n) + Au (n)/n -1 As (n) = Au (n) (1 +1/n) -1
SJCET, Palai
Algorithm Analysis and Design (R 606) Successful searches Best- (1) (only one element is compared) Average- (log n) Worst- (log n)
55
Unsuccessful searches Best, Average and Worst- (log n)
DIVIDE-AND-CONQUER MATRIX MULTIPLICATION ALGORITHM

The product of two n x n matrices X and Y is a third , n x n matrix Z=XY , with (i,j)th entry n Zij=Xik Ykj k=1 In general XY, not the same as YX; matrix multiplication is not commutative. The formula above implies an O (n3) algorithm for matrix multiplication: there are n2 entries to be computed, and each takes linear time. For quite a while, this was widely believed to be the best running time possible, and it was even proved that no algorithm which used just additions and multiplications could do better. It was therefore a source of great excitement when in 1969, Strassen announced a signi_cantly more ef_cient algorithm, based upon divide-and-conquer. Matrix multiplication is particularly easy to break into subproblems, because it can be performed blockwise. To see what this means, carve X into four n/2 x n/2 blocks, and also Y:
A B C D , E F
X=
Y=
G H
Then their product can be expressed in terms of these blocks, and is exactly as if the blocks were single elements.
XY=
A B
AE+BG AF+BH
SJCET, Palai

C D G H CE+DG CF+DH
56
We now have a divide-and-conquer strategy: to compute the size-n, product XY, recursively compute eight size-n/2 products AE,BG,AF,BH,CE,DG,CF,DH and then do a few O(n2) time additions. The total running time is described by the recurrence relation T(n)= 8T(n/2) + O(n2), which comes out to O(n3), the same as for the default algorithm. However, an improvement in the time bound is possible, and as with integer multiplication, it relies upon algebraic tricks. It turns out that XY can be computed from just seven sub problems. P1 = A(F H) P2 P3 P4 P5 P6 P7
= = = = = =
(A + B)H (C + D)E D(G E) (A + D) (E + H) (B - D) (G + H) (A - C) (E + F)
XY =
P +P P +P 5 4 2 6 P +P 3 4
P +P 1 2 P +P P P 1 5 3 7
This translates into a running time of T(n)= 7T(n/2) + O(n2), which by the result is O(nlog2 7 ) O(n2.81).
SJCET, Palai
57
STRESSENS MATRIX MULTIPLICATION

Strassen showed that 2x2 matrix multiplication can be accomplished in 7 multiplication and 18 additions or subtractions. This reduce can be done by Divide and Conquer Approach.Divide the input data S in two or more disjoint subsets S1, S2. Solve the subproblems recursively. Combine the solutions for S1, S2, , into a solution for S. The base case for the recursion are subproblems of constant size.Analysis can be done using recurrence equations.Divide matrices in sub-matrices and recursively multiply sub-matrices Let A, B be two square matrices over a ring R. We want to calculate the matrix product C as
If the matrices A, B are not of type 2n x 2n we fill the missing rows and columns with zeros. We partition A, B and C into equally sized block matrices
with
then
With this construction we have not reduced the number of multiplications. We still need 8 multiplications to calculate the Ci,j matrices, the same number of multiplications we need when using standard matrix multiplication. Now comes the important part. We define new matrices
SJCET, Palai
58
which are then used to express the Ci,j in terms of Mk. Because of our definition of the Mk we can eliminate one matrix multiplication and reduce the number of multiplications to 7 (one multiplication for each Mk) and express the Ci,j as
We iterate this division process n-times until the submatrices degenerate into numbers (group elements). Practical implementations of Strassen's algorithm switch to standard methods of matrix multiplication for small enough submatrices, for which they are more efficient. The particular crossover point for which Strassen's algorithm is more efficient depends on the specific implementation and hardware. It has been estimated that Strassen's algorithm is faster for matrices with widths from 32 to 128 for optimized implementations and 60,000 or more for basic implementations. Numerical analysis The standard matrix multiplications takes approximately 2n3 arithmetic operations (additions and multiplications); the asymptotic complexity is O(n3).
SJCET, Palai

N
59
Ci , j
k 1
ai ,k bk , j
N N N
Thus T ( N )
i 1 j 1 k 1
cN 3
O( N 3 )
The number of additions and multiplications required in the Strassen algorithm can be calculated as follows: let f(k) be the number of operations for a matrix. Then by
recursive application of the Strassen algorithm, we see that f(k) = 7f(k 1) + l4k, for some constant l that depends on the number of additions performed at each application of the algorithm. Hence f(k) = (7 + o(1))k, i.e., the asymptotic complexity for multiplying matrices of size n = 2k using the Strassen algorithm is
. The reduction in the number of arithmetic operations however comes at the price of a somewhat reduced numerical stability.
Algorithm
void matmul(int *A, int *B, int *R, int n) { if (n == 1) { (*R) += (*A) * (*B); } else { matmul(A, B, R, n/4);
SJCET, Palai
Algorithm Analysis and Design (R 606) matmul(A, B+(n/4), R+(n/4), n/4); matmul(A+2*(n/4), B, R+2*(n/4), n/4); matmul(A+2*(n/4), B+(n/4), R+3*(n/4), n/4); matmul(A+(n/4), B+2*(n/4), R, n/4); matmul(A+(n/4), B+3*(n/4), R+(n/4), n/4); matmul(A+3*(n/4), B+2*(n/4), R+2*(n/4), n/4); matmul(A+3*(n/4), B+3*(n/4), R+3*(n/4), n/4); } }
60
SJCET, Palai
61
MERGE SORT
Merge sort is an example of divide and conquer algorithm. Its worst case complexity is O(nlog n). To understand merge sort assume that elements are arranged in the non decreasing order. Assume a sequence of n elements a[1],a[2],...a[n]. Split them into two sets like a[1],a[2],..a[n/2] and a[(n/2)+1],...a[n]. Each set is individually sorted, and the resulting sorted sequences are merged to produce a single sorted sequence of n elements. Merge-sort is based on the divide-and-conquer paradigm. The Merge-sort algorithm can be described in general terms as consisting of the following three steps: 1. Divide Step: If given array A has zero or one element,it is already sorted. Otherwise, divide A into two arrays, each containing about half of the elements of A. 2. Recursion Step: Recursively sort array A1 and A2. 3. Conquer Step: Combine the elements back in A by merging the sorted arrays A1 and A2 into a sorted sequence. Consider the following example: Consider the array of ten elements a [1:10] = (310,285,179,652,351,423,861,254,450,520) The merge sort algorithm first divides the array as follows a[1:5] and a[6:10] and then a[1:3] and a[4:5] and a[6:8] and a[9:10] and then into a[1:2] and a[3:3] and a[6:7] and a[8:8] and finally it will look like a[1:1],a[2:2],a[3:3],a[4:4],a[5:5],a[6:6],a[7:7],a[8:8],a[9:9],a[10:10] Pictorially the file can now be viewed as (310|285|179|652,351|423,861,254,450,520) Elements a[1] and a[2] are merged as (285,310|179|652,351|423,861,254,450,520)
SJCET, Palai
Algorithm Analysis and Design (R 606) And then a[3] is merged with a[1:2] (179,285, 310 |652,351|423,861,254,450,520) next a[4] and a[5] (179,285, 310 |351,652|423,861,254,450,520)and then a[1:3] and a[4:5] (179,285, 310,351,652|423,861,254,450,520) Repeated recursive calls are invoked producing the following array (179,285, 310 ,351,652|254,423, 450,520,861) and finally (179,254,285,310, 351,423, 450,520,652,861) At this point there are two sorted subarrays and final merge produces the fully sorted result
62
1, 10
1, 5
6, 10
1, 3
4, 5
6, 8
9, 10
1, 2
3, 3
4, 4
5, 5
6, 7
8, 8
9, 9
10, 10
1, 1
2, 2
6, 6
7, 7
If the time for the merging operation is proportional to n, then the computing time for merge sort is described by the recurrence relation a n= 1a a constant SJCET, Palai
Algorithm Analysis and Design (R 606) T(n)= 2T(n/2)+cn n>1, c a constant
63
When n is a power of 2, n=2k we can solve this equation by successive substitutions: T(n) = 2(2T(n4)+cn/2)+cn = 4T(n/4)+2cn = 4(2T(n/8)+cn/4) + 2cn . . = 2kT(1) + kcn = an+cn log n It is easy to see that if 2k < n <2k+1, then T(n)<=T(2k+1). Therefore T(n)=O (nlogn). 24 MergeSort describes the process very succinctly using recursion and a function Merge which merges two sorted sets. Before executing MergeSort , the n elements should be placed in a[1:n]. Then MergeSort (1,n) causes the keys to be rearranged into nondecreasing order in a. Algorithm MergeSort(low,high) //a[low:high] is a global array to be sorted. //Small(P) is true if there is only one element to sort. //In this case the list is already sorted. { if(low<high) then // If there are more than one element. { //Divide P into subproblems.
SJCET, Palai
Algorithm Analysis and Design (R 606) //Find where to split the set . mid:=[(low+high)/2]; //Solve the subproblems. MergeSort(low,high); MergeSort(mid+1,high); //Combine the solutions. Merge(low,mid,high); } } Algorithm Merge(low,mid,high) //a[low:high] is a global array containing two sorted subsets in //a[low:mid] and in a[mid+1:high].The goal is to merge these two // sets into single set residing in a[low:high].b[] is an auxiliary global //array. { h:=low; i:=low; j:=mid+1; while ((h mid) and (j high)) do { if (a[h] a[j]) then { b[i]:=a[h]; h:=h+1; } else Department of Computer Science & Engineering
64
SJCET, Palai
Algorithm Analysis and Design (R 606) { b[i]:=a[j]; j:=j+1; } i:=i+1; } if(h>mid) then for k:=j to high do { b[i]:=a[k]; i:=i+1; } else for k:=h to mid do { b[i]:=a[k]; i:=i+1; } for k:=low to high do a[k]:=b[k]; }
65
SJCET, Palai
66
COMPLEXITY ANALYSIS OF MERGE SORT

Merge sort is an ideal example of the divide and conquer strategy in which the given set of elements is split into two equal-sized sets and the combining operation is the merging of two sorted sets into one. Given a sequence of n elements a[1],.....,a[n], the general idea is to imagine them to split into two sets a[1],...a[n/2] and a[n/2+1],...a[n].Each set is individually sorted and the resulting sorted sequences are merged to produce a single sorted sequence of n elements. MergeSort describes this process using recursion and a function Merge which merges two sorted sets. Example: Consider an array of ten elements a[1:10] = (310,285,179,652,351,423,861,254,450,520). Algorithm MergeSort begins by splitting a[] into two subarrays each of size five (a[1:5] and a[6:10]). The elements in a[1:5] are then split into two subarrays of size three (a[1:3]) and two (a[4:5]). Then the items in a[1:3] are split into subarrays of size two (a[1:2]) and one (a[3:3]). The two values in a[1:2] are split final time into one-element subarrays , and now the merging begins. A record of the subarrays is implicitly maintained by the recursive mechanism. Pictorially the file can be viewed as ( 310 | 285 | 179 | 652 , 351 | 423 , 861 , 254 , 450 , 520 ) where vertical bars indicate the boundaries of subarrays. Elements a[1] and a[2] are merged to yield ( 285 , 310 | 179 | 652 , 351 | 423 , 861 , 254 , 450 , 520 ) Then a[3] is merged with a[1:2] and ( 179 , 285 , 310 | 652 , 351 | 423 , 861 , 254 , 450 , 520 ) is produced. Next elements a[4] and a[5] are merged: ( 179 , 285 , 310 | 351 , 652 | 423 , 861 , 254 , 450 , 520 ) And then a[1:3] and a[4:5]: ( 179 , 285 , 310 , 351 , 652 | 423 , 861 , 254 , 450 , 520 ) Department of Computer Science & Engineering SJCET, Palai
67
At this point the algorithm has returned to the first invocation of MergeSort and is about to process the second recursive call. Repeated recursive calls are invoked producing the following subarrays: ( 179 , 285 , 310 , 351 , 652 | 423 | 861 | 254 | 450 , 520 )
Elements a[6] and a[7] are merged. Then a[8] is merged with a[6:7]: ( 179 , 285 , 310 , 351 , 652 | 254 , 423 , 861 | 450 , 520 )
Next a[9] and a[10] are merged and then a[6:8] and a[9:10]: ( 179 , 285 , 310 , 351 , 652 | 254 , 423 , 450 , 520 , 861 )
At this point there are two sorted subarrays and the final merge produces the fully sorted result ( 179 , 254 , 285 , 310 , 351 , 423 , 450 , 520 , 652 , 861 )
Figure (1.1) is a tree that represents the sequence of recursive calls that are produced by MergeSort when it is applied to ten elements. The pair of values in each node are the values of parameters low and high. The splitting continues until the sets containing a single element are produced. Figure (1.2) is a tree representing the calls to produce Merge by MergeSort. For example, the node containing 1 , 2 and 3 represents the merging of a[1:2] with a[3].
SJCET, Palai
68
1,10
1,5
6,10
1,3 1,2 1,1 2,2 3,3 4,4
4,5 5,5 6,6

Tree of calls of Merge ( Fig 1.1 )
6,8 6,7 7,7 8,8 9,9
9,10 10,10
1,1,2
6,6,7
1,2,3
4,4,5
6,7,8
9,9,10
1,3,5
6,8,10
1,5,10
Tree of calls of MergeSort(1,10) ( Fig 1.2 )
SJCET, Palai
69
If the time for the merging operation is proportional to n, then the computing time for merge sort is described by the recurrence relation
T(n) = { a { 2T(n/2) + cn
n=1 , a a constant n>1 , c a constant
When n is a power of 2, n = 2^k, we can solve this equation by successive substitutions:
T(n) = 2(2T(n/4)+ cn/2) +cn = 4T(n/4) + 2cn = 4(2T(n/8) + cn/4) + 2cn . . . = 2^kT(1) + kcn = an + cn logn
It is easy to see that if 2^k < n < 2^(k+1) , then T(n) < T(2^k+1).
T(n) = O(n logn)
SJCET, Palai
70
QUICK SORT
Quicksort is one of the fastest and simplest sorting algorithms . It works recursively by a divide-and-conquer strategy.
Idea
First, the sequence to be sorted a is partitioned into two parts, such that all elements of the first part b are less than or equal to all elements of the second part c (divide). Then the two parts are sorted separately by recursive application of the same procedure (conquer). Recombination of the two parts yields the sorted sequence (combine). Figure 1 illustrates this approach.
Figure 1: Quicksort(n)
The first step of the partition procedure is choosing a comparison element x. All elements of the sequence that are less than x are placed in the first part, all elements greater than x are placed in the second part. For elements equal to x it does not matter into which part they come. In the following algorithm it may also happen that an element equal to x remains between the two parts.
SJCET, Palai
71
After partitioning the sequence, quicksort treats the two parts recursively by the same procedure. The recursion ends whenever a part consists of one element only.
Program
The following Java program implements quicksort. void quicksort (int[] a, int lo, int hi) { // lo is the lower index, hi is the upper index // of the region of array a that is to be sorted int i=lo, j=hi, h; int x=a[(lo+hi)/2];
// partition do { while (a[i]<x) i++; while (a[j]>x) j--; if (i<=j) { h=a[i]; a[i]=a[j]; a[j]=h; i++; j--; } } while (i<=j);
// recursion if (lo<j) quicksort(a, lo, j); if (i<hi) quicksort(a, i, hi); }
Analysis
The best-case behavior of the quicksort algorithm occurs when in each recursion step the partitioning produces two parts of equal length. In order to sort n elements, in this case the
SJCET, Palai
72
running time is in (n log(n)). This is because the recursion depth is log(n) and on each level there are n elements to be treated (Figure 2 a). The worst case occurs when in each recursion step an unbalanced partitioning is produced, namely that one part consists of only one element and the other part consists of the rest of the elements (Figure 2 c). Then the recursion depth is n-1 and quicksort runs in time (n2). In the average case a partitioning as shown in Figure 2 b is to be expected. The choice of the comparison element x determines which partition is achieved. Suppose that the first element of the sequence is chosen as comparison element. This would lead to the worst case behavior of the algorithm when the sequence is initially sorted. Therefore, it is better to choose the element in the middle of the sequence as comparison element. Even better would it be to take the n/2-th greatest element of the sequence (the median). Then the optimal partition is achieved. Actually, it is possible to compute the median in linear time [AHU 74]. This variant of quicksort would run in time O(n log(n)) even in the worst case. However, the beauty of quicksort lies in its simplicity. And it turns out that even in its simple form quicksort runs in O(n log(n)) on the average. Moreover, the constant hidden in the Onotation is small. Therefore, we trade this for the (rare) worst case behavior of (n2). Proposition: The time complexity of quicksort is in (n log(n)) (n2) In the average case and in In the worst case
Conclusions
Quicksort turns out to be the fastest sorting algorithm in practice. It has a time complexity of (n log(n)) on the average. However, in the (very rare) worst case quicksort is as slow as Bubblesort, namely in (n2). There are sorting algorithms with a time complexity of O(n log(n)) even in the worst case, e.g. Heapsort and Mergesort. But on the average, these algorithms are by a constant factor slower than quicksort. It is possible to obtain a worst case complexity of O(n log(n)) with a variant of quicksort (by choosing the median as comparison element). But this algorithm is on the average and in the
SJCET, Palai
73
worst case by a constant factor slower than Heapsort or Mergesort; therefore, it is not interesting in practice. The quicksort algorithm was invented by C.A.R. Hoare in 1962. I learned about it from "The C Programming Language, Second Edition" by Brian W. Kernighan and Dennis M. Ritchie (page 87). I found the K&R code a bit hard to follow; I believe I've come upon a better way of explaining the algorithm, using a few figures of speech. The point of any sort routine is to take an array of objects and arrange the objects in some kind of sequence. We might, for example, want to take this collection of numbers:
06 34 69 33 75 64 04 74 25 95 15 58 78 36 51 73 13 27
and put them into numerical order, with the smallest number first, like this:
04 06 13 15 25 27 33 34 36 51 58 64 69 73 74 75 78 95
The general approach of the quick sort is to select an element from the middle (or from close to the middle) of an array and then put all other elements which are less than or equal to the selected element to its left and all elements which are greater than the selected element to its right. The selected element is called the "pivot element" because the other elements, figuratively speaking, turn around it. This crude "sorting" around the pivot element yields two sub-arrays: a left one and a right one. If, for example, we start with the original array above and use "25" as the pivot element, our first sorting yields:
[ 04 06 15 13 ] 25 [ 64 34 74 69 95 33 58 78 36 51 73 75 27 ]
The next step is for quicksort to call itself to have the left and right sub-arrays sorted. Of course, a "sub array" with zero elements or only one element is already in proper order and does not need to be sorted. The interesting part of the quick sort is how it comes up with the sub arrays. I will illustrate. From our original array, take out the element in (or close to) the middle of the array for a pivot element. This, figuratively speaking, leaves a hole in the array:
SJCET, Palai
Algorithm Analysis and Design (R 606) 06 34 69 33 75 64 04 74 __ 95 15 58 78 36 51 73 13 27
74
p.e. = 25
Now "move the hole" to the first position in the array by moving the current occupant of that position into the current hole's position:
__ 34 69 33 75 64 04 74 06 95 15 58 78 36 51 73 13 27
p.e. = 25
Now we get to work. We start with the first element to the right of the hole. If it is greater than the pivot element, we do nothing. If it is less than or equal to the pivot element, then we want to put the element in the hole and then re-establish the hole immediately to the right of the moved element. The first element of our sample array that needs to be moved is "04":
+-----------------+ | V | |
04 34 69 33 75 64 __ 74 06 95 15 58 78 36 51 73 13 27
+--------------+ | | | V
04 _ 69 33 75 64 34 74 06 95 15 58 78 36 51 73 13 27
p.e. = 25
After we similarly process "06" we will have: 04 06 __ 33 75 64 34 74 69 95 15 58 78 36 51 73 13 27
p.e. = 25
At the end of the process, we have: Department of Computer Science & Engineering SJCET, Palai
75
04 06 15 13 __ 64 34 74 69 95 33 58 78 36 51 73 75 27
p.e. = 25
As you can see, all elements to the left of the hole are less than the pivot element; all elements to the right of the hole are greater than the pivot element. To finish up, we sort the left sub-array then the right sub-array, then we stuff the pivot element back into the hole and we are done.
Algorithm partition (a,m,p) //Within a[m], a[m+1],,a[p-1]the elements are rearranged in such //a manner that if initially t=a[m], then after completion a[q] =t for //some q between m and p-1,a[k]<=t form<=k<q, and a[k]>=t for //q<k<p. q is returned. Set a[p]=. { v :=a[m]; I :=m; j :=p; repeat { repeat I:=i+1; until ( a[i]>=v); repeat j:=j-1; until(a[j]<=v);
SJCET, Palai
Algorithm Analysis and Design (R 606) if(i<j) then interchange (a,i,j); }until(i>=j); a[m]:=a[j]; a[j]:=v; return j; } Algorithm interchange (a,i,j) // exchange a[i] with a[j]. { p :=a[i]; a[i] :=a[j]; a[j] :=p; }
76
In quick sort, the division into sub arrays is made so that the sorted sub arrays do not need to be merged later. This is accomplished by rearranging the elements in a[1:n] such that a[i]<=a[j] for all I between 1 and m and all j between m+1 and n for some m,1<=m<=n. Thus the elements in a[1:m] and a[m+1:n] can be independently sorted. No merge is needed. The rearrangement of the element is accomplished by picking some element of a[],say t =a[s],and then reordering the other elements so that all elements appearing before t in a[1:n] are less than or equal to t and all elements appearing after t are greater than or equal to t. This is rearranging is referred to as partitioning. Function Partition of Algorithm accomplishes an in-place partitioning of the elements of a[m:p-1].It is assumed that a[p]>=a[m] and that a[m] is the partitioning elements. If m=1 and p-1=n, then a[n+1] must be defined and must be greater than or equal to all elements in a[1:n]. The assumption that a[m] is the partition element is merely for convenience; other choices for the partitioning element than the first item in the set are better in practice. The function Interchange (a, i, j) exchanges a[i] with a[j]. Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) Algorithm Quick Sort (p,q)
77
//Sorts the elements a[p],,a[q] which reside in the global array //a[1:n] into ascending order; a[n+1] is considered to be defined and //must be >= all the elements in a[1:n]. { if (p<q) then //If there are more than one element { // divide P into two subproblems. j:=Partition(a,p,q+1); //j is the position of the partitioning element. // Solve the subproblems. QuickSort (p,j-1); Quick Sort(j+1,q); // There is no need for combining solutions. } }
Using Hoares clever method of partitioning a set of elements about a chosen element, we can directly devise a divide-and-conquer method for completely sorting n elements. Following a call to the function Partition, two sets S1 and S2 are produced. All elements in S1 are less than or equal to the element in S2. Hence S1 and S2 can be sorted independently. Each set is sorted by reusing the function Partition
SJCET, Palai
78
Quicksort
Quicksort
Quicksort in action on a list of numbers. The horizontal lines are pivot values.
Class
Sorting algorithm
Data structure
Varies
Worst case performance
(n2)
Best case performance
(nlogn)
Average case performance
(nlogn) comparisons
Worst case space complexity
Varies by implementation
Optimal
Sometimes
Quicksort is a well-known sorting algorithm developed by C. A. R. Hoare that, on average, makes (nlogn) (big O notation) comparisons to sort n items. However, in the worst case, it makes (n2) comparisons. Typically, quicksort is significantly faster in practice than other (nlogn) algorithms, because its inner loop can be efficiently implemented on most architectures, and in most real-world data, it is possible to make design choices which minimize the probability of requiring quadratic time. Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) Quicksort is a comparison sort and, in efficient implementations, is not a stable sort.
79
Algorithm
Quicksort sorts by employing a divide and conquer strategy to divide a list into two sub-lists. The steps are: 1. Pick an element, called a pivot, from the list. 2. Reorder the list so that all elements which are less than the pivot come before the pivot and so that all elements greater than the pivot come after it (equal values can go either way). After this partitioning, the pivot is in its final position. This is called the partition operation. 3. Recursively sort the sub-list of lesser elements and the sub-list of greater elements. The base case of the recursion are lists of size zero or one, which are always sorted. In simple pseudocode, the algorithm might be expressed as this:
function quicksort(array) var list less, greater if length(array) 1 return array select and remove a pivot value pivot from array for each x in array if x pivot then append x to less else append x to greater return concatenate(quicksort(less), pivot, quicksort(greater))
Notice that we only examine elements by comparing them to other elements. This makes quicksort a comparison sort. This version is also a stable sort (assuming that the "for each" method retrieves elements in original order, and the pivot selected is the last among those of equal value). The correctness of the partition algorithm is based on the following two arguments:
SJCET, Palai
80
At each iteration, all the elements processed so far are in the desired position: before the pivot if less than or equal to the pivot's value, after the pivot otherwise (loop invariant). Each iteration leaves one fewer element to be processed (loop variant). The correctness of the overall algorithm follows from inductive reasoning: for zero or one element, the algorithm leaves the data unchanged; for a larger data set it produces the concatenation of two parts, elements less than or equal to the pivot and elements greater than it, themselves sorted by the recursive hypothesis. The disadvantage of the simple version above is that it requires (n) extra storage space, which is as bad as merge sort. The additional memory allocations required can also drastically impact speed and cache performance in practical implementations. There is a more complex version which uses an in-place partition algorithm and can achieve the complete sort using O(nlogn) space use on average (for the call stack):
function partition(array, left, right, pivotIndex) pivotValue := array[pivotIndex] swap array[pivotIndex] and array[right] // Move pivot to end storeIndex := left for i from left to right 1 if array[i] pivotValue swap array[i] and array[storeIndex] storeIndex := storeIndex + 1 swap array[storeIndex] and array[right] // Move pivot to its final place return storeIndex
In-place partition in action on a small list. The boxed element is the pivot element, blue elements are less or equal, and red elements are larger. This is the in-place partition algorithm. It partitions the portion of the array between indexes left and right, inclusively, by moving all elements less than or equal to a[pivotIndex] to the beginning of the subarray, leaving all the greater elements following them. In the process it also finds the final position for the pivot element, which it returns. It temporarily moves the pivot element to the end of the subarray, so that it doesn't get in the way. Because it only uses exchanges, the final list has the same elements as the original list. Notice that an element may be exchanged multiple times before reaching its final place. Department of Computer Science & Engineering SJCET, Palai
81
This form of the partition algorithm is not the original form; multiple variations can be found in various textbooks, such as versions not having the storeIndex. However, this form is probably the easiest to understand. Once we have this, writing quicksort itself is easy:
procedure quicksort(array, left, right) if right > left select a pivot index (e.g. pivotIndex := left) pivotNewIndex := partition(array, left, right, pivotIndex) quicksort(array, left, pivotNewIndex - 1) quicksort(array, pivotNewIndex + 1, right)
However, since partition reorders elements within a partition, this version of quicksort is not a stable sort.
Formal analysis
From the initial description it's not obvious that quicksort takes (nlogn) time on average. It's not hard to see that the partition operation, which simply loops over the elements of the array once, uses (n) time. In versions that perform concatenation, this operation is also (n). In the best case, each time we perform a partition we divide the list into two nearly equal pieces. This means each recursive call processes a list of half the size. Consequently, we can make only logn nested calls before we reach a list of size 1. This means that the depth of the call tree is (logn). But no two calls at the same level of the call tree process the same part of the original list; thus, each level of calls needs only (n) time all together (each call has some constant overhead, but since there are only (n) calls at each level, this is subsumed in the (n) factor). The result is that the algorithm uses only (nlogn) time. An alternate approach is to set up a recurrence relation for the T(n) factor, the time needed to sort a list of size n. Because a single quicksort call involves (n) factor work plus two recursive calls on lists of size n / 2 in the best case, the relation would be: The master theorem tells us that T(n) = (nlogn).
SJCET, Palai
82
In fact, it's not necessary to divide the list this precisely; even if each pivot splits the elements with 99% on one side and 1% on the other (or any other fixed fraction), the call depth is still limited to 100logn, so the total running time is still (nlogn). In the worst case, however, the two sublists have size 1 and n 1 (for example, if the array consists of the same element by value), and the call tree becomes a linear chain of n nested calls. The ith call does (n i) work, and . The recurrence relation is: T(n) = (n) + T(0) + T(n 1) = O(n) + T(n 1) This is the same relation as for insertion sort and selection sort, and it solves to T(n) = (n2). Given knowledge of which comparisons are performed by the sort, there are adaptive algorithms that are effective at generating worst-case input for quicksort on-the-fly, regardless of the pivot selection strategy.
Randomized quicksort expected complexity

Randomized quicksort has the desirable property that it requires only (nlogn) expected time, regardless of the input. Suppose we sort the list and then divide it into four parts. The two parts in the middle will contain the best pivots; each of them is larger than at least 25% of the elements and smaller than at least 25% of the elements. If we could consistently choose an element from these two middle parts, we would only have to split the list at most 2log2n times before reaching lists of size 1, yielding an (nlogn) algorithm. A random choice will only choose from these middle parts half the time. However, this is good enough. Imagine that you are flipping a coin over and over until you get k heads. Although this could take a long time, on average only 2k flips are required, and the chance that you won't get k heads after 100k flips is highly improbable. By the same argument, quicksort's recursion will terminate on average at a call depth of only 2(2log2n). But if its average call depth is (logn), and each level of the call tree processes at most n elements, the total amount of work done on average is the product, (nlogn). Note that the algorithm does not have to verify that the pivot is in the middle half - if we hit it any constant fraction of the times, that is enough for the desired complexity.
SJCET, Palai
83
The outline of a formal proof of the O(nlogn) expected time complexity follows. Assume that there are no duplicates as duplicates could be handled with linear time pre- and postprocessing, or considered cases easier than the analyzed. Choosing a pivot, uniformly at random from 0 to n 1, is then equivalent to choosing the size of one particular partition, uniformly at random from 0 to n 1. With this observation, the continuation of the proof is analogous to the one given in the average complexity section.
SJCET, Palai
84
Average complexity
Even if pivots aren't chosen randomly, quicksort still requires only (nlogn) time over all possible permutations of its input. Because this average is simply the sum of the times over all permutations of the input divided by n factorial, it's equivalent to choosing a random permutation of the input. When we do this, the pivot choices are essentially random, leading to an algorithm with the same running time as randomized quicksort. More precisely, the average number of comparisons over all permutations of the input sequence can be estimated accurately by solving the recurrence relation: Here, n 1 is the number of comparisons the partition uses. Since the pivot is equally likely to fall anywhere in the sorted list order, the sum is averaging over all possible splits. This means that, on average, quicksort performs only about 39% worse than the ideal number of comparisons, which is its best case. In this sense it is closer to the best case than the worst case. This fast average runtime is another reason for quicksort's practical dominance over other sorting algorithms.
Space complexity
The space used by quicksort depends on the version used. Quicksort has a space complexity of (logn), even in the worst case, when it is carefully implemented such that in-place partitioning is used. This requires (1). After partitioning, the partition with the fewest elements is (recursively) sorted first, requiring at most (logn) space. Then the other partition is sorted using tail recursion or iteration. The version of quicksort with in-place partitioning uses only constant additional space before making any recursive call. However, if it has made (logn) nested recursive calls, it needs to store a constant amount of information from each of them. Since the best case makes at most (logn) nested recursive calls, it uses (logn) space. The worst case makes (n) nested recursive calls, and so needs (n) space; Sedgewick's improved version using tail recursion requires (logn) space in the worst case. Department of Computer Science & Engineering SJCET, Palai
85
We are eliding a small detail here, however. If we consider sorting arbitrarily large lists, we have to keep in mind that our variables like left and right can no longer be considered to occupy constant space; it takes (logn) bits to index into a list of n items. Because we have variables like this in every stack frame, in reality quicksort requires ((logn)2) bits of space in the best and average case and (nlogn) space in the worst case. This isn't too terrible, though, since if the list contains mostly distinct elements, the list itself will also occupy (nlogn) bits of space. The not-in-place version of quicksort uses (n) space before it even makes any recursive calls. In the best case its space is still limited to (n), because each level of the recursion uses half as much space as the last, and Its worst case is dismal, requiring space, far more than the list itself. If the list elements are not themselves constant size, the problem grows even larger; for example, if most of the list elements are distinct, each would require about O(logn) bits, leading to a best-case (nlogn) and worst-case (n2logn) space requirement.
Variants
There are three variants of quicksort that are worth mentioning: Balanced quicksort: choose a pivot likely to represent the middle of the values to be sorted, and then follow the regular quicksort algorithm. External quicksort: The same as regular quicksort except the pivot is replaced by a buffer. First, read the M/2 first and last elements into the buffer and sort them. Read the next element from the beginning or end to balance writing. If the next element is less than the least of the buffer, write it to available space at the beginning. If greater than the greatest, write it to the end. Otherwise write the greatest or least of the buffer, and put the next element in the buffer. Keep the maximum lower and minimum upper keys written to avoid resorting middle elements that are in order. When done, write the buffer. Recursively sort the smaller partition, and loop to sort the remaining partition.
SJCET, Palai
86
Three-way radix quicksort (also called multikey quicksort): is a combination of radix sort and quicksort. Pick an element from the array (the pivot) and consider the first character (key) of the string (multikey). Partition the remaining elements into three sets: those whose corresponding character is less than, equal to, and greater than the pivot's character. Recursively sort the "less than" and "greater than" partitions on the same character. Recursively sort the "equal to" partition by the next character (key).
Comparison with other sorting algorithms

Quicksort is a space-optimized version of the binary tree sort. Instead of inserting items sequentially into an explicit tree, quicksort organizes them concurrently into a tree that is implied by the recursive calls. The algorithms make exactly the same comparisons, but in a different order. The most direct competitor of quicksort is heapsort. Heapsort is typically somewhat slower than quicksort, but the worst-case running time is always (nlogn). Quicksort is usually faster, though there remains the chance of worst case performance except in the introsort variant. If it's known in advance that heapsort is going to be necessary, using it directly will be faster than waiting for introsort to switch to it. Quicksort also competes with mergesort, another recursive sort algorithm but with the benefit of worst-case (nlogn) running time. Mergesort is a stable sort, unlike quicksort and heapsort, and can be easily adapted to operate on linked lists and very large lists stored on slow-to-access media such as disk storage or network attached storage. Although quicksort can be written to operate on linked lists, it will often suffer from poor pivot choices without random access. The main disadvantage of mergesort is that, when operating on arrays, it requires (n) auxiliary space in the best case, whereas the variant of quicksort with in-place partitioning and tail recursion uses only (logn) space. (Note that when operating on linked lists, mergesort only requires a small, constant amount of auxiliary storage.)
SJCET, Palai
87
MODULE 3
SJCET, Palai
88
THE GREEDY METHOD CONTROL ABSTRACTION

The greedy method suggests that one can devise an algorithm that works in stages, considering one input at a time. At each stage, a decision is made regarding whether a particular input is in an optimal solution. This is done by considering the inputs in an order determined by some selection procedure. If the inclusion of the next input into the partially constructed optimal solution will result in an infeasible solution, then this input is not assed to the partial solution. Otherwise, it is added. The selection procedure itself is based on some optimization measure. The measure may be the objective function. In fact, several different optimization measures may be plausible for a given problem. Most of these, however, will result in algorithms that generate suboptimal solutions. This version of the greedy technique is called the subset paradigm. The function Select selects an input from a[ ] and removes it. The selected inputs value is assigned to x. Feasible is a Boolean-valued function that determines whether x can be included into the solution vector. The function Union combines x with the solution and updates the objective function. The function Greedy describes the essential way that a greedy algorithm will look, once a particular problem is chosen and the functions Select, Feasible, and Union are properly implemented. Algorithm Greedy (a,n) //a[1 : n] contains the n inputs. { Solution :=;// Initialize the solution.
for i:=1 to n do { x:=Select(a); if Fesible(solution,x) then solution := Union(solution,x); } return solution;
}
Department of Computer Science & Engineering SJCET, Palai
89
GENERAL KAPSACK PROBLEM

In the knapsack problem, we have n objects and a knapsack or a bag. Object i has a weight wi and the knapsack has a capacity m. If a fraction xi , 0 xi 1, of object i is placed into the knapsack, then a profit of pixi is earned. The objective is to obtain a filling of the knapsack that maximizes the total profit earned. Since the knapsack capacity is m, we require the total weight of all chosen objects to be at most m. Formally, the problem can be stated as maximize pixi 1 in Subject to wixi m 1i n And 0 xi 1, 1i n
The profit and weight are positive numbers. This is done by using greedy method.
ALGORITHM ALGORITHM greedyknapsack(m, n ) / / p[ 1.. n] , w[ 1.. n] / / pi/ wi pi+1/ wi+1 / / x[1 . n] solution { for i=1 to n do x[1] =0.0 ; u=m ; for i= 1 to n do { If ( w[i] > u) then break ; x[i] = 1.0 ; u = u w[i] ; } If ( i n ) then x[i] = u/ w[i] ; } Department of Computer Science & Engineering SJCET, Palai
90
If p1 / w1 p2 /w2 . pn / wn , then greedy knapsack generate an Optimum solution to the given instant of knapsack problem. All optimum solutions will fill the knapsack exactly complexity of knapsack algorithm is 0( n).
Greedy algorithms
Similarly to dynamic programming, greedy algorithms are used to solve optimization problems. In contrast to dynamic programming, no global optimal solution is computed, but rather locally optimal decisions are taken. That is why the approach is called greedy.
Example: Knapsack Problem I

2 possible greedy-strategies: select the most valuable element in each step that still fits into the knapsack: Knapsack capacity: 15 Object Size Value g1 3 3 g2 4 5 g3 6 8 g4 7 9 choose: g4; value: 9; remaining capacity: 8 choose: g4; value: 9; remaining capacity: 1 y not optimal
Example: Knapsack Problem II

select the relatively most valuable element (max( v(gi )s(gi ) ) ) This will lead to the optimal solution in the case described above. However, this need not always be the case! Knapsack capacity: 50 Object Size Value vs g1 30 90 3 g2 40 100 2,5 g3 50 110 2,2 The strategy will choose g1 (value 90). Department of Computer Science & Engineering SJCET, Palai
91
However, the optimal would be g3 (value 110) Greedy algorithms do not always provide the optimal solution. In exchange, they are far less expensive than dynamic programming. Complexity of the greedy-knapsack: O(C)
Optimal greedy algorithms

There are problems where greedy algorithms produce optimal solutions. Example: resource planning problem. Given: set of activities S = {1, . . . , n} that want to use some resource, for example a lecture hall. The resource can only be used by one activity at a time. Each activity i has a starting time si and a completion time fi with si < fi . The activity takes place in the time interval [si , fi ), i.e., the next activity can start at time fi . Compatibility of activities: compatible(i , j) :, si _ fj _ sj _ fi . Problem: compute the largest possible set of compatible Activitie
OPTIMAL STORAGE ON TAPES
Optimal storage on tapes I

Given: n programs P1, . . . , Pn with lengths l1, . . . , ln. The programs are stored on a tape in order Pi1 , . . . , Pin . Time to access Program Pij : T(i1, . . . , in, j) = li1 + + lij . Average access time: T(i1, . . . , in) = 1n Xn
j=1
T(i1, . . . , in, j) = 1n Xn
j=1
Xj
i=1
li j = 1n Xn
j=1
(n j + 1)lij
Optimal storage on tapes II

How to store the programs on the tape so that the average access time becomes minimal? Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) Let n be fixed; hence 1n can be ignored. Example: Let P1, P2, P3 have lengths 17, 5, 10. order acess time P1P2P3 17 + (17 + 5) + (17 + 5 + 10) = 71 P1P3P2 17 + (17 + 10) + (17 + 10 + 5) = 76 P2P1P3 59 P3P1P2 69 P2P3P1 52 P3P2P1 52
92
Optimal storage on tapes III

Fact: If for the lenghts l1, . . . , ln the relation l1 _ l2 _ _ ln holds, then the average access time is minimized by the order P1 . . . Pn. Hence, a greedy strategy that always selects the shortest program leads to the minimal average access time.
What have we learnt on greedy algorithms?

Greedy algorithms treat optimization problems by taking locally optimal decisions. Hence, they may fail to determine the global optimum. Greedy algorithms need less time than, e.g., dynamic programming. There are problems where a greedy strategy leads to an optimal solution.
SPANNING TREES-MINIMUM COST SPANNING TREES

Definition: Let G= (V, E) be an undirected connected graph. A graph t= (V, E) of G is a spanning tree of G if T is a tree. Spanning trees have many applications. For example, they can be used to obtain independent set of circuit equation for an electric network. Another application of spanning trees arises from the property that a spanning tree is a minimal subgraph G of G such that V(G) =V(G) and G is connected. A minimal subgraph is one with fewest numbers of edges. Any connected graph with n vertices must have least n-1 edges and all connected graphs with n-1 edges are trees. If the nodes of G represent cities and the edges represent possible communication links connecting two cities, Department of Computer Science & Engineering SJCET, Palai
93
then the minimum number of links needed to connect the cities is n-1. The spanning trees of G represent all feasible choices.
In practical situations, the edges have weights assigned to them. These weights may represent the cost of construction, the length of the link, and so on. Given such weighted graph, one would then wish to select cities to have minimum total cost or minimum total length. In either case the links selected have to form a tree. If this is not so, then selection of links contain cycle. Removal of any of the links on this cycle results in a link selection of less cost connecting all the cities. So that the minimum cost spanning tree of G can be obtained. The cost of the spanning tree is the sum of all the edges in that tree.
28 8 2 14 16
10
7 6 24 18 25 5 4
12
22
(a)
SJCET, Palai
94
10 14
2 16
7 6
25
5 4
12
22
(b)
Figure (a) represents a graph and (b) represents the minimum cost spanning tree.
PRIMS ALGORITHM
At first a peak is chosen in random order ,which for simplicity we accept it as V(1).This way two sets of pointers are initialized ,the 0={1} and P={2...n}. The O set (the O is taken from the Greek word Oristiko which means Terminal), will always contain the pointers of those peaks which are terminally attached in the T tree. The V(1) peak has already been attached in the T tree. The P set( P is taken from the Greek word Prosorino which means Temporary) contains the rest of the pointers for the peaks, P={1...n}-O which are those pointers who have not been terminally connected with a node of T, that means they are not attached in the tree. In every execution of the Prim Algorithm a new peak will be connected to the T tree, not always with their numbering order, for example the V(4) peak can be connected to the tree before the V(2) peak. The corresponding pointer of the newly connected peak will be deleted
SJCET, Palai
95
from P set and will be inserted to the O set. When all peaks are connected there will be O={1,...n} and P=0.This of course means the end of the algorithm. The new peak every time will be chosen by using greedy method ,among all sides of G which connect peaks already inserted in the T (pointers in the O set ) tree with the rest of the peaks (pointers in the P set ),we choose one with minimum cost. If the chosen one is e(ij) then i belongs in the O set , V(i) peak is already in the T tree, j belongs in the P set , and V(j) peak has not been attached in the T tree yet. We put V(j) in the T tree, we change the O set by putting the j pointer, and we also change the P set by removing the j pointer. This may seem to you extremely complicated but it is easily understood by a set of examples. Pseudocode For The Prim Algorithm. INPUT :n,c[e(ij)],i,j belonging to {1,...,n}. OUTPUT :p(j) j=2,...,n (pointer of peaks j father in the T tree). STEPS 1. :(initializations). O={1} (V(1) root of the T tree). P={2,...,n} For every j belonging to P :e(j):=c[e(j1)] , p(j)=1 ( all peaks connected to the root. By definition of the cost function :e(j)=infinite when V(j) does not connect to V(1).). 2. Choose a k for which e(k)<=e(j) for every j belonging to P In case of tight choose the smaller one.
Exchange the O set with the set produced by the union of the O set and {k} . Exchange the P set with the set produced by the difference of the P set and {k} .(P<-P-{k}) If P=0 then stop. 3. For every j belonging to P compare e(j) with c[e(kj)]. If e(j) >c[e(kj)] exchange e(j) <-c(e(kj)).Go back to Step 1.
SJCET, Palai
Algorithm Analysis and Design (R 606) An example for Prims algorithm
96
shortest paths from v0 to all destinations
SJCET, Palai
97
1 2
5 4
(a)
10
7 6
5 4
SJCET, Palai
98
10
7 6
5 4
(c)
12
10 14
7 6
5 4
(d)
12
SJCET, Palai
99
10 14
2 16
7 6
5 4
(e)
12
10 14
2 16
7 6
5 4
12
22
(f)
SJCET, Palai
100
Figure (a) shows the current graph with no edges selected. Edge (1, 6) is the first edge considered. It is included in the spanning tree being built. This yields the graph of fig (b). Next edge (3, 4) is selected and included in the tree (fig(c)). The next edge to be considered is (2, 7). Its inclusion in tree is being built does not create a cycle, so we get the graph of fig (d). Edge (2, 3) is considered next and included in the home fig (e). Of the edges not yet considered, (7, 4) has the least cost. It is considered next. Its inclusion in the tree results in a cycle, so this edge is discarded. Edge (5,4) is next edge to be added to the tree being built. This results in the configuration of fig (f). The next edge to be considered is (7, 5). It is discarded as it creates a cycle. Finally, edge (6, 5) is considered and included in the tree being built. This completes the spanning tree. The resulting tree has the cost 99.
KRUSKALS ALGORITHM
ALGORITHM ALGORITHM Kruskal ( E, cost, n, t) / / E is the set of edges in G. G has n vertices. Cost [u, v] is the cost of / / edge (u, v). t is the set of edges in the minimum cost spanning tree. / / the final cost is returned. { Construct a heap out of the edge costs using Heapify ; for i := n do parent [i] := -1 ; / / Each vertex is in a different set. i := 0 ; mincost := 0.0 ; while ( i < n 1 ) and ( heap not empty )) do { Delete a minimum cost edge (u, v) from the heap and reheapify using Adjust ; j := Find (u) ; k := Find (v) ; if (j k) then { i := i+1 ; t[ i, 1] := u ; t [i, 2 ] := v ; mincost := mincost + cost [u, v] ; Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) Union (j, k) ; } } If ( i n 1 ) then write ( no spanning tree) ; else return mincost; }
101
COMPLEXITY The set t is the set of edges to be included in the minimum cost spanning tree and i is the number of edges in t. The set t can be represented as a sequential list using a two dimensional array t [ 1: n 1, 1 : 2]. Edge (u, v) can be added to t by the assignments t[ i, 1] := u ; and y[ i, 2] := v; . In the while loop of line 10, edges are removed from the heap one by one in nondecreasing order of cost. Line 14 determines the sets containing u and v. If j k then vertices u and v are in different sets and edge (u, v) is included into t. The sets containing u and v are combined in line 20. If u = v the edge (u, v) is discarded as its inclusions into t would create a cycle. Line 23 determines whether a spanning tree was found. It follows that i n 1 iff the graph G is not connected. The computing time is O( | E | log |E|), where e is the edge set of G.
JOB SEQUENCING WITH DEADLINES

If we are given a set of n jobs. Associated with job i is integer deadline di 0 and a profit pi>0. For any job i the profit pi is earned if the job is completed by its deadline. To complete a job one has to process the job on the machine for one unit of time. Only one machine is available for processing jobs. A feasible solution for this problem is a subset J of such that each job in this can be completed by its deadline. The value of a feasible solution J is the sum of the profits of the jobs in J, or iJ pi an optimal solution is a feasible solution with maximum value.
SJCET, Palai
102
Example:- Let n=4,(p1,p2,p3,p4)=(100,10,15,27) and (d1,d2,d3,d4)=(2,1,2,1). The feasible solutions and their values are: No 1. 2. 3. 4. 5. 6. 7. 8. 9. Feasible solution (1,2) (1,3) (1,4) (2,3) (3,4) (1) (2) (3) (4) Processing sequence 2,1 1,3 or 3,1 4,1 2,3 4,3 1 2 3 4 Value 110 115 127 25 42 100 10 15 27
Solution 3 is optimal. In this solution only jobs 1 and 4 are processed and the value is 127. These jobs must be processed in the order job 4 followed by job 1. Thus the processing of job begins at time zero and that of job 1 is completed at time 2. High level description of job sequencing algorithm Algorithm Greedy job(d,J,n) //J is a set of jobs that can be completed by their deadlines. { J:={1}; for i:=2 to n do { if(all jobs in J U {i} can be completed by their headlines) then J:=J U {i}; } }
SJCET, Palai
Algorithm Analysis and Design (R 606) Greedy algorithm for sequencing unit time jobs with deadlines and profits Algorithm JS(d,j,n) //d[i]1, 1in are the deadlines,n1. The jobs are ordered such that p[1]p[2]...p[n]. //J[i] is the ith job in the optimal solution,1ik. //Also at termination d[J][i]]d[J[i+1]],1ik. { d[0] :=J[0]:=0;//initialize. J[1]:=1;// Include job 1. k:=1; for i:=2 to n do { //consider jobs in nonincreasing order of p[i]. // Find position for i and check feasibility of insertion. r:=k; while((d[J[r]]>d[i]) and (d[J][r]]r)) do r:=r-1; if((d[J[r]]d[i]) and (d[i]>r)) then { //Insert i into J[ ]. for q:=k to (r+1) step-1 do J[q+1]:=J[q]; J[r+1]:=i; k:=k+1; } } return k; }
103
The fuction JS is a correct implemention of the greedy-based method. Since d[i]1,the job with largest pi will always be in the greedy solution. As the jobs are in the nonincreasing Department of Computer Science & Engineering SJCET, Palai
104
order of pis,line 6 in the above algorithm includes the job with largest pi. The for loop of the line 8 considers the remaining job in the jobs in the order required by the greedy algorithm method. At all times, the set of jobs already included in the solution is maintained in J. If J[i],1ik, is the set already included,then J is such that d[J][i]]d[J[i+1]],1ik. When job i is being considered, the while loop determines where in J this job has to be inserted. The worst case computing time of Algorithm JS is (n2). The computing time of JS can be reduced from (n2) to nearby O(n2) by using disjoint set union and find algorithms and a different method to determine the feasibility of a partial solution.
SJCET, Palai
105
MODULE 4
SJCET, Palai
106
DYNAMIC PROGRAMMING
Dynamic programming is an algorithm design method that can be used when the solution to a problem can be viewed as the result of a sequence of decisions.Below are some of the examples of problems that can be viewed this way. 1)KNAPSACK-The solution to the knapsack problem can be viewed as the result of a sequence of decisions.We have to decide the values of xi,1 i n.An optimal sequence of decisions maximizes the objective function pixi.It also satisfies the constraints wixim and 0 xi 1. 2)OPTIMAL MERGE PATTERNS-An optimal merge pattern tells us which pair of files should be merged at each step.An optimal sequence of decisions is a least cost sequence. 3)SHORTEST PATH-One way to find a shortest path from vertex i to vertex j in a directed graph G is to decide which vertex should be the second vertex,which the third and so on.An optimal sequence of decisions is one that results in a path of least length.
For some of the problems that may be viewed in this way,an optimal sequence of decisions can be found by making the decisions one at a time and never making an erroneous decision.This is true for all problems solvable by the greedy method.For many problems it is not possible to make stepwise decisions in such a manner that the sequence of decisions made is optimal.
One way to solve problems for which it is not possible to make a sequence of stepwise decisions leading to an optimal decision sequence is to try all possible decision sequences.We could enumerate all decision sequences and then pick out the best.But the time and space requirements may be prohibitive.Dynamic programming often drastically reduces the amount of enumeration by avoiding the enumeration of some decision sequences that cannot possibly be optimal.In dynamic programming an optimal sequence of decisions is obtained by making explicit appeal to the principle of optimality.
PRINCIPLE OF OPTIMALITY The principle of optimality states that an optimal sequence of decisions has the property that whatever the initial state and decisions are,the remaining decisions must constitute an optimal decision sequence with regard to the state resulting from the first decision.
SJCET, Palai
107
Thus, the essential difference between the greedy method and dynamic programming is that in the greedy method only one decision sequence is ever generated.In dynamic programming ,many decision sequences may be generated.However,sequences containing suboptimal subsequences cannot be optimal(if the principle of optimality holds)and so will not be generated as far as possible. Example:SHORTEST PATH Consider the shortest path problem.Assume that i,i1,i2,.........ik,j is a shortest path from i to j.Starting with the initial vertex i,a decision has been made to go to vertex i1.Following this decision the problem state is defined by vertex i1 and we need to find a path from i1 to j.It is clear that the sequence i1,i2,.......ik,j must constitute a shortest i1 to j path.If not ,let i1,r1,r2,........rq,j be a shortest i1 to j path.Then i,i1,r1,.......,rq,j is an i to j path that is shorter than the path i,i1,i2,.......,ik,j.Therefore the principle of optimality applies for this problem. Because of the use of principle of optimality , decision sequences containing subsequences that are suboptimal are not considered. Although the total number of different decision sequences is exponential in the number of decisions(if there are d choices for each of the n decisions to be made then there are dn possible sequences),dynamic programming algorithms often have a polynomial complexity.
Another important feature of dynamic programming approach is that optimal solutions to subproblems are retained so as to avoid recomputing their values.The use of these tabulated values make it natural to recast the recursive equations into an iterative algorithm.
MULTISTAGE GRAPHS
EXPLANATION Theory A multistage graph G = (V, E) is a directed graph in which the vertices are partitioned into k>=2 disjoint sets Vi, 1<=i<=k. If (u, v) is an edge in E, then u Vi and v Vi+1 for some i, 1<=i<k. The sets V1 and Vk are such that | V1|=|Vk|=1. Let s and t be the vertex in V1 and Vk. The vertex s is the source, and t the sink. Let c (i,j) be the cost of edge (i, j).The cost of a path from s to t is the sum of the costs of the edges on the path.
SJCET, Palai
108
The multistage graph problem is to find a minimum-cost path from s to t. Each set Vi defines a stage in the graph. Because of the constraints on E, every path from s to t starts in stage 1, goes to stage 2, then to stage 3, then stage 4, etc., and eventually terminates in stage k. A dynamic programming formulation for a k-stage graph problem is obtained by using the principle of optimality. According to this principle a path (i, k) is said to be optimal if and only if the intermediate paths (i, j) and (j, k) are also optimal. The multistage graph problem has two approaches namely, forward and backward . In forward approach the resulting path is obtained by moving in the direction of destination from source, even though it appears in reverse. In the latter case it from destination to source tracing the lowest cost path. Equation
The cost of a path (i, j) is given by the equation
cost (i, j) = min {c (j, l) + cost (i+1, l)} l Vi+1 (j, l) E where j is a vertex in Vi.
Here cost (i, j) means i th level and the cost of edge from j th node to the last node.
EXAMPLE Consider the following figure. The graph is a 5 stage graph, with 12 vertices and the cost of each node is given. The aim is to find the minimum cost path from the first node to the last node. Figure V1 V2 V3 V4 V5
SJCET, Palai
109
2 9 7 1 3 4 2 5
11 8 11 1
4 6 2
6
5 2
9
4
7 7 7
5
10 00 0
5 6
12
11
Cost Calculation cost (4, 9) =4 cost (4,10) =2 cost (4,11) =5 cost (3,6) =cost from 6 th to 12 th node = min { 5+ cost (4, 9),6+ cost (4, 10) }=7 cost (3,7) = min { 4+ cost (4, 9),3+ cost (4, 10) }=5 cost (3,8) = min { 5+ cost (4, 10),6+ cost (4, 11) }=7 cost (2,2) = min { 4+ cost (3, 6),2+ cost (3, 7), 1+ cost (3, 8) }=7 cost (2,3) = min { 2+ cost (3, 6),7+ cost (3, 7)}=9 cost (2,4) = 11+cost (3, 8)=18 cost (2,5) = min { 11+ cost (3, 7),8+ cost (3, 8)}=15 cost (1,1) = min { 9+ cost (2, 2),7+ cost (2,3), 11+ cost (2, 4) ,2+ cost (2, 5)}=16 By selecting the least cost edges from each group we can determine which all edges will be involved in the minimum cost path.
Thus the minimum cost path in forward approach is (1, 2, 7, 10, 12) and in backward approach it is (1, 3, 6, 10, 12).Both have a cost of 16 each.
FORWARD APPROACH
EXPLANATION:
SJCET, Palai
110
A multistage graph G = (V, E) is a directed graph in which the vertices are partitioned into k>=2 disjoint sets Vi, 1<=i<=k. If (u, v) is an edge in E, then u Vi and v Vi+1 for some i, 1<=i<k. The sets V1 and Vk are such that | V1|=|Vk|=1. Let s and t be the vertex in V1 and Vk. The vertex s is the source, and t the sink. Let c (i,j) be the cost of edge (i, j).The cost of a path from s to t is the sum of the costs of the edges on the path. The multistage graph problem is to find a minimum-cost path from s to t. Each set Vi defines a stage in the graph. Because of the constraints on E, every path from s to t starts in stage 1, goes to stage 2, then to stage 3, then stage 4, etc., and eventually terminates in stage k.
A dynamic programming formulation for a k-stage graph problem is obtained by using the principle of optimality. According to this principle a path (i, k) is said to be optimal if and only if the intermediate paths (i, j) and (j, k) are also optimal.
The multistage graph problem has two approaches namely, forward and backward . In forward approach the resulting path is obtained by moving in the direction of destination from source, even though it appears in reverse. In the latter case it from destination to source tracing the lowest cost path. By using the forward approach,we obtain the cost of a path (i, j) which is given by the following equation:
cost (i, j) = min {c (j, l) + cost (i+1, l)} l Vi+1 (j, l) E , where j is a vertex in Vi.
Here cost (i, j) means i th level and the cost of edge from j th node to the last node. Consider the example given below. This is a 5 stage graph, with 12 vertices and the cost of each node is given. The aim is to find the minimum cost path from the first node to the last node. This can be done by using the forward approach as shown below: EXAMPLE:
SJCET, Palai
111
V1
V2
V3
V4
V5
2 9 7 1 3 4 2 5
11 8 11 1
4 6 2
6
5 2
9
4
7 7 7
5
10 00 0
5 6
12 2
11
CALCULATING THE MINIMUM COST: cost (4, 9) = cost from 4th node to the 9th node = 4 cost (4,10) = 2 cost (4,11) =5 cost (3,6) = min { 6+ cost (4, 9), 6+ cost (4, 10) }= 7 cost (3,7) = min { 4+ cost (4, 9), 3+ cost (4, 10) }= 5 cost (3,8) = min { 5+ cost (4, 10), 6+ cost (4, 11) }= 7 cost (2,2) = min { 4+ cost (3, 6), 2+ cost (3, 7), 1+ cost (3, 8) }= 7
While calculating the value of cost (2,2) the values of cost (3,6), cost (3,7) and cost (3,8) have been reused so as to avoid their re-computation.
cost (2,3) = min { 2+ cost (3, 6), 7+ cost (3, 7)}= 9
cost (2,4) = min{11+cost (3, 8)}= 18 Department of Computer Science & Engineering SJCET, Palai
112
cost (2,5) = min { 11+ cost (3, 7), 8+ cost (3, 8)}= 15
cost (1,1) = min { 9+ cost (2, 2), 7+ cost (2,3), 11+ cost (2, 4) , 2+ cost (2, 5)}= 16
A minimum cost s to t path has a cost of 16. Thus, the cost of forward approach is= 16.
By selecting the least cost edges from each group we can determine which all edges will be involved in the minimum cost path.
Thus the minimum cost path in forward approach is (1, 2, 7, 10, 12) which has the cost of 16. It is represented using dotted lines.
This is the multistage graph forward approach problem.
BACKWARD APPROACH :
The Multistage graph problem can be solved using backward approach in the following manner. Let bp(i,j) be a minimum-cost path from vertex s to a vertex j in Vi . let bcost(i,j) be the cost of bp(i,j). From the backward approach we obtain
bcost(i,j) = min {bcost(i-1,l) + c(l,j)} l Vi-1 <l,j> E
Algorithm in psedocode corresponding to obtain a minimum-cost s-t graph is BGraph.(Algorithm 1.1) The first subscript on bcost, p and d are omitted for the same reasons as before. This algorithm has the same complexity as FGraph provided G is now represented by itsinverse adjacency lists (i.e., for each vertex v we have a list of vertices w such that <w,v> E
SJCET, Palai
Algorithm Analysis and Design (R 606) ALGORITHM
113
1. Algorithm BGraph(G,k,n,p) 2. // Same function as FGraph 3. { 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. } } / / Find a minimum-cost path. p[1] := 1; p[k] :=n; for j := k 1 to 2 do p[j] := d[p[j+1]]; bcost[1] :=0.0; <r,j> for j :=2 to n do { / / Compute bcost[j]. Let r be such that is an edge of G is an edge of and bcost is an edge of G and bcost[r] + c[r,j] is minimum; bcost[j] :=bcost[r] + c[r,j]; d[j] := r;
It should be easy to see that both FGraph and BGraph work correctly even on a more generalized version of multistage graphs. In this generalization, the graph is permitted to have edges <u,v> such that u Vi , v Vj and i< j. In the pseudocodes FGraph and BGraph, bcost(i,j) is set to for any <i,j> E. When programming these pseudocodes, one could use the maximum allowable floating point number for . If the weight of any such edge is added to some other costs, a floatin point overflow might occur. Care should be taken to avoid such overflows.
Eg: d(B, T) = min{9+d(D, T), 5+d(E, T), 16+d(F, T)} = min{9+18, 5+13, 16+2} = 18. d(C, T) = min{ 2+d(F, T) } = 2+2 = 4 Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) d(S, T) = min{1+d(A, T), 2+d(B, T), 5+d(C, T)} = min{1+22, 2+18, 5+4} = 9. The above way of reasoning is called backward reasoning.
114
9 5
D E
d(D, T) d(E, T)
16
d(F, T)
Backward (forward reasoning) d(S, A) = 1 d(S, B) = 2 d(S, C) = 5 d(S,D)=min{d(S, A)+d(A, D),d(S, B)+d(B, D)} = min{ 1+4, 2+9 } = 5 d(S,E)=min{d(S, A)+d(A, E),d(S, B)+d(B, E)} = min{ 1+11, 2+5 } = 7 d(S,F)=min{d(S, A)+d(A, F),d(S, B)+d(B, F)} = min{ 2+16, 5+2 } = 7 d(S,T) = min{d(S, D)+d(D, T),d(S,E)+d(E,T), d(S, F)+d(F, T)} = min{ 5+18, 7+13, 7+2 } =9 Department of Computer Science & Engineering
approach
SJCET, Palai
Algorithm Analysis and Design (R 606) Forward approach and backward approach:
115
Note that if the recurrence relations are formulated using the forward approach then the relations are solved backwards . i.e., beginning with the last decision On the other hand if the relations are formulated using the backward approach, they are solved forwards. Sample Graph and Simulation of the Algorithm
Stage I
Stage II
Stage III
Stage IV
Stage V
Stage VI
8 2 5
12
1 3 6
9
13 15
10 00 0 11
14
SIMULATION OF THE SOLUTION USING BACKWARD COSTS Format: COST(stage, node) = minimum cost of traveling to the node in stage from the source(node 1). STEP I COST(I,1) = 0 STEP II COST(II,2) = COST(I,1) + cost(1,2) = 0 + 10 = 10 Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) COST(II,3) = COST(I,1) + cost(1,3) = 0 + 20 = 20 COST(II,4) = COST(I,1) + cost(1,4) = 0 + 30 = 30 STEP III COST(III,5) = min{COST(II,2) + cost(2,5), COST(II,3) + cost(3,5), COST(II,4) + cost(4,5)} =min{10 +10, 20+40, 30+} = 20---via the path 1-2-5 COST(III,6) = min{COST(II,2) + cost(2,6), COST(II,3) + cost(3,6), COST(II,4) + cost(4,6)} =min{10 +20, 20+, 30+40} = 20---via the path 1-3-6 COST(III,7) = min{COST(II,2) + cost(2,7), COST(II,3) + cost(3,7), COST(II,4) + cost(4,7)} =min{10 +30, 20+50, 30+30} = 40---via the path 1-2-7 STEP IV COST(IV,8)=min{COST(III,5) + cost(5,8), COST(III,6) + cost(6,8), COST(III,7) + cost(7,8)} =min{20+10, 20+ ,40+ } Department of Computer Science & Engineering
116
SJCET, Palai
Algorithm Analysis and Design (R 606) =30---via the path 1-2-5-8 COST(IV,9)=min{COST(III,5) + cost(5,9), COST(III,6) + cost(6,9), COST(III,7) + cost(7,9)} =min{20+20, 20+20 ,40+ } =40---via the path 1-2-5-9 or via the path 1-3-6-9 COST(IV,10)=min{COST(III,5) + cost(5,10), COST(III,6) + cost(6,10), COST(III,7) + cost(7,10)} =min{20+10, 20+ ,40+ } =30---via the path 1-2-5-10 COST(IV,11)=min{COST(III,5) + cost(5,11), COST(III,6) + cost(6,11), COST(III,7) + cost(7,11)} =min{20+30, 20+30 ,40+30 } =50---via the path 1-2-5-11 or via the path 1-3-6-11 STEP V COST(V,12)=min{COST(IV,8) + cost(8,12), COST(IV,9) + cost(9,12), COST(IV,10) + cost(10,12), COST(IV,11) + cost(11,12)} =min{30+10 ,40+ ,30+,50+} = 40via the path 1-2-5-8-12 Department of Computer Science & Engineering
117
SJCET, Palai
Algorithm Analysis and Design (R 606) COST(V,13)=min{COST(IV,8) + cost(8,13), COST(IV,9) + cost(9,13), COST(IV,10) + cost(10,13), COST(IV,11) + cost(11,13)} =min{30+20 ,40+20 ,30+10,50+10} = 40via the path 1-2-5-8-13 COST(V,14)=min{COST(IV,8) + cost(8,14), COST(IV,9) + cost(9,14), COST(IV,10) + cost(10,14), COST(IV,11) + cost(11,14)} =min{30+30 ,40+10 ,30+20,50+30} =50---via the path 1-2-5-10-14 or via 1-2-5-9-14 or 1-3-6-9-14 STEP VI COST(VI,15)=min{COST(V,12)+cost(12,15), COST(V,13)+cost(13,15), COST(V,14)+cost(14,15)} =min{40+20 ,40+10 ,50+30} =50---via the path 1-2-5-8-13-15
118
SJCET, Palai
119
ALL PAIRS SHORTEST PATH

Finding all pairs shortest path consists of finding the shortest distance between every pair of nodes in a possibly directed graph. Various means of doing so are known, and the following list gives a few of the common methods: Floyd-Warshall algorithm is an elegant, quickly implementable O(n3) algorithm (Assumes absence of negatively-weighed cycles). Johnson's algorithm is harder to implement, but might perform better for sparse graphs. Algorithms for single source problem might also be repeatedly used, although they often perform worse or are harder to optimize
FLOYD-WARSHELL ALGORITHM
In computer science, the FloydWarshall algorithm (sometimes known as the WFI Algorithm or RoyFloyd algorithm, since Bernard Roy described this algorithm in 1959) is a graph analysis algorithm for finding shortest paths in a weighted, directed graph. A single execution of the algorithm will find the shortest paths between all pairs of vertices. The FloydWarshall algorithm is an example of dynamic programming
Algorithm
The Floyd-Warshall algorithm compares all possible paths through the graph between each pair of vertices. It is able to do this with only V3 comparisons. This is remarkable considering that there may be up to V2 edges in the graph, and every combination of edges is tested. It does so by incrementally improving an estimate on the shortest path between two vertices, until the estimate is known to be optimal. Consider a graph G with vertices V, each numbered 1 through N. Further consider a function shortestPath(i,j,k) that returns the shortest possible path from i to j using only vertices 1 through k as intermediate points along the way. Now, given this function, our goal is to find the shortest path from each i to each j using only nodes 1 through k + 1. There are two candidates for this path: either the true shortest path only uses nodes in the set (1...k); or there exists some path that goes from i to k + 1, then from k + 1 to j that is better. Department of Computer Science & Engineering SJCET, Palai
120
We know that the best path from i to j that only uses nodes 1 through k is defined by shortestPath(i,j,k), and it is clear that if there were a better path from i to k + 1 to j, then the length of this path would be the concatenation of the shortest path from i to k + 1 (using vertices in (1...k)) and the shortest path from k + 1 to j (also using vertices in (1...k)). Therefore, we can define shortestPath(i,j,k) in terms of the following recursive formula
This formula is the heart of Floyd Warshall. The algorithm works by first computing shortestPath(i,j,1) for all (i,j) pairs, then using that to find shortestPath(i,j,2) for all (i,j) pairs, etc. This process continues until k=n, and we have found the shortest path for all (i,j) pairs using any intermediate vertices.
Pseudocode
Conveniently, when calculating the kth case, one can overwrite the information saved from the computation of k 1. This means the algorithm uses quadratic memory. Be careful to note the initialization conditions: 1 /* Assume a function edgeCost(i,j) which returns the cost of the edge from i to j 2 3 (infinity if there is none). Also assume that n is the number of vertices and edgeCost(i,i)=0
4 */ 5 6 int path[][]; 7 /* A 2-dimensional matrix. At each step in the algorithm, path[i][j] is the shortest path 8 from i to j using intermediate vertices (1..k-1). Each path[i][j] is initialized to
9 edgeCost(i,j) or infinity if there is no edge between i and j. 10 */ 11 12 procedure FloydWarshall () 13 for k: = 1 to n Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) 14 15 for each (i,j) in {1,..,n}2 path[i][j] = min ( path[i][j], path[i][k]+path[k][j] );
121
Analysis
To find all n2 of from those of requires 2n2 bit operations. Since we begin with , , ..., ,
and compute the sequence of n zero-one matrices the total number of bit operations used is
. Therefore, the complexity of
the algorithm is (n3) and can be solved by a deterministic machine in polynomial time
Applications and generalizations

The FloydWarshall algorithm can be used to solve the following problems, among others: Shortest paths in directed graphs (Floyd's algorithm). Transitive closure of directed graphs (Warshall's algorithm). In Warshall's original formulation of the algorithm, the graph is unweighted and represented by a Boolean adjacency matrix. Then the addition operation is replaced by logical conjunction (AND) and the minimum operation by logical disjunction (OR). Finding a regular expression denoting the regular language accepted by a finite automaton (Kleene's algorithm) Inversion of real matrices (Gauss-Jordan algorithm). Optimal routing. In this application one is interested in finding the path with the maximum flow between two vertices. This means that, rather than taking minima as in the pseudocode above, one instead takes maxima. The edge weights represent fixed constraints on flow. Path weights represent bottlenecks; so the addition operation above is replaced by the minimum operation. Testing whether an undirected graph is bipartite. Fast computation of Pathfinder Networks. Maximum Bandwidth Paths in Flow Networks
SJCET, Palai
122
JOHNSONS ALGORITHM
Johnson's algorithm is a way to find shortest paths between all pairs of vertices in a sparse directed graph. It allows some of the edge weights to be negative numbers, but no negativeweight cycles may exist.
Johnson's algorithm consists of the following steps: 1. First, a new node q is added to the graph, connected by zero-weight edge to each other node. 2. Second, the Bellman-Ford algorithm is used, starting from the new vertex q, to find for each vertex v the least weight h(v) of a path from q to v. If this step detects a negative cycle, the algorithm is terminated. 3. Next the edges of the original graph are reweighted using the values computed by the Bellman-Ford algorithm: an edge from u to v, having length w(u,v), is given the new length w(u,v) + h(u) h(v). 4. Finally, for each node s, Dijkstra's algorithm is used to find the shortest paths from s to each other vertex in the reweighted graph. In the reweighted graph, all paths between a pair s and t of nodes have the same quantity h(s) -h(t) added to them, so a path that is shortest in the original graph remains shortest in the modified graph and vice versa. However, due to the way the values h(v) were computed, all modified edge lengths are non-negative, ensuring the optimality of the paths found by Dijkstra's algorithm. The distances in the original graph may be calculated from the distances calculated by Dijkstra's algorithm in the reweighted graph by reversing the reweighting transformation.
Analysis
The time complexity of this algorithm, using Fibonacci heaps in the implementation of Dijkstra's algorithm, is O(V2log V + VE): the algorithm uses O(VE) time for the Bellman-Ford stage of the algorithm, and O(V log V + E) for each of V instantiations of Dijkstra's algorithm. Department of Computer Science & Engineering SJCET, Palai
123
Thus, when the graph is sparse, the total time can be faster than the Floyd-Warshall algorithm, which solves the same problem in time O(V3).
Example
The first three stages of Johnson's algorithm are depicted in the illustration below.
The graph on the left of the illustration has two negative edges, but no negative cycles. At the center is shown the new vertex q, a shortest path tree as computed by the Bellman-Ford algorithm with q as starting vertex, and the values h(v) computed at each other node as the length of the shortest path from q to that node. Note that these values are all non-positive, because q has a length-zero edge to each vertex and the shortest path can be no longer than that edge. On the right is shown the reweighted graph, formed by replacing each edge weight w(u,v) by w(u,v) + h(u) h(v). In this reweighted graph, all edge weights are non-negative, but the shortest path between any two nodes uses the same sequence of edges as the shortest path between the same two nodes in the original graph. The algorithm concludes by applying Dijkstra's algorithm to each of the four starting nodes in the reweighted graph.
SJCET, Palai
124
SHORTEST PATH PROBLEM
A graph with 6 vertices and 7 edges In graph theory, the shortest path problem is the problem of finding a path between two vertices (or nodes) such that the sum of the weights of its constituent edges is minimized. An example is finding the quickest way to get from one location to another on a road map; in this case, the vertices represent locations and the edges represent segments of road and are weighted by the time needed to travel that segment. Formally, given a weighted graph (that is, a set V of vertices, a set E of edges, and a realvalued weight function f : E R), and one element v of V, find a path P from v to each v' of V so that
is minimal among all paths connecting v to v' . The problem is also sometimes called the single-pair shortest path problem, to distinguish it from the following generalizations: The single-source shortest path problem, in which we have to find shortest paths from a source vertex v to all other vertices in the graph. The single-destination shortest path problem, in which we have to find shortest paths from all vertices in the graph to a single destination vertex v. This can be reduced to the single-source shortest path problem by reversing the edges in the graph.
SJCET, Palai
125
The all-pairs shortest path problem, in which we have to find shortest paths between every pair of vertices v, v' in the graph. These generalizations have significantly more efficient algorithms than the simplistic approach of running a single-pair shortest path algorithm on all relevant pairs of vertices.
Algorithms
The most important algorithms for solving this problem are: Dijkstra's algorithm solves the single-pair, single-source, and single-destination shortest path problems. Bellman-Ford algorithm solves single source problem if edge weights may be negative. A* search algorithm solves for single pair shortest path using heuristics to try to speed up the search. Floyd-Warshall algorithm solves all pairs shortest paths. Johnson's algorithm solves all pairs shortest paths, and may be faster than FloydWarshall on sparse graphs. Perturbation theory finds (at worst) the locally shortest path.
Applications
Shortest path algorithms are applied to automatically find directions between physical locations, such as driving directions on web mapping websites like Mapquest or Google Maps. If one represents a nondeterministic abstract machine as a graph where vertices describe states and edges describe possible transitions, shortest path algorithms can be used to find an optimal sequence of choices to reach a certain goal state, or to establish lower bounds on the time needed to reach a given state. For example, if vertices represents the states of a puzzle like a Rubik's Cube and each directed edge corresponds to a single move or turn, shortest path algorithms can be used to find a solution that uses the minimum possible number of moves. Department of Computer Science & Engineering SJCET, Palai
126
In a networking or telecommunications mindset, this shortest path problem is sometimes called the min-delay path problem and usually tied with a widest path problem. For example, the algorithm may seek the shortest (min-delay) widest path, or widest shortest (min-delay) path. A more lighthearted application is the games of "six degrees of separation" that try to find the shortest path in graphs like movie stars appearing in the same film. Other applications include "operations research, plant and facility layout, robotics, transportation, and VLSI design".
Related problems
For shortest path problems in computational geometry, see Euclidean shortest path. The traveling salesman problem is the problem of finding the shortest path that goes through every vertex exactly once, and returns to the start. Unlike the shortest path problem, this problem is NP-complete and, as such, is believed not to be efficiently solvable (see P = NP problem) . The problem of finding the longest path in a graph is also NP-complete. The Canadian traveller problem and the stochastic shortest path problem are generalizations where either the graph isn't completely known to the mover, changes over time, or where actions (traversals) are probabilistic. The problems of recalculation of shortest paths arises if some graph transformations (e.g., shrinkage of nodes) are made with a graph
SJCET, Palai
127
LOWER BOUND THEORY

Our main task for each problem is to obtain a correct and efficient solution. If two algorithms solving the same problem were discovered and their times differed by an order of magnitude, then the one with the smaller order was generally regarded as superior. To establish a given algorithm in the most efficient way, a function g(n) , that is a lower bound on the time, is discovered. If we have an algorithm whose computing time is the same order as g(n), then we know that asymptotically we can do no better. There is a mathematical notation for expressing lower bounds. If f(n) is the time for some algorithm, then we write f(n)=(g(n)) to mean that g(n) is a lower bound for f(n). Formally this equation can be written if there exist positive constants c and n0 such that |f(n)| c|g(n)| for all n > n0. In addition to developing lower bounds to within a constant factor, we are also concerned with determining more exact bounds. Deriving good lower bounds is often more difficult than devising efficient algorithms. Perhaps this is because a lower bound states a fact about all possible algorithms for solving a problem. Usually we cannot enumerate and analyze all these algorithms, so lower bound proofs are often hard to obtain. However , for many problems it is possible to easily observe that a lower bound identical to n exists, where n is the number of inputs( or possibly outputs) to the problem. For example, consider all algorithms that find the maximum of an unordered set of n integers. Clearly every integer must be examined at least once, so (n) is a lower bound for any algorithm that solves this problem. Or, suppose we wish to find an algorithm that efficiently multiplies two nn matrices. Then (n2) is a lower bound on any such algorithm since there are 2n2 inputs that must be examined and n2 outputs that must be computed. Bounds such as these are often reffered to as trivial lower bounds because they are so easy to obtain. We know how to find the maximum of n elements by an algorithm that uses only n-1 comparisons so there is no gap between the upper and lower bounds for this problem. But for matrix multiplication the best known algorithm requires O(n2+) operations ( > 0) , and so there is no reason to believe that a better method cannot be found.
SJCET, Palai
128
COMPARISON TREES FOR SEARCHING AND SORTING

A comparison tree for binary search on an array is a binary tree that depicts all possible search paths.i.e it shows all different sequences of comparisons undertaken by binary search when searching for keys that may or may not be present in the array.
X:A(1)
Failure
X: A(2)
Failure
X:A(n)
Failure
Failure
Suppose that we are given a set S of distinct values on which an ordering relation < holds. The sorting problem calls for determining a permutation of the integers 1 to n, say p(1) to p(n), such that the n distinct values from S stored in A[1:n] satisfy A[p(1)] < A[p(2)] <. .. .< A[p(n)]. The ordered searching problem asks whether a given element x E S occurs within the elements in A[1:n], then we are to determine an I between 1 and n such that A[i] = x. The merging problem assumes that two ordered sets of distinct inputs from S are given in A[1:m] and B[1:n] such that A[1] < . . . < A[m] and B[1] < . . . B[n]; these m+n values are to be rearranged into an array C[1:m+n] so that C[1] < . . . < C[m+n]. For all these problems we restrict the class of algorithms we are considering to those which work solely by making comparisons between elements. No arithmetic involving elements is permitted, though it is possible for the algorithm to move elements around.Thes algorithms are referred to as
SJCET, Palai
129
comparison based algorithms. We rule out algorithms such as radix sort that decompose the values into subparts. a) Ordered Searching In obtaining the lower bound for the ordered searching problem, we consider only those comparison based algorithms in which every comparison between two elements of S is of the type compare x and A[i]. Each internal node in the binary tree represents a comparison between x and an A[i], or x > A[i].If x=A[i],the algorithm terminates. The left branch is taken if x < A[i], and the right is taken if x > A[i].If the algorithm terminates following a left or right branch , then no I has been found such that x = A[i] and the algorithm must declare the search successful. b) Sorting We can describe any sorting algorithm that satisfies the restrictions of the comparison tree model by a binary tree. Consider the case in which the n numbers A[1:n] to be sorted are distinct.Now,any comparison between A[i] and A[j] must result in one of two possibilities: either A[i] < A[j] or A[i] > A[j].So, the comparison tree is a binary tree in which each internal node is labeled by the pair i:j which represents the comparison of A[i] with A[j].If A[i] < A[j] , then the algorithm proceeds down the left branch of the tree; otherwise right branch. The external nodes represent termination of the algorithm. Associate with every path from the root to an external node is a unique permutation. c) Selection Any comparison tree that models comparison-based algorithms for finding the maximum of n elements has at least 2^ n-1 external nodes since each path from the root to an external node must contain at least n-1 internal nodes. This implies at least n-1 comparisons for otherwise at least two of the input items never lose a comparison and the largest is not yet found.
Comparison trees
comparison tree n! leaves : every permutation must be a leaf -case # comparisons = height of tree
SJCET, Palai
130
a1a2a3 : a1< a2?
a1a2a3 : a2<a3 ?
a2a1a3 : a1< a3?
a1a2a3
a1a3a2 : a1< a3?
a2a1a3 a2a3a1 : a2< a3?
a1a3a2
a3a1a2
a2a3a1
a3a2a1
Comparison tree for insertion sort of three items
LOWER BOUND FOR COMPARISON-BASED SORTING

It is generally much more difficult to show that an algorithm is best possible for a certain task, than to come up with a good algorithm for that task. In the latter case, one need only argue about one algorithm the one that is supposedly good; in the former case, one must argue about all possible algorithms even those which one knows nothing about! To answer the question whats the best way to do a job we must start by fixing the set of tools that may be used to do the job. For the problem at hand, sorting, we will take pair wise comparisons of keys as the available tools. Of course, this leaves out of consideration Bin sorting (which is not based on key comparisons). But, in a sense, we can justify this on the ground that Bin sorting is not a general sorting method: it is applicable only if keys have a particular form. At any rate, its important to keep in mind that the result Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) we are about to present only applies to sorting algorithms based on comparisons. Suppose we want to sort n keys K1 , K2 , . . . , Kn . Lets assume that all keys are distinct, so that for any Ki , Kj where i j , either Ki < Kj or Ki > Kj. The main theoretical device well use to analyze our problem is a decision (or comparison) tree. This is a useful way of representing any comparison-based algorithm that sorts n keys (for any given n). Before introducing the formal definition, lets develop some intuition about decision trees by studying an example.
131
Example 1: Below is a decision tree that corresponds to one possible algorithm for sorting 3 keys, K1 , K2 , K3 .
1 <
1:2
>
2:3 2:3
<
>
< >
1:3 123 1: 3 321
<
>
>
213 231
<
132 312
SJCET, Palai
132
The internal nodes of this tree correspond to comparisons the algorithm makes. The labels in those nodes, specify which keys are to be compared. For example the label "1:2" of the root indicates that K1 is to be compared with K2. Depending on the outcome of the comparison in a node, the left or the right branch out of that node is taken. The label of each leaf is the permutation specifying how to rearrange the keys to get them sorted. For example, consider the node with label "2,3,1". This means that K2<K3<K1. Note that the fact this is the order of the three keys is implied by the outcomes of the comparisons made in the path from the root to that leaf: To get to that leaf we must go from the root to the right (signifying that K1>K2); then from node "2:3" to the left (signifying that K2<K3) and from node "1:3" to the right (signifying that K1>K3). These three comparisons imply that K2<K3<K1. This is the case for the path from the root to any -2- leaf. Thus the decision tree specifies the sequence of comparisons that the algorithm will perform to sort 3 keys. An execution of the algorithm for a specific input (of 3 keys) corresponds to a path from root to a leaf. For example, suppose that the input keys are K c, K a, K3=b
(where a < b < c). The execution starts at the root of the tree. The node there specifies that K1 and K2 are to be compared. Since K1>K2, we take the right branch from the root and arrive at the "2:3" node. This indicates we must compare K2 to K3 and since K2<K3, we take the left branch and arrive at the "1:3" node. By comparing K1 and K3 we discover that K1>K3 and therefore we take the right branch which leads us to the leaf labeled "2,3,1". This indicates that the keys in sorted order are K2<K3<K1, as indeed is the case. A more programming language-like description of the algorithm corresponding to the above decision tree is: if K1<K2 then if K2<K3 then sorted order of keys is K1 , K2 , K3 else {K2>K3} if K1<K3 then sorted order of keys is K1 , K3 , K2 else {K1>K3} sorted order of keys is K3 , K1 , K2 else {K1>K2} if K2<K3 then if K1<K3 then sorted order of keys is K2 , K1 , K3 else {K1>K3} sorted order of keys is K2 , K3 , K1 else {K2>K3} sorted order of keys is K3 , K2 , K1 Using the intuition we have gained by studying this example, let us now give the formal Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) definition of a decision tree. Definition 1: A decision tree of order n is a binary tree so that (1) It has n! leaves, each labeled by a different permutation of 1, 2, . . . , n. (2) Internal nodes are labeled by pairs of indices of the form i : j 1, 2, . . . , n) there is either a node p (i) : p (i from that node to its left child or p (i from that node to its right child nodes too but must contain at least these.) The idea is that leaves correspond to the possible outcomes of sorting n distinct keys. Internal node i : j corresponds to the comparison of Ki and Kj. If the outcome is Ki < Kj then the left subtree of the node i : j contains the subsequent comparisons made until the order of the keys is determined. Symmetrically, if the outcome is Ki > Kj , the right subtree of i : j contains the subsequent comparisons. Part (3) of the definition essentially requires that every relationship determined by the algorithm must have been established by actual comparisons: the algorithm cannot guess. Any comparison-based algorithm for sorting n keys corresponds to a decision tree of i j n.
133
(3) In the path from the root to a leaf p (1) p (2) . . . p (n) (p is a permutation of in which case the path goes i) in which case the path goes i < n. (The path may contain other
order n. The execution of the algorithm on input sequence of keys K1, K2, . . . , Kn follows the path from the root to the leaf labeled by the permutation p such that Kp (1) < Kp (2) < . . . < Kp (n). Note that for a given sorting algorithm we need a different -3decision tree for each n to represent all possible executions of that algorithm when the input consists of n keys. Example 2: Consider the algorithm for insertion sort: INSERTION -SORT(A) begin for i := 2 to n do for j := i down to 2 do if A[ j] < A[ j 1] then A[ j else break end A[ j 1]
SJCET, Palai
134
WORST CASE LOWER BOUND ON THE NUMBER OF COMPARISONS

Let Cn denote the minimum number of comparisons required to sort n keys. For any decision tree, the worst case number of comparisons required by the algorithm represented by that tree is precisely the height of the tree. Lemma: Any binary tree of height h has at most 2h leaves. Proof: Trivial induction on h. By definition, any decision tree of order n is a binary tree with n! leaves. Thus, by the lemma, it must have height at least approximation says that n Cn n (n/e)n. Thus, n (n/e)n n/e)n = n log n n log e n log n). n Cn n
from which it follows that Cn
AVERAGE CASE LOWER BOUND ON THE NUMBER OF COMPARISONS

Consider a decision tree of order n representing some sorting algorithm. If we assume that all initial arrangements of the keys to be sorted are equally likely, the average case number of comparisons performed by that algorithm is equal to the external path length of the decision tree divided by the number of leaves in the tree. The following fact can be easily proved: Fact 2: The tree that minimizes the external path length has all leaves in at most two depths d and d 1, for some d. By Facts 1 and 2 we may assume that in the decision tree that minimizes the average number of comparisons, all leaves have depth d or d 1 for some d. Let Cn be the minimum average case number of comparisons needed to sort n keys. Also, let Nd and Nd1 be the number of leaves in the corresponding decision tree at depth d and d 1 respectively. Recall that the number of leaves in T is n!. Thus we have Cn = [(d 1) Nd d Nd ]/n! (1) But we have: Nd Nd1 = n! (2) and Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) Nd Nd1 =2d (3)
135
Equation (2) says that the total number of leaves in T is n!. For equation (3) note that if we gave 2 children to each leaf of depth d 1, we would get the maximum possible number of nodes at depth d, which is 2d . Recall that the external path length of the tree is the sum of the depths of all leaves. -5Solving (2) and (3) for Nd and Nd1, we get: Nd =2 n and Nd1 =2d n! (5) Substituting (4) and (5) in (1) we obtain: Cn = [(d 1) Nd d Nd ]/n! = (d n n But d d )/n! n = log n n n! 2e )/n! d (4)
Cn = (n! log n n = log n
Cn n n log n). n log n) comparisons n(n/e)n n/e)n = n log n n log e.
Hence, Cn
Therefore, any comparisonboth in the worst and in the average case.
ORACLES AND ADVERSARY ARGUMENTS
One of the proof techniques that is useful for obtaining lower bounds consists of making use of an oracle. The most famous oracle in the history was called the Delphic oracle, located in Delphi in Greece. This oracle can still be found, situated in the side of a hill embedded in some rocks. In olden times people would approach the oracle and ask it a question. After some period of time elapsed, the oracle would reply and a caretaker would interpret the oracles answer.
SJCET, Palai
136
A similar phenomenon takes place when we use an oracle to establish a lower bound. Given some model of computation such as comparison trees, the oracle tells us the outcome of each comparison. To derive a good lower bound, the oracle tries its best to cause the algorithm to work as hard as it can. It does this by choosing as the outcome of the next test, the result that causes the most work to be required to determine the final answer. And by keeping tracks of the work that is done, a worst-case lower bound for the problem can be derived.
MERGING
Now we consider the merging problem. Given the sets A[l:m] and B[l:n], Where the item in A and the item in B are sorted, we investigate lower bounds for algorithms that merge these two sets to give a single sorted set. As was the case of sorting, we assume that all the m+n elements are distinct and that A[1]<A[2]..<A[m] and B[1]<B[2].<B[n]. It is possible that after these two sets are merged, the n elements of B can be inter leaved within A in every possible way. Elementary combinatorics tells us that there are ( ) ways that the
As and Bs can merge together while preserving the ordering within A and B. For example, if m=3, n=2, A[i]=x,A[2]=y,A[3]=z,B[1]=u,B[2]=v, there are ( )=10 ways in which A and
B merge: u,v,x,y,z;u,x,v,y,z;u,x,y,v,z;u,x,y,z,u;x,u,v,y,z;x,u,y,v,z;x,u,y,z,v;x,y,u,v,z;x,y,u,z,v; and x,y,u,v. Thus if we use comparison trees as our model for merging algorithm, then there will be external nodes, and therefore at least [log ] comparisons are required by any comparison-
based merging algorithm. The conventional merging algorithm takes m+n-1 comparisons. If we let MERGE(m,n) be the minimum number of comarisons needed to merge m items, then we have the inequality [log ] MERGE(m,n)-1
The exercises show that these upper and lower bounds can get arbitrarily far apart as m gets much smaller than n. This should not be a surprise becoz the conventional algorithm is designed to work best when m and n binary insertion would required the fewest number of comparisons needed to merge A[1] into B[1],..,B[n]. When m and n are equal, the lower bound given by the comparison tree model is too and the number of comparisons for the conventional merging algorithm can be shown to be optimal.
SJCET, Palai
Algorithm Analysis and Design (R 606) Theorm: MERGE(m,m)=2m-1, for m1.
137
Proof: Consider any algorithm that merges the two sets A[1] <..< A[m] and B[1]<.<B[m]. We already have an algorithm that requires 2m-1 comparisons. If we can show that MERGE(m,m)2m-1, then the theorm follows. Consider any comparison-based algorithm for solving A[1]<B[2]<A[2]<<B[m]<A[m], that is, for which the Bs and As alternate. Any merging algorithm must make each of the 2m-1 comparisons B[1]:A[1],A[1]:B[2],B[2]:A[2],..,B[m]:A[m] while merging the given input. To see this, supose that a comparison of type B[i]:A[i] is not made for some i. Then the algorithm cannot distinguish b/w the previous ordering and the one in which B[1]<A[1]<<A[i-1]<A[i]<B[i]<B[i+1]<.<B[m]<A[m]. So the algorithm will not neccessarly merge the As and Bs properly. If a comparison of type A[i]:B[i+1] is not made, then the algorithm will not be able to distinguish b/w the cases in which B[1]<A[1]<B[2]<<B[m]<A[m] and in which B[1]<A[1]<B[2]<A[2]<A[i1]<B[i]<B[i+1]<A[i]<a[i+1]<.<B[m]<A[m]. So any algorithm must make all 2m-1 comparisons to produce this final result. The theorm follows.
INSERTION SORT
If the first few objects are already sorted, an unsorted object can be inserted in the sorted set in proper place. This is called insertion sort. An algorithm consider the elements one at a time, inserting each in its suitable place among those already considered (keeping them sorted). Insertion sort is an example of an incremental algorithm; it builds the sorted sequence one number at a time. INSERTION_SORT (A) 1. For j = 2 to length [A] do 2. 3. 4. 5. 6. key = A[j] {Put A[j] into the sorted sequence A[1 . . j-1] i j -1 while i > 0 and A[i] > key do A[i+1] = A[i] SJCET, Palai
Algorithm Analysis and Design (R 606) 7. 8. Analysis Best-Case i = i-1 A[i+1] = key
138
The while-loop in line 5 executed only once for each j. This happens if given array A is already sorted. T(n) = an + b = O(n) It is a linear function of n. Worst-Case The worst-case occurs, when line 5 executed j times for each j. This can happens if array A starts out in reverse order T(n) = an2 + bc + c = O(n2) It is a quadratic function of n.
SJCET, Palai
139
The graph shows the n2 complexity of the insertion sort.
Stability Since multiple keys with the same value are placed in the sorted array in the same order that they appear in the input array, Insertion sort is stable.
Extra Memory This algorithm does not require extra memory. For Insertion sort we say the worst-case running time is (n2), and the best-case running time is (n). Insertion sort use no extra memory it sort in place. The time of Insertion sort is depends on the original order of a input. It takes a time in (n2) in the worst-case, despite the fact that a time in order of n is sufficient to solve large instances in which the items are already sorted.
SJCET, Palai
Algorithm Analysis and Design (R 606) Implementation void insertionSort(int numbers[], int array_size) { int i, j, index; for (i=1; i < array_size; i++) { index = numbers[i]; j = i; while ((j > 0) && (numbers[j-1] > index)) { numbers[j] = numbers[j-1]; j = j - 1; } numbers[j] = index; } }
140
SJCET, Palai
141
Adversary argument
Suppose we have an algorithm we think efficient. Image an adversary tries to prove otherwise. At each point in the algorithm, whenever an decision (i.e., key comparison) is made, the adversary tells us the result of the decision. The adversary chooses the answer which tries to force the algorithm work hard (i.e., do a lot of decision, or to say, the answer releases as less new information as possible). You can think the adversary is constructing a bad input while it is answering the questions. The only requirement on the answers is that they must be internally consistent. If the adversary can force the algorithm to perform f(n) steps, then f(n) is the lower bound, i.e, at least how many steps in the worst case. Simply Put: Playing a guessing game between you and your friend. You are to pick up a date, and the friend will try to guess the date by asking YES/NO questions. Your purpose is forcing your friend to ask as many questions as possible. To question is it in winter, your answer should be NO. To question is the first letter of the months name in the first half of the alphabet? Your answer should be YES. Idea: You did not pick up a date in advance at all, but Construct a date according to the questions. The The requirement is that the finally constructed date should be consistent to all your answers to the questions. Looks like cheating, but it is a good way to find the lower bound.
SJCET, Palai
142
Lower Bound for Algorithms that remove at most One Inversion per Comparison
Insertion Sort, either does nothing or moves the key in the jth slot to the (j+1)st slot. By moving the key in the jth slot up one slot, we have remedied the fact that x (number being compared with) should come before the key. However, this is all we have accomplished. We show that all sorting algorithms that sort only by comparisons of keys, and accomplish such a limited amount of rearranging after each time, require at least quadratic time. We obtain our results under the assumption that the keys to be sorted are distinct. Clearly, the worst case bound still holds true with this restriction removed because a lower bound on the worst-case performance from some inputs of some subsets is also a lower bound when all inputs are considered. In general, we are concerned with sorting n distinct keys that come from any ordered set. However, without loss of generality, we can assume that the keys to be sorted are simply the positive integers 1,2,.,n, because we can substitute 1 for the smallest key, 2 for the second smallest, and so on. For example, suppose we have alpha input, [Ralph, Clyde, Dave]. We can associate 1 with Clyde, 2 with Dave and 3 with Ralph. To obtain the equivalent input, [3, 1, 2]. Any algorithm that sorts these integers only by comparison of keys would have to do the same number of comparisons to sort the three names. A permutation of the first n positive integers can be thought of as an ordering of those integers. Because there are n! Permutations, of the first n positive integers. There are n! different orderings of those integers. For example the following six permutations are all the ordering of the first three positive integers: [1,2,3] [1,3,2] [2,1,3] [2,3,1] [3,1,2] [3,2,1] This means that n! Different inputs (to a sorting algorithm) containing n different keys. These six permutations are the different inputs of size three (3). We denote a permutation by [k1,k2,,kn]. That is ki is the integer at the ith position. For the permutation [3, 1, 2], for example, K1=3, k2=1, k3=2 R and s must be integers between 1 and n such that s > r. Department of Computer Science & Engineering SJCET, Palai
143
Given a permutation, the pair (s, r) is an inversion in either the permutation or its transpose but not both. Showing that there are n ( n - 1 ) / 2 such pairs of integers between 1 and n is left as an exercise. This means that a permutation and its transpose have exactly, n ( n - 1 ) / 2 inversions between them. So the average no of inversions in a permutation and its transpose is,
1 n(n 1) * 2 2 n(n 1) 4
Therefore, we consider all permutations equally probable for the input, the average number of inversions in the input is also n (n-1) / 4. Because we assume that the algorithm removes at most one inversion after each comparison, on an average it must do at least this many comparisons to remove all inversion sand thereby sort the input. Insertion Sort removes at most the inversion consisting of S[j] and x after each comparison, and therefore this algorithm is in the class of algorithms addressed by Theorem. Because Insertion Sorts worst case time complexity is n ( n 1 ) / 2 and its average case time complexity is about n2/4, its about as good as we can hope to do (as far as comparisons of keys are concerned) with algorithms that sort only by comparisons of keys and remove at least one inversion after each comparison.
SELECTION SORT
This type of sorting is called "Selection Sort" because it works by repeatedly element. It works as follows: first find the smallest in the array and exchange it with the element in the first position, then find the second smallest element and exchange it with the element in the second position, and continue in this way until the entire array is sorted.
SELECTION_SORT (A) for i 1 to n-1 do min j i; min x A[i] for j i + 1 to n do If A[j] < min x then min j j Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) min x A[j] A[min j] A [i] A[i] min x
144
Selection sort is among the simplest of sorting techniques and it work very well for small files. Furthermore, despite its evident "nave approach "Selection sort has a quite important application because each item is actually moved at most once, Section sort is a method of choice for sorting files with very large objects (records) and small keys. The worst case occurs if the array is already sorted in descending order. Nonetheless, the time require by selection sort algorithm is not very sensitive to the original order of the array to be sorted: the test "if A[j] < min x" is executed exactly the same number of times in every case. The variation in time is only due to the number of times the "then" part (i.e., min j j; min x A[j] of this test are executed.
The Selection sort spends most of its time trying to find the minimum element in the "unsorted" part of the array. It clearly shows the similarity between Selection sort and Bubble sort. Bubble sort "selects" the maximum remaining elements at each stage, but wastes some effort imparting some order to "unsorted" part of the array. Selection sort is quadratic in both the worst and the average case, and requires no extra memory. For each i from 1 to n - 1, there is one exchange and n - i comparisons, so there is a total of n -1 exchanges and (n -1) + (n -2) + . . . + 2 + 1 = n(n -1)/2 comparisons. These observations hold no matter what the input data is. In the worst case, this could be quadratic, but in the average case, this quantity is O(n log n). It implies that the running time of Selection sort is quite insensitive to the input.
Implementation
void selectionSort(int numbers[], int array_size) { int i, j; int min, temp;
for (i = 0; i < array_size-1; i++) Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) { min = i; for (j = i+1; j < array_size; j++) { if (numbers[j] < numbers[min]) min = j; } temp = numbers[i]; numbers[i] = numbers[min]; numbers[min] = temp; } }
145
SELECTION OF KTH SMALLEST ELEMENT

In computer science, a selection algorithm is an algorithm for finding the k-th smallest number in a list (such a number is called the kth order statistic.) This includes the cases of finding the minimum, maximum, and median elements. There are worst-case linear time selection algorithms. Selection is a subproblem of more complex problems like the nearest neighbor problem and shortest path problems. A worst-case linear algorithm for the general case of selecting the kth largest element was published by Blum, Floyd, Pratt, Rivest, and Tarjan in their 1973 paper Time bounds for selection, sometimes called BFPRT after the last names of the authors. The algorithm that it is based on was conceived by the inventor of quicksort,C.A.R.Hoare, and is known as Hoare's selection algorithm or quickselect. In quicksort, there is a subprocedure called partition that can, in linear time, group a list (ranging from indices left to right) into two parts, those less than a certain element, and those greater than or equal to the element. Here is pseudocode that performs a partition about the element list[pivotIndex]: Algorithm function partition(list, left, right, pivotIndex) pivotValue := list[pivotIndex] swap list[pivotIndex] and list[right] // Move pivot to end Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) storeIndex := left for i from left to right-1 if list[i] < pivotValue swap list[storeIndex] and list[i] storeIndex := storeIndex + 1 swap list[right] and list[storeIndex] // Move pivot to its final place return storeIndex
146
In quicksort, we recursively sort both branches, leading to best-case (n log n) time. However, when doing selection, we already know which partition our desired element lies in, since the pivot is in its final sorted position, with all those preceding it in sorted order preceding it and all those following it in sorted order following it. Thus a single recursive call locates the desired element in the correct partition: Algorithm function select(list, left, right, k) select pivotIndex between left and right pivotNewIndex := partition(list, left, right, pivotIndex) if k = pivotNewIndex return list[k] else if k < pivotNewIndex return select(list, left, pivotNewIndex-1, k) else return select(list, pivotNewIndex+1, right, k) Note the resemblance to quicksort; indeed, just as the minimum-based selection algorithm is a partial selection sort, this is a partial quicksort, generating and partitioning only O(log n) of its O(n) partitions. This simple procedure has expected linear performance, and, like quicksort, has quite good performance in practice. It is also an in place algorithm, requiring only constant memory overhead, since the tail recursion can be eliminated with a loop like this: Algorithm function select(list, left, right, k) loop select pivotIndex between left and right pivotNewIndex := partition(list, left, right, pivotIndex) Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) if k = pivotNewIndex return list[k] else if k < pivotNewIndex right := pivotNewIndex-1 else left := pivotNewIndex+1
147
Like quicksort, the performance of the algorithm is sensitive to the pivot that is chosen. If bad pivots are consistently chosen, this degrades to the minimum-based selection described previously, and so can require as much as O(n2) time. David Musser describes a "median-of-3 killer" sequence that can force the well-known median-of-three pivot selection algorithm to fail with worst-case behavior .
The algorithm works as follows: 1. Find the minimum value in the list 2. Swap it with the value in the first position 3. Repeat the steps above for remainder of the list (starting at the second position) Effectively, we divide the list into two parts: the sublist of items already sorted, which we build up from left to right and is found at the beginning, and the sublist of items remaining to be sorted, occupying the remainder of the array. Here is an example of this sort algorithm sorting five elements: 64 25 12 22 11 11 25 12 22 64
11 12 25 22 64 11 12 22 25 64
SJCET, Palai
148
Selection sort can also be used on list structures that make add and remove efficient, such as a linked list. In this case it's more common to remove the minimum element from the remainder of the list, and then insert it at the end of the values sorted so far. For example: 64 25 12 22 11 11 64 25 12 22 11 12 64 25 22 11 12 22 64 25 11 12 22 25 64
SJCET, Palai
149
MODULE 5
SJCET, Palai
150
BACKTRACKING
Many problems with searching a set of solutions or which ask for an optimal solution satisfying some constraints can be solved using the backtracking formulation. In many applications of the backtrack method, the desired solution is expressible as an n-tuple(x1,..,xn), where the xi are chosen from some from finite set Si. Often the problem to be solved calls for finding one vector that maximizes (or minimizes or satisfies) a criterion function P(x1,..,xn). Sometimes it seeks all vectors that satisfy P. The criterion function P is the inequality a[xi]a[xi+1] for 1 i < n. The set Si is finite and includes the integers 1 through n. Many of the problems we solve using backtracking require that all the solutions satisfy a complex set of constraints. For any problem these constraints can be classified into two categories: explicit and implicit. The explicit constraints depend on the particular instance I of the problem being solved. All tuples that satisfy the explicit constraints define a possible solution space for I. the implicit constraints are the rules that determine which of the tuples in the solution space of I satisfy the criterion function. Thus implicit constraints describe the way in which xi must relate to each other.
CONTROL ABSTRACTION
(i) Recursive backtracking algorithm Algorithm Backtrack(k) //This schema describes the backtracking process using recursion. On entering, the first //k-1 values x[1], x[2],..,x[k-1] of the solution vector x[1:n] have been assigned. x[ ] and //n are global. { for (each x[k] T(x[1],..,x[k-1]) do { if (Bk(x[1],x[2],..,x[k])!=0) then {
SJCET, Palai
Algorithm Analysis and Design (R 606) if (x[1],x[2],..,x[k] is a path to an answer node) then write (x[1:k]); if (k<n) then backtrack(k+1); } } } Explanation
151
Let (x1,x2,..xi) be a path from the root to a node in a state space tree. Let T(x1,x2,..xi) be the set of all possible values for xi+1 such that (x1,x2,..xi+1) is also a path to a problem state. T(x1,x2,..xn) =. We assume the existence of bounding function Bi+1 (expressed as predicates) such that if Bi+1(x1,x2,..xi+1) is false for a path (x1,x2,..xi+1) from the root node to a problem state, then the path cannot be extended to reach an answer node. Thus the candidates for position i+1 of the solution vector (x1,x2,..xn) are those values which are generated by T and satisfy Bi+1. The above recursive algorithm is initially invoked by Backtrack(1);. The solution vector (x1,x2,..xn), is treated as a global array x[1:n]. All the possible elements for the kth position of the tuple that satisfy Bk are generated, one by one, and adjoined to the current vector (x1,x2,..xk-1). Each time xk is attached, a check is made to determine whether a solution has been found. Then the algorithm is recursively invoked. When the for loop is exited, no more values for xk exist and the current copy of Backtrack ends. The last unresolved call now resumes.
(ii)General iterative backtracking method Algorithm IBacktrack(n) //This schema describes the backtracking process. //All solutions are generated in x[1:n] and printed as soon as they are determined. { Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) k:= 1; while(k0) do {
152
if (there remains an untried x[k] T(x[1],x[2],..,x[k-1]) and Bk (x[1],..,x[k]) is true) then { if (x[1],.x[k] is a path to answer node) then write (x[1:k]); k:=k+1;//Consider the next set. } else k:=k-1; //Backtrack to the previous set. } } Explanation T() will yield the set of possible values that can be placed as the first component x[1] of the solution vector. The component x[1] will take on those for which the bounding function B1x(1) is true. The variable k is continually incremented and a solution vector is grown until either a solution in found or no untried value of x k remains. When k is decremented, the algorithm must resume the generation of possible elements for the kth position that have not yet been tried. Conclusion The efficiency of both algorithms depends very much on four factors: 1. The time to generate next xk 2. The number of xk satisfying the explicit constraints 3. The time for the bounding functions Bk 4. The number of xk satisfying the Bk
SJCET, Palai
Algorithm Analysis and Design (R 606) Constraints Solutions must satisfy a set of constraints Explicit vs. implicit Definition 1: Explicit constraints are rules that restrict each xi to take on values only from a given set. eg) xi 0 or Si = { all nonnegative real numbers }
153
xi = 0 or 1 or Si = { 0, 1 } li xi ui or Si = {a : li a ui }
All tuples satisfying the explicit constraints define a possible solution space for I (I=problem instance) Definition 2: The implicit constraints are rules that determine which of the tuples in the solution space of I satisfy the criterion function. Thus implicit constrains describe the way in which the xi must relate to each other.
Classic combinatorial problem Place eight queens on an 8*8 chessboard so that no two attack Without loss of generality, assume that queen i is placed on row i All solutions represented as 8-tuples (x1,x2,,x8) where xi is the column on which queen i is placed Constraints Explicit constraints Si={1,2,,8} So, solution space consists of 88 8-tuples Implicit constraints No two xis can be the same (By this, solution space reduced from 88 to 8!) No two queens can be on the same diagonal
SJCET, Palai
154
BOUNDING FUNCTIONS
The backtrack algorithm as its virtue the ability to yield the same answer with far fewer than m trials. Its basic idea is to build solution vector, one competent at a time and to use modified criterion functions Pi(x1,,xn) called bounding functions, to test whether the vector being formed has any chance of success. The major advantage of this method is this: if it is realized that the partial vector (x1, x2,.., xi) can in no way lead to an optimal solution, then mi+1mn possible test vectors can be ignored entirely.
N-QUEENS PROBLEM
The n-queens problem is a generalization of the 8-queens problem. Here n queens are to be placed on an nxn chessboard so that no two attack, that is no two queens are on the same row, column, or diagonal. The solution space consists of all n! permutations of the n-tuple (1,2,,n). The tree is called permutation tree. The edges are labeled by possible values of xi. Edges from level 1 to level 2 nodes specify the values for x1. Edges from level i to i+1 are labeled with values of xi. The solution space is defined by all the paths from the root node to a leaf node. For eg. If n=4, then there will be 4! =24 leaf nodes in the tree. Eg: 8-queens problem A classic problem is to place eight queens on an 8x8 chessboard so that no two attack, that is, so that no two of them are on the same row, column, or diagonal.
SJCET, Palai
155
Bounding function No two queens placed on the same column xis are distinct No two queens placed on the same diagonal how to test? The same value of row-column or row+column Supposing two queens on (i,j) and (k,l) i-j=k-l (i.e., j-l=i-k) or i+j=k+l (i.e., j-l=k-i) So, two queens placed on the same diagonal iff |j-l| = |i-k|
SJCET, Palai
156
This example shows how the backtracking works(4-Queens)
SJCET, Palai
157
TREE STRUCTURE
Tree organization of the 4-queens solution space. Nodes are numbered as in depth first search.
The n-queens problem is a generalization of the 8-queens problem. Now n queens are to be placed on an nn chess board so that no two attack; that is, no two queens are on the same row, column or diagonal.Generalising our discussions, the solution space consists of all n! permutations of n-tuple (1, 2.., n).The above figure shows a possible tree organization for the case n=4. A tree such as this is called a permutation tree. The edges are labeled by possible values of xi. Edges from level 1 to level 2 nodes specify the value for x1.Thus the leftmost sub tree contains all solutions with x1=1 and x2=2,and so on. Edges from level to level i+1are labeled with the value of xi.the solution space is defined by all paths from the root node to a leaf node. There are 4! =24 nodes in the tree.
ALGORITHM
Algorithm Place (k, ) // Returns true if a queen can be placed in the kth row and //ith column. Otherwise it returns false. x[] is a //global array whose first (k 1) values have been set. //Abs (p) returns the absolute value of p. Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) { for j: =1 to k-1 do if ((x [j] = ) // Two in the same column or (Abs (x [j] ) = Abs (j-k)) ) // or in the same diagonal then return false; return true; }
158
Algorithm Nqueens (k, n) // Using backtracking, this procedure prints all // possible placements of n queens on an n x n // chessboard so that they are nonattacking. { for : =1 to n do { if Place (k, ) then { x [k] : = ; if (k=n) then write (x[1:n]); else Nqueens (k+1,n); } } }
SJCET, Palai
159
SUM OF SUBSETS
Given positive numbers wi, 1in, and m, find all subsets of the wi whose sum is m eg) (w1,w2,w3,w4) = (11,13,24,7) and m=31 Solutions are (11,13,7) and (24,7) Representation of solution vector Variable-sized By giving indices In the above example, (1,2,4) and (3,4) Explicit constraints: xi { j | j is integer and 1jn} Implicit constraints: no two be the same, the sums are m Fixed-sized
n-tuple (x1,x2,,xn) where xi{0,1}, 1in In the above example, (1,1,0,1) and (0,0,1,1)
Variable tuple size Suppose we are given n distinct positive numbers(usually called weights) and we desire to find all combinations of these numbers whose sums are m.This is called sum of subsets problem. We could formulate this problem using either fixed-or variable-sized tuple Given positive numbers wi,1<=i<=n, and m, this problem calls for finding all subsets of the wi whose sums are m.For example if n=4,( w1,w2,w3,w4)=(11,13,24,7), and m=31,then the desired subsets are (11,13,7) and (24,7).Rather than represent the solution vector by the wi which sum to m we could represent the solution vector by giving the indices of these wi.Now the two solutions are described by the vectors (1,2,4) and (3,4).In general all solutions are ktuples (x1,x2,.....xk), 1<=k<=n and different solutions may have different sized tuples.
SJCET, Palai

Solution space defined by all paths from the root to any node
160
Fixed tuple size: Solution space defined by all paths from the root to a leaf node Bounding function Bk(x1,..,xk)=true iff
Assuming wis in nondecreasing order, (x1,..,xk) cannot lead to an answer node if
SJCET, Palai
161
So, the bounding function is
Backtracking algorithm for the sum of sunsets Invoked by SumOfSubsets(0,1,i=1,nwi)
SJCET, Palai

Example:n=6, w[1:6]={5,10,12,13,15,18}, m=30
162
Terminology A node which has been generated and all of whose children have not yet been generated is called a live node The live node whose children are currently being generated is called E-node A dead node is a generated node which is not to be expanded further or all of whose children have been generated Two different ways of tree generation Backtracking Depth first node generation with bounding functions Branch-and-bound E-node remains E-node until it is dead Four factors for efficiency of the backtracking algorithms
Time to generate the next xk Number of xk satisfying the explicit constraints Time for the bounding function Bk Number of xk satisfying Bk
SJCET, Palai
163
A solution space organization for the sum of subsets problem.Nodes are numbered in BFS
* The first three relatively independent of the problem instance, and the last dependent
X1=1
1 1 X1=4 X1=2 3 3 X2=4 X2=3 9 99 X3=4 15 6 10 11 11 1 4 4 5 5
2 2 X2=2 X 6 6 X3=3 12 13 X3=4 14 6 7 7 X3=4 8
X2=4
X4=4 16 Fig 1
The tree of fig 1 represents corresponds to variable tuple size formulation.The edges are labelled such that an edge from a level i node to a level i+1 node represents a value for xi.At each node, the solution space is partitioned into subsolution spaces.The solution space is defined by all paths from the root node to any node in the tree,since any such path corresponds to a subset satisfying the explicit constraints.Thr possible paths are (1),(1,2),(1,2,3),(1,2,3,4),(1,2,4),(1,3,4),(2),(2,3) and so on.Thus the leftmost subtree defines all subsets containing w1,the next subtree defines all subsets containing w2 but not w1 and so on. Department of Computer Science & Engineering SJCET, Palai
164
Another organization for the sum of subsets problem.Nodes are numbered in D-search
X1=1
1 1
X1=0
X2=1
2 12 1
3 1 X2=0 X2=1 4 1 X3=0 21 11 X4=1 25 22 X4=0 1 22 0000 1 == X3=0 X3=1 1 12 13 11 X4= 10 15 1 11 1 0 1 X3=1 6 11 X4 11 =1 8 1 11 X2=0
X3=1 26 22 6 1
18 11 81 X3=0 27 12 1 X3=1 20 20 1
19 19 1
5 11
30 11
X4=1 31 28 X4=0 11 22 8 1
X4=1 29 24 X4=0 11 24 1
11 X4=1 2 X4= X4= 23 1 16 1 17 0 14 14 1 61 1 1 1 1
FIG 2
The tree of fig 2 corresponds to the fixed tuple size formation.Edges from level i nodes level i+l nodes are labelled with the value of xi w hich is either zero or one.All paths from root to a leaf node defining the solution space .The left subtree of the root defines all subsets containing w1,the right subtree defines all subsets not containing w1, and so on.Now there are 24 leaf nodes.
SJCET, Palai
165
KNAPSACK PROBLEM
Explanation: Given n positive weights wi , n positive profits pi, and a positive number m that is the knapsack capacity, this problem calls for choosing a subset of the weights such that iin wixi m and iin pixi is maximized. The xis constitute a zero-one-valued vector. The solution space for this problem consists of the 2n distinct ways to assign zero or one values to the x is. Thus the solution space is the same as that for the sum of subsets proble. Two possible tree organizations are possible. One corresponds to the fixed tuple size formulation and the other to the variable tuple size formulation. Backtracking algorithms for the knapsack problem can arrived by using either of these two state space trees. Regardless of which is used, bounding functions are needed to help kill some live nodes without expanding them. A bounding function for this problem is obtained by using an upper bound on the value of the best feasible solution obtainable by expanding the given live node and any of its descendants. If this upper bound is not higher than the value of the best solution determined so far, then that live node can be killed. Now by using the fixed tuple size formulation, if at node Z the values of xi, 1ik, have already been determined, then an upper bound for Z can be obtained by relaxing the requirement xi=0 or 1 to 0xi 1 for k+1in and using the greedy algorithm to solve the relaxed problem. Function Bound(cp,cw,k) determines an upper bound on the best solution obtainable by expanding any node Z at level k+1 of the state space tree. The object weights and profits are w[i] and p[i]. It is assumed that p[i]/w[i]p[i+1]/w[i+1], 1i<n. From Bound it follows that the bound for a feasible left child of a node Z is the same as that for Z. Hence, the bounding function need not be used whenever the backtracking algorithm makes a move to the left child of a node. The resulting algorithm is BKnap. It was obtained form the recursive backtracking schema. Initially set fp;=-1;. This algorithm is invoked as BKnap(1,0,0); When fp -1, x[i]. 1 i n, is such that ni=1 p[i]x[i] = fp. In lines 8 to 18 children are generated. In line 20, Bound is used to test whether a right child should be generated. The Department of Computer Science & Engineering SJCET, Palai
166
path y[i], 1 i k, is the path to the current node. The current weight cw = k-1i=1 w[i]y[i] and cp= be. So far, all the backtracking algorithms have worked on a static state space tree. i=1k-1 p[i]y[i]. In lines 13 to 17 and 23 to 27 the solution vector is updated if need
Algorithm: Bounding Function Algorithm Bound(cp,cw,k) //cp is the current profit total, cw is the current //weight total; k is the index of the last removed //item; and m is the knapsack size. { b:= cp; c:=cw; for i:=k+1 to n do { c:=c+w[i]; if(c<m) then b:=b+p[i]; else return b+(1-(c-m)/w[i])*p[i]; } return b; }
Algorithm: Backtracking solution to the 0/1 knapsack problem. Algorithm Bknap(k,cp,cw) //m is the size of the knapsack; n is the number of weights. //and profits. w[] and p[] are the weights and profits. //p[i]/w[i] p[i+1]/w[i+1]. fw is the final weight of //knapsack; fp is the final maximum profit. x[k] = 0 if w[k] //is not in the knapsack; else x[k]=1. { // Generate left child. if (cw + w[k] m) then Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) { y[k]:=1; if(k<n) then BKnap(k+1,cp+p[k], cw +w[k]); if ((cp+p[k] > fp) and (k=n)) then { fp:=cp +p[k]; fw:=cw+w[k]; for j:=1 to k do x[j]:=y[j]; } } // Generate right child. if (Bound(cp,cw,k)fp) then { y[k] :=0; if (k<n) then BKnap(k+1,cp,cw); if((cp>fp) and (k=n)) then { fp:=cp; fw:=cw; for j:=1 to k do x[j]:=y[j]; } } }
167
ALGORITHM
// CP-Current Profit; //K-index of last removed access { b=cp;c=cw; for P=K+1 to n do { c=c+w[i]; if(c<n)the b=b+p[i];
SJCET, Palai
Algorithm Analysis and Design (R 606) else return b+(1-(c-m)/w[i])+p[i]; } Return(b); }
168
Algorithm B knap(K,cp,cw)
{ If(w+w[K]<=m) then { Y[K]=1; If(K<n)then B knap(K+1,cp+p[K],cw+w[K]); If((cp=p[k])>fp)and(K=n)then { fp=cp+p[K];fw=cw+w[K]; for j=1 to K do x[j]=y[j]; } } If(Bound(cp,w,K)>=fp) then { Y[K]=0; If(k<n)then Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) B knap(K+1,cp,cw); If((cp>fp)and(k=n)) then { fp=cp; fw=cw; for j=1 to k do x[j]=y[j]; } } }
169
BRANCH AND BOUND ALGORITHM TECHNIQUE Introduction

Branch and bound is another algorithm technique that we are going to present in our multipart article series covering algorithm design patterns and techniques. B&B, as it is often abbreviated, is one of the most complex techniques and surely cannot be discussed in its entirety in a single article. Thus, we are going to focus on the so-called A* algorithm that is the most distinctive B&B graph search algorithm. If you have followed this article series then you know that we have already covered the most important techniques such as backtracking, the greedy strategy, divide and conquer, dynamic programming, and even genetic programming. As a result, in this part we will compare branch and bound with the previously mentioned techniques as well. It is really useful to understand the differences. Branch and bound is an algorithm technique that is often implemented for finding the optimal solutions in case of optimization problems; it is mainly used for combinatorial and discrete global optimizations of problems. In a nutshell, we opt for this technique when the domain of
SJCET, Palai
170
possible candidates is way too large and all of the other algorithms fail. This technique is based on the en masse elimination of the candidates. You should already be familiar with the tree structure of algorithms. Out of the techniques that we have learned, both the backtracking and divide and conquer traverse the tree in its depth, though they take opposite routes. The greedy strategy picks a single route and forgets about the rest. Dynamic programming approaches this in a sort of breadth-first search variation (BFS). Now if the decision tree of the problem that we are planning to solve has practically unlimited depth, then, by definition, the backtracking and divide and conquer algorithms are out. We shouldn't rely on greedy because that is problem-dependent and never promises to deliver a global optimum, unless proven otherwise mathematically. As our last resort we may even think about dynamic programming. The truth is that maybe the problem can indeed be solved with dynamic programming, but the implementation wouldn't be an efficient approach; additionally, it would be very hard to implement. You see, if we have a complex problem where we would need lots of parameters to describe the solutions of sub-problems, DP becomes inefficient. Branch and bound is a systematic method for solving optimization problems B&B is a rather general optimization technique that applies where the greedy method and dynamic programming fail. However, it is much slower. Indeed, it often leads to exponential time complexities in the worst case. On the other hand, if applied carefully, it can lead to algorithms that run reasonably fast on average. The general idea of B&B is a BFS-like search for the optimal solution, but not all nodes get expanded (i.e., their children generated). Rather, a carefully selected criterion determines which node to expand and when, and another criterion tells the algorithm when an optimal solution has been found. The basic concept underlying the branch-and-bound technique is to divide and conquer. Since the original large problem is hard to solve directly,it is divided into smaller and smaller subproblems until these subproblems can be conquered. The dividing (branching) is done by partitioning the entire set of feasible solutions into smaller and smaller subsets.The conquering (fathoming) is done partially by (i) giving a bound for the best solution in the subset;(ii) discarding the subset if the bound indicates that it cant contain an optimal solution.These
SJCET, Palai
171
three basic steps branching, bounding, and fathoming are illustrated on the following example.
Branch-and-Bound Algorithms A counter-part of the backtracking search algorithm which, in the absence of a cost criteria, the algorithm traverses a spanning tree of the solution space using the breadth-first approach. That is, a queue is used, and the nodes are processed in first-in-first-out order Procedure Expand(E) begin /* Generate all the children of E; */ I := E->I; X,p: nodepointer; S[1:n]: Boolean; /* S is a bitmap set initialized to 0*/ /* S will contain all the jobs that have been assigned by the partial path from the root to E */ p := E; while (p is not the root) do S[p->J] := 1; p := p-> Parent; endwhile for job=1 to n do if S[job] = 0 then X := new(node); X->I := I + 1; X->J := job; X->Parent := E; X->CC := E->CC + AX->I,X->J-mX->I; Insert(X,H); endif endfor Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) end .
172
If a cost criteria is available, the node to be expanded next (i.e., the branch) is the one with the best cost within the queue. In such a case, the cost function may also be used to discard (i.e., the bound) from the queue nodes that can be determined to be expensive. A priority queue is needed here. Cost-Based Tree Traversal of branch and bound A function can be considered to be a tree generator, if when given a node X and index i it produces the ith child of the node. The following function produces a complete binary tree of 11 nodes.
The recursive function provided for deriving permutations is another example of a function that may be used to generate trees. Besides for a tree generator function, we also need a cost function to decide in what order to traverse the nodes when searching for a solution. The algorithm proceeds in the following manner. 1. Initialization: The root of the of the tree is declared to be alive. 2. Visit: The cost criteria decides which of the live nodes is to process next. 3. Replacement: The chosen node is removed from the set of live nodes, and its children are inserted into the set. The children are determined by the tree generator function. 4. Iteration: The visitation and replacement steps are repeated until no alive nodes are left. Department of Computer Science & Engineering SJCET, Palai
173
In the case of backtracking the cost criteria assumes a last-in-first-out (LIFO) function, which can be realized with a stack memory. A first-in-first-out cost criteria implies the FIFO branch-and-bound algorithm, and it can be realized with queue memory. A generalization to arbitrary cost criteria is the basis for the priority branch-and-bound algorithm, and a priority queue memory can be employed to realize the function.
FIFO BRANCH AND BOUND ALGORITHM

FIFO branch and bound algorithm for the job sequencing problem can begin with upper= infinity. While implementing FIFO B&B algorithm it is not economical to kill live nodes with c> upper each time upper is updated. Each solution is assumed to be expressible as an array X[1:n] (as was seen in Backtracking). A predictor, called an approximate cost function CC, is assumed to have been defined. A live node is a node that has not been expanded.A dead node is a node that has been expanded. The expanded node (or E-node for short) is the live node with the best CC value. Branch and Bound is a general search method. Starting by considering the root problem (the original problem with the complete feasible region), the lower-bounding and upper-bounding procedures are applied to the root problem.If the bounds match, then an optimal solution has been found and the procedure terminates. Otherwise, the feasible region is divided into two or more regions, these subproblems partition the feasible region. The algorithm is applied recursively to the subproblems. If an optimal solution is found to a subproblem, it is a feasible solution to the full problem, but not necessarily globally optimal. If the lower bound for a node exceeds the best known feasible solution, no globally optimal solution can exist in the subspace of the feasible region represented by the node. Therefore, the node can be removed from consideration. The search proceeds until all nodes have been solved or pruned, or until some specified threshold is met between the best solution found and the lower bounds on all unsolved subproblems. A counter-part of the backtracking search algorithm which, in the absence of a cost criteria, the algorithm traverses a spanning tree of the solution space using the breadth-first approach. That is, a queue is used, and the nodes are processed in firstin-first-out order. If a cost criteria is available, the node to be expanded next (i.e., the branch) is the one with the best cost within the queue. In such a case, the cost function may also be used to discard (i.e., the bound) from the queue nodes that can be determined to be expensive. A priority queue is needed here.
SJCET, Palai
174
The general FIFO B&B algorithm follows:
Procedure B&B() begin E: nodepointer; E := new(node); -- this is the root node which -- is the dummy start node H: heap; -- A heap for all the live nodes
-- H is a min-heap for minimization problems, -- and a max-heap for maximization problems. while (true) do if (E is a final leaf) then -- E is an optimal solution print out the path from E to the root; return; endif Expand(E); if (H is empty) then report that there is no solution; return; endif E := delete-top(H); endwhile end
Procedure Expand(E) begin - Generate all the children of E; - Compute the approximate cost value CC of each child; - Insert each child into the heap H; end
SJCET, Palai
175
We need to define the full record of a node .We need to fully implement the Expand procedure Every node corresponds to something like X[i]=j, which signifies that the job X[i] assigned to person i is j. Every node must store its CC value. Each node must point to its parent so that when an optimal leaf is generated, the path from that leaf to the root can be traced and printed out as the optimal solution.Therefore, a node record structure should look like:
Record node Begin Parent: nodepointer; I: integer; -- person I J: integer; -- Job J is assigned to person I CC: real; End
Take the 2nd CC formula: CC(X at level k) = cost so far + sumni=k+1mi where mi is the minimum of row i. observe that if X is a pointer to a node, then X->CC = X->Parent->CC + AX->I,A->J - mX->I Write a piece of code that computes the mis for i=1,2,...,n Code for Expand(E): Procedure Expand(E) begin /* Generate all the children of E; */ I := E->I; X,p: nodepointer; S[1:n]: Boolean; /* S is a bitmap set initialized to 0*/ /* S will contain all the jobs that have been assigned by the partial path from the root to E */ p := E; while (p is not the root) do Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) S[p->J] := 1; p := p-> Parent; endwhile for job=1 to n do if S[job] = 0 then X := new(node); X->I := I + 1; X->J := job; X->Parent := E; X->CC := E->CC + AX->I,X->J-mX->I; Insert(X,H); endif endfor end.
176
Although a number of algorithms have been proposed for the integer linear programming problem, the FIFO branch-and-bound technique has proven to be reasonably efficient on practical problems, and it has the added advantage that it solves continuous linear programs as sub problems.The technique is also used in a lot of software in global optimization.
Fifo B&B Cost-Based Tree Traversal
A function can be considered to be a tree generator, if when given a node X and index i it produces the ith child of the node.The following function produces a complete binary tree of 11 nodes.
SJCET, Palai
177
The recursive function provided for deriving permutations is another example of a function that may be used to generate trees. Besides for a tree generator function, we also need a cost function to decide in what order to traverse the nodes when searching for a solution. The algorithm proceeds in the following manner. 1. Initialization: The root of the of the tree is declared to be alive. 2. Visit: The cost criteria decides which of the live nodes is to process next. 3. Replacement: The chosen node is removed from the set of live nodes, and its children are inserted into the set. The children are determined by the tree generator function. 4. Iteration: The visitation and replacement steps are repeated until no alive nodes are left. In the case of backtracking the cost criteria assumes a last-in-first-out (LIFO) function, which can be realized with a stack memory. A first-in-first-out cost criteria implies the FIFO branch-and-bound algorithm, and it can be realized with queue memory. A generalization to arbitrary cost criteria is the basis for the priority branch-and-bound algorithm, and a priority queue memory can be employed to realize the function.
Search the tree using a breadth-first search (FIFO branch and bound). Search the tree as in a bfs, but replace the FIFO queue with a stack (LIFO branch and bound). Replace the FIFO queue with a priority queue (least-cost (or max priority) branch and bound). The priority of a node p in the queue is based on an estimate of the likelihood that the answer node is in the subtree whose root is p. FIFO branch and bound finds solution closest to root.Backtracking may never find a solution because tree depth is infinite (unless repeating configurations are eliminated).
Least-cost branch and bound directs the search to parts of the space most likely to contain the answer. So it could perform better than backtracking.
SJCET, Palai
178
LIFO BRANCH AND BOUND ALGORITHM

LIFO branch and bound algorithm for the job sequencing problem can begin with upper= infinity. While implementing LIFO B&B algorithm it is not economical to kill live nodes with c> upper each time upper is updated. Each solution is assumed to be expressible as an array X[1:n] (as was seen in Backtracking). A predictor, called an approximate cost function CC, is assumed to have been defined. A live node is a node that has not been expanded.A dead node is a node that has been expanded. The expanded node (or E-node for short) is the live node with the best CC value. Branch and Bound is a general search method. Starting by considering the root problem (the original problem with the complete feasible region), the lower-bounding and upper-bounding procedures are applied to the root problem.If the bounds match, then an optimal solution has been found and the procedure terminates. Otherwise, the feasible region is divided into two or more regions, these subproblems partition the feasible region. The algorithm is applied recursively to the subproblems. If an optimal solution is found to a subproblem, it is a feasible solution to the full problem, but not necessarily globally optimal. If the lower bound for a node exceeds the best known feasible solution, no globally optimal solution can exist in the subspace of the feasible region represented by the node. Therefore, the node can be removed from consideration. The search proceeds until all nodes have been solved or pruned, or until some specified threshold is met between the best solution found and the lower bounds on all unsolved subproblems. A counter-part of the backtracking search algorithm which, in the absence of a cost criteria, the algorithm traverses a spanning tree of the solution space using the breadth-first approach. That is, a queue is used, and the nodes are processed in firstin-first-out order. If a cost criteria is available, the node to be expanded next (i.e., the branch) is the one with the best cost within the queue. In such a case, the cost function may also be used to discard (i.e., the bound) from the queue nodes that can be determined to be expensive. A priority queue is needed here. A general LIFO B&B algorithm :
Procedure B&B() begin E: nodepointer; E := new(node); -- this is the root node which -- is the dummy start node Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) H: heap; -- A heap for all the live nodes
179
-- H is a min-heap for minimization problems, -- and a max-heap for maximization problems. while (true) do if (E is a final leaf) then -- E is an optimal solution print out the path from E to the root; return; endif Expand(E); if (H is empty) then report that there is no solution; return; endif E := delete-top(H); endwhile end
Procedure Expand(E) begin - Generate all the children of E; - Compute the approximate cost value CC of each child; - Insert each child into the heap H; end
We need to define the full record of a node .We need to fully implement the Expand procedure Every node corresponds to something like X[i]=j, which signifies that the job X[i] assigned to person i is j. Every node must store its CC value. Each node must point to its parent so that when an optimal leaf is generated, the path from that leaf to the root can be traced and printed out as the optimal solution.Therefore, a node record structure should look like:
Record node Begin Department of Computer Science & Engineering SJCET, Palai
Algorithm Analysis and Design (R 606) Parent: nodepointer; I: integer; -- person I J: integer; -- Job J is assigned to person I CC: real; End
180
Take the 2nd CC formula: CC(X at level k) = cost so far + sumni=k+1mi where mi is the minimum of row i. observe that if X is a pointer to a node, then X->CC = X->Parent->CC + AX->I,A->J - mX->I Write a piece of code that computes the mis for i=1,2,...,n Although a number of algorithms have been proposed for the integer linear programming problem, the LIFO branch-and-bound technique has proven to be reasonably efficient on practical problems, and it has the added advantage that it solves continuous linear programs as sub problems.The technique is also used in a lot of software in global optimization.
Lifo B&B Cost-Based Tree Traversal
A function can be considered to be a tree generator, if when given a node X and index i it produces the ith child of the node.The following function produces a complete binary tree of 11 nodes.
The recursive function provided for deriving permutations is another example of a function that may be used to generate trees. Department of Computer Science & Engineering SJCET, Palai
181
Besides for a tree generator function, we also need a cost function to decide in what order to traverse the nodes when searching for a solution. The algorithm proceeds in the following manner. 5. Initialization: The root of the of the tree is declared to be alive. 6. Visit: The cost criteria decides which of the live nodes is to process next. 7. Replacement: The chosen node is removed from the set of live nodes, and its children are inserted into the set. The children are determined by the tree generator function. 8. Iteration: The visitation and replacement steps are repeated until no alive nodes are left. In the case of backtracking the cost criteria assumes a last-in-first-out (LIFO) function, which can be realized with a stack memory. A last-in-first-out cost criteria implies the LIFO branch-and-bound algorithm, and it can be realized with queue memory. A generalization to arbitrary cost criteria is the basis for the priority branch-and-bound algorithm, and a priority queue memory can be employed to realize the function. Search the tree using a breadth-first search (FIFO branch and bound). Search the tree as in a bfs, but replace the FIFO queue with a stack (LIFO branch and bound). Replace the FIFO queue with a priority queue (least-cost (or max priority) branch and bound). The priority of a node p in the queue is based on an estimate of the likelihood that the answer node is in the subtree whose root is p. LIFO branch and bound finds solution closest to root.Backtracking may never find a solution because tree depth is infinite (unless repeating configurations are eliminated). Least-cost branch and bound directs the search to parts of the space most likely to contain the answer. So it could perform better than backtracking.
SJCET, Palai
182
LC CONTROL ABSTRACTION
listnode= record { Listnode *next,*parent; Float cost; } Algorithm LCSearch(t) //Search t for an answer node. { If *t is an answer node then output t and return E:=t; //E-node Initalize the list of live nodes to be empty; Repeat { for each child x of E do { If x is an answer node then output the path from x to t and return; Add(x);//x is a new live node. (x->parent):=E;//Pointer for path to root } If there are no more live nodes then { Write(No answer node);return; } E:=Least(); }until(false); }
SJCET, Palai
183
15- PUZZLE
The 15 puzzle consists of 15 squares numbered from 1 to 15 that are placed in a box leaving one position out of the 16 empty. The goal is to reposition the squares from a given arbitrary starting arrangement by sliding them one at a time into the configuration shown above. For some initial arrangements, this rearrangement is possible, but for others, it is not.
The n-puzzle is known in various versions, including the 8 puzzle, the 15 puzzle, and with various names. It is a sliding puzzle that consists of a frame of numbered square tiles in random order with one tile missing. If the size is 33, the puzzle is called the 8-puzzle or 9puzzle, and if 44, the puzzle is called the 15-puzzle or 16-puzzle. The object of the puzzle is to place the tiles in order (see diagram) by making sliding moves that use the empty space. Theorem: The goal state is reachavle from intial state iff 16 i=1less(i)+x is even. For any state let less(i) be the number of tiles j such that j<I and position(j)>position(i).Let position(i) be the position number in the intial state of the tile numbered i.
SJCET, Palai
184
First ten steps in depth first search
SJCET, Palai
185
TRAVELLING SALESMAN PROBLEM

The Travelling Salesman problem (TSP) is a problem in combinatorial optimization studied in operations research and theoretical computer science. Given a list of cities and their pairwise distances, the task is to find a shortest possible tour that visits each city exactly once. It is used as a benchmark for many optimization methods. The TSP has several applications even in its purest formulation, such as planning, logistics, and the manufacture of microchips. Slightly modified, it appears as a sub-problem in many areas, such as genome sequencing. In these applications, the concept city represents, for example, customers, soldering points, or DNA fragments, and the concept distance represents travelling times or cost, or a similarity measure between DNA fragments.
Asymmetric and symmetric

In the symmetric TSP, the distance between two cities is the same in each direction. Thus, the underlying structure is an undirected graph between; especially, each tour has the same length in both directions. In the asymmetric TSP, the distance from one city to the other need not equal the distance in the other direction, in general, there may not even be a connection in the other direction.
Problem:
You are given a list of n cities along with the distances between each pair of cities. The goal is to find a tour which starts at the first city, visits each city exactly once and returns to the first city, such that the distance traveled is as small as possible. This problem is known to be NP-complete , i.e. no serial algorithm exists that runs in time polynomial in n, only in time exponential in n, and it is widely believed that no polynomial time algorithm exists. In practice, we want to compute an approximate solution, i.e. a single tour whose length is as short as possible, in a given amount of time. More formally, we are given a graph G=(N,V,W) consisting of a set N of n nodes (or cities), a set of edges V = {(i,j)} connecting cities, and a set of nonnegative weights W = {w(i,j)} giving the length of edge (i,j) (distance from city i to city j). The graph is directed, so that an Department of Computer Science & Engineering SJCET, Palai
186
edge (i,j) may only be traversed in the direction from i to j, and edge (j,i) may or may not exist. Similarly, w(i,j) does not necessarily equal w(j,i), if both edges exist.
Branch-and-Bound for TSP

A simple improvement on the algorithm prunes the search tree by observing that if a partial tour is already longer than the best solution found so far, there is no reason to continue searching that path.
Naive Branch-and-Bound Solution of TSP
w = w(1,2) + w(2,3) + w(3,4) + ... + w(n-1,n) + w(n,1) Best_S_so_far = ( n, [ 1, 2, 3, ... , n-1, n ], w ) S = ( 1, [ 1 ], 0 ) Search( S, Best_S_so_far ) print Best_S_so_far
procedure Search( S, Best_S_so_far )
let ( k, [ i1, i2, ... , ik ], w ) = S let ( n, [ i1B, i2B, ... , inB ], wB ) = Best_S_so_far if k = n then new_w = w + w(ik,i1) if new_w < wB then Best_S_so_far = ( k, [ i1, i2, ... , ik ], new_w ) end if else for all j not in [ i1, i2, ... , ik ] new_w = w + w(ik,j) if new_w < wB then New_S = ( k+1, [ i1, i2, ... , ik, j ], new_w ) Search( New_S, Best_S_so_far ) end if end for endif return end
Better Branch and Bound Algorithm for TSP
SJCET, Palai
187
Another strategy for searching the solution space is to repeatedly divide it into two parts: those with a given edge and those without the edge. The search tree would unfold as follows:
----------| all solns | ----------/ ---------------| solns with e_i | ---------------/ \ \ ----------------| solns w/out e_i | ----------------/ \
----------- ---------- ---------- ----------| with e_j | | w/out e_j | | with e_k | | w/out e_k | ----------- ---------- ---------- -----------
Bounding the solutions: Assume the input is given as a dense adjacency matrix. We will put infinities, rather than zeros on the diagonal to avoid traversing these self-edges.
i\j 1 2 3 4 5 6 7
\ ________________________________________ 1 2 3 4 5 6 7 |Inf 3 93 77 13 42 36 33 21 16 56 9 16 28 7 25 57 34 25 91 57 7
| 4 Inf | 45 | 39 | 28 | 3 | 44
17 Inf 90 46 88 26
80 Inf 88 18 33
33 Inf 46
92 Inf
27 84
39 Inf
We can subtract a constant from a given row or column, as long as the values remain nonnegative. This changes the weight of each tour, but not the set of legal tours or their relative weights. We therefore normalize the matrix by subtracting the minimum value in each row from the row and the minimum value of each column from the column. This results in a matrix with at least one zero in every row and column.
i\j 1 2 3 4 5 6 7
\ ________________________________________ 1 2 3 |Inf 0 83 66 9 37 19 30 17 0 6 12 12 50 26 5
| 0 Inf | 29
1 Inf
SJCET, Palai

4 5 6 7 | 32 | 3 | 0 | 18 83 21 85 0 66 Inf 56 8 0 49 0 0 80 28 0
188
7 Inf 42 0
89 Inf 58
13 Inf
Any solution must use one entry from every row and every column, so the sum of the values we just subtracted is a lower bound on the weight of solution. In this case, we subtracted [3 4 16 7 25 3 26] from the rows and then [0 0 7 1 0 0 4] from the columns, so the lower bound on the weight of any solution is 96. Representing the set of solutions: The adjacency matrix can be used to represent the set of solutions. When an edge is chosen for the solution, the row and column containing that edge is deleted. When an edge is eliminated from the solution, its entry is changed to infinity so that it will never be chosen. In the above example, assume we choose to split the search space on the edge from 4 to 6. For the right subtree, which represents all solutions not containing (4,6), we replace the (4,6) entry by infinity. The minimum value in row 4 is now 32, so we can renormalize the matrix and improve the lower bound to 96+32 = 128. Column 6 has another 0 entry, so it remains unchanged. The matrix for the right subtree is:
i\j 1 2 3 4 5 6 7
\ ________________________________________ 1 2 3 4 5 6 7 |Inf 0 83 66 9 37 19 30 17 0 6 12 12 50 26 5 48 28 0
| 0 Inf | 29 | 0 | 3 | 0 | 18
1 Inf 51 21 85 0
34 Inf 56 8 0
17 Inf 0
7 Inf 42 0
89 Inf 58
13 Inf
The left subtree represents all solutions containing (4,6). We therefore delete row 4 and column 6 from the matrix and renormalize. In addition, since we have used edge (4,6), edge (6,4) is no longer usable, so we replace the (6,4) with infinity.The matrix can now be renormalized, in this case by subtracting 3 from the row with i=5 (now the 4th row in the smaller matrix), yielding a lower bound of 96+3=99 and the matrix:
i\j 1 2 3 4 5 7
\ __________________________________ 1 |Inf 0 83 9 30 50
SJCET, Palai

2 3 5 6 7 | 0 Inf | 29 | 0 | 0 | 18 66 37 19 17 0 26 5 25 0
189
1 Inf 18 85 0 53
4 Inf 89
8 Inf 0 0
58 Inf
The process outlined so far can be used by repeatedly following these steps. Following the left branches we are guaranteed to reach some solution in n levels. Following the right branch, we will eventually have a matrix full of infinities, at which point the lower bound Choice of the splitting edge: In general, the right branches represent a larger set of solutions than the left branches. Therefore, in choosing the splitting edge, we look for something that will raise the lower bound of the right-hand subtree as much as possible. In the example, edge (4,6) was chosen because its value was zero and the next larger value in row 4 was 32. There are other zero entries in the matrix, for example, (3,5), but the improvement in the lower bound for that case would have been only 1 (for row 3) plus 17 (for column 5). Therefore, the general rule is to search for the zero entry at (i,j) that maximizes the increase in the lower bound (largest sum of minimum in row i and minimum in column j, not counting the zero at (i,j)). Insertion of infinities in left-branches: When expanding a left branch in the search tree for edge (i,j), we noted that edge (j,i) should be made infinity. In general, we are trying to avoid the creation of non-covering cycles in the tour, i.e., cycles that traverse only a subset of the nodes. For the example above, assume that the left-most search path unfolds with the following edge splits: (4,6), (3,5), and (2,1). At this point the partial solution contains three disconnected paths; all cycles were prevented by marking (6,4), (5,3), and (1,2) as infinity.
SJCET, Palai
190
----------| all solns | ----------/ -----------| with (4,6)| -----------/ -----------| with (3,5)| -----------/ -----------| with (2,1)| -----------\ -----------| w/out (2,1)| -----------\ ------------| w/out (3,5)| ------------\ ------------| w/out (4,6)| -------------
The next choice of splitting edge is (1,4), so according to our above rule (4,1) should be changed to infinity. However, at this point the adjacency matrix does not contain a row for 4, since it was removed with the (4,6) step. Moreover, the partial solution is now: 2-1-4-6 &3-5.
SJCET, Palai

Algorithm Analysis and Design

Uploaded by

Algorithm Analysis and Design

Uploaded by

Algorithm Analysis and Design (R 606)

Department of Computer Science & Engineering

Introduction and Complexity

Divide and Conquer

Department of Computer Science & Engineering