Distributed & Parallel Algorithms

(CS 18301)

Coordinator: Dr. Ashish Kumar Maurya

Teacher : Shailendra Kumar Singh

By:- Shailendra Kumar Singh, CSE Department, MNNIT

1 Allahabad
 Course/Subject Outline
 Algorithms
 Distributed Processing
 Parallel Processing
 Flynn’s classification and Amdahl’s law
 Abstract Model of Parallel Computing
 Parallel Random Access Machine
 Topologies
 Parallel Sorting and Searching
 Parallel Matrix Operations
 Graph and List Application

By:- Shailendra Kumar Singh, CSE Department, MNNIT

2 Allahabad
What are algorithms?
 Informally, an algorithm is any well-defined
computational procedure that takes some
value, or set of values, as input and produces
some value, or set of values, as output.

 An algorithm is thus a sequence of

computational steps that transform the input
into the output.

By:- Shailendra Kumar Singh, CSE Department, MNNIT

3 Allahabad
Why study algorithms ?
 Algorithms help us to understand scalability.

 Performance often draws the line between

what is feasible and what is impossible.

 Algorithmic mathematics provides a language

for talking about program behavior.

By:- Shailendra Kumar Singh, CSE Department, MNNIT

4 Allahabad
Algorithm Analysis
 Analyzing an algorithm means, predicting the resources
that the algorithm requires. Occasionally, resources
such as memory, communication bandwidth, or
computer hardware are of primary concern, but most
often it is computational time that we want to
measure.

 Number of algorithms can be written to solve a single

problem. So by analyzing the algorithms, we can find
the best solution (algorithm) to the problem.
By:- Shailendra Kumar Singh, CSE Department, MNNIT
5 Allahabad
Order of growth
• Any algorithm is expected to work fast for any
input size.
• For smaller input size our algorithm will work fine
but for higher input size the execution time is
much higher.
• By increasing the size of n (input size) we can
analyze how well our algorithm works.

By:- Shailendra Kumar Singh, CSE Department, MNNIT

6 Allahabad
 Cont..
Let input size, n=5 and we have to sort the list of
elements for e.g. 25,29,10,15,2.

• So for n=5 our algorithm will work fine but what if

n=5000?
• So our algorithm will take much longer time to sort
the elements or cause small delays to give the
result.
• How the behavior of algorithm changes with the no.
of inputs will give the analysis of the algorithm and
By:- Shailendra Kumar Singh, CSE Department, MNNIT
7 is Allahabad
called the Order of Growth.
 Order of Growth
1. Big O (Oh)
2. Big Ω (Omega)
3. Big θ (Theta)
4. Small o
5. Small ω

By:- Shailendra Kumar Singh, CSE Department, MNNIT

8 Allahabad
 What kinds of problems are solved by
algorithms?
 Sorting
 E commerce
 Graph searching
 …

By:- Shailendra Kumar Singh, CSE Department, MNNIT

9 Allahabad
 Cont..
The problem of sorting

Input: Sequence 〈a1, a2, ..., an〉 of numbers.

Output: Permutation 〈a'1, a '2, ..., a'n〉
such that a'1≤ a'2≤ ...≤ a'n .

 Example:
Input: 8 2 4 9 3 6
Output: 2 3 4 6 8 9
By:- Shailendra Kumar Singh, CSE Department, MNNIT
10 Allahabad
 Insertion sort
 Given array divided into two sub-arrays.
 Pick first value from unsorted sub-array and store in
sorted sub-array.
 Sorted
 Unsorted

Example

By:- Shailendra Kumar Singh, CSE Department, MNNIT

11 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

12 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

13 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

14 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

15 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

16 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

17 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

18 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

19 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

20 Allahabad
Insertion sort algorithm

By:- Shailendra Kumar Singh, CSE Department, MNNIT

21 Allahabad
 Time Complexity
INSERTION-SORT (A) Cost Times

1 for j <- 2 to length[A] C1 2,3…(n+1)= n

2 do key <- A[j] C2 n-1

3 i <- j - 1 C3 n-1
4 while i > 0 and A[i] > key C4
5 do A[i + 1] <- A[i] C5
6 i <- i - 1 C6
7 A[i + 1] <- key
C7 n-1
By:- Shailendra Kumar Singh, CSE Department, MNNIT
22 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

23 Allahabad
 Cont..

By:- Shailendra Kumar Singh, CSE Department, MNNIT

24 Allahabad
 Cont..
INSERTION-SORT (A) Cost Times

1 for j <- 2 to length[A] C1 2,3…(n+1)= n

2 do key <- A[j] C2 n-1

3 i <- j - 1 C3 n-1
4 while i > 0 and A[i] > key C4 (n-1), ((n-1)n)/2
5 do A[i + 1] <- A[i] C5 0 , ((n-1)n)/2
6 i <- i - 1 C6 0 , ((n-1)n)/2
7 A[i + 1] <- key
C7 n-1
By:- Shailendra Kumar Singh, CSE Department, MNNIT
25 Allahabad
 Cont..
Best Worst
Cost Times
C1*n C1*n
C1 2,3…(n+1)= n
C2 n-1 C2*n-1 C2*n-1

C3 n-1 C3*n-1 C3*n-1

C4 (n-1), ((n-1)n)/2 C4*n-1 C4*(n^2-n)/2

C5 0 , ((n-1)n)/2 C5*0 C5*(n^2-n)/2

C6 0 , ((n-1)n)/2 C6*0 C6*(n^2-n)/2
C7 n-1 C7*n-1 C7*n-1

By:- Shailendra Kumar Singh, CSE Department, MNNIT

=an+b =a’n^2+b’n+c
26 Allahabad
 Time & Space Complexity
Length of A=n

Best Case= an+b => n+1 => O(n)

Worst Case= a’n^2+b’n+c => n^2+n+1=> O(n^2)

eg. n=100, n^2=10,000

Space Complexity= 3 => O(1)

By:- Shailendra Kumar Singh, CSE Department, MNNIT

27 Allahabad
 Big O
o It represents the maximum time that the algorithm takes for its
execution.
o Worst Case
o Upper bound

o Growth rate of f(n) is not more than growth rate of g(n).

there exist c>0 and n0>0 , n ≥ n0
f(n) ≤ c.g(n) => f(n) = O(g(n))
Eg. 2n=O(n) => 2n ≤ c.n; 2 ≤c;
2n ≤2*n => n ≥1 => n0=1
By:- Shailendra Kumar Singh, CSE Department, MNNIT
f(n)=2n+1 ≤ c.( ?) { c ≥3, n ≥1}
28 Allahabad
 Big Ω
 It represents the minimum time that the algorithm
takes for its execution.
 Best case
 Lower bound

 Growth rate of f(n) is not less than growth rate of g(n).

there exist c>0 and n0>0 , n ≥ n0
f(n) ≥ c. g(n) f(n) = Ω (g(n))
Eg. F(n)=2n+1; 2n+1 ≥c.(?) 2n+1 ≥ c.(n) { c ≤ 2, n ≥0 }
By:- Shailendra Kumar Singh, CSE Department, MNNIT
29 Allahabad
 Big θ
 It represents the average time an algorithm takes for its
execution.
 Average case
 Exact time

 Let c1 . g(n) and c2 . g(n) are two constant multiples of

g(n). c1, c2 and n0>0 , n ≥ n0
0 ≤ c1.g(n) ≤ f(n) ≤ c2. g(n) f(n) = θ(g(n))
Eg. c1.g(n) ≤ 2n+1≤ c2. g(n) c1.n ≤ 2n+1≤ c2.n
By:- Shailendra Kumar Singh, CSE Department, MNNIT
30 Allahabad
Small o and small ω
Small o
f(n) < c.g(n) => f(n) = o(g(n))
If for all c>0 and n0>0 , n ≥ n0
Eg. 2n=O(n) but 2n ≠ o(n);
2n < c.n 2n < 2.1 * n {c>2}; 2n=o(n^2) {c>0}
Small ω
f(n) > c. g(n) => f(n) = Ω (g(n))
For all c>0 and n0>0 , n ≥ n0
{ Iff g(n)=o(f(n)) } 2n=o(n^2) g(n)=2n, f(n)=n^2
f(n) > c. g(n) n^2 > c(2n) {c>0}
By:- Shailendra Kumar Singh, CSE Department, MNNIT
31 Allahabad