Divide and Conquer
APPLIED ALGORITHMS
1 Background
2 Merge Sort
3 Binary Search
4 Binary Search Over Values
5 Other Types of D&C
3 / 35
Divide and Conquer
D&C is a problem-solving paradigm in which a problem is made simpler by
‘dividing’ it into smaller parts and then conquering each part.
There are usually 3 main steps:
1 DIVIDE: split the problem into one or more smaller subproblems -
usually by half or nearly half
2 CONQUER: solve each of these subproblems recursively - which are
now easier
3 COMBINE: combine the solutions to the subproblems into a solution
of the given problem
4 / 35
Standard divide and conquer algorithms
Quicksort
Mergesort
Karatsuba algorithm
Strassen algorithm
Many algorithms from computational geometry
I Convex hull
I Closest pair of points
5 / 35
Applications of D&C
Solving difficult problems: breaking the problem into sub-problems,
solving the trivial cases and combining sub-problems to the original
problem
Parallelism: naturally adapted for execution in multi-processor
machines, especially shared-memory systems where the
communication of data between processors does not need to be
planned in advance, because distinct sub-problems can be executed
on different processors
Memory access: naturally tend to make efficient use of memory
caches. Once a sub-problem is small enough, it and all its
sub-problems can, in principle, be solved within the cache, without
accessing the slower main memory
...
6 / 35
Analysis of D&C
Described by recursive equation
Suppose T (n) is the running time on a problem of size n
(
Θ(1) if n ≤ nc
T (n) =
aT (n/b) + D(n) + C (n) if n ≥ nc ,
where
a: number of subproblems
n/b: size of each subproblem
D(n): cost of divide operation
C (n): cost of combination operation
7 / 35
Time complexity
1 void solve ( int n ) {
2 if ( n == 0)
3 return ;
4
5 solve ( n /2);
6 solve ( n /2);
7
8 for ( int i = 0; i < n ; i ++) {
9 // some constant time operations
10 }
11 }
What is the time complexity of this divide and conquer algorithm?
Usually helps to model the time complexity as a recurrence relation:
I T (n) = 2T (n/2) + n
8 / 35
Time complexity
But how do we solve such recurrences?
Usually simplest to use the Master theorem when applicable
I It gives a solution to a recurrence of the form T (n) = aT (n/b) + f (n)
in asymptotic terms
I All of the divide and conquer algorithms mentioned so far have a
recurrence of this form
The Master theorem tells us that T (n) = 2T (n/2) + n has
asymptotic time complexity O(n log n)
The recurrence tree method is also very useful to solve such
recurrences.
9 / 35
Decrease and conquer
Sometimes we’re not actually dividing the problem into many
subproblems, but only into one smaller subproblem
Usually called decrease and conquer
The most common example of this is binary search
10 / 35
Merge Sort
Divide: divide the n-element sequence into two subproblems of n/2
elements each
Conquer: sort the two subsequences recursively using merge sort. If
the length of a sequence is 1, do nothing since it is already in order
Combine merge the two sorted subsequences to produce the sorted
answer
11 / 35
Merge Sort – Merge Function
Merge is the key operation in merge sort
Suppose the (sub)sequence(s) are stored in the array A. Moreover,
A[p . . . q] and A[q + 1 . . . r ] are two sorted subsequences.
MERGE(A,p,q,r) will merge the two subsequences into sorted
sequence A[p . . . r ]
MERGE(A,p,q,r) takes Θ(r − p + 1)
1 MERGE_SORT (A ,p , r ){
2 if (p < r ) {
3 q =( p + r )/2
4 MERGE_SORT (A ,p , q )
5 MERGE_SORT (A , q +1 , r )
6 MERGE (A ,p ,q , r )}
7 }
Call to MERGE-SORT(A,1,n) (suppose n=length(A))
12 / 35
Analysis of Merge Sort
Divide: D(n) = Θ(1)
Conquer: a = 2, b = 2, so 2T (n/2)
Combine C (n) = Θ(n)
(
Θ(1) if n = 1
T (n) =
2T (n/2) + Θ(n) if n > 1
(
c if n = 1
T (n) =
2T (n/2) + cn if n > 1
T (n) = O(n log n) by Recursive Tree or Master Theorem
13 / 35
Binary Search
We have a sorted array of elements, and we want to check if it
contains a particular element x
Algorithm:
1 Base case: the array is empty, return false
2 Compare x to the element in the middle of the array
3 If it’s equal, then we found x and we return true
4 If it’s less, then x must be in the left half of the array
1 Binary search the element (recursively) in the left half
5 If it’s greater, then x must be in the right half of the array
1 Binary search the element (recursively) in the right half
14 / 35
Binary search
1 bool binary_search ( const vector < int > & arr , int lo , int hi , int x ){
2 if ( lo > hi ) {
3 return false ;
4 }
5
6 int m = ( lo + hi ) / 2;
7 if ( arr [ m ] == x ) {
8 return true ;
9 } else if ( x < arr [ m ]) {
10 return binary_search ( arr , lo , m - 1 , x );
11 } else if ( x > arr [ m ]) {
12 return binary_search ( arr , m + 1 , hi , x );
13 }
14 }
15
16 binary_search ( arr , 0 , arr . size () - 1 , x );
T (n) = T (n/2) + 1
O(log n)
15 / 35
Binary search - iterative
1 bool binary_search ( const vector < int > & arr , int x ) {
2 int lo = 0 ,
3 hi = arr . size () - 1;
4
5 while ( lo <= hi ) {
6 int m = ( lo + hi ) / 2;
7 if ( arr [ m ] == x ) {
8 return true ;
9 } else if ( x < arr [ m ]) {
10 hi = m - 1;
11 } else if ( x > arr [ m ]) {
12 lo = m + 1;
13 }
14 }
15
16 return false ;
17 }
16 / 35
Binary search over integers
This might be the most well known application of binary search, but
it’s far from being the only application
More generally, we have a predicate p : {0, . . . , n − 1} → {T , F }
which has the property that if p(i) = T , then p(j) = T for all j > i
Our goal is to find the smallest index j such that p(j) = T as quickly
as possible
i 0 1 ··· j −1 j j +1 ··· n−2 n−1
p(i) F F ··· F T T ··· T T
We can do this in O(log(n) × f ) time, where f is the cost of
evaluating the predicate p, in the same way as when we were binary
searching an array
17 / 35
Binary search over integers
1 int lo = 0 ,
2 hi = n - 1;
3
4 while ( lo < hi ) {
5 int m = ( lo + hi ) / 2;
6
7 if ( p ( m )) {
8 hi = m ;
9 } else {
10 lo = m + 1;
11 }
12 }
13
14 if ( lo == hi && p ( lo )) {
15 printf ( " lowest index is % d \ n " , lo );
16 } else {
17 printf ( " no such index \ n " );
18 }
18 / 35
Binary search over integers
Find the index of x in the sorted array arr
1 bool p ( int i ) {
2 return arr [ i ] >= x ;
3 }
Later we’ll see how to use this in other ways
19 / 35
Binary search over reals
An even more general version of binary search is over the real numbers
We have a predicate p : [lo, hi] → {T , F } which has the property that
if p(i) = T , then p(j) = T for all j > i
Our goal is to find the smallest real number j such that p(j) = T as
quickly as possible
Since we’re working with real numbers (hypothetically), our [lo, hi]
can be halved infinitely many times without ever becoming a single
real number
Instead it will suffice to find a real number j 0 that is very close to the
correct answer j, say not further than EPS = 2−30 away
We can do this in O(log( hi−lo
EPS )) time in a similar way as when we
were binary searching an array
20 / 35
Binary search over reals
1 double EPS = 1e -10 ,
2 lo = -1000.0 ,
3 hi = 1000.0;
4
5 while ( hi - lo > EPS ) {
6 double mid = ( lo + hi ) / 2.0;
7
8 if ( p ( mid )) {
9 hi = mid ;
10 } else {
11 lo = mid ;
12 }
13 }
14
15 printf ( " %0.10 lf \ n " , lo );
21 / 35
Binary search over reals
This has many cool numerical applications
Find the square root of x
1 bool p ( double j ) {
2 return j * j >= x ;
3 }
Find the root of an increasing function f (x)
1 bool p ( double x ) {
2 return f ( x ) >= 0.0;
3 }
This is also referred to as the Bisection method
22 / 35
Practicing problem
Pie
23 / 35
Binary search the answer
It may be hard to find the optimal solution directly, as we saw in the
example problem
On the other hand, it may be easy to check if some x is a solution or
not
A method of using binary search to find the minimum or maximum
solution to a problem
Only applicable when the problem has the binary search property: if i
is a solution, then so are all j > i
p(i) checks whether i is a solution, then we simply apply binary
search on p to get the minimum or maximum solution
24 / 35
Other types of divide and conquer
Binary search is very useful, can be used to construct simple and
efficient solutions to problems
But binary search is only one example of divide and conquer
Let’s explore two more examples
25 / 35
Binary exponentiation
We want to calculate x n , where x, n are integers
Assume we don’t have the built in pow method
Naive method:
1 int pow ( int x , int n ) {
2 int res = 1;
3 for ( int i = 0; i < n ; i ++) {
4 res = res * x ;
5 }
6
7 return res ;
8 }
This is O(n), but what if we want to support large n efficiently?
26 / 35
Binary exponentiation
Let’s use divide and conquer
Notice the three identities:
I x0 = 1
I x n = x × x n−1
I x n = x n/2 × x n/2
Or in terms of our function:
I pow (x, 0) = 1
I pow (x, n) = x × pow (x, n − 1)
I pow (x, n) = pow (x, n/2) × pow (x, n/2)
pow (x, n/2) is used twice, but we only need to compute it once:
I pow (x, n) = pow (x, n/2)2
27 / 35
Binary exponentiation
Let’s try using these identities to compute the answer recursively
1 int pow ( int x , int n ) {
2 if ( n == 0) return 1;
3 return x * pow (x , n - 1);
4 }
28 / 35
Binary exponentiation
Let’s try using these identities to compute the answer recursively
1 int pow ( int x , int n ) {
2 if ( n == 0) return 1;
3 return x * pow (x , n - 1);
4 }
How efficient is this?
I T (n) = 1 + T (n − 1)
28 / 35
Binary exponentiation
Let’s try using these identities to compute the answer recursively
1 int pow ( int x , int n ) {
2 if ( n == 0) return 1;
3 return x * pow (x , n - 1);
4 }
How efficient is this?
I T (n) = 1 + T (n − 1)
I O(n)
28 / 35
Binary exponentiation
Let’s try using these identities to compute the answer recursively
1 int pow ( int x , int n ) {
2 if ( n == 0) return 1;
3 return x * pow (x , n - 1);
4 }
How efficient is this?
I T (n) = 1 + T (n − 1)
I O(n)
I Still just as slow...
28 / 35
Binary exponentiation
What about the third identity?
I n/2 is not an integer when n is odd, so let’s only use it when n is even
1 int pow ( int x , int n ) {
2 if ( n == 0) return 1;
3 if ( n % 2 != 0) return x * pow (x , n - 1);
4 int st = pow (x , n /2);
5 return st * st ;
6 }
How efficient is this?
29 / 35
Binary exponentiation
What about the third identity?
I n/2 is not an integer when n is odd, so let’s only use it when n is even
1 int pow ( int x , int n ) {
2 if ( n == 0) return 1;
3 if ( n % 2 != 0) return x * pow (x , n - 1);
4 int st = pow (x , n /2);
5 return st * st ;
6 }
How efficient is this?
I T (n) = 1 + T (n − 1) if n is odd
I T (n) = 1 + T (n/2) if n is even
29 / 35
Binary exponentiation
What about the third identity?
I n/2 is not an integer when n is odd, so let’s only use it when n is even
1 int pow ( int x , int n ) {
2 if ( n == 0) return 1;
3 if ( n % 2 != 0) return x * pow (x , n - 1);
4 int st = pow (x , n /2);
5 return st * st ;
6 }
How efficient is this?
I T (n) = 1 + T (n − 1) if n is odd
I T (n) = 1 + T (n/2) if n is even
I Since n − 1 is even when n is odd:
I T (n) = 1 + 1 + T ((n − 1)/2) if n is odd
29 / 35
Binary exponentiation
What about the third identity?
I n/2 is not an integer when n is odd, so let’s only use it when n is even
1 int pow ( int x , int n ) {
2 if ( n == 0) return 1;
3 if ( n % 2 != 0) return x * pow (x , n - 1);
4 int st = pow (x , n /2);
5 return st * st ;
6 }
How efficient is this?
I T (n) = 1 + T (n − 1) if n is odd
I T (n) = 1 + T (n/2) if n is even
I Since n − 1 is even when n is odd:
I T (n) = 1 + 1 + T ((n − 1)/2) if n is odd
I O(log n)
I Fast!
29 / 35
Binary exponentiation
Notice that x doesn’t have to be an integer, and ? doesn’t have to be
integer multiplication...
It also works for:
I Computing x n , where x is a floating point number and ? is floating
point number multiplication
I Computing An , where A is a matrix and ? is matrix multiplication
I Computing x n (mod m), where x is a matrix and ? is integer
multiplication modulo m
I Computing x ? x ? · · · ? x, where x is any element and ? is any
associative operator
All of these can be done in O(log(n) × f ), where f is the cost of
doing one application of the ? operator
30 / 35
Fibonacci words
Recall that the Fibonacci sequence can be defined as follows:
I fib1 = 1
I fib2 = 1
I fibn = fibn−2 + fibn−1
We get the sequence 1, 1, 2, 3, 5, 8, 13, 21, . . .
There are many generalizations of the Fibonacci sequence
One of them is to start with other numbers, like:
I f1 = 5
I f2 = 4
I fn = fn−2 + fn−1
We get the sequence 5, 4, 9, 13, 22, 35, 57, . . .
What if we start with something other than numbers?
31 / 35
Fibonacci words
Let’s try starting with a pair of strings, and let + denote string
concatenation:
I g1 = A
I g2 = B
I gn = gn−2 + gn−1
Now we get the sequence of strings:
I A
I B
I AB
I BAB
I ABBAB
I BABABBAB
I ABBABBABABBAB
I BABABBABABBABBABABBAB
I ...
32 / 35
Fibonacci words
How long is gn ?
I len(g1 ) = 1
I len(g2 ) = 1
I len(gn ) = len(gn−2 ) + len(gn−1 )
Looks familiar?
len(gn ) = fibn
So the strings become very large very quickly
I len(g10 ) = 55
I len(g100 ) = 354224848179261915075
I len(g1000 ) =
434665576869374564356885276750406258025646605173717
804024817290895365554179490518904038798400792551692
959225930803226347752096896232398733224711616429964
409065331879382989696499285160037044761377951668492
28875
33 / 35
Fibonacci words
Task: Compute the ith character in gn
34 / 35
Fibonacci words
Task: Compute the ith character in gn
Simple to do in O(len(n)), but that is extremely slow for large n
34 / 35
Fibonacci words
Task: Compute the ith character in gn
Simple to do in O(len(n)), but that is extremely slow for large n
Can be done in O(n) using divide and conquer
34 / 35
Practicing problem
Fibonacci Words
35 / 35
