Introduction to Algorithms

6.046J/18.401J/SMA5503

Lecture 2
Prof. Erik Demaine

Solving recurrences
The analysis of merge sort from Lecture 1 required us to solve a recurrence. Recurrences are like solving integrals, differential equations, etc. o Learn a few tricks. Lecture 3: Applications of recurrences.

Day 3

Introduction to Algorithms

L2.2

Substitution method
The most general method: 1. Guess the form of the solution. 2. Verify by induction. 3. Solve for constants. Example: T(n) = 4T(n/2) + n [Assume that T(1) = (1).] Guess O(n3) . (Prove O and separately.) Assume that T(k) ck3 for k < n . Prove T(n) cn3 by induction.
Day 3 Introduction to Algorithms L2.3

Example of substitution
T (n) = 4T (n / 2) + n 4c ( n / 2 ) 3 + n = ( c / 2) n 3 + n desired residual = cn3 ((c / 2)n3 n) cn3 desired whenever (c/2)n3 n 0, for example, if c 2 and n 1. residual
Day 3 Introduction to Algorithms L2.4

Example (continued)
We must also handle the initial conditions, that is, ground the induction with base cases. Base: T(n) = (1) for all n < n0, where n0 is a suitable constant. For 1 n < n0, we have (1) cn3, if we pick c big enough.
This bound is not tight!
Day 3 Introduction to Algorithms L2.5

A tighter upper bound?

We shall prove that T(n) = O(n2). Assume that T(k) ck2 for k < n: T (n) = 4T (n / 2) + n 4cn 2 + n = O(n) Wrong! We must prove the I.H.

= cn 2 ( n) [ desired residual ] cn 2 for no choice of c > 0. Lose!

Day 3 Introduction to Algorithms L2.6

A tighter upper bound!

IDEA: Strengthen the inductive hypothesis. Subtract a low-order term. Inductive hypothesis: T(k) c1k2 c2k for k < n. T (n) = 4T (n / 2) + n 4(c1 (n / 2) 2 c2 (n / 2) + n = c1n 2 2c2 n + n = c1n 2 c2 n (c2 n n) c1n 2 c2 n if c2 > 1.

Pick c1 big enough to handle the initial conditions.

Day 3 Introduction to Algorithms L2.7

Recursion-tree method
A recursion tree models the costs (time) of a recursive execution of an algorithm. The recursion tree method is good for generating guesses for the substitution method. The recursion-tree method can be unreliable, just like any method that uses ellipses (). The recursion-tree method promotes intuition, however.
Day 3 Introduction to Algorithms L2.8

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2:

Day 3

Introduction to Algorithms

L2.9

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2: T(n)

Day 3

Introduction to Algorithms

L2.10

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2: n2 T(n/4) T(n/2)

Day 3

Introduction to Algorithms

L2.11

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 T(n/16) T(n/8) (n/2)2 T(n/8) T(n/4)

Day 3

Introduction to Algorithms

L2.12

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 (n/16)2 (1)
Day 3 Introduction to Algorithms L2.13

(n/2)2 (n/8)2 (n/4)2

(n/8)2

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 (n/16)2 (1)
Day 3 Introduction to Algorithms L2.14

n2

(n/2)2 (n/8)2 (n/4)2

(n/8)2

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 (n/16)2 (1)
Day 3 Introduction to Algorithms L2.15

n2
5 n2 16

(n/2)2 (n/8)2 (n/4)2

(n/8)2

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 (n/16)2 (1)
Day 3 Introduction to Algorithms L2.16

n2
5 n2 16 25 n 2 256

(n/2)2 (n/8)2 (n/4)2

(n/8)2

Example of recursion tree

Solve T(n) = T(n/4) + T(n/2) + n2: n2 (n/4)2 (n/16)2

n2
5 n2 16 25 n 2 256

5 3 16

(n/2)2 (n/8)2 (n/4)2

(n/8)2

(1)
Day 3

Total = n = (n2)

5 + 5 2 1 + 16 16

+L geometric series
L2.17

( ) +( )

Introduction to Algorithms

The master method

The master method applies to recurrences of the form T(n) = a T(n/b) + f (n) , where a 1, b > 1, and f is asymptotically positive.

Day 3

Introduction to Algorithms

L2.18

Three common cases

Compare f (n) with nlogba: 1. f (n) = O(nlogba ) for some constant > 0. f (n) grows polynomially slower than nlogba (by an n factor). Solution: T(n) = (nlogba) . 2. f (n) = (nlogba lgkn) for some constant k 0. f (n) and nlogba grow at similar rates. Solution: T(n) = (nlogba lgk+1n) .
Day 3 Introduction to Algorithms L2.19

Three common cases (cont.)

Compare f (n) with nlogba: 3. f (n) = (nlogba + ) for some constant > 0. f (n) grows polynomially faster than nlogba (by an n factor), and f (n) satisfies the regularity condition that a f (n/b) c f (n) for some constant c < 1. Solution: T(n) = ( f (n)) .

Day 3

Introduction to Algorithms

L2.20

Examples
Ex. T(n) = 4T(n/2) + n a = 4, b = 2 nlogba = n2; f (n) = n. CASE 1: f (n) = O(n2 ) for = 1. T(n) = (n2). Ex. T(n) = 4T(n/2) + n2 a = 4, b = 2 nlogba = n2; f (n) = n2. CASE 2: f (n) = (n2lg0n), that is, k = 0. T(n) = (n2lg n).
Day 3 Introduction to Algorithms L2.21

Examples
Ex. T(n) = 4T(n/2) + n3 a = 4, b = 2 nlogba = n2; f (n) = n3. CASE 3: f (n) = (n2 + ) for = 1 and 4(cn/2)3 cn3 (reg. cond.) for c = 1/2. T(n) = (n3). Ex. T(n) = 4T(n/2) + n2/lg n a = 4, b = 2 nlogba = n2; f (n) = n2/lg n. Master method does not apply. In particular, for every constant > 0, we have n = (lg n).
Day 3 Introduction to Algorithms L2.22

General method (Akra-Bazzi)

T (n) = aiT (n / bi ) + f (n)
i =1 k k

Let p be the unique solution to

(ai /bip ) = 1.
Then, the answers are the same as for the master method, but with np instead of nlogba. (Akra and Bazzi also prove an even more general result.)
Day 3 Introduction to Algorithms L2.23

i =1

Idea of master theorem

Recursion tree:

a f (n/b) f (n/b) f (n/b) a h = logbn f (n/b2) f (n/b2) f (n/b2)

f (n)

f (n) a f (n/b) a2 f (n/b2) nlogba (1)

L2.24

(1)
Day 3

#leaves = ah = alogbn = nlogba

Introduction to Algorithms

Idea of master theorem

Recursion tree:

a f (n/b) f (n/b) f (n/b) a h = logbn f (n/b2) f (n/b2) f (n/b2)

CASE 1: The weight increases CASE 1: The weight increases geometrically from the root to the geometrically from the root to the (1) leaves. The leaves hold aaconstant leaves. The leaves hold constant fraction of the total weight. fraction of the total weight.
Day 3 Introduction to Algorithms

f (n)

f (n) a f (n/b) a2 f (n/b2) nlogba (1) (nlogba)

L2.25

Idea of master theorem

Recursion tree:

a f (n/b) f (n/b) f (n/b) a h = logbn f (n/b2) f (n/b2) f (n/b2)

f (n)

f (n) a f (n/b) a2 f (n/b2) nlogba (1) (nlogbalg n)

L2.26

(1)
Day 3

CASE 2: (k = 0) The weight CASE 2: (k = 0) The weight is approximately the same on is approximately the same on each of the logbbn levels. each of the log n levels.
Introduction to Algorithms

Idea of master theorem

Recursion tree:

a f (n/b) f (n/b) f (n/b) a h = logbn f (n/b2) f (n/b2) f (n/b2)

CASE 3: The weight decreases CASE 3: The weight decreases geometrically from the root to the geometrically from the root to the (1) leaves. The root holds aaconstant leaves. The root holds constant fraction of the total weight. fraction of the total weight.
Day 3 Introduction to Algorithms

f (n)

f (n) a f (n/b) a2 f (n/b2) nlogba (1) ( f (n))

L2.27

Conclusion

Next time: applying the master method. For proof of master theorem, see CLRS.

Day 3

Introduction to Algorithms

L2.28

Appendix: geometric series

1 x n +1 for x 1 1 + x + x2 + L + xn = 1 x
1 1+ x + x +L = for |x| < 1 1 x
2
Return to last slide viewed.

Day 3

Introduction to Algorithms

L2.29