18-645/SP07: How to Write Fast Code

Assignment 1
Due Date: Thu Jan 31 6:00pm
http://www.ece.cmu.edu/∼pueschel/teaching/18-645-CMU-spring08/course.html

Submission instructions: If you have an electronic version of your assignment (preferably made using
LATEX, but other forms, including scanned copies are okay), email them to: <schellap+18645-assign1@andrew.cmu.edu>.
Paper submissions need to be dropped off with one of the TAs at PH-B10 or with Carol Patterson at PH-B15.
Late submissions will not be graded.

1. (9 pts) Show that the following identities hold by determining the explicit constants c and n0 that are
a part of the definition of O.

(a) n + 1 = O(n)
(b) n3 + 2n2 + 3n + 4 = O(n3 )
(c) n5 = O(nlog2 n )

Solution:

(a) n + 1 = O(n)
For all n ≥ 1,
n + 1 ≤ n + n = 2n.
Thus, for c = 2, n0 = 1, we have n + 1 ≤ cn for all n ≥ n0 .
(b) n3 + 2n2 + 3n + 4 = O(n3 )
The trick is to get rid of all lower terms by converting them into multiples of n3 .
For all n ≥ 1:

n3 + 2n2 + 3n + 4 ≤ n3 + 2n3 + 3n3 + 4n3

≤ 10n3 .

Therefore, we can choose c = 10, n0 = 1 in the definition of O.

(c) n5 = O(nlog2 n )
Since 5 ≤ log2 (n), for n ≥ 25 , we have for n ≥ 32,

n5 ≤ nlog2 (n) .

Hence, we can choose c = 1, n0 = 32.

2. (14 pts) You know that O(n + 1) = O(n). Similarly, simplify the following as much as possible and
briefly justify.
(a) O(100)
(b) O(100 + (1/n))
√
(c) O(3n2 + n)
(d) O(log3 (n))
(e) O(n2.1 + n2 log(n))
(f) O(mn + n)
(g) O(m log(n) + n log(m) + n)
Solution:
(a) O(100)
Constants don’t matter, so O(100) = O(1).

18-645 SP 2007 / Assignment 1 Pg 1 of 6 Electrical & Computer Engineering

Instructor: Markus Püschel Carnegie Mellon University
 (b) O(100 + (1/n))
As n grows, (1/n) goes negligible compared to 100 because limn→∞ 1/n
100 = 0. Therefore O(100 +
(1/n)) = O(1).
√
(c) O(3n2 + n)
√ √ √
n is negligible compared to 3n2 because limn→∞ 3nn2 = 0. Hence O(3n2 + n) = O(n2 ).
(d) O(log3 (n))
As seen in class, O(log3 (n)) = O(log(n)).
(e) O(n2.1 + n2 log(n))
n2 log(n)
n2 log(n) is negligible compared to n2.1 because limn→∞ n2.1 = 0. Therefore O(n2.1 +
n2 log(n)) = O(n2.1 ).
(f) O(mn + n)
n
n is negligible compared to mn because limm→∞ mn = 0. Hence O(mn + n) = O(mn).
(g) O(m log(n) + n log(m) + n)
n
n is negligible compared to n log(m) because limm→∞ n log(m) = 0. m log(n) and n log(m) are not
further comparable. Therefore O(m log(n) + n log(m) + n) = O(m log(n) + n log(m))
3. (9 pts)
(i) In class, you learned that Θ(loga n) = Θ(logb n) for a, b > 1. Does Θ(an ) = Θ(bn ) hold? Justify
your answer.
(ii) Show that for k > 0, α > 1: nk = O(αn ) (i.e., polynomial functions grow slower than exponential
functions).
(iii) Find a function f (n) such that f (n) = O(1), f (n) > 0 for all n, and f (n) 6= Θ(1). Justify the
answer.
Solution:
(i) This is true only if a = b. Otherwise, let’s assume a > b. We show that an 6= O(bn ) through a
proof by contradiction:
Assume an = O(bn ). Then, by defintion of O, there is a constant c and n0 such that for all
n ≥ n0 :
an ≤ cbn .
This implies (by applying the base-a log on both sides),
n ≤ n loga b + loga c.
Since a > b and thus loga b < 1, we can solve for n as
n ≤ loga c/(1 − loga b), for all n ≥ n0 ,
which is a contradiction (it does not hold since n grows to infinity); thus, the original assumption
is wrong and we have an 6= O(bn ) as desired.
(ii) We consider limx→∞ xk /αx and apply L’Hospital’s rule
http://mathworld.wolfram.com/LHospitalsRule.html
k times. Remember that the derivative of αx is loge (α)αx :
xk kxk−1 k!
lim x
= lim = ... = lim = 0.
x→∞ α x→∞ loge (α)αx x→∞ (loge (α))k αx

This means, if we choose any c > 0, then there is an n0 such that

nk /αk < c,
which implies nk < cαk as desired.

18-645 SP 2007 / Assignment 1 Pg 2 of 6 Electrical & Computer Engineering

Instructor: Markus Püschel Carnegie Mellon University
 (iii) f (n) = 1/n. Obviously 0 < 1/n < 1 for n ≥ 1 and thus 1/n = O(1). Assume now that also
1/n = Ω(1) and show that this leads to a contradiction. 1/n = Ω(1) means that there is a
constant c > 0 and n0 such that

c · 1 ≤ 1/n, for n ≥ n0 ,

which is obviously wrong (whenever n > 1/c).

Not easy to find an algorithm with 1/n as cost function :-).

4. (18 pts) Give asymptotic bounds (O, Ω, or Θ) for T (n) in each of the following recurrences. Make
your bounds as tight as possible. Justify your answers.

(a) T (n) = 2T (n/2) + n2 .

(b) T (n) = T (9n/10) + n.
(c) T (n) = 16T (n/4) + n2 .
(d) T (n) = 7T (n/3) + n2 .
√
(e) T (n) = 2T (n/4) + n.
(f) T (n) = 4T (n/2) + n2 log n.

Solution:

(a) T (n) = 2T (n/2) + n2 . a = 2, b = 2, f (n) = n2 . This is Case 3. T (n) = Θ(n2 ).

(b) T (n) = T (9n/10) + n. a = 1, b = 10/9, f (n) = n. This is Case 3. T (n) = Θ(n).
(c) T (n) = 16T (n/4) + n2 . a = 16, b = 4, f (n) = n2 . This is Case 2. T (n) = Θ(n2 log n).
(d) T (n) = 7T (n/3) + n2 . a = 7, b = 3, f (n) = n2 . This is Case 3. T (n) = Θ(n2 ).
√ √ √
(e) T (n) = 2T (n/4) + n. a = 2, b = 4, f (n) = n. This is Case 2. T (n) = Θ( n log n).
(f) T (n) = 4T (n/2) + n2 log n. This problem does not fall into any of the cases.

5. (10 pts) Solve 4(a) exactly for T (1) = 1 asumming n = 2k .

Solution: Let n = 2k and uk = T (2k ). The problem is now uk = 2uk−1 + 22k and u0 = 1.
To get some intuition, we unwind the first terms of the sequence.

uk = 2uk−1 + 22k
= 2(2uk−2 + 22k−2 ) + 22k
= 22 uk−2 + 22k + 22k−1
= 23 uk−3 + 22k + 22k−1 + 22k−2

Clearly uk = 2k + 22k + · · · + 2k+1 (proof is done by induction.).

uk = 2k + 22k + · · · + 2k+1
= 2k + 2k+1 (1 + · · · + 2( k − 1))
1 − 2k
= 2k + 2k+1
1−2
= 2k + 2k+1 (2k − 1)
= 22k+1 − 2k

By returning to the standard notation, we finaly find: T (n) = 2n2 − n.

18-645 SP 2007 / Assignment 1 Pg 3 of 6 Electrical & Computer Engineering

Instructor: Markus Püschel Carnegie Mellon University
 6. (15 pts) Consider two polynomials: h(x) = hn−1 xn−1 + ... + h0 and p(x) = pn−1 xn−1 + ... + p0 of the
same degree n − 1.
Compute the exact (arithmetic) cost

C(n) = (number of additions, number of multiplications)

for multiplying the polynomials

(a) by definition;
(b) using the Karatsuba algorithm, recursively applied, assuming n = 2k .

Solution:

(a) By definition, it is necessary to multiply each factor by all others. Therefore, Cm (n) = n2 .
One needs to figure out that there are as many additions as there are multiplications except that
one of the multiplication goes for free in each monomials of the final polynomial(of degree 2n − 2).
Therefore Ca (n) = n2 − 2n + 1.
(b) As seen in class, C(2) = (4, 3) with Karatsuba’s algorithms.
We now denote pm (x) is a polynomial of degree m in the variable x. To compute the recursive
Karatsuba, there are two methods that are equivalent in the number of computations. Namely
p2m+1 (x) = pm (x) + xm+1 pm x or p2m+1 (x) = pm (x2 ) + xpm x2 . We take the first one but we
would find the same results with the other one.

p2m+1 (x)p2m+1 (x) = (pm (x) + xm+1 pm x)(pm (x) + xm+1 pm x)

= pm (x)pm (x) + x2m+2 pm (x)pm (x)+
xm+1 ((pm (x) + pm (x))(pm (x) + pm (x)) − pm (x)pm (x) − pm (x)pm (x))

From here, we can see that we need 3 multiplications of degree m polynomials, two addtions of
degree m polynomials, two additions of degree 2m polynomials and 2m scalar additions (first
term and third term don’t overlap and their sum has all terms except a term in x2m+1 . Third
term has 2m + 1 monomials, between xm+1 and x3m+1 ).
Therefore, a multiplication of two degree 2m + 1 polynomials require 3 multiplications of degree
m polynomials and 2(m + 1) + 2(2m + 1) + 2m = 8m + 4 scalar additions.
Therefore, a multiplication of two degree 2k − 1 polynomials require 3 multiplications of degree
2k−1 − 1 polynomials and 2k+2 − 4 scalar additions.
By induction, it is clear that Cm (2k ) = 3k . Therefore Cm (n) = nlog2 (3)

18-645 SP 2007 / Assignment 1 Pg 4 of 6 Electrical & Computer Engineering

Instructor: Markus Püschel Carnegie Mellon University
 The number of additions is a bit more tricky:
Ca (2k ) = 2k+2 − 4 + 3Ca (2k−1 )
= 2k+2 + 3 ∗ 2(k−1)+2 − (4 + 3 ∗ 4) + 32 Ca (2k−2 )
X
h≤k−2
= (3h 22+k−h − 3h 4) + 3k−1 Ca (21 )
h=0

X
h≤k−2 X
h≤k−2
= (3h 22+k−h ) − (3h 4) + 3k−1 ∗ 4
h=0 h=0

X
h≤k−2 X
h≤k−2
= 22+k (3/2)h − 4 (3h ) + 3k−1 ∗ 4
h=0 h=0
k−1
1 − (3/2) 1 − 3k−1
= 22+k −4 + 3k−1 ∗ 4
1 − 3/2 1−3
= 23+k ((3/2)k−1 − 1) − 2(3k−1 − 1) + 3k−1 ∗ 4
= 16 ∗ 3k−1 − 2 ∗ 3k−1 + 2 + 4 ∗ 3k−1 − 23+k
= 18 ∗ 3k−1 − 23+k + 2
= 6 ∗ 3k − 8 ∗ 2k + 2
Therefore Ca (n) = 6nlog2 (3) − 8n + 2.
7. (15 pts) Solve the recurrence f0 = 1, f1 = 1, fn = fn−1 + 2fn−2 , using the method of generating
functions.
Solution: Our generating function is:
X
F (x) = fn xn
n≥0

Step 1: X X X
fn xn = fn−1 xn + 2 fn−2 xn
Step 2:
X X X
fn xn = fn−1 xn + 2 fn−2 xn
n≥2 n≥2 n≥2
X
F (x) = f0 + f1 + fn xn
n≥2
X X
F (x) = 1+x+ fn−1 xn + 2 fn−2 xn
n≥2 n≥2
X X
n−1
F (x) = 1+x+x fn−1 x + 2x2 fn−2 xn−2
n≥2 n≥2
X X
k 2
F (x) = 1+x+x fk x + 2x fk xk
k≥1 k≥0
X X
k 2
F (x) = 1+x+x fk x + 2x fk xk − xf0 x0
k≥0 k≥0
2
F (x) = 1 + xF (x) + 2x F (x)
Step 3:
F (x) = 1 + xF (x) + 2x2 F (x)
1 1
F (x) = 2
=
(1 − x − 2x ) (1 + x)(1 − 2x)

18-645 SP 2007 / Assignment 1 Pg 5 of 6 Electrical & Computer Engineering

Instructor: Markus Püschel Carnegie Mellon University
 Step 4:
A B
F (x) = +
(1 + x) (1 − 2x)
A(1 − 2x) + B(1 + x) = 1
A+B = 1
−2A + B = 0
Solution A = 1/3, B = 2/3

Step 5:
1X 2X n n
F (x) = (−1)n xn + 2 x
3 3
n≥0 n≥0

Step 6:
1 2 1
fn = (−1)n + 2n = (2n+1 + (−1)n )
3 3 3
8. (10 pts) Consider a polynomial of the third degree: a(x) = a0 + xa1 + x2 a2 + x3 a3 .

(a) Compute the exact (arithmetic) cost

D = (number of additions, number of multiplications) :

for evaluating a(x) at a point x = x0

i. by definition (in a straightforward way, without using any tricks)
ii. if the polynomial is expressed as: a(x) = a0 + x(a1 + x(a2 + xa3 ))
(b) Now, determine the cost D of evaluating an n-th degree polynomials p(x) = a0 + · · · + an xn using
the same trick as in ii. above. The method is called Horner’s scheme.
Solution:

(a) i. By definition, there is 3 additions and 6 multiplications (without reusing already computed
powers of x) or 3 additions and 5 multiplications (with reuse of powers of x)
ii. With this expression it goes down to 3 additions and 3 multiplications.
(b) By induction, Horner’s scheme evaluates a polynom of degree n in n additions and n multiplica-
tions.

18-645 SP 2007 / Assignment 1 Pg 6 of 6 Electrical & Computer Engineering

Instructor: Markus Püschel Carnegie Mellon University