2.

A LGORITHM A NALYSIS

‣ Computational Tractability
‣ asymptotic order of growth
‣ implementing Gale–Shapley survey of common running times
‣
2. A LGORITHM A
NALYSIS
computational tractability
asymptotic order of growth
implementing Gale–Shapley survey of common running times
A strikingly modern thought

“ As soon as an Analytic Engine exists, it will necessarily guide the future

course of the science. Whenever any result is sought by its aid, the question
will arise—By what course of calculation can these results be arrived at by
the machine in the shortest time? ” — Charles Babbage (1864)

how many times do you

have to turn the crank?

Analytic Engine
3
Models of computation: Turing machines

Deterministic Turing machine. Simple and idealistic model.

Running time. Number of steps.

Memory. Number of tape cells utilized.

Caveat. No random access of memory.

・ Single-tape TM requires ≥ n steps
2
to detect n-bit palindromes.
・Easy to detect palindromes in ≤ cn steps on a real computer.
4
Models of computation: word RAM

assume w ≥ log2 n
Word RAM.
・Each memory location and input/output cell stores a w-bit integer. Primitive
・operations: arithmetic/logic operations, read/write memory, array indexing,
following a pointer, conditional branch, …

constant-time C-style operations

(w = 64)
input … memory

program a[i] i

.
.
.
output …

Running time. Number of primitive operations.

Memory. Number of memory cells utilized.

Caveat. At times, need more refined model (e.g., multiplying n-bit integers). 5
Brute force

Brute force. For many nontrivial problems, there is a natural brute-force search
algorithm that checks every possible solution.
・ Typically takes 2 steps (or worse) for inputs of size n.
n

・Unacceptable in practice.

Ex. Stable matching problem: test all n! perfect matchings for stability. 6
Polynomial running time

Desirable scaling property. When the input size doubles, the algorithm should slow
down by at most some multiplicative constant factor C.

Def. An algorithm is poly-time if the above scaling property holds.

There exist constants c > 0 and d > 0 such that,

for every input of size n, the running time of the algorithm is bounded
d corresponds
above by c n primitive computational steps.
to C = 2 d

von Neumann Nash Gödel Cobham Edmonds Rabin

(1953) (1955) (1956) (1964) (1965) (1966)

7
Polynomial running time

We say that an algorithm is efficient if it has a polynomial running time.

Theory. Definition is (relatively) insensitive to model of computation.

Practice. It really works!

・The poly-time algorithms that people develop have both

small constants and small exponents.

・Breaking through the exponential barrier of brute force typically

exposes some crucial structure of the problem. n120

Map graphs in polynomial time

Map graphs in polynomial time
Exceptions. Some poly-time algorithms in the wild Department of Computer
Mikkel
Thorup
Science, University of Copenhag
Universitetsparken
Mikkel Thorup 1, DK-2100 Copenhagen East, Denma
have galactic constants and/or huge exponents. Department of Computer Science, University of mthorup@diku.dk
Copenhagen Universitetsparken 1, DK-2100 Copenhagen
East, Denmark
mthorup@diku.dk

Abstract
Chen, Grigni, andAbstractPapadimitriou (WADS’97 and
Chen, Grigni, and Papadimitriou (WADS’97 and STOC’98)
STOC’98) have introduced a modified notion of planarity,
where two faces are considered haveadjacent
introduced a modified
if they share atnotion of planarity, where two
least one point.
The corresponding faces are
abstract graphs are considered
called mapadjacent if they share at least one point.
Thewhether
corresponding abstract Figure 1. Large
graphs.
Chen et.al. raised the question of map graphs can graphs are called map graphs.

Q. Which would you prefer: 20 n or120 n 1 + 0.02 ln n ?

be Chen et.al. raised the question of whether map graphs can be
recognized in polynomial time. They showed that the decision Figure 1. Large cliques in
maps is in NP and presentedrecognized
problem a polynomialin polynomial
time time. They showed that the decision
addition, the hamantach may
algorithm problem
for the special case where we allow is in4NP
at most andtopresented a polynomial time algorithm
faces
for the special case where we allow at most 4 faces to intersect touching all three corners. In
intersect
in any point — if only 3 are allowed to intersect in a point, we addition, the hamantach may have at most two triangle faces
all the different types of large
get the usual planar graphs. in any point — if only 3 are allowed to intersect
touching in a point,
all three corners. we In [5] there is a classification
Chen et.al. conjectured that get
mapthe usualcan
graphs planar graphs.
be recognized ofall the different types of large showed that recognizing map
cliques in maps.
Chen et.al. conjectured that map graphs can be recognized recognition can beChen
doneet.al.
in sin
[5] in polynomial time, and in this paper, their conjecture is settled showed that recognizing map graphs is in NP, hence that the
in polynomial time, and in this paper, their conjecture is settled they conjectured that, in fact,
affirmatively. recognition can be done in singly exponential time.
affirmatively. However, polynomial time. They suppo
they conjectured that, in fact, map graphs can be recognized
that if we allow at most 4
1. Introduction in
polynomial time. They supported their conjecture by
faces
showing resulting map graphs can be
that if we allow at most 4 faces to meet in any single point,
1. Introduction
Recently Chen, Grigni, and Papadm ithe
rec
orituresulting map[4, agprhs5] suggestedcan begenera
8 the
this paper, we settle
recognized in polynomial
a graph, we can decide in
thetime.
studyInof a modified notion of palnya.tirthis paper,Thewe selt basic hteframe-general
polyn conjecture, showing
Recently Chen, Grigni, and Papadimitriou [4, 5] suggested The algorithm can easily be
Worst-case analysis
Worst case. Running time guarantee for any input of size n.
・Generally captures efficiency in practice.
・Draconian view, but hard to find effective alternative.

Exceptions. Some exponential-time algorithms are used widely in practice because the
worst-case instances don’t arise.

simplex algorithm Linux grep k-means algorithm

9
Other types of analyses

Probabilistic. Expected running time of a randomized algorithm.

Ex. The expected number of compares to quicksort n elements is ~ 2n ln n.

Amortized. Worst-case running time for any sequence of n operations. Ex. Starting from
an empty stack, any sequence of n push and pop operations takes O(n) primitive
computational steps using a resizing array.

Also. Average-case analysis, smoothed analysis, competitive analysis, ...

10
 2. A LGORITHM A
NALYSIS
‣ computational tractability
‣ asymptotic order of growth
‣ implementing Gale–Shapley
‣ survey of common running
times

SECTION 2.2
Big O notation
Upper bounds. f (n) is O(g(n)) if there exist constants c > 0 and n0 ≥ 0 such that
0 ≤ f (n) ≤ c · g (n) for all n ≥ n0.
c · g(n)

Ex. f (n) = 32n2 + 17n + 1. f (n)

・ f (n) is O(n ).
2
choose c = 50, n = 1
0

・f (n) is neither O(n) nor O(n log n).

n0 n

2
Typical usage. Insertion sort makes O(n ) compares to sort n elements.

12
Analysis of algorithms: quiz 1

Let f(n) = 3n +2 17 n log2 n + 1000. Which of the following are true?

A. f (n) is O(n 2). choose c = 1020, n0 = 1

B. f (n) is O(n 3). choose c = 1020, n0 = 1

C. Both A and B.

D. Neither A nor B.

13
Big O notational abuses
One-way “equality.” O(g(n)) is a set of functions, but computer scientists often write f
(n) = O(g(n)) instead of f (n) ∈ O(g(n)).

Ex. Consider g1(n) = 5n 3and g2(n) = 3n . 2

・ We have g1(n) = O(n ) and g2(n) = O(n ).

3 3

・But, do not conclude g (n) = g2(n).

1

Domain and codomain. f and g are real-valued functions.

・ The domain is typically the natural numbers: ℕ → ℝ.
・Sometimes we extend to the reals: ℝ ≥ 0 → ℝ. input size, recurrence relations Or restrict

・to a subset. plotting, limits, calculus

Bottom line. OK to abuse notation in this way; not OK to misuse it.

14
Big O notation: properties

Reflexivity. f is O( f ).

Constants. If f is O(g) and c > 0, then c f is O(g).

Products. If f 1 is O(g 1) and f 2 is O(g 2), then f 1 f 2 is O(g 1 g2).

Pf.

・∃ c > 0 and n1 ≥ 0 such that 0 ≤ f 1(n) ≤ c1 · g1(n) for all n ≥ n1.

・∃ c > 0 and n2 ≥ 0 such that 0 ≤ f 2(n) ≤ c2 · g2(n) for all n ≥ n2.
・Then, 0 ≤ f 1(n) · f 2(n) ≤ c1 · c2 · g1(n) · g2(n) for all n ≥ max { n1, n2 }.
1
▪
2 c n0

Sums. If f 1 is O(g 1) and f2 is O(g 2), then f 1 + f 2 is O(max {g 1, g2}).

ignore lower-order terms
Transitivity. If f is O(g) and g is O(h), then f is O(h).

Ex. f (n) = 5n3 + 3n 2 + n + 1234 is O(n ).3

15
Big Omega notation

Lower bounds. f (n) is Ω(g(n)) if there exist constants c > 0 and n0 ≥ 0 such that
f (n) ≥ c · g(n) ≥ 0 for all n ≥ n0.
f (n)

Ex. f (n) = 32n2 + 17n + 1. c · g(n)

・ f (n) is both Ω(n 2) and Ω(n). choose c = 32, n0 = 1

・ f (n) is not Ω(n3).

n0 n

Typical usage. Any compare-based sorting algorithm requires Ω(n log n) compares in
the worst case.

Vacuous statement. Any compare-based sorting algorithm requires at least O(n

log n) compares in the worst case.

16
Analysis of algorithms: quiz 2

Which is an equivalent definition of big Omega notation?

A. f (n) is Ω(g(n)) if g (n) is O( f (n)).

B. f (n) is Ω(g(n)) if there exists a constant c > 0 such that f (n) ≥ c · g(n) ≥ 0 for
infinitely many n.

C. Both A and B.

D. Neither A nor B.

17
Big Theta notation
Tight bounds. f (n) is Θ (g(n)) if there exist constants c1 such > 0, c2 > 0, and n0 ≥ 0
that 0 ≤ c1 · g(n) ≤ f (n) ≤ · g(n) for all n ≥ n0. c2 · g(n)
c2
f (n)
Ex. f (n) = 32n2 + 17n + 1. c1 · g(n)
・ f (n) is Θ (n2 ). choose c = 32, c = 50, n = 1
1 2 0

・ f (n) is neither Θ (n) nor Θ (n ).3

n0 n

Typical usage. Mergesort makes Θ (n log n) compares to sort n elements.

between ½ n log 2 n
and n log2 n

19
Analysis of algorithms: quiz 3

Which is an equivalent definition of big Theta notation?

A. f (n) is Θ (g(n)) if f (n) is both O( g (n)) and Ω( g (n)).

f (n)
B. f (n) is Θ (g(n)) if lim = c >for0 some constant 0 < c < ∞.
n g
(n)

C. Both A and B.

D. Neither A nor B. counterexample

2n n
f (n) =
3n n

g(n) = n

f (n) is O(n) but limit does not

exist
20
Asymptotic bounds and limits

f (n)
Proposition. If lim = c >for0 some constant 0 < c < ∞ then f (n) is Θ (g(n)).
n g
(n)
Pf.

・By definition of the limit, for any ε > 0, there exists n0 such that
f (n)
c c+
g
(n)
for all n ≥ n0.
・ Choose ε = ½ c > 0.

・Multiplying by g(n) yields 1/2 c · g(n) ≤ f (n) ≤ 3/2 c · g(n) for all n ≥ n0.
・Thus, f (n) is Θ (g(n)) by definition, with c1 = 1/2 c and c2 = 3/2 c. ▪

f (n)
Proposition. If lim = 0 , then f (n) is O(g(n)) but not Ω(g(n)).
n g
(n)
f (n)
Proposition. If lim = , then f (n) is Ω(g(n)) but not O(g(n)).
n g
(n) 21
Asymptotic bounds for some common functions

d d
Polynomials. Let f (n) = a0 + a1 n + … + ad n with ad > 0. Then, f (n) is Θ (n ).
Pf. d
a0 + a 1 n + . . . + a d
lim n = ad > 0
n nd

Logarithms. loga n is Θ (logb n) for every a > 1 and every b > 1.

Pf. log a n no need to specify base
= log b n (assuming it is a constant)
1
log b a

d
Logarithms and polynomials. loga n is O(n ) for every a > 1 and every d > 0.
Pf.
lim log da = 0
n nn

d ) for every
Exponentials and polynomials. n is O(r n r > 1 and every d > 0.
Pf. nd
lim n = 0
n r

Θ (n log n).
Factorials. n! is 2 Pf.
Stirling’s formula: n n
n! 2 n
e 22
Big O notation with multiple variables

Upper bounds. f (m, n) is O(g(m, n)) if there exist constants c > 0, m0 ≥ 0, and n0 ≥
0 such that 0 ≤ f (m, n) ≤ c · g (m, n) for all n ≥ n0 and m ≥ m0.

Ex. f (m, n) = 32mn2 + 17mn + 32n3.

・ f (m, n) is both O(mn + n ) and O(mn ).
2 3 3

・ f (m, n) is O(n3 ) if a precondition to the problem implies m ≤ ・ f n.

3 2
(m, n) is neither O(n ) nor O(mn ).

Typical usage. In the worst case, breadth-first search takes O(m + n) time to find a
shortest path from s to t in a digraph with n nodes and m edges.

23
 2. A LGORITHM A
NALYSIS
‣ computational tractability
‣ asymptotic order of growth
‣ implementing Gale–Shapley
‣ survey of common running
times

SECTION 2.3
Efficient implementation

Goal. Implement Gale–Shapley to run in O(n ) time.2

GALE–SHAPLEY (preference lists for n hospitals and n students)

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
____________________________________________________________________________________________________________________

INITIALIZE M to empty matching.

WHILE (some hospital h is unmatched)
s ← first student on h’s list to whom h has not yet proposed.
IF (s is unmatched)
Add h–s to matching M.
ELSE IF (s prefers h to current partner hʹ)
Replace hʹ–s with h–s in matching M.
ELSE
s rejects h.

RETURN stable matching M.

____________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________
____________________________________________________________________________________________________________________

25
Efficient implementation

Goal. Implement Gale–Shapley to run in O(n ) time.2

Representing hospitals and students. Index hospitals and students 1, …, n.

Representing the matching.

・Maintain two arrays student[h] and hospital[s].
- if h matched to s, then student[h] = s and hospital[s] = h
- use value 0 to designate that hospital or student is unmatched
・Can add/remove a pair from matching in O(1) time.

・Maintain set of unmatched hospitals in a queue (or stack).

・Can find an unmatched hospital in O(1) time.

26
Data representation: making a proposal

Hospital makes a proposal.

・Key operation: find hospital’s next favorite student.

・For each hospital: maintain a list of students, ordered by preference. For each
・hospital: maintain a pointer to student for next proposal.

next proposal to

hospital h 3 4 1 5 2 null

least favorite
favorite

Bottom line. Making a proposal takes O(1) time.

27
Data representation: accepting/rejecting a proposal
Student accepts/rejects a proposal.
・ Does student s prefer hospital h to hospital hʹ ?
・For each student, create inverse of preference list of hospitals.

pref[]
1 st 2 nd 3 rd 4 th 5 th 6 th 7 th 8 th

8 3 7 1 4 5 6 2
student prefers hospital 4 to 6 since
rank[4] < rank[6]

1 2 3 4 5 6 7 8
rank[]
4 th 8 th 2 nd 5 th 6 th 7 th 3 rd 1 st

for i = 1 to n rank[pref[i]] = i

2
Bottom line. After Θ (n ) preprocessing time (to create the n ranking arrays), it takes O(1)
time to accept/reject a proposal. 28
Stable matching: summary

2
Theorem. Can implement Gale–Shapley to run in O(n ) time.
Pf.
・ Θ (n2 ) preprocessing time to create the n ranking arrays. ・ There are O(n )
2 ▪
proposals; processing each proposal takes O(1) time.

Theorem. In the worst case, any algorithm to find a stable matching must query the
2
hospital’s preference list Ω(n ) times.

29
 2. A LGORITHM A
NALYSIS
‣ computational tractability
‣ asymptotic order of growth
‣ implementing Gale–Shapley
‣ survey of common running
times

SECTION 2.4
Constant time

Constant time. Running time is O(1).

bounded by a constant,
Examples. which does not depend on input size n

・ Conditional branch.

・Arithmetic/logic operation.
・Declare/initialize a variable.
・Follow a link in a linked list.
・Access element i in an array.
・Compare/exchange two elements in an array.
・…

31
Linear time

Linear time. Running time is O(n).

Merge two sorted lists. Combine two sorted linked lists A = a1, a2, …, an and B = b1, b2,
…, bn into a sorted whole.

O(n) algorithm. Merge in mergesort.

i ← 1; j ← 1.
WHILE (both lists are nonempty)
IF (ai ≤ bj) append ai to output list and increment i.
ELSE append bj to output list and increment j.
Append remaining elements from nonempty list to output list.
32
TARGET SUM

TARGET-SUM. Given a sorted array of n distinct integers and an integer T, find two
that sum to exactly T ?

input
−20 10 20 30 35 40 60 70 T = 60
(sorted)

i j

33
Logarithmic time

Logarithmic time. Running time is O(log n).

Search in a sorted array. Given a sorted array A of n distinct integers and an integer x, find
index of x in array.

remaining elements
O(log n) algorithm. Binary search.
・Invariant: If x is in the array, then x is in A[lo .. hi]. After k
・iterations of WHILE loop, (hi − lo + 1) ≤ n / 2 k k ≤ 1 + log2 n.

lo ← 1; hi ← n.
WHILE (lo ≤ hi)
mid ← ⎣(lo + hi) / 2⎦.
IF (x < A[mid]) hi ← mid − 1.

ELSE IF (x > A[mid]) lo ← mid + 1.

ELSE RETURN mid.
RETURN −1.
35
 Logarithmic time

O(log n)

https://www.facebook.com/pg/npcompleteteens
36
SEARCH IN A SORTED ROTATED ARRAY

SEARCH-IN-SORTED-ROTATED-ARRAY. Given a rotated sorted array of n distinct integers

and an element x, determine if x is in the array.

sorted circular array

20
95 30

35
90

50
85

60
80

75 65
67

sorted rotated array

80 85 90 95 20 30 35 50 60 65 67 75

1 2 3 4 5 6 7 8 9 10 11 12

37
 Linearithmic time
Linearithmic time. Running time is O(n log n).

Sorting. Given an array of n elements, rearrange them in ascending order.

O(n log n) algorithm. Mergesort.

a[]
lo hi 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15M E R G E S O
R T E X A M P L E
a, aux, 0, 0, 1) E M R G E S O R T E X A M P L E
a, aux, 2, 2, 3) E M G R E S O R T E X A M P L E
aux, 0, 1, 3) E G M R E S O R T E X A M P L E
a, aux, 4, 4, 5) E G M R E S O R T E X A M P L E
a, aux, 6, 6, 7) E G M R E S O R T E X A M P L E
aux, 4, 5, 7) E G M R E O R S T E X A M P L E
ux, 0, 3, 7) E E G M O R R S T E X A M P L E
a, aux, 8, 8, 9) E E G M O R R S E T X A M P L E
a, aux, 10, 10, 11) E E G M O R R S E T A X M P L E
aux, 8, 9, 11) E E G M O R R S A E T X M P L E
a, aux, 12, 12, 13) E E G M O R R S A E T X M P L E
a, aux, 14, 14, 15) E E G M O R R S A E T X M P E L
aux, 12, 13, 15) E E G M O R R S A E T X E L M P
ux, 8, 11, 15) E E G M O R R S A E E L M P T X
, 0, 7, 15) A E E E E G L M M O P R R S T X
39
Trace of merge results for top-down mergesort
LARGEST EMPTY INTERVAL

LARGEST-EMPTY-INTERVAL. Given n timestamps x1, …, xn on which copies of a file

arrive at a server, what is largest interval when no copies of file arrive?

40
Quadratic time

Quadratic time. Running time is O(n ). 2

Closest pair of points. Given a list of n points in the plane (x1, y1), …, (xn, yn), find the
pair that is closest to each other.

O(n2) algorithm. Enumerate all pairs of points (with i < j).

min ← ∞.
FOR i = 1 TO n
FOR j = i + 1 TO n
d ← (xi − xj)2 + (yi − 2

yj) .
IF (d < min)

min ← d.

Remark. Ω(n )2seems inevitable, but this is just an illusion. [see §5.4]
42
Cubic time

Cubic time. Running time is O(n ). 3

3-SUM. Given an array of n distinct integers, find three that sum to 0.

O(n3) algorithm. Enumerate all triples (with i < j < k).

FOR i = 1 TO n

FOR j = i + 1 TO n
FOR k = j + 1 TO n
IF (ai + aj + ak = 0)
RETURN (ai, aj, ak).

Remark. Ω(n )3seems inevitable, but O(n ) is not

2
hard. [see next slide]
43
3-SUM

3-SUM. Given an array of n distinct integers, find three that sum to 0.

O(n3) algorithm. Try all triples.

O(n2) algorithm.

44
Polynomial time

k
Polynomial time. Running time is O(n ) for some constant k > 0.

Independent set of size k. Given a graph, find k nodes such that no two are joined by
an edge.
k is a constant

O(nk) algorithm. Enumerate all subsets of k nodes.

FOREACH subset S of k nodes:

Check whether S is an independent
set. IF (S is an independent set)
RETURN S.
independent set of size 3

・ Check whether S is an independent set of size k takes O(k ) time. 2

・Number of k-element subsets = n kn ( n= 1)( n k2)(k · ·1)(k· ( n k + 1) ・
2) · · ·
nk
k
2 k k
O(k n / k!) = O(n ). 1 !

poly-time for k = 17, but not practical 46

Exponential time

k
Exponential time. Running time is O(2 ) for nsome constant k > 0.

Independent set. Given a graph, find independent set of max cardinality.

O(n2 2n) algorithm. Enumerate all subsets of n elements.

S* ← ∅.
FOREACH subset S of n nodes:
Check whether S is an independent set. IF
(S is an independent set and ⎢S⎟ > ⎢S*⎟)
S* ← S. independent set of max cardinality

RETURN S*.

47
Exponential time

k
Exponential time. Running time is O(2 ) for nsome constant k > 0.

Euclidean TSP. Given n points in the plane, find a tour of minimum length. O(n � n!)

algorithm. Enumerate all permutations of length n.

π* ←
∅.
FOREACH permutation π of n points:
Compute length of tour corresponding to π.
IF (length(π) < length(π*))
π* ← π.
for simplicity, we’ll assume Euclidean
RETURN π*. distances are rounded to nearest integer (to avoid
issues with infinite precision)

48
Analysis of algorithms: quiz 4

Which is an equivalent definition of exponential time?

A. O( 2n doesn’t include 3 n

B. O(2cn ) for some constant c > 0. n doesn’t include n! = 2

includes 3 but Θ (n log n)

C. Both A and B.

D. Neither A nor B.

49