Introduction to

Algorithms and
Data Structures

Markus Bläser
Saarland University
Draft—February 3, 2011 and forever
2

Markus Bläser 2009–11

1 Introduction

1.1 Binary search

Consider the following—nowadays pretty outdated—example: You are look-
ing for a particular CD of a particular interpret. Because you are well-bred,
you store your CD alphabetically sorted in your shelf. So you can scan all
your CDs from the left to the right until you find the one that you are look-
ing for. If your collection were not alphabetically sorted, then this would
be the best that you can do. But since your CDs are sorted, you can look
at the CD (roughly) in the middle of the shelf. If the CD you are looking
for is alphabetically smaller, then you know that it is in the left half of your
collection. Otherwise, it is in the right half of the collection. In this way,
you halved the search space by looking at only one CD. This is a huge im-
provement over the scanning from left to right method. Here looking at one
CD decreases your search space only by one.
From this example, we will learn a lot about the spirit of this course. We
will
• find algorithmic solutions that are faster than the “obvious” ones,
• prove that the algorithms terminate and produce the correct result,
and
• rigorously analyze the running time of the algorithms.
Let us abstract a little bit from the CD example. Our input is an array
a[1..n] and the entries in the array are sorted, i.e., a[1] < a[2] < · · · < a[n].
We do not care about the exact type of the array, we only need that we can
compare the values. Algorithm 1 implements the method sketched above.
The variables ` and r store the current left and right border of the search
space and m is the index of the current element in the middle. If a[m] = x,
then we found the element. Otherwise, m becomes the new left or right
border.
The text between /* . . . */ in line 2 is a loop invariant. The loop invariant
holds before each iteration of the loop. This is of course not automatically
true, we have to design the program in such a way and, of course, we have
to prove this.
In the case above, we have to declare what a[0] and a[n+1] is. We will set
a[0] = −∞ and a[n + 1] = +∞. These are fictious elements, we need them
for the invariant to be true and for our proofs to work, but the algorithm
can never access them or compute with the values −∞ or ∞.

3
4 1. Introduction

Algorithm 1 Binary-search
( array a[1..n], a[1] < a[2] < · · · < a[n], element x
Input: Sorted
m if there is an 1 ≤ m ≤ n with a[m] = x
Output:
−1 otherwise
1: ` := 0; r := n + 1;
2: while ` + 1 < r do /* 0 ≤ ` < r ≤ n + 1 and a[`] < x < a[r] */
3: m := b `+r
2 c;
4: if a[m] = x then
5: return m;
6: if a[m] < x then
7: ` := m
8: else
9: r := m;
10: return −1;

If we start with this, then before the first iteration of the while loop, our
invariant is certainly true. Now we show that if the invariant is true before
the current iteration, then it will also be true after the iteration (that is,
before the next iteration). Note that ` ≤ m ≤ r and in lines 7 and 8, either
` or r is set to m. Thus 0 ≤ ` ≤ r ≤ n + 1 holds after the iteration. If ` = r,
then this would mean that ` + 1 = r before the iteration. But in this case,
the loop would not have been executed. Thus the first part of the invariant
is true. For the second part, note that it is certainly true in the beginning,
since by our convention, a[0] = −∞ and a[n + 1] = +∞.1 Now assume that
the invariant was true before the current iteration. If a[m] = x, then we stop
and there is no next iteration. If a[m] 6= x, then in lines 6 to 9, ` and r are
chosen in such a way that the invariant holds at the end of the iteration.
If the while loop is left, then either a[m] = x or ` + 1 = r. In the first
case, the algorithm returns m. In the second case, the algorithm returns −1.
But we know that a[`] < x < a[r] from our invariant. But since there are no
further array elements between a[`] and a[r], x is not in the array. Thus the
algorithm is correct.
The algorithm also terminates: In each iteration either ` is increased or
r is reduced. Hence, the algorithm will terminate.

Theorem 1.1 Binary-search always terminates and returns the correct out-
put.

Often, it is very convenient to state algorithms in a recursive fashion.

While we can transform an iterative algorithm into a recursive one and vice
1
If you still feel uneasy with this convention, you can replace it by a case distinction.
The second part of the invariant then reads ((` = 0)∨(a[`] < x))∧((r = n+1)∨(x < a[r])).
I hope that this convinces you of the usefulness of the fictious elements.

Markus Bläser 2009–11

1.2. Machine model 5

versa, the recursive one is often easier to analyse, but might be slower in
pratice.

Algorithm 2 Binary-search (recursive)

( array a[1..n], a[1] < a[2] < · · · < a[n], element x
Input: Sorted
m if there is an 1 ≤ m ≤ n with a[m] = x
Output:
−1 otherwise
1: if n = 0 then
2: return −1;
3: m := b n+12 c;
4: if a[m] = x then
5: return m;
6: if a[m] < x then
7: s := Binary-search(a[m + 1..n], x);
8: if s = −1 then
9: return −1
10: else
11: return m + s;
12: else
13: return Binary-search(a[1..m − 1], x);

If n = 0, then our array is empty and there is nothing to do. Otherwise,

we compute m as before. (0 and n+1 are the borders.) If a[m] is our element,
we return m. If not, we call Binary search recursively. In lines 7–11, we deal
with the case that x is (potentially) in the second half of the array. If the
recursive call returns −1, that is, x was not found, then we return −1. If
something else is returned, that is, x was found, then we return m + s, since
we searched in the second half. In line 13, we deal with the case that x is
(potentially) in the first half. Here we can just pass the value s, since no
index shift is required.
Note that we rely on an intelligent compiler (you!) when writing our
pseudocode. For instance, Binary-search(a[m + 1...n], x) calls the routine
with the subarray of length n − m. Inside the recursive call, the entries are
again indexed from 1, . . . , n − m. Furthermore, we always know the length
of the array.

1.2 Machine model

We formulated the algorithms for binary search in pseudocode. Pseudocode
has no precise syntax. There are while and for loops, if-then-else statements
and returns statement. We only need to informally declare variables. For the
moment, the only built-in type are integers and the Boolean values true and
false. We saw comparsion operations like = or <, we will allow arithmetic

Markus Bläser 2009–11

6 1. Introduction

operations like +, −, ·, /, and rem, where the latter two are integer division
and remainder. := denotes the assignment operator. Pseudocode is very
intuitive. For instance, you should be able to transfer the pseudocode of
binary search into real code of the programming language of your choice
without any problems.
When we want to analyze the running time of an algorithm written in
pseudocode, we have to charge costs for different operations. Since we want
to abstract from different programming languages and architectures, there
is no point in discussing whether an addition is more costly than a multipli-
cation, so all statements in pseudocode take one unit of time.2
There is one exception: We have to bound the values that we can store
in integer variables. Otherwise we could create huge numbers by repeated
squaring: We start with 2, multiply it with itself, we get 4, multiply it
with itself and get 16 and so on. With n multiplications, we can create the
n
number of 22 , which has an exponential number of bits. We could even have
efficient algorithms for factoring integers, a problems that is supposed to be
hard; modern cryptographic systems are based on this assumption. We can
even have efficient algorithms for so-called NP-hard problems, which you will
encounter soon in the lecture “Grundzüge der Theoretischen Informatik”, see
the first assignement. (And even this is not the end of the story. . . ) So
we will require that our integer variables can only store values that are
polynomially large in the size of the data that we want to process. In the
binary search algorithm, n is the size of the input (or n + 1 if you also want
to count x.) Our integers can hold values up to some polynomial in n (of
our choice), say, n2 . This is enough to address all the elements of the array,
so this is a reasonable model. If we want to have larger integers, we have
to simulate them by an array of small integers (like bignum packages do
this in reality). In our binary search example, the values of the variables `,
m, and r are always bounded by n + 1. The values of the a[i] and x are
part of the input. If they were large, then n would not be an appropriate
measure for the size of the input but rather the sum of the bit lengths of the
array elements and x. But the array elements and x play a special role in
binary search, since we only access them via comparisons. So there actual
size does not matter at all, we could even assume that they are real numbers
or something else, as long as there is a total order on them.
The underlying architecture is the so-called Word RAM, RAM stands
for random access machine, the prefix Word indicates that we can store
numbers that are polynomially large in the input size in the cells. This is
a mathematical abstraction of the von Neumann architecture, a pioneering
computer architecture that still influences modern architectures. But since
2
This oversimplification turned out to be very fruitful, since we now can concentrate
one finding better algorithms and not trying to avoid multiplications because they are
so costly. Nevertheless, if we want to do an implementation in a concrete programming
language of an algorithm given in pseudocode, you have to think about such details.

Markus Bläser 2009–11

1.3. Asymptotic growth of functions 7

pseudocode is so intuitive, we do not need to go into further details here.

If you want to learn more about the underlying model, it is a good idea to
read Section 2.2 of the book by Mehlhorn and Sanders.
Modern computer architectures are more complicated, they have differ-
ent levels of caches, which one should exploit when implementing these al-
gorithms. But so-called external memory algorithms are beyond the scope
of this lecture.

1.3 Asymptotic growth of functions

Next, we want to analyze the running time of binary search. Usually we
do not care too much about constant factors. First of all, this has been a
very fruitful approach in the past decades. It is usually much better to first
care about the asymptotic behavior. For example, if looking at the CD in
the middle of the shelf takes three times longer than looking at a CD when
you scan them one after an other, because you first have to determine the
middle, even for small-sized CD collections, binary search is preferable.
Second, it is very hard to precisely calculate the exact running times. Do
we count the number of actual cycles? The number of statements? Do dif-
ferent statements have different costs? By only caring about the asymptotic
behaviour, we are abstracting away from these details and we can concen-
trate on finding good abstract algorithms. Of course, when it comes down to
implementing an algorithm, we have to think about the underlying architec-
ture. And it might turn out that for your concrete architecture and typical
input size, an asymptotically slower algorithm is preferable.
Let R≥0 denote the set of all nonnegative real numbers. We call a function
f : N → R≥0 asymptotically positive if it is positive almost everywhere,
i.e., f (n) = 0 only holds for finitely many n. Whenever we investigate
the asymptotic growth of functions, we will tacitly assume that they are
asymptotically positive, which is fine, since we are estimating the running
time of algorithms. The number of steps that an algorithm can make is
always in integer, so running times are always functions N → N. By allowing
functions N → R≥0 , some calculations will become easier .

Definition 1.2 Let f : N → R≥0 be a function that is asymptotically posi-

tive.

1. O(f ) = {g : N → R≥0 | there are c > 0 and n0 ∈ N

such that g(n) ≤ cf (n) for all n ≥ n0 },

2. Ω(f ) = {g : N → R≥0 | there are c > 0 and n0 ∈ N

such that g(n) ≥ cf (n) for all n ≥ n0 },

3. Θ(f ) = O(f ) ∩ Ω(g),

Markus Bläser 2009–11

8 1. Introduction

4. o(f ) = {g : N → R≥0 | for all c > 0 there is an n0 ∈ N

such that g(n) ≤ cf (n) for all n ≥ n0 },

5. ω(f ) = {g : N → R≥0 | for all c > 0 there is an n0 ∈ N

such that g(n) ≥ cf (n) for all n ≥ n0 }.

Above c can be an arbitrary positive real number.

If g ∈ O(f ), then g does not asymptotically grow faster than f . If

g ∈ Ω(f ), then g grows asymptotically at least as fast as f . If g = Θ(f ),
then the asymptotic growth of f and g is the same. If g ∈ o(f ), then f grows
asymptotically strictly faster than g; if g ∈ ω(f ), then g grows asymptotically
strictly faster than f .

Example 1.3 1. 2n2 + 1 ∈ O(n2 ), since 2n2 + 1 ≤ 3n2 for all n ≥ 1.

2. 2n2 + 1 ∈ Ω(n2 ), because 2n2 + 1 ≥ n2 for all n ≥ 0. Thus 2n2 + 1 =

Θ(n2 ).

Remark 1.4 A little warning: In some books, a different definition of Ω(f )

is used, namely, g(n) ≥ cf (n) has to hold infinitely often (instead of almost
everywhere). The idea is that the O-notation is used to give upper bounds
for the running time. A reasonable upper bound should hold for almost all
inputs. The Ω-notation is used to state lower bounds. One can consider an
algorithm bad if it has a particular running time on infinitely many inputs
(and not on almost all).

Lemma 1.5 1. If f ∈ O(g), then g ∈ Ω(f ) and vice versa.

2. If f ∈ o(g), then g ∈ ω(f ) and vice versa.

Proof. We only prove the first statement, the second is proven in the
same way. If f ∈ O(g), then there is a constant c and an n0 such that
f (n) ≤ cg(n) for all n ≥ n0 . But this means that g(n) ≥ 1c f (n) for all
n ≥ n0 . Hence g ∈ Ω(f ). This argument is reversible, which shows the “vice
versa”-part (replace c by 1c ).

Exercise 1.1 Prove that Θ(f ) = {g : N → R≥0 | there are c, c0 > 0 and n0 ∈ N
such that c0 f (n) ≤ g(n) ≤ cf (n) for all n ≥ n0 }.

Syntactic sugar 1.6 Although O(f ) and all the other classes are sets of
functions, we will often treat them like a function. “Algorithm A has running
time O(n2 )” means that the running time of A is a function in O(n2 ). Many
books also write g = O(f ) instead of g ∈ O(f ) and O(g) = O(f ) instead of
O(g) ⊆ O(f ). I will try to avoid this.

Markus Bläser 2009–11

1.4. Running time analysis 9

1.4 Running time analysis

1.4.1 On foot
To prove that binary search terminates, we showed that r − ` decreases in
every iteration. To analyze the running time, we have to bound the amout
by which r − ` decreases.
But what does it mean “to analyze the running time”?. We can of course
count the number of steps for each possible input. But this can get very
complicated. Therefore, we will always puts inputs into groups of the same
size and measure the running time for them. For binary search, n, the length
of the array, is a natural choice. We will consider worst case running times.
This means that we try to find an upper bound for all running times of the
binary search algorithm on inputs of length n. In other words, we want to
bound the maximum of all running times on inputs of length n.3

Lemma 1.7 Let λ and ρ be the value of ` and r before an iteration of the
loop, and λ0 and ρ0 after the iteration.4 Then

ρ0 − λ0 − 1 ≤ b(ρ − λ − 1)/2c.

Proof. Let µ = b(ρ+λ)/2c. In one iteration, either the value of r becomes

µ or the value of λ becomes µ. Therefore,

ρ0 − λ0 − 1 ≤ max{ρ − µ − 1, µ − λ − 1}
1
≤ max{ρ − ((ρ + λ)/2 − ) − 1, (ρ + λ)/2 − λ − 1}
2
= max{(ρ − λ − 1)/2, (ρ − λ)/2 − 1}
= (ρ − λ − 1)/2.

Since ρ0 + λ0 − 1 is a natural number, we can even round down the right hand
side.

Exercise 1.2 Convince yourself that for all n ∈ N,

3
Considering worst case running times has been proven very fruitful. It is usually
easier to analyze and it bounds the running time of every input. Nevertheless, we will
occasionally consider average case running times, that is, we will try to estimate the
average of the running times of all inputs of length n.
4
You should habe learned that it is important to distinguish between the variable (part
of the syntax) and its value (part of the semantic). ` is the variable name, λ and λ0 are two
values. Since you got older, we will often use the same symbol for the variable and for its
value, if there is no or only little danger of confusion, and sometimes even if there is real
danger of confusion. For instance, at the beginning of this paragraph, we wrote that r − `
decreases. Correctly, we had to write the value of r minus the value of ` decreases. But
since it is sooo convenient, we will often write the name of the variable and mean the value.
Take this chance to write all the things you ever wanted to write in the “Programmierung
1+2” lectures and still get credits for it.

Markus Bläser 2009–11

10 1. Introduction

1. d n2 e = b n+1
2 c,
n 1
2. 2 − 2 ≤ b n2 c ≤ n2 .

Thus in every step, the quantity r − ` − 1 is halved. In the beginning,

we have r − ` − 1 = n. If n = 0, then there are no iterations. Assume that
n > 0. After i iterations, r − ` − 1 ≤ bn/2i c. We stop, if ` + 1 ≥ r, that
is r − ` − 1 ≤ 0. This is equivalent to n/2i < 1. (r − ` − 1 is an integer!)
The smallest integer i0 such that n/2i0 < 1 is i0 = blog2 nc + 1. Therefore,
Binary-search performs at most blog2 nc + 1 iterations. Since every iteration
consists of a constant number of operations, we get the following theorem.

Theorem 1.8 The running time of Binary-search is O(log2 n).

Exercise 1.3 Binary-search looks at blog2 nc + 1 elements of the array.

Prove that this is optimal, i.e., every algorithm that correctly decides whether
x is in the array or not has to look at ≥ blog2 nc + 1 elements.

1.4.2 By solving recurrences

We now invest a little more to get a nicer derivation of the running time.
We will essentially only analyze one step of the binary search algorithm and
then let our theorems do the rest.

Syntactic sugar 1.9 For f, g : N → R≥0 , f ◦ g : N → R≥0 is the function

given by n 7→ f (dg(n)e). We extend this to functions of larger arity in the
obvious way.

Lemma 1.10 Let g1 , . . . , g` : N → R≥0 be functions such that gλ (m) ≤ m−1

for all m > 0 and 1 ≤ λ ≤ `. Let c ∈ R≥0 and h : N → R≥0 . Then the
recurrence
(
c if n = 0
f (n) =
f (g1 (n)) + · · · + f (g` (n)) + h(n) if n > 0

has a unique solution f : N → R≥0 .

Proof. We construct the solution f inductively.

Induction base: f (0) := c fulfills the equation and is the only possible solu-
tion.
Induction step: We assume that we already defined all f (m) with m < n
and that there was only one possible solution. Since gλ (n) ≤ n − 1 for all
λ, f (gλ (n)) is defined and it is unique. Thus there is only one way to define
f (n), namely f (n) := f (g1 (n)) + · · · + f (g` (n)) + h(n).

Markus Bläser 2009–11

1.4. Running time analysis 11

Remark 1.11 The same is true (with the same proof ), if the values of
f (0), f (1), . . . , f (i) are given. In this case gλ (m) ≤ m − 1 has to hold only
for m > i.

Remark 1.12 Again with the same proof, one can show that if fˆ satisfies
the inequalities fˆ(n) ≤ fˆ(g1 (n))+· · ·+ fˆ(g` (n))+h(n) if n > 0 and fˆ(0) ≤ c,
then fˆ(n) ≤ f (n) for all n where f is the unique solution of the equation.

Throughout this lecture, log : N → R≥0 will denote the function

(
0 if n = 0
n 7→
blog2 (n)c + 1 if n > 0

Here log2 : R → R is the “ordinary” logarithm to base 2. log n essentially

is the number of bits of the binary expansion of n. (0 is represented by the
empty word.)

Exercise 1.4 Show that log n = dlog2 (n + 1)e.

Lemma 1.13 The function log is the unique solution of the equation
(
f (b n2 c) + 1 if n > 0
f (n) =
0 if n = 0

Proof. By Lemma 1.10, the equation has a unique solution, since b n2 c < n
for n > 0. The function log is monotonically increasing. It is constant on
the sets {2ν , . . . , 2ν+1 − 1} for all ν and log(2ν+1 ) = log(2ν+1 − 1) + 1.
We claim that if n ∈ {2ν , . . . , 2ν+1 − 1}, then b n2 c ∈ {2ν−1 , . . . , 2ν − 1}.
Since n → b n2 c is a monotonically increasing function, it is sufficient to check
this for the borders:

b2ν /2c = b2ν−1 c = 2ν−1 ,

b(2ν+1 − 1)/2c = b2ν − 1/2c = 2ν − 1.

Therefore,
log(bn/2c) + 1 = log(n)
for all n > 0.

Corollary 1.14 Let c ∈ R≥0 . The function c · log n is the unique solution
of (
f (b n2 c) + c if n > 0
f (n) =
0 if n = 0

Markus Bläser 2009–11

12 1. Introduction

1
Proof. By dividing the equations by c, we see that c · f = log.
Now we look at the recursive version of binary search. Let t : N → N
denote an upper bound for the number of comparisons5 made by binary
search. t fulfills

t(0) = 0
t(n) ≤ t(bn/2c) + 1.

The second equation follows from Lemma 1.7. Thus by Lemma 1.13, t(n) ≤
log(n) for all n.

Disclaimer

Algorithms and data structures are a wide field. There are way
more interesting and fascinating algorithms and data structures than
would ever fit into this course. There are even way more important
and basic algorithms and data structures than would fit into this
course. Therefore, I recommend that you study related material in
text books to get a broader understanding. The books by Cormen,
Leiserson, Rivest, and Stein and by Mehlhorn and Sanders are
excellent, but there are many more.

5
For the ease of presentation, we count the “a[m] = x” and “a[m] < x” comparisons
as one comparisons. Think of it as having a three-valued comparisons that returns the
results “smaller”, “equal”, or “larger”.

Markus Bläser 2009–11

2 Sorting

2.1 Selection-sort
To work properly, binary search requires that the array is sorted. How fast
can we sort an array? A simple algorithm works as follows: Look for the
minimum element of the array, put it at the front. Then sort the rest of
the array in the same way. Look at the minimum element of the remaining
array and put it at the front, and so on. In this way, after m iterations, the
first m elements are sorted and contain the m smallest elements of the array.
Algorithm 3 shows one implementation of this. It is written recursively, but
one can easily replace the recursion by one for loop (does not look so nice,
but usually yields a faster code).
The correctness of the algorithm is easily shown by induction.

Lemma 2.1 After executing Selection_sort(a[1..n]), a[1] ≤ a[2] ≤ · · · ≤

a[n] holds.

Proof. By induction on n.
Induction base: If n ≤ 1, then there is nothing to sort and selection sort
does nothing.
Induction step: By the induction hypothesis, envoking Selection_sort(a[2..n])
sorts the subarray a[2..n]. Since a[1] is the smallest among the elements in
the array, the complete array a[1..n] is sorted, too.

Algorithm 3 Selection-sort
Input: array a[1..n]
Output: afterwards, a[1] ≤ a[2] ≤ · · · ≤ a[n] holds
1: if n > 1 then
2: i := Select-minimum(a[1..n])
3: h := a[1]; a[1] := a[i]; a[i] := h;
4: Selection-sort(a[2..n]);

The subroutine Select-minimum can be implemented as follows. The

index i stores the index of the current minimum. In the for loop, we compare
every element with the current minimum and update the index i if a smaller
element is found.
The correctness follows from the invariant of the for loop. Again, this
invariant can be easily shown by induction. To train a little bit, we will do

13
14 2. Sorting

the proof formally. Later, we will often just state the invariant and give a
hint how to prove it, assuming that you can do the proof on your own.

Lemma 2.2 The invariant in Algorithm 4 is fulfilled before every iteration

of the for loop.

Proof. The proof is by induction on j.

Induction base: When j = 2, then the invariant is trivially fulfilled, since
a[1] is the minimum of a[1..j − 1].
Induction step: Assume that a[i] is the minimum of a[1..j − 1], that is, the
invariant was true before the (j − 1)th execution of the loop. If j > n, there
is nothing to prove. Otherwise, in the loop, a[i] is compared with a[j] and i
is set to j if a[j] < a[i]. Thus a[i] is now the minimum of a[1..j]. Therefore,
the invariant is true before the jth execution, too.

Algorithm 4 Select-minimum
Input: array a[1..n]
Output: index i such that a[i] = min1≤j≤n a[j]
1: i = 1
2: for j = 2, . . . , n do /* i ≤ j − 1 ∧ a[i] is the min of a[1..j − 1] */
3: if a[j] < a[i] then
4: i := j
5: return i;

How many comparisons does Selection-sort do? Select-minimum does

n−1 comparisons when selecting the minimum of an array of length n. Thus
we get the following recursive equation for the number c(n) of comparisons
of selection sort:
(
0 if n = 0, 1
c(n) =
n − 1 + c(n − 1) if n > 1

We can determine c(n) by the so-called iteration method, an “on foot” method,
which accidently turns out to be elegant here:

c(n) = n − 1 + c(n − 1)
= n − 1 + n − 2 + c(n − 2)
= n − 1 + n − 2 + n − 3 + · · · + 1 + c(1)
n−1
X
= i
i=1
1
= n(n − 1) = O(n2 ).
2
Thus the overall running time of selection sort is O(n2 ).

Markus Bläser 2009–11

2.2. Merge sort 15

Excursus: Gauß’s formula for consecutive integer sums

Karl Friedrich Gauß (1777–1855) was a German mathematician, astronomer, and
geodesist. If he had lived today, he would also have been a computer scientist for
sure.
One hears the following anecdote quite often:1 At the age of seven, Gauß
attended the Volksschule. To have some spare time, the teacher Büttner told the
pupils to sum up the numbers from 1 to 100. Gauß quickly got the answer 5050 by
recognizing
P100 that 1 + 100 = 101, 2 + 99 = 101, . . . 50 + 51 = 101, in other words,
1
i=1 i = 2 100 · 101. Poor Büttner.
·

2.2 Merge sort

There are other algorithms that sort. Insertion sort is another natural al-
gorithm. It works the way people sort playing cards. We take one element
after another and insert it into the correct position. To find the position,
we compare it with the elements sorted so far until we find the right posi-
tion. Then we copy the rest of the array one place to the right and store
the current element into the free position. Inserting the ith element requires
O(i) steps, since either every element in the list of already sorted elements
is either compared to the current element or the element is moved. There-
fore, the number of steps is bounded by O(n2 ) (by the same analysis as for
selection sort).
Can we sort faster? Merge sort is a faster algorithm. We start with the
following subproblem. We have two arrays that are already sorted and want
to combine them into one sorted array. Algorithm 5 does the job. We assume
that our two arrays are stored in one larger array a[1..n]. The index t marks
the “border” between these two arrays, i.e., a[1..t] is sorted and a[t + 1..n]
is sorted. We have two indices i and j which mark the smallest element of
the two arrays that have not been processed so far. In the beginning, i = 1
and j = t + 1. Then we compare a[i] with a[j] and add the smaller one to
the array b. The condition “j = n + 1 or i ≤ t and a[i] < a[j]” also checks
whether one of the two arrays a[1..t] and a[t + 1..n] is already empty. We
assume that pseudocode does lazy evaluation. In this way, we do not try to
access array elements that are out of the boundary.
We can easily show by induction that the invariant of the for loop is
correct. From this it follows that a[1..n] is sorted in the end. For every
comparison that merge does, one element is added to b. For the last element,
we do not need a comparison. Thus merge does at most n − 1 comparisons
and the overall running time is O(n).
1
In this lecture notes, you occasionally find anecdotes about mathematicians and com-
puter scientists. I cannot guarantee that they are true but they are entertaining. This
particular one can be found on Wikipedia, which indicates that it is false.

Markus Bläser 2009–11

16 2. Sorting

Algorithm 5 Merge
Input: array a[1..n], integer t such that a[1..t] and a[t + 1..n] are sorted
Output: afterwards, a[1..n] is sorted
i := 1; j := t + 1
for k = 1, . . . , n do
/* b[1..k − 1] is sorted and all elements in b[1..k − 1] are
smaller than the elements in a[i..t] and a[j..n] */
if j = n + 1 or i ≤ t and a[i] < a[j] then
b[k] := a[i];
i := i + 1;
else
b[k] := a[j];
j := j + 1
a[1..n] := b[1..n];

Now we can sort as follows (Algorithm 6): If n ≤ 1, there is nothing to

sort. Otherwise, we divide the input array a[1..n] into two halves of (almost)
the same size. Then we recursively sort these two arrays. An easy induction
in n shows that merge sort indeed sorts.

Algorithm 6 Merge-sort
Input: array a[1..n]
Output: afterwards, a[1] ≤ a[2] ≤ · · · ≤ a[n] holds
if n > 1 then
m := bn/2c;
Merge-sort(a[1..m]);
Merge-sort(a[m + 1..n]);
Merge(a[1..n], m);

What can we say about the running time? Let us estimate the number
of comparisons c(n) that merge sort does on arrays of length n in the worst
case. We have
(
0 if n ≤ 1
c(n) ≤ n n
c(b 2 c) + c(d 2 e) + n − 1 if n > 1

since we split the array into two halves and need an additional n − 1 com-
parisons for merging. Since c is certainly monotonically increasing, we can
replace the second inequality by

c(n) ≤ 2c(d n2 e) + n − 1.

Markus Bläser 2009–11

2.2. Merge sort 17

c(n)
Let us divide the inequality by n − 1 and set ĉ(n) = n−1 for n ≥ 1. We get
c(n) c(d n2 e)
ĉ(n) = ≤2· +1
n−1 n−1
c(d n2 e)
≤2· +1
2(d n2 e − 1)
n
= ĉ(d e) + 1.
2
We “almost” know ĉ, since it “almost” fulfills the same recursive inequality
as log. In particular, ĉ(n) ≤ log(n − 1) for n > 0, because the function
n 7→ log(n − 1) fulfills the same recurrence:
log(n − 1) = log(b(n − 1)/2c) + 1 = log(dn/2e − 1) + 1 = log(dn/2e − 1) + 1.
Therefore, merge sorts makes c(n) = (n − 1)ĉ(n) = (n − 1) log(n − 1) com-
parsions in the worst case and the overall running is O(n log n).

Divide and Conquer

Divide and conquer is a general design principle. It consists of three

steps:

Divide: Split the problem into smaller subproblems.

In the case of merge sort, these are the two sublists.

Conquer: Solve the subproblems recursively.

This is done by the two recursive calls.

Combine: Combine the solutions of the subproblems to get a solu-

tion of the original problem.
This is done by the procedure merge.

Of course, the structure of the problem has to be suitable to allow a

divide and conquer approach.

Excursus: Sorting in practice

In practice, one does not just sort numbers. Sorting algorithms are for instance
used to sort database records according to some key. The rest of the record is
so-called satellite data. Instead of storing the whole records/objects in the array,
one usually only store a reference or pointer to the records/objects and just moves
the references in the array.
When sorting really large array, memory management can be more important
than minimising the number of comparisons, since fetching blocks from and writing
them to the hard disk is a really costly task. There are algorithms for this, too,
but we will not talk about them in this lecture.

Markus Bläser 2009–11

3 More sorting procedures

A sorting algorithms is called in place if it only stores a constant number of

values outside of the array. Merge sort is not an in place sorting algorithm,
since the merge procedure uses a second array. So merge sort needs twice as
much storage to run as is needed to store the data. For large data sets, this
can be problematic.

3.1 Heap sort

Heap sort is a sorting algorithm that is in place and whose running time is
also O(n log n) in the worst case. We introduce another important principle
of algorithm design, namely, data structures. Data structures are used to
manage the data during the execution of the algorithm. The data is struc-
tured in such a way that operations that are crucial for the algorithms can
be performed in an efficient manner. You can think of it as a class. We have
several methods which are described by their input/output behaviour and
we have upper bounds for their running times.

3.1.1 Heaps
A (binary) heap A can be viewed as an ordered binary tree all levels of which
are completely filled except for the bottom one. Each node of the tree stores
one element of our array. We have three basic functions

Parent(i) returns the parent node of i, if i has a parent.

Left(i) returns the left child of i, if i has a left child

Right(i) return the right child of i, if i has right child

So far, there is nothing special about heaps. The crucial things is the
they satisfy the heap property:

Heap property
A[Parent(i)] ≥ A[i] for all i except the root

18
3.1. Heap sort 19

1
26

2 3
23 16

4 5 6 7
19 16 11 12

8 9 10
15 13 10

Figure 3.1: A heap. The large numbers in the circles are the elements stored
in the heap. The small numbers next to the circle is the corresponding
position in the array.

That is, the value of every node is larger than the values of its two children.
Using induction, we get that for every node i in the heap, all nodes that are
below A[i] have a value that is at most A[i]. Warning! Do not confuse
our heaps with the heap in a JAVA environment.
Heaps can be efficiently implemented: Assume we want to store n ele-
ments in an array A. We just store them into in an array A[1..n]. We have
a variable heap-size which stores the number of elements in the heap.1 The
nodes are just the indices {1, . . . , heap-size}. The array may contain more
elements (with larger index), since later on, we will build a heap step by step
starting with a heap of size 1.
The value of the root is stored in A[1]. It has two children, we store their
value in A[2..3]. These two children have altogether four children. We can
store their values in A[4..7]. In general, the ith “layer” of the tree is stored in
A[2i ..2i+1 − 1]. (The root is the 0th layer.) The last layer might be shorter
and is stored in A[2h−1 ..heap-size]. Here h = log(heapsize) − 1 is the height
of the tree, the number of edges on a longest path that goes from the root
to a leaf. (Because of the structure of heaps, every path from the root to a
leaf has length either h − 1 or h.)
The methods Parent, Left, and Right can be easily implemented as fol-
lows:
It is easy to verify that the functions Left and Right map the set {2i , . . . , 2i+1 −
1} (level i) into the set {2i+1 , . . . , 2i+2 − 1} (level i + 1). Left is nothing else
1
Yes, I know, never ever have a global variable floating around somewhere. But for
pseudocode, this is okay. We just want to understand how heaps work. I am pretty sure
that with your knowledge from the “Programmierung 1+2” lectures, you can do a very
nice state-of-the art implementation.

Markus Bläser 2009–11

20 3. More sorting procedures

Algorithm 7 Parent(i)
1: return bi/2c

Algorithm 8 Left(i)
1: return 2i

but a left shift and adding the least significant bit 0, Right does the same but
adds a 1. In particular these implementations are very efficient and short,
so in reality, you could declare them as inline. Since we use the Left, Right,
and Parent procedures only inside other procedure, we do not check whether
the result is still a valid node, i.e, whether it is in {1, . . . , heap-size}; this has
to be done by the calling procedures.

3.1.2 Establishing the Heap property

The function Heapify is a crucial procedure to ensure the heap property. It
gets a node i as input. We promise that the two subtrees that have the
children Left(i) and Right(i) as roots satisfies the heap property, but the
subtree with root i may not. After execution, also the tree with root i
satisfies the heap property.
After the execution of lines 1–7, h ∈ {i, `, r} is chosen such that

A[h] = max{A[i], A[`], A[r]}.

Note that we check in lines 4 and 6 whether the children are indeed present.
If h = i, then the heap property is also satisfied by the subtree with root i
and we do not have to do anything. Otherwise, we exchange A[i] with A[h].
Now A[i] contains the largest element of the subtree with root i. The subtree
whose root is not h is not affected at all and still satisfies the heap property.
The other subtree with root h almost fulfills the heap property: The two
subtrees Left(h) and Right(h) fulfill the heap property but A[h] might not
be the largest element. But this is precisely the input situation of Heapify,
so we can apply recursion.
What’s the running time of Heapify? Since every call results in at most
one recursive call, the number of recursive calls is bounded by the height of
the subtree with root i. In each call, only a constant number of operations
are executed. Thus, the total running time is O(log n).

Exercise 3.1 Analyze the running time of Heapify by formulating a recur-

rence and solving it.

Algorithm 9 Right(i)
1: return 2i + 1

Markus Bläser 2009–11

3.1. Heap sort 21

Algorithm 10 Heapify
Input: heap A, index i such that the heap property is fulfilled for the sub-
heaps with roots Left(i) and Right(i)
Output: afterwards, the heap property is also fulfilled at i
1: ` := Left[i]
2: r := Right[i]
3: h := i
4: if ` ≤ heap-size and A[`] > A[i] then
5: h=`
6: if r ≤ heap-size and A[r] > A[h] then
7: h=r
8: if h 6= i then
9: swap(A[i], A[h])
10: Heapify(A, h)

3.1.3 Bulding a heap from scratch

Now assume we have an array A[1..n] and we want to convert it into a heap.
We can use the procedure Heapify in a bottom-up manner. Because the
indices {bn/2c, . . . , n} are all leaves, the subtree with a root j ≥ bn/2c is
a heap. Then we apply Heapify and ensure the heap property in a layer
by layer fashion. The correctness of the approach can be easily proven by
reverse induction on i.

Algorithm 11 Build-heap
Input: array A[1..n]
Output: afterwards, A satisfies the heap property
1: heap-size := n
2: for i = bn/2c, . . . , 1 do
3: Heapify(A, i)

What is the running time of Build-heap? An easy upper bound can

be obtained as follows: The running time of each Heapify call is O(log n)
and there are bn/2c such calls, therefore, the total running is bounded by
O(n log n). This bound is correct, since it is an upper bound, but we can do
better.
The crucial observation is for a lot of subtrees, the height is not log n
but much smaller. dn/2e nodes are leaves, the corresponding subtrees have
height 0. The leaves have dn/4e parents, the height of the corresponding
trees is 1, etc.

Exercise 3.2 Show that in general, a heap with n nodes has at most dn/2h+1 e
subtrees of height h.

Markus Bläser 2009–11

22 3. More sorting procedures

Therefore, we can bound the running of build-heap by

log n−1 l log n−1 ∞
! !
X n m X h X h
· O(h) ⊆ O n · ⊆O n· .
2h+1 2h 2h
h=0 h=0 h=0
We know that
∞
X 1
xh = .
1−x
h=0
for all x with |x| < 1 (geometric series). Taking derivates with respect to x,
and multiplying with x, we get
∞
X x
h · xh = .
(1 − x)2
h=0
P∞ h
Therefore, h=0 2h = 2 and
∞
!
X h
O n· ⊆ O(n)
2h
h=0

3.1.4 Heap sort

To sort an array A[1..n] using heaps, we first build a heap. Now we can easily
get the largest element of the array. It is A[1]. We exchange it with the last
element of the array. A[1] is now in the right place. The array A[1..n − 1]
is almost a heap, only the element A[1] may violate the heap property. But
this is exactly the situation for Heapify! After the call in line 5, A[1..n − 1]
is a heap and we can repeat the process.

Algorithm 12 Heap sort

Input: array A[1..n]
Output: afterwards, A is sorted
1: Build-heap(A)
2: for i := n, . . . , 2 do
3: Swap(A[1], A[i])
4: heap-size := heap-size − 1
5: Heapify(A, 1) /* A[i..n] is sorted and contains the n − i + 1
largest elements */

The correctness is again easily shown by induction. The analysis of the

running time is not hard either. The for loop is executed n − 1 times,
the Heapify call needs time O(log n). Therefore, the total running times is
O(n log n). Heap sort is a clever implementation of the selection sort idea.
Exercise 3.3 In the analysis of Build-heap, we got a better time bound when
estimating the running times of the Heapify calls more carefully. Does the
same work for Heap sort?

Markus Bläser 2009–11

3.2. Quick sort 23

3.2 Quick sort

Quick sort is an inplace sorting algorithm that is widely used in practice. Its
worst case running time is quadratic, but it performs very well in practice.
One can also show that the average running time is O(n log n). Since we
need some probability theory to prove this, we defer this proof until you
learn these things in MfI 3 and just state the algorithm.
Quick sort gets an array A and bounds ` and r. After execution, the
subarray A[`..r] is sorted. Quick sort works in a divide and conquer manner.
We divide the array into two parts and sort them recursively.

Algorithm 13 Quick sort

Input: array A[1..n], indices `, r
Output: afterwards, A[`..r] is sorted
1: if ` < r then
2: q := Partition(A, `, r)
3: Quicksort(A, `, q − 1)
4: Quicksort(A, q + 1, r)

Compared to merge sort, instead of just cutting the array into two halves,
sort recursively, and then merge, quick sort divides the array into two parts
in a more clever way such that no merging is needed after the recursive calls.
The function Partition does the following. It chooses a pivot element p
, that is, any element of A[`..r]. We simply choose p := A[`].2 Then we
rearrange A in such a way that A[1..q − 1] only contains elements ≤ p, A[q]
contains p, and A[q + 1..r] contains only elements ≥ p. Finally, we return q.
We now simply need to sort A[1..q − 1] and A[q + 1..r] recursively and are
done.
The function Partition can easily implement in time O(n). With a little
care, we can even do it inplace.
In the first line, the pivot element is chosen. We take the first element.
Then we move the pivot temporarily to the end. Then we run through the
array and compare whether A[i] ≤ p. If this is the case, then A[i] has to be
moved to the left. So we swap it with A[s]. The index s stores the current
border between the “lower” part and the “upper” part. Finally, we just put
the pivot again in the middle.

2
Choosing a pivot element in practice is somewhere between art and magic, since it
shall be chosen in such a way that the two parts are roughly balanced. Taking the first
element is often a bad choice in pratice, in particular, if the array is partly sorted.

Markus Bläser 2009–11

24 3. More sorting procedures

Algorithm 14 Partition
Input: array A[1..n], incides ` ≤ r
Output: afterwards, there is an index q such that all elements in A[`..q − 1]
are ≤ A[q] and the ones in A[q + 1..r] are ≥ A[q]. q is returned.
1: p := A[`]
2: swap(A[`], A[r])
3: s := `
4: for i := `, . . . , r − 1 do
5: if A[i] ≤ p then
6: swap(A[i], A[s])
7: s := s + 1
8: swap(A[s], A[r])
9: return s;

Markus Bläser 2009–11

4 Selection problems

Our procedure for finding the minimum or maximum needs n − 1 compar-

isons. We want to show that this is optimal.

4.1 Lower bounds

Let X be a finite set. Recall that a (binary) relation is a subset R ⊆ X × X.
R is irreflexive, if for all x ∈ X, (x, x) ∈
/ R. R is antisymmetric if for all
x 6= y, (x, y) ∈ R implies (y, x) ∈ / R. R is transitive, if for all x, y, z ∈ X,
(x, y) ∈ R and (y, z) ∈ R implies (x, z) ∈ R. A partial order on X is a
relation R ⊆ X × X that is irreflexive, antisymmetric, and transitiv. R is a
total order if in addition, (x, y) ∈ R or (y, x) ∈ R for all x 6= y.

Exercise 4.1 1. Show that the usual relation < on N is a total order.

2. What about ≤?1

3. Show that “a is a nontrivial divisor of b” is a partial order. (A divisor

is called nontrival if a 6= 1 and a 6= b.)

Instead of writing (x, y) ∈ R, we will often write xRy. With an R, this

does not look nice, on the other hand x < y looks nicer than (x, y) ∈<.
An element x is minimal with respect to R if for all y 6= x, (y, x) ∈ / R. x
is maximal with respect to R, if for all y 6= x, (x, y) ∈ / R.
We can represent relations as directed graphs, more precisely, (X, R) is
a directed graph. Since R is irreflexive, G has no self-loops. Antisymmetry
means that there is no cycle of length two, that is, two edges (x, y) and
(y, x). Transitivity means that whenever there is a directed path from x to
y, there is also an edge (x, y). (From the definition of transivity, this follows
directly for paths of length 2. The rest follows by induction on the path
length.) Altogether, this means that G is acyclic, i.e, does not contain any
directed cycles. Thus, every order induces an acyclic graph. On the other
hand, every acyclic graph induces a partial order by taking the transitive
closure.
A comparison tree is an ordered binary tree T = (V, E). Each node that
is not a leaf is labeled with a comparison x?y, where x, y are two variables
from a set X = {x1 , . . . , xn } of variables. For reasons that will become clear
1
Some authors prefer orders to be reflexive, that is, (x, x) ∈ R for all x ∈ X, instead
of irreflexive.

25
26 4. Selection problems

in one or two sentences, the left child of a node v is denoted by v< and the
right child is denoted by v> . With every node v of the graph, we associate
a relation R(v), which is defined inductively. If v is the root of the tree,
then R(v) = ∅, that is, we do not know anything about the relation between
the elements in X. Assume that we have defined R(v) for some node v and
that v is labeled with the comparison x?y. Then R(v< ) is the transitive
closure of R(v) ∪ {(x, y)}. In the same way, R(v> ) is the transitive closure of
R(v)∪{(y, x)}. So R(v) stores all the information about the relation between
the variables in X that the comparison tree has gathered so far (including
what it can deduce by transitivity). We always assume that the comparison
tree does nothing stupid, that is, it will never compare an x with itself nor
does it compare an x with a y with (x, y) ∈ R(v) or (y, x) ∈ R(v).
Let π be a permutation in Sn . For a relation R on x, the relation Rπ is
defined by
(xi , xj ) ∈ Rπ : ⇐⇒ (xπ(i) , xπ(j) ) ∈ R
for all i, j, that is, the relation is the same up to changing the roles of the
x1 , . . . , x n .
What does it mean that a comparison tree computes a particular relation
R on X? We have our variables X = {x1 , . . . , xn }. There is a hidden total
order, that is, there is a permutation π ∈ X such that xπ(1) < xπ(2) < · · · <
xπ(n) but the comparison tree does not know this permutation. Now we
“run” the comparison tree in the obvious way. At every node v, two elements
xi , xj ∈ X are compared. If xi < xj according to the order given by π, we
go to v< , otherwise we go to v> . Eventually, we get to a leaf `. We require
that R ⊆ R(`)π , that is, the comparisons tree finds the relation R between
the elements up the the renaming given by π.
In the case of sorting, the relation computed at every leaf is simply a total
order on X. That is, the relation computed by the tree is S = {(xi , xj ) | i <
j}.
In the case of minimum computation, the comparison tree shall be able
to tell at every leaf which one of the xj is the minimum. That is, the relation
computed by the tree is M = {(x1 , xi ) | 2 ≤ i ≤ n}. This means that x1 is
minimal and no other element is.
Every algorithm that we studied so far was comparison-based, that is,
we only accessed the elements of the array by comparisons. Therefore, ev-
ery algorithm yields a comparison tree by just running the algorithm and
whenever two elements are compared, we add a node to the comparison tree
and branch on the two possible outcomes. (If the algorithms does something
stupid and compares two elements where the outcome is already known, then
we do not add a node and there is no need to branch.) The number of com-
parisons that our algorithm does in the worst case is nothing else than the
height of the constructed comparisons tree.

Exercise 4.2 Formally prove the following: Let A be a sorting algorithm

Markus Bläser 2009–11

4.1. Lower bounds 27

x1 ?x2

x1 ?x3 x2 ?x3

{(x1 , x2 ), (x1 , x3 )} {(x1 , x2 ), (x3 , x1 ), (x3 , x2 )} {(x2 , x1 ), (x2 , x3 )} {(x2 , x1 ), (x3 , x2 ), (x3 , x1 )}

Figure 4.1: A comparison tree which finds the minimum of three elements.
The left child of a node v is the child v< , right child is the node v> . (Node
names are not drawn.) Each node that is not a leaf is labeled with a compar-
ison. Below each leaf, there is the relation that is computed at the leaf. At
two leaves, the tree learns the complete order between the three elements,
at the other two, it only knows that one is smaller than the other two, but
it does not know the relation between the other two. But this is fine, since
the tree shall only find the minimum.

which makes at most t(n) comparisons. Then there is a comparison tree

with height bounded by t(n) which sorts.

Comparison trees itself are not a good machine model for sorting, since
they are non-uniform: For most comparison trees, there is no compact way
to represent them! But we will use comparison trees only to show lower
bounds. And since every sorting algorithm in pseudocode gives a comparison
tree, this also yields lower bounds for the number of comparisons made by
algorithms in pseudocode.

4.1.1 Sorting
Theorem 4.1 Every comparison that sorts the elements in X = {x1 , . . . , xn }
has at least n! leaves.

Proof. If a comparison tree sorts, then two different permutations π and

π 0 obviously have to lead to different leaves ` and `0 , because S(`)π and
S(`0 )π0 have to be the same relation S, which is a total ordering. There are
n! permutations of {1, . . . , n}.
Since a tree with N leaves has at height at least log N − 1, every pseu-
docode algorithm that sorts by using comparisons needs log(n!) − 1 compar-

Markus Bläser 2009–11

28 4. Selection problems

isons. A rough estimate for n! is

nn
2
≤ n! ≤ nn
2
(why?). Because log2 n ≤ log n ≤ log2 n + 1, we get the lower bound
nn n n
2
log(n!) ≥ log2 = log2 n − ∈ Ω(n log n).
2 2 2
A more precise estimate for n! is Stirling’s formula, which says that
√  n n
n! = 2πn e λn
e
1 1
for some λn ∈ ( 12n+1 , 12n ). With this, we get

1 log2 e
log(n!) ≥ n log2 n − n log2 e + log2 (2πn) + ≥ n log n − n(log2 e + 1).
2 12n + 1
Merge sort uses (n − 1) log(n − 1) ≤ n log n comparisons. So up to a linear
lower order term, merge sort uses the minimum number of comparisons.
Let us first make the proof a little more complicated. In this way, the
method becomes applicable to a wider range of problems. Consider a com-
parison tree T with node set V . A function w : V → R is a potential function
if w(v) ≤ w(v< ) + w(v> ) for all nodes v that are not a leaf.

Lemma 4.2 Let T be a comparison tree and w be a potential function. If

w(v) ≤ B for all leaves v, then the height of T is ≥ log2 ( w(r)
B ) where r is the
root of T .

Proof. We prove the following more general statement: For every node
v, the height of the subtree with root v is ≥ log2 ( w(v)
B ). The proof is by
induction on the height of the subtree.
Induction base: If v is a leaf, then w(v) ≤ B, hence w(v)/B ≤ 1 and
log2 ( w(r)
B ) ≤ 0.
Induction step: Let v be any node that is not a leaf. Since w(v) ≤ w(v< ) +
w(v> ), we have w(v) 2 ≤ max{w(v< ), w(v> )}. W.l.o.g. let w(v< ) ≥ w(v> ).
By the induction hypothesis, we know that the height of the subtree with
root v< is at least log( w(v <)
B ). The height of the subtree with root v therefore
is at least
     
w(v< ) w(v< ) w(v)
log2 + 1 = log2 2 · ≥ log2 .
B B B

We call a permutation π consistent with an order R on X, if, whenever

(xi , xj ) ∈ R, then xi < xj in the hidden total order given by the permutation
π. That is, R does not contradict the total order induced by π.

Markus Bläser 2009–11

4.2. Upper bounds for selecting the median 29

Consider the following function s : V → R defined by

s(v) = the number of permutations that are consistent with R(v).

We claim that s is a potential function: If π is consistent with R(v), then π

is either consistent with R(v< ) or R(v> ). Thus s(v) = s(v< ) + s(v> ). We
have s(r) = n! for the root r and s(`) = 1 for every leaf `. Therefore, the
height of the tree is a least log2 (n!) − 1.

4.1.2 Minimum selection

We can use potential functions to show that selecting the minimum requires
at least n − 1 comparisons. While this is intuitively clear, since one “has to
look at every element”, we could for instance just compare pairs of elements
(n/2 comparisons) and have looked at every element and still do not know
the minimum. Consider the following function

m(v) = 2number of minima in R(v) .

Assume that at a node v, R(v) has i minima. Then both R(v< ) and R(v> )
have at least i−1 minima, since a comparison can kill at most one minimum.
Because 2i = 2i−1 + 2i−1 , m is a potential function. At the root r, we have
m(r) = n, since every element is a minimum. At a leaf v, m(v) = 1, since
n
our tree finds the minimum. Therefore, the height is ≥ log2 ( 22 ) = n − 1.

Theorem 4.3 Every comparison tree that finds the minimum has height
≥ n − 1.

4.2 Upper bounds for selecting the median

Selecting the minimum can be done with n − 1 comparisons. What about
selecting the median, the element in middle. Given n elements and a total
order on it, the median is the unique element x such that b(n−1)/2c elements
are smaller than x and d(n−1)/2e elements are larger than x. Or what about
selecting the ith largest elements for any i ∈ {1, . . . , n}? We can of course
sort the elements using any sorting algorithm and then pick the ith largest
element. But we can do it with O(n) comparisons for any i.
The algorithm above is written in Pseudopseudocode. Pseudopseudocode
uses natural language to describe the algorithm. Pseudopseudocode is a very
advanced language, since the user has to be very careful. Whenever you are in
doubt whether some piece of Pseudopseudocode is o.k., ask yourself whether
you can write a program in Pseudocode that does the same. When analysing
the running time of a program in Pseudopseudocode, you have to analyze
the running time of the corresponding program in Pseudocode.

Markus Bläser 2009–11

30 4. Selection problems

Algorithm 15 Select
Input: array a[1..n], integer i
Output: the ith largest element in a
1: If n ≤ 60, then find the ith largest element by sorting.
2: Divide the n elements in bn/5c groups of 5 elements. At most 4 elements
remain.
3: Find the median of each of the bn/5c groups by sorting.
4: Recursively call Select to find the median m of the bn/5c medians.
(Once we found m, we forget about the groups.)
5: Use the procedure Partition on a with m as the pivot element.
6: Let q be the position of m in the array.
7: If q = i, then return m.
8: If i < q, then call Select(a[1..q − 1], i). Else call Select(a[q + 1..n], i − q).

The algorithm Select first groups the elements into bn/5c groups of five
elements. From each of the groups it computes the median for instance just
by sorting. (With Mergesort this takes 12 comparisons; there is also an
(optimal) comparison tree for selecting the median with height 6). We use
Select recursively to find the median m of the bn/5c medians and use m as
a pivot. m is a good pivot, because it yields a rather balanced partition.

3 3
Lemma 4.4 q − 1 ≥ 10 n − 3 and n − q ≥ 10 n − 3.

Proof. Let r = bn/5c be the number of groups with 5 elements.

Since m is the median of the medians, d 12 (r − 1)e medians are larger and
1
b 2 (r − 1)e medians are smaller than m. For every median, there are two
elements that are larger than the median and two that are smaller (in his
group). And there are two elements that are larger than m and two elements
that are smaller. Altogether, there are at least 3 · d 21 (r − 1)e + 2 elements
that are larger than m, namely the larger medians, the two elements in each
group that are larger than the median, and the two larger elements in the
group of m, and 3 · b 21 (r − 1)c + 2 elements that are smaller. Since the later
quantity is smaller, we only have to bound this one:
   
1 1 1
3· (r − 1) + 2 ≥ 3 · (r − 1) − +2
2 2 2
   
1 n 4 1
≥3· − −1 − +2
2 5 5 2
3
≥ n − 3.
10

By selecting the median of medians, we found a good pivot for the Par-
3
tition procedure. Since the size of each partition is at least 10 n − 3, it can

Markus Bläser 2009–11

4.2. Upper bounds for selecting the median 31

7
be, on the other hand, at most 10 n + 2 (we can subtract one more for the
median of medians).
Let t(n) be the number of comparisons made by select on arrays of length
n. We have
j n k   
7 jnk
t(n) = t +t n + 2 + 12 + n, (4.1)
5 10 5

if n > 60, since we have one recursive call with an array of length b n5 c for
finding the median of medians and another recursive call on the partition
that contains the true median. We need 12 · n5 comparisons to find the
 

medians in the groups and n − 1 comparisons within partition. If n ≤ 60,

we have t(n) ≤ (n − 1) log(n − 1) ≤ 8n.

Lemma 4.5 Let 1 , . . . , ` ∈ Q be positive. Let d, e ∈ Q be positive. Let

1
N ≥ 1−λ
for all 1 ≤ λ ≤ `. Assume that 1 + · · · + ` + N` < 1. If
(
t(d1 ne) + · · · + t(d` ne) + d · n if n > N
t(n) ≤
e·n otherwise

d
then t(n) ≤ c · n where c = max{ 1−( ` , e}.
1 +···+` + N )

Proof. Induction base: If n ≤ N , then t(n) = e · n ≤ c · n.

Induction step: Let n > N . By the induction hypothesis, we can assume
that t(m) ≤ cm for all m < n. By the choice of N , dλ ne < n. Thus we can
apply the induction hypothesis to all t(dλ N e). We have

t(n) ≤ t(d1 ne) + · · · + t(d` ne) + d · n

≤ c · d1 ne + · · · + c · d` ne + d · n
≤ c · (1 + · · · + ` )n + c · ` + d · n
 
` d
≤ c · n · 1 + · · · + ` + + .
N c
| {z }
≤1

7
We can use Lemma 4.5 to solve (4.1). We can bound 10 n + 2 from above
11 1 11 2 29
by 15 n for n > 60. Since 5 + 15 + 60 = 30 < 1, we get that t(n) ≤ c · n with
c = 132.

Remark 4.6 The actual value for c is rather coarse. We can get the term
`
N as small as we want by making N larger. In the same way, we can bound
7 7
10 n + 2 by h · n for an h that is arbitrarily close to 10 .

Markus Bläser 2009–11

5 Elementary data structures

Data structures store data in a clever way to support certain operations. We

already saw heaps that stored the data in such a way that we could quickly
find the minimum (or maximum).
In this chapter we will review some of the basic data structures. I am
sure that you have seen them in the “Programmierung 1+2” lectures.

5.1 Stacks
Stacks provide two operations. We can insert an element and we can remove
it. These two operations are usually called Push and Pop. Stacks use the
last-in-first-out principle. They work like stacks in real life: When you want
to put something on the stack, you can only put it on the top. And when
you want to remove something from the stack, you can only take the element
on the top.
To implement a stack, all you need is an array S[1..N ] and a variable
top. This variable stores the index of the last element put on the stack. In
the beginning, we set top to 0.
The Algorithm IsEmpty checks, whether there are elements on the stack.

Algorithm 16 IsEmpty
Input: stack S
Output: 1 if S is empty, 0 otherwise
1: if top = 0 then
2: return 1
3: else
4: return 0

Push pushes an element x on the stack. It first increases top by one and
then stores x in the new position. It also checks whether the stack is full. (If
for some reason, we know that there will be no stack overflow, one can also
dismiss this check for efficiency reasons.)
Pop removes on element from the stack and returns it. It also checks
whether there is an element to pop.
All the operations supported by the stack take O(1) time. Often, a Peek
procedure is provided in addition. Peek returns the top element without
removing it. Peek can be simulated by popping the top element and pushing
it back.

32
5.2. Queues 33

Algorithm 17 Push
Input: stack S, element x
Output: adds x to S
1: if top ≥ N then
2: “error”
3: else
4: top = top + 1;
5: S[top] = x;

Algorithm 18 Pop
Input: stack S
Output: returns and removes the top element of S
1: if IsEmpty(S) then
2: “error”
3: else
4: top := top − 1
5: return S[top+1 ]

5.2 Queues
Queues do the same as stacks: We can store elements and retrieve them.
However, the storing princicple is first-in-first-out. Queues work like queues
in real life, say, in the mensa. If a student arrives, he or she goes to the end
of the queue. Whenever a meal is handed out, the first student in the queue
leaves happily (?) with his or her meal.
To implement a queue, we need an array Q[1..N ]. We have two variables,
head and tail . The variable head stores the index of the elements that is the
first in the queue, tail the index of position after the queue where the next
arriving element will be put. In the beginning, head and tail will be set to
1.
The queue will be arranged in a circular way in Q. If head ≤ tail , then
the queue consists of the elements Q[head ..tail − 1]. If tail < head , then the
queue are the elements Q[head ..N ] together with the elements Q[1..tail − 1].
In this way, we do not have to shift the queues by one whenener an element
is dequeued. (Unlike queues in real life, which usually move forward when
the first element is removed. One exception might be very bad meals.)
The queue is empty, when head = tail . The queue is full, when head =
tail + 1. In the latter case, there is still one free cell, but if we occupied
this one, then head = tail , which we could not distinguish from the empty
queue. (This is an information theoretic problem: there are n + 1 possible
numbers of elements that can be stored in the queue, namely 0, 1, . . . , n.
On the other hand, the difference between head and tail can only attain n
different values.) If we want to uses this last cell, then we should introduce

Markus Bläser 2009–11

34 5. Elementary data structures

a variable count which counts the length of the queue. However, the first
solution usually gives a slightly more efficient code.
Procedure IsEmpty checks whether the queue is empty.

Algorithm 19 IsEmpty
Input: queue Q
Output: 1 if Q is empty, 0 otherwise
1: if head = tail then
2: return 1
3: else
4: return 0

Procedure Enqueue appends x to the queue. First, it checks whether the

queue if full. The second part of the disjunction checks the case that the
queue occupies Q[1..N −1]. If the queue is not full, then we store x in Q[tail ]
and increase tail by 1. If tail will get larger than N , then we set it to 1.

Algorithm 20 Enqueue
Input: queue Q, element x
Output: adds x to Q
1: if head = tail + 1 or head = 1 and tail = N then
2: “error”
3: else
4: Q[tail ] = x
5: if tail = N then
6: tail = 1
7: else
8: tail = tail + 1

Procedure Dequeue returns the first element and works in a similar man-
ner like Enqueue. All three procedures have running time O(1).

5.3 Linked lists

A linked list stores elements in a linear order. Unlike an array, we do not have
direct access to the elements via the indices. The elements are linked via
references. This yields a very flexible structure though not every operation
is very efficient. There are singly linked list and doubly linked list. In a
singly linked list, for every element in the list, we only store a reference to
the next object in the list. In a doubly linked list, we also store a reference
to the previous object in the list.
For every element x in a doubly linked list, there are three basic methods:
Prev(x): is a reference to the previous element in the list (not present in a
singly linked list).

Markus Bläser 2009–11

5.3. Linked lists 35

Algorithm 21 Dequeue
Input: queue Q
Output: the first element of Q
1: if head = tail then
2: “error”
3: else
4: x := Q[head ]
5: if head = N then
6: head := 1
7: else
8: head := head + 1
9: return x

Next(x): is a reference to the next element

Key(x): is the key of the element

Usually, there is more data to store than the key. So we also store a reference
to the element data. But this is not important for our considerations. For the
first element h in the list, the head, Prev(h) = NULL is a null reference. For
the last element in the list, the tail, Next(x) = NULL. There is a reference
head which points to the first element of the list.
The procedure List-search finds an element in the list with a given key,
if there is such an element, otherwise it returns the NULL reference. In the
worst case, it has to scan the whole list, therefore, the running time is O(n).
Note that the while loop either stops when we found the key k or we are
sure that there is no node with key k.

Algorithm 22 List-search
Input: a list L, a key k
Output: (a reference to) an element x with Key(x) = k, if there exists one,
NULL otherwise
1: x := head
2: while x 6= NULL and Key[x] 6= k do
3: x := Next(x)
4: return x.

The procedure List-insert adds an element to the front of the list. It takes
O(1) time. If we want to insert the element at a specific place, say after an
element y (maybe we want to keep the list sorted), the same algorithm works,
we just have to replace the reference head by Next(y).
Finally, the procedure List-delete deletes an element given by a reference
from a list. Its running time is O(1). If, however, the element is given by a
key, then we first have to use List-search to find the element.

Markus Bläser 2009–11

36 5. Elementary data structures

Algorithm 23 List-insert
Input: a list L, an element x
Output: appends x to the front of the list
1: Next(x) := head
2: if head 6= NULL then
3: Prev(head ) := x
4: head := x
5: Prev(x) := NULL

Algorithm 24 List-delete
Input: a list L, an element x
Output: removes x from the list
1: if Prev(x) 6= NULL then
2: Next(Prev(x)) := Next(x)
3: else
4: head := Next(x)
5: if Next(x) 6= NULL then
6: Prev(Next(x)) := Prev(x)

Stacks

IsEmpty: O(1) time, checks whether the stack is empty

Push: O(1) time, adds an element to the stack

Pop: O(1) time, removes an element to the stack

Stacks use the last-in-first-out principle.

Queues

IsEmpty: O(1), checks whether queue is empty

Enqueue: O(1), adds an element to the queue

Dequeue: O(1), removes an element from the queue

Queues use the first-in-first-out principle.

Markus Bläser 2009–11

5.3. Linked lists 37

Doubly linked lists

List-search: O(n), finds an element with a given key

List-insert: O(1), adds an element to the front of the list

List-delete: O(1), removes an element given by a reference

Markus Bläser 2009–11

6 Binary search trees

In many applications, like data bases, the data is dynamic: Elements are
inserted or deleted and we try to maintain the data in such a way that other
operations can be supported efficiently. Linked lists for instance support
such operations but the search is not efficient in lists.
Binary search trees are a simple data structure that support many dy-
namic set operations quite efficiently, in particular

Search: Finds an element with a particular key value (or detects that
there is no such element).

Minimum: Finds the element with the smallest key

Maximum: Finds the element with the largest key

Pred: Finds the element before a given element (according to the

keys)

Succ: Finds the element after a given element

Insert: Inserts a new element

Delete: Deletes a given element

Binary search trees are rooted and ordered binary trees that fulfill the
binary search tree property (see below). The running time of the methods
above is bounded by the height of the trees. The height of binary search
trees “behaves like” the running time of quick sort. In the good cases and
even on the average, the height is logarithmic. In the worst case, however,
it is linear. There are variations of binary search trees that are guarantued
to have height that is bounded by O(log n), n being the number of nodes in
the tree (= number of elements stored). We will study one of them in the
next chapter.
We represent trees as a linked data structure. There are four basic meth-
ods:

38
 39

Parent(v) returns the parent of v (and NULL, if v is the root).

Left(v) returns the left child of v (and NULL, if v has no left child).

Right(v) returns the right child of v (and NULL, if v has no right

child).

Key(v) returns the key of v according to which the elements will be

sorted.

Binary search trees fulfill the binary search tree property:

Binary search tree property

Key(v) ≤ Key(w) for all nodes w in the right subtree of v,
Key(v) ≥ Key(w) for all nodes w in the left subtree if v.

Every subtree of a binary search tree T is again a binary search tree. If

x is a node in T , we denote the subtree of T with root x by T (x).
The binary search tree property enables us to output the elements sorted
by key. This is done via a so-called inorder walk.

Algorithm 25 Inorder-walk
Input: A node x of a binary search tree T
Output: prints the elements of T (x) in increasing key order
1: if x 6= NULL then
2: Inorder-walk(Left(x))
3: output x;
4: Inorder-walk(Right(x))

Exercise 6.1 1. Prove that Inorder-walk is correct.

2. Prove that the running time of Inorder-walk is O(n) where n is the

number of elements in T .

Exercise 6.2 1. Can you reconstruct the binary search tree from the out-
put of Inorder-walk?

2. A preorder walk first outputs the root and then deals with the left and
right subtrees, i.e., lines 2 and 3 of the procedure Inorder-walk are
exchanged. Can you reconstruct the binary search tree from the output
of a preorder walk?

Markus Bläser 2009–11

40 6. Binary search trees

26

19 29

11 23 27 33

13 21 35

20 22

Figure 6.1: A binary search tree

6.1 Searching

If given a key k, we can easily check whether there is a node v in T with

Key(v) = k. We just start at the root r of T . If Key(r) = v, then we are
done. If Key(r) < k, then we have to look in the left subtree and otherwise
in the right subtree.

Algorithm 26 BST-search
Input: node x, key k
Output: a node y with Key(y) = k if such a y exists, NULL otherwise
1: if x = NULL or k = Key[x] then
2: return x
3: if k < Key(y) then
4: return BST-search(Left(x), k)
5: else
6: return BST-search(Right(x), k)

Lines 1 and 2 deal with case when we either found our node or a node
with key k is not in the tree. In the latter case, x = NULL, i.e, we were in
a node v without any left or right child and called BST-search with Left(v)
or Right(v), respectively, which is NULL. The running time of BST-search
is bounded by O(h) where h is the height of T .

Markus Bläser 2009–11

6.2. Minimum and maximum 41

6.2 Minimum and maximum

We can find the minimum in a binary search tree by always going to the left
child until we reach a node that has no left child.

Algorithm 27 BST-minimum
Input: a node x in a binary search tree T
Output: the node in T (x) with minimum key
1: if Left(x) 6= NULL then
2: return BST-minimum(Left(x))
3: return x

Lemma 6.1 BST-minimum is correct.

Proof. The proof is by induction in the height of T (x).

Induction base: All elements in T (Right(x)) have a key that is ≥ Key(x).
Thus, if Left(x) = NULL, then x must be the element with the smallest key.
Induction step: Assume that Left(x) 6= NULL. By the induction hypothesis,
BST-minimum(Left(x)) returns the minimum element y of T (Left(x)). But
since x and all elements in T (Right(x)) have a larger key by the binary search
property, y is also the minimum element of T (x).
The correctness of the procedures in the next sections can be shown in
a similar way, though the proofs get more complicated. But since binary
search trees occurred in the “Programmierung 1+2” lectures, we will skip
these proofs here.

Exercise 6.3 Write the corresponding procedure for the Maximum.

6.3 Successor and predecessor

Often it is useful to find the successor of a given node x. This means that we
are looking for the node with the smallest key that is greater than Key(x)
(assuming that all keys are distinct). This successor is completely determined
by the position in the tree, so we can find it without comparing any elements.
(And this also shows that everything works if the keys are not all different...)
If a node x has a right child, that is Right(v) 6= NULL, then the successor
simply is the node in T (Right(x)) with minimum key. If x does not have a
right child, then the successor of x is the lowest anchestor of x that has a
left child that is also an anchestor of x.
BST-successor computes the successor of a node x. The second part
deals with the case that x has no right child. We go up the paths from x
to the root. y is always the parent of x. We stop when either y = NULL
or x = Left(y). In the first case, x has no successor and NULL is returned.

Markus Bläser 2009–11

42 6. Binary search trees

Algorithm 28 BST-successor
Input: node x in a binary search tree T
Output: the successor of x, if x has a one, NULL otherwise
1: if Right(x) 6= NULL then
2: return BST-minimum(Right(x))
3: y := Parent(x)
4: while y 6= NULL and x 6= Left(y) do
5: x := y
6: y := Parent(y)
7: return y

In the second case, y is the lowest ancestor the left child of which is also an
ancestor. The running time is again linear in the height of T .

Exercise 6.4 Write the corresponding procedure for finding the predecessor.

6.4 Insertion and deletion

Now we want to insert a node z. Key(z) is initalized, but Left(z), Right(z)
and Parent(z) are all NULL. To insert a node in a binary search tree, similar
to the procedure BST-seach, we trace down a path from the root until we
would have to go to a child that is not present. This is the place where z
has to be inserted.

Algorithm 29 BST-insert
Input: node x of a binary search tree T , new node z
Output: inserts z into T (x) mainting the binary search tree property
1: if x = NULL then
2: z becomes the root of T
3: else
4: if Key(z) < Key(x) then
5: if Left(x) 6= NULL then
6: BST-insert(Left(x), z)
7: else
8: Parent(z) := x
9: Left(x) := z
10: else
11: if Right(x) 6= NULL then
12: BST-insert(Right(x), z)
13: else
14: Parent(z) := x
15: Right(x) := z

Markus Bläser 2009–11

6.4. Insertion and deletion 43

In lines 1 and 2, we check whether the tree is empty. In this case, z is

the new root of T . This check is only done once, since we ensure that in all
recursive calls that x ist not NULL. So one gets a somewhat more efficient
code by checking this in the beginning once and then removing this check in
the recursive calls. In line 3, we test whether z belongs to the left or right
subtree of x. Then there are two cases: If x has a corresponding child, we
do recursion. Otherwise, we found the place to insert z.
For deletion, we are given a tree T and a node x in T . We consider three
different cases:

x has no child: In this case, we can just remove x.

x has one child: In this case, we can just remove x and connect the child
of x to the parent of x. (If x does not have a parent, i.e., is the root,
then the child of x becomes the new root.

x has two children: Then we first search the successor of x. The successor
y of x has at most one child. So we first delete y as in one of the first
two cases and then replace x by y.

In lines 1 to 4, we select the node to delete. In line 19, we also have to

copy anything else that is stored into the node.

Exercise 6.5 Prove that the insertion and deletion procedures keep the bi-
nary search tree property.

Theorem 6.2 Searching, minimum and maximum computation, predecessor

and successor computation and insert and delete run in time O(h) on binary
search trees of height h.

Markus Bläser 2009–11

44 6. Binary search trees

Algorithm 30 BST-delete
Input: node x in a binary search tree T
Output: deletes x from T
1: if Left(x) = NULL or Right(x) = NULL then
2: y := x
3: else
4: y = BST-successor(x)
5: if Left(y) 6= NULL then
6: v := Left(y)
7: else
8: v := Right(y)
9: if v 6= NULL then
10: Parent(v) := Parent(y)
11: if Parent(y) = NULL then
12: v becomes the root of T
13: else
14: if y = Left(Parent(y)) then
15: Left(Parent(y)) := v
16: else
17: Right(Parent(y)) := v
18: if y 6= x then
19: Key(x) := Key(y)

26

19 29

11 23 27 33

13 21 35

20 22

Figure 6.2: Deletion of a leaf (with key 22). It is simply removed

Markus Bläser 2009–11

6.4. Insertion and deletion 45

26

19 29

11 23 27 33

13 21 35

20 22

Figure 6.3: Deletion of a node with one child (with key 33). The node is
removed and the child becomes a child of the parent.

26

19 29

11 23 27 33

13 21 35

20 22

Figure 6.4: Deletion of a node with two childs (with key 19). We search the
succecessor (key 20). The successor is removed as in the first two cases and
its content is copied into the node the shall be deleted.

Markus Bläser 2009–11

7 AVL trees

Binary search trees of height h support a lot of operations in time O(h).

While one can show that on the average, a binary search tree with n elements
has height O(log n), this need not be true in the worst case, in particular, if
we insert elements one after another that are already partially sorted. There
are several ways to create search trees that have logarithmic height even in
the worst case. One such an example are AVL trees.
Let T be a binary search tree. Whenever a child of a node x is missing, say
Left(x) = NULL, then we add a virtual leaf to this node. In this way, every
node has either degree two or is a virtual leaf. Nodes that are not virtual
leaves (i.e., the nodes from the original binary search tree) are called internal
nodes. Only internal nodes carry data. We need these virtual leaves for the
analysis, they are not needed in the implementation (a NULL reference is
interpreted as a virtual leaf).
For every node x in T , Height(x) is the height of the subtree T (x) with
root x. The virtual leaves have height 0. For every internal node x, the
balance of x is defined as

Bal(x) = Height(T (Left(x))) − Height(T (Right(x))).

An AVL tree is a binary search tree with the extra property that for every
internal node x, the height of the subtree with root Left(x) and the height
of the subtree with root Right(x) differ by at most one.

AVL tree property

1. An AVL tree fulfills the binary search tree property.

2. For every internal node v, Bal(v) ∈ {−1, 0, 1}.

For the static operations like searching, finding minimal elements, etc.
we can use the implementations for binary search trees. We will also use the
insert and delete procedure of binary search trees, but afterwards, we have
to ensure that the AVL property is restored.
AVL trees are named after their inventors, G. M. Adelson-Velsky and E.
M. Landis.
If we want implement AVL trees, we do not have to store the heights, it
is sufficient to store the balance. This information can be stored in two bits
which can often be squeezed somewhere in.

46
7.1. Bounds on the height 47

7.1 Bounds on the height

Definition 7.1 The Fibonacci numbers Fn are defined recursively by
F0 = 0,
F1 = 1,
Fn = Fn−1 + Fn−2 .
Exercise 7.1 Show that
√ !n √ !n !
1 1+ 5 1− 5
Fn = √ −
5 2 2

for all n (hint: induction on n). This is called the Moivre-Binet formula.

Excursus: Fibonacci numbers

Fibonacci numbers described the growth of a population living under ideal (?)
conditions. Assume you start with a pair of rabbits. After one month, they bear
another pair of rabbits at the end of every month. Every new pair does the same
after one month of waiting and rabbits never die. Then Fi is the number of pairs
of rabbits in the ith month.
√
Note that the base 1−2 5 of the second term is in (−1, 0). Therefore,
this term goes to 0 very fast and oscillates around zero. The growth of the
Fibonacci numbers is dominated by the first term, the base of which is in
(1, 2). Thus, the growth is exponential.
Lemma 7.2 For every AVL tree T with n internal nodes, the height of T is
bounded by O(log n).
Proof. We show by induction on n that every AVL tree of height h has
at least Fh+1 virtual leaves.
Induction base: An AVL tree of height 0 has 1 virtual leaf. An AVL tree of
height 1 has 2 virtual leaves.
Induction step: Let T be an AVL tree with height h. Let T1 and T2 be its
subtrees. One of them, say T1 , has height h − 1, the other one has height
≥ h − 2 by the AVL tree property. By the induction hypothesis, T1 has at
least Fh leaves and T2 has at least Fh−1 leaves. Therefore, T has a least
Fh + Fh−1 = Fh+1 leaves.
The claim of the lemma follows from this: Let T be an AVL tree with n
internal nodes (and n + 1 virtual leaves) and height h. We have
√ !h+1 √ !h+1 √ !h+1
 
1  1+ 5 1− 5 1 1 + 5
n+1 ≥ Fh+1 = √ − ≥ √ .
5 2 2 2 5 2

Therefore h ∈ O(log n).

Markus Bläser 2009–11

48 7. AVL trees

x y

y x

T1 T3

T2 T3 T1 T2

Figure 7.1: A left rotation around x transforms the tree on the left hand side
into the tree on the right hand side. A right rotation around y transforms
the tree on the right hand side into the tree on the left hand side.

7.2 Restoring the AVL tree property

Since every AVL tree is a binary search tree, it automatically supports the
static methods Search, Minimum, Maximum, Pred, and Succ. Since their
running time is O(h) and the height h of any AVL tree is O(log n), their
running time is O(log n). Insert and Deletion are also performed in the same
way, however, after inserting or deleting a node, the AVL tree property might
be violated and we have to restore it.

7.2.1 Rotations

Rotations are used to restructure binary search trees. A left rotation around
x transforms the tree on the left hand side in Figure 7.1 into the tree on the
right hand side. A right rotation around y transforms the tree on the right
hand side in Figure 7.1 into the tree on the left hand side. In this sense, left
and right rotations are inverse operations.
If the tree on the left hand side fulfills the binary search tree property,
then the tree T1 contains all nodes the key of which is smaller than Key(x).
The tree T2 contains all node with a key between Key(x) and Key(y). And
T3 has all nodes with a key greater than Key(y). But then the tree on the
right hand side fulfills the binary search tree property, too. The same is true
for the right rotation.

Lemma 7.3 Left and right rotations preserve the binary search tree prop-
erty.

Markus Bläser 2009–11

7.2. Restoring the AVL tree property 49

7.2.2 Insertion
When a node is inserted, it always becomes a leaf in the tree. But because
we added the virtual leaves, inserting a node means that a virtual leaf is
replaced by a virtual node with height 1 having two virtual leaves. So a tree
of height 0 is replaced by a tree of height 1. This means, that potentially
the AVL tree property is violated.

Observation 7.4 After inserting a node y, the AVL tree property can only
be violated at nodes on the path from y to the root.

Algorithm 31 restores the AVL property. y is the current node the height
of which is increased. If y is the left child of its parent x, then we have to
increase the balance of x by 1 because the height of the left subtree of x
increased. If y is the right child, then the balance of x is decreased by 1. If
the balance of x becomes 0 in this way, then we are done, because the height
of T (x) did not change.1

Algorithm 31 AVL-insert-repair
Input: AVL tree T , a node y inserted into T
Output: afterwards, the AVL property is restored
1: x := Parent(y)
2: while x 6= NULL do
3: if y = Left(x) then
4: Bal(x) := Bal(x) + 1
5: else
6: Bal(x) := Bal(x) − 1
7: if Bal(x) = 0 then
8: return
9: if Bal(x) = 2 or Bal(x) = −2 then
10: Restore the AVL property using a rotation or double rotation (see
Figure 7.1 and 7.2)
11: return
12: y := x; x := Parent(y)

If the balance of x became +1 or −1, then the height of T (x) was in-
creased by 1, so we need to inspect the parent of x in the next iteration of
the while loop. If the balance of x became +2 or −2, then the AVL tree
property is violated and we have to repair it. We only consider the case
when the balance of x is −2, the other case is completely symmetric. If the
balance of x became −2, then y = Right(x) and the balance of y is either
+1 or −1. We distinguish between these to cases:
1
The balance of x was +1 or −1 before the insertion, that is, the height of one of its
subtrees was one less than the height of the other. After the insertion, both of them have
the same height now. Since the node above x do not “see” this difference, we can stop.

Markus Bläser 2009–11

50 7. AVL trees

Bal(y) = −1 after the insertion: In this case, we perform a left rotation

as in Figure 7.1. The following table show all relevant parameters
before the insertion, after the insertion, and after the rotation. We set
h := Height(T1 ). The height of the other subtrees are then determined
by the balances of x and y. Because we assumed that Bal(y) = −1
after the insertion, the insertion took place in T3 .

before insertion after insertion after rotation

Bal(x) −1 −2 −1
Bal(y) 0 −1 0
Height(T1 ) h h h
Height(T2 ) h+1 h+1 h+1
Height(T3 ) h+1 h+2 h+2
Height(T (x)) h+3 h+4 h+2
Height(T (y)) h+2 h+3 h+3

After the rotation, all balances are in {+1, 0, −1}. Furthermore, since
Height(T (x)) before the insertion is the same as Height(T (y)) after the
insertion, we can stop, because there cannot be any further violations
above y.
Bal(y) = 1 after the insertion: In this case, we perform a left double rotation
as depicted in Figure 7.2. We set h := Height(T1 ). The height of the
other subtrees are then determined by the balances of x, y, and z.
Because we assumed that Bal(y) = 1 after the insertion, the insertion
took place in T (z):

before insertion after insertion after rotation

Bal(x) −1 −2 0 or +1
Bal(y) 0 1 −1 or 0
Bal(z) 0 +1 or −1 0
Height(T1 ) h h h
Height(T2 ) h−1 h − 1 or h h − 1 or h
Height(T3 ) h−1 h − 1 or h h − 1 or h
Height(T4 ) h h h
Height(T (x)) h+2 h+3 h+1
Height(T (y)) h+1 h+2 h+1
Height(T (z)) h h+1 h+2

After the double rotation, all balances are in {+1, 0, −1}. Furthermore,
the height of T (x) before the insertion is the same as the height of T (z)
after the insertion. Therefore, the procedure can stop.

In both cases, the balances are restored and moreover, we can leave the
procedure.

Markus Bläser 2009–11

7.2. Restoring the AVL tree property 51

x z

y x y

T1
z

T4 T1 T2 T3 T4

T2 T3

Figure 7.2: A left double rotation. You can think of a double left rotation as
a right rotation around y followed by a left rotation around x. This explains
the name double rotation. A double rotation also preserve the binary search
tree property.

7.2.3 Deletion
Deletion is performed similar to insertion. We use the delete method from
binary search trees and then move the path up to the root and try to restore
the AVL tree property
When we delete a node in a binary search tree, three cases can occur.
Either it is a leaf in the binary tree, that means, it is an internal node with
two virtual leaves in our AVL tree. Or it is an internal node with only one
child. In an AVL tree, this means that one of its children is a virtual leaf.2
Or it is an node note with two children that are internal nodes, too. Then
we delete its successor and copy the content of the successor to the node.
Let v be the node deleted. We assume that there are no references from
nodes in the tree to v but the Parent(v) still references to the former parent
of v in T . After deletion, we call the method AVL-delete-repair with v. The
procedure works in a similar way to AVL-insert-repair by going up the path
from v to the root. However, some things have to be changed: First of all,
the balance of x has to be adapted in the opposite way in lines 4 and 6.
Then, we can stop if the balance of x becomes −1 or 1. This means that the
balance was 0 before and the height of one of the subtrees was decreased.
But since this does not change the total height of the tree, then we stop.
If the balance becomes 2 or −2, then things get more complicated. As-
sume that Bal(x) = −2, the other case is treated similarly. In this case, v is
a left child of x. Let y be the right child of x. We distinguish three cases:
2
Note that by the AVL tree property, the other child then is an internal node with two
virtual leaves.

Markus Bläser 2009–11

52 7. AVL trees

Warning! We use the same figures as in the case of insertion. How-

ever, the node deleted is in left subtree of x. Moreover, y is not the currently
inspected by AVL-delete-repair. (This is v which is the root of T1 .)

Bal(y) = −1: We we use a left rotation like in Figure 7.1. Let h =

Height(T1 ).

before deletion after deletion after rotation

Bal(x) −1 −2 0
Bal(y) −1 −1 0
Height(T1 ) h h−1 h−1
Height(T2 ) h−1 h−1 h−1
Height(T3 ) h h h
Height(T (x)) h+2 h+2 h
Height(T (y)) h+1 h+1 h+1

Since the height of T (x) before the deletion is larger than the height
of T (y) after the rotation, we have to go on and move up the path to
the root.
Bal(y) = 0: Again, we do a left rotation. Let h := Height(T1 ).

before deletion after deletion after rotation

Bal(x) −1 −2 0
Bal(y) 0 0 1
Height(T1 ) h h−1 h−1
Height(T2 ) h h h
Height(T3 ) h h h
Height(T (x)) h+2 h+2 h+1
Height(T (y)) h+1 h+1 h+2

Since the height of T (x) before the deletion is the same as the height
of T (y) after the rotation, we can stop in this case.
Bal(y) = +1: In this case, we perform a left double rotation. Now we look
at at Figure 7.2. Let h := Height(T1 ).

before deletion after deletion after rotation

Bal(x) −1 −2 0 or +1
Bal(y) +1 +1 0 or −1
Bal(z) −1, 0, or +1 −1, 0, or +1 0
Height(T1 ) h h−1 h−1
Height(T4 ) h−1 h−1 h−1
Height(T (x)) h+2 h+2 h
Height(T (y)) h+1 h+1 h
Height(T (z)) h h h+1

Markus Bläser 2009–11

7.2. Restoring the AVL tree property 53

All balances are restored, but the since the height of T (x) before the
deletion is larger than the height of T (z) after the rotation, we have
to go up.

Algorithm 32 AVL-delete-repair
Input: AVL tree T , a node v deleted from T
Output: afterwards, the AVL property is restored
1: x := Parent(v)
2: while x 6= NULL do
3: if v = Left(x) then
4: Bal(x) := Bal(x) − 1
5: else
6: Bal(x) := Bal(x) + 1
7: if Bal(x) = 1 or Bal(x) = −1 then
8: return
9: if Bal(x) = 2 or Bal(x) = −2 then
10: Restore the AVL property using a rotation or double rotation
11: v := x; x := Parent(v)

Both repair procedures trace a path of length O(log n). Each rotation
takes time O(1), since we have to update a constant number of references.

Theorem 7.5 Searching, minimum and maximum computation, predecessor

and successor computation and insert and delete run in time O(log n) on a
AVL tree with n internal nodes.

Markus Bläser 2009–11

8 Binomial Heaps

Mergeable heaps are heaps that support a union operation. That is, given
heaps H and H 0 , we can merge them into one big heap that contains all
elements of H and H 0 (and, of course, satisfies the heap property). Take our
ordinary heaps from heap-sort. To implement the union operation, we can
just concatenate the two arrays and call the Heapify procedure. However,
this takes time O(n) which is too slow for large sets of data. In this chapter,
we will introduce binomial heaps which support all heap operations in time
O(log n) and, in addition, the union operation in time O(log n), too. In
particular, binomial heaps support the following operations:

Make-heap(): creates a new heap that is empty.

Minimum(H): returns a pointer/reference to an element in H that

has a minimum key (but leaves the element in H).

Extract-min(H): removes an element with the minimum key from

H and returns (a pointer/reference to) the node.

Insert(H, x): inserts x into H.

Delete(H, x): deletes x from H.

Union(H, H 0 ): creates a new heap that contains all elements from H

and H 0 .

Decrease-key(H, x, k): assigns the new key value k to the element x

provided that k is smaller than the current value.

The Decrease-key procedure is useful, if we want to implement mergable

priority queues.

8.1 Binomial trees

A binomial tree Bk of order k is an ordered tree that is defined recursively:
1. B0 is the tree with one node.

2. The tree Bk consists of two copies of Bk−1 . The root of one copy
becomes the the leftmost child of the root of the other copy.

54
8.1. Binomial trees 55

Note that Bk only “describes a structure”, later we will fill the nodes with
data.

Lemma 8.1 The following statements are true:

1. Bk has 2k nodes.

2. The height of Bk is k.

3. There are ki nodes at depth i, 0 ≤ i ≤ k.

4. The root has k children and all other nodes have strictly less.

Proof. The proof of each of the statements is by induction on k.

1. B0 has 1 = 20 nodes. Now assume that Bk−1 has 2k−1 nodes (induction
hypothesis). Bk consists of two copies of Bk−1 and has 2 · 2k−1 = 2k
nodes.

2. B0 has height 0. Assume that Bk−1 has height k − 1. The height of

Bk is the height of Bk−1 plus 1, hence it is k − 1 + 1 = k.

3. In B0 , there is exactly one node at depth

0. Now assume that in Bk−1 ,
there are exactly i nodes at depth k−1i . How many nodes are at depth
k−1


i in Bk ? There are i in the copy of Bk−1 that contains the root
of Bk and k−1


i−1 in the one that does not contain the root. Altogether,
these are      
k−1 k−1 k 1
+ = .
i i−1 i

4. In B0 , the root has no children and there are no further nodes. Assume
that in Bk−1 , the root has k − 1 children and all other nodes have less
children. The root in Bk has k children. All other nodes have less
children, since they belong to one copy of Bk−1 and no further edges
are added except the one joining the two copies.

Binary trees were easy to store. Every node had three fields containing
pointers/references to the left and right child and the parent. Binomial trees
have an unbounded number of children. We can of course keep a list of all
children at every node. More compact is the so-called left-child-right-sibling
representation: Every node has a pointer/reference to its parent, its left-most
child, and its sibling to the right. If the node is the root, then the parent
1
How does one prove this identity? Either you use brute force or you can do the
following: The right hand side is the number of possibilities to choose i out of k items.
Now mark one of the k items. To take i of the k items, you either can take the marked
one and i − 1 out of the k − 1 remaining items or you do not take the marked one but i
out of the k − 1 remaining ones. But this is precisely the sum on the left-hand side.

Markus Bläser 2009–11

56 8. Binomial Heaps

B0 B1 B2 B3

Figure 8.1: The binomial trees B0 , B1 , B2 , and B3

P
C S

P P P
C S C S C S

Figure 8.2: The left-child-right-sibling representation of a node in a tree with

three children. The three types of pointers are marked with P (parent), C
(child), and S (sibling).

pointer is NULL. If the node is a leaf, then the pointer to the left-most child
is NULL. If it is the last (right-most) child, then the pointer to the right
silbing is NULL. We will also store at each node x its number of children
and denote this quantitiy by Degree(x).

8.2 Binomial heaps

A binomial heap is a set of binomial trees. Every tree in this set has the
structure of a binomial tree but it contains additional data like the key and
a pointer/reference to satellite data. A binomial heap satisfies the binomial
heap property:

Markus Bläser 2009–11

8.3. Operations 57

Binomial heap property

1. Every binomial tree in the heap is heap-ordered : the key of the

parent is smaller than or equal to the key of the children.

2. The binomial heap contains at most one copy of Bk for every

k.

Technically, we store the trees in a linked list. It turns out to be useful

to store the trees in a list in ascending order with respect to their sizes (or
degrees of the roots, which gives the same order). To link the trees together
in the list, we can use the Silbling pointer/reference of the roots, since it is
not used otherwise. There is one distinguished pointer/reference Head(H)
which points to the first tree in the list.
The number n of elements in the binomial heap H P`uniquely determines
i
the orders of the binomial trees in the heap. If n = i=0 bi 2 is the binary
expansion of n, then H will contain the trees Bi with bi = 1, since the binary
expansion is unique. In particular, every binomial heap with n elements has
at most log n trees.

8.3 Operations
8.3.1 Make heap
To build a heap, we just initialize H as the empty list. The running time is
obviously O(1).

Algorithm 33 Make-BH
Input: binomial heap H
Output: empties H
Head(H) := NULL (Old-fashioned languages like C/C++ require that
you free the memory)

8.3.2 Minimum
By item 1 of the binomial heap property, the minimum is stored in one of
the roots of the trees. Therefore we only have to search the roots of the
trees. The next algorithm is a simple minimum selection algorithm on lists.
n runs through the list and in x and min, we store the current minimum and
its value. Since a binomial heap contains ≤ log n trees, the running time of
BH-Minimum is O(log n).

Markus Bläser 2009–11

58 8. Binomial Heaps

Algorithm 34 BH-Minimum
Input: binomial heap H
Output: the minimum element stored in H
n := Head(H)
x := n
if n 6= NULL then
min := Key(n)
n := Next(n)
while n 6= NULL do
if Key(n) < min then
min := Key(n)
x := n
n := Next(n)
return x

8.3.3 Union
The union routine is the heart of binomial heaps. All further operations are
somehow based on this routine. As a subroutine, we need a procedure that
takes two Bk−1 and produces a Bk . We move the references/pointers in the
obvious way.

Algorithm 35 BT-union
Input: the roots x and y of two copies of Bk−1
Output: the trees are linked together to form a Bk with root y
Parent(x) := y
Sibling(x) := Child(y)
Child(y) := x
Degree(y) := Degree(y) + 1

Remark 8.2 If Key(y) ≤ Key(x) and both trees above are heap-ordered,
then the resulting tree is heap-ordered, too.

To union two binomial heaps H1 and H2 , we first merged the two lists
of trees into one list. The trees are ordered with ascending degrees. By
the binomial heap property, every tree Bk occurs at most once in each Hi ,
thus it occurs at most twice in the merged list. We will go through the list
from left to right. Whenever we encounter two copies of Bk , we will merge
them into one Bk+1 by exploiting BT-union. In this way, we will ensure that
left to the current position, every Bk appears at most once and right to the
current position, every Bk appears at most twice. The only exception is the
current position: The Bk in this position may appear three times! How can
this happen when in the beginning, every Bk only appears at most twice?

Markus Bläser 2009–11

8.3. Operations 59

Well, we can have two Bk−1 followed by two Bk in the merged list. The two
Bk−1 will be merged to one Bk and then there are three Bk ’s in a row. We
leave the first one of them in the list and merge the other two to one Bk+1
in the next step.
There will be three pointers/references in our procedure, prev , cur , and
next. They point to the previous, current, and next tree in the merged list.
We distinguish three cases. The third one splits into two subcases.

Degree(cur ) 6= Degree(next): In this case, the degree are different. We do

not have to merge the current tree and can move on with the next tree.

Degree(cur ) = Degree(next) = Degree(Sibling(next)): In this case, we have

three trees of the same degree in a row and we are standing on the first
one. We go one step to the right and then will merge the next two
trees in the next step.

Degree(cur ) = Degree(next) 6= Degree(Sibling(next)): We merge the two

trees of the same degree to form a tree of larger degree. This case has
two subcases: If the first tree has the root with smaller key, then it
becomes the root of the new tree. Otherwise, the root of the second
tree, becomes the new root.

If case 2 is executed, then case 3 is executed in the next iteration of the

while-loop.

Lemma 8.3 The following invariant holds after executing cases 1 or 3:

1. The degree of all trees left to cur is < Degree(cur).

2. The degrees of all trees left to cur are distinct.

3. Right to cur, the degree of all trees is ≥ Degree(cur).

4. Right to cur, every degree appears at most twice, Degree(cur) might

appear three times.

Proof. The proof is by induction on the number of executions of the body

of the while loop. Before the first execution of the while loop, the invariant
is certainly fulfilled.
If case 1 is executed, then Degree(cur ) < Degree(next), so the invariant
is still valid after setting cur := next.
If case 3 is executed, then Degree(prev ) < Degree(cur ) = Degree(next)
or Degree(prev ) = Degree(cur ) = Degree(next). In the latter case, case 2
has been executed before. Now cur and next are merged into one tree of
bigger degree. After this, Degree(prev ) < Degree(cur ). It might now be the
case that Degree(cur ) = Degree(next) = Degree(Sibling(next)). But right

Markus Bläser 2009–11

60 8. Binomial Heaps

Algorithm 36 BH-union
Input: binomial heaps H1 , H2
Output: H1 and H2 merged
H := Make-BH()
Merge the two lists of binomial trees of H1 and H2 into one list with
ascending degrees.
prev := NULL; cur := Head(H); next := Sibling(cur );
while next 6= NULL do
if (Degree(cur ) 6= Degree(next)) or (Sibling(next) 6= NULL) and
(Degree(Sibling(next)) = Degree(cur )) then /* cases 1,2 */
prev := cur
cur := next
else
if Key(cur ) ≤ Key(next) then /* case 3a */
Sibling(cur ) := Sibling(next)
BT-union(next, cur )
else /* case 3b */
if prev = NULL then
Head(H) := next
else
Sibling(prev ) := next
BT-union(cur , next)
cur := next
next := Sibling(cur )

Markus Bläser 2009–11

8.3. Operations 61

to this, every degree occurs at most twice. Hence the invariant is still fulfilled
after setting cur := next.
Thus, BH-union is correct. It runs in time O(log n) where n is the number
of elements in the output heap, since the heaps only have ≤ log n trees and
we only spend O(1) time in the while loop per tree.

8.3.4 Insert
With the union operation, inserting an element x into a binomial heap H is
very easy: We pack x into a binomial heap and then merge this heap with
H. The overall running time is obviously O(log n).

Algorithm 37 BH-insert
Input: binomial heap H, element x
Output: inserts x into H
H1 := Make-BH()
Parent(x) := Child(x) := Sibling(x) := NULL
Degree(x) := 0
Head(H1 ) := x
H := BH-union(H, H1 )

8.3.5 Extract-min
We saw already that an element with a minimum key can only be a root of a
tree. But if we extract the minimum, then we destroy the binomial tree. The
next lemma states that the remaining parts of the tree still form a binomial
heap. Thus we can merge the resulting two heaps.

Lemma 8.4 The tree Bk is obtained by attaching Bk−1 , . . . , B0 to a single

node (the root) as children from left to right.

Proof. The proof is by induction on k.

Induction base: If k = 0, then there is nothing to prove.
Induction step: Assume that the statement is true for Bk−1 . We obtain
Bk by taking two copies of Bk−1 and making the root of one of them the
left-most child of the other. By the induction hypothesis, we can replace the
Bk−1 the root of which became the root of Bk by a root with Bk−2 , . . . , B0
attached to it. It follows that Bk is a node with Bk−1 , Bk−2 , . . . , B0 attached
to it.
The algorithm is written in pseudopseudocode. The running time is
dominated by the BH-union call. Thus it is O(log n).

Markus Bläser 2009–11

62 8. Binomial Heaps

Algorithm 38 BH-extract-min
Input: binomial heap H
Output: returns an element with minimum key and removes it from H
Use the Minimum procedure to find a root x with minimum key.
Remove the corresponding tree from H.
H1 := Make-BH()
H1 := list of subtress of x in reverse order
H := BH-union(H, H1 )
return x

8.3.6 Delete
Exercise 8.1 Implement the Decrease-key procedure.

With the Decrease-key operation, deleting given elements is easy. We

set its key to −∞ and just call Extract-minimum. −∞ is a value that is
smaller than any other key. Either one has to artifically increase the range
of the key values. Or one just merges the code of Decrease-key and Extract-
minimum explicitly. In the latter case, we do not need the value −∞. We
just introduced it to produce shorter code.

Algorithm 39 BH-delete
Input: binomial heap H, element x in H
Output: x is removed from H
BH-decrease-key(H, x, −∞)
BH-extract-min(H)

Markus Bläser 2009–11

9 Fibonacci heaps

Fibonacci heaps provide the same operations as binomial heaps, but with
a better running time for most of the operations. For some of the opera-
tions, this is only achieved on the average in a sense made precise in the
next section. Fibonacci heaps achieve this improvement by being lazy; bi-
nomal heaps ensure that after every operation, the binomial heap property
is fulfilled. Fibonacci heaps, on the other hand, tidy up only once after a
while.1

9.1 Amortized analysis

Consider the following problem: Given an array a[0..`−1] which we interpret
as an ` bit binary counter, what is the running time of increasing the counter
by 1? The algorithm below performs this increase. Each entry a[i] either
holds the value 0 or 1 and a[0] is the least significant bit. Of course, in the
worst case, every bit is set to 1 and we have to flip all n bits (and get an
overflow error). But on the average, these costs are much lower. The average
costs are also called the amortized costs.

Algorithm 40 Increasing a binary counter

Input: a binary counter a[0..` − 1]
Output: a is increased by 1
1: i := 0
2: while a[i] = 1 and i ≤ ` − 1 do
3: a[i] := 0
4: i := i + 1
5: if i ≤ ` − 1 then
6: a[i] := 1
7: else
8: output “overflow”

9.1.1 The aggregate method

How can we determine the amortized costs? When using the aggregate
method, we show that for all n, every sequence of n operations takes ≤ t(n)
1
As opposed to real life, being lazy sometimes pays off for data structures.

63
64 9. Fibonacci heaps

time. The amortized costs per operation is therefore t(n) n . Here t(n) is the
number of times we increase an entry of a. We assume that n ≤ 2` .
Assume our counter a is initialized with 0 in all entries. The least signif-
icant bit a[0] is changed every time we perform an increase. a[1], however,
is only changed every second time, namely, when a[0] = 1. a[2] is changed
only every fourth time, when a[0] = a[1] = 1. In general, a[i] is changed only
every 2i th time. Therefore, the total time is
n n
X n X 1
t(n) = b ic ≤ n · b i c ≤ 2n
2 2
i=0 i=0

and the amortized costs are

t(n)
≤ 2.
n
You saw this method already when analyzing the running time of Build-
heap for binary heaps. Now you know a fancy name for this, too.

9.1.2 The potential method

The potential method is an advanced method to estimate the amortized costs
of an operation. For the binary counter example, it is overkill, but it will be
useful for analyzing Fibonacci heaps.
With every “state” of the counter, we associate a potential. (Do not
confuse this with the potential function used when analyzing comparison
trees!) Let ci be the true costs of the ith operation and Φi be the potential
after the ith operation, then the amortized costs are defined as

ai = ci + Φi − Φi−1

(Think of this as the “true costs” plus the “potential difference”.) Then the
sum of the amortized costs is
n
X n
X n
X
ai = (ci + Φi − Φi−1 ) = Φn − Φ0 + ci ,
i=1 i=1 i=1

in other words,
n
X n
X
ci = ai + Φ 0 − Φn .
i=1 i=1

If we design the potential function in such a way that Φ0 − Φn ≤ 0, then

the sum of the ai bounds the total costs (the sum of the ci ). Often, we will
choose the potential function in such a way that it is always nonnegative and
Φ0 = 0 Then Φ0 − Φn ≤ 0 trivially holds.
It is up to you to design a good potential function. In the case of a binary
counter, the potential Φi of the counter is the number of ones in it. Let ti

Markus Bläser 2009–11

9.2. Fibonacci heaps 65

be the number of ones in the counter after the ith increase. We set Φi = ti .
Then Φ0 = 0 and Φi ≥ 0 for all i. We have

Φi − Φi−1 ≤ −ci + 2

because if the ith operations takes ci steps, then ci − 1 ones are replaced by
zeros and one zero is replaced by a one. Thus

ai = ci + Φi − Φi−1 ≤ 2.

With the potential method, it is also very easy to bound the costs of n
increase operations when we start in any counter state, it is simply

2n + Φ0 − Φn ≤ 2n + Φ0 .

Since Φ0 ≤ `, the total costs are O(n) as soon as n = Ω(`).

Warning! We do not have to implement the potential function, we
only need it for the analysis! There is no need to keep a variable potential
in Algorithm 40 and update it every time!

9.2 Fibonacci heaps

Fibonacci heaps are a collection of unordered trees. For the moment it is
okay to think of them as binomial trees with the only difference that the
tree is unordered. The trees are heap-ordered. We extend the left-child-
right-sibling representation: The children of a node is not stored in a singly
linked linear list but in a doubly linked circular list. (You could call this
“some-child-left-sibling-right-sibling” representation.) Also the roots of the
trees in the Fibonacci heap are stored in a circular list. We do, however, not
require that every type of tree appears at most once in the tree. We will be
lazy and try to “clean up” the heap as few times as possible. There will be
a pointer min to one of the roots in the circular list and it will always point
to the root with the minimum key.
With this structure, it is easy (that means in O(1) time) to perform the
following operations:

Link: Given the roots of two trees, we combine them into one tree making
the root with the smaller key the root of the new tree and the other
root one of its children. Note that the new tree is heap-ordered if both
input trees are. See Figure 9.1 for an illustration.

Unlink: is the opposite of Link. Given a the root r of a tree and one of the
children x of the root, we remove the subtree with root x from the tree
with root r.

Markus Bläser 2009–11

66 9. Fibonacci heaps

Figure 9.1: The link procedure. The link operation takes two trees and makes
the the root with the larger key (on the righthand side) the child of the other
root (on the left hand side). Note that we did not draw the children of the
root on the righthand side. The unlink procedure is the opposite operation.
It removes a particular subtree from the doubly linked list.

Figure 9.2: The splice procedure.

Splice: It takes two circular lists of trees and combines them into one cir-
cular list of trees. See Figure 9.2.

Furthermore, nodes can be marked in a Fibonacci heap. A node is marked

if it has already lost one of its children. The potential of a Fibonacci heap
consisting of t trees with m marked nodes is

t + 2m.

We will analyze the amortized cost of an arbitrary sequence of operations.

Φi = ti + 2mi denotes the potential after the ith operation, where ti is the
number of roots after the ith operation and mi is the number of marked
nodes. If several heaps are involved in the operation, the potential is the
sum of the individual potentials.
More precisely, the potential function will be some constant time the
potential defined above to compensate for constant factors in the running
times.

9.3 Operations
We first implement the procedures Insert, Union, Minimum, and Extract-
min. If we only perform these operations, then the trees in the Fibonacci
heaps will always be (unordered) binomial trees and no nodes will be marked.
Nevertheless, we will explicitly write the term mi in the potential, because
everything is also valid if there are marked nodes.

Markus Bläser 2009–11

9.3. Operations 67

9.3.1 Minimum
The procedure Minimum is easy to implement, we just have to return min.
The actual costs for this are O(1) and there is no change in the potential.
Therefore the amortized costs are O(1), too.

9.3.2 Union
The procedure Union takes the two heaps and splices them together to one
heaps as in Figure 9.2. Then we choose min as the minimum of the two
minima. That’s it! We do not tidy everything up as we did for binomial
heaps. We even do not sort the sizes of the trees. The actual costs are O(1).
There is no change in potential, since the number of trees in the union is the
sum of the number of heaps in the input heaps. Therefore, the amortized
costs are O(1), too.

9.3.3 Insert
To insert an element, we create a heap of size one (using Make-heap, which
we frankly did not implement so far) and then use union. This takes O(1)
time. The new root has one child more, so the potential goes up by one.
The amortized costs are O(1 + 1) = O(1).

9.3.4 Extract-min
Extract-min is the first interesting procedure. We can find the minimum
using min. We unlink the corresponding tree T from the heap F . When we
remove the root of T , we get a ring of trees, which is nothing else than a
Fibonacci heap. If we splice this ring together with F . The following step
is the only time when we tidy up: Whenever we find two trees in the heap
the root of which has the same degree, we link them. Note that if the two
trees are unordered binomial trees, then we will get an unordered binomial
tree again. So after each linking, we still have a ring of unordered binomial
trees.

Algorithm 41 FH-Extract-min
Input: Fibonacci heap F
Output: the minimum is extracted and returned
1: Unlink the tree T with root min from F
2: Remove the root of T to obtain a circular list of trees.
Splice this list with F .
3: As long as there are two roots with the same degree in F , we link them
together.
4: Recompute min and return the old minimum.

Markus Bläser 2009–11

68 9. Fibonacci heaps

How fast can we implement this scheme? Steps 1 ans 2 take time O(1).
It is very easy to implement step 3 by first sorting the trees by size (recall
that a Fibonacci heap is lazy and does not store the trees sorted) and then
proceed in a way similar to the union procedure for binomial heaps. But
Step 3 can be even implemented in time linear in the number of roots after
splicing: We have an array D. D[i] stores a reference/pointer to a root with
degree i if we have found one so far. For each root r, we run the following
program:
1: i := Degree[r]
2: while D[i] 6= NULL do
3: r0 := D[i]
4: D[i] := NULL
5: r := Link(r, r0 )
6: i := i + 1
7: D[i] := r
If D[i] = NULL in the beginning, then we simply set D[i] := r. Oth-
erwise, we link r and D[i]. Then we get a tree with degree i + 1. If it is
the first one of this degree, then we set D[i + 1] := r, otherwise we repeat
the process. Since every root it put into D exactly once, the running time is
linear in the number of roots.
Let d(n) be an upper bound for the degree of any node in any Fibonacci
heap on n nodes. Since every tree currently is an unordered binomial heap,
d(n) ≤ log n. Before we extract the minimum, we have ti−1 roots. After
splicing, we have ≤ ti−1 + d(n) many roots. After linking all roots of the
same degree, we have ti roots. Since all roots have different degrees then,
ti ≤ d(n). The total cost of FH-Extract-min is

O(ti−1 + d(n)).

The potential difference is

ti + 2mi − ti−1 − 2mi−1 = ti − ti−1 ≤ d(n) − ti−1 .

Note that mi = mi−1 . By scaling the potential function appropriately (to

compensate for the constant in the O-notation), we get that the amortized
costs are proportional to ti−1 + d(n) + d(n) − ti−1 = 2d(n) and thus they are

O(d(n)) ≤ O(log n).2

Proposition 9.1 For a Fibonacci heap, the operations Minimum, Union,

and Insert take constant time, even in the worst case. The operation Extract-
min takes amortized time O(log n).
2
We will later show that even with deletions and delete-min operations, d(n) ∈ O(log n).
Therefore, this bound remains valid.

Markus Bläser 2009–11

9.4. More operations 69

9.4 More operations

The Decrease-key and the Delete operation work in a similar way. They both
use a subroutine called Cascading-cut.

9.4.1 Cascading cuts

Algorithm 42 Cascading-cut
Input: Fibonacci heap F , node x
1: if x is not marked then
2: Mark(x)
3: return
4: y := Parent(x).
5: if y 6= NULL then
6: Unmark(x)
7: Unlink T (x)
8: Link T (x) to the cycle of roots of F
9: Update min
10: Cascading-cut(F, y)

Cascading cut is called when a node lost one of its children. Cascading-
cut gets a Fibonacci heap F and a node x. If x is not marked (that is, it did
not loose a child so far), then x is simply marked. If x is marked, then we
remove the tree T (x) from its current tree and link it to the cycle of roots
and x is unmarked. Then we proceed recursively with the parent of x (which
now lost one of its children, namely x) until we reach a root.

9.4.2 Decrease-key
Let x be the node, the key of which shall be decreased. We remove the
subtree T (x) and link it to the cycle of roots. Since x is now a root, we
can decrease its key without violating the heap property. Maybe we need to
update min. Since the former parent of x lost a child, we call Cascading-cut.
Let ci be the number of cuts performed. This includes the initial cut of
T (x) and all cuts performed by Cascading-cut. The costs of Decrease-key
is proportional to ci . After calling Decrease-key, the number of trees in the
cycle of roots increases by ci , there is one new tree for every cut. Therefore

ti = ti−1 + ci .

Furthermore, at most one node is newly marked and ci − 1 nodes are un-
marked. Hence,

mi = mi−1 − (ci − 1) + 1 = mi − ci + 2.

Markus Bläser 2009–11

70 9. Fibonacci heaps

Algorithm 43 FH-decrease-key
Input: Fibonacci heap F , node x, key k with k ≤ Key(x)
Output: Key(x) is set to k
1: y = Parent(x)
2: Key(x) := k
3: if y 6= NULL then
4: Unlink the tree T (x)
5: Link T (v) to the cycle of roots
6: Update min
7: if y 6= NULL then
8: Cascading-cut(F, y)

The potential difference is

Φi − Φi−1 = ti − ti−1 + 2(mi − mi−1 ) = −ci + 4.

Therefore, the amortized costs are proportional to

ai = ci + Φi − Φi−1 = 4

and thus they are constant. (Because we choose the factor 2 in the term
2mi , we get a constant here!)

9.4.3 Delete
Delete is very similar to Decrease-key. Let x be the node to be deleted. We
first remove T (x) as before. Instead of adding T (x) to the cycle of roots,
we first remove x and splice the cycle of the children of x with the cycle of
roots of F . We update min. Since the former parent of x lost a child, we
call Cascading-cut.
Let ci be the number of cuts performed. This includes the initial cut
of T (x) and all cuts performed by Cascading-cut. The costs of Delete is
proportional to d(n) + ci . d(n) for recomputing min 3 , and ci for performing
the cuts. After calling Delete, the number of trees in the cycle of roots
increases by d(n) + ci , there is one new tree for every cut. Therefore

ti ≤ ti−1 + d(n) + ci .

Furthermore, at most one node is marked and ci − 1 nodes are unmarked.

Hence,
mi = mi−1 − (ci − 1) + 1.
3
We add at most d(n) new roots to the cycle. They all have to be compared with
the current minimum. However, this does not work if the node x that we delete is the
current minimum! In this case, we use Extract-min instead. I overlooked this special case
in a previous version of these lecture notes. There is a simpler (= shorter Pseudocode)
implementation of FH-delete: Decrease the key of x to −∞ and then use extract min!

Markus Bläser 2009–11

9.5. A bound on the number of leaves 71

Algorithm 44 FH-delete
Input: Fibonacci heap F , node x
Output: x is deleted
1: if x = min then
2: Fh-extract-min(x)
3: else
4: y = Parent(x)
5: Unlink the tree T (x)
6: Remove x
7: Splice the cycle of children of x with the cycle of roots of F
8: Update min
9: if y 6= NULL then
10: Cascading-cut(F, y)

The potential difference is

Φi − Φi−1 = ti − ti−1 + 2(mi − mi−1 ) = d(n) − ci + 4.
Therefore, the amortized costs are proportional to
ai = d(n) + ci + Φi − Φi−1 = 2d(n) + 4.
We will prove that d(n) ∈ O(log n) in the next section. Therefore, deletion
has amortized costs O(log n).
Our analysis of the four other operations are still valid if we allow the
operations Decrease-key and Delete. too. Therefore, we obtain the final
result.

Theorem 9.2 For a Fibonacci heap, the operations Minimum, Union, In-
sert, and Decrease-key take constant time, the first three even in the worst
case. The operations Extract-min and Delete take amortized time O(log n).

9.5 A bound on the number of leaves

Lemma 9.3 Let x be a node in a Fibonacci heap with Degree[x] = d. Let
y1 , . . . , y` be the children of x in order in which they were attached to x.
Then Degree[y1 ] ≥ 0 and Degree[yi ] ≥ i − 2 for all i ≥ 2.

Proof. Let i ≥ 2. When yi was linked to x, then x had y1 , . . . , yi−1 as

children and hence, Degree[x] ≥ i − 1. Two nodes are only linked together,
if they have the same degree. Therefore, Degree[yi ] ≥ i − 1 at that time.
Since yi still is a child of x, it has lost at most one child so far. Therefore,
Degree[yi ] ≥ i − 2.
Let F be a Fibonacci heap. Let sk denote the minimum size of any
subtree of F having a root of degree k.

Markus Bläser 2009–11

72 9. Fibonacci heaps

Lemma 9.4 For all k ≥ 2, sk ≥ Fk+2 .

Proof. The proof is by induction on k.

Induction base: We have s0 = 1, s1 = 2, and s2 = 3.
Induction step: Let k ≥ 3. By Lemma 9.3 and the induction hypthesis,
k
X k
X k
X
sk ≥ s0 + si−2 ≥ 2 + Fi = 1 + Fi = Fk+2 .
i=2 i=2 i=0

Pk
Exercise 9.1 Prove that Fk+2 = 1 + i=0 Fi for all k. (Hint: Induction on
k)

Corollary 9.5 d(n) ∈ O(log n) for all Fibonacci heaps with n nodes

Proof. For every node of degree k, we have Fk+2 ≤ n. As we have seen

before, k ∈ O(log n).

Markus Bläser 2009–11

10 Master theorem

We solved recurrence relations twice, when analyzing merge sort and the
linear time selection algorithm. Everything we learned there is essentially
sufficient to show the following general purpose theorem. It is usually called
“master theorem” (but should be called “slave theorem” since it encourages
you to turn your brain off).
You should know the theorem and should be able to apply it. The proof
was not presented in the lecture and is not relevant for the exam.

Theorem 10.1 (Master theorem) Let a ≥ 1, b > 1, c, and n0 be natural

numbers, let f : N → N be a function and let t be defined by
(
a · t(dn/be) + f (n) if n > n0
t(n) =
c if n ≤ n0 .

Then the following holds:

1. If f (n) = O(nlogb a− ) for some  > 0, then t(n) = O(nlogb (a) ).

2. If f (n) = Θ(nlogb a ), then t(n) = O(nlogb (a) log n)

3. If f (n) = Ω(nlogb a+ ) and a f (dn/be) ≤ df (n) for some constant d < 1
and all sufficiently large n, then t(n) = O(f (n)).

Proof overview: Consider the recursion tree. At the root, we have work
f (n) to do. The root has a children. At every children, we have work roughly
f (n/b), given a total work of af (n/b). In general, at recursion depth i, the
total work amounts to ai f (n/bi ).
The function f (n) = nlogb a fulfills nlogb a = a(n/b)logb a . For this function,
the total work at every recursion level is the same, namely nlogb a . There are
logb n levels yielding a total amount of nlogb a logb n.
If f grows (polynomially) slower than nlogb a , then the work at thes leaves
dominates, since the work increases geometrically. There are alogb n = nlogb a
leaves. The work at each leaf is constant.
If f grows (polynomially) faster than nlogb a , then the work at the root
dominates. It is f (n).

Exercise 10.1 Show that if f fulfills f (dn/be) ≤ df (n) for some constant
d < 1 and all sufficiently large n, then f (n) = Ω(nlogb a+ )

73
74 10. Master theorem

Lemma 10.2 Let f, f 0 : N → N and let t and t0 be the solutions of the

recurrence in the master theorem for f and f 0 , respectively.
1. Assume that f (n) ≤ f 0 (n) for all n. Then t(n) ≤ t0 (n) for all n.

2. If f is monotone, so is t.

Exercise 10.2 Proof the lemma above (Hint: standard induction on n).

Let B = {bi | i ∈ N} be the set of all powers of b. If n ∈ B, then

dn/be = n/b ∈ B, too. Therefore, the restriction of t|B to B is also defined
by the recurrence of the master theorem.
For the first two cases of the master theorem, our strategy will be to
solve the recurrence only for t|B and f |B . The next lemma shows that this
solutions also bounds t on N.

Lemma 10.3 If t is monotone and t(n) ≤ γ · ne for all n ∈ B and constants

γ and e, then t ∈ O(ne ).

Proof. Let n ∈ N. Let i = dlogb ne. Since t is monotone, t(n) ≤ t(bi ).

Therefore,
t(n) ≤ γbe·i ≤ γbe logb n+e = γbe · ne .
Since γbe is a constant, we are done.
Proof of Theorem 10.1. We start with the first two cases. Let e :=
logb a, respectively. By Lemmas 10.2 and 10.3, it is sufficient if we show that
t|B (n) ≤ γ · ne (first case) and t|B (n) ≤ γ · ne log n.
By dividing by γ · ne , the recurrence relation becomes
t(n) t(n/b) f (n)
e
≤a· e
+
γ·n γ·n γ · ne
t(n)
Let t̂(n) = γ·ne.

We have  n e  n e
γne = γbe =a
b b
In the first case, the recurrence becomes

t̂(n) ≤ t̂(n/b) + α · n−

for some constant α. By Exercise 10.3, t̂(n) = O(1). Therefore, t(n) =

O(nlogb a ).
In the second case, the recurrence becomes

t̂(n) ≤ t̂(n/b) + 1

Using a proof similar to the one of Lemma 1.13, we get t̂(n) = O(log n).
Hence, t(n) = O(nlogb a log n).

Markus Bläser 2009–11

 75

In the third case, we do not restrict ourselves to B. We divide the

recurrence by f (n) and get

t(n) t(dn/be)
≤a· +1
f (n) f (n)
t(n)
Let t̂(n) = f (n) . By assumption, f (n) ≤ a·f (dn/be)/d. Thus, the recurrence
becomes
t̂(n) = d · t̂(dn/be) + 1.
As in the proof of Lemma 4.5, we get t̂(n) = O(1). Thus t(n) = O(f (n)).

Exercise 10.3 Show that if s(n) = s(n/b) + n− for all n ∈ B \ {1} and
s(1) = 1, then s(n) = O(1). (Hint: Show by induction on n, that s(n) =
Plog n−1 i
n i=0b b .)

Markus Bläser 2009–11

11 Some algorithm design tech-
niques

In this chapter, we will learn two algorithm design techniques that are well-
suited for optimisation problem with a certain “substructure property”.

11.1 Dynamic programming

Let G = (V, E) be an undirected graph. Let n = |V | and m = |E|. A node
coloring is a mapping c : V → N+ . If im c ⊆ {1, . . . , k}, then c is called a
k-coloring. A coloring c of a graph G = (V, E) is called proper if c(u) 6= c(v)
for all {u, v} ∈ E. The chromatic number χ(G) of a graph G is the smallest
number k such that there is a k-coloring of G.

Exercise 11.1 Let G = (V, E) be a graph with |V | = n. Prove the following:

1. 1 ≤ χ(G) ≤ n
2. χ(G) = 1 if and only if E = ∅
3. χ(G) = 2 if and only if G is bipartite.
(A graph is bipartite if we can partition V = L ∪ R with L ∩ R = ∅
such that for every edge {u, v} ∈ E, u ∈ L and v ∈ R or vice versa.)

Given a graph G = (V, E) and an integer k as an input, how can we

decide whether G has a proper k-coloring? First, we can try all proper k-
colorings. We can bound their number by k n . The total running time of
the naive algorithm is O(k n · m), since for each coloring, we have to check
whether it is proper or not. Backtracking is an alternative, however, it is
not clear how to bound the running time.
For a set U ⊆ V , the induced subgraph G[U ] = (U, E[U ]), where E[U ]
is the set of all edges {u, v} ∈ E with u, v ∈ U . We call a set U ⊆ V
independent if G[U ] has no edges.
Dynamic programming computes the chromatic numbers of induced sub-
graphs of G, starting with the subgraphs of size one and then increasing the
size of the subgraphs step by step. We set
C(G[{v}]) = 1 for all v ∈ V (11.1)
For any set U we define
C(G[U ]) = min{C(G[U \ I]) + 1 | ∅ ( I ( U is an independent set} (11.2)

76
11.1. Dynamic programming 77

These two equations immediately yield an algorithm: Run over all sets U ,
starting with the sets of size 1, then of size 2, and so on. For each set U with
|U | > 1 run over all subsets I and check whether it is independent. If yes,
then check whether C(G[U \ I]) + 1 is smaller than the smallest number of
colors we found so far.

Lemma 11.1 For all U , χ(G[U ]) = C(G[U ]).

Proof. The proof is by induction on |U |.

Induction base: If |U | = 1, then this is obvious.
Induction step: Let |U | > 1. Let c be an optimum proper coloring for G[U ],
that is, a coloring with χ(G[U ]) =: k colors. Every color class forms an
independent set, in particular c−1 (k) =: I is independent. c restricted to
U \ I is a (k − 1)-coloring and therefore, χ(G[U \ I]) ≤ χ(G[U ]) − 1. By the
induction hypothesis, C(G[U \ I]) = χ(G[U \ I]). We have

C(G[U ]) ≤ C(G[U \ I]) + 1 ≤ χ(G[U ])

Since there is also a coloring with C(G[U ]) colors by construction, C(G[U ]) =

χ(G[U ]).
The running time is proportional to
n  
X n i
2 · m = m · 3n ,
i
i=0

which is independent of k!1 ni is the number of sets U of size i and 2i is

the number of subsets I of a set U of size i. (We have included the empty
set to get a nicer formula for the running time.)

11.1.1 Finding the sequence of operations

The algorithm above only computes the number of colors needed. What can
we do if we want to compute a corresponding coloring? C(G[U ]) is computed
by looking at all C(G[U \ I]). If we store a set IU for which the minimum
is attained for every C(G[U ]), then we can easily construct a coloring. We
start at C(G[V ]). We give the set IV color 1. Then we look at C(G[V \ IV ]).
The set stored there gets color 2 and so on.

11.1.2 When does dynamic programming help?

Dynamic programming helps whenever we can divide our problems (here
computing C(G[U ])) into subproblems (here C(G[U \ I])) From the optimal
1
There are even faster algorithms, for instance, one can show that there are at most
n/3
3 independent sets in any graph.

Markus Bläser 2009–11

78 11. Some algorithm design techniques

solutions of the subproblems, we must be able to reconstruct an optimal

solution of our original problem.
In order to be efficient, we also need that the subproblems “overlap”. In
this way, the overall work is dramatically reduced.
The term “programming” in “dynamic programming” does not mean de-
signing a program or writing some code, it means solving a problem by using
a tableaux.

11.1.3 Memoization
The dynamic programming approach is bottom-up. If we want to go top-
down by making recursive calls, we can use a technique called memoization.
We have a global array C(G[U ]). Whenever we compute C(G[U ]) during
a recursive call, we store the result in C(G[U ]). Before making a recursive
call, we look in C(G[U ]) whether this value has already been computed. This
avoids unnecessary recursive calls. This essentially fills the table T top-down
instead of bottom-up.

11.2 Greedy algorithms

11.2.1 Interval scheduling and interval graph coloring
Consider the following problem: We have a set of intervals Ii = [si , fi ),
1 ≤ i ≤ n. In the interval scheduling problem, we want to find as many
intervals as possible that are pairwise disjoint. You can think of each Ii
being a lecture with starting time si and finishing time fi . You want to
attend as many lectures as possible. 2
A related problem is the following: Instead of finding a maximum set of
non-intersecting lectures, we want to assign the lectures to lecture halls in
such a way that at most one lecture takes place in a lecture hall at any point
in time. Of course, we want to use the least number of lecture halls. The
interval graph coloring problem formalizes this. Given a set V of intervals
[si , fi ), 1 ≤ i ≤ n, the corresponding intervall graph G = (V, E) is obtained
as follows: There is an edge between two intervals I and J iff I ∩ J 6= ∅.
(This means that the corresponding activities cannot take place in the same
location, since their time slots intersect.) A proper coloring of an interval
graphs is a solution to our scheduling problem: Each color corresponds to a
lecture hall. All lectures with the same color can be scheduled to the same
location, since by the definition of proper coloring, they are not connected
by an edge and hence their time slots do not intersect. To minimize the
2
The intervalls are closed on the one side and are open on the other to reflect the fact
that a lecture that goes from 2 to 4 and a lecture from 4 to 6 are usually assumed to be
compatible.

Markus Bläser 2009–11

11.2. Greedy algorithms 79

number of locations, we have to seek for a proper coloring with a minimum

number of locations.

11.2.2 The greedy method

The interval scheduling and graph coloring problem can be solved by a so-
called greedy algorithm. Greedy algorithms are greedy in a literal sense,
they always make the choice that looks best to them at the moment. In
the interval scheduling problem, we will always take the interval with the
smallest finishing time that does not intersect with the intervals already
chosen, that is, occupies the least amount of time.

Algorithm 45 Greedy-scheduling
Input: a set of intervals with starting and finishing times si ≤ fi , 1 ≤ i ≤ n
Output: a subset T of non-intersecting intervals that is as large as possible
1: T := ∅
2: while there is an interval that does not intersect any interval in T do
3: Choose the non-intersecting interval I that has the smallest finishing
time.
4: T := T ∪ {I}
5: return T

If we sort S in advance with respect to finishing times, then we only

look at each activity once and we can achieve a running time of O(n log n)
(including sorting).
The next lemma shows that the greedy approach here works.
Lemma 11.2 Let Iji be the interval that is selected in the ith iteration of
the while loop, 1 ≤ i ≤ t. If there is a set Si = {j1 , . . . , ji , h1 , . . . , h` } such
that Ip ∩ Iq = ∅ of all p, q ∈ Si , p 6= q, then after a suitable permutation of
the h1 , . . . , h` , Ŝi = {j1 , . . . , ji , ji+1 , h2 , . . . , h` } is also a set of indices such
that Ip ∩ Iq = ∅ for all p, q ∈ Ŝi , p 6= q.
Proof. Since Iji+1 was chosen by Greedy-schedule in the (i+1)th iteration,
we have fji+1 ≤ fhs , 1 ≤ s ≤ `. Each Ihs does not intersect with any Ijr ,
1 ≤ r ≤ i. Hence if fji+1 > fhs , then Greedy-schedule would have chosen
Ihs instead. Iji+1 can have a nonempty intersection with at most one Ihs .
Otherwise two of the Ih1 , . . . , Ih` would intersect, which cannot be true by
assumption. Let w.l.o.g. Ih1 be this interval. Since Iji+1 does not intersect
with any of the other intervals, also not with Ijr , since Iji+1 was chosen by
Greedy-schedule. Therefore, Ŝi is a set of indices such that the corresponding
intervals do not intersect.

Theorem 11.3 Greedy schedule is correct, that is, if there is a schedule S

with k intervals, then Greedy-schedule will find a schedule with at least k
intervals.

Markus Bläser 2009–11

80 11. Some algorithm design techniques

Proof. Let j1 , . . . , jm be the intervals chosen by Greedy-schedule and let

S = {q1 , . . . , qk }. By applying Lemma 11.2 repeatedly and swapping in j1 ,
j2 , and so on, we get that S 0 = {j1 , . . . , jm , qk−m+1 , . . . , qk } is a schedule
with k intervals, too. Can m < k? No, because S 0 is a schedule and because
fji ≤ fqk−m+1 by the greedy choice, greedy would also have taken Iqk−m+1 ,
which it has not. Therefore, m = k.
Next we come to the interval graph coloring problem. For the ease of
presentation, we assume that the starting and the finishing times are natural
numbers between 0 and N . Let
C := max #{i | x ∈ Ii }.
0≤x≤N

C is the maximal number of intervals that overlap in a single point. All these
intervals are connected with an edge in the interval graph, therefore, each
of these intervals needs a different color. Therefore, every proper coloring of
an interval graph needs at least C colors.
On the other hand, there is always a coloring with C colors and we can
find it by a greedy algorithm. We consider the intervals by ascending starting
time and always give the next interval a color that does not create a conflict.

Algorithm 46 Greedy-coloring
Input: an interval graph G = V, E
Output: a proper coloring of the nodes in V
1: sort the intervals by starting time.
2: initially, all intervals are uncolored
3: for j = 1, . . . n do
4: Let F be the colors of intervals from I1 , . . . , Ij−1 that intersect Ij .
5: Choose a color from {1, . . . , C} \ F and color Ij with this color.

Since we sort the intervals by starting time, it is also very easy to keep
track of the intervals that intersect with Ij . In this way, we get an algorithm
with O(n log n) running time.
The algorithm will never color intervals that are connected by an edge
with the same color. It might happen that an interval remains uncolored,
since there are no colors left in {1, . . . , C} \ F . Can this happen? Since
the intervals are sorted by starting time, all the intervals that intersect Ij
intersect in the point sj . By the definition of C, #F < C. Therefore, Ij is
properly colored.

Theorem 11.4 Greedy-coloring is correct.

11.2.3 When does the greedy approach work?

Greedy algorithms are quite similar to dynamic programming. We reduce
our problem to the solution of a subproblem. But in dynamic programming,

Markus Bläser 2009–11

11.2. Greedy algorithms 81

we reduce the solution of our problem to the solution of several problems

and we do not know which of them will give the best solution to our original
problem. So we have to solve them all. If greedy algorithms works, it can
reduce the solution to the original problem to the solution of one problem.
So usually it runs much faster, typically in linear time.

Markus Bläser 2009–11

12 Graph algorithms

A(n undirected) graph G is a pair (V, E), where V is a finite set, the set
of nodes or vertices, and E is a set of two-element subsets (= unordered
pairs) of V , the so-called edges. If e = {u, v} ∈ E, we say that u and v are
connected by the edge e or that u and v are adjacent. The edge e is called
incident on u and v.
In a directed graph G = (V, E), edges are ordered pairs, i.e., E ⊆ V × V .
We say that an edge e = (u, v) leaves u and enters v or is incident from u
and incident to v. e is called an edge from u to v. An edge of the form (v, v)
is called a self-loop. We can also allow self loops in undirected graphs, they
are sets of size one. Note that in a directed graph, there can be two edges
between two nodes, one from u to v and the other one from v to u.
A sequence v0 , v1 , . . . , vk is called a walk from v0 to vk in G = (V, E) if
{vi , vi+1 } ∈ E for all 0 ≤ i < k. (In the case of a directed graph, (vi , vi+1 ) ∈
E, that is, all edges in the path have to point “in the same direction”.) A
walk is called a (simple) path if all nodes in the path are distinct. k is called
the length of the path. (Some books use path as a synomym for walk and
simple path for path.)

Exercise 12.1 Show that if two vertices are connected by a walk, then they
are connected by a path.

A cycle is a walk such that v0 = vk , k > 0 (if G is directed) or k > 1 (if

G is undirected), and the vertices v0 , . . . , vk−1 are pairwise distinct. (Some
books call this a simple cycle.) A graph is called acyclic if it contains no
cycle.
A node is u is reachable from v if there is a path from v to u. If G is
undirected, then the “is reachable from” relation is symmetric.

Exercise 12.2 Show that in undirected graph, the “is reachable from” rela-
tion is an equivalence relation.

The connected components of an undirected graph are the equivalence

classes of the “is reachable from” relation. Two nodes u and v are in the
same component if and only if there is a path between u and v. A graph is
called connected if there is only one equivalence class.
If G is directed, then the “is reachable from” relation is no longer an
equivalence relation. But the relation “u is reachable from v and v is reach-
able from u” is an equivalence relation. (Prove this!) The equivalence classes

82
12.1. Data structures for representing graphs 83

of this relation are the strongly connected components. A directed graph is

strongly connected if there is only one equivalence class. It is called weakly
connected if the underlying undirected graph (the graph that we get when
we replace every directed edge (u, v) by {u, v} and possible removing double
edges) is connected.
A tree is an acyclic connected undirected graph. A forest is an acyclic
undirected graph. A graph G0 = (V 0 , E 0 ) is a subgraph of G = (V, E) if
V 0 ⊆ V and E 0 ⊆ E. A tree that is a subgraph of G and that contains all
nodes of G is called a spanning tree.

12.1 Data structures for representing graphs

How can we store a graph? It is very convenient to represent the nodes
in V as numbers 1, 2, . . . , n where n = |V |. There are two standard ways
of storing the edges of a graph, the adjacency-list representation and the
adjacency-matrix representation.
In the adjacency-list representation, for every node i, there is a list Li . If
there is an edge e = {i, j} or e = (i, j), we add the element j to Li . In other
words, if there is an edge from i to j, then we store the vertex j in the list
Li . In the case of an undirected graph, we either store the edge {i, j} twice,
namely in Li and Lj , or we make the convention that an edge is only stored
in the list Li , if i < j, and in Lj , if j < i, respectively.
In the adjacency-matrix representation, we store the edges in a matrix
(or two-dimensional array) A:
(
1 if {i, j} or (i, j) ∈ E, respectively
A[i, j] =
0 otherwise

If G is undirected, then A is symmetric.

First of all, the adjacency-list representation needs O(|V | + |E|) cells of
memory, so the space consumption is asymptotically optimal. (Note that
|E| can be much smaller than |V | if the graph is not connected. Therefore,
we cannot just write O(|E|).) The adjacency-matrix representation always
needs O(|V |2 ) cells of memory. If the graph has relatively few edges (so
called sparse graphs), then the adjacency-list representation is preferably
with respect to space consumption.
If we want to find out whether there is an edge between two nodes (i, j),
we can answer this question in O(1) time with a adjacency-matrix represen-
tation. With a adjacency-matrix-representation, however, we have to scan
the whole list Li (or Lj ) in the worst case, so the time is O(di ) where di is
the degree of i, that is, the number of edges that are incident with i.
In short, if one can afford it, the adjacency-matrix representation is ad-
vantageous because of its simplicity. However, often we are confronted with

Markus Bläser 2009–11

84 12. Graph algorithms

huge sparse graphs (think of web pages as nodes and links as edges). Here
adjacency-matrices are way too large.
Both representation can be easily extended to weighted graphs. A weighted
graph G = (V, E, w) has in addition a function w : E → Q that assigns each
edge a weight. In the adjacency-list representation, we simply store the
corresponding weight w(i, j) together with j in the list. In the adjacency-
matrix, we just store the weight w(i, j) instead of 1. If an edge does not
exist, we can, depending on the application, either store 0 or ∞ or a special
value indicating that there is no edge.

12.2 Breadth first search

Breath first search and depth first search are two important way to explore a
graph and they are the basis of many other graph algorithms. We are given
a graph G = (V, E) and a node s ∈ V , the start node or source. We want
to explore the graph in the sense that we want to discover all nodes that are
reachable from s by a path.
We start in s. The nodes that we first can discover from s are the nodes
v such that there is an edge (s, v) ∈ E, the so-called neighbours of s.1 Then
we can discover the neighbours of the neighbours of s and so on until we
discover all nodes reachable from s.
Breadth first search first explores all nodes that are close to s. Nodes
have one of three possible states: undiscovered, discovered, finished. In the
beginning, s is discovered and all other nodes are undiscovered. A node
becomes discovered if we encounter it for the first time. Then its state
changes to discovered. If all neighbours of a node have been discovered, its
state finally changes to finished.
Breadth first search first marks every node v as undiscovered , the distance
d[v] to s is set to ∞, and the predecessor p[v] is set to NULL. Since the only
node that we know so far is s, we set state[s] = discovered and d[s] := 0.
We initalize a queue Q with s, Q will contain all discovered nodes that are
not finished. As long as Q is not empty, we remove the next node v from it.
All its neighbours that are have not been discovered are marked discovered
and are put into the Q. v becomes the predecessors of these nodes and their
distance is set to d[v] + 1. Since all of its neighbours are discovered, v is
marked finished .

1
Breadth first search works for undirected and directed graphs. We only present it for
directed graphs, since every undirected graph can be represented by a directed graph by
replacing each undirected edge {u, v} be two directed edges (u, v) and (v, u).

Markus Bläser 2009–11

12.2. Breadth first search 85

Algorithm 47 Breadth first search

Input: graph G = (V, E), source s ∈ V
Output: every node that is reachable from s is marked as finished
p[v] contains the node from which v was discovered
d[v] contains the distance from s to v
1: for each v ∈ V do
2: state[v] := undiscovered
3: p[v] := NULL
4: d[v] := ∞
5: state[s] := discovered
6: d[s] := 0
7: let Q be an empty queue
8: Enqueue(Q, s)
9: while Q is not empty do
10: v := Dequeue(Q)
11: for each node u adjacent to v do
12: if state[u] = undiscovered then
13: state[u] := discovered
14: d[u] := d[v] + 1
15: p[u] := v
16: Enqueue(Q, u)
17: state[v] := finished

Markus Bläser 2009–11

86 12. Graph algorithms

12.2.1 Correctness
Let G = (V, E) be a graph. The shortest path distance δ(x, y) of two nodes
x, y ∈ V is

δ(x, y) = the length of a shortest path from x to y.

If there is no path from x to y, then δ(x, y) = ∞.

Exercise 12.3 Show that δ fulfills the triangle inequality, i.e,

δ(x, z) ≤ δ(x, y) + δ(y, z)

for all x, y, z ∈ V .

Lemma 12.1 Let G = (V, E) be a graph and s ∈ V . Then

δ(s, y) ≤ δ(s, x) + 1

for all (x, y) ∈ E.

Proof. If there is a path p from s to x, then there is also one from s to y,

namely, p extended by y, Therefore δ(s, y) ≤ δ(s, x) + 1. If there is no path
from s to x, then δ(s, x) = ∞ and the inequality is trivially fulfilled.

Let Ui = {v ∈ V | δ(s, v) = i} for i ∈ N be the set of all vertices that

have distance i from s. We have U0 = {s}.

Lemma 12.2 If BFS is executed on G = (V, E), then all vertices of Ui are
inserted into the queue before all vertices in Ui+1 and after all vertices in
Ui−1 , if i > 0.

Proof. The proof is by induction on i.

Induction base: : Clear, since U0 = {s} and s is the first vertex in the queue
and every vertex enters the queue at most once.
Induction step: : For the induction step, we assume that the statement is
true for all 0 ≤ j ≤ i. By the induction hypothesis (applied repeatedly), first
all vertices of U0 enter the queue, then all of U1 and finally all of Ui .
Now new nodes are put into the queue if a node is removed from the
queue. If a node v is removed from the queue, all its undiscovered neighbours
are put into the queue. If v is in Uj , then all its neighbours are in U0 ∪ U1 ∪
· · · ∪ Uj+1 . Neighbours in U0 ∪ · · · ∪ Uj−1 were put before v in the queue,
therefore they are already finished and not put into the queue. Neighbours
in Uj are also neighbours of some node u ∈ Uj−1 . Hence they were already
put into the queue by u or some other node in Uj−1 , since all nodes in Uj−1
were removed from the queue before v. Therefore, the only nodes that can
be put into the queue by v are its neighbours in Uj+1 .

Markus Bläser 2009–11

12.2. Breadth first search 87

Now we apply the argument above for j = i. Nodes from Ui+1 can only
be put into the queue when we remove vertices from Ui . When we remove
the first vertex of Ui , only vertices from Ui are in the queue and we start
putting the vertices from Ui+1 into the queue. Since every vertex in Ui+1 has
a neighbour in Ui , all vertices of Ui+1 are in the queue after the last vertex
of Ui is removed.

Theorem 12.3 BFS works correct, that is, after termination, d[v] = δ(s, v)
for all v ∈ V . Moreover, for every node v with δ(s, v) < ∞, a shortest path
from s to p[v] together with the edge (p[v], v) is a shortest path from s to v.

Note that d[v] = δ(s, v) is sufficient to prove the correctness, since a node
v is marked as finished iff d[v] < ∞ and if a node is marked as finished, then
p[v] contains the node v was discovered from.
Proof. If v is not reachable from s, then v is never put into the queue,
hence d[v] = ∞.
We now show by induction on i that d[v] = i for all v ∈ Ui .
Induction base: : This is true by construction for U0 .
Induction step: : Assume that d[v] = i for all v ∈ Ui . All nodes u in Ui+1
are put into the queue by a node in Ui by Lemma 12.2. This means that
d[u] = i + 1. This completes the induction step.
Finally note that for every node v ∈ Ui+1 , p[v] ∈ Ui . Therefore a shortest
path to p[v] has length i. If we extend this path by (p[v], v), then we get a
path of length i + 1, which must be a shortest path.

Corollary 12.4 Let V 0 = {v ∈ V | p[v] 6= NULL}∪{s} and E 0 = {(p[v], v) |

v ∈ V 0 \ {s}}. Then G0 = (V 0 , E 0 ) is a tree such that every path from s to v
is a shortest path.

The proof of the corollary follows from repeatedly using the “moreover”-
part of Theorem 12.3. Such a tree G0 is called a shortest path tree.

12.2.2 Running time analysis

The initialization phase (lines 1–6) is linear in the number of nodes. Since
we only put undiscovered nodes into the Q and mark them as finished when
we remove them, every node is put at most once into the queue. Since every
node is queued only once, we look at every edge also at most once. If we
have a adjacency-list representation, then the running time of breadth first
search is O(|V | + |E|). In other words, the running time is linear in the size
of the graph. If we have an adjacency-list representation, then the running
time is O(|V |2 ).

Markus Bläser 2009–11

88 12. Graph algorithms

12.3 Depth first search

Depth first search also explores all nodes that are reachable from a given
node s. But instead of storing the discovered node in a queue, it uses a
stack (or recursion). Everything else is the same, our vertices can have one
of three states undiscovered , discovered , and finished . And for every node
v, we store the node p[v] from which it was discovered.
Since nodes will not be discovered along shortest paths, we will not store
the distances to s. Instead, there is a global variable t which is used as a
clock. It is increased by one whenever a new vertex is discovered or one is
finished. Every vertex gets two time stamps, the discovery time d[v] and
the finishing time f [v]. These time stamps can be quite useful, see e.g. the
exercises.
Algorithm 48 Depth-first-search
Input: graph G = (V, E), node x
uses a global variable t as clock
Output: marks all nodes reachable from x as finished
stores for each such node the discovery time d[v] and finishing time f [v]
1: state[x] := discovered ;
2: t := t + 1; d[x] := t
3: for each y adjacent to x do
4: if state[y] = undiscovered then
5: p[y] = x
6: Depth-first-search(y)
7: state[x] := finished
8: t := t + 1; f [x] := t;

Before we call Depth-first-search(G, s), we have to initialize the arrays

state and p. Since we look at every vertex and every edge at most one, the
running time is O(|V | + |E|).
To explore all nodes in a graph, for example, when we want to compute
connected components, we can pack the depth-first-search procedure into a
for loop. The procedure DFS-explore does not only explore what is reachable
from a single node, but when it explored all reachable nodes, it starts with
another undiscovered vertex. (We could have done the same with BFS.) In
this way, we do not only get a depth first search tree but a depth first search
forest. The trees in these forests are essentially the recursion trees.

Theorem 12.5 After execution of DFS-explore, for any two vertices x 6= y,

the intervals [d[x], f [x]] and [d[y], f [y]] are either disjoint or one is contained
in the other.

Proof. W.l.o.g. assume that d[x] < d[y]. If f [x] < d[y] then the intervals
are disjoint and we are done.

Markus Bläser 2009–11

12.3. Depth first search 89

Algorithm 49 DFS-explore
Input: a graph G
Output: every node is visited
start and finishing times are computed
1: for each v ∈ V do
2: state[v] := undiscovered
3: p[v] := NULL
4: t := 0
5: for each x ∈ V do
6: if state[x] = undiscovered then
7: Depth-first-search(x)

If d[y] < f [x], then x’s state is discovered when y is discovered. Therefore
x is a predessor in the recursion tree of y. Therefore f [x] < f [y].

Markus Bläser 2009–11

13 Minimum spanning trees

Let G = (V, E, w) be a connected weighted undirected graph. A tree T that

is a subgraph of G and contains every node of G is called a spanning tree.
Let ET be the edges of T . The weight of T is defined as
X
w(T ) = w(e).
e∈ET

A spanning tree is a minimum spanning tree of G if its weight is minimal

among all spanning trees of G.
If we think of the vertices as towns and the cost on the edges as the
costs of building a road between the corresponding towns, then a minimum
spanning tree is the cheapest way of building streets such that any pair of
towns is connected. (We do not allow any junctions outside the towns. If
this is allowed, then we get so-called Steiner trees. But this is a more difficult
problem.)
A cut of G is a partition of the vertices into two nonempty sets (S, V \S).
An edge e = {u, v} crosses the cut, if u ∈ S and v ∈ V \ S or vice versa. A
cut respects a set E 0 of edges if no edge in E 0 crosses the cut.
The following theorem will be useful, when analysing algorithms for min-
imum spanning trees.

Theorem 13.1 Let F be a set of edges that is contained in a minimum

spanning tree of G = (V, E, w). Let (S, V \ S) be a cut that respects F .
Let e be an edge of minimum weight crossing the cut. Then F ∪ {e} is also
contained in a minimum spanning tree.

Proof. Let T = (V, ET ) be a minimum spanning tree that contains F .

Let e = {u, v} an edge of minimum weight crossing the cut (S, V \ S). If T
contains e, too, then we are done.
Otherwise, the graph (V, ET ∪ {e}) contains exactly one cycle consisting
of the edge e and a path P from u to v in T . Since u and v are on opposite
sides of the cut, there is a least one edge f = {x, y} in P crossing the cut.
Because (V, ET ∪ {e}) contains only one cycle, T 0 = (V, ET ∪ {e} \ {f }) is
again acyclic. It is still connected, since the nodes x and y are connected by
the remaining edges of P together with e. Therefore, T 0 is a spanning tree.
By the choice of e, w(e) ≤ w(f ). Thus

w(T 0 ) = w(T ) − w(f ) + w(e) ≤ w(T ).

90
13.1. Data structures for (disjoint) sets 91

Since T is a minimum spanning tree, w(T 0 ) = w(T ). Because F respects the

/ F . By construction, T 0 contains F ∪ {e}. Hence, T 0 is a minimum
cut, f ∈
spanning tree, we are looking for.

13.1 Data structures for (disjoint) sets

For one of the algorithms that we will design in this chapter, we need a
data structure to maintain so-called disjoint dynamic sets. We have a finite
universe U of n elements and we want to store a collection of disjoint sub-
sets S1 , . . . , S` ⊆ U . Each of the sets Si is represented by a distinguished
element in it, the so-called representative. Since the sets are disjoint, the
representative is unique. The following operations shall be supported:

Make-set(x): Creates a new set that contains x (and no other element). x

is not allowed to appear in any other set that we created so far.

Union(x, y): Computes the union of the two sets that contain x and y,
respectively. We assume that x and y are in different sets. The old
sets are destroyed.

Find(x): Finds the representative of the unique set that x belongs to. If x
is not in any set, then NULL is returned.

One easy implementation is to use linked lists. Each set is stored in a

linked list, the first element in the list is the representative. Make-set can be
implemented with constant running time and also Union needs only constant
running time since we just have to append the lists. (We store an additional
pointer to the last element.) However, finding the representative needs Θ(n)
time in the worst case since we have to go from element to element until
we reach the head of the list. To speed up the Find procedure, we can add
to each element of the list a pointer to the first element. Then we can find
the representative in time O(1). But now the union needs time Θ(n) in the
worst case, since we might have to redirect Θ(n) pointers.
A very good compromise are binomial heaps. Recall that binomial heaps
are a lists of binomial trees. Make-set (called Make-BH) and Union are
already implemented and have running times O(1) and O(log n). As the
representative element, we can take the minimum of the heap, which can be
found in time O(log n) (cf. Aufgabe 6.3).

13.2 Kruskal’s algorithm

Krukal’s algorithm is a greedy algorithm. In each step it chooses the lightest
edge that does not create a cycle. We start with n trees, each consisting of

Markus Bläser 2009–11

92 13. Minimum spanning trees

Algorithm 50 Kruskal-MST
Input: connected weighted undirected graph G = (V, E, w)
Output: a minimum spanning tree T of G
1: Sort the edges by increasing weight
2: for each vertex v ∈ V do
3: Make-set(v)
4: ET := ∅
5: for each edge e = {u, v} ∈ E, in order by increasing weight do
6: if Find(u) 6= Find(v) then
7: ET := ET ∪ {e}
8: Union(u, v)
9: return (V, ET )

a single vertex. Then trees are joined until in the end, we have a single tree.
We use a disjoint set data structure to store the intermediate forests.

Theorem 13.2 Kruskal’s algorithm returns a minimum spanning tree.

Proof. We prove by induction on the number of edges that ET is a subset

of some minimum spanning tree. Since in the end, (V, ET ) is connected, we
obtained a minimum spanning tree.
Induction base: In the beginning, ET is empty.
Induction step: Now assume that |ET | > 0. By the induction hypothesis,
ET is contained in some minimum spanning tree. If |ET | = n − 1, we are
done. Otherwise, let e = {u, v} be the edge that is chosen next. Let S be
the set of all nodes in the set of u. The set S respects ET by construction
and e crosses the cut (S, V \ S). By Theorem 13.1, there is a spanning tree
that contains ET ∪ {e}.
Sorting the edges takes time O(|E| log |E|). Initialitzation of the forest
takes time O(|V |). In the second for-loop, we iterate over all edges. For each
edge, we perform a constant number of disjoint set operations, each taking
time O(log |V |). Therefore, the total running time is O(|E| log |E|). (Note
that |E| ≥ |V | − 1, since G is connected.)

13.3 Prim’s algorithm

Prim’s algorithm also works in a greedy fashion, too. But it does not grow
several components, it just extends one component by successively adding
new nodes. (Prim’s algorithm was invented, as the name suggests, by R.C.
Prim. But it was also invented earlier by V. Jarnik.)
We store all vertices in a priority queue. As long as a vertex y is not
adjacent to a vertex of the spanning tree selected so far, its key is set to ∞.

Markus Bläser 2009–11

13.3. Prim’s algorithm 93

Algorithm 51 Prim-MST
Input: connected weighted undirected graph G = (V, E, w)
Output: a minimum spanning tree T of G
1: Choose an arbitrary root r ∈ V .
2: Key[r] := 0; Key[v] := ∞ for all v ∈ V \ {r}.
3: Let Q be a min-priority queue filled with the vertices in V .
4: p[r] := NULL
5: while Q is not empty do
6: x := Extract-min(Q)
7: if p[x] 6= NULL then
8: ET := ET ∪ {p[x], x}
9: for each vertex y adjacent to x do
10: if y ∈ Q and w({x, y}) < Key[y] then
11: p[y] := x
12: Key[y] := w({x, y})
13: return (V, ET )

If it becomes adjacent to some vertex x, then its key becomes the weight of
w({x, y}). If it becomes a adjacent to a new vertex, then we only set the key
to the new weight if this will decrease the key. (In particular, we can always
use the Decrease-key procedure for this.) In this way, Key[x] is the cost of
adding x to the tree grown so far. In p[y] we always store the vertex x such
that Key[y] = w({x, y}).
What do we achieve by this? Well, Prim’s algorithm is supposed to work
as follows: We always choose the minimum weight edge such that extends the
current tree, that is, joins a new node to the tree. If x := Extract-min(Q),
then the edge {p[x], x} will be precisely this edge. By using a priority queue,
we get a efficient implemenation of this process. The only exception is in
the beginning, when we extract the root r from the queue. In this case, we
do not add an edge to the tree.
Why is Prim’s algorithm correct? The edge chosen is always a minimum
weight edge that crosses the cut consisting of the vertices chosen so far
and the remaining vertices in the Q. Now, the same proof as the one of
Theorem 13.2 works.
Assume that we implement the priority queue using binary heaps. The
initialization needs time O(|V |). The outer for-loop is executed |V | times.
An Extract-min operation needs time O(log |V |), so the cost of all Extract-
min operations is O(|V | log |V |). The inner for-loop is executed |E| times
altogether. The test y ∈ Q can be implemented in constant time by using
a Boolean array. The Decrease-Key operation has running time O(log |V |).
Thus the overall running time is O(|V | log |V | + |E| log |V |) ⊆ O(|E| log |V |).
Note that log |V | ∈ Θ(log |E|), so this is not an improvement over Kruskal’s
algorithm (at least not theoretically). However, if we use Fibonacci heaps to

Markus Bläser 2009–11

94 13. Minimum spanning trees

implement the queue, we can perform a Decrease-key operation in amortized

time O(1). This means that all |E| Decrease-key operations need running
time O(|E|) altogether. This gives a total running time of O(|V | log |V |+|E|).

Markus Bläser 2009–11

14 Shortest paths

Let G = (V, E, w) be an edge-weighted directed graph and let s, t ∈ V . Let

P = (v0 , . . . , v` ) be a path from s to t, that is, v0 = s, v` = t, and for all
0 ≤ λ <P`, (vλ , vλ+1 ) ∈ E. The (shortest path) weight of such a path is
w(P ) = `−1 1
λ=0 w((vλ , vλ+1 )). How can we compute the shortest path from
s to t? (This is an important problem if your are a nagivation system or a
travel planer.)
First there is the question whether the shortest walk from s to t is always
a path. This is certainly the case if all weights are nonnegative. What if
there are negative weights? Consider a walk from s to t that is not a (simple)
path. Then this walk contains a cycle C. If the weight of C is nonnegative,
then we can remove this cycle and get a new walk whose weight is equal
to (if w(C) = 0) or shorter than (if w(C) > 0) the weight of the original
walk. If the weight of C is negative, then the walk gets shorter if we go
through C once more. In this case, there is no shortest walk from s to t.
In summary, if on every walk from s to t there is no negative cycle, then
the weight δ(s, t) of shortest walk from s to t is well-defined and is always
attained by a path (which has at most |V | − 1 edges). If there is a walk with
a negative cycle from s to t, then we set δ(s, t) = −∞. Luckily, in many
practical applications, nonnegative weights do not appear at all.

Exercise 14.1 Show that if (v0 , v1 , . . . , v` ) is a shortest path from v0 to v` ,

then (vi , vi+1 , . . . , vj ) is a shortest path from vi to vj .

Exercise 14.2 Show that for any three nodes u, v, and x, δ(u, v) ≤ δ(u, x)+
w((x, v)).

14.1 Relaxing an edge

All algorithms that we know for computing a shortest path from s to t, also
compute a shortest path from s to v for every other node v ∈ V . This seems
wasteful at a first glance, but if you think about it, there could be a path
with many edges which nevertheless has small weight. And to be sure that
there is no such path you have to look at all nodes. (In pratical applications,
it is important to find stop-criteria and heuristics which edges to consider
first. If you want to go from Saarbrücken to Homburg, you usually only
1
“shortest” means a path with minimum weight in this chapter and not minimum
number of edges.

95
96 14. Shortest paths

have to consider edges to cities that are geographically near to Saarbrücken

to Homburg.) Such a type of problem is called single source shortest path,
we want to compute the shortest path from a given start node (source) to
all other nodes. In the all-pair shortest path problem, we want to compute
a shortest path between every pair of vertices.
Our algorithms will be quite similar to breadth-first-search. We have an
array d and an array p. d[v] will always be an upper bound for δ(s, v). In
the beginning, we initialize d[v] := ∞ for v 6= s and d[s] := 0. p[v] will be a
predecessor of v, in particular, we will always have

d[v] ≤ d[p[v]] + w((p[v], v)). (14.1)

provided that p[v] 6= NULL. We initialize p[v] := NULL for all v.

The following procedure Relax2 tests, whether we can improve the cur-
rent estimate d[u]:

Algorithm 52 Relax
Input: nodes u and v
Output: d[v] and p[v] are updated
if d[v] > d[u] + w((u, v)) then
d[v] := d[u] + w((u, v))
p[v] := u

If we initialize d and p as above and then perform any number of calls to

Relax, then the following statements are true:

1. For every edge (u, v) ∈ E, d[v] ≤ d[u] + w((u, v)),

2. For every v ∈ V , δ(s, v) ≤ d[v],

3. The values d[v] decrease monotonically (over time),

4. If d[u] = δ(s, u) and there is a shortest path from s to v such that

the edge (u, v) is on this path, then d[v] = δ(s, v) after the call of
Relax(u, v).

All the statements are true in the beginning. The first one can be easily
proven by induction. A call of Relax(u, v) only changes d[v]. When d[v] is
changed, then d[v] ≤ d[u] + w((u, v)) now holds. When d[v] is not changed,
then d[v] ≤ d[u] + w((u, v)) was true even before.
For the second statement, assume that the call of Relax(u, v) is the first
call such that d[v] < δ(s, v) for some node v. Then

d[u] + w((u, v)) = d[v] < δ(s, v) ≤ δ(s, u) + w((u, v)).

2
The term Relax is used mostly for historical reasons. We rather tighten the upper
bound.

Markus Bläser 2009–11

14.2. Dijkstra’s algorithm 97

The last inequality follows from Exercise 14.2. Therefore, d[u] < δ(s, u)
which contradicts the assumption that Relax(u, v) was the first call with
d[v] < δ(s, v).
The third statement is clear by the property of Relax.
For the fourth statement, note that after executing Relax(u, v), then d[v]
is δ(s, u) + w((u, v)) which is the weight of a shortest path from s to v.

14.2 Dijkstra’s algorithm

Dijkstra’s algorithm assumes that all weights are nonnegative. It is maybe
the most popular single source shortest path algorithm.

Algorithm 53 Dijkstra
Input: edge-weighted directed graph G = (V, E, w), w(e) ≥ 0 for all e,
source s ∈ V
Output: d[v] = δ(s, v) for all v and p encodes a shortest path tree
1: Initialize d[s] = 0, d[v] = ∞ for all v 6= s, and p[v] = NULL for all v.
2: Let Q be a min-priority queue filled with all vertices from V using d[v]
as keys.
3: while Q is not empty do
4: x := Extract-min(Q)
5: for each y with (x, y) ∈ E do
6: Relax(x, y)

Note that the Relax procedure now involves a Decrease-min call. Before
we come to the correctness, let us first analyze the running time. If we imple-
ment Q by an ordinary array, then the Insert and Decrease-min operations
that time O(1) while Extract-min takes O(|V |). Every node is inserted and
extracted once. For every edge, we have one Decrease-key (= Relax) opera-
tion. Thus, the total running time is O(|V |2 + |E|) which is linear in the size
of the graph if the graph is dense, that is, |E| = Θ(|V |2 ). If we implement Q
with binary heaps, then every operation takes time O(log |V |) giving a total
running time of O(|V | log |V | + |E| log |V |). And finally, we could also use
Fibonacci heaps; then we get a running time of O(|V | log |V | + |E|).
The correctness of Dijkstra’s algorithm follows immediately from the fol-
lowing lemma.

Lemma 14.1 For all v that are extracted from Q, d[v] = δ(s, v).

Proof. The first node that is extracted is s. We have d[s] = 0 = δ(s, s).
Now let u be the first node that will be extracted such that d[u] 6= δ(s, u).
We have u 6= s. By property 2 in Section 14.1, d[u] > δ(s, u).

Markus Bläser 2009–11

98 14. Shortest paths

Consider a shortest path P from s to u. Since s is not in the queue and

u is (still) in the queue, there must be an edge (x, y) on the path such that
x is not in the queue but y is. (x can be s and y can be u.)
Since x is removed from the queue before u, we have d[x] = δ(s, x). Since
Relax(x, y) was called, when x was removed, we have
d[y] ≤ d[x] + w((x, y)) = δ(s, x) + w((x, y)) = δ(s, y).
The second equality follows from the fact that the subpath from s to y of P
is a shortest path. Thus d[y] = δ(s, y) by property 2.
Since the weights are nonnegative, δ(s, y) ≤ δ(s, u). Therefore,
d[y] = δ(s, y) ≤ δ(s, u) < d[u].
But u is removed before y from Q, therefore d[u] ≤ d[y], a contradiction.

14.3 Bellman-Ford algorithm

The Bellman-Ford algorithm also works with negative weights. If it detects
an negative cycle, it will return an error message.

Algorithm 54 Bellman-Ford
Input: edge-weighted directed graph G = (V, E, w), source s ∈ V
Output: d[v] = δ(s, v) for all v and p encodes a shortest path tree
1: Intialize d and p
2: for i := 1, . . . , |V | − 1 do
3: for each e = (u, v) ∈ E do
4: Relax(u, v)
5: for each (u, v) ∈ E do do
6: if d[v] > d[u] + w((u, v)) then
7: error “negative cycle”

The running time of the Bellman-Ford algorithm is obviously O(|V |·|E|).

Why is it correct? Assume that there is no path with a negative cycle from
s to v. Consider a shortest path P = (v0 , . . . , v` ) from s to v (s = v0 and
v = v` ). Before the first iteration of the outer for-loop, δ(s, v0 ) = d[v0 ].
After the first iteration of the outer loop, the edge (v0 , v1 ) is relaxed in the
inner loop, since every edge is relaxed there. By property 4, δ(s, v1 ) = d[v1 ].
After the second iteration, (v1 , v2 ) is relaxed and δ(s, v2 ) = d[v2 ]. Since P
has at most |V | − 1 edges, δ(s, v` ) = d[v` ] holds in the end.
Intuitively speaking, if there is a negative cycle, then we can relax along
this cycle an infinite number of times (|V | is sufficient) and lower the values
d[v] every time.
Exercise 14.3 Formally prove that if there is a negative cycle in G, then
the Bellman-Ford algorithm will detect this.

Markus Bläser 2009–11

15 Hash tables

Often we only want to support the three basic operations insert, search, and
delete. This is called a dynamic set or dictionary problem.

Insert(S, x): adds x to S

Search(S, k): finds an element with key k

Delete(S, x): removes x from S.

All these operations should run extremely fast, that is, O(1). And they
should be even fast in pratice.

15.1 Direct-address tables

The simplest way to store the elements is direct addressing. Our keys come
from a universe U = {0, 1, . . . , m − 1}. We assume that m is not too large
and that no two elements have the same key. We store the elements in a
direct address table, that is, an array T [0..m − 1]. We have
(
Pointer/Reference to x if x ∈ S
T [Key(x)] =
NULL otherwise

So we store (a pointer/reference to) an element x in T [Key(x)]. In this way,

it is very easy to realize the three operations above in constant time. But
there is a problem: If the number of possible keys is much larger than the
size of a typical set S that we want to store, then we are wasting a lot of
space. Consider for instance a phone book. Our keys are strings of, say,
length 15. So there are 2615 possible keys! But only a very tiny fraction of
them corresponds to real names.

15.2 Hash tables

If our universe U of keys is large, say #U = N , but we want to store a
relatively small number of elements, say n, then direct addressing performs
poorly. Instead, we use a so-called hash function

h : U → {0, . . . , m − 1}.

99
100 15. Hash tables

m usually will be of roughly the same size as n. Instead of storing an element

x in T [Key(x)], like in direct addressing, we store it in T [h(Key(x))]. In this
way, an array T [0..m − 1] is sufficient. Since m is comparable to n, we do
not waste too much space. T is called a hash table.
Does this solve our problem? The answer is no (or not yet), since there
might be keys k 6= k 0 with h(k) = h(k 0 ), so two elements would be stored in
the same entry of T . In fact, this is unavoidable if N > m by the pigeonhole
principle. Even stronger, there is an index i ∈ {0, . . . , m − 1} such that
h(k) = i holds for at least dN/me keys k.
One possible solution to this problem is chaining. Instead of storing a
pointer/reference to an element in T [i], we store a pointer/reference to a list
that contains all elements x in S with h(Key(x)) = i.
We can implement our three operations as follows (in pseudopseudocode).

Algorithm 55 CH-insert
Input: hash table T , element x
Output: inserts x into T
1: insert x at the head of T [h(Key(x))].

Insertion runs in time O(1), since we insert x at the head of T [h(Key(x))].

We do not test whether x is already in T .

Algorithm 56 CH-search
Input: hash table T , key k
Output: returns an element with key k if one exists, NULL otherwise
1: search for key k in T [h(k)]

Searching might take time O(number of elements in T [h(k)]). In the

worst case (there is no element with key k), we have search the whole list
T [h(k)].

Algorithm 57 CH-delete
Input: hash table T , element x
Output: deletes x from T
1: delete x from the list T [h(Key(x)]

Deletion can be done in time O(1), if we assume that the lists are doubly
linked. Note that we assume that the element x is given. If we only have
the key, we first have to search for the element.
The total space requirement for T is O(m + n), the number of slots in T
plus the number of elements stored in the table.
The worst case running time for searching, however, is O(n), which is
inacceptable. The worst case occurs if all elements hash to the same entry

Markus Bläser 2009–11

15.2. Hash tables 101

T [i]. This can indeed happen, since by the pigeon hole principle, there is an
i such that dN/me keys hash to T [i].
But can this happen often? Of course, there are degenerate hash func-
tions h that map all keys to the same slots T [i]. We assume that h distributes
the keys evenly among the table, that is, #{k ∈ U | h(k) = i} ≤ dN/me.

Lemma 15.1 Assume that h distributes the keys evenly among the table. If
a random set S ⊆ U of size n is stored in T , then the expected number of
elements in any entry of T is ≤ d N n
me · N .

Proof. Let i be an index in {0, . . . , m−1}. Consider the following random

variable
X(S) = #{k ∈ S | h(k) = i}.
We want to bound its expected value. For a key k ∈ U , let
(
1 if k ∈ S and h(k) = i
Yk (S) =
0 otherwise
Obiously, X
X(S) = Yk (S)
k∈U
and " #
X X
E[X] = E Yk = E[Yk ].
k∈U k∈U
The last equality follows from the fact the expected value is linear. Next,
we just have to compute E[Yk ]. But since Yk is {0, 1}-valued (such random
variables are also called indicator variables), we have
E[Yk ] = Pr[Yk = 1],
where the probability is taken over all choices of S. If h(k) 6= i, then Pr[Yk =
1] = 0. Therefore, only keys k with h(k) = i can contribute to the expected
n
value. If h(k) = i, then Pr[Yk = 1] = N , since we choose n elements out of
n
N at random and the probability that we hit k is N . Hence,
 
X X N n
E[X] = E[Yk ] = E[Yk ] ≤ · .
m N
k∈U k:h(k)=i

The quantity d N n n
m e · N is approximately m . This is called the load factor.
Under the assumption of the lemma above, then running time of searching
n
is O(1 + m ). If the number of slots in T and the number of elements are
linearly related, i.e., n = Θ(m), then searching can be done in constant time.
Finding a good hash function is an art, since inputs usually are not
random. So hash functions in practise often include some knowledge of the
input distribution.

Markus Bläser 2009–11

102 15. Hash tables

Example 15.2 We interpret our keys as natural numbers ∈ {0, . . . , N − 1}.

1. h(k) = k mod m
This function distributes the keys evenly. If m = 2t is a power of two,
then it is often a bad hash function, since h(k) are just the lower order
bits of k. If m is a prime “not too close to a power of two”, then it
usually performs well.

2. h(k) = bm(k · C − bk · Cc), where C ∈ (0, 1) is a constant.

The expression in the brackets multiplies k by C and then removes the
integral part. Experiments show that some
√ constants work better than
others, a good choice seems to be C = ( 5 − 1)/2.

Markus Bläser 2009–11

16 More on hashing

16.1 Universal hashing

If the set S being inserted is chosen uniformly at random, then hashing
performs well on the average under mild conditions on h (see Lemma 15.1).
However, real data is usually not chosen uniformly at random. To overcome
this, we now choose the hash function at random! This will have the same
effect if the family of hash function we choose our function from is “nice”.

Definition 16.1 Let H be a family of functions U → {0, . . . , m − 1}. H is

called universal if for any two keys x 6= y,

1
Pr [h(x) = h(y)] ≤ .
h∈H m

Above, h is drawn uniformly at random from H.

Exercise 16.1 Show that the set of all functions U → {0, . . . , m − 1} is

universal.

Universal hashing gives good upper bounds on the expected runnning

time of the Search procedure.

Theorem 16.2 Let H be a universal family of hash functions. If a set

S ⊆ U with n elements is stored in a hash table T of size m with chaining
using a function h drawn uniformly at random from H, then the expected list
length is ≤ 1 + n−1
m .

Proof. Let x be a key from S. Let X be the random variable defined by

X(h) = {k ∈ S | h(k) = h(k)}.

We want to show that E[X] ≤ 1 + n−1

m . For each k ∈ S, let Yk be the
indicator variable (
1 if h(k) = h(x),
Yk (h) =
0 otherwise.
Obviously X
X(h) = Yk (h).
k∈S

103
104 16. More on hashing

and " #
X X
E[X] = E Yk = E[Yk ].
k∈S k∈S

The last equality follows from the fact that the expected value is linear.
Since Yk is an indicator variable,
(
1
≤m if k 6= x
E[Yk ] = Pr [Yk = 1] = Pr [h(k) = h(x)]
h∈H h∈H = 1 if k = x

The last equation follows from the fact that H is universal. Thus,
X n−1
E[X] = E[Yx ] + E[Yk ] ≤ 1 + .
m
k∈S\{x}

While the set of all functions forms a universal family, it is a bad family for
practical purposes: The family is hugh; it has size mN . Furthermore, there
is no better way than representing the functions by a table of its values. A
good family of universal should have the following properties.

• The family should not be too large.

• We should be able to efficiently choose a function a random from the

family. (We enhance our Pseudocode language by a function Rand(n)
which returns a random integer between 0 and n − 1.)
Efficiently means in time polynomial in log N . This induces a bound
on the size of the family.

• There should be one algorithm that efficiently evaluates every function

of the family.

Example 16.3 Here is a better family of universal hash functions. We as-

sume that U = {0, . . . , 2ν − 1} and m = 2µ . Alternatively, we can interpret
U as bit strings or {0, 1}-vectors of length ν and {0, . . . , m − 1} as bits string
or {0, 1}-vectors of length µ. Recall that {0, 1} together with addition and
multiplication modulo 2 forms a field. A matrix M ∈ {0, 1}µ×ν induces a
linear mapping {0, 1}ν → {0, 1}µ . Our family of hash functions Hlin will be
the set of linear functions {0, 1}ν → {0, 1}µ or alternatively, the set of all
µ × ν-matrices over {0, 1}.

Choosing a function at random from Hlin means that we pick a random

matrix from a big bag containing all µ × ν-matrices with {0, 1} entries. The
following “local” process generates the same distribution: We flip a random
coin for each entry and set it to 0 or 1 depending of the outcome. (This
works since Hlin contains all matrices.)

Markus Bläser 2009–11

16.2. Open addressing 105

Theorem 16.4 Hlin is universal.

Proof. Let x, y ∈ {0, 1}ν , x 6= y. Let x1 , . . . , xν and y1 , . . . , yν be the

entries of x and y, respectively. W.l.o.g. x1 6= y1 . W.l.o.g. x1 = 0 and y1 = 1.
Let h be a function in Hlin with matrix M . Let c1 , . . . , cν be the columns of
M . We have
ν
X ν
X
h(x) = M x = ci xi = ci xi
i=1 i=2
ν
X ν
X
h(y) = M y = ci yi = c1 + ci yi
i=1 i=2

We have
n
X
h(x) = h(y) ⇐⇒ c1 = ci (xi − yi ).
i=2
Pn
How many matrices M fulfill c1 = i=2 ci (xi − yi )? The righthand side
determines c1 completely. HencePout of the 2µ possible choices for c1 , only
one will be a solution of c1 = ni=2 ci (xi − yi ). Consequently, Pr[h(x) =
1
h(y)] = 21µ = m .
To choose a random function form Hlin , we just have to fill a matrix with
0 and 1 chosen randomly. We can evaluate the hash function by multiplying
the matrix with the key we want to hash.

16.2 Open addressing

Like chaining, open addressing is a way to resolve conflicts. All elements are
stored directly in the hash table T ; there are no lists stored outside the table.
In this way, our hash table can become full and no further elements can be
added. The load factor can never exceed 1, as it can with chaining. How
do we resolve conflicts? When we want to insert an element in an occupied
table entry, we look for another place in the table. More specifically, we
compute a sequence of positions and check or probe the hash table until
we find an empty slot to put the key. (Note: hashing occasionally has an
alchemy-flavour; this is one of these sections.)
Formally, we are given a function

h : U × {0, . . . , m − 1} → {0, . . . , m − 1}.

We assume that for all keys k ∈ U , the sequence h(k, 0), h(k, 1), . . . , h(k, m−
1) is a permutation of the number 0, 1, . . . , m − 1. We first try to store the
element in T [h(k, 0)]. If this field is occupied, we try T [h(k, 1)] and so on.

Markus Bläser 2009–11

106 16. More on hashing

For this reason, the sequence h(k, 0), h(k, 1), . . . , h(k, m − 1) is also called a
probing sequence. If we do not find an empty field, we claim that the table
is full.
The procedure Insert and Search can be easily realized as follows.

Algorithm 58 OA-Insert
Input: hash table T , key k
Output: inserts k into T , if T is not full
1: i := 0
2: j := h(k, 0)
3: while i < m do
4: if T [j] = NULL then
5: T [j] := k
6: return
7: else
8: i := i + 1
9: j := h(k, i)
9: error “Table full”

Algorithm 59 OA-Search
Input: hash table T , key k
Output: j if k is stored in T [j], NULL if no such j exists
1: i := 0
2: j := h(k, 0)
3: while i < m and T [j] 6= NULL do
4: if T [j] = k then
5: return j
6: else
7: i := i + 1
8: j := h(k, i)
8: return NULL

What about deletion of elements? This is a real problem. We cannot

just set T [i] := NULL to delete an element, since afterwards, the search
procedure might not be able to find elements that are still in the table. One
solution is to not remove elements from the table but mark them as deleted
by using an extra flag. If there are many deletions, then the table is quite
full although it only stores a small number of keys. But if only a few deletion
operations are expected (or none at all), this works well.
How can one construct functions h : U ×{0, . . . , m−1} → {0, . . . , m−1}.
Here are some examples:

Linear problem: Let h0 : U → {0, . . . , m − 1} be an “ordinary” hash func-

Markus Bläser 2009–11

16.3. Perfect hashing 107

tion. We set
h(k, i) := h0 (k) + i mod m.
h(k, 0), . . . , h(k, m − 1) is a permutation of 0, . . . , m − 1, so h meets
our requirements. In fact, h(k, 0), . . . , h(k, m − 1) is a cylic shift. This
means that altogether, h produces at most m different probing se-
quences.
A big problem of linear probing is primary clustering: Assume that h0
distributes the keys evenly among the entries of T . Consider an empty
table and we insert an element. Since h0 distributes the keys evenly,
every cell of T has the same chance of getting the element. Let i be the
cell to which the element goes. Now we add a second element. Every
cell of T again has a 1/m chance of getting the element except for one,
i + 1. This cell has a chance of 2/m, since the element goes there if it
goes to cell i (which is occupied) or cell i + 1. In general, if j cells are
occupied before a particular cell, then the probability that an element
will be stored in this cell is (j + 1)/m. Such long runs are bad since
they increase the running time of search.

Quadratic probing: Quadratic probing uses functions of the form

h(k, i) = h0 (k) + ai + bi2 mod m

with appropriate constants a and b. It avoids primary clustering (and

replaces it by a milder form called, guess what, secondary clustering).
On the other hand, it also generates at most m different probe se-
quences.

Double hashing: Here we use two hash functions h1 , h2 : U → {0, . . . , m −

1} and set
h(k, i) = h1 (k) + ih2 (k) mod m
If h1 and h2 are carefully chosen, then we get Θ(m2 ) different probe
sequences.

16.3 Perfect hashing

Up to now, all our upper bounds on the search time are only true in expec-
tation. For universal hashing, the expected search time for any element is
bounded by O(1 + (n − 1)/m) but that does not exclude that for a particular
choice of h, the search time is much higher.
In the following, we assume that we are given a fixed set of keys S and
deletions are not allowed. (This is not an unrealistic scenario, think of a
dictionary or phone book.) A hash function h is called perfect for S if every
call of the Search procedure has running time O(1).

Markus Bläser 2009–11

108 16. More on hashing

Lemma 16.5 Let H be a universal family of hash functions and m = n2 .

We have
1
Pr[for all x 6= y: h(x) 6= h(y)] ≥ .
2

Proof. For every pair x 6= y, we have Pr[h(x) = h(y)] ≤ 1/m. Therefore,

Pr[for all x 6= y: h(x) 6= h(y)] = 1 − Pr[there are x 6= y: h(x) = h(y)]

X
≥1− Pr[h(x) 6= h(y)]
x6=y
 
n 1
≥1− ·
2 m
1
≥ ,
2
where the sum in the second line is over all unordered pairs.
If we can accept the fact that the size of the table is quadratic in the
number of stored keys, then half of the functions h ∈ H are perfect, since
they map every key to a distinct cell. However, if n is large, then we want
m to be linear in n. We can achieve this by a two stage construction (see
below).
How can we construct such an h? We can just draw a function at random.
With probability 1/2, it will be perfect. We can check this, since we know
the set S of keys explicitly. If h is not perfect, we draw another function and
so on. The probability that we do not find a perfect function after i such
steps is ≤ 21i which goes to zero rapidly.
To get down to linear space, we first choose a hash function h : U →
{0, . . . , n − 1} from a universal family. h will have collisions, let m0 be the
number of keys that are mapped to 0 by h, m1 be the number of keys that
are mapped to 1 and so on. Let Si be the set of keys that are hashed to i. We
use a hash function hi : Si → {0, . . . , m2i − 1} as constructed in Lemma 16.5.
A key x with h(x) = i is hashed to (i, hi (x)). Finally the hash values (i, j),
0 ≤ iP ≤ n − 1 and 0 ≤ j ≤ mi are identified with {0, . . . , M − 1} where
M = n−1 2
i=0 mi . Altogether, we have a two-stage process: First apply h and
then the appropriate hi . We have no collisions by construction.
We are done if we can choose h in such a way that
n−1
X 1
Pr[ m2i > 4n] < .
h 2
i=0

In this case, we can choose h such that the total table length is ≤ 4n. It is
sufficient to show the following lemma. (Use Markov inequality! Or do it on
foot: If for at least half of the hash functions, the length is > 4n, then the
expected length is > 12 · 4n = 2n, a contradiction.)

Markus Bläser 2009–11

16.3. Perfect hashing 109

Lemma 16.6 E[ n−1 2

P
i=0 mi ] < 2n.

Proof. For x, y ∈, let Zx,y be the random variable defined by

(
1 if h(x) = h(y),
Zx,y (h) =
0 otherwise.

We have
"n−1 #  
X XX
E m2i = E  Zx,y  ,
i=0 x∈S y∈S

because for every 0 ≤ i ≤ n − 1, every pair (x, y) ∈ Si × Si contributes 1 to

the sum on the right hand side yielding a total contribution of m2i . By the
linearity of expectation,
 
XX XX
E Zx,y  = n + E[Zx,y ]
x∈S y∈S x∈S y6=x
1
≤ n + n(n − 1) ·
n
< 2n

Markus Bläser 2009–11