Lecture Notes -- Data Structures

These lecture notes are designed for on-line reference and review. Please do not print
them on university computing facilities!!

• Lecture 1 -- Data Structures and Programming

• Lecture 2 -- Software Engineering and Top-Down Design
• Lecture 3 -- Stacks and Queues
• Lecture 4 -- Pointers and Dynamic Memory Allocation
• Lecture 5 -- Linked Stacks and Queues
• Lecture 6 -- Calculator Algorithms
• Lecture 7 -- Linked Lists: Insertion and Deletion
• Lecture 8 -- Do the Shuffle
• Lecture 9 -- Recursive and Doubly Linked Lists
• Lecture 10 -- Recursion and Backtracking
• Lecture 11 -- Applications of Recursion
• Lecture 12 -- Abstraction and Modules
• Lecture 13 -- Object-Oriented Programming
• Lecture 14 -- Simulations
• Midterm 1 -- Answer Key
• Lecture 15 -- Asymptotics
• Lecture 16 -- Introduction to Sorting
• Lecture 17 -- Mergesort and Quicksort
• Lecture 18 -- Heapsort and Priority Queues
• Lecture 19 -- Sequential and Binary Search
• Lecture 20 -- Access Formulas and Arrays
• Lecture 21 -- Hashing
• Lecture 22 -- Binary Search Trees
• Midterm 2 -- Answer Key
• Lecture 23 -- Random Search Trees
• Lecture 24 -- AVL Trees These will not be covered in class, but the
implementation may be useful in doing the final program.
• Lecture 25 -- Red-Black Trees
• Lecture 26 -- Splay Trees
• Lecture 27 -- Graphs
 Data Structures and Programming
Lecture 1
Steven S. Skiena

Why Data Structures?

In my opinion, there are only three important ideas which must be mastered to write
interesting programs.

• Iteration - Do, While, Repeat, If

• Data Representation - variables and pointers
• Subprograms and Recursion - modular design and abstraction

At this point, I expect that you have mastered about 1.5 of these 3.

It is the purpose of Computer Science II to finish the job.

Data types vs. Data Structures

A data type is a well-defined collection of data with a well-defined set of operations on it.

A data structure is an actual implementation of a particular abstract data type.

Example: The abstract data type Set has the operations EmptySet(S), Insert(x,S),
Delete(x,S), Intersection(S1,S2), Union(S1,S2), MemberQ(x,S), EqualQ(S1,S2),
SubsetQ(S1,S2).

This semester, we will learn to implement such abstract data types by building data
structures from arrays, linked lists, etc.

Modula-3 Programming

Control Structures: IF-THEN-ELSE, CASE-OF

Iteration Constructs: REPEAT-UNTIL (at least once), WHILE-DO (at least 0), FOR-DO
(exactly n times).

Elementary Data Types: INTEGER, REAL, BOOLEAN, CHAR

Enumerated Types: COINSIDE = {HEADS, TAIL, SIDE}

Operations: +, -, <, >, #

Elementary Data Structures

ArraysThese let you access lots of data fast. (good)

You can have arrays of any other data type. (good)

However, you cannot make arrays bigger if your program decides it needs more space.
(bad)

RecordsThese let you organize non-homogeneous data into logical packages to keep
everything together. (good)

These packages do not include operations, just data fields (bad, which is why we need
objects)

Records do not help you process distinct items in loops (bad, which is why arrays of
records are used)

SetsThese let you represent subsets of a set with such operations as intersection, union,
and equivalence. (good)

Built-in sets are limited to a certain small size. (bad, but we can build our own set data
type out of arrays to solve this problem if necessary)

Subroutines

Subprograms allow us to break programs into units of reasonable size and complexity,
allowing us to organize and manage even very long programs.

This semester, you will first encounter programs big enough that modularization will be
necessary for survival.

Functions are subroutines which return values, instead of communicating by parameters.

Abstract data types have each operation defined by a subroutine.

Subroutines which call themselves are recursive. Recursion provides a very powerful
way to solve problems which takes some getting used to.

Such standard data structures as linked lists and trees are inherently recursive data
structures.

Parameter Passing

There are two mechanisms for passing data to a subprogram, depending upon whether the
subprogram has the power to alter the data it is given.
In pass by value, a copy of the data is passed to the subroutine, so that no matter what
happens to the copy the original is unaffected.

In pass by reference, the variable argument is renamed, not copied. Thus any changes
within the subroutine effect the original data. These are the VAR parameters.

Example: suppose the subroutine is declared Push(VAR s:stack, e:integer) and called
with Push(t,x). Any changes with Push to e have no effect on x, but changes to s effect t.

Generic Modula-3 Program I

MODULE Prim EXPORTS Main;

(* Prime number testing with Repeat *)

IMPORT SIO;

VAR candidate, i: INTEGER;

BEGIN
SIO.PutText("Prime number test\n");
REPEAT
SIO.PutText("Please enter a positive number; enter 0 to quit. ");
candidate:= SIO.GetInt();
IF candidate > 2 THEN
i:= 1;
REPEAT
i:= i + 1
UNTIL ((candidate MOD i) = 0) OR (i * i > candidate);
IF (candidate MOD i) = 0 THEN
SIO.PutText("Not a prime number\n")
ELSE
SIO.PutText("Prime number\n")
END; (*IF (candidate MOD i) = 0 ...*)
ELSIF candidate > 0 THEN
SIO.PutText("Prime number\n") (*1 and 2 are prime*)
END; (*IF candidate > 2*)
UNTIL candidate <= 0;
END Prim.

Generic Modula-3 Program II

MODULE Euclid2 EXPORTS Main; (*17.05.94. LB*)

(* The Euclidean algorithm (with controlled input):
Compute the greatest common divisor (GCD) *)

IMPORT SIO;

VAR
a, b: INTEGER; (* input values *)
x, y: CARDINAL; (* working variables *)

<*FATAL SIO.Error*>
BEGIN (*statement part*)
SIO.PutText("Euclidean algorithm\nEnter 2 positive numbers: ");

a:= SIO.GetInt();
WHILE a <= 0 DO
SIO.PutText("Please enter a positive number: ");
a:= SIO.GetInt();
END; (*WHILE a < 0*)

b:= SIO.GetInt();
WHILE b <= 0 DO
SIO.PutText("Please enter a positive number: ");
b:= SIO.GetInt();
END; (*WHILE b < 0*)

x:= a; y:= b; (*x and y can be changed by the algorithm*)

WHILE x # y DO
IF x > y THEN x:= x - y ELSE y:= y - x END;
END; (*WHILE x # y*)

SIO.PutText("Greatest common divisor = ");

SIO.PutInt(x);
SIO.Nl();
END Euclid2.

Programming Proverbs

KISS - ``Keep it simple, stupid.'' - Don't use fancy features when simple ones suffice.

RTFM - ``Read the fascinating manual.'' - Most complaints from the compiler can be
solved by reading the book. Logical errors are something else.

Make your documentation short but sweet. - Always document your variable
declarations, and tell what each subprogram does.

Every subprogram should do something and hide something - If you cannot concisely
explain what your subprogram does, it shouldn't exist. This is why I write the header
comments before I write the subroutine.

Program defensively - Add the debugging statements and routines at the beging, because
you know you are going to need them later.

A good program is a pretty program. - Remember that you will spend more time reading
your programs than we will.

Perfect Shuffles

1 1 1 1 1 1 1 1 1
2 27 14 33 17 9 5 3 2
3 2 27 14 33 17 9 5 3
4 28 40 46 49 25 13 7 4
5 3 2 27 14 33 17 9 5
 6 29 15 8 30 41 21 11 6
7 4 28 40 46 49 25 13 7
8 30 41 21 11 6 29 15 8
9 5 3 2 27 14 33 17 9
10 31 16 34 43 22 37 19 10
11 6 29 15 8 30 41 21 11
12 32 42 47 24 38 45 23 12
13 7 4 28 40 46 49 25 13
14 33 17 9 5 3 2 27 14
15 8 30 41 21 11 6 29 15
16 34 43 22 37 19 10 31 16
17 9 5 3 2 27 14 33 17
18 35 18 35 18 35 18 35 18
19 10 31 16 34 43 22 37 19
20 36 44 48 50 51 26 39 20
21 11 6 29 15 8 30 41 21
22 37 19 10 31 16 34 43 22
23 12 32 42 47 24 38 45 23
24 38 45 23 12 32 42 47 24
25 13 7 4 28 40 46 49 25
26 39 20 36 44 48 50 51 26
27 14 33 17 9 5 3 2 27
28 40 46 49 25 13 7 4 28
29 15 8 30 41 21 11 6 29
30 41 21 11 6 29 15 8 30
31 16 34 43 22 37 19 10 31
32 42 47 24 38 45 23 12 32
33 17 9 5 3 2 27 14 33
34 43 22 37 19 10 31 16 34
35 18 35 18 35 18 35 18 35
36 44 48 50 51 26 39 20 36
37 19 10 31 16 34 43 22 37
38 45 23 12 32 42 47 24 38
39 20 36 44 48 50 51 26 39
40 46 49 25 13 7 4 28 40
41 21 11 6 29 15 8 30 41
42 47 24 38 45 23 12 32 42
43 22 37 19 10 31 16 34 43
44 48 50 51 26 39 20 36 44
45 23 12 32 42 47 24 38 45
46 49 25 13 7 4 28 40 46
47 24 38 45 23 12 32 42 47
48 50 51 26 39 20 36 44 48
49 25 13 7 4 28 40 46 49
50 51 26 39 20 36 44 48 50
51 26 39 20 36 44 48 50 51
52 52 52 52 52 52 52 52 52

Software Engineering and Top-Down Design

Lecture 2
Steven S. Skiena
Software Engineering and Saddam Hussain

Think about the Patriot missiles which tried to shoot down SCUD missiles in the Persian
Gulf war and think about how difficult it is to produce working software!

1. How do you test a missile defense system?

2. How do you satisfy such tight constraints as program speed, computer
size/weight, flexibility to recognize different types of missiles?
3. How do you get hundreds of people to work on the same program without getting
chaos?

Even today, there is great controversy about how well the missiles actually did in the war.

Testing and Verification

How do you know that your program works? Not by testing it!

``Testing reveals the presence, but not the absence of bugs.'' - Dijkstra

Still, it is important to design test cases which exercise the boundary conditions of the
program.

Example: Linked list insertion. The boundary cases include:

• insertion before the first element.

• insertion after the last element.
• insertion into the empty list.
• insertion between element (the general case).

Test Case Generation

In the Microsoft Excel group, there is one tester for each programmer! Types of test cases
include:

Boundary cases - Make sure that each line of code and branch of IF is executed at least
once.

Random data - Automatically generated test data can be useful to test user patterns you
might otherwise not consider, but you must verify that the results are correct!

Other users - People who were not involved in writing the program will have vastly
different ideas of how to use it.

Adversaries - People who want to attack your program and have access to the source for
often find bugs by reading it.
Verification

But how can we know that our program works? The ideal way is to mathematically prove
it.

For each subprogram, there is a precise set of preconditions which we assume is satisfied
by the input parameters, and a precise set of post-conditions which are satisfied by the
output parameters.

If we can show that any input satisfying the preconditions is always transformed to output
satisfying the post conditions, we have proven the subprogram correct.

Top-Down Refinement

To correctly build a complicated system requires first setting broad goals and refining
them over time. Advantages include:

A hierarchy hides information - This permits you to focus attention on only a manageable
amount of detail.

With the interfaces defined by the hierarchy, changes can be made without effecting the
rest of the structure - Thus systems can be maintained without being ground to a halt.

Progress can be made in parallel by having different people work on different

subsections - Thus you can organize to build large systems.

Stepwise Refinement in Programming

The best way to build complicated programs is to construct the hierarchy one level at a
time, finally writing the actual functions when the task is small enough to be easily done.

• Build a prototype to throw away, because you will, anyway.

• Anything difficult put off for last. If necessary, decompose it into another level of
detail.

Most of software engineering is just common sense, but it is very easy to ignore common
sense.

Building a Military Threat

Module Build-Military: The first decision is now to organize it, not what type of tank to
buy.

Several different organizations are possible, and in planning we should investigate each
one:
 • Offense, Defense
• Army, Navy, Air Force, Marines, Coast Guard ...

Procedure Army: Tanks, Troops, Guns ...

Procedure Troops: Training, Recruitment, Supplies

Top-Down Design Example

``Teaching Software Engineering is like telling children to brush their teeth.'' -

anonymous professor.

To make this more concrete, lets outline how a non-trivial program should be structured.

Suppose that you wanted to write a program to enable a person to play the game
Battleship against a computer.

Tell me what to do!

What is Battleship?

Each side places 5 ships on a grid, and then takes turns guessing grid points
until one side has covered all the ships:

For each query, the answer ``hit'', ``miss'', or ``you sunk my battleship'' must be given.

There are two distinct views of the world, one reflecting the truth about the board, the
other reflecting what your opponent knows.

Program: Battleship

Interesting subproblems are: display board, generate query, respond to query, generate
initial configuration, move-loop (main routine).

What data structure should we use? Two-dimensional arrays.

How do I enforce separation between my view and your view?

Data Structures and Programming

Lecture 3
Steven S. Skiena

Stacks and Queues

The first data structures we will study this semester will be lists which have the property
that the order in which the items are used is determined by the order they arrive.

• Stacks are data structures which maintain the order of last-in, first-out
• Queues are data structures which maintain the order of first-in, first-out

Queues might seem fairer, which is why lines at stores are organized as queues instead of
stacks, but both have important applications in programs as a data structure.

Operations on Stacks

The terminology associated with stacks comes from the spring loaded plate containers
common in dining halls.

When a new plate is washed it is pushed on the stack.

When someone is hungry, a clean plate is popped off the stack.

A stack is an appropriate data structure for this task since the plates don't care about when
they are used!

Maintaining Procedure Calls

Stacks are used to maintain the return points when Modula-3 procedures call other
procedures which call other procedures ...

Jacob and Esau

In the biblical story, Jacob and Esau were twin brothers where Esau was born first and
thus inherited Issac's birthright. However, Jacob got Esau to give it away for a bowl of
soup, and so Jacob went to become a patriarch of Israel.

But why was Jacob justified in so tricking his brother???

Rashi, a famous 11th century Jewish commentator, explained the problem by saying
Jacob was conceived first, then Esau second, and Jacob could not get around the narrow
tube to assume his rightful place first in line!

• Therefore Rebecca was modeled by a stack.

• ``Push'' Issac, Push ``Jacob'', Push ``Esau'', Pop ``Esau'', Pop ``Jacob''

Abstract Operations on a Stack

• Push(x,s) and Pop(x,s) - Stack s, item x. Note that there is no search operation.
• Initialize(s), Full(s), Empty(s), - The latter two are Boolean queries.
Defining these abstract operations lets us build a stack module to use and reuse without
knowing the details of the implementation.

The easiest implementation uses an array with an index variable to represent the top of
the stack.

An alternative implementation, using linked lists is sometimes better, for it can't ever
overflow. Note that we can change the implementations without the rest of the program
knowing!

Declarations for a stack

INTERFACE Stack; (*14.07.94 RM, LB*)

(* Stack of integer elements *)

TYPE ET = INTEGER; (*element type*)

PROCEDURE Push(elem : ET); (*adds element to top of stack*)

PROCEDURE Pop(): ET; (*removes and returns top element*)
PROCEDURE Empty(): BOOLEAN; (*returns true if stack is empty*)
PROCEDURE Full(): BOOLEAN; (*returns true if stack is full*)

END Stack.

Stack Implementation

MODULE Stack; (*14.07.94 RM, LB*)

(* Implementation of an integer stack *)

CONST
Max = 8; (*maximum number of elements on stack*)

TYPE
S = RECORD
info: ARRAY [1 .. Max] OF ET;
top: CARDINAL := 0; (*initialize stack to empty*)
END; (*S*)

VAR stack: S; (*instance of stack*)

PROCEDURE Push(elem:ET) =
(*adds element to top of stack*)
BEGIN
INC(stack.top); stack.info[stack.top]:= elem
END Push;

PROCEDURE Pop(): ET =
(*removes and returns top element*)
BEGIN
DEC(stack.top); RETURN stack.info[stack.top + 1]
END Pop;

PROCEDURE Empty(): BOOLEAN =

 (*returns true if stack is empty*)
BEGIN
RETURN stack.top = 0
END Empty;

PROCEDURE Full(): BOOLEAN = (*returns true if stack is full*)

BEGIN
RETURN stack.top = Max
END Full;

BEGIN
END Stack.

Using the Stack Type

MODULE StackUser EXPORTS Main; (*14.02.95. LB*)

(* Example client of the integer stack *)

FROM Stack IMPORT Push, Pop, Empty, Full;

FROM SIO IMPORT Error, GetInt, PutInt, PutText, Nl;
<* FATAL Error *> (*suppress warning*)

BEGIN
PutText("Stack User. Please enter numbers:\n");
WHILE NOT Full() DO
Push(GetInt()) (*add entered number to stack*)
END;
WHILE NOT Empty() DO
PutInt(Pop()) (*remove number from stack and return
it*)
END;
Nl();
END StackUser.

FIFO Queues

Queues are more difficult to implement than stacks, because action happens at both ends.

The easiest implementation uses an array, adds elements at one end, and moves all
elements when something is taken off the queue.

It is very wasteful moving all the elements on each DEQUEUE. Can we do better?

More Efficient Queues

Suppose that we maintaining pointers to the first (head) and last (tail) elements in the
array/queue?

Note that there is no reason to explicitly clear previously unused cells.

Now both ENQUEUE and DEQUEUE are fast, but they are wasteful of space. We need a
array bigger than the total number of ENQUEUEs, instead of the maximum number of
items stored at a particular time.

Circular Queues

Circular queues let us reuse empty space!

Note that the pointer to the front of the list is now behind the back pointer!

When the queue is full, the two pointers point to neighboring elements.

There are lots of possible ways to adjust the pointers for circular queues. All are tricky!

How do you distinguish full from empty queues, since their pointer positions might be
identical? The easiest way to distinguish full from empty is with a counter of how many
elements are in the queue.

FIFO Queue Interface

INTERFACE Fifo; (*14.07.94 RM, LB*)

(* A queue of text elements *)

TYPE ET = TEXT; (*element type*)

PROCEDURE Enqueue(elem:ET); (*adds element to end*)

PROCEDURE Dequeue(): ET; (*removes and returns first element*)
PROCEDURE Empty(): BOOLEAN; (*returns true if queue is empty*)
PROCEDURE Full(): BOOLEAN; (*returns true if queue is full*)

END Fifo.

Priority Queue Implementation

MODULE Fifo; (*14.07.94 RM, LB*)

(* Implementation of a fifo queue of text elements *)

CONST
Max = 8; (*Maximum number of elements in FIFO
queue*)

TYPE
Fifo = RECORD
info: ARRAY [0 .. Max - 1] OF ET;
in, out, n: CARDINAL := 0;
END; (*Fifo*)

VAR w: Fifo; (*contains a FIFO queue*)

PROCEDURE Enqueue(elem:ET) =
(*adds element to end*)
BEGIN
 w.info[w.in]:= elem; (*stores new element*)
w.in:= (w.in + 1) MOD Max; (*increments in-pointer in ring*)
INC(w.n); (*increments number of stored
elements*)
END Enqueue;

PROCEDURE Dequeue(): ET =
(*removes and returns first element*)
VAR e: ET;
BEGIN
e:= w.info[w.out]; (*removes oldest element*)
w.out:= (w.out + 1) MOD Max; (*increments out-pointer in ring*)
DEC(w.n); (*decrements number of stored
elements*)
RETURN e; (*returns the read element*)
END Dequeue;

Utility Routines

PROCEDURE Empty(): BOOLEAN =

(*returns true if queue is empty*)
BEGIN
RETURN w.n = 0;
END Empty;

PROCEDURE Full(): BOOLEAN =

(*returns true if queue is full*)
BEGIN
RETURN w.n = Max
END Full;

BEGIN
END Fifo.

User Module

MODULE FifoUser EXPORTS Main; (*14.07.94. LB*)

(* Example client of the text queue. *)

FROM Fifo IMPORT Enqueue, Dequeue, Empty, Full; (* operations of the

queue *)
FROM SIO IMPORT Error, GetText, PutText, Nl;
<* FATAL Error *> (*supress warning*)

BEGIN
PutText("FIFO User. Please enter texts:\n");
WHILE NOT Full() DO
Enqueue(GetText())
END;
WHILE NOT Empty() DO
PutText(Dequeue() & " ")
END;
Nl();
END FifoUser.
Other Queues

Double-ended queues - These are data structures which support both push and pop and
enqueue/dequeue operations.

Priority Queues(heaps) - Supports insertions and ``remove minimum'' operations which

useful in simulations to maintain a queue of time events.

We will discuss simulations in a future class.

Pointers and Dynamic Memory Allocation

Lecture 4
Steven S. Skiena

Pointers and Dynamic Memory Allocation

Although arrays are good things, we cannot adjust the size of them in the middle of the
program.

If our array is too small - our program will fail for large data.

If our array is too big - we waste a lot of space, again restricting what we can do.

The right solution is to build the data structure from small pieces, and add a new piece
whenever we need to make it larger.

Pointers are the connections which hold these pieces together!

Pointers in Real Life

In many ways, telephone numbers serve as pointers in today's society.

• To contact someone, you do not have to carry them with you at all times. All you
need is their number.
• Many different people can all have your number simultaneously. All you need do
is copy the pointer.
• More complicated structures can be built by combining pointers. For example,
phone trees or directory information.

Addresses are a more physically correct analogy for pointers, since they really are
memory addresses.
Linked Data Structures

All the dynamic data structures we will build have certain shared properties.

• We need a pointer to the entire object so we can find it. Note that this is a pointer,
not a cell.
• Each cell contains one or more data fields, which is what we want to store.
• Each cell contains a pointer field to at least one ``next'' cell. Thus much of the
space used in linked data structures is not data!
• We must be able to detect the end of the data structure. This is why we need the
NIL pointer.

Pointers in Modula-3

A node in a linked list can be declared:

type
pointer = REF node;
node = record
info : item;
next : pointer;
end;

var
p,q,r : pointer; (* pointers *)
x,y,z : node; (* records *)

Note circular definition. Modula-3 lets you get away with this because it is a reference
type. Pointers are the same size regardless of what they point to!

We want dynamic data structures, where we make nodes as we need them. Thus
declaring nodes as variables are not the way to go!

Dynamic Allocation

To get dynamic allocation, use new:

p := New(ptype);

New(ptype) allocates enough space to store exactly one object of the type ptype. Further,
it returns a pointer to this empty cell.

Before a new or otherwise explicit initialization, a pointer variable has an arbitrary value
which points to trouble!

Warning - initialize all pointers before use. Since you cannot initialize them to explicit
constants, your only choices are
 • NIL - meaning explicitly nothing.
• New(ptype) - a fresh chunk of memory.
• assignment to some previously initialized pointer of the same type.

Pointer Examples

Example: p := new(node); q := new(node);

p.x grants access to the field x of the record pointed to by p.

p^.info := "music";
q^.next := nil;

The pointer value itself may be copied, which does not change any of the other fields.

Note this difference between assigning pointers and what they point to.

p := q;

We get a real mess. We have completely lost access to music and can't get it back!
Pointers are unidirectional.

Alternatively, we could copy the object being pointed to instead of the pointer itself.

p^ := q^;

What happens in each case if we now did:

p^.info := "data structures";

Where Does the Space Come From?

Can we really get as much memory as we want without limit just by using New?

No, because there are the physical limits imposed by the size of the memory of the
computer we are using. Usually Modula-3 systems let the dynamic memory come from
the ``other side'' of the ``activation record stack'' used to maintain procedure calls:

Just as the stack reuses memory when a procedure exits, dynamic storage must be
recycled when we don't need it anymore.

Garbage Collection

The Modula-3 system is constantly keeping watch on the dynamic memory which it has
allocated, making sure that something is still pointing to it. If not, there is no way for you
to get access to it, so the space might as well be recycled.
The garbage collector automatically frees up the memory which has nothing pointing to
it.

It frees you from having to worry about explicitly freeing memory, at the cost of leaving
certain structures which it can't figure out are really garbage, such as a circular list.

Explicit Deallocation

Although certain languages like Modula-3 and Java support garbage collection, others
like C++ require you to explicitly deallocate memory when you don't need it.

Dispose(p) is the opposite of New - it takes the object which is pointed to by p and makes
it available for reuse.

Note that each dispose takes care of only one cell in a list. To dispose of an entire linked
structure we must do it one cell as a time.

Note we can get into trouble with dispose:

Of course, it is too late to dispose of music, so it will endure forever without garbage
collection.

Suppose we dispose(p), and later allocation more dynamic memory with new. The cell
we disposed of might be reused. Now what does q point to?

Answer - the same location, but it means something else! So called dangling references
are a horrible error, and are the main reason why Modula-3 supports garbage collection.

A dangling reference is like a friend left with your old phone number after you move.
Reach out and touch someone - eliminate dangling references!

Security in Java

It is possible to explicitly dispose of memory in Modula-3 when it is really necessary, but

it is strongly discouraged.

Java does not allow one to do such operations on pointers at all. The reason is security.

Pointers allow you access to raw memory locations. In the hands of skilled but evil
people, unchecked access to pointers permits you to modify the operating system's or
other people's memory contents.

Java is a language whose programs are supposed to be transferred across the Internet to
run on your computer. Would you allow a stranger's program to run on your machine if
they could ruin your files?
 Linked Stacks and Queues
Lecture 5
Steven S. Skiena

Pointers about Pointers

var p, q : ^node;

p = new(node) creates a new node and sets p to point to it.

p describes the node which is pointed to by p.

p .item describes the item field of the node pointed to by p.

dispose(p) returns to the system the memory used by the node pointed to by p. This is not
used because of Modula-3 garbage collection.

NIL is the only value a pointer can have which is not an address.

Linked Stacks

The problem with array-based stacks are that the size must be determined at compile
time. Instead, let's use a linked list, with the stack pointer pointing to the top element.

To push a new element on the stack, we must do:

p^.next = top;
top = p;

Note this works even for the first push if top is initialized to NIL!

Popping from a Linked Stack

To pop an item from a linked stack, we just have to reverse the operation.

p = top;
top = top^.next;
p^.next = NIL; (*avoid dangling reference*)

Note again that this works in the boundary case of one item on the stack.
Note that to check we don't pop from an empty stack, we must test whether top = NIL
before using top as a pointer. Otherwise things crash or segmentation fault.

Linked Stack in Modula-3

MODULE Stacks; (*14.07.94 RM, LB*)

(* Implementation of the abstract, generic stack. *)
REVEAL
T = BRANDED REF RECORD
info: ET; next: T;
END; (*T*)

PROCEDURE Create(): T = (*creates and intializes a new stack*)

BEGIN
RETURN NIL; (* a new, empty stack is simply NIL *)
END Create;

PROCEDURE Push(VAR stack: T; elem:ET) =

(*adds element to stack*)
VAR new: T := NEW(T, info:= elem, next:= stack); (*create element*)
BEGIN
stack:= new (*add element at top*)
END Push;

PROCEDURE Pop(VAR stack: T): ET =

(*removes and returns top element, or NIL for empty stack*)
VAR first: ET := NIL; (* Pop returns NIL for empty stack*)
BEGIN
IF stack # NIL THEN
first:= stack.info; (*copy info from first element*)
stack:= stack.next; (*remove first element*)
END; (*IF stack # NIL*)
RETURN first;
END Pop;

PROCEDURE Empty(stack: T): BOOLEAN =

(*returns TRUE for empty stack*)
BEGIN
RETURN stack = NIL
END Empty;

BEGIN
END Stacks.

Generic Stack Interface

INTERFACE Stacks; (*14.07.94 RM, LB*)

(* Abstract generic stack. *)

TYPE
T <: REFANY; (*type of stack*)
ET = REFANY; (*type of elements*)

PROCEDURE Create(): T; (*creates and intializes a

new stack*)
 PROCEDURE Push(VAR stack: T; elem: ET); (*adds element to stack*)
PROCEDURE Pop(VAR stack: T): ET; (*removes and returns top
element, or
NIL for empty stack*)
PROCEDURE Empty(stack: T): BOOLEAN; (*returns TRUE for empty
stack*)

END Stacks.

Generic Stacks Client

MODULE StacksClient EXPORTS Main; (* LB *)

(* Example client of both the generic stack and the type FractionType.
This program builds up a stack of fraction numbers as well as of
complex numbers.
*)

IMPORT Stacks;
IMPORT FractionType;
FROM Stacks IMPORT Push, Pop, Empty;
FROM SIO IMPORT PutInt, PutText, Nl, PutReal, PutChar;

TYPE
Complex = REF RECORD r, i: REAL END;

VAR
stackFraction: Stacks.T:= Stacks.Create();
stackComplex : Stacks.T:= Stacks.Create();

c: Complex;
f: FractionType.T;

BEGIN (*StacksClient*)
PutText("Stacks Client\n");

FOR i:= 1 TO 4 DO
Push(stackFraction, FractionType.Create(1, i)); (*stores numbers
1/i*)
END;

FOR i:= 1 TO 4 DO
Push(stackComplex, NEW(Complex, r:= FLOAT(i), i:= 1.5 * FLOAT(i)));
END;

WHILE NOT Empty(stackFraction) DO

f:= Pop(stackFraction);
PutInt(FractionType.Numerator(f));
PutText("/");
PutInt(FractionType.Denominator(f), 1);
END;
Nl();

WHILE NOT Empty(stackComplex) DO

c:= Pop(stackComplex);
PutReal(c.r);
 PutChar(':');
PutReal(c.i);
PutText(" ");
END;
Nl();

END StacksClient.

Linked Queues

Queues in arrays were ugly because we need wrap around for circular queues. Linked
lists make it easier.

We need two pointers to represent our queue - one to the rear for enqueue operations,
and one to the front for dequeue operations.

Note that because both operations move forward through the list, no back pointers are
necessary!

Enqueue and Dequeue

To enqueue an item :

p^.next := NIL;
if (back = NIL) then begin (* empty queue *)
front := p; back := p;
end else begin (* non-empty queue *)
back^.next := p;
back := p;
end;

To dequeue an item:

p := front;
front := front^.next;
p^.next := NIL;
if (front = NIL) then back := NIL; (* now-empty queue *)

Building the Calculator

Lecture 6
Steven S. Skiena

Reverse Polish Notation

HP Calculators use reverse Polish notation or postfix notation. Instead of the conventional
a + b, we write A B +.

Our calculator will do the same. Why? Because it is the easiest notation to implement!

The rule for conversion is to read the expression from left to right. When we see a
number, push it on the operation stack. When we see an operation, pop the last two
numbers on stack, do the operation, and push the result on the stack.

Look Ma, no parentheses!

Algorithms for the calculator

To implement addition, we add digits from right to left, with the carry one place if the
sum is greater than 10.

Note that the last carry can go beyond one or both numbers, so you must handle this
special case.

To implement subtraction, we work on digits from right to left, and borrow 10 if

necessary from the digit to the left.

A borrow from the leftmost digit is complicated, since that gives a negative number.

This is why I suggest completing addition first before worrying about subtraction.

I recommend to test which number has a larger absolute value, subtract from that, and
then adjust the sign accordingly.

Parsing the Input

There are several possible ways to handle the problem of reading in the input line and
parsing it, i.e. breaking it into its elementary components of numbers and operators.

The way that seems best to me is to read the entire line as one character string in a
variable of type TEXT.

As detailed in your book, you can use the function Text.Length(S) to get the length of this
string, and the function Text.GetChar(S,i) to retreive any given character.
Useful functions on characters include the function ORD(c), which returns the integer
character code of c. Thus ORD(c) - ORD('0') returns the numerical value of a digit
character.

You can test characters for equality to identify special symbols.

Standard I/O

The easiest way to read and write from the files is to use I/O redirection from UNIX.

Suppose calc is your binary program, and it expects input from the keyboard and output
to the screen. By running calc < filein at the command prompt, it will take its input from
the file filein instead of the keyboard.

Thus by writing your program to read from regular I/O, you can debug it interactively
and also run my test files.

Programming Hints

1. Write the comments first, for your sake.

2. Make sure your main routine is abstract enough that you can easily see what the
program does.
3. Isolate the details of your data structures to a few abstract operations.
4. Build good debug print routines first.

List Insertion and Deletion

Lecture 7
Steven S. Skiena

Search, Insert, Delete

There are three fundamental operations we need for any database:

• Insert: add a new record at a given point

• Delete: remove an old record
• Search: find a record with a given key

We will see a wide variety of different implementation of these operations over the
course of the semester.

How would you implement these using an array?

With linked lists, we can creating arbitrarily large structures, and never have to move any
items.

Most of these operations should be pretty simple now that you understand pointers!

Searching in a Linked List

Procedure Search(head:pointer, key:item):pointer;

Var
p:pointer;
found:boolean;
Begin
found:=false;
p:=head;
While (p # NIL) AND (not found) Do
Begin
If (p^.info = key) then
found = true;
Else
p = p^.next;
End;
return p;
END;

Search performs better when the item is near the front of the list than the back.

What happens when the item isn't found?

Insertion into a Linked List

The easiest way to insert a new node p into a linked list is to insert it at the front of the
list:

p^.next = front;
front = p;

To maintain lists in sorted order, however, we will want to insert a node between the two
appropriate nodes. This means that as we traverse the list we must keep pointers to both
the current node and the previous node.

MODULE Intlist; (*16.07.94. RM, LB*)

(* Implementation of sorted integer lists. *)

REVEAL (*reveal inner structure of T*)

T = BRANDED REF RECORD
key: INTEGER; (*key value*)
next: T := NIL; (*pointer to next element*)
END; (*T*)

PROCEDURE Create(): T =
(* returns a new, empty list *)
BEGIN
 RETURN NIL; (*creation is trivial; empty list is NIL*)
END Create;

PROCEDURE Insert(VAR list: T; value:INTEGER) =

(* inserts new element in list and maintains order *)
VAR
current, previous: T;
new: T := NEW(T, key:= value); (*create new element*)
BEGIN
IF list = NIL THEN list:= new (*first element*)
ELSIF value < list.key THEN (*insert at beginning*)
new.next:= list; list:= new;
ELSE (*find position for insertion*)
current:= list;
previous:= current;
WHILE (current # NIL) AND (current.key <= value) DO
previous:= current; (*previous hobbles after*)
current:= current.next;
END;
(*after the loop previous points to the insertion point*)
new.next:= current;
(*current = NIL if insertion point is the end*)
previous.next:= new; (*insert new element*)
END; (*IF list = NIL*)
END Insert;

Make sure you understand where these cases come from and can verify why all of them
work correct.

Deletion of a Node

To delete a node from a singly linked-list, we must have pointers to both the node-to-die
and the previous node, so we can reconnect the rest of the list.

PROCEDURE Remove(VAR list: T; value:INTEGER; VAR found: BOOLEAN) =

(* deletes (first) element with value from sorted list,
or returns false in found if the element was not found *)
VAR
current, previous: T;
BEGIN
IF list = NIL THEN found:= FALSE
ELSE (*start search*)
current:= list; previous:= current;
WHILE (current # NIL) AND (current.key # value) DO
previous:= current; (*previous hobbles after*)
current:= current.next;
END;
(*holds: current = NIL or current.key = value, but not both*)
IF current = NIL THEN
found:= FALSE (*value not found*)
ELSE
found:= TRUE; (*value found*)
IF current = list THEN
list:= current.next (*element found at beginning*)
ELSE
 previous.next:= current.next
END;
END; (*IF current = NIL*)
END; (*IF list = NIL*)
END Remove;

Passing Procedures as Arguments

Note the passing of a procedure as a parameter - it is legal, and useful to make more
general functions, for example a sort routine for both increasing and decreasing order, or
any order.

PROCEDURE Iterate(list: T; action: Action) =

(* applies action to all elements (with key value as parameter) *)
BEGIN
WHILE list # NIL DO
action(list.key);
list:= list.next;
END;
END Iterate;

BEGIN (* Intlist *)
END Intlist.

Pointers and Parameter Passing

Pointers provide, for better or (usually) worse, and alternate way to modify parameters.
Let us look at two different ways to swap the ``values'' of two pointers.

Procedure Swap1(var p,q:pointer);

Var r:pointer;
begin
r:=q;
q:=p;
p:=r;
end;

This is perhaps the simplest and best way - we just exchange the values of the pointers...

Alternatively, we could swap the values of what is pointed to, and leave the pointers
unchanged.

Procedure Swap2(p,q : pointer);

var tmp : node;
begin
tmp := q^; (*1*)
q^ := p^; (*2*)
p^ := tmp; (*3*)
end;

After step (*1*):

After step (*2*):

After step (*3*):

Side Effects of Pointers

If swap2, since we do not change the values of p and q, they do not need to be var
parameters!

However, copying the values did not do the same thing as copying the pointers, because
in the first case the physical location of the data changed, while in the second the data
stayed put.

If data which is pointed to moves, the value of what is pointed to can change!

Moral: you must be careful about the side effects of pointer operations!!!

C language does not have var parameters. All side effects are done by passing pointers.
Additional pointer operations in C language help make this practical.

Doing the Shuffle

Lecture 8
Steven S. Skiena

Programming Style

Although programming style (like writing style) is a somewhat subjective thing, there is a
big difference between good and bad.

The good programmer doesn't just strive for something that works, but something that
works elegantly and efficiently; something that can be maintained and understood by
others.

Just like a good writer rereads and rewrites their prose, a good programmer rewrites their
program to make it cleaner and better.

To get a better sense of programming style, let's critique some representative solutions to
the card-shuffling assignment to see what's good and what can be done better.

Ugly looking main

MODULE card EXPORTS Main;
IMPORT SIO;
TYPE
index=[1..200];
Start=ARRAY index OF INTEGER;
Left=ARRAY index OF INTEGER;
Right=ARRAY index OF INTEGER;
Final=ARRAY index OF INTEGER;
VAR
i,j,times,mid,k,x: INTEGER;
start: Start;
left: Left;
right: Right;
final: Final;
BEGIN
SIO.PutText("deck size shuffles\n");
SIO.PutText("--------- --------\n");
SIO.PutText(" 200 ");
SIO.PutInt( times );

REPEAT (*Repeat the following until perfect shuffle*)

i:=1; (*original deck*)

WHILE i<=200 DO
start[i]:=i;
i:=i+1;
END;

j:=1; (*splits into two decks*)

mid:=100;
WHILE (j<=100) DO
left[j]:=start[j];
right[j]:=start[j+mid];
j:=j+1;
END;
x:=1;
k:=1; (*shuffle them into one deck*)
WHILE k<=200 DO
final[k]:=left[x];
final[k+1]:=right[x];
k:=k+2;
END;

UNTIL start[2]=final[2]; (*check if complete shuffle*)

times:=times+1;
END card.

There are no variable or block comments. This program would be hard to understand.

This is an ugly looking program - the structure of the program is not reflected by the
white space.

Indentation and blank lines appear to be added randomly.

There are no subroutines used, so everything is one big mess.

See how the dependence on the number of cards is used several times within the body of
the program, instead of just in one CONST.

What Does shufs Do?

PROCEDURE shufs( nn : INTEGER )= (* shuffling procedure *)

VAR
i : INTEGER; (* index variable *)
count : INTEGER; (* COUNT variable *)

BEGIN

FOR i := 1 TO 200 DO (* reset this array *)

shuffled[i] := i;
END;

count := 0; (* start counter from 0 *)

REPEAT
count := count + 1;
FOR i := 1 TO 200 DO (* copy shuffled ->
tempshuf *)
tempshuf[i] := shuffled[i];
END;

FOR i := 1 TO nn DO (* shuffle 1st half *)

shuffled[2*i-1] := tempshuf[i];
END;

FOR i := nn+1 TO 2*nn DO (* shuffle 2nd half *)

shuffled[2*(i-nn)] := tempshuf[i];
END;

UNTIL shuffled = unshuffled ; (* did it return to

original? *)

(* print out the data *)

Wr.PutText(Stdio.stdout , "2*n= " & Fmt.Int(2*nn) & " \t" );
Wr.PutText(Stdio.stdout , Fmt.Int(count) & "\n" );

END shufs;

Every subroutine should ``do'' something that is easily described. What does shufs do?

The solution to such problems is to write the block comments for the subroutine does
before writing the subroutine.

If you can't easily explain what it does, you don't understand it.

How many comments are enough?

MODULE Shuffles EXPORTS Main;
IMPORT SIO;

TYPE
Array= ARRAY [1..200] OF INTEGER; (*Create an integer array from *)
(*1 to 200 and called Array *)
VAR
original, temp1, temp2: Array; (*Declare original,temp1 and *)
(*temp2 to be Array *)
counter: INTEGER; (*Declare counter to be integer*)

(********************************************************************)
(* This is a procedure called shuffle used to return a number of *)
(* perfect shuffle. It input a number from the main and run the *)
(* program with it and then return the final number of perfect shuffle
*)
(********************************************************************)

PROCEDURE shuffle(total: INTEGER) :INTEGER =

VAR
half, j, p: INTEGER; (*Declare half, j, p to be
integer *)
BEGIN
FOR j:= 1 TO total DO
original[j] := j;
temp1[j] := j;
END; (*for*)
half := total DIV 2;
REPEAT
j := 0;
p := 1;
REPEAT
j := j + 1;
temp2[p] := temp1[j]; (* Save the number from the first half
*)
(* of the original array into temp2
*)
p := p + 1;
temp2[p] := temp1[half+j]; (* Save the number from the last
half*)
(* of the original array into temp2
*)
p := p + 1;
UNTIL p = total + 1; (*REPEAT_UNTIL used to make a new array of
temp1*)
INC (counter); (* increament counter when they shuffle once
*)
FOR i := 1 TO total DO
temp1[i] := temp2[i];
END; (* FOR loop used to save all the elements from temp2 to
temp1 *)
UNTIL temp1 = original; (* REPEAT_UNTIL, when two array match
exactly *)
(* same then quick *)
RETURN counter; (* return the counter *)
END shuffle; (* end procedure shuffle *)
(********************************************************************)
(* This is the Main for shuffle program that prints out the numbers *)
(* of perfect shuffles necessary for a deck of 2n cards *)
(********************************************************************)

BEGIN
...
END Shuffles. (* end the main program called Shuffles *)

This program has many comments which should be obvious to anyone who can read
Modula-3.

More useful would be enhanced block comments telling you what the program is done
and how it works.

The ``is it completely reshuffled yet?'' test is done cleanly, although all of the 200 cards
are tested regardless of deck size.

The shuffle algorithm is too complicated. Algorithms must be pretty, too

MODULE prj1 EXPORTS Main;

IMPORT SIO;
CONST
n : INTEGER = 100; (*size of split deck*)

TYPE
nArray = ARRAY[1..n] OF INTEGER; (*n sized deck type*)
twonArray = ARRAY[1..2*n] OF INTEGER; (*2n sized deck type*)

VAR
merged : twonArray; (*merged deck*)
count : INTEGER;

PROCEDURE shuffle(size:INTEGER; VAR merged:twonArray)=

VAR
topdeck, botdeck : nArray; (*arrayed split decks*)
BEGIN
FOR i := 1 TO size DO
topdeck[i] := merged[i]; (*split entire deck*)
botdeck[i] := merged[i+size]; (*into top, bottom decks*)
END;
FOR j := 1 TO size DO
merged[2*j-1] := topdeck[j]; (*If odd then 2*n-1
position.*)
merged[2*j] := botdeck[j]; (*If even then 2*n position*)
END;
END shuffle;

PROCEDURE printout(count:INTEGER; size:INTEGER)=

BEGIN
SIO.PutInt(size);
SIO.PutText(" ");
 SIO.PutInt(count);
SIO.PutText(" \n");
END printout;

PROCEDURE checkperfect(merged:twonArray; i:INTEGER) : BOOLEAN=

VAR
size : INTEGER;
check : BOOLEAN;
BEGIN
check := FALSE;
size := 0;
REPEAT
INC(size, 1); (*check to see if*)
IF merged[size+1] - merged[size] = 1 THEN (*deck is perfectly*)
check := TRUE; (*shuffled, if so *)
END; (*card progresses by
1*)
UNTIL (check = FALSE OR size - 1 = i);
RETURN check;
END checkperfect;

Checkperfect is much more complicated than it need be; just check whether merged[i] =
i. You can return without the BOOLEAN variable.

A good thing is that the deck size is all a function of a CONST.

The shuffle is slightly wasteful of space - two extra full arrays instead of two extra half
arrays.

Why does this work correctly?

BEGIN
SIO.PutLine("Welcome to Paul's card shuffling program!");
SIO.PutLine(" DECK SIZE NUMBER OF SHUFFLES ");
SIO.PutLine(" _________________________________ ");
num_cards := 2;
REPEAT
counter := 0;
FOR i := 1 TO (num_cards) DO
deck[i] :=i;
END; (*initializes deck*)
REPEAT
deck := Shuffle(deck,num_cards);
INC(counter);
UNTIL deck[2] = 2;
SIO.PutInt(num_cards,16); SIO.PutInt(counter,19);
SIO.PutText("\n");
INC(num_cards,2);
(*increments the number of cards in deck by 2.*)
UNTIL ( num_cards = ((2*n)+2));
END ShuffleCards.
Why we know that this stopping condition suffices to get us all the cards in the right
position. This should be proven prior to use.

Why use a Repeat loop when For will do?

Program Defensively

I am starting to see the wreckage of several programs because students are not building
their programs to be debugged.

• Add useful debug print statements! Have your program describe what it is doing!
• Document what you think your program does! Otherwise, how do you know
whine it is doing it!
• Build your program in stages! Thus you localize your bugs, and make sure you
understand simple things before going on to complicated things.
• Use spacing to show the structure of your program. A good program is a pretty
program!

Recursive and Doubly Linked Lists

Lecture 9
Steven S. Skiena

Recursive List Implementation

The basic insertion and deletion routines for linked lists are more elegantly written using
recursion.

PROCEDURE Insert(VAR list: T; value:INTEGER) =

(* inserts new element in list and maintains order *)
VAR new: T; (*new node*)
BEGIN
IF list = NIL THEN
list:= NEW(T, key := value) (*list is empty*)
ELSIF value < list.key THEN (*proper place found: insert*)
new := NEW(T, key := value);
new.next := list;
list := new;
ELSE (*seek position for insertion*)
Insert(list.next, value);
END; (*IF list = NIL*)
END Insert;

PROCEDURE Remove(VAR list:T; value:INTEGER; VAR found:BOOLEAN) =

(* deletes (first) element with value from sorted list,
or returns false in found if the element was not found *)
 BEGIN
IF list = NIL THEN (*empty list*)
found := FALSE
ELSIF value = list.key THEN (*elemnt found*)
found := TRUE;
list := list.next
ELSE (*seek for the element to delete*)
Remove(list.next, value, found);
END;
END Remove;

Doubly Linked Lists

Often it is necessary to move both forward and backwards along a linked list. Thus we
need another pointer from each node, to make it doubly linked.

List types are analogous to dance structures:

• Conga line - singly linked list.

• Chorus line - doubly linked list.
• Hora circle - double linked circular list.

Extra pointers allow the flexibility to have both forward and backwards linked lists:

type
pointer = REF node;
node = record
info : item;
front : pointer;
back : pointer;
end;

Insertion

How do we insert p between nodes q and r in a doubly linked list?

p^.front = r;
p^.back = q;
r^.back = p;
q^.front = p;

It is not absolutely necessary to have pointer r, since r = q .front, but it makes it cleaner.

The boundary conditions are inserting before the first and after the last element.

How do we insert before the first element in a doubly linked list (head)?

p^.back = NIL;
p^.front = head;
head^.back = p;
head = p; (* must point to entire structure *)
Inserting at the end is similar, except head doesn't change, and a back pointer is set to
NIL.

Linked Lists: Pro or Con?

The advantages of linked lists include:

• Overflow can never occur unless the memory is actually full.

• Insertions and deletions are easier than for contiguous (array) lists.
• With large records, moving pointers is easier and faster than moving the items
themselves.

The disadvantages of linked lists include:

• The pointers require extra space.

• Linked lists do not allow random access.
• Time must be spent traversing and changing the pointers.
• Programming is typically trickier with pointers.

Recursion and Backtracking

Lecture 10
Steven S. Skiena

Recursion

Recursion is a wonderful, powerful way to solve problems.

Elegant recursive procedures seem to work by magic, but the magic is same reason
mathematical induction works!

Example: Prove .

For n=1, , so its true. Assume it is true up to n-1.

Example: All horses are the same color! (be careful of your basis cases!)
The Tower of Hanoi

MODULE Hanoi EXPORTS Main; (*18.07.94*)

(* Implementation of the game Towers of Hanoi. *)

PROCEDURE Transfer(from, to: Post) =

(*moves a disk from post "from" to post "to"*)
BEGIN
WITH f = posts[from], t = posts[to] DO
INC(t.top);
t.disks[t.top]:= f.disks[f.top];
f.disks[f.top]:= 0;
DEC(f.top);
END; (*WITH f, t*)
END Transfer;

PROCEDURE Tower(height:[0..Height] ; from, to, between: Post) =

(*Does the job through recursive calls on itself*)
BEGIN
IF height > 0 THEN
Tower(height - 1, from, between, to);
Transfer(from, to);
Display();
Tower(height - 1, between, to, from);
END;
END Tower;

BEGIN (*main program Hanoi*)

posts[Post.Start].top:= Height;
FOR h:= 1 TO Height DO
posts[Post.Start].disks[h]:= Height - (h - 1)
END;
Tower(Height, Post.Start, Post.Finish, Post.Temp);
END Hanoi.

To count the number of moves made,

Recursion not only made a complicated problem understandable, it made it easy to

understand.

Combinatorial Objects

Many mathematical objects have simple recursive definitions which can be exploited
algorithmically.

Example: How can we build all subsets of n items? Build all subsets of n-1 items, copy
the subsets, and add item n to each of the subsets in one copy but not the other.

Once you start thinking recursively, many things have simpler formulations, such as
traversing a linked list or binary search.
Gray codes

We saw how to generate subsets recursively. Now let us generate them in an interesting
order.

All subsets of can be represented as binary strings of length n, where bit

i tells whether i is in the subset or not.

Obviously, all subsets must differ in at least one element, or else they would be identical.
An order where they differ by exactly one from each other is called a Gray code.

For n=1, {},{1}.

For n=2, {},{1},{1,2},{2}.

For n=3, {},{1},{1,2},{2},{2,3},{1,2,3},{1,3},{3}

Recursive construction algorithm: Build a Gray Code of , make a

reverse copy of it, append n to each subset in the reverse copy, and stick the two together!

Formulating Recursive Programs

Think about the base cases, the small cases where the problem is simple enough to solve.

Think about the general case, which you can solve if you can solve the smaller cases.

Unfortunately, many of the simple examples of recursion are equally well done by
iteration, making students suspicious.

Further, many of these classic problems have hidden costs which make recursion seem
expensive, but don't be fooled!

Factorials

PROCEDURE Factorial (n: CARDINAL): CARDINAL =

BEGIN
IF n = 0 THEN
RETURN 1 (* trivial case *)
ELSE
RETURN n * Factorial(n-1) (* recursive branch *)
END (* IF*)
END Factorial;

Be sure you understand how the parameter passing mechanism works.

Would this program work if n was a VAR parameter?

Fibonacci Numbers

The Fibonacci numbers are given by the recurrence relation .

PROCEDURE Fibonacci(n : CARDINAL) : CARDINAL =

BEGIN (* Fibonacci *)
IF n <= 1 THEN
RETURN 1
ELSE
RETURN Fibonacci(n-1) + Fibonacci(n-2) (*n > 1*)
END (* IF *)
END Fibonacci;

How much time does this elementary Fibonacci function take?

Implementing Recursion

Part of the mystery of recursion is the question of how the machine keeps everything
straight.

How come local variables don't get trashed?

The answer is that whenever a procedure or function is called, the local variables are
pushed on a stack, so the new recursive call is free to use them.

When a procedure ends, the variables are popped off the stack to restore them to where
they were before the call.

Thus the space used is equal to the depth of the recursion, since stack space is reused.

Tail Recursion

Tail recursion costs space, but not time. It can be removed mechanically and is by some
compilers.

Moral: Do not be afraid to use recursion if the algorithm is efficient.

The overhead of recursion vs. maintaining your own stack is too small to worry about.

By being clever, you can sometimes save stack space. Consider the following variation of
Quicksort:

If (p-1 < h-p) then

 Qsort(1,p)

Qsort(p,h)

else

Qsort(p,h)

Qsort(1,p)

By doing the smaller half first, the maximum stack depth is in the worst case.

Applications of Recursion

You may say, ``I just want to get a job and make lots of money. What can recursion do
for me?

We will look at three applications

• Backtracking
• Game Tree Search
• Recursion Descent Compilation

The N-Queens Problem

Backtracking is a way to solve hard search problems.

For example, how can we put n queens on an board so that no two queens attack
each other?

Tree Pruning

Backtracking really pays off when we can prove a node early in the search tree.

Thus we need never look at its children, or grandchildren, or great....

We apply backtracking to big problems, so the more clever we are, the more time we
save.

There are total sets of eight squares but no two queens can be in the same row.
There are ways to place eight queens in different rows. However, since no two queens
can be in the same column, there are only 8! permutations of columns, or only 40,320
possibilities.

We must also be clever to test as quickly as possible the new queen does not violate a
diagonal constraint

Applications of Recursion
Lecture 11
Steven S. Skiena

Game Trees

Chess playing programs work by constructing a tree of all possible moves from a given
position, so as to select the best possible path.

The player alternates at each level of the tree, but at each node the player whose move it
is picks the path that is best for them.

A player has a forced loss if lead down a path where the other guy wins if they play
correctly.

This is a recursive problem since we can always maximize, by just changing perspective.

In a game like chess, we will never reach the bottom of the tree, so we must stop at a
particular depth.

Alpha-beta Pruning

Sometimes we don't have to look at the entire game tree to get the right answer:

No matter what the red score is, it cannot help max and thus need not be looked at.

An advanced strategy called alpha-beta running reduces search accordingly.

Recursive Descent Compilation

Compilers do two useful things

• They identify whether a program is legal in the language.

• They translate it into assembly language.

To do either, we need a precise description of the language, a BNF grammar which gives
the syntax. A grammar for Modula-3 is given throughout your text.

The language definition can be recursive!!

Our compiler will follow the grammar to break the program into smaller and smaller
pieces.

When the pieces get small enough, we can spit out the appropriate chunk of assembly
code.

To avoid getting into infinite loops, we place our trust in the fellow who wrote the
grammar. Proper design can ensure that there are no such troubles.

Abstraction and Modules

Lecture 12
Steven S. Skiena

Abstract Data Types

It is important to structure programs according to abstract data types: collections of data

with well-defined operations on it

Example: Stack or Queue. Data: A sequence of items Operations: Initialize, Empty?,

Full?, Push, Pop, Enqueue, Dequeue

Example: Infinite Precision Integers. Data: Linked list of digits with sign bit. Operations:
Print number, Read Number, Add, Subtract, Multiply, Divide, Exponent, Module,
Compare.

Abstract data types add clarity by separating the definitions from the implementations.

What Do We Want From Modules?

Separate Compilation - We should be able to break the program into smaller files.
Further, we shouldn't need the source for each Module to link it together, just the
compiled object code files.

Communicate Desired Information Between Modules - We should be able to define a

type or procedure in one module and use it in another.

Information Hiding - We should be able to define a type or procedure in one module and
forbid using it in another! Thus we can clearly separate the definition of an abstract data
type from its implementation!

Modula-3 supports all of these goals by separating interfaces (.i3 files) from
implementations (.m3 files).

Example: The Piggy Bank

Below is an interface file to:

INTERFACE PiggyBank; (*RM*)

(* Interface to a piggy bank:

You can insert money with "Deposit". The only other permissible
operation is smashing the piggy bank to get the ``money back''
The procedure "Smash" returns the sum of all deposited amounts
and makes the piggy bank unusable.
*)

PROCEDURE Deposit(cash: CARDINAL);

PROCEDURE Smash(): CARDINAL;

END PiggyBank.

Note that this interface does not reveal where or how the total value is stored, nor how to
initialize it.

These are issues to be dealt with within the implementation of the module.

Piggy Bank Implementation

MODULE PiggyBank; (*RM/CW*)

(* Implementation of the PiggyBank interface *)

VAR contents: INTEGER; (* state of the piggy bank *)

PROCEDURE Deposit(cash: CARDINAL) =

(* changes the state of the piggy bank *)
BEGIN
<*ASSERT contents >= 0*> (* piggy bank still okay? *)
contents := contents + cash
END Deposit;
PROCEDURE Smash(): CARDINAL =
VAR oldContents: CARDINAL := contents; (* contents before smashing *)
BEGIN
contents := -1; (* smash piggy bank *)
RETURN oldContents
END Smash;

BEGIN
contents := 0 (* initialization of state variables in body *)
END PiggyBank.

A Client Program for the Bank

MODULE Saving EXPORTS Main; (*RM*)

(* Client of the piggy bank:

In a loop the user is prompted for the amount of deposit.

Entering a negative amount smashes the piggy bank.
*)

FROM PiggyBank IMPORT Deposit, Smash;

FROM SIO IMPORT GetInt, PutInt, PutText, Nl, Error;

<*FATAL Error*>

VAR cash: INTEGER;

BEGIN (* Saving *)
PutText("Amount of deposit (negative smashes the piggy bank): \n");
REPEAT
cash := GetInt();
IF cash >= 0 THEN
Deposit(cash)
ELSE
PutText("The smashed piggy bank contained $");
PutInt(Smash());
Nl()
END;
UNTIL cash < 0
END Saving.

Interface File Conventions

Imports describe what procedures a given module makes available.

Exports describes what we are willing to make public, ultimately including the ``MAIN''
program.

By naming files with the same .m3 and .i3 names, the ``ezbuild'' make command can start
from the file with the main program, and final all other relevant files.
Ideally, the interface file should hide as much detail about the internal implementation of
a module from its users as possible. This is not easy without sophisticated language
features.

Hiding the Details

INTERFACE Fraction; (*RM*)

(* defines the data type for rational numbers *)

TYPE T = RECORD
num : INTEGER;
den : INTEGER;
END;

PROCEDURE Init (VAR fraction: T; num: INTEGER; den: INTEGER := 1);

(* Initialize "fraction" to be "num/den" *)

PROCEDURE Plus (x, y : T) : T; (* x + y *)

PROCEDURE Minus (x, y : T) : T; (* x - y *)
PROCEDURE Times (x, y : T) : T; (* x * y *)
PROCEDURE Divide (x, y : T) : T; (* x / y *)

PROCEDURE Numerator (x : T): INTEGER; (* returns the numerator of x

*)
PROCEDURE Denominator (x : T): INTEGER; (* returns the denominator of
x *)

END Fraction.

Note that there is a dilemma here. We must make type T public so these procedures can
use it, but would like to prevent users from accessing (or even knowing about) the fields
num and dem directly.

Subtypes and REFANY

Modula-3 permits one to declare subtypes of types, A <: B, which means that anything of
type A is of type B, but everything of type type B is not necessarily of type A.

This proves important in implementing advanced object-oriented features like

inheritance.

REFANY is a pointer type which is a supertype of any other pointer. Thus a variable of
type REFANY can store a copy of any other pointer.

This enables us to define public interface files without actually revealing the guts of the
fraction type implementation.

Fraction type with REFANY

INTERFACE FractionType; (*19.12.94. RM, LB*)

(* defines the data type of rational numbers, compare with Example
10.10! *)

TYPE T <: REFANY; (*T is a subtype of Refany; its structure is

hidden*)

PROCEDURE Create (numerator: INTEGER; denominator: INTEGER := 1): T;

PROCEDURE Plus (x, y : T) : T; (* x + y *)
PROCEDURE Minus (x, y : T) : T; (* x - y *)
PROCEDURE Mult (x, y : T) : T; (* x * y *)
PROCEDURE Divide (x, y : T) : T; (* x : y *)
PROCEDURE Numerator (x : T): INTEGER;
PROCEDURE Denominator (x : T): INTEGER;

END FractionType.

Somewhere within a module we must reveal the implementation of type T. This is done
with a REVEAL statement:

MODULE FractionType; (*19.12.94. RM, LB*)

(* Implementation of the type FractionType. Compare with Example
10.12. In this version the structure of elements of the type is
hidded in the interface. The structure is revealed here.
*)

REVEAL T = BRANDED REF RECORD (*opaque structure of T*)

num, den: INTEGER
END; (*T*)

...

The Key Idea about REFANY

With generic pointers, it becomes necessary for type checking to be done a run-time,
instead of at compile-time as done to date.

This gives more flexibility, but much more room for you to hang yourself. For example:

TYPE
Student = REF RECORD lastname,firstname:TEXT END;
Address = REF RECORD street:TEXT; number:CARDINAL END;

VAR
r1 : Student;
r2 := NEW(Student, firstname:="Julie", lastname:="Tall");
adr := NEW(Address, street:="Washington", number:="21");
any := REFANY;

BEGIN
any := r2; (* always a safe assignment *)
r1 := any; (* legal because any is of type student *)
adr := any; (* produces a run-time error, not compile-time
*)
You should worry about the ideas behind generic implementations (why does Modula-3
do it this way?) more than the syntactic details (how does Modula-3 let you do this?). It is
very easy to get overwhelmed by the detail.

Generic Types

When we think about the abstract data type ``Stack'' or ``Queue'', the implementation of
th the data structure is pretty much the same whether we have a stack of integers or reals.

Without generic types, we are forced to declare the type of everything at compile time.
Thus we need two distinct sets of functions, like PushInteger and PushReal for each
operation, which is waste.

Object-Oriented programming languages provide features which enable us to create

abstract data types which are more truly generic, making it cleaner and easier to reuse
code.

Object-Oriented Programming
Lecture 13
Steven S. Skiena

Why Objects are Good Things

Modules provide a logical grouping of procedures on a related topic.

Objects provide a logical grouping of data and associated operations.

The emphasis of modules is on procedures; the emphasis of objects is on data. Modules

are verbs followed by nouns: Push(S,x), while objects are nouns followed by verbs:
S.Push(x).

This provides only an alternate notation for dealing with things, but different notations
can sometimes make it easier to understand things - the history of Calculus is an example.

Objects do a great job of encapsulating the data items within, because the only access to
them is through the methods, or associated procedures.

Stack Object

MODULE StackObj EXPORTS Main; (*24.01.95. LB*)

(* Stack implemented as object type. *)
 IMPORT SIO;

TYPE
ET = INTEGER; (*Type of elements*)
Stack = OBJECT
top: Node := NIL; (*points to stack*)
METHODS
push(elem:ET):= Push; (*Push implements push*)
pop() :ET:= Pop; (*Pop implements pop*)
empty(): BOOLEAN:= Empty; (*Empty implements empty*)
END; (*Stack*)
Node = REF RECORD
info: ET; (*Stands for any
information*)
next: Node (*Points to the next node in
the stack
*)
END; (*Node*)

PROCEDURE Push(stack: Stack; elem:ET) =

(*stack: receiver object (self)*)
VAR
new: Node := NEW(Node, info:= elem); (*Element instantiate*)
BEGIN
new.next:= stack.top;
stack.top:= new; (*new element added to top*)
END Push;

PROCEDURE Pop(stack: Stack): ET =

(*stack: receiver object (self)*)
VAR first: ET;
BEGIN
first:= stack.top.info; (*Info copied from first
element*)
stack.top:= stack.top.next; (*first element removed*)
RETURN first
END Pop;

PROCEDURE Empty(stack: Stack): BOOLEAN =

(*stack: receiver object (self)*)
BEGIN
RETURN stack.top = NIL
END Empty;

VAR
stack1, stack2: Stack := NEW(Stack); (*2 stack objects created*)
i1, i2: INTEGER;
BEGIN
stack1.push(2); (*2 pushed onto stack1*)
stack2.push(6); (*6 pushed onto stack2*)
i1:= stack1.pop(); (*pop element from stack1*)
i2:= stack2.pop(); (*pop element from stack2*)
SIO.PutInt(i1);
SIO.PutInt(i2);
SIO.Nl();
END StackObj.
Object-Oriented Programming

Object-oriented programming is a popular, recent way of thinking about program

organization.

OOP is typically characterized by three major ideas:

• Encapsulation - objects incorporate both data and procedures.

• Inheritance - classes (object types) are arranged in a hierarchy, and each class
inherits but specializes methods and data from its ancestors.
• Polymorphism - a particular object can take on different types at different times.
We saw this with REFANY variables whose types depend upon what is assigned
it it (dynamic binding).

Inheritance

When we define an object type (class), we can specify that it be derived from (subtype to)
another class. For example, we can specialize the Stack object into a GarbageCan:

TYPE
GarbageCan = Stack OBJECT
OVERRIDES
pop():= Yech; (* Remove something from can?? *)
dump():= RemoveAll; (* Discard everything from can *)
END; (*GarbageCan*)

The GarbageCan type is a form of stack (you can still push in it the same way), but we
have modified the pop and dump methods.

This subtype-supertype relation defines a hierarchy (rooted tree) of classes. The

appropriate method for a given object is determined at run time (dynamic binding)
according to the first class at or above the current class to define the method.

OOP and the Calculator Program

How might object-oriented programming ideas have helped in writing the calculator
program?

Many of you noticed that the linked stack type was similar to the long integer type, and
wanted to reuse the code from one in another.

The following type hierarchy shows one way we could have exploited this, by creating
special stack methods push and pop, and overwriting the add and subtract methods for
general long-integers.

Philosophical issue: should Long-Integer be a subtype of Positive-Long-Integer or visa

versa?
Why didn't I urge you to do it this way? In my opinion, the complexity of mastering and
using the OOP features of Modula-3 would very much overwhelm the code savings from
such a small program. Object-oriented features differ significantly from language to
language, but the basic principles outlined here are fairly common.

However, you should see why inheritance can be a big win in organizing larger programs.

Simulations
Lecture 14
Steven S. Skiena

Simulations

Often, a system we are interested in may be too complicated to readily understand, and
too expensive or big to experiment with.

• What direction will an oil spill move in the Persian Gulf, given certain weather
conditions?
• How much will increases in the price of oil change the American unemployment
rate?
• Now much traffic can an airport accept before long delays become common?

We can often get good insights into hard problems by performing mathematical
simulations.

Scoring in Jai-alai

Jai-alai is a Basque variation of handball, which is important because you can bet on it in
Connecticut. What is the best way to bet?

The scoring system in use in Connecticut is very interesting. Eight players or teams
appear in each match, numbered 1 to 8. The players are arranged in a queue, and the top
two players in the queue play each other. The winner gets a point and keeps playing, the
loser goes to the end of the queue. Winner is the first one to get to 7 points.

This scoring obviously favors the low numbered players. For fairness, after the first trip
through the queue, each point counts two.

But now is this scoring system fair?

Simulating Jai-Alai
1 PLAYS 2
1 WINS THE POINT, GIVING HIM 1

1 PLAYS 3
3 WINS THE POINT, GIVING HIM 1

4 PLAYS 3
3 WINS THE POINT, GIVING HIM 2

5 PLAYS 3
3 WINS THE POINT, GIVING HIM 3

6 PLAYS 3
3 WINS THE POINT, GIVING HIM 4

7 PLAYS 3
3 WINS THE POINT, GIVING HIM 5

8 PLAYS 3
8 WINS THE POINT, GIVING HIM 1

8 PLAYS 2
2 WINS THE POINT, GIVING HIM 2

1 PLAYS 2
2 WINS THE POINT, GIVING HIM 4

4 PLAYS 2
2 WINS THE POINT, GIVING HIM 6

5 PLAYS 2
5 WINS THE POINT, GIVING HIM 2

5 PLAYS 6
5 WINS THE POINT, GIVING HIM 4

5 PLAYS 7
7 WINS THE POINT, GIVING HIM 2

3 PLAYS 7
7 WINS THE POINT, GIVING HIM 4

8 PLAYS 7
7 WINS THE POINT, GIVING HIM 6

1 PLAYS 7
7 WINS THE POINT, GIVING HIM 8

WIN-PLACE-SHOW IS 7 2 3

BETTER THAN AVERAGE TRIFECTAS: 1 TRIALS

WIN PLACE SHOW OCCURRENCES

7 2 3

Is the Scoring Fair?

How can we test if the scoring system is fair?

We can simulate a lot of games and see how often each player wins the game!

But when player A plays a point against player B, how do we decide who wins? If the
players are all equally matched, we can flip a coin to decide. We can use a random
number generator to flip the coin for us!

What data structures do we need?

• A queue to maintain the order of who is next to play.

• An array to keep track of each player's score during the game.
• A array to keep track of how often a player has won so far.

Simulation Results

Jai-alai Simulation Results

Pos win %wins place %places show %shows

1 16549 16.55 17989 17.99 15123 15.12
2 16207 16.21 17804 17.80 15002 15.00
3 13584 13.58 16735 16.73 14551 14.55
4 12349 12.35 13314 13.31 13786 13.79
5 10103 10.10 10997 11.00 13059 13.06
6 10352 10.35 7755 7.75 11286 11.29
7 9027 9.03 8143 8.14 9007 9.01
8 11829 11.83 7263 7.26 8186 8.19

total games = 100000

Compare these to the actual win results from Berenson's Jai-alai 1983-1986:

1 14.1%, 2 14.6%, 3 12.8%, 4 11.5%, 5 12.0%, 6 12.4%, 7 11.1%, 8 11.3%

Were these results good?

Yes, but not good enough to bet with! The matchmakers but the best players in the
middle, so as to even the results. A more complicated model will be necessary for better
results.

Limitations of Simulations

Although simulations are good things, there are several reasons to be skeptical of any
results we get.

Is the underlying model for the simulation accurate?

Are the implicit assumptions reasonable, or are there biases?

How do we know the program is an accurate implementation of the given model?

After all, we wrote the simulation because we do not know the answers! How do you
debug a simulation of two galaxies colliding or the effect of oil price increases on the
economy?

So much rides on the accuracy of simulations it is critical to build in self-verification

tests, and prove the correctness of implementation.

Random Number Generator

We have shown that random numbers are useful for simulations, but how do we get
them?

First we must realize that there is a philosophical problem with generating random
numbers on a deterministic machine.

``Anyone who considers arithmetical methods of producing random digits is , of course,

in a state of sin.'' - John Von Neumann

What we really want is a good way to generate pseudo-random numbers, a sequence

which has the same properties as a truly random source.

This is quite difficult - people are lousy at picking random numbers. Note that the
following sequence produces 0's + 1's with equal frequency but does not look like a fair
coin:

Even recognizing random sequences is hard. Are the digits of pseudo-random?

Should all those palindromes (535, 979, 46264, 383) be there?

The Middle Square Method

Von Neumann suggested generating random numbers by taking a big integer, squaring it,
and using the middle digits as the seed/random number.

It looks random to me... But what happens when the middle digits just happen to be
0000000000? From then on, all digits will be zeros!
Linear Congruential Generators

The most popular random number generators, because of simplicity, quality, and small
state requirements are linear congruential generators.

If is the last random number we generated, then

The quality of the numbers generated depends upon careful selection of the seed and
the constants a, c, and m.

Why does it work? Clearly, the numbers are between 0 and m-1. Taking the remainder
mod m is like seeing where a roulette ball drops in a wheel with m slots.

Asymptotics
Lecture 15
Steven S. Skiena

Analyzing Algorithms

There are often several different algorithms which correctly solve the same problem.
How can we choose among them? There can be several different criteria:

• Ease of implementation
• Ease of understanding
• Efficiency in time and space

The first two are somewhat subjective. However, efficiency is something we can study
with mathematical analysis, and gain insight as to which is the fastest algorithm for a
given problem.

Time Complexity of Programs

What would we like as the result of the analysis of an algorithm? We might hope for a
formula describing exactly how long a program implementing it will run.
Example: Binary search will take milliseconds on an array of n
elements.

This would be great, for we could predict exactly how long our program will take. But it
is not realistic for several reasons:

1. Dependence on machine type - Obviously, binary search will run faster on a

CRAY than a PC. Maybe binary search will now take ms?
2. Dependence on language/compiler - Should our time analysis change when
someone uses an optimizing compiler?
3. Dependence of the programmer - Two different people implementing the same
algorithm will result in two different programs, each taking slightly differed
amounts of time.
4. Should your time analysis be average or worst case? - Many algorithms return
answers faster in some cases than others. How did you factor this in? Exactly
what do you mean by average case?
5. How big is your problem? - Sometimes small cases must be treated different from
big cases, so the same formula won't work.

Time Complexity of Algorithms

For all of these reasons, we cannot hope to analyze the performance of programs
precisely. We can analyze the underlying algorithm, but at a less precise level.

Example: Binary search will use about iterations, where each iteration takes time
independent of n, to search an array of n elements in the worst case.

Note that this description is true for all binary search programs regardless of language,
machine, and programmer.

By describing the worst case instead of the average case, we saved ourselves some nasty
analysis. What is the average case?

Algorithms for Multiplications

Everyone knows two different algorithms for multiplication: repeated addition and digit-
by-digit multiplication.

Which is better? Let's analyze the complexity of multiplying an n-digit number by an m-

digit number, where .
In repeated addition, we explicity use that . Thus adding an
n-digit + m-digit number, requires ``about'' n+m steps, one for each digit.

How many additions can we do in the worst case? The biggest n-digit number is all nines,
and .

The total time complexity is the cost per addition times the number of additions, so the

total complexity .

Digit-by-Digit Multiplication

Since multiplying one digit by one other digit can be done by looking up in a
multiplication table (2D array), each step requires a constant amount of work.

Thus to multiply an n-digit number by one digit requires ``about'' n steps. With m ``extra''
zeros (in the worst case), ``about'' n + m steps certainly suffice.

We must do m such multiplications and add them up - each add costs as much as the
multiplication.

The total complexity is the cost-per-multiplication * number-of-multiplications + cost-

per-addition * number-of- multiplication .

Which is faster?

Clearly the repeated addition method is much slower by our analysis, and the difference
is going to increase rapidly with n...

Further, it explains the decline and fall of Roman empire - you cannot do digit-by-digit
multiplication with Roman numbers!

Growth Rates of Functions

To compare the efficiency of algorithms then, we need a notation to classify numerical
functions according to their approximate rate of growth.

We need a way of exactly comparing approximately defined functions. This is the big Oh
Notation:

If f(n) and g(n) are functions defined for positive integers, then f(n)= O(g(n)) means that

there exists a constant c such that for all sufficiently large positive
integers.

The idea is that if f(n)=O(g(n)), then f(n) grows no faster (and possibly slower) than g(n).

Note this definition says nothing about algorithms - it is just a way to compare numerical
functions!

Examples

Example: is . Why? For all n > 100, clearly

, so it satisfies the definition for c=100.

Example: is not . Why? No matter what value of c you pick,

is not true for n>c!

In the big Oh Notation, multiplicative constants and lower order terms are unimportant.
Exponents are important.

Ranking functions by the Big Oh

The following functions are different according to the big Oh notation, and are ranked in
increasing order:

O(1) Constant growth

 Logarithmic growth (note:independent of base!)

Polynomial growth: ordered by exponent

O(n) Linear Growth

Quadratic growth

Exponential growth

Why is the big Oh a Big Deal?

Suppose I find two algorithms, one of which does twice as many operations in solving the
same problem. I could get the same job done as fast with the slower algorithm if I buy a
machine which is twice as fast.

But if my algorithm is faster by a big Oh factor - No matter how much faster you make
the machine running the slow algorithm the fast-algorithm, slow machine combination
will eventually beat the slow algorithm, fast machine combination.

I can search faster than a supercomputer for a large enough dictionary, If I use binary
search and it uses sequential search!
An Application: The Complexity of Songs

Suppose we want to sing a song which lasts for n units of time. Since n can be large, we
want to memorize songs which require only a small amount of brain space, i.e. memory.

Let S(n) be the space complexity of a song which lasts for n units of time.

The amount of space we need to store a song can be measured in either the words or

characters needed to memorize it. Note that the number of characters is

since every word in a song is at most 34 letters long - Supercalifragilisticexpialidocious!

What bounds can we establish on S(n)? S(n) = O(n), since in the worst case we must
explicitly memorize every word we sing - ``The Star-Spangled Banner''

The Refrain

Most popular songs have a refrain, which is a block of text which gets repeated after each
stanza in the song:

Bye, bye Miss American pie

Drove my chevy to the levy but the levy was dry
Them good old boys were drinking whiskey and rye
Singing this will be the day that I die.

Refrains made a song easier to remember, since you memorize it once yet sing it O(n)
times. But do they reduce the space complexity?

Not according to the big oh. If

Then the space complexity is still O(n) since it is only halved (if the verse-size = refrain-
size):
The k Days of Christmas

To reduce S(n), we must structure the song differently.

Consider ``The k Days of Christmas''. All one must memorize is:

On the kth Day of Christmas, my true love gave to me,

On the First Day of Christmas, my true love gave to me, a partridge in a pear tree

But the time it takes to sing it is

If , then , so . 100 Bottles of Beer

What do kids sing on really long car trips?

n bottles of beer on the wall,

n bottles of beer.
You take one down and pass it around
n-1 bottles of beer on the ball.

All you must remember in this song is this template of size , and the current value
of n. The storage size for n depends on its value, but bits suffice.

This for this song, .

Uh-huh, uh-huh

Is there a song which eliminates even the need to count?

That's the way, uh-huh, uh-huh

I like it, uh-huh, huh

Reference: D. Knuth, `The Complexity of Songs', Comm. ACM, April 1984, pp.18-24
 Introduction to Sorting
Lecture 16
Steven S. Skiena

Sorting

Sorting is, without doubt, the most fundamental algorithmic problem

1. Supposedly, 25% of all CPU cycles are spent sorting

2. Sorting is fundamental to most other algorithmic problems, for example binary
search.
3. Many different approaches lead to useful sorting algorithms, and these ideas can
be used to solve many other problems.

What is sorting? It is the problem of taking an arbitrary permutation of n items and

rearranging them into the total order,

Knuth, Volume 3 of ``The Art of Computer Programming is the definitive reference of

sorting.

Issues in Sorting

Increasing or Decreasing Order? - The same algorithm can be used by both all we need
do is change to in the comparison function as we desire.

What about equal keys? - Does the order matter or not? Maybe we need to sort on
secondary keys, or leave in the same order as the original permutations.

What about non-numerical data? - Alphabetizing is sorting text strings, and libraries
have very complicated rules concerning punctuation, etc. Is Brown-Williams before or
after Brown America before or after Brown, John?

We can ignore all three of these issues by assuming a comparison function which
depends on the application. Compare (a,b) should return ``<'', ``>'', or ''=''.

Applications of Sorting

One reason why sorting is so important is that once a set of items is sorted, many other
problems become easy.
SearchingBinary search lets you test whether an item is in a dictionary in time.

Speeding up searching is perhaps the most important application of sorting.

Closest pairGiven n numbers, find the pair which are closest to each other.

Once the numbers are sorted, the closest pair will be next to each other in sorted order, so
an O(n) linear scan completes the job.

Element uniquenessGiven a set of n items, are they all unique or are there any duplicates?

Sort them and do a linear scan to check all adjacent pairs.

This is a special case of closest pair above.

Frequency distribution - ModeGiven a set of n items, which element occurs the largest
number of times?

Sort them and do a linear scan to measure the length of all adjacent runs.

Median and SelectionWhat is the kth largest item in the set?

Once the keys are placed in sorted order in an array, the kth largest can be found in
constant time by simply looking in the kth position of the array.

How do you sort?

There are several different ideas which lead to sorting algorithms:

• Insertion - putting an element in the appropriate place in a sorted list yields a

larger sorted list.
• Exchange - rearrange pairs of elements which are out of order, until no such pairs
remain.
• Selection - extract the largest element form the list, remove it, and repeat.
• Distribution - separate into piles based on the first letter, then sort each pile.
• Merging - Two sorted lists can be easily combined to form a sorted list.

Selection Sort

In my opinion, the most natural and easiest sorting algorithm is selection sort, where we
repeatedly find the smallest element, move it to the front, then repeat...

* 5 7 3 2 8
2 * 7 3 5 8
 2 3 * 7 5 8
2 3 5 * 7 8
2 3 5 7 * 8

If elements are in an array, swap the first with the smallest element- thus only one array is
necessary.

If elements are in a linked list, we must keep two lists, one sorted and one unsorted, and
always add the new element to the back of the sorted list.

Selection Sort Implementation

MODULE SimpleSort EXPORTS Main; (*1.12.94. LB*)

(* Sorting and text-array by selecting the smallest element *)

TYPE
Array = ARRAY [1..N] OF TEXT;
VAR
a: Array; (*the array in which to search*)
x: TEXT; (*auxiliary variable*)
last, (*last valid index *)
min: INTEGER; (* current minimum*)

BEGIN

...

FOR i:= FIRST(a) TO last - 1 DO

min:= i; (*index of smallest element*)
FOR j:= i + 1 TO last DO
IF Text.Compare(a[j], a[min]) = -1 THEN (*IF a[i] < a[min]*)
min:= j
END;
END; (*FOR j*)
x:= a[min]; (* swap a[i] and a[min] *)
a[min]:= a[i];
a[i]:= x;
END; (*FOR i*)

...

END SimpleSort.

The Complexity of Selection Sort

One interesting observation is that selection sort always takes the same time no matter
what the data we give it is! Thus the best case, worst case, and average cases are all the
same!

Intuitively, we make n iterations, each of which ``on average'' compares n/2, so we

should make about comparisons to sort n items.

To do this more precisely, we can count the number of comparisons we make.

To find the largest takes (n-1) steps, to find the second largest takes (n-2) steps, to find
the third largest takes (n-3) steps, ... to find the last largest takes 0 steps.

An advantage of the big Oh notation is that fact that the worst case time is
obvious - we have n loops of at most n steps each.

If instead of time we count the number of data movements, there are n-1, since there is
exactly one swap per iteration.

Insertion Sort

In insertion sort, we repeatedly add elements to a sorted subset of our data, inserting the
next element in order:

* 5 7 3 2 8
5 * 7 3 2 8
3 5 * 7 2 8
2 3 5 * 7 8
2 3 5 7 * 8

InsertionSort(A)

for i = 1 to n-1 do

j=i
 while (A[j] > A[j-1]) do swap(A[j],A[j-
1])

In inserting the element in the sorted section, we might have to move many elements to
make room for it.

If the elements are in an array, we scan from bottom to top until we find the j such that

, then move from j+1 to the end down one to make room.

If the elements are in a linked list, we do the sequential search until we find where the
element goes, then insert the element there. No other elements need move!

Complexity of Insertion Sort

Since we do not necessarily have to scan the entire sorted section of the array, the best,
worst, and average cases for insertion sort all differ!

Best case: the element always gets inserted at the end, so we don't have to move
anything, and only compare against the last sorted element. We have (n-1) insertions,
each with exactly one comparison and no data moves per insertion!

What is this best case permutation? It is when the array or list is already sorted! Thus
insertion sort is a great algorithm when the data has previously been ordered, but slightly
messed up.

Worst Case Complexity

Worst case: the element always gets inserted at the front, so all the sorted elements must
be moved at each insertion. The ith insertion requires (i-1) comparisons and moves so:

What is the worst case permutation? When the array is sorted in reverse order.

This is the same number of comparisons as with selection sort, but uses more movements.
The number of movements might get important if we were sorting large records.

Average Case Complexity

Average Case: If we were given a random permutation, the chances of the ith insertion

requiring comparisons are equal, and hence 1/i.

The expected number of comparisons is for the ith insertion is:

Summing up over all n keys,

So we do half as many comparisons/moves on average!

Can we use binary search to help us get below time?

Mergesort and Quicksort

Lecture 17
Steven S. Skiena

Faster than O( ) Sorting?

Can we find a sorting algorithm which does significantly better than comparing each pair
of elements? If not, we are doomed to quadratic time complexity....

Since sorting large numbers of items is such an important problem - an

algorithm is the way to go!

Logarithms
It is important to understand deep in your bones what logarithms are and where they
come from.

A logarithm is simply an inverse exponential function. Saying is equivalent to

saying that .

Exponential functions, like the amount owed on a n year mortgage at an interest rate of

per year, are functions which grow distressingly fast. Thus inverse exponential
functions, ie. logarithms, grow refreshingly slowly.

Binary search is an example of an algorithm. After each comparison, we can

throw away half the possible number of keys. Thus twenty comparisons suffice to find
any name in the million-name Manhattan phone book!

If you have an algorithm which runs in time, take it, because this is blindingly
fast even on very large instances.

Properties of Logarithms

Recall the definition, .

Asymptotically, the base of the log does not matter:

Thus, , and note that

is just a constant.

Asymptotically, any polynomial function of n does not matter:Note that

since , and

.
Any exponential dominates every polynomial. This is why we will seek to avoid
exponential time algorithms.

Federal Sentencing Guidelines

2F1.1. Fraud and Deceit; Forgery; Offenses Involving Altered or Counterfeit Instruments
other than Counterfeit Bearer Obligations of the United States.

(a) Base offense Level: 6

(b) Specific offense Characteristics

(1) If the loss exceeded $2,000, increase the offense level as follows:

The federal sentencing guidelines are designed to help judges be consistent in assigning
punishment. The time-to-serve is a roughly linear function of the total level.

However, notice that the increase in level as a function of the amount of money you steal
grows logarithmically in the amount of money stolen.
This very slow growth means it pays to commit one crime stealing a lot of money, rather
than many small crimes adding up to the same amount of money, because the time to
serve if you get caught is much less.

The Moral: ``if you are gonna do the crime, make it worth the time!''

Mergesort

Given two sorted lists with a total of n elements, at most n-1 comparisons are required to
merge them into one sorted list. Repeatedly compare the top elements on each list.

Example: and .

No more comparisons are needed once the list is empty.

Fine, but how do we get the smaller sorted lists to start with? We do merges of even
smaller lists!

Working backwards, we eventually get to lists of one element, which are by definition
sorted!

Mergesort Example

Note that on each iteration, the size of the sorted lists doubles, form 1 to 2 to 4 to 8 to 16
...to n.
How many doublings (or iterations) does it take before the entire array of size n is sorted?
Answer: .

How much work do we do per iteration?

In merging the lists of 1 element, we have merges, each requiring 1 comparison,

for a total of comparisons.

In merging the lists of 2 elements, we have merges, each requiring at most 3

comparisons, for a total of comparisons.

...

In merging the lists of 2 elements, we have merges, each requiring at most

comparisons, for a total of .

This is always less than n per stage!!! If we make at most n comparisons in each of

stages, we make at most comparisons in total!

Make sure you understand why mergesort is - it is the conceptually

simplest algorithm we will see.

Space Requirements for Mergesort

How much extra space (over the space used to represent the input elements) do we need
to do mergesort?

It is easy to merge two sorted linked lists without using any extra space.

However, to merge two sorted arrays (or portions of an array), we must use a third array
to store the result of the merge. This avoids steping on elements we have not needed yet:

Example: Merge ((4,5,6), (1,2,3)).

QuickSort
Although Mergesort is , it is somewhat inconvienient to implementate
using arrays, since we need space to merge.

In practice, the fastest sorting algorithm is Quicksort, which uses partitioning as its main
idea.

Example: Pivot about 10.

17 12 6 19 23 8 5 10 - before

6 8 5 10 23 19 12 17 - after

Partitioning places all the elements less than the pivot in the left part of the array, and all
elements greater than the pivot in the right part of the array. The pivot fits in the slot
between them.

Note that the pivot element ends up in the correct place in the total order!

Partitioning the elements

Once we have selected a pivot element, we can partition the array in one linear scan, by
maintaining three sections of the array: < pivot, > pivot, and unexplored.

Example: pivot about 10

| 17 12 6 19 23 8 5 | 10
| 5 12 6 19 23 8 | 17
5 | 12 6 19 23 8 | 17
5 | 8 6 19 23 | 12 17
5 8 | 6 19 23 | 12 17
5 8 6 | 19 23 | 12 17
5 8 6 | 23 | 19 12 17
5 8 6 ||23 19 12 17
5 8 6 10 19 12 17 23

As we scan from left to right, we move the left bound to the right when the element is
less than the pivot, otherwise we swap it with the rightmost unexplored element and
move the right bound one step closer to the left.

Since the partitioning step consists of at most n swaps, takes time linear in the number of
keys. But what does it buy us?

1. The pivot element ends up in the position it retains in the final sorted order.
2. After a partitioning, no element flops to the other side of the pivot in the final
sorted order.
Thus we can sort the elements to the left of the pivot and the right of the pivot
independently!

This gives us a recursive sorting algorithm, since we can use the partitioning approach to
sort each subproblem.

Quicksort Implementation

MODULE Quicksort EXPORTS Main; (*18.07.94. LB*)

(* Read in an array of integers, sort it using the Quicksort algorithm,
and output the array.

See Chapter 14 for the explanation of the file handling and Chapter
15 for exception handling, which is used in this example.
*)

IMPORT SIO, SF;

VAR
out: SIO.Writer;

TYPE
ElemType = INTEGER;
VAR
array: ARRAY [1 .. 10] OF ElemType;

PROCEDURE InArray(VAR a: ARRAY OF ElemType) RAISES {SIO.Error} =

(*Reads a sequence of numbers. Passes SIO.Error for bad file
format.*)
VAR
in:= SF.OpenRead("vector"); (*open input file*)
BEGIN
FOR i:= FIRST(a) TO LAST(a) DO a[i]:= SIO.GetInt(in) END;
END InArray;

PROCEDURE OutArray(READONLY a: ARRAY OF ElemType) =

(*Outputs an array of numbers*)
BEGIN
FOR i:= FIRST(a) TO LAST(a) DO SIO.PutInt(a[i], 4, out) END;
SIO.Nl(out);
END OutArray;

PROCEDURE Quicksort(VAR a: ARRAY OF ElemType; left, right: CARDINAL)

=
VAR
i, j: INTEGER;
x, w: ElemType;
BEGIN

(*Partitioning:*)
i:= left; (*i iterates upwards from
left*)
j:= right; (*j iterates down from
right*)
x:= a[(left + right) DIV 2]; (*x is the middle element*)
 REPEAT
WHILE a[i] < x DO INC(i) END; (*skip elements < x in left
part*)
WHILE a[j] > x DO DEC(j) END; (*skip elements > x in right
part*)
IF i <= j THEN
w:= a[i]; a[i]:= a[j]; a[j]:= w; (*swap a[i] and a[j]*)
INC(i);
DEC(j);
END; (*IF i <= j*)
UNTIL i > j;

(*recursive application of partitioning to subarrays:*)

IF left < j THEN Quicksort(a, left, j) END;

IF i < right THEN Quicksort(a, i, right) END;

END Quicksort;

BEGIN
TRY (*grasps bad file
format*)
InArray(array); (*read an array in*)
out:= SF.OpenWrite(); (*create output file*)
OutArray(array); (*output the array*)
Quicksort(array, 0, NUMBER(array) - 1); (*sort the array*)
OutArray(array); (*display the array*)
SF.CloseWrite(out); (*close output file to
make it permanent*)
EXCEPT
SIO.Error => SIO.PutLine("bad file format");
END; (*TRY*)
END Quicksort.

Best Case for Quicksort

Since each element ultimately ends up in the correct position, the algorithm correctly
sorts. But how long does it take?

The best case for divide-and-conquer algorithms comes when we split the input as evenly
as possible. Thus in the best case, each subproblem is of size n/2.

The partition step on each subproblem is linear in its size. Thus the total effort in

partitioning the problems of size is O(n).

The recursion tree for the best case looks like this:
The total partitioning on each level is O(n), and it take levels of perfect partitions to
get to single element subproblems. When we are down to single elements, the problems

are sorted. Thus the total time in the best case is .

Worst Case for Quicksort

Suppose instead our pivot element splits the array as unequally as possible. Thus instead
of n/2 elements in the smaller half, we get zero, meaning that the pivot element is the
biggest or smallest element in the array.

Now we have n-1 levels, instead of , for a worst case time of , since the

first n/2 levels each have elements to partition.

Thus the worst case time for Quicksort is worse than Heapsort or Mergesort.

To justify its name, Quicksort had better be good in the average case. Showing this
requires some fairly intricate analysis.

The divide and conquer principle applies to real life. If you will break a job into pieces, it
is best to make the pieces of equal size!

Intuition: The Average Case for Quicksort

The book contains a rigorous proof that quicksort is in the average case. I
will instead give an intuitive, less formal explanation of why this is so.

Suppose we pick the pivot element at random in an array of n keys.

Half the time, the pivot element will be from the center half of the sorted array.

Whenever the pivot element is from positions n/4 to 3n/4, the larger remaining subarray
contains at most 3n/4 elements.

If we assume that the pivot element is always in this range, what is the maximum number
of partitions we need to get from n elements down to 1 element?
good partitions suffice.

At most levels of decent partitions suffices to sort an array of n elements.

But how often when we pick an arbitrary element as pivot will it generate a decent
partition?

Since any number ranked between n/4 and 3n/4 would make a decent pivot, we get one
half the time on average.

If we need levels of decent partitions to finish the job, and half of random
partitions are decent, then on average the recursion tree to quicksort the array has

levels.

Since O(n) work is done partitioning on each level, the average time is .

More careful analysis shows that the expected number of comparisons is

What is the Worst Case?

The worst case for Quicksort depends upon how we select our partition or pivot element.
If we always select either the first or last element of the subarray, the worst-case occurs
when the input is already sorted!

A B D F H J K
B D F H J K
D F H J K
F H J K
H J K
J K
K

Having the worst case occur when they are sorted or almost sorted is very bad, since that
is likely to be the case in certain applications.

To eliminate this problem, pick a better pivot:

1. Use the middle element of the subarray as pivot.

2. Use a random element of the array as the pivot.
3. Perhaps best of all, take the median of three elements (first, last, middle) as the
pivot. Why should we use median instead of the mean?
Whichever of these three rules we use, the worst case remains . However,
because the worst case is no longer a natural order it is much more difficult to occur.

Is Quicksort really faster than Mergesort?

Since Mergesort is and selection sort is , there is no debate about

which will be better for decent-sized files.

But how can we compare two algorithms to see which is faster? Using the
RAM model and the big Oh notation, we can't!

When Quicksort is implemented well, it is typically 2-3 times faster than mergesort or
heapsort. The primary reason is that the operations in the innermost loop are simpler. The
best way to see this is to implement both and experiment with different inputs.

Since the difference between the two programs will be limited to a multiplicative
constant factor, the details of how you program each algorithm will make a big
difference.

If you don't want to believe me when I say Quicksort is faster, I won't argue with you. It
is a question whose solution lies outside the tools we are using. The best way to tell is to
implement them and experiment.

Combining Quicksort and Insertion Sort

When we compare the expected number of comparisons for Quicksort + Insertion sort, a
funny thing happens for small n:

Why not take advantage of this, and switch over to insertion sort when the size of the
subarray falls below a certain threshhold?

Why not indeed? But how do we find the right switch point to optimize performance?
Experiments are more useful than analysis here.

Randomization
Suppose you are writing a sorting program, to run on data given to you by your worst
enemy. Quicksort is good on average, but bad on certain worst-case instances.

If you used Quicksort, what kind of data would your enemy give you to run it on?
Exactly the worst-case instance, to make you look bad.

But instead of picking the median of three or the first element as pivot, suppose you
picked the pivot element at random.

Now your enemy cannot design a worst-case instance to give to you, because no matter
which data they give you, you would have the same probability of picking a good pivot!

Randomization is a very important and useful idea. By either picking a random pivot or
scrambling the permutation before sorting it, we can say:

``With high probability, randomized quicksort runs in time.''

Where before, all we could say is:

``If you give me random input data, quicksort runs in expected time.''

Since the time bound how does not depend upon your input distribution, this means that
unless we are extremely unlucky (as opposed to ill prepared or unpopular) we will
certainly get good performance.

Randomization is a general tool to improve algorithms with bad worst-case but good
average-case complexity.

The worst-case is still there, but we almost certainly won't see it.

Priority Queues and Heapsort

Lecture 18
Steven S. Skiena

Who's Number 2?

In most sports playoffs, a single elimination tournament is used to decide the

championship.
The Marlins were clearly the best team in the 1997 World Series, since they were the
only one without a loss. But who is number 2? The Giants, Braves, and Indians all have
equal claims, since only the champion beat them!

Each game can be thought of as a comparison. Given n keys, we would like to determine
the k largest values. Can we do better than just sorting all of them?

In the tournament example, each team represents an leaf of the tree and each game is an
internal node of the tree. Thus there are n-1 games/comparisons for n teams/leaves.

Note that the champion is identified even though no team plays more than
games!

Lewis Carroll, author of ``Alice in Wonderland'', studied this problem in the 19th century
in order to design better tennis tournaments!

We will seek a data structure which will enable us to repeatedly identify the largest key,
and then delete it to retrieve the largest remaining key.

This data structure is called a heap, as in ``top of the heap''.

Binary Heaps

A binary heap is defined to be a binary tree with a key in each node such that:

1. All leaves are on, at most, two adjacent levels.

2. All leaves on the lowest level occur to the left, and all levels except the lowest are
completely filled.
3. The key in the root is all its children, and the left and right subtrees are again
binary heaps. (This is a recursive definition)

Conditions 1 and 2 specify the shape of the tree, while condition 3 describes the labeling
of the nodes tree.

Unlike the tournament example, each label only appears on one node.

Note that heaps are not binary search trees, but they are binary trees.

Heap Test

Where is the largest element in a heap?

Answer - the root.

Where is the second largest element?

Answer - as the root's left or right child.

Where is the smallest element?

Answer - it is one of the leaves.

Can we do a binary search to find a particular key in a heap?

Answer - No! A heap is not a binary search tree, and cannot be effectively used for
searching.

Why Do Heaps Lean Left?

As a consequence of the structural definition of a heap, each of the n items can be

assigned a number from 1 to n with the property that the left child of node number k has a
number 2k and the right child number 2k+1.

Thus we can store the heap in an n element array without pointers!

If we did not enforce the left constraint, we might have holes, and need room for
elements to store n things.

This implicit representation of trees saves memory but is less flexible than using pointers.
For this reason, we will not be able to use them when we discuss binary search trees.

Constructing Heaps

Heaps can be constructed incrementally, by inserting new elements into the left-most
open spot in the array.

If the new element is greater than its parent, swap their positions and recur.

Since at each step, we replace the root of a subtree by a larger one, we preserve the heap
order.

Since all but the last level is always filled, the height h of an n element heap is bounded
because:
so .

Doing n such insertions takes , since each insertion takes at most

time.

Deleting the Root

The smallest (or largest) element in the heap sits at the root.

Deleting the root can be done by replacing the root by the nth key (which must be a leaf)
and letting it percolate down to its proper position!

The smallest element of (1) the root, (2) its left child, and (3) its right child is moved to
the root. This leaves at most one of the two subtrees which is not in heap order, so we
continue one level down.

After steps of O(1) time each, we reach a leaf, so the deletion is completed in

time.

This percolate-down operation is called often Heapify, for it merges two heaps with a
new root.

Heapsort

An initial heap can be constructed out on n elements by incremental insertion in

time:

Build-heap(A)

for i = 2 to n do

HeapInsert(A[i], A)
Exchanging the maximum element with the last element and calling heapify repeatedly

gives an sorting algorithm, named Heapsort.

Heapsort(A)

Build-heap(A)

for i = n to 1 do

swap(A[1],A[i])

n = n - 1

Heapify(A,1)

Advantages of heapsort include:

• No extra space (Quicksort needs a stack)

• No worst case trouble.
• Simpler to get fast and correct than Quicksort.

The Lesson of Heapsort

Always ask yourself, ``Can we use a different data structure?''

Selection sort scans throught the entire array, repeatedly finding the smallest remaining
element.
For i = 1 to n

A: Find the smallest of the first n-i+1 items.

B: Pull it out of the array and put it first.

Using arrays or unsorted linked lists as the data structure, operation A takes O(n) time and
operation B takes O(1).

Using heaps, both of these operations can be done within time, balancing the
work and achieving a better tradeoff.

Priority Queues

A priority queue is a data structure on sets of keys supporting the operations: Insert(S, x)
- insert x into set S, Maximum(S) - return the largest key in S, and ExtractMax(S) - return
and remove the largest key in S

These operations can be easily supported using a heap.

• Insert - use the trickle up insertion in .

• Maximum - read the first element in the array in O(1).
• Extract-Max - delete first element, replace it with the last, decrement the element

counter, then heapify in .

Application: Heaps as stacks or queues

• In a stack, push inserts a new item and pop removes the most recently pushed
item.
• In a queue, enqueue inserts a new item and dequeue removes the least recently
enqueued item.

Both stacks and queues can be simulated by using a heap, when we add a new time field
to each item and order the heap according it this time field.

• To simulate the stack, increment the time with each insertion and put the
maximum on top of the heap.
• To simulate the queue, decrement the time with each insertion and put the
maximum on top of the heap (or increment times and keep the minimum on top)
This simulation is not as efficient as a normal stack/queue implementation, but it is a cute
demonstration of the flexibility of a priority queue.

Discrete Event Simulations

In simulations of airports, parking lots, and jai-alai - priority queues can be used to
maintain who goes next.

In a simulation, we often need to schedule events according to a clock. When someone is

born, we may then immediately decide when they will die, and we will have to be
reminded when to bury them!

The stack and queue orders are just special cases of orderings. In real life, certain people
cut in line.

Sweepline Algorithms in Computational Geometry

In the priority queue, we will store the points we have not yet encountered, ordered by x
coordinate. and push the line forward one stop at a time.

Greedy Algorithms

In greedy algorithms, we always pick the next thing which locally maximizes our score.
By placing all the things in a priority queue and pulling them off in order, we can
improve performance over linear search or sorting, particularly if the weights change.

Example: Sequential strips in triangulations.

Sequential and Binary Search

Lecture 19
Steven S. Skiena

Sequential Search

The simplest algorithm to search a dictionary for a given key is to test successively
against each element.

This works correctly regardless of the order of the elements in the list. However, in the
worst case all elements might have to be tested.

Procedure Search(head:pointer, key:item):pointer;

Var
 p:pointer;
found:boolean;
Begin
found:=false;
p:=head;
While (p # NIL) AND (not found) Do
Begin
If (p^.info = key) then
found = true;
Else
p = p^.next;
End;
return p;
END;

With and Without Sentinels

A sentinel is a value placed at the end of an array to insure that the normal case of
searching returns something even if the item is not found. It is a way to simplify coding
by eliminating the special case.

MODULE LinearSearch EXPORTS Main; (*1.12.94. LB*)

(* Linear search without a sentinel *)

...

i:= FIRST(a);
WHILE (i <= last) AND NOT Text.Equal(a[i], x) DO INC(i) END;

IF i > last THEN

SIO.PutText("NOT found");
ELSE
SIO.PutText("Found at position: ");
SIO.PutInt(i)
END; (*IF i > last*)
SIO.Nl();
END LinearSearch.

The sentinel insures that the search will eventually succeed:

MODULE SentinelSearch EXPORTS Main; (*27.10.93. LB*)

(* Linear search with sentinel. *)

...

(* Do search *)
a[LAST(a)]:= x; (*sentinel at position N+1*)
i:= FIRST(a);
WHILE x # a[i] DO INC(i) END;

(* Output result *)
IF i = LAST(a) THEN
SIO.PutText("NOT found");
 ELSE
SIO.PutText("Found at position: "); SIO.PutInt(i)
END;
SIO.Nl();
END SentinelSearch.

Weighted Sequential Search

Sometimes sequential search is not a bad algorithm, especially when the list isn't long.
After all, sequential search is easier to implement than binary search, and does not require
the list to be sorted.

However, if we are going to do a sequential search, what order do we want the elements?
Sorted order in a linked list doesn't really help, except maybe to help us stop early if the
item isn't in the list.

Suppose you were organizing your personal phone book for sequential search. You
would want your most frequently called friends to be at the front: In sequential search,
you want the keys ordered by frequency of use!

Why? If is the probability of searching for the ith key, which is a distance from the
front, the expected search time is

which is minimized by placing the list in decreasing probability of access order.

For the list (Cheryl,0.4), (Lisa,0.25), (Lori,0.2), (Lauren,0.15), the expected search time
is:

If access probability had been uniform, the expected search time would have been

So I win using this order, and win even more if the access probabilities are furthered
skewed.

But how do I find the probabilities?

Self-Organizing Lists
Since it is often impractical to compute usage frequencies, and because usage frequencies
often change in the middle of a program (locality), we would like our data structure to
automatically adjust to the distribution.

Such data structures are called self-organizing.

The idea is to use a heuristic to move an element forward in the list whenever it is
accessed. There are two possibilities:

• Move forward one is the ``conservative'' approach. (1,2,3,4,5) becomes (1,2,4,3,5)

after a Find(4).
• Move to front is the ``liberal'' approach. (1,2,3,4,5) becomes (4,1,2,3,5) after a
Find(4).

Which Heuristic is Better?

Move-forward-one can get caught in traps which won't fool move-to-front:

For list (1,2,3,4,5,6,7,8), the queries Find(8), Find(7), Find(8), Find(7), ... will search the
entire list every time. With move-to-front, it averages only two comparisons per query!

In fact, it can be shown that the total search time with move-to-front is never more than
twice the time if you knew the actual probabilities in advance!!

We will see self-organization again later in the semester when we talk about splay trees.

Let's Play 20 Questions!

1. 11.
2. 12.
3. 13.
4. 14.
5. 15.
6. 16.
7. 17.
8. 18.
9. 19.
10. 20.

Binary Search

Binary Search is an incredibly powerful technique for searching an ordered list. It is

familiar to everyone who uses a telephone book!

The basic algorithm is to find the middle element of the list, compare it against the key,
decide which half of the list must contain the key, and repeat with that half.

Two requirements to support binary search:

 • Random access of the list elements, so we need arrays instead of linked lists.
• The array must contain elements in sorted order by the search key.

Why Do Twenty Questions Suffice?

With one question, I can distinguish between two words: A and B; ``Is the key ?''

With two questions, I can distinguish between four words: A,B,C,D; ``Is the ?''

Each question I ask em doubles the number of words I can search in my dictionary.

, which is much larger than any portable dictionary!

Thus I could waste my first two questions because

Exponents and Logs

Recall the definitions of exponent and logarithm from high school:

Thus exponentiation and logarithms are inverse functions, since .

Note that the logarithm of a big number is a much smaller number.

Thus the number of questions we must ask is the base two logarithm of the size of the
dictionary.

Implementing Binary Search

Although the algorithm is simple to describe informally, it is tricky to produce a working

binary search function. The first published binary search algorithm appeared in 1946, but
the first correct published program appeared in 1962!

The difficulty is maintaining the following two invariants with each iteration:

• The key must always remain between the low and high indices.
• The low or high indice must advance each iteration.
The boundary cases are very tricky: zero elements left, one elements left, two elements
left, and an even or odd number of elements!

Versions of Binary Search

There are at least two different versions of binary search, depending upon whether we
want to test for equality at each query or only at the end.

For the later, suppose we want to search for ``k'':

iteration bottom top mid

---------------------------------------
1 2 14 (1+14)/2=7
2 1 7 (1+7)/2=4
3 5 7 (5+7)/2=6
4 6 7 (7+7)/2=7

Since , 7 is the right spot. However, we must now test if entry[7]='k'. If

not, the item isn't in the array.

Alternately, we can test for equality at each comparison. Suppose we search for ``c'':

iteration bottom top mid

------------------------------------
1 1 14 (1+14)/2 = 7
2 1 6 (1+6)/2 = 3
3 1 2 (1+2)/2 = 1
4 2 2 (2+2)/2 = 2

Now it will be found!

Recursive Binary Search Implementation

PROCEDURE Search( READONLY array: ARRAY [0 .. MaxInd - 1] OF INTEGER;

left, right: [0 .. MaxInd - 1];
argument: INTEGER): [0..MaxInd] =
(*Implements binary search in an array*)
VAR
middle := left + (right - left) DIV 2;

BEGIN (* binary search *)

IF argument = array[middle] THEN (*found*)
RETURN middle
ELSIF argument < array[middle] THEN (*search in left half*)
IF left < middle THEN
RETURN Search(array, left, middle - 1, argument)
ELSE (*left boundary reaches
middle: not found*)
RETURN MaxInd
END (*IF left < middle*)
ELSE (*search in right half*)
 IF middle < right THEN
RETURN Search(array, middle + 1, right, argument)
ELSE (*middle reaches right
boundary: not found*)
RETURN MaxInd
END (*IF middle < right*)
END (*IF argument = array[middle]*)
END Search;

Arrays and Access Formulas

Lecture 20
Steven S. Skiena

One-dimensional Arrays

The easiest way to view a one - dimensional array is as a contiguous block of memory
locations of length (# of array elements) (size of each element)

Because the size (in bytes) of each element is the same, the compiler can translated
A[500] into the address of the record

If A points to the first location of of k-byte records, then

This is the access formula for a one-dimensional array.

Two-Dimensional Arrays

How does the compiler know where to store element A[i,j] of a two-dimensional array?
By chopping the matrix into rows, it can be stored like a one- dimensional array:

If A points to the first location of A[l1..h1,l2..h2] of k-byte records, then:

Is this access formula for row-major or column-major order, assuming the first index
gives the row?

For three dimensions, cut the matrix into two dimensional slabs, and use the previous
formula. For k-dimensional arrays, we can find a similar formula by induction.
Thus we can access any element in a k-dimensional array in O(k) time, which is constant
for any reasonably dimension.

Fortran stores its arrays in column-major order, while most other languages use row-
major order. But why might we really need to know what is going on under the hood?

In C language, pointers are usually used to cruise through arrays. Cruising through a 2D
array meaningfully requires knowing the order of the elements.

Also, in a computer with virtual memory or a cache, it is often faster to access elements if
they are close to the last one we have read. Knowing the access function lets us choose
the right way to order nested loops.

(*row-major*)
(*column-major*)

Do i=1 to n Do j=

Do j=1 to n Do i=

A[i,j] = 0

Triangular Tables

By playing with our own access functions we can build efficient arrays of whatever shape
we want, including triangular and banded arrays.

Triangular tables prove useful for representing any symmetric function, such as the
distance from A to B, D[a,b] = D[b,a]. Thus we can save almost half the memory of a
rectangular array by storing it as a triangle

The access formula is:

since the identity can be proven by induction.

Faster than Binary Search?

Binary search takes time to find a particular key in a sorted array. It can be
shown that, in the worst case, no faster algorithm exists. So how might we do faster?

This is not a contradiction. Suppose we wanted to search on a field containing an ID

number between 1 and the number of records. Rather than doing a binary search on this
field, why not use it as an index in an array!

Accessing such an array element is O(1) instead of !

Interpolation Search

Binary search is only optimal when you know nothing about your data except that it is
sorted!

When you look up AAA in the telephone book, you don't start in the middle. We use our
understanding of how things are named in the real world to choose where to prove next.
Such an algorithm is called an interpolation search, since we are interpolating(guessing)
where the key should be.

Interpolation search is only as good as our guesses. If we do not understand the data as
well as you think, interpolation search can be very slow - recall the Shifflett's of
Charlottesville!

With interpolation search, the cost of making a good guess might overwhelm the
reduction in the number of guesses, so watch out!

The Key Ideas on Access Formulas

A pointer tells us exactly where in memory an item is.

An array reference A[i] lets us quickly calculate exactly where the ith element of A is in
memory, knowing only i, the starting location of A, and the size of each array item.

Any time we can compute the exact position for an item in memory by a simple access
formula, we can find it as quickly as we can compute the formula!

Must Array Indices be Integers?

We have seen that binary search is slower than table lookup. Why can't the entire world
be one big array?

One reason is that many of the fields we wish to search on are not integers, for example,
names in a telephone book. What address in the machine is defined by ``Skiena''?

To compute the appropriate address we need a function to map arbitrary keys to

addresses. Such hash functions form the basis of an important search technique, hashing!

Hashing
Lecture 21
Steven S. Skiena

Hashing

One way to convert form names to integers is to use the letters to form a base ``alphabet-
size'' number system:

To convert ``STEVE'' to a number, observe that e is the 5th letter of the alphabet, s is the
19th letter, t is the 20th letter, and v is the 22nd letter.

Thus ``Steve''

Thus one way we could represent a table of names would be to set aside an array big
enough to contain one element for each possible string of letters, then store data in the
elements corresponding to real people. By computing this function, it tells us where the
person's phone number is immediately!!

What's the Problem?

Because we must leave room for every possible string, this method will use an incredible
amount of memory. We need a data structure to represent a sparse table, one where
almost all entries will be empty.

We can reduce the number of boxes we need if we are willing to put more than one thing
in the same box!

Example: suppose we use the base alphabet number system, then take the remainder
Now the table is much smaller, but we need a way to deal with the fact that more than
one, (but hopefully every few) keys can get mapped to the same array element.

The Basics of Hashing

The basics of hashing is to apply a function to the search key so we can determine where
the item is without looking at the other items. To make the table of reasonable size, we
must allow for collisions, two distinct keys mapped to the same location.

• We a special hash function to map keys (hopefully uniformly) to integers in a

certain range.
• We set up an array as big as this range, and use the valve of the function as the
index to store the appropriate key. Special care must be taken to handle collisions
when they occur.

There are several clever techniques we will see to develop good hash functions and deal
with the problems of duplicates.

Hash Functions

The verb ``hash'' means ``to mix up'', and so we seek a function to mix up keys as well as
possible.

The best possible hash function would hash m keys into n ``buckets'' with no more than

keys per bucket. Such a function is called a perfect hash function

How can we build a hash function?

Let us consider hashing character strings to integers. The ORD function returns the
character code associated with a given character. By using the ``base character size''
number system, we can map each string to an integer.

The First Three SSN digits Hash

The first three digits of the Social Security Number

The last three digits of the Social Security Number

What is the big picture?

1. A hash function which maps an arbitrary key to an integer turns searching into
array access, hence O(1).
2. To use a finite sized array means two different keys will be mapped to the same
place. Thus we must have some way to handle collisions.
 3. A good hash function must spread the keys uniformly, or else we have a linear
search.

Ideas for Hash Functions

• Truncation - When grades are posted, the last four digits of your SSN are used,
because they distribute students more uniformly than the first four digits.
• Folding - We should get a better spread by factoring in the entire key. Maybe
subtract the last four digits from the first five digits of the SSN, and take the
absolute value?
• Modular Arithmetic - When constructing pseudorandom numbers, a good trick for
uniform distribution was to take a big number mod the size of our range. Because
of our roulette wheel analogy, the numbers tend to get spread well if the tablesize
is selected carefully.

Prime Numbers are Good Things

Suppose we wanted to hash check totals by the dollar value in pennies mod 1000. What
happens?

, , and

Prices tend to be clumped by similar last digits, so we get clustering.

If we instead use a prime numbered Modulus like 1007, these clusters will get broken:
, , and .

In general, it is a good idea to use prime modulus for hash table size, since it is less likely
the data will be multiples of large primes as opposed to small primes - all multiples of 4
get mapped to even numbers in an even sized hash table!

The Birthday Paradox

No matter how good our hash function is, we had better be prepared for collisions,
because of the birthday paradox.

Assuming 365 days a year, what is the probability that exactly two people share a
birthday? Once the first person has fixed their birthday, the second person has 365
possible days to be born to avoid a collision, or a 365/365 chance.

With three people, the probability that no two share is . In

general, the probability of there being no collisions after n insertions into an m-element
table is
When m = 366, this probability sinks below 1/2 when N = 23 and to almost 0 when

The moral is that collisions are common, even with good hash functions.

What about Collisions?

No matter how good our hash functions are, we must deal with collisions. What do we do
when the spot in the table we need is occupied?

• Put it somewhere else! - In open addressing, we have a rule to decide where to

put it if the space is already occupied.
• Keep a list at each bin! - At each spot in the hash table, keep a linked list of keys
sharing this hash value, and do a sequential search to find the one we need. This
method is called chaining.

Collision Resolution by Chaining

The easiest approach is to let each element in the hash table be a pointer to a list of keys.

Insertion, deletion, and query reduce to the problem in linked lists. If the n keys are
distributed uniformly in a table of size m/n, each operation takes O(m/n) time.

Chaining is easy, but devotes a considerable amount of memory to pointers, which could
be used to make the table larger. Still, it is my preferred method.

Open Addressing

We can dispense with all these pointers by using an implicit reference derived from a
simple function:

If the space we want to use is filled, we can examine the remaining locations:

1. Sequentially

2. Quadratically

3. Linearly

The reason for using a more complicated scheme is to avoid long runs from similarly
hashed keys.
Deletion in an open addressing scheme is ugly, since removing one element can break a
chain of insertions, making some elements inaccessible.

Performance on Set Operations

With either chaining or open addressing:

• Search - O(1) expected, O(n) worst case.

• Insert - O(1) expected, O(n) worst case.
• Delete - O(1) expected, O(n) worst case.

Pragmatically, a hash table is often the best data structure to maintain a dictionary.
However, the worst-case running time is unpredictable.

The best worst-case bounds on a dictionary come from balanced binary trees, such as red-
black trees.

Tree Structures
Lecture 21
Steven S. Skiena

Trees

``I think that I shall never see a poem as lovely as a tree.

Poems are wrote by fools like me, but only G-d can make a tree.''
- Joyce Kilmer

We have seen many data structures which allow fast search, but not fast, flexible update.

Sorted Tables - search, O(n) insertion, O(n) deletion.

Hash Tables - The number of insertions are essentially bounded by the table size, which
must be specified in advance. Worst case O(n) search.

Binary trees will enable us to search, insert, and delete fast, without predefining the size
of our data structure!

How can we get this flexibility?

The only data structure we have seen which allows fast insertion/ deletion is the linked
list, with updates in O(1) time but search in O(n) time.

To get search time, we used binary search, meaning we always had a choice
of two next elements to look at.

To combine these ideas, we want a ``linked list'' with two pointers per node! This is the
basic idea behind search trees!

Rooted Trees

We can use a recursive definition to specify what we mean by a ``rooted tree''.

A rooted tree is either (1) empty, or (2) consists of a node called the root, together with
two rooted trees called the left subtree and right subtree of the root.

A binary tree is a rooted tree where each node has at most two descendants, the left child
and the right child.

A binary tree can be implemented where each node has left and right pointer fields, an
(optional) parent pointer, and a data field.

Rooted trees in Real Life

Rooted trees can be used to model corporate heirarchies and family trees.

Note the inherently recursive structure of rooted trees. Deleting the root gives rise to a
certain number of smaller subtrees.

In a rooted tree, the order among ``brother'' nodes matters. Thus left is different from
right. The five distinct binary trees with five nodes:

Binary Search Trees

A binary search tree is a binary tree where each node contains a key such that:

• All keys in the left subtree precede the key in the root.
• All keys in the right subtree succeed the key in the root.
• The left and right subtrees of the root are again binary search trees.

Left: A binary search tree. Right: A heap but not a binary search tree.

For any binary tree on n nodes, and any set of n keys, there is exactly one labeling to
make it a binary search tree!!
Binary Tree Search

Searching a binary tree is almost like binary search! The difference is that instead of
searching an array and defining the middle element ourselves, we just follow the
appropriate pointer!

The type declaration is simply a linked list node with another pointer. Left and right
pointers are identical types.

TYPE
T = BRANDED REF RECORD
key: ElemT;
left, right: T := NIL;
END; (*T*)

Dictionary search operations are easy in binary trees. The algorithm works because both
the left and right subtrees of a binary search tree are binary search trees - recursive
structure, recursive algorithm.

Search Implementation

PROCEDURE Search(tree: T; e: ElemT): BOOLEAN =

(*Searches for an element e in tree.
Returns TRUE if present, else FALSE*)
BEGIN
IF tree = NIL THEN
RETURN FALSE (*not found*)
ELSIF tree.key = e THEN
RETURN TRUE (*found*)
ELSIF e < tree.key THEN
RETURN Search(tree.left, e) (*search in left tree*)
ELSE
RETURN Search(tree.right, e) (*search in right tree*)
END; (*IF tree...*)
END Search;

This takes time proportional to the height of the tree, O(h). Good, balanced trees have

height , while bad, unbalanced trees have height O(n).

Building Binary Trees

To insert a new node into an existing tree, we search for where it should be, then replace
that NIL pointer with a pointer to the new node.

Each NIL pointer defines a gap in the space of keys!

The pointer in the parent node must be modified to remember where we put the new
node.
Insertion Routine

PROCEDURE Insert(VAR tree: T; e: ElemT) =

BEGIN
IF tree = NIL THEN
tree:= NEW(T, key:= e); (*insert at proper place*)
ELSIF e < tree.key THEN
Insert(tree.left, e) (*search place in left tree*)
ELSE
Insert(tree.right, e) (*search place in right tree*)
END; (*IF tree...*)
END Insert;

Tree Shapes and Sizes

Suppose we have a binary tree with n nodes.

How many levels can it have? At least and at most n.

How many pointers are in the tree? There are n nodes in tree, each of which has 2
pointers, for a total of 2n pointers regardless of shape.

How many pointers are NIL, i.e ``wasted''? Except for the root, each node in the tree is
pointed to by one tree pointer Thus the number of NILs is

, for .

Traversal of Binary Trees

How can we print out all the names in a family tree?

An essential component of many algorithms is to completely traverse a tree data

structure. The key is to make sure we visit each node exactly once.

The order in which we explore each node and its children matters for many applications.

There are six permutations of {left, right, node} which define traversals. The most
interesting traversals are inorder {left, node, right}, preorder {node, left, right},
postorder {left, right, node},

Why do we care about different traversals? Depending on what the tree represents,
different traversals have different interpretations.

An in-order traversals of a binary serach tree sorts the keys!

Inorder traversal: 748251396, Preorder traversal: 124785369, Postorder traversal:

784529631
Reverse Polish notation is simply a post order traversal of an expression tree, like the one
below for expression 2+3*4+(3*4)/5.

PROCEDURE Traverse(tree: T; action: Action;

order := Order.In; direction := Direction.Right) =

PROCEDURE PreL(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
action(x.key, depth);
PreL(x.left, depth + 1);
PreL(x.right, depth + 1);
END; (*IF x # NIL*)
END PreL;

PROCEDURE PreR(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
action(x.key, depth);
PreR(x.right, depth + 1);
PreR(x.left, depth + 1);
END; (*IF x # NIL*)
END PreR;

PROCEDURE InL(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
InL(x.left, depth + 1);
action(x.key, depth);
InL(x.right, depth + 1);
END; (*IF x # NIL*)
END InL;

PROCEDURE InR(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
InR(x.right, depth + 1);
action(x.key, depth);
InR(x.left, depth + 1);
END; (*IF x # NIL*)
END InR;

PROCEDURE PostL(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
PostL(x.left, depth + 1);
PostL(x.right, depth + 1);
action(x.key, depth);
END; (*IF x # NIL*)
END PostL;

PROCEDURE PostR(x: T; depth: INTEGER) =

BEGIN
IF x # NIL THEN
PostR(x.right, depth + 1);
PostR(x.left, depth + 1);
 action(x.key, depth);
END; (*IF x # NIL*)
END PostR;

BEGIN (*Traverse*)
IF direction = Direction.Left THEN
CASE order OF
| Order.Pre => PreL(tree, 0);
| Order.In => InL(tree, 0);
| Order.Post => PostL(tree, 0);
END (*CASE order*)
ELSE (* direction = Direction.Right*)
CASE order OF
| Order.Pre => PreR(tree, 0);
| Order.In => InR(tree, 0);
| Order.Post => PostR(tree, 0);
END (*CASE order*)
END (*IF direction*)
END Traverse;

Deletion from Binary Search Trees

Insertion was easy because the new node goes in as a leaf and only its parent is affected.

Deletion of a leaf is just as easy - set the parent pointer to NIL. But what if the node to be
deleted is an interior node? We have two pointers to connect to only one parent!!

Deletion is somewhat more tricky than insertion, because the node to die may not be a
leaf, and thus effect other nodes.

Case (a), where the node is a leaf, is simple - just NIL out the parents child pointer.

Case (b), where a node has one chld, the doomed node can just be cut out.

Case (c), relabel the node as its predecessor (which has at most one child when z has two
children!) and delete the predecessor!

PROCEDURE Delete(VAR tree: T; e: ElemT): BOOLEAN =

(*Deletes an element e in tree.
Returns TRUE if present, else FALSE*)

PROCEDURE LeftLargest(VAR x: T) =
VAR y: T;
BEGIN
IF x.right = NIL THEN (*x points to largest element left*)
y:= tree; (*y now points to target node*)
tree:= x; (*tree assumes the largest node to
the left*)
x:= x.left; (*Largest node left replaced by its
left subtree*)
tree.left:= y.left; (*tree assumes subtrees ...*)
tree.right:= y.right; (*... of deleted node*)
ELSE (*Largest element left not found*)
 LeftLargest(x.right) (*Continue search to the right*)
END;
END LeftLargest;

BEGIN
IF tree = NIL THEN RETURN FALSE
ELSIF e < tree.key THEN RETURN Delete(tree.left, e)
ELSIF e > tree.key THEN RETURN Delete(tree.right, e)
ELSE (*found*)
IF tree.left = NIL THEN
tree:= tree.right;
ELSIF tree.right = NIL THEN
tree:= tree.left;
ELSE (*Target node has two nonempty
subtrees*)
LeftLargest(tree.left) (*Search in left subtree*)
END; (*IF tree.left...*)
RETURN TRUE
END; (*IF tree...*)
END Delete;

Random Search Trees

Lecture 23
Steven S. Skiena

How good are Random Trees?

Who have seen that binary trees can have heights ranging from to n. How tall are
they on average?

By using an intuitive argument, like I did with quicksort. I will convince you a random
tree is usually quite close to balanced. The text contains a more rigorous proof, which
you should look at.

Consider the first insertion into an empty tree. This node becomes the root and never
changes. Since in a binary search tree all keys less than the root go in the left subtree, the
root acts as a partition or pivot element!

Let's say a key is a 'good' pivot element if it is in the center half of the sorted space of
keys. Half of the time, our root will be a 'good' pivot element.

The next insertion will form the root of a subtree, and will be drawn at random from the
items either > root or < root. Again, half the time each insertion will be a 'good' partition
of the appropriate subset of keys.
The bigger half of a good partition contains at most 3n/4 items. Thus the maximum depth
of good splits k is:

so .

Doubling the depth to account for bad splits still makes in on average!

On average, random search trees are very good - more careful analysis shows the average
height after n insertions is . Since , this is only 39%
more than a perfectly balanced tree.

Of course, if we get unlucky and insert keys in sorted order, we are doomed to the worst
case performance.

insert(a)

insert(b)

insert(c)

insert(d)

What we want is an insertion/deletion procedure which adjusts the tree a little after each
insertion, keeping it close enough to balanced so the maximum height is logarithmic, but
flexible enough so we can still update fast!

Perfectly Balanced Trees

Perfectly balanced trees require a lot of work to maintain:

If we insert the key 1, we must move every single node in the tree to rebalance it, taking

time.

Therefore, when we talk about "balanced" trees, we mean trees whose height is

, so all dictionary operations (insert, delete, search, min/max,

successor/predecessor) take time.

Red-Black trees are binary search trees where each node is assigned a color, where the

coloring scheme helps us maintain the height as .

AVL Trees
Lecture 24
Steven S. Skiena

AVL Trees

An AVL tree is a binary search tree in which the heights of the left and right subtrees of
the root differ by at most 1, and the left and right subtrees are again AVL trees.

Therefore, we can label each node of an AVL tree with a balance factor as well as a key:

• ``='' - both subtrees of the node are of equal height

• ``/'' - the left subtree is one taller than the right subtree
• ``
'' - the right subtree is one taller than the left subtree.

AVL trees are named after their inventors, the Russians G.M. Adel'son-Velshi, and E.M.
Laudis in 1962.

These are the most unbalanced possible AVL trees with a skew always to the right.

By maintaining the balance of each node (i.e. the subtree below it) when we insert a new
node, we can easily see whether or not to take action!

The balance is more useful than maintaining the height of each node because it is a
relative, not absolute measure. Thus we can move subtrees around without affecting their
balance, even if they end up at different heights.
How good are AVL trees?

To find out how bad they can be, we want to find what the minimum number of modes a

tree of height h can have. If is a minimum node AVL tree, its left and right subtrees
must themselves be minimum node AVL trees of smaller size. Further, they should differ
in height by 1 to take advantage of AVL freedom.

Counting the root node,

Such trees are called Fibonacci trees and .

Thus the worse case AVL tree is almost as good as a random tree - on average it is very
close to an optional tree.

Why are Fibonacci trees of logarithmic height?

Recall that the Fibonacci numbers are defined , ,

.

Since we are adding the last two numbers together, we are more than doubling the next-
to-last and somewhat less that doubling the last number.

In fact, , so a tree with nodes has height

AVL Trees Interface

INTERFACE AVLTree; (*08.07.94. CW, LB*)

(* Balanced binary search tree, subtype of "BinaryTree.T" *)

IMPORT BinaryTree;
TYPE T <: BinaryTree.T; (*T is a subtype of BinaryTree.T *)

END AVLTree.

AVL Trees Implementation

MODULE AVLTree EXPORTS AVLTree, AVLTreeRep; (*08.07.94. CW*)

(* Implementation of the balanced binary search tree as subtype of
"BinaryTree.T". The methods "insert" and "delete" are overwritten
to keep the tree balanced when elements are inserted or
deleted. The other methods are inhereted from the supertype.
*)

IMPORT BinaryTree, BinTreeRep;

REVEAL
T = BinaryTree.T BRANDED OBJECT
OVERRIDES
delete:= Delete;
insert:= Insert;
END;

PROCEDURE Insert(tree: T; e: REFANY) =

PROCEDURE RR (VAR root: BinTreeRep.NodeT) =

(*simple rotation right*)
VAR left:= root.left;
BEGIN
root.left:= left.right;
left.right:= root;
NARROW(root, NodeT).balance:= 0;
root:= left;
END RR;

PROCEDURE RL (VAR root: BinTreeRep.NodeT) =

(*simple rotation left*)
VAR right:= root.right;
BEGIN
root.right:= right.left;
right.left:= root;
NARROW(root, NodeT).balance:= 0;
root:= right;
END RL;

PROCEDURE RrR (VAR root: BinTreeRep.NodeT) =

(*double rotation right*)
VAR right:= root.left.right;
BEGIN
root.left.right:= right.left;
right.left:= root.left;
IF NARROW(right, NodeT).balance = -1
THEN NARROW(root, NodeT).balance:= +1
ELSE NARROW(root, NodeT).balance:= 0
END;
IF NARROW(right, NodeT).balance = +1
THEN NARROW(root.left, NodeT).balance:= -1
ELSE NARROW(root.left, NodeT).balance:= 0
END;
root.left:= right.right;
right.right:= root;
root:= right;
END RrR;

PROCEDURE RrL (VAR root: BinTreeRep.NodeT) =

 (*double rotation left*)
VAR left:= root.right.left;
BEGIN
root.right.left:= left.right;
left.right:= root.right;
IF NARROW(left, NodeT).balance = +1
THEN NARROW(root, NodeT).balance:= -1
ELSE NARROW(root, NodeT).balance:= 0
END;
IF NARROW(left, NodeT).balance = -1
THEN NARROW(root.right, NodeT).balance:= +1
ELSE NARROW(root.right, NodeT).balance:= 0
END;
root.right:= left.left;
left.left:= root;
root:= left;
END RrL;

PROCEDURE InsertBal(VAR root: BinTreeRep.NodeT; new: REFANY;

VAR bal: BOOLEAN) =
BEGIN
IF root = NIL
THEN
root:= NEW(NodeT, info:= new, balance:= 0);

ELSIF tree.compare(new, root.info)<0 THEN

InsertBal(root.left, new, bal);
IF NOT bal THEN (* bal stops recursion*)
WITH done=NARROW(root, NodeT).balance DO
CASE done OF
|+1=> done:= 0; bal:= TRUE; (*insertion ok*)
| 0=> done:= -1; (*still balanced, but
continue*)
|-1=>
IF NARROW(root.left, NodeT).balance = -1
THEN RR(root)
ELSE RrR(root)
END;
NARROW(root, NodeT).balance:= 0;
bal:= TRUE; (*after rotation tree
ok*)
END; (*CASE*)
END (*WITH*)
END (*IF*)

ELSE
InsertBal(root.right, new, bal);
IF NOT bal THEN (* bal is set to stop the recurs.
adjustm. of balance *)
WITH done=NARROW(root, NodeT).balance DO
CASE done OF
|-1=> done:= 0; bal:= TRUE; (*insertion ok *)
| 0=> done:= +1; (*still balanced, but
continue*)
|+1=>
IF NARROW(root.right, NodeT).balance = +1
THEN RL(root)
 ELSE RrL(root)
END;
NARROW(root, NodeT).balance:= 0;
bal:= TRUE; (*after rotation tree ok*)
END; (*CASE*)
END (*WITH*)
END (*IF*)
END;
END InsertBal;

VAR balanced:= FALSE;

BEGIN (*Insert*)
InsertBal(tree.root, e, balanced)
END Insert;

PROCEDURE Delete(tree: T; e: REFANY): REFANY =

PROCEDURE RR (VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN) =
(*simple rotation right*)
VAR left:= root.left;
BEGIN
root.left:= left.right;
left.right:= root;
IF NARROW(left, NodeT).balance = 0
THEN
NARROW(root, NodeT).balance:= -1;
NARROW(left, NodeT).balance:= +1;
bal:= TRUE;
ELSE
NARROW(root, NodeT).balance:= 0;
NARROW(left, NodeT).balance:= 0; (*depth changed:
continue*)
END;
root:= left;
END RR;

PROCEDURE RL (VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN) =
(*simple rotation left*)
VAR right:= root.right;
BEGIN
root.right:= right.left;
right.left:= root;
IF NARROW(right, NodeT).balance = 0
THEN
NARROW(root, NodeT).balance:= +1;
NARROW(right, NodeT).balance:= -1;
bal:= TRUE;
ELSE
NARROW(root, NodeT).balance:= 0;
NARROW(right, NodeT).balance:= 0; (*depth changed:
continue*)
END;
root:= right;
END RL;
PROCEDURE RrR (VAR root: BinTreeRep.NodeT) =
(*double rotation right*)
VAR right:= root.left.right;
BEGIN
root.left.right:= right.left;
right.left:= root.left;
IF NARROW(right, NodeT).balance = -1
THEN NARROW(root, NodeT).balance:= +1
ELSE NARROW(root, NodeT).balance:= 0
END;
IF NARROW(right, NodeT).balance = +1
THEN NARROW(root.left, NodeT).balance:= -1
ELSE NARROW(root.left, NodeT).balance:= 0
END;
root.left:= right.right;
right.right:= root;
root:= right;
NARROW(right, NodeT).balance:= 0;
END RrR;

PROCEDURE RrL (VAR root: BinTreeRep.NodeT) =

(*double rotation left*)
VAR left:= root.right.left;
BEGIN
root.right.left:= left.right;
left.right:= root.right;
IF NARROW(left, NodeT).balance = +1
THEN NARROW(root, NodeT).balance:= -1
ELSE NARROW(root, NodeT).balance:= 0
END;
IF NARROW(left, NodeT).balance = -1
THEN NARROW(root.right, NodeT).balance:= +1
ELSE NARROW(root.right, NodeT).balance:= 0
END;
root.right:= left.left;
left.left:= root;
root:= left;
NARROW(left, NodeT).balance:= 0;
END RrL;

PROCEDURE BalanceLeft(VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN) =
BEGIN
WITH done = NARROW(root, NodeT).balance DO
CASE done OF
|-1=> done:= 0; (*new depth: continue*)
| 0=> done:= 1; bal:= TRUE; (*balanced ->ok*)
|+1=> (*balancing needed*)
IF NARROW(root.right, NodeT).balance >= 0
THEN RL(root, bal)
ELSE RrL(root)
END
END (*CASE*)
END (*WITH*)
END BalanceLeft;

PROCEDURE BalanceRight(VAR root: BinTreeRep.NodeT;

 VAR bal: BOOLEAN) =
BEGIN
WITH done = NARROW(root, NodeT).balance DO
CASE done OF
|+1=> done:= 0; (*new depth: continue*)
| 0=> done:= -1; bal:= TRUE; (*balanced ->ok*)
|-1=> (*balancing needed*)
IF NARROW(root.left, NodeT).balance <= 0
THEN RR(root, bal)
ELSE RrR(root)
END
END (*CASE*)
END (*WITH*)
END BalanceRight;

PROCEDURE DeleteSmallest(VAR root: BinTreeRep.NodeT;

VAR bal: BOOLEAN): REFANY =
VAR deleted: REFANY;
BEGIN
IF root.left = NIL
THEN
deleted:= root.info;
root:= root.right;
RETURN deleted;
ELSE
deleted:= DeleteSmallest(root.left, bal);
IF NOT bal THEN BalanceLeft(root, bal) END;
RETURN deleted;
END;
END DeleteSmallest;

PROCEDURE Delete(VAR root: BinTreeRep.NodeT; elm: REFANY;

VAR bal: BOOLEAN): REFANY =
VAR deleted: REFANY;
BEGIN
IF root = NIL
THEN RETURN NIL

ELSIF tree.compare(root.info, elm)>0 THEN

deleted:= Delete(root.left, elm, bal);
IF deleted # NIL
THEN
IF NOT bal THEN BalanceLeft(root, bal) END;
RETURN deleted;
ELSE RETURN NIL;
END

ELSIF tree.compare(root.info, elm)<0 THEN

deleted:= Delete(root.right, elm, bal);
IF deleted # NIL
THEN
IF NOT bal THEN BalanceRight(root, bal) END;
RETURN deleted;
ELSE RETURN NIL;
END

ELSE
 deleted:= root.info;
IF root.left = NIL
THEN
root:= root.right;
ELSIF root.right = NIL THEN
root:= root.left;
ELSE
root.info:= DeleteSmallest(root.right, bal);
IF NOT bal THEN BalanceRight(root, bal) END;
END;
RETURN deleted;
END;
END Delete;

VAR balanced:= FALSE;

BEGIN (*Delete*)
RETURN Delete(tree.root, e, balanced)
END Delete;

BEGIN
END AVLTree.

Deletion from AVL Trees

We have seen that AVL trees are for insertion and query.

But what about deletion?

Don't ask! Actually, you can rebalance an AVL tree in but it is more
complicated than insertion.

We will later study B-trees, where deletion is simpler, so don't worry about the details of
deletions form AVL trees.

Red-Black Trees
Lecture 25
Steven S. Skiena

Red-Black Tree Definition

Red-black trees have the following properties:

1. Every node is colored either red or black.

 2. Every leaf (NIL pointer) is black.
3. If a node is red then both its children are black.
4. Every single path from a node to a decendant leaf contains the same number of
black nodes.

What does this mean?

If the root of a red-black tree is black can we just color it red?

No! For one of its children might be red.

If an arbitrary node is red can we color it black?

No! Because now all nodes may not have the same black height.

What tree maximizes the number of nodes in a tree of black height h?

What does a red-black tree with two real nodes look like?

Not (1) - consecutive reds Not (2), (4) - Non-Uniform black height

Red-Black Tree Height

Lemma: A red-black tree with n internal nodes has height at most .

Proof: Our strategy; first we bound the number of nodes in any subtree, then we bound
the height of any subtree.

We claim that any subtree rooted at x has at least - 1 internal nodes, where bh(x)
is the black height of node x.

Proof, by induction:

Now assume it is true for all tree with black height < bh(x).

If x is black, both subtrees have black height bh(x)-1. If x is red, the subtrees have black
height bh(x).

Therefore, the number of internal nodes in any subtree is

Now, let h be the height of our red-black tree. At least half the nodes on any single path
from root to leaf must be black if we ignore the root.

Thus and , so .

This implies that ,so . height6pt width4pt

Therefore red-black trees have height at most twice optimal. We have a balanced search
tree if we can maintain the red-black tree structure under insertion and deletion.

Rotations

The basic restructuring step for binary search trees are left and right rotation:

1. Rotation is a local operation changing O(1) pointers.

2. An in-order search tree before a rotation stays an in-order search tree.
3. In a rotation, one subtree gets one level closer to the root and one subtree one
level further from the root.

Rotation Implementation

PROCEDURE RR (VAR root: BinTreeRep.NodeT) =

(*simple rotation right*)
VAR left:= root.left;
BEGIN
root.left:= left.right;
left.right:= root;
root:= left;
END RR;

PROCEDURE RL (VAR root: BinTreeRep.NodeT) =

(*simple rotation left*)
VAR right:= root.right;
BEGIN
root.right:= right.left;
right.left:= root;
root:= right;
END RL;

Note the in-order property is preserved.

Red-Black Insertion
Since red-black trees have height, if we can preserve all properties of such
trees under insertion/deletion, we have a balanced tree!

Suppose we just did a regular insertion. Under what conditions does it stay a red-black
tree?

Since every insertion take places at a leaf, we will change a black NIL pointer to a node
with two black NIL pointers.

To preserve the black height of the tree, the new node must be red. If its new parent is
black, we can stop, otherwise we must restructure! How can we fix two reds in a row?

It depends upon our uncle's color:

If our uncle is red, reversing our relatives' color either solves the problem or pushes it
higher!

Note that after the recoloring:

1. The black height is unchanged.

2. The shape of the tree is unchanged.
3. We are done if our great-grandparent is black.

If we get all the way to the root, recall we can always color a red-black tree's root black.
We always will, so initially it was black, and so this process terminates.

The Case of the Black Uncle

If our uncle was black, observe that all the nodes around us have to be black:

Solution - rotate right about B:

Since the root of the subtree is now black with the same black-height as before, we have
restored the colors and can stop!

Double Rotations

A double rotation can be required to set things up depending upon the left-right turn
sequence, but the principle is the same.

Deletion from Red-Black Trees

Recall the three cases for deletion from a binary tree:

Case (a) The node to be deleted was a leaf;

Case (b) The node to be deleted had one child;

Case (c) relabel to node as its successor and delete the successor.

Deletion Color Cases

Suppose the node we remove was red, do we still have a red-black tree?

Yes! No two reds will be together, and the black height for each leaf stays the same.

However, if the dead node y was black, we must give each of its decendants another
black ancestor. If an appropriate node is red, we can simply color it black otherwise we
must restructure.

Case (a) black NIL becomes ``double black'';

Case (b) red becomes black and black becomes ``double black'';

Case (c) red becomes black and black becomes ``double black''.

Our goal will be to recolor and restructure the tree so as to get rid of the ``double black''
node.

In setting up any case analysis, we must be sure that:

1. All possible cases are covered.

2. No case is covered twice.

In the case analysis for red-black trees, the breakdown is:

Case 1: The double black node x has a red brother.

Case 2: x has a black brother and two black nephews.

Case 3: x has a black brother, and its left nephew is red and its right nephew is black.

Case 4: x has a black brother, and its right nephew is red (left nephew can be any color).

Conclusion

Red-Black trees let us implement all dictionary operations in . Further, in no

case are more than 3 rotations done to rebalance. Certain very advanced data structures
have data stored at nodes which requires a lot of work to adjust after a rotation -- red-
black trees ensure it won't happen often.
Example: Each node represents the endpoint of a line, and is augmented with a list of
segments in its subtree which it intersects.

We will not study such complicated structures, however.

Splay Trees
Lecture 26
Steven S. Skiena

What about non-uniform access?

AVL/red-black trees give us worst case query and update operations, by

keeping a balanced search tree. But when I access with non-uniform probability, a
skewed tree might be better:

• I call Eve with probability .75

• I call Lisa with probability .05
• I call Wendy with probability .20

Expected cost of left tree:

Expected cost of right tree:

In real life, it is difficult to obtain the actual probabilities, and they keep changing. What
can we do?

Self-organizing Search Trees

We can apply our self-organizing heuristics to search trees, as we did with linked lists.
Whenever we access a node, we can either:

• Move-forward-one (conservative heuristic)

• Move-to-front (liberal heuristic)

Once again, move-to-front proves better at adjusting to changing distributions.

Moving a made to the front of a search tree means making it the root!

To get a particular node to the root we can do a sequence of rotations!

Splay trees use the move-to-front heuristic on each search / query.

Splay Trees

To search or insert into a splay tree, we first perform the operation as if it was a random
tree. After it is found or inserted, perform a splay operation to move the given key to the
root.

A splay operation consists of a sequence of double rotations until the node is within one
level of the root, where at most one single rotation suffices to finish the job.

The choice of which double rotation to do depends upon our relationship to our
grandparent - a single rotation is performed only when we have no grandparent!

The cases: and .

Splay Tree Example

Example: Splay(a)

At the conclusion, a is the root and the tree is more balanced.

Note that the tree would not have become more balanced had we just used single
rotations to promote a to the root, instead of double rotations.

How good are Splay Trees?

Sleator and Tarjan showed that if the keys are accessed with a uniform distribution, the

cost for any sequence of n splay operations is , so the amortized cost is

per operation!

This is better than expected since there is no probability involved! If we get

an expensive splay step (i.e. moving up an non-balanced tree) it meant we did enough
cheap operations before this that we can pay for the differences out of our savings!

Further, if the distribution is non-uniform, we get amortized costs within a constant factor
of the best possible tree!

All of this is done without keeping any balance or color information - amazing!
 Graphs
Lecture 27
Steven S. Skiena

Graphs

A graph G consists of a set of vertices V together with a set E of vertex pairs or edges.

Graphs are important because any binary relation is a graph, so graphs can be used to
represent essentially any relationship.

Example: A network of roads, with cities as vertices and roads between cities as edges.

Example: An electronic circuit, with junctions as vertices as components as edges.

To understand many problems, we must think of them in terms of graphs!

The Friendship Graph

Consider a graph where the vertices are people, and there is an edge between two people
if and only if they are friends.

This graph is well-defined on any set of people: SUNY SB, New York, or the world.

What questions might we ask about the friendship graph?

• If I am your friend, does that mean you are my friend?

A graph is undirected if (x,y) implies (y,x). Otherwise the graph is directed. The
``heard-of'' graph is directed since countless famous people have never heard of
me! The ``had-sex-with'' graph is presumably undirected, since it requires a
partner.

• Am I my own friend?

An edge of the form (x,x) is said to be a loop. If x is y's friend several times over,
that could be modeled using multiedges, multiple edges between the same pair of
vertices. A graph is said to be simple if it contains no loops and multiple edges.

• Am I linked by some chain of friends to the President?

A path is a sequence of edges connecting two vertices. Since Mel Brooks is my

father's-sister's-husband's cousin, there is a path between me and him!
 • How close is my link to the President?

If I were trying to impress you with how tight I am with Mel Brooks, I would be
much better off saying that Uncle Lenny knows him than to go into the details of
how connected I am to Uncle Lenny. Thus we are often interested in the shortest
path between two nodes.

• Is there a path of friends between any two people?

A graph is connected if there is a path between any two vertices. A directed graph
is strongly connected if there is a directed path between any two vertices.

• Who has the most friends?

The degree of a vertex is the number of edges adjacent to it.

• What is the largest clique?

A social clique is a group of mutual friends who all hang around together. A
graph theoretic clique is a complete subgraph, where each vertex pair has an edge
between them. Cliques are the densest possible subgraphs. Within the friendship
graph, we would expect that large cliques correspond to workplaces,
neighborhoods, religious organizations, schools, and the like.

• How long will it take for my gossip to get back to me?

A cycle is a path where the last vertex is adjacent to the first. A cycle in which no
vertex repeats (such as 1-2-3-1 verus 1-2-3-2-1) is said to be simple. The shortest
cycle in the graph defines its girth, while a simple cycle which passes through
each vertex is said to be a Hamiltonian cycle.

Data Structures for Graphs

There are two main data structures used to represent graphs.

Adjacency MatricesAn adjacency matrix is an matrix, where M[i,j] = 0 iff there is

no edge from vertex i to vertex j

It takes time to test if (i,j) is in a graph represented by an adjacency matrix.

Can we save space if (1) the graph is undirected? (2) if the graph is sparse?

Adjacency ListsAn adjacency list consists of a array of pointers, where the ith
element points to a linked list of the edges incident on vertex i.
To test if edge (i,j) is in the graph, we search the ith list for j, which takes , where
is the degree of the ith vertex.

Note that can be much less than n when the graph is sparse. If necessary, the two
copies of each edge can be linked by a pointer to facilitate deletions.

Tradeoffs Between Adjacency Lists and Adjacency Matrices

Both representations are very useful and have different properties, although adjacency
lists are probably better for most problems.

Traversing a Graph

One of the most fundamental graph problems is to traverse every edge and vertex in a
graph. Applications include:

• Printing out the contents of each edge and vertex.

• Counting the number of edges.
• Identifying connected components of a graph.

For efficiency, we must make sure we visit each edge at most twice.

For correctness, we must do the traversal in a systematic way so that we don't miss
anything.

Since a maze is just a graph, such an algorithm must be powerful enough to enable us to
get out of an arbitrary maze.

Marking Vertices

The idea in graph traversal is that we must mark each vertex when we first visit it, and
keep track of what have not yet completely explored.
For each vertex, we can maintain two flags:

• discovered - have we ever encountered this vertex before?

• completely-explored - have we finished exploring this vertex yet?

We must also maintain a structure containing all the vertices we have discovered but not
yet completely explored.

Initially, only a single start vertex is considered to be discovered.

To completely explore a vertex, we look at each edge going out of it. For each edge
which goes to an undiscovered vertex, we mark it discovered and add it to the list of work
to do.

Note that regardless of what order we fetch the next vertex to explore, each edge is
considered exactly twice, when each of its endpoints are explored.

Correctness of Graph Traversal

Every edge and vertex in the connected component is eventually visited.

Suppose not, ie. there exists a vertex which was unvisited whose neighbor was visited.
This neighbor will eventually be explored so we would visit it:

Traversal Orders

The order we explore the vertices depends upon what kind of data structure is used:

• Queue - by storing the vertices in a first-in, first out (FIFO) queue, we explore the
oldest unexplored vertices first. Thus our explorations radiate out slowly from the
starting vertex, defining a so-called breadth-first search.
• Stack - by storing the vertices in a last-in, first-out (LIFO) stack, we explore the
vertices by lurching along a path, constantly visiting a new neighbor if one is
available, and backing up only if we are surrounded by previously discovered
vertices. Thus our explorations quickly wander away from our starting point,
defining a so-called depth-first search.

The three possible colors of each node reflect if it is unvisited (white), visited but
unexplored (grey) or completely explored (black).

Breadth-First Search

BFS(G,s)
for each vertex do

color[u] = white

, ie. the distance from s

p[u] = NIL, ie. the parent in the BFS tree

color[u] = grey

d[s] = 0

p[s] = NIL

while do

u = head[Q]

for each do
 if color[v] = white then

color[v] = gray

d[v] = d[u] + 1

p[v] = u

enqueue[Q,v]

dequeue[Q]

color[u] = black

Depth-First Search

DFS has a neat recursive implementation which eliminates the need to explicitly use a
stack.

Discovery and final times are sometimes a convenience to maintain.

DFS(G)

for each vertex do

color[u] = white
 parent[u] = nil

time = 0

for each vertex do

if color[u] = white then DFS-VISIT[u]

Initialize each vertex in the main routine, then do a search from each connected
component. BFS must also start from a vertex in each component to completely visit the
graph.

DFS-VISIT[u]

color[u] = grey (*u had been white/undiscovered*)

discover[u] = time

time = time+1

for each do

if color[v] = white then

 parent[v] = u

DFS-VISIT(v)

color[u] = black (*now finished with u*)

finish[u] = time

time = time+1

SUNY at Stony BrookMidterm 1

CSE 214 - Data Structures October 10, 1997

Midterm Exam

Name: Signature:
ID #: Section #:

INSTRUCTIONS:

• You may use either pen or pencil.

• Check to see that you have 4 exam pages plus this cover (5 total).
• Look over all problems before starting work.
• Your signature above signs the CSE 214 Honor Pledge: ``On my honor as a
student I have neither given nor received aid on this exam.''
• Think before you write.
• Good luck!!

1) (25 points) Assume that you have the linked structure on the left, where each node
contains a .next field consisting of a pointer, and the pointer p points to the structure as
shown. Describe the sequence of Modula-3 pointer manipulations necessary to convert it
to the linked structure on the right. You may not change any of the .info fields, but you
may use temporary pointers tmp1, tmp2, and tmp3 if you wish.

Many different solutions were possible, including:

tmp1 := p;
p := p.next;
p^.next^.next := tmp1;

2) (30 points) Write a procedure which ``compresses'' a linked list by deleting

consecutive copies of the same character from it. For example, the list
(A,B,B,C,A,A,A,C,A) should be compressed to (A,B,C,A,C,A). Thus the same character
can appear more than once in the compressed list, only not successively. Your procedure
must have one argument as defined below, a VAR parameter head pointing to the front of
the linked list. Each node in the list has .info and .next fields.

PROCEDURE compress(VAR head : pointer);

Many different solutions are possible, but recursive solutions are particularly clean and
elegant.
PROCEDURE compress(VAR head : pointer);
VAR
second : pointer; (* pointer to next element *)

BEGIN
IF (head # NIL) THEN
second := head^.next;
IF (second # NIL)
IF (head^.info = second^.info) THEN
head^.next = second^.next;
compress(head);
ELSE
compress(head^.next);
END;
END;
END;
END;

3) (20 points) Provide the output of the following program:

MODULE strange; EXPORTS main;

IMPORT SIO;

TYPE
ptr_to_integer = REF INTEGER;
VAR
a, b : ptr_to_integer;

PROCEDURE modify(x : ptr_to_integer; VAR y : ptr_to_integer);

begin
x^ := 3;
 SIO.PutInt(a^);
SIO.PutInt(x^); SIO.Nl();
y^ := 4;
SIO.PutInt(b^);
SIO.PutInt(y^); SIO.Nl();
end;

begin
a := NEW(INTEGER); b := NEW(INTEGER);
a^ := 1;
b^ := 2;
SIO.PutInt(a^);
SIO.PutInt(b^); SIO.Nl();
modify(a,b);
SIO.PutInt(a^);
SIO.PutInt(b^); SIO.Nl();

end.

Answers:
1 2
3 3
4 4
3 4

4) (25 points)

Write brief essays answering the following questions. Your answer must fit completely in
the space allowed

(a) Explain the difference between objects and modules? ANSWER: Several answers
possible, but the basic differences are (1) the notation to use them, and (2) that objects
encapsulate both procedures and data where modules are procedure oriented. (b) What is
garbage collection? ANSWER: The automatic reuse of dynamic memory which, because
of pointer dereferencing, is no longer accessible. (c) What might be an advantage of a
doubly-linked list over a singly-linked list for certain applications? ANSWER: Additional
flexibility in moving both forward and in reverse on a linked list. Specific advantages
include being able to delete a node from a list given just a pointer to the node, and
efficiently implementing double-ended queues (supporing push, pop, enqueue, and
dequeue).

SUNY at Stony BrookMidterm 2

CSE 214 - Data Structures November 21, 1997

Midterm Exam
Name: Signature:
ID #: Section #:

INSTRUCTIONS:

• You may use either pen or pencil.

• Check to see that you have 5 exam pages plus this cover (6 total).
• Look over all problems before starting work.
• Your signature above signs the CSE 214 Honor Pledge: ``On my honor as a
student I have neither given nor received aid on this exam.''
• Think before you write.
• Good luck!!

1) (20 points) Show the state of the array after each pass by the following sorting
routines. You do not have to show the array after every move or comparison, but only
after each execution of the main sorting loop or recursive call. Sort in increasing order.

-------------------------------------------------------------
| 34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
-------------------------------------------------------------

(a) Insertion Sort

10 points

-------------------------------------------------------------
| 34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 34 \ 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 34 | 125 \ 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 5 | 34 | 125 \ 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 5 | 19 | 34 | 125 \ 87 | 243 | 19 | -3 | 117 | 36 |
| 5 | 19 | 34 | 87 | 125\ 243 | 19 | -3 | 117 | 36 |
| 5 | 19 | 34 | 87 | 125| 243 \ 19 | -3 | 117 | 36 |
| 5 | 19 | 19 | 34 | 87 | 125| 243 \ -3 | 117 | 36 |
| -3 | 5 | 19 | 19 | 34 | 87 | 125| 243 \ 117 | 36 |
| -3 | 5 | 19 | 19 | 34 | 87 | 117| 125| 243 \ 36 |
| -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117| 125| 243 |
-------------------------------------------------------------

(b) Quicksort (pivot on rightmost element)

10 points - there are many variants

 ------------------------------------------------------------
| 34 | 125 | 5 | 19 | 87 | 243 | 19 | -3 | 117 | 36 |
| 34 | 5 | 19 | 19 | -3 | 36 | 125 | 87 | 243 | 117 |
| -3 | 34 | 5 | 19 | 19 | 36 | 87 | 117 | 125 | 243 |
| -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117 | 125 | 243 |
| -3 | 5 | 19 | 19 | 34 | 36 | 87 | 117| 125| 243 |
-------------------------------------------------------------

2) (25 points) In class, we discussed two different heuristics for self-organizing

sequential search. With the move-to-front heuristic, on a query we perform a sequential
search for the appropriate element, which when found is moved from its current position
to the front of the list. With the move-forward-one heuristic, on a query we perform a
sequential search for the appropriate element, which when found is moved one element
closer to the front of the list. For both algorithms, if the search key is already at the front
of the list, no change occurs.

(a) Write a function to implement sequential search in a linked list, with the move-to-
front heuristic. You may assume that there are at least two elements in the list and that the
item is always found.

PROCEDURE ListSearch (VAR p : pointer) : pointer =

var q, head:pointer;

head := p;
q := p;

if (q^.info = key) then

return(q);
else
p := p^.next;
while (p^.info # key) do
p := p^.next;
q := q^.next;
end
q^.next := p^.next;
p^.next := head;
head := p;
return (p);
end
points for searching, 10 points for move to front.

(b) Which of these two heuristics is better suited for implementation with arrays? Why?

5 points

move-forward-one is better for arrays since it can be done via one swap.

3) (15 points) Assume you have an array with 11 elements that is to be used to store data
as an hash table. The hash function computes the number mod 11. Given the following
list of insertions to the table:
 2 4 13 18 22 31 33 34 42 43 49
Show the resulting table after the insertions for each of the following hashing collision
handling methods.

a) Show the resulting table after the insertions for chaining. (array of linked lists)

10 points

0 - 22, 33 1 - 34 2 - 2, 13 3 - 4 - 4 5 - 49 6 - 7 - 18 8 - 9 - 31, 42 10 - 43

b) List an advantage and a disadvantage of chaining compared to open addressing

5 points.

Advantages - deletion is easier and hash table cannot be filled.

Disadvantages - the links use up memory which can go to a bigger hash table.

4) (20 points) Write brief essays answering the following questions. Your answer must
fit completely in the space allowed

(a) Is f(n) = O(g(n)) if and ? Show why or why not. points

No! There is no constant such that .

(b) Consider the following variant of insertion sort. Instead of using sequential search to
find the position of the next element we insert into the sorted array, we use a binary
search. We then move the appropriate elements over to create room for the new insertion.
What is the worst case number of element comparisons performed using this version of
insertion sort on n items (big Oh)? points

(c) What is the worst case number of element movements performed using the above
version of insertion sort on n items (big Oh)?

6 points

I took off more points for inconsistancies between the answers...

5) (20 points) The integer square root of integer n (SQRT(n)) is the largest integer x such
that . For example, SQRT(8) = 2, while SQRT(9) = 3.
Write a Modula-3 function to compute SQRT(n). For full credit, your algorithm should
run in time. Partial credit will be given for an algorithm.

(Hint: think about the ideas behind binary search)

PROCEDURE sqrt(n : INTEGER):INTEGER =

var
low, high, mid : integer;

low := 1;
high := n;

while (high - low) > 1 do

mid := (high+low) div 2;
if (mid * mid) > n then low := mid+1;
else high := mid;
end;

return (mid);
end;