Stony Brook University

Department of Computer Science

CSE 353: Machine Learning
(Spring 2015)
Instructor: Luis E. Ortiz

Programming Homework 1: Model Selection

and Evaluation, Cross Validation, Nearest
Neighbors, Neural Networks, and SVMs;
Application: Classification for Optical
Handwritten Digit Recognition
Recognizing Handwritten Digits Automatically
In this homework you will apply several multi-class classification machine-learning techniques (i.e.,
nearest neighbors, neural networks, and SVMs) to a stylized version of the problem of optical
recognition of handwritten digits.
The data set of examples is the result of preprocessing 32x32 bitmap images of hanwritten digits
(i.e., the representation of each image is originally a 32x32 matrix of pixels, each pixel taking value
1 or 0 corresponding to a black or white pixel, respectively). The result of the preprocessing is an
8x8 grayscale image where each pixel takes an integer value from 0 to 16. 1
The following is an example for the digit 9:
Original Image

Transformed Image

For each preprocessed image, each pixel corresponds to an attribute taking one of 16 values.
Thus, the data has 64 attributes. Although each attribute is an integer from 0 to 16, you must
treat each attribute as real-valued. There are 10 classes corresponding to each of digit.
You are provided five (5) data files.
optdigits train.dat contains (a permuted version of) the original training data.
optdigits train trans.dat contains the transformed image version of the examples in the
file optdigits train.dat.
optdigits test trans.dat contains the transformed image version of the test examples.
optdigits trial.dat contains an example of each digit from the original validation set.
1

The reason for the preprocessing is to normalize the data to help correct for small distortions and reduce
dimensionality. The resulting images provide a good approximation of the original images for classification purposes.
Please visit http://archive.ics.uci.edu/ml/datasets/Optical+Recognition+of+Handwritten+Digits at the UCI
Machine Learning Repository [Lichman, 2013] for more information about this data set.

Page 1 of 8

Programming Homework 1

CSE 353: Machine Learning

Spring 2015

optdigits trial trans.dat contains the transformed image version of the examples from
optdigits trial.dat.
Each file contains one data example per line. Each line contains 65 integers separated by a
single empty space. The first 64 integers correspond to the values of each of the attributes (i.e., a
number from 0 to 16) and the last integer is the example class label (i.e., the corresponding digit
0, 1, . . . , 9). The training and test data sets have 3823 and 1797 examples, respectively.
You are asked to implement the technique of K-nearest neighbors classification using Euclidean
distance as the distance metric and apply it, using K = 1 and K = 3, to the data set of optical
handwritten digits described above. As a tie-breaking rule during classification, select the lowest
digit as the class label (i.e., if there is a tie between 3 and 7, pick 3 as the label).

Applying K-Nearest Neighbors

For this part, you will implement a K-nearest neighbor classifier for K = 1 and K = 3.

1-Nearest Neighbor
Here, you will evaluate the performance of a 1-nearest neighbor classifier by producing its learning
graph, which is a plot of the test error as a function of the number of training examples (i.e., the size
of the training set). In particular, compute and display/plot the learning graph for training sets consisting of the first m = 10, 50, 100, 500, 1000, 3823(all) examples in the optdigits train trans.dat
file. Recall that the test error is the misclassification error obtained from evaluating the classifier
on the test data. The file optdigits test trans.dat contains the test data, which in this case is
the transformed version of the images.

3-Nearest Neighbors
For each image in the trial set, provide the indexes to the 3-nearest neighbor examples in the
training dataset. The index to an example is the row, or line number, of the file with the training
dataset.
1. For each transformed image example in the trial data set given in the optdigits trial trans.dat
file, identify the indexes to the 3-nearest neighbors in the training dataset in the transformed
space, using the training data file of transformed images given in optdigits train trans.dat.
2. List, in increasing order of Euclidean distance, the indexes to the 3-nearest neighbors you
identified for each of the 10 exemplars trial dataset, include their respective labels. Also
provide the output labels of the corresponding 3-nearest-neighbors classifier for each of the
trial examples. How many of the trial examples are correctly classified by the 3-nearestneighbors classifier?
3. (Optional) Display the corresponding original 32x32 pixels, B&W images, which you can find
in the optdigits train.dat file, of the 3-nearest neighbors of each of the 10 exemplars in the
trial data set, along with their respective labels. Display the images as a row, starting with
the exemplar itself (whose original image you get from the file optdigits trial.dat), and

Page 2 of 8

Programming Homework 1

CSE 353: Machine Learning

Spring 2015

followed by the original images for the 3-nearest neighbors, in increasing order of Euclidean
distance.
NOTE: The first 1024 binary values of each line in the files optdigits train.dat and
optdigits trial.dat encode the original bitmap images as bit vectors. You need to appropriately reshape that vector to obtain the original bitmap image in matrix form.

Applying Neural Networks

You will apply the neural-network code provided, including implementations of ForwardProp
and BackProp. The subsection entitled The ML Neural-Network Black-box provides a
description of how to use the code.
Model and Implementation Details. All the neural network units are sigmoidal. The error
function used for BackProp is the mean squared error.
Neural Network Architectures: You will use two neural network architectures, depending
on whether you are using the data in its transformed or original form:
1. For the transformed data, you will use a perceptron with 64 input units and 10 output units.
For learning, you will run BackProp using a learning rate = 1 for a maximum of 1000
rounds. (Note that we are denoting the learning rate of BackPropby , not , to avoid
confusion with the SVM example-weights which are most commonly denoted by .)
2. For the original data, you will use a neural network with 1 hidden layer, 1024 input units, 64
units in the hidden layer and 10 output units. For learning, you will run BackProp using a
learning rate = 21 for a maximum of 200 rounds.
Weight Initialization and Extra Stopping Conditions: The implementation of BackProp provided in the source code initializes each neural-network weight (including thresholds)
with a number independently drawn at random from the uniform distribution over [1, 1]. The
implementation stops BackProp whenever, from one round to the next, the change in the error
function value is 106 or if the maximum change in neural network weight values is 104 .
Evaluation and Report. The implementation provided
1. records
(a) the training and test error function values and
(b) the training and test misclassification error rate
for the neural network at each round of BackProp; and
2. keeps track of the neural network with lowest training error during any round of BackProp.
For each of the two neural-network architectures described above, use the information that the
neural-network code provides to
1. plot
(a) the training and test error function values together in one graph and
Page 3 of 8

Programming Homework 1

CSE 353: Machine Learning

Spring 2015

(b) the training and test misclassification error rate together on a separate graph
as a function of the number of rounds of BackProp. Include these two graph plots, along
with a brief discussion of the results, in your report.
2. report the value of the error function on the training and test data, and the train and test
misclassification error rate. Tabulate and include those values in your report.
3. evaluate each example in the trial data set on each neural network learned and report the
corresponding output classification.

The ML Neural-Network Black-box

You will use an implementation of neural networks supplied with the programming homework files.
Neural-Network Learning Code. The bash-shell script learner nnet script is a wrapper for
the BackProp algorithm for learning neural networks. This is how it works. Say that given some
training data, we want to learn a neural network, defined in terms of matrices Wr+1,r corresponding
to the weights on the connections between consecutive layers r and r + 1. If the neural network
has H + 2 layers, where H is the number of hidden layers, r = 0 corresponds to the input layer,
and r = H + 2 corresponds to the output, then for each r = 0, . . . , H + 1, Wr+1,r would have
nr+1 rows and nr columns, corresponding to the number of units in each layer, respectively. In this
homework, we want to learn two neural networks, one using the transformed digit data and the
other using the untransformed/original digit data.
To learn the first network for the transformed data, first copy the files optdigits tra trans.dat
and optdigits cv trans.dat, corresponding to the training and test files for the transformed version of the digits data, into the files D.dat and D cv.dat, respectively, and place them in the same
directory as the neural-network code provided. You then run the following bash-shell script as
./learn nnet script [64 10] 1000 1 1e-4 1e-6
which will produce the following files corresponding to a perceptron (i.e., two-layer neural network
corresponding to the input and output units, without any hidden layers) formed of 64 input units
and 10 output units, obtained by running BackProp on the training data file D.dat using a
maximum of 1000 iterations, a learning-rate/step-size value of 1, a value of 104 as a stopping
threshold on the minimum change in the weight parameter values between iterations, and a value
of 106 as a stopping threshold on the minimum change in the training error-function values
between iterations.
best nnet W 1 0.dat
last nnet W 1 0.dat
train err.dat
test err.dat

best nnet err final.dat

last nnet err final.dat
train err rate.dat
test err rate.dat

The first two rows above corresponds to the weight files for each consecutive layer and error-related
information for the best and last neural network obtained during BackProp. The weight file
best nnet W 1 0.dat contains the weights parameters between the first and second layer as a table
with 10 rows and 65 columns (64 + 1 to account for the offset threshold parameter that is input
to each unit of each layer). The error information file best nnet err final.dat contains a single
Page 4 of 8

Programming Homework 1

CSE 353: Machine Learning

Spring 2015

row of values of the error function on the training and test data, followed by the values of the
training and test error rates of the neural network. The case for the file last nnet W 1 0.dat
and last nnet err final.dat is similar. (You will not be using the files last nnet W 1 0.dat
and last nnet err final.dat in this homework.) The last two rows of the table of files above
corresponds to the training and test error function values and rates that the neural networks found
at each iteration of BackProp achieve. Each one of those files is a sequence of values (written
as a single column in the file) for each round of BackProp. You can use those values to plot the
evolution of the respective error values during BackProp.
Neural-Network Classification Code. Given the weight-parameter files for a neural network
obtained using the learning bash-shell script just described, we can use the neural network for
classification. To do this, we first copy all the inputs of all the examples we want to classify in
the file X.dat, one example per row, and one column per input feature in the file. Suppose the
neural network architecture is a perceptron with 64 inputs units and 10 output units. Then copy
the weight parameters into the file W 1 0.dat. (For example, if we want to classify using the best
neural network found during BackProp in the example above, we would simply copy the content
of the file best nnet W 1 0.dat into the file W 1 0.dat.) Then we execute the following bash-shell
scripts
./classify nnet script [64 10]
which will produce the file Y.dat with the classification output that the neural network we are
evaluating assigns to each of the examples in the file X.dat; the classification of each example in
X.dat appears as a single column in Y.dat, in the same order as given in X.dat.
On the Neural-Network Black-box. The implementation of the actual neural-network algorithms is in a Matlab-like language called Octave. The wrapper assumes that you have installed
either Octave or Matlab. 2 You need to edit the line in the script corresponding to the call to
Matlab or Octave, and replace it with the correct command call and path to the Matlab or Octave
program installed in your system. (The source code was not tested in Matlab; it may need minor
modifications to run in that environment.) Once you have specified a program and edited the script
accordingly, the wrapper will call the corresponding program for you.

Applying Support Vector Machines (SVMs)

You will now evaluate the performance of SVMs on the original/untransformed digit data. NOTE:
As for neural networks, you do not need to implement an SVM learning algorithm. You only
need to use one already provided for you along with the programming homework files. Please see
the subsection entitile The ML SVM Black-box for more information on how to use the
implementation.
Model and Implementation Details. The following are experimental details on the application
of the learning algorithm and the different SVM instances you will consider.
2

GNU Octave is freely distributed software. See http://www.gnu.org/software/octave/ for download information.

Page 5 of 8

Programming Homework 1

CSE 353: Machine Learning

Spring 2015

One-against-all Approach to Multiclass Problems: You will use a one-against-all approach to cast the L-class classification problem into L individual binary classification problems.
In this approach, for each class c, you will learn an SVM using the provided code to discriminate between class c and the rest, where class c examples correspond to the positive examples (y = +1), while examples of the other classes are the negative examples (y = 1). Suppose that
Pthe SMV (l)binary(l)classifier learned this way for class c is Hc (x) = sign (fc (x)), where
fc (x) = m
l=1 l (c)y K(x , x) + b(c). Then, we built an overall multi-class classifier H as
H(x) Hc (x) where c arg max fc (x) .
c=1,...,L

Kernel functions: Recall the following kernel functions.

polynomial kernel of degree d: K(x, z) = (1 + x z)d
Gaussian/RBF kernel with inverse-variance parameter 1/ 2 : K(x, z) = exp 12 kx zk2
For this homework, you will consider the following specific instances of SVMs:
polynomial kernels of three different degrees d = 1 (i.e., linear kernel), 4 and 10
Gaussian/RBF kernel with three different inverse-variance parameter = 0.1, 0.027 and 0.001
Initialization and Stopping Conditions: The implementation initializes the dual variables/example weights as (1) = 0. You must run the SVM learning algorithm for a minimum of
15 iteration. Stop after a maximum of 5000 iterations or whenever the relative change in value of
the dual objective function g evaluated at the dual variables (t) and (t+1) obtained at consecutive
iteration t and t + 1 of the learning algorithm satisfies
g((t+1) ) g((t) )
103 .
g((t) )
Evaluation and Report. Perform the following steps, for each SVM instance listed above, using
the original/untransformed digit data only.
Report the training and test misclassification error rate for each SVM-based multiclass classifier H learned
Also report the training and test misclassification error rate for each binary classifier SVM
Hk learned for class c
Evaluate each example in the trial data set on each SVM learned and report the corresponding
output classification

Page 6 of 8

Programming Homework 1

CSE 353: Machine Learning

Spring 2015

The ML SVM Black-box

You will use an implementation of SVMs supplied with the programming homework files.
SVM Learning Code. The bash-shell script learner svm script is a wrapper for the implementation of a projected-gradient learning algorithm for multiclass SVMs using a one-against-all
approach. The code also evaluates the training and test misclassification error rate using the train
and test sets given as input. Finally, the script also outputs the classification of a given trial set of
examples. All relevant information appears through standard output (i.e., stdout).
To use the script, first copy the training, test, and trial datasets into the files D.dat, D cv.dat
and D trial.dat, respectively. Then you may need to modify the file of the corresponding kernel
function to set the corresponding parameter. For example, if you want to use a polynomial kernel,
then you may need to modify the polynomial-degree parameter : in the file kernel poly.m, change
the line
d = 1;
to the appropriate degree. (The line above corresponds to a linear kernel.) So for a quadratic
kernel, that line should be
d = 2;
which changes the degree of the polynomial kernel to 2. Similarly, to use a Gaussian kernel, you
must change the following line in file kernel gauss.m,
inv var = 2.7e-2;
corresponding to the inverse-variance parameter accordingly.
Having copied the respective data file and adapted the appropriate kernel function, executing,
for instance, the bash-shell script
./learn svm script poly 5000 15 1e-3 10
will output all the relevant results of running a projected-gradient learning algorithm, using a
polynomial kernel, for a maximum of 5000 iterations, a minimum of 15 iterations, a value of 103
for the minimum relative improvement in the dual objective function for the SVM, and for a
problem with 10 output labels. To use the Gaussian kernel instead of the polynomial kernel in the
learning script, replace the first script input, poly, by the string gauss in the command line
given above.
On the SVM Black-box. Just like for neural networks, the implementation of the SVM algorithm is in a Matlab-like language called Octave. The wrapper assumes that you have installed
either Octave or Matlab. 3 You need to edit the line in the script corresponding to the call to
Matlab or Octave, and replace it with the correct command call and path to the Matlab or Octave
program installed in your system. (The source code was not tested in Matlab; it may need minor
modifications to run in that environment.) Once you have specified a program and edited the script
accordingly, the wrapper will call the corresponding program for you.

GNU Octave is freely distributed software. See http://www.gnu.org/software/octave/ for download information.

Page 7 of 8

Programming Homework 1

CSE 353: Machine Learning

Spring 2015

What to Turn In
You need to submit the following (electronically via Blackboard):
1. A written report (in PDF) that includes the information/plots/graphs/images requested
above along with a brief discussion of all your results. For example, for nearest-neighbors,
include plots, list of indexes, ordered by Euclidean distance, classifiers output labels, answer
to question, and, for the optional part, a grid of images, properly formatted per the layout
instructions given above, along with a brief discussion. In your discussion, compare and
contrast the different classifier models and learning algorithms based on your results.
2. All your code and executable (as a tared-and-gziped compressed file), with instructions on
how to run your program. A platform-independent executable is preferred; otherwise, also
provide instructions on how to compile your program. Please use standard tools/compilers/etc.,
generally available in most popular platforms.
Collaboration Policy: It is OK to discuss the homework with peers outside your own group,
but each group of students must write and turn in their own report, code, etc., based on their own
work. In addition, every member of the team must be able to answer any question on any aspect of
the submitted report and source code.

References
M. Lichman. UCI machine learning repository, 2013. URL http://archive.ics.uci.edu/ml.

Page 8 of 8