Artificial Intelligence
Binary Classifiers for Multi-
class Classification Problems
Pham Viet Cuong
Dept. Control Engineering & Automation, FEEE
Ho Chi Minh City University of Technology
Binary Classifiers for Multi-class Classification Problems
ü Classification: A predictive modeling problem that involves assigning a
class label to an example
ü Binary classification: Classification tasks with two classes
ü Multi-class classification: Classification tasks with more than two classes
ü Binary classifiers:
v Support Vector Machines
v Perceptron
v Logistic Regression
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 2
Binary Classifiers for Multi-class Classification Problems
ü Multi-class classifications problems:
v Handwritten digit (0, 1, . . ., 9)
v Image classification ILSVRC2012
v ...
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 3
Binary Classifiers for Multi-class Classification Problems
ü Binary classifiers for multi-class classifications problems:
v One-vs-One
v One-vs-Rest (One-vs-All)
v Hierachical
v Binary coding
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 4
Binary Classifiers for Multi-class Classification Problems
ü One-vs-Rest:
v Split a multi-class classification into one binary classification problem
per class
v A binary classifier is trained on each binary classification problem
v Predictions are made using the most confident model
v Example
§ Three class: sun, moon, star
§ Three binary classification problem
• Binary Classification Problem 1: sun vs [moon, star]
• Binary Classification Problem 2: moon vs [sun, star]
• Binary Classification Problem 3: star vs [sun, moon]
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 5
Binary Classifiers for Multi-class Classification Problems
ü Each model predicts a class membership probability (probability-like
score)
ü Class index with largest score (argmax) is used to predict a class
ü Common used for Logistic Regression, Perceptron
ü Cons:
v Requires one model for each class (issue for large datasets, slow
models, or very large numbers of classes)
v Scale of confidence values may differ between binary classifiers
v Unbalanced distributions between the sets of positives and negatives
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 6
Binary Classifiers for Multi-class Classification Problems
ü One-vs-One:
v Split a multi-class classification into one binary classification problem
per each pair of classes
v Fit a binary classifier on each binary classification problem
v Example:
§ Three class: sun, moon, star
§ Three binary classification problem
• Binary Classification Problem 1: sun vs moon
• Binary Classification Problem 2: sun vs star
• Binary Classification Problem 3: moon vs star
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 7
Binary Classifiers for Multi-class Classification Problems
ü Each data point: classified according to a majority vote
ü If each model predicts a class membership probability: class index with
largest sum score (argmax) is predicted the class label
ü Cons:
v Number of binary models: C*(C-1)/2, C: # classes
v More models than the one-vs-rest
v Requires one model for each pair of class (issue for large datasets,
slow models, or very large numbers of classes)
v Suffers from ambiguities (some regions of input space may receive
the same number of votes)
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 8
Binary Classifiers for Multi-class Classification Problems
ü Hierarchical classification:
v Divide the output space i.e. into a tree
v Each parent node is divided into multiple child nodes until each child
node represents one class
v Example
Flat classification
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 9
Binary Classifiers for Multi-class Classification Problems
ü Binary coding:
v Encode class label by binary numbers
v # binary classifiers = log2(C) (C: # classes)
x = ceil(x): smallest integer greater than or equal to x
v Least binary classifiers required
v What if a bit incorrectly determinded?
v What if # classes not a power of 2?
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 10
Binary Classifiers for Multi-class Classification Problems
ü Linearly separable data:
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 11
Artificial Neural Networks
ü References
v https://machinelearningmastery.com/one-vs-rest-and-one-vs-one-for-
multi-class-classification/
v https://en.wikipedia.org/wiki/Artificial_neural_network
Pham Viet Cuong - Dept. Control Eng. & Automation, FEEE, HCMUT 12
