UNIT-5

Classification and
Prediction
Jiawei Han, Micheline Kamber, and Jian Pei
University of Illinois at Urbana-Champaign &
Simon Fraser University
©2011 Han, Kamber & Pei. All rights reserved.
1
UNIT - V

Classification and Prediction : Classification by

decision tree induction, Bayesian Classification,
Rule-based Classification, Associative Classification

2
 What is
classification?
• A bank loans officer needs analysis of customer data
in order to learn which loan applicants are ―safe
and which are ―risky for the bank.
• A marketing manager at a company needs data to
help guess whether a customer with a given profile
will buy a new computer.
• A medical researcher wants to analyze breast cancer
data in order to predict which one of three specific
treatments a patient should receive.

3
 In each of these examples, the data analysis task is
classification, where a model or classifier is
constructed to predict categorical labels,such as
 ―safe or ―risky for the loan application data;
 ―yes or ―no for the marketing data;
 ―treatment A ―treatment B or ―treatment C for
the medical data.
 Suppose that the marketing manager would like to
predict how much a given customer will spend
during a sale at a company.

This data analysis task is an example of numeric
prediction, where the model constructed predicts a
continuous-valued function, or ordered value, as
opposed to a categorical label.

4
 Supervised vs. Unsupervised
Learning
 Supervised learning (classification)
 Supervision: The training data (observations,
measurements, etc.) are accompanied by labels indicating
the class of the observations
 New data is classified based on the training set
 Unsupervised learning (clustering)
 The class labels of training data is unknown
 Given a set of measurements, observations, etc. with the
aim of establishing the existence of classes or clusters in
the data
5
 Prediction Problems:
Classification vs. Numeric
 Classification
Prediction
 predicts categorical class labels
 constructs a model based on the training set and the values

in a classifying attribute and uses it in classifying new data

 Numeric Prediction
 models continuous-valued functions, i.e., predicts

unknown or missing values

 Regression analysis is a statistical methodology that is most

often used for numeric prediction

 Typical applications
 Credit/loan approval:

 Medical diagnosis: if a tumor is cancerous or benign

 Fraud detection: if a transaction is fraudulent

6
 Classification—A Two-Step
Process
 Model construction: describing a set of predetermined classes

Each tuple/sample is assumed to belong to a predefined class,
as determined by the class label attribute

The set of tuples used for model construction is training set

The model is represented as classification rules, decision trees,
or mathematical formulae
 Model usage: for classifying future or unknown objects

Estimate accuracy of the model

The known label of test sample is compared with the
classified result from the model

Accuracy rate is the percentage of test set samples that are
correctly classified by the model

Test set is independent of training set (otherwise overfitting)
7
  If the accuracy is acceptable, use the model to classify new
data
 Note: If the test set is used to select models, it is called
validation (test) set

8
 Process (1): Model
Construction

Classification
Algorithms
Training
Data

NAME RANK YEARS TENURED Classifier

Mike Assistant Prof 3 no (Model)
Mary Assistant Prof 7 yes
Bill Professor 2 yes
Jim Associate Prof 7 yes IF rank = ‘professor’
Dave Assistant Prof 6 no
OR years > 6
Anne Associate Prof 3 no
THEN tenured = ‘yes’
9
Process (2): Using the Model in
Prediction

Classifier

Testing
Data Unseen Data

(Jeff, Professor, 4)
NAME RANK YEARS TENURED
Tom Assistant Prof 2 no Tenured?
Merlisa Associate Prof 7 no
George Professor 5 yes
Joseph Assistant Prof 7 yes
10
 Decision Tree Induction
 A decision tree algorithm known as ID3 (Iterative
Dichotomiser).
 A decision tree is a flowchart-like tree structure,
 Where each internal node (nonleaf node) denotes a test on an
attribute,
 Each branch represents an outcome of the test
 Each leaf node (or terminal node) holds a class label
 The topmost node in a tree is the root node
 Tree is constructed in a top-down recursive divide-and-
conquer manner

11
12
 Attributes are categorical (if continuous-valued, they are
discretized in advance)
 Test attributes are selected on the basis of a heuristic or
statistical measure (e.g., Information gain or Gini Index)

13
 Decision Tree Induction: An
Example
age income student credit_rating buys_computer
<=30 high no fair no
 Training data set: Buys_computer <=30 high no excellent no
 The data set follows an example of 31…40 high no fair yes
>40 medium no fair yes
Quinlan’s ID3 (Playing Tennis) >40 low yes fair yes
 Resulting tree: >40 low yes excellent no
31…40 low yes excellent yes
age? <=30 medium no fair no
<=30 low yes fair yes
>40 medium yes fair yes
<=30 medium yes excellent yes
<=30 overcast
31..40 >40 31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no

student? yes credit rating?

no yes excellent fair

no yes no yes
14
15
16
17
 Decision tree algorithm- ID3/
Example 1

April 27, Data Mining: Concepts and 18

April 27, Data Mining: Concepts and 19
April 27, Data Mining: Concepts and 20
April 27, Data Mining: Concepts and 21
22
23
24
25
26
27
28
29
30
 Gini Index
 If a data set D contains examples from n classes, gini index, gini(D)
is defined as n 2
gini( D) 1  p j
j 1
where pj is the relative frequency of class j in D
 If a data set D is split on A into two subsets D1 and D2, the gini
index gini(D) is defined as |D | |D |
gini A ( D)  1 gini( D1)  2 gini( D 2)
|D| |D|
 Reduction in Impurity:
gini( A) gini( D)  giniA ( D)
 The attribute provides the smallest ginisplit(D) (or the largest
reduction in impurity) is chosen to split the node (need to
enumerate all the possible splitting points for each attribute)
31
32
 Overfitting and Tree Pruning
 Overfitting: An induced tree may overfit the training data
 Too many branches, some may reflect anomalies due to

noise or outliers
 Poor accuracy for unseen samples

 Two approaches to avoid overfitting

 Prepruning: Halt tree construction early ̵ do not split a node

if this would result in the goodness measure falling below a

threshold

Difficult to choose an appropriate threshold
 Postpruning: Remove branches from a “fully grown” tree—

get a sequence of progressively pruned trees


Use a set of data different from the training data to
decide which is the “best pruned tree” 33
 Rainforest: Training Set and Its
AVC Sets

Training Examples AVC-set on Age AVC-set on income

age income studentcredit_rating
buys_computerAge Buy_Computer income Buy_Computer

<=30 high no fair no yes no

yes no
<=30 high no excellent no
high 2 2
31…40 high no fair yes <=30 2 3
31..40 4 0 medium 4 2
>40 medium no fair yes
>40 low yes fair yes >40 3 2 low 3 1
>40 low yes excellent no
31…40 low yes excellent yes
AVC-set on
<=30 medium no fair no AVC-set on Student
credit_rating
<=30 low yes fair yes
student Buy_Computer Buy_Computer
>40 medium yes fair yes
Credit
<=30 medium yes excellent yes yes no
rating yes no
31…40 medium no excellent yes yes 6 1 fair 6 2
31…40 high yes fair yes no 3 4 excellent 3 3
>40 medium no excellent no
34
 Bayesian Classification:
Why?
 A statistical classifier: performs probabilistic prediction, i.e.,
predicts class membership probabilities
 Foundation: Based on Bayes’ Theorem.
 Performance: A simple Bayesian classifier, naïve Bayesian
classifier, has comparable performance with decision tree and
selected neural network classifiers
 Incremental: Each training example can incrementally
increase/decrease the probability that a hypothesis is correct —
prior knowledge can be combined with observed data
 Standard: Even when Bayesian methods are computationally
intractable, they can provide a standard of optimal decision
making against which other methods can be measured
35
 Bayes’ Theorem: Basics
M
 Total probability Theorem: P(B)   P(B | Ai )P( Ai )
i 1

 Bayes’ Theorem: P( H | X) P(X | H ) P( H ) P(X | H )P( H ) / P(X)

P(X)
 Let X be a data sample (“evidence”): class label is unknown
 Let H be a hypothesis that X belongs to class C
 Classification is to determine P(H|X), (i.e., posteriori probability): the
probability that the hypothesis holds given the observed data sample X
 P(H) (prior probability): the initial probability

E.g., X will buy computer, regardless of age, income, …
 P(X): probability that sample data is observed
 P(X|H) (likelihood): the probability of observing the sample X, given that
the hypothesis holds

E.g., Given that X will buy computer, the prob. that X is 31..40,
medium income
36
 Prediction Based on Bayes’
Theorem
 Given training data X, posteriori probability of a hypothesis H,
P(H|X), follows the Bayes’ theorem

P(H | X) P(X | H ) P( H ) P(X | H )P( H ) / P(X)

P(X)
 Informally, this can be viewed as
posteriori = likelihood x prior/evidence
 Predicts X belongs to Ci iff the probability P(Ci|X) is the highest
among all the P(Ck|X) for all the k classes
 Practical difficulty: It requires initial knowledge of many
probabilities, involving significant computational cost
37
Bayes’ Theorem

38
39
40
41
42
Classification Is to Derive the Maximum
Posteriori
 Let D be a training set of tuples and their associated class
labels, and each tuple is represented by an n-D attribute vector
X = (x1, x2, …, xn)
 Suppose there are m classes C1, C2, …, Cm.
 Classification is to derive the maximum posteriori, i.e., the
maximal P(Ci|X)
 This can be derived from Bayes’ theorem
P(X | C )P(C )
P(C | X)  i i
i P(X)

 Since P(X) is constant for all classes, only

P(C | X) P(X | C )P(C )
i i i
needs to be maximized

43
 Naïve Bayes Classifier: Training
Dataset
age income studentcredit_rating
buys_compu
<=30 high no fair no
Class: <=30 high no excellent no
C1:buys_computer = ‘yes’ 31…40 high no fair yes
C2:buys_computer = ‘no’ >40 medium no fair yes
>40 low yes fair yes
>40 low yes excellent no
Data to be classified:
31…40 low yes excellent yes
X = (age <=30,
<=30 medium no fair no
Income = medium, <=30 low yes fair yes
Student = yes >40 medium yes fair yes
Credit_rating = Fair) <=30 medium yes excellent yes
31…40 medium no excellent yes
31…40 high yes fair yes
>40 medium no excellent no
44
 Naïve Bayes Classifier: An
Example age income studentcredit_rating
buys_comp
<=30 high no fair no
<=30 high no excellent no
31…40 high no fair yes

P(Ci): P(buys_computer = “yes”) = 9/14 = 0.643 >40
>40
>40
medium
low
low
no fair
yes fair
yes excellent
yes
yes
no

P(buys_computer = “no”) = 5/14= 0.357 31…40

<=30
low
medium
yes excellent
no fair
yes
no
<=30 low yes fair yes

Compute P(X|Ci) for each class >40
<=30
medium yes fair
medium yes excellent
yes
yes
31…40 medium no excellent yes

P(age = “<=30” | buys_computer = “yes”) = 2/9 = 0.222 31…40

>40
high
medium
yes fair
no excellent
yes
no

P(age = “<= 30” | buys_computer = “no”) = 3/5 = 0.6

P(income = “medium” | buys_computer = “yes”) = 4/9 = 0.444
P(income = “medium” | buys_computer = “no”) = 2/5 = 0.4
P(student = “yes” | buys_computer = “yes) = 6/9 = 0.667
P(student = “yes” | buys_computer = “no”) = 1/5 = 0.2
P(credit_rating = “fair” | buys_computer = “yes”) = 6/9 = 0.667
P(credit_rating = “fair” | buys_computer = “no”) = 2/5 = 0.4
 X = (age <= 30 , income = medium, student = yes, credit_rating = fair)
P(X|Ci) : P(X|buys_computer = “yes”) = 0.222 x 0.444 x 0.667 x 0.667 = 0.044
P(X|buys_computer = “no”) = 0.6 x 0.4 x 0.2 x 0.4 = 0.019
P(X|Ci)*P(Ci) : P(X|buys_computer = “yes”) * P(buys_computer = “yes”) = 0.028
P(X|buys_computer = “no”) * P(buys_computer = “no”) = 0.007
Therefore, X belongs to class (“buys_computer = yes”) 45
Rule-Based Classification

46
 Rule-Based Classification

• A rule-based classifier uses a set of IF-THEN rules for

classification.
• An IF-THEN rule is an expression of the form

LHS:- rule antecedent or precondition.

RHS:- rule consequent.

47
 If the condition (i.e., all the attribute tests) in a rule antecedent
holds true for a given tuple, we say that the rule antecedent is
satisfied (or simply, that the rule is satisfied)
 If the condition in antecedent is available then that the rule
covers the tuple

48
 Quality of rule is decided by coverage and accuracy
 A rule R can be assessed by its coverage and accuracy
 Given a tuple, X, from a class labeled data set, D,
 let ncovers be the number of tuples covered by R;
 ncorrect be the number of tuples correctly classified by R;
 and |D| be the number of tuples in D.
 We can define the coverage and accuracy of R as

49
How rule-based classifier Work

50
51
Solution to first conflict(tuple satisfy both rule)

 we need a conflict resolution strategy to figure out which rule

gets to fire and assign its class prediction
 There are many possible strategies.
 Size ordering and
 Rule ordering
Size Ordering
 The size ordering scheme assigns the highest priority to the
triggering rule that has the “toughest” requirements,
 Where toughness is measured by the rule antecedent size.
 That is, the triggering rule with the most attribute tests is fired
Rule Ordering
 scheme prioritizes the rules beforehand
 The ordering may be class-based or rule-based

52
Rule-Based Ordering
•Individual rules are rank bases on quality of rule
•Rules are unorder

Class-Based Ordering
•Rules that belong to the same class appear together
•Rules are ordered

53
 Solution to second conflict(tuple satisfy no rule)

• Default rule can be set up to specify a default class, based on

a training set
• This may be the class in majority or the majority class of the
tuples that were not covered by any rule.
• The default rule is evaluated at the end, if and only if no
other rule covers X.
• The condition in the default rule is empty. In this way, the
rule fires when no other rule is satisfied.

54
Building Classification Rule

Rule Characteristic:-
Mutually exclusive :-
Mutually exclusive means that we cannot have rule conflicts
here because no two rules will be triggered for the same tuple.
(We have one rule per leaf, and any tuple can map to only one
leaf.)
Exhaustive:-
means there is one rule for each possible attribute–value
combination, so that this set of rules does not require a default
rule.

55
Direct Method
•Rules are extracted from data
•Example:- Sequential Covering Algorithm
Indirect Method
•Extract rules from other classification Method
•Example:-Decision Tree, Neural Network

56
 Sequential Covering Algorithm

• IF-THEN rules can be extracted directly from the training

data
• The name comes from the notion that the rules are
learned sequentially
• Rules are learned one at a time.
• Each time a rule is learned, the tuples covered by the
rule are removed, and the process repeats on the
remaining tuples

57
58
59
60
Rule Growing(How are rule learn?

61
 Learn_One_Rule() needs a measure of rule quality.
 Accuracy may seem like an obvious choice at first, but
consider Example
 Thus, R2 has greater accuracy than R1, but it is not the better
rule because of its small coverage.

62
Indirect Method

63
 Rule Extraction from a Decision
Tree
 Rules are easier to understand than large
trees age?
 One rule is created for each path from the <=30 31..40 >40
root to a leaf student? credit rating?
yes
 Each attribute-value pair along a path forms a
no yes excellent fair
conjunction: the leaf holds the class
no yes no yes
prediction
 Rules are mutually exclusive and exhaustive
 Example: Rule extraction from our buys_computer decision-tree
IF age = young AND student = no THEN buys_computer = no
IF age = young AND student = yes THEN buys_computer = yes
IF age = mid-age THEN buys_computer = yes
IF age = old AND credit_rating = excellent THEN buys_computer = no
IF age = old AND credit_rating = fair THEN buys_computer = yes
64
 Associative Classification(ARM+Classification)

• Association rules are mined in a two-step process

• The first step searches for patterns of attribute–value pairs
that occur repeatedly in a data set(also referred to as frequent
patterns).
• The second step analyzes the frequent itemset to generate
association rules.
• All association rules must satisfy certain criteria regarding their
“accuracy” (or confidence) and the proportion of the data set
that they actually represent (referred to as support).
• For example, the following is an association rule mined from a
data set, D, shown with its confidence and support:

65
• From a classification point of view, this is akin to rule accuracy.
• For example
• A confidence of 93% for Rule means that 93% of the customers in
D who are young and have an OK credit rating belong to the class
buys computer = yes.
• The percentage of tuples in D satisfying the rule antecedent and
having class label C is called the support of R.
• A support of 20% for Rule above means that 20% of the
customers in D are young, have an OK credit rating

66
67
68
69
 Classifier Evaluation Metrics:
Confusion Matrix
Confusion Matrix:
Actual class\Predicted class C1 ¬ C1
C1 True Positives (TP) False Negatives (FN)
¬ C1 False Positives (FP) True Negatives (TN)

Example of Confusion Matrix:

Actual class\Predicted buy_computer buy_computer Total
class = yes = no
buy_computer = yes 6954 46 7000
buy_computer = no 412 2588 3000
Total 7366 2634 10000
 Given m classes, an entry, CMi,j in a confusion matrix indicates
# of tuples in class i that were labeled by the classifier as class j
 May have extra rows/columns to provide totals
70
 Accuracy, Error Rate,
Sensitivity and Specificity
A\P C ¬C  Class Imbalance Problem:
C TP FN P  One class may be rare, e.g.
¬C FP TN N
fraud, or HIV-positive
P’ N’ All
 Significant majority of the

 Classifier Accuracy, or negative class and minority of

recognition rate: percentage of the positive class
test set tuples that are correctly  Sensitivity: True Positive
classified recognition rate
Accuracy = (TP + TN)/All 
Sensitivity = TP/P
 Error rate: 1 – accuracy, or  Specificity: True Negative

Error rate = (FP + FN)/All recognition rate


Specificity = TN/N
71
 Precision and Recall, and F-
measures
 Precision: exactness – what % of tuples that the classifier
labeled as positive are actually positive

 Recall: completeness – what % of positive tuples did the

classifier label as positive?
 Perfect score is 1.0
 Inverse relationship between precision & recall

F measure (F1 or F-score): harmonic mean of precision and
recall,


Fß: weighted measure of precision and recall

assigns ß times as much weight to recall as to precision

72
 Classifier Evaluation Metrics:
Example

Actual Class\Predicted class cancer = yes cancer = no Total Recognition(%)

cancer = yes 90 210 300 30.00 (sensitivity
cancer = no 140 9560 9700 98.56 (specificity)
Total 230 9770 10000 96.40 (accuracy)
 Precision = 90/230 = 39.13% Recall = 90/300 = 30.00%

73