\((

P A N G . NI N G T A N
M i c h i g a nS t a t eU n i v e r s i t y

MICHAELSTEINBACH
U n i v e r s i toyf M i n n e s o t a

V I P I N K UM A R
U n i v e r s i toy f M i n n e s o t a
a n d A r m yH i g h P e r f o r m a n c e
ComputinR g e s e a r cC henter

+f.f_l crf.rfh. .W if f

aqtY6l$
t.T.R.C.
i'&'ufe61ttt1/.
Y \ t.\ $t,/,1'
'48!
n,5 \. 7\ V

Boston San Francisco NewYork

London Toronto Sydney Tokyo Singapore Madrid
MexicoCity Munich Paris CapeTown HongKong Montreal
 G.R
r+6,q If you purchased this book within the United States or Canada you should be aware that it has been
wrongfirlly imported without the approval of the Publishel or the Author.
- {)gq*
3 AcquisitionsEditor Matt Goldstein

T3 ProjectEditor Katherine Harutunian

Production Supervisor Marilyn Lloyd
Production Services Paul C. Anagnostopoulos of Windfall Software

Loo6 Marketing Manager

Copyeditor
Michelle Brown
Kathy Smith
Proofreader IenniferMcClain
Technicallllustration GeorgeNichols
Cover Design Supervisor Joyce Cosentino Wells
Cover Design Night & Day Design
Cover Image @ 2005 Rob Casey/Brand X pictures
hepress and Manufacturing Caroline Fell
Printer HamiltonPrinting

Access the latest information about Addison-Wesley titles from our iWorld Wide Web site:
http ://www. aw-bc.com/computing

Many of the designations used by manufacturers and sellers to distiriguish their products
are claimed as trademarks. where those designations appear in this book, and Addison-
Wesley was aware of a trademark claim, the designations have been printed in initial caps
or all caps.

The programs and applications presentedin this book have been incl,[rdedfor their
instructional value. They have been tested with care, but are not guatanteedfor any
particular purpose. The publisher does not offer any warranties or representations,nor does
it accept any liabilities with respect to the programs or applications.

Copyright @ 2006 by Pearson Education, Inc.

For information on obtaining permission for use of material in this work, please submit a
written request to PearsonEducation, Inc., Rights and Contract Department, 75 Arlington
Street,Suite 300, Boston, MA02II6 or fax your requestto (617) g4g-j047.

All rights reserved. No part of this publication may be reproduced, stored in a retrieval
system, or transmitted, in any form or by any means, electronic, mechanical, photocopying,
recording, or any other media embodiments now known or hereafter to become known,
without the prior written permission of the publisher. printed in the united Statesof
America.

lsBN0-321-42052-7

2 3 4 5 67 8 9 10-HAM-O807 06
our famili,es
Preface
Advances in data generation and collection are producing data sets of mas-
sive size in commerce and a variety of scientific disciplines. Data warehouses
store details of the salesand operations of businesses,Earth-orbiting satellites
beam high-resolution images and sensordata back to Earth, and genomicsex-
periments generate sequence,structural, and functional data for an increasing
number of organisms. The ease with which data can now be gathered and
stored has created a new attitude toward data analysis: Gather whatever data
you can whenever and wherever possible. It has become an article of faith
that the gathered data will have value, either for the purpose that initially
motivated its collection or for purposes not yet envisioned.
The field of data mining grew out of the limitations of current data anal-
ysis techniques in handling the challengesposedlby these new types of data
sets. Data mining does not replace other areas of data analysis, but rather
takes them as the foundation for much of its work. While some areas of data
mining, such as association analysis, are unique to the field, other areas, such
as clustering, classification, and anomaly detection, build upon a long history
of work on these topics in other fields. Indeed, the willingness of data mining
researchersto draw upon existing techniques has contributed to the strength
and breadth of the field, as well as to its rapid growth.
Another strength of the field has been its emphasis on collaboration with
researchers in other areas. The challenges of analyzing new types of data
cannot be met by simply applying data analysis techniques in isolation from
those who understand the data and the domain in which it resides. Often, skill
in building multidisciplinary teams has been as responsiblefor the successof
data mining projects as the creation of new and innovative algorithms. Just
as, historically, many developments in statistics were driven by the needs of
agriculture, industry, medicine, and business, rxrany of the developments in
data mining are being driven by the needs of those same fields.
This book began as a set of notes and lecture slides for a data mining
course that has been offered at the University of Minnesota since Spring 1998
to upper-division undergraduate and graduate Students. Presentation slides
viii Preface

and exercisesdeveloped in these offerings grew with time and served as a basis
for the book. A survey of clustering techniques in data mining, originally
written in preparation for research in the area, served as a starting point
for one of the chapters in the book. Over time, the clustering chapter was
joined by chapters on data, classification, association analysis, and anomaly
detection. The book in its current form has been class tested at the home
institutions of the authors-the University of Minnesota and Michigan State
University-as well as several other universities.
A number of data mining books appeared in the meantime, but were not
completely satisfactory for our students primarily graduate and undergrad-
uate students in computer science,but including students from industry and
a wide variety of other disciplines. Their mathematical and computer back-
grounds varied considerably, but they shared a common goal: to learn about
data mining as directly as possible in order to quickly apply it to problems
in their own domains. Thus, texts with extensive mathematical or statistical
prerequisiteswere unappealing to many of them, as were texts that required a
substantial database background. The book that evolved in responseto these
students needsfocusesas directly as possible on the key conceptsof data min-
ing by illustrating them with examples, simple descriptions of key algorithms,
and exercises.

Overview Specifically,this book provides a comprehensiveintroduction to

data mining and is designedto be accessibleand useful to students, instructors,
researchers,and professionals. Areas covered include data preprocessing,vi-
sualization, predictive modeling, associationanalysis, clustering, and anomaly
detection. The goal is to present fundamental concepts and algorithms for
each topic, thus providing the reader with the necessarybackground for the
application of data mining to real problems. In addition, this book also pro-
vides a starting point for those readerswho are interested in pursuing research
in data mining or related fields.
The book coversfive main topics: data, classification,associationanalysis,
clustering, and anomaly detection. Except for anomaly detection, each of these
areas is covered in a pair of chapters. For classification, association analysis,
and clustering, the introductory chapter covers basic concepts, representative
algorithms, and evaluation techniques, while the more advanced chapter dis-
cussesadvanced concepts and algorithms. The objective is to provide the
reader with a sound understanding of the foundations of data mining, while
still covering many important advanced topics. Becauseof this approach, the
book is useful both as a learning tool and as a reference.
 Preface ix

To help the readers better understand the concepts that have been pre-
sented, we provide an extensive set of examples, figures, and exercises. Bib-
Iiographic notes are included at the end of each chapter for readers who are
interested in more advanced topics, historically important papers, and recent
trends. The book also contains a comprehensivesubject and author index.

To the Instructor As a textbook, this book is suitable for a wide range

of students at the advanced undergraduate or graduate level. Since students
come to this subject with diverse backgrounds that may not include extensive
knowledgeof statistics or databases,our book requires minimal prerequisites-
no database knowledge is needed and we assume only a modest background
in statistics or mathematics. To this end, the book was designed to be as
self-contained as possible. Necessarymaterial from statistics, linear algebra,
and machine learning is either integrated into the body of the text, or for some
advanced topics, covered in the appendices.
Since the chapters covering major data mining topics are self-contained,
the order in which topics can be covered is quite flexible. The core material
is covered in Chapters 2, 4, 6, 8, and 10. Although the introductory data
chapter (2) should be covered first, the basic classification, association analy-
sis, and clustering chapters (4, 6, and 8, respectively) can be covered in any
order. Becauseof the relationship of anomaly detection (10) to classification
(4) and clustering (8), these chapters should precede Chapter 10. Various
topics can be selected from the advanced classification, association analysis,
and clustering chapters (5, 7, and 9, respectively) to fit the schedule and in-
terests of the instructor and students. We also advise that the lectures be
augmented by projects or practical exercisesin data mining. Although they
are time consuming, such hands-on assignmentsgreatly enhance the value of
the course.

Support Materials The supplementsfor the book are available at Addison-

Wesley's Website www.aw.con/cssupport. Support materials available to all
readers of this book include

o PowerPoint lecture slides

o Suggestionsfor student projects
o Data mining resourcessuch as data mining algorithms and data sets
o On-line tutorials that give step-by-stepexamplesfor selecteddata mining
techniquesdescribedin the book using actual data sets and data analysis
software
x Preface

Additional support materials, including solutions to exercises,are available

only to instructors adopting this textbook for classroom use. Please contact
your school's Addison-Wesley representative for information on obtaining ac-
cessto this material. Comments and suggestions,as well as reports of errors,
can be sent to the authors through dnbook@cs.unm.edu.

Acknowledgments Many people contributed to this book. We begin by

acknowledging our families to whom this book is dedicated. Without their
patience and support, this project would have been impossible.
We would like to thank the current and former students of our data mining
groups at the University of Minnesota and Michigan State for their contribu-
tions. Eui-Hong (Sam) Han and Mahesh Joshi helped with the initial data min-
ing classes.Some ofthe exercisesand presentation slides that they created can
be found in the book and its accompanying slides. Students in our data min-
ing groups who provided comments on drafts of the book or who contributed
in other ways include Shyam Boriah, Haibin Cheng, Varun Chandola, Eric
Eilertson, Levent Ertoz, Jing Gao, Rohit Gupta, Sridhar Iyer, Jung-Eun Lee,
Benjamin Mayer, Aysel Ozgur, Uygar Oztekin, Gaurav Pandey, Kashif Riaz,
Jerry Scripps, Gyorgy Simon, Hui Xiong, Jieping Ye, and Pusheng Zhang. We
would also like to thank the students of our data mining classesat the Univer-
sity of Minnesota and Michigan State University who worked with early drafbs
of the book and provided invaluable feedback. We specifically note the helpful
suggestionsof Bernardo Craemer, Arifin Ruslim, Jamshid Vayghan, and Yu
Wei.
Joydeep Ghosh (University of Texas) and Sanjay Ranka (University of
Florida) class tested early versions of the book. We also received many useful
suggestionsdirectly from the following UT students: Pankaj Adhikari, Ra-
jiv Bhatia, FbedericBosche, Arindam Chakraborty, Meghana Deodhar, Chris
Everson, David Gardner, Saad Godil, Todd Hay, Clint Jones, Ajay Joshi,
Joonsoo Lee, Yue Luo, Anuj Nanavati, Tyler Olsen, Sunyoung Park, Aashish
Phansalkar, Geoff Prewett, Michael Ryoo, Daryl Shannon, and Mei Yang.
Ronald Kostoff (ONR) read an early version of the clustering chapter and
offered numerous suggestions. George Karypis provided invaluable IATEX as-
sistance in creating an author index. Irene Moulitsas also provided assistance
with IATEX and reviewed some of the appendices. Musetta Steinbach was very
helpful in finding errors in the figures.
We would like to acknowledge our colleagues at the University of Min-
nesota and Michigan State who have helped create a positive environment for
data mining research. They include Dan Boley, Joyce Chai, Anil Jain, Ravi
 Preface xi

Janardan, Rong Jin, George Karypis, Haesun Park, William F. Punch, Shashi
Shekhar, and Jaideep Srivastava. The collaborators on our many data mining
projects, who also have our gratitude, include Ramesh Agrawal, Steve Can-
non, Piet C. de Groen, FYan Hill, Yongdae Kim, Steve Klooster, Kerry Long,
Nihar Mahapatra, Chris Potter, Jonathan Shapiro, Kevin Silverstein, Nevin
Young, and Zhi-Li Zhang.
The departments of Computer Scienceand Engineering at the University of
Minnesota and Michigan State University provided computing resourcesand
a supportive environment for this project. ARDA, ARL, ARO, DOE, NASA,
and NSF provided research support for Pang-Ning Tan, Michael Steinbach,
and Vipin Kumar. In particular, Kamal Abdali, Dick Brackney, Jagdish Chan-
dra, Joe Coughlan, Michael Coyle, Stephen Davis, Flederica Darema, Richard
Hirsch, Chandrika Kamath, Raju Namburu, N. Radhakrishnan, James Sido-
ran, Bhavani Thuraisingham, Walt Tiernin, Maria Zemankova, and Xiaodong
Zhanghave been supportive of our research in data mining and high-performance
computing.
It was a pleasure working with the helpful staff at Pearson Education. In
particular, we would like to thank Michelle Brown, Matt Goldstein, Katherine
Harutunian, Marilyn Lloyd, Kathy Smith, and Joyce Wells. We would also
like to thank George Nichols, who helped with the art work and Paul Anag-
nostopoulos, who provided I4.T[X support. We are grateful to the following
Pearson reviewers: Chien-Chung Chan (University of Akron), Zhengxin Chen
(University of Nebraska at Omaha), Chris Clifton (Purdue University), Joy-
deep Ghosh (University of Texas, Austin), Nazli Goharian (Illinois Institute
of Technology), J. Michael Hardin (University of Alabama), James Hearne
(Western Washington University), Hillol Kargupta (University of Maryland,
Baltimore County and Agnik, LLC), Eamonn Keogh (University of California-
Riverside), Bing Liu (University of Illinois at Chicago), Mariofanna Milanova
(University of Arkansas at Little Rock), Srinivasan Parthasarathy (Ohio State
University), Zbigniew W. Ras (University of North Carolina at Charlotte),
Xintao Wu (University of North Carolina at Charlotte), and Mohammed J.
Zaki (RensselaerPolvtechnic Institute).
Gontents
Preface vll

Introduction 1
1.1 What Is Data Mining? 2
7.2 Motivating Challenges 4
1.3 The Origins of Data Mining 6
1.4 Data Mining Tasks 7
1.5 Scope and Organization of the Book 11
1.6 Bibliographic Notes 13
t.7 Exercises 16

Data 19
2.I Types of Data 22
2.1.I Attributes and Measurement 23
2.L.2 Types of Data Sets . 29
2.2 Data Quality 36
2.2.I Measurement and Data Collection Issues 37
2.2.2 IssuesRelated to Applications 43
2.3 Data Preprocessing 44
2.3.L Aggregation 45
2.3.2 Sampling 47
2.3.3 Dimensionality Reduction 50
2.3.4 Feature Subset Selection 52
2.3.5 Feature Creation 55
2.3.6 Discretization and Binarization 57
2.3:7 Variable Tlansformation . 63
2.4 Measuresof Similarity and Dissimilarity . . . 65
2.4.L Basics 66
2.4.2 Similarity and Dissimilarity between Simple Attributes . 67
2.4.3 Dissimilarities between Data Objects . 69
2.4.4 Similarities between Data Objects 72
xiv Contents

2.4.5 Examples of Proximity Measures 73

2.4.6 Issuesin Proximity Calculation 80
2.4.7 Selecting the Right Proximity Measure 83
2.5 BibliographicNotes 84
2.6 Exercises 88

Exploring Data 97
3.i The Iris Data Set 98
3.2 Summary Statistics 98
3.2.L Frequenciesand the Mode 99
3.2.2 Percentiles 100
3.2.3 Measuresof Location: Mean and Median 101
3.2.4 Measuresof Spread: Range and Variance 102
3.2.5 Multivariate Summary Statistics 704
3.2.6 Other Ways to Summarize the Data 105
3.3 Visualization 105
3.3.1 Motivations for Visualization 105
3.3.2 General Concepts 106
3.3.3 Techniques 110
3.3.4 Visualizing Higher-Dimensional Data . 724
3.3.5 Do's and Don'ts 130
3.4 OLAP and Multidimensional Data Analysis 131
3.4.I Representing Iris Data as a Multidimensional Array 131
3.4.2 Multidimensional Data: The General Case . 133
3.4.3 Analyzing Multidimensional Data 135
3.4.4 Final Comments on Multidimensional Data Analysis 139
3 . 5 Bibliographic Notes 139
3 . 6 Exercises 747

Classification:
Basic Concepts, Decision Tlees, and Model Evaluation L45
4.1 Preliminaries 746
4.2 General Approach to Solving a Classification Problem 748
4.3 Decision Tlee Induction 150
4.3.1 How a Decision Tlee Works 150
4.3.2 How to Build a Decision TYee 151
4.3.3 Methods for Expressing Attribute Test Conditions 155
4.3.4 Measuresfor Selecting the Best Split . 158
4.3.5 Algorithm for Decision Tlee Induction 164
4.3.6 An Examole: Web Robot Detection 166
 Contents xv

4.3.7 Characteristics of Decision Tlee Induction 168

4.4 Model Overfitting 172
4.4.L Overfitting Due to Presenceof Noise L75
4.4.2 Overfitting Due to Lack of Representative Samples L77
4.4.3 Overfitting and the Multiple Comparison Procedure 178
4.4.4 Estimation of Generalization Errors 179
4.4.5 Handling Overfitting in Decision Tlee Induction 184
4.5 Evaluating the Performance of a Classifier 186
4.5.I Holdout Method 186
4.5.2 Random Subsampling . . . 187
4.5.3 Cross-Validation 187
4.5.4 Bootstrap 188
4.6 Methods for Comparing Classifiers 188
4.6.L Estimating a Confidence Interval for Accuracy 189
4.6.2 Comparing the Performance of Two Models . 191
4.6.3 Comparing the Performance of Two Classifiers 192
4.7 BibliographicNotes 193
4.8 Exercises 198

5 Classification: Alternative Techniques 207

5.1 Rule-Based Classifier 207
5.1.1 How a Rule-Based Classifier Works 209
5.1.2 Rule-Ordering Schemes 2II
5.1.3 How to Build a Rule-Based Classifier 2r2
5.1.4 Direct Methods for Rule Extraction 2r3
5.1.5 Indirect Methods for Rule Extraction 22L
5.1.6 Characteristics of Rule-Based Classifiers 223
5.2 Nearest-Neighbor classifiers 223
5.2.L Algorithm 225
5.2.2 Characteristics of Nearest-Neighbor Classifiers 226
5.3 Bayesian Classifiers 227
5.3.1 Bayes Theorem 228
5.3.2 Using the Bayes Theorem for Classification 229
5.3.3 Naive Bayes Classifier 23L
5.3.4 Bayes Error Rate 238
5.3.5 Bayesian Belief Networks 240
5.4 Artificial Neural Network (ANN) 246
5.4.I Perceptron 247
5.4.2 Multilayer Artificial Neural Network 25r
5.4.3 Characteristics of ANN 255
xvi Contents

5.5 Support Vector Machine (SVM) 256

5.5.1 Maximum Margin Hyperplanes 256
5.5.2 Linear SVM: SeparableCase 259
5.5.3 Linear SVM: Nonseparable Case 266
5.5.4 Nonlinear SVM . 270
5.5.5 Characteristics of SVM 276
5 . 6 Ensemble Methods 276
5.6.1 Rationale for Ensemble Method 277
5.6.2 Methods for Constructing an Ensemble Classifier 278
5.6.3 Bias-Variance Decomposition 28r
5.6.4 Bagging 283
5.6.5 Boosting 285
5.6.6 Random Forests 290
5.6.7 Empirical Comparison among Ensemble Methods 294
o.t Class Imbalance Problem 294
5.7.1 Alternative Metrics 295
5.7.2 The Receiver Operating Characteristic Curve 298
5.7.3 Cost-SensitiveLearning . . 302
5.7.4 Sampling-BasedApproaches . 305
c . 6 Multiclass Problem 306
5 . 9 Bibliographic Notes 309
5 . 1 0Exercises 315

Association Analysis: Basic Concepts and Algorithms 327

6.1 Problem Definition . 328
6.2 Flequent Itemset Generation 332
6.2.I The Apri,ori Principle 333
6.2.2 FbequentItemset Generation in the Apri,ori, Algorithm . 335
6.2.3 Candidate Generation and Pruning . . . 338
6.2.4 Support Counting 342
6.2.5 Computational Complexity 345
6.3 Rule Generatiorr 349
6.3.1 Confidence-BasedPruning 350
6.3.2 Rule Generation in Apri,ori, Algorithm 350
6.3.3 An Example: Congressional Voting Records 352
6.4 Compact Representation of FbequentItemsets 353
6.4.7 Maximal Flequent Itemsets 354
6.4.2 Closed Frequent Itemsets 355
6.5 Alternative Methods for Generating Frequent Itemsets 359
6.6 FP-Growth Alsorithm 363
 Contents xvii

6.6.1 FP-tee Representation 363

6.6.2 Frequent Itemset Generation in FP-Growth Algorithm . 366
6.7 Evaluation of Association Patterns 370
6.7.l Objective Measuresof Interestingness 37r
6.7.2 Measures beyond Pairs of Binary Variables 382
6.7.3 Simpson'sParadox 384
6.8 Effect of Skewed Support Distribution 386
6.9 Bibliographic Notes 390
6.10 Exercises 404

7 Association Analysis: Advanced Concepts 4L5

7.I Handling Categorical Attributes 415
7.2 Handling Continuous Attributes 4t8
7.2.I Discretization-Based Methods 418
7.2.2 Statistics-BasedMethods 422
7.2.3 Non-discretizalion Methods 424
7 . 3 Handling a Concept Hierarchy 426
7.4 Seouential Patterns 429
7.4.7 Problem Formulation 429
7.4.2 Sequential Pattern Discovery 431
7.4.3 Timing Constraints 436
7.4.4 Alternative Counting Schemes 439
7.5 Subgraph Patterns 442
7.5.1 Graphs and Subgraphs . 443
7.5.2 Frequent Subgraph Mining 444
7.5.3 Apri,od-like Method 447
7.5.4 Candidate Generation 448
7.5.5 Candidate Pruning 453
7.5.6 Support Counting 457
7.6 Infrequent Patterns 457
7.6.7 Negative Patterns 458
7.6.2 Negatively Correlated Patterns 458
7.6.3 Comparisons among Infrequent Patterns, Negative Pat-
terns, and Negatively Correlated Patterns 460
7.6.4 Techniquesfor Mining Interesting Infrequent Patterns 461
7.6.5 TechniquesBased on Mining Negative Patterns 463
7.6.6 TechniquesBased on Support Expectation . 465
7.7 Bibliographic Notes 469
7.8 Exercises 473
xviii Contents

Cluster Analysis: Basic Concepts and Algorithms 487

8.1 Overview 490
8.1.1 What Is Cluster Analysis? 490
8.I.2 Different Types of Clusterings . 49r
8.1.3 Different Types of Clusters 493
8.2 K-means 496
8.2.7 The Basic K-means Algorithm 497
8.2.2 K-means: Additional Issues 506
8.2.3 Bisecting K-means 508
8.2.4 K-means and Different Types of Clusters 510
8.2.5 Strengths and Weaknesses 510
8.2.6 K-means as an Optimization Problem 513
8.3 Agglomerative Hierarchical Clustering 515
8.3.1 Basic Agglomerative Hierarchical Clustering Algorithm 516
8.3.2 Specific Techniques 518
8.3.3 The Lance-Williams Formula for Cluster Proximity . 524
8.3.4 Key Issues in Hierarchical Clustering . 524
8.3.5 Strengths and Weaknesses 526
8 . 4 DBSCAN 526
8.4.1 Tladitional Density: Center-BasedApproach 527
8.4.2 The DBSCAN Algorithm 528
8.4.3 Strengths and Weaknesses 530
8 . 5 Cluster Evaluation 532
8.5.1 Overview 533
8.5.2 Unsupervised Cluster Evaluation Using Cohesion and
Separation 536
8.5.3 Unsupervised Cluster Evaluation Using the Proximity
Matrix 542
8.5.4 Unsupervised Evaluation of Hierarchical Clustering . 544
8.5.5 Determining the Correct Number of Clusters 546
8.5.6 Clustering Tendency 547
8.5.7 Supervised Measures of Cluster Validity 548
8.5.8 Assessingthe Significanceof Cluster Validity Measures . 553
8.6 Bibliograph ic Notes ooo
8.7 Exercises 559

Cluster Analysis: Additional Issues and Algorithms 569

9.1 Characteristics of Data, Clusters, and Clustering Algorithms . 570
9.1.1 Example: Comparing K-means and DBSCAN . . . . . . 570
9.1.2 Data Characteristics 577
 Contents xix

9.1.3 ClusterCharacteristics
. . 573
9.L.4 GeneralCharacteristicsof ClusteringAlgorithms 575
9.2 Prototype-BasedClustering 577
9.2.1 F\zzy Clustering 577
9.2.2 ClusteringUsing Mixture Models 583
9.2.3 Self-OrganizingMaps (SOM) 594
9.3 Density-BasedClustering 600
9.3.1 Grid-BasedClustering 601
9.3.2 SubspaceClustering 604
9.3.3 DENCLUE: A Kernel-BasedSchemefor Density-Based
Clustering 608
9.4 Graph-BasedClustering 612
9.4.1 Sparsification 613
9.4.2 Minimum SpanningTlee (MST) Clustering . . . 674
9.4.3 OPOSSUM:Optimal Partitioning of SparseSimilarities
UsingMETIS 616
9.4.4 Chameleon: Hierarchical Clustering with Dynamic
Modeling 616
9.4.5 Shared Nearest Neighbor Similarity 622
9.4.6 The Jarvis-Patrick Clustering Algorithm 625
9.4.7 SNN Density 627
9.4.8 SNN Density-Based Clustering 629
9.5 Scalable Clustering Algorithms 630
9.5.1 Scalability: General Issuesand Approaches 630
9.5.2 BIRCH 633
9.5.3 CURE 635
9.6 Which Clustering Algorithm? 639
9.7 Bibliographic Notes 643
9.8 Exercises 647

10 Anomaly Detection 651

10.1 Preliminaries 653
10.1.1 Causesof Anomalies 653
10.1.2 Approaches to Anomaly Detection 654
10.1.3 The Use of Class Labels 655
10.1.4 Issues 656
10.2 Statistical Approaches 658
t0.2.7 Detecting Outliers in a Univariate Normal Distribution 659
10.2.2 OutliersinaMultivariateNormalDistribution . . . . . 661
10.2.3 A Mixture Model Approach for Anomaly Detection. 662
xx Contents

10.2.4 Strengths and Weaknesses 665

10.3 Proximity-Based Outlier Detection 666
10.3.1 Strengths and Weaknesses 666
10.4 Density-Based Outlier Detection 668
10.4.1 Detection of Outliers Using Relative Density 669
70.4.2 Strengths and Weaknesses 670
10.5 Clustering-Based Techniques 67L
10.5.1 Assessingthe Extent to Which an Object Belongs to a
Cluster 672
10.5.2 Impact of Outliers on the Initial Clustering 674
10.5.3 The Number of Clusters to Use 674
10.5.4 Strengths and Weaknesses 674
10.6 Bibliographic Notes 675
10.7 Exercises 680

Appendix A Linear Algebra 685

A.1 Vectors b6i)
A.1.1 Definition 685
4.I.2 Vector Addition and Multiplication by a Scalar 685
A.1.3 Vector Spaces 687
4.7.4 The Dot Product, Orthogonality, and Orthogonal
Projections 688
A.1.5 Vectors and Data Analysis 690
42 Matrices 691
A.2.1 Matrices: Definitions 691
A-2.2 Matrices: Addition and Multiplication by a Scalar 692
4.2.3 Matrices: Multiplication 693
4.2.4 Linear tansformations and Inverse Matrices 695
4.2.5 Eigenvalue and Singular Value Decomposition . 697
4.2.6 Matrices and Data Analysis 699
A.3 Bibliographic Notes 700

Appendix B Dimensionality Reduction 7OL

8.1 PCA and SVD 70I
B.1.1 Principal ComponentsAnalysis (PCA) 70L
8.7.2 SVD . 706
8.2 Other Dimensionality Reduction Techniques 708
8.2.I Factor Analysis 708
8.2.2 Locally Linear Embedding (LLE) . 770
8.2.3 Multidimensional Scaling, FastMap, and ISOMAP 7I2
 Contents xxi

8.2.4 Common Issues 715

B.3 Bibliographic Notes 7L6

Appendix C Probability and Statistics 7L9

C.1 Probability 7L9
C.1.1 Expected Values 722
C.2 Statistics 723
C.2.L Point Estimation 724
C.2.2 Central Limit Theorem 724
C.2.3 Interval Estimation 725
C.3 Hypothesis Testing 726

Appendix D Regression 729

D.1 Preliminaries 729
D.2 Simple Linear Regression 730
D.2.L Least SquareMethod 731
D.2.2 Analyzing RegressionErrors 733
D.2.3 Analyzing Goodnessof Fit 735
D.3 Multivariate Linear Regression 736
D.4 Alternative Least-SquareRegressionMethods 737

Appendix E Optimization 739

E.1 Unconstrained Optimizafion 739
E.1.1 Numerical Methods 742
8.2 Constrained Optimization 746
E.2.I Equality Constraints 746
8.2.2 Inequality Constraints 747

Author Index 750

Subject Index 758

Copyright Permissions 769

 1

Introduction
Rapid advances in data collection and storage technology have enabled or-
ganizations to accumulate vast amounts of data. However, extracting useful
information has proven extremely challenging. Often, traditional data analy-
sis tools and techniques cannot be used because of the massive size of a data
set. Sometimes,the non-traditional nature of the data means that traditional
approachescannot be applied even if the data set is relatively small. In other
situations, the questions that need to be answeredcannot be addressedusing
existing data analysis techniques, and thus, new methods need to be devel-
oped.
Data mining is a technology that blends traditional data analysis methods
with sophisticated algorithms for processinglarge volumes of data. It has also
opened up exciting opportunities for exploring and analyzing new types of
data and for analyzing old types of data in new ways. In this introductory
chapter, we present an overview of data mining and outline the key topics
to be covered in this book. We start with a description of some well-known
applications that require new techniques for data analysis.

Business Point-of-sale data collection (bar code scanners, radio frequency

identification (RFID), and smart card technology) have allowed retailers to
collect up-to-the-minute data about customer purchasesat the checkout coun-
ters of their stores. Retailers can utilize this information, along with other
business-critical data such as Web logs from e-commerce Web sites and cus-
tomer service records from call centers, to help them better understand the
needs of their customers and make more informed businessdecisions.
Data mining techniques can be used to support a wide range of business
intelligence applications such as customer profiling, targeted marketing, work-
flow management, store layout, and fraud detection. It can also help retailers
2 Chapter 1 lntroduction

answer important business questions such as "Who are the most profitable
customers?" "What products can be cross-soldor up-sold?" and "What is the
revenue outlook of the company for next year?)) Some of these questions mo-
tivated the creation of association analvsis (Chapters 6 and 7), a new data
analysis technique.

Medicine, Science, and Engineering Researchersin medicine, science,

and engineering are rapidly accumulating data that is key to important new
discoveries. For example, as an important step toward improving our under-
standing of the Earth's climate system, NASA has deployed a seriesof Earth-
orbiting satellites that continuously generate global observations of the Iand
surface, oceans, and atmosphere. However, because of the size and spatio-
temporal nature of the data, traditional methods are often not suitable for
analyzing these data sets. Techniquesdevelopedin data mining can aid Earth
scientists in answering questions such as "What is the relationship between
the frequency and intensity of ecosystem disturbances such as droughts and
hurricanes to global warming?" "How is land surface precipitation and temper-
ature affected by ocean surface temperature?" and "How well can we predict
the beginning and end of the growing seasonfor a region?"
As another example, researchersin molecular biology hope to use the large
amounts of genomic data currently being gathered to better understand the
structure and function of genes. In the past, traditional methods in molecu-
lar biology allowed scientists to study only a few genes at a time in a given
experiment. Recent breakthroughs in microarray technology have enabled sci-
entists to compare the behavior of thousands of genesunder various situations.
Such comparisons can help determine the function of each gene and perhaps
isolate the genesresponsiblefor certain diseases.However, the noisy and high-
dimensional nature of data requires new types of data analysis. In addition
to analyzing gene array data, data mining can also be used to address other
important biological challengessuch as protein structure prediction, multiple
sequencealignment, the modeling of biochemical pathways, and phylogenetics.

1.1 What Is Data Mining?

Data mining is the processof automatically discovering useful information in
large data repositories. Data mining techniques are deployed to scour large
databases in order to find novel and useful patterns that might otherwise
remain unknown. They also provide capabilities to predict the outcome of a
 1.1 What Is Data Mining? 3

future observation, such as predicting whether a newly arrived.customer will

spend more than $100 at a department store.
Not all information discovery tasks are consideredto be data mining. For
example, Iooking up individual records using a database managemenr sysrem
or finding particular Web pages via a query to an Internet search engine are
tasks related to the area of information retrieval. Although such tasks are
important and may involve the use of the sophisticated algorithms and data
structures, they rely on traditional computer sciencetechniques and obvious
features of the data to create index structures for efficiently organizing and
retrieving information. Nonetheless,data mining techniques have been used
to enhance information retrieval systems.

Data Mining and Knowledge Discovery

Data mining is an integral part of knowledge discovery in databases
(KDD), which is the overall process of converting raw data into useful in-
formation, as shown in Figure 1.1. This process consists of a series of trans-
formation steps, from data preprocessing to postprocessing of data mining
results.

Information

Figure1,1. Theprocess
of knowledge
discovery
indatabases
(KDD).

The input data can be stored in a variety of formats (flat files, spread-
sheets, or relational tables) and may reside in a centralized data repository
or be distributed across multiple sites. The purpose of preprocessing is
to transform the raw input data into an appropriate format for subsequent
analysis. The steps involved in data preprocessinginclude fusing data from
multiple sources, cleaning data to remove noise and duplicate observations,
and selecting records and features that are relevant to the data mining task
at hand. Because of the many ways data can be collected and stored, data
4 Chapter 1 Introduction

preprocessingis perhaps the most laborious and time-consuming step in the

overall knowledge discovery process.
,,Closing the loop" is the phrase often used to refer to the process of in-
tegrating data mining results into decision support systems. For example,
in business applications, the insights offered by data mining results can be
integrated with campaign management tools so that effective marketing pro-
motions can be conducted and tested. Such integration requires a postpro-
cessing step that ensuresthat only valid and useful results are incorporated
into the decision support system. An example of postprocessingis visualiza-
tion (see Chapter 3), which allows analysts to explore the data and the data
mining results from a variety of viewpoints. Statistical measuresor hypoth-
esis testing methods can also be applied during postprocessingto eliminate
spurious data mining results.

L.2 Motivating Challenges

As mentioned earlier, traditional data analysis techniques have often encoun-
tered practical difficulties in meeting the challengesposed by new data sets.
The following are some of the specific challengesthat motivated the develop-
ment of data mining.

Scalability Becauseof advancesin data generation and collection, data sets

with sizes of gigabytes, terabytes, or even petabytes are becoming common.
If data mining algorithms are to handle these massive data sets, then they
must be scalable. Many data mining algorithms employ special search strate-
gies to handle exponential search problems. Scalability may also require the
implementation of novel data structures to accessindividual records in an ef-
ficient manner. For instance, out-of-core algorithms may be necessary when
processingdata sets that cannot fit into main memory. Scalability can also be
improved by using sampling or developing parallel and distributed algorithms.

High Dimensionality It is now common to encounter data sets with hun-

dreds or thousands of attributes instead of the handful common a few decades
ago. In bioinformatics, progress in microarray technology has produced gene
expression data involving thousands of features. Data sets with temporal
or spatial components also tend to have high dimensionality. For example,
consider a data set that contains measurements of temperature at various
locations. If the temperature measurementsare taken repeatedly for an ex-
tended period, the number of dimensions (features) increasesin proportion to
 L.2 Motivating Challenges 5

the number of measurementstaken. Tladitional data analysis techniques that

were developed for low-dimensional data often do not work well for such high-
dimensional data. Also, for some data analysis algorithms, the computational
complexity increasesrapidly as the dimensionality (the number of features)
increases.

Heterogeneous and Complex Data TYaditional data analysis methods

often deal with data sets containing attributes of the same type, either contin-
uous or categorical. As the role of data mining in business,science,medicine,
and other flelds has grown, so has the need for techniques that can handle
heterogeneousattributes. Recent years have also seen the emergenceof more
complex data objects. Examples of such non-traditional types of data include
collections of Web pagescontaining semi-structured text and hyperlinks; DNA
data with sequential and three-dimensional structure; and climate data that
consists of time series measurements (temperature, pressure, etc.) at various
locations on the Earth's surface. Techniquesdeveloped for mining such com-
plex objects should take into consideration relationships in the data, such as
temporal and spatial autocorrelation, graph connectivity, and parent-child re-
lationships between the elementsin semi-structured text and XML documents.

Data ownership and Distribution Sometimes, the data needed for an

analysis is not stored in one location or owned by one organization. Instead,
the data is geographically distributed among resourcesbelonging to multiple
entities. This requires the development of distributed data mining techniques.
Among the key challenges faced by distributed data mining algorithms in-
clude (1) how to reduce the amount of communication neededto perform the
distributed computatior, (2) how to effectively consolidate the data mining
results obtained from multiple sources,and (3) how to address data security
issues.

Non-traditional Analysis The traditional statistical approach is based on

a hypothesize-and-test paradigm. In other words, a hypothesis is proposed,
an experiment is designed to gather the data, and then the data is analyzed
with respect to the hypothesis. Unfortunately, this processis extremely labor-
intensive. Current data analysis tasks often require the generation and evalu-
ation of thousands of hypotheses,and consequently,the development of some
data mining techniques has been motivated by the desire to automate the
process of hypothesis generation and evaluation. Furthermore, the data sets
analyzed in data mining are typically not the result of a carefully designed
6 Chapter 1 Introduction

experiment and often represent opportunistic samplesof the data, rather than
random samples. Also, the data sets frequently involve non-traditional types
of data and data distributions.

1.3 The Origins of Data Mining

Brought together by the goal of meeting the challenges of the previous sec-
tion, researchersfrom different disciplines began to focus on developing more
efficient and scalabletools that could handle diverse types of data. This work,
which culminated in the field of data mining, built upon the methodology and
algorithms that researchershad previously used. In particular, data mining
draws upon ideas, such as (1) sampling, estimation, and hypothesis testing
from statistics and (2) search algorithms, modeling techniques, and learning
theories from artificial intelligence, pattern recognition, and machine learning.
Data mining has also been quick to adopt ideas from other areas, including
optimization, evolutionary computing, information theory, signal processing,
visualization, and information retrieval.
A number of other areas also play key supporting roles. In particular,
database systems are needed to provide support for efficient storage, index-
ing, and query processing. Techniquesfrom high performance (parallel) com-
puting are often important in addressing the massive size of some data sets.
Distributed techniques can also help addressthe issue of size and are essential
when the data cannot be gathered in one location.
Figure 1.2 shows the relationship of data mining to other areas.

1.2.Datamining
Figure ofmany
asa conlluence disciplines.
 L.4 Data Mining Tasks 7

1.4 Data Mining Tasks

Data mining tasks are generally divided into two major categories:

Predictive tasks. The objective of thesetasks is to predict the value of a par-

ticular attribute based on the values of other attributes. The attribute
to be predicted is commonly known as the target or dependent vari-
able, while the attributes used for making the prediction are known as
the explanatory or independent variables.

Descriptive tasks. Here, the objective is to derive patterns (correlations,

trends, clusters, trajectories, and anomalies) that summarize the un-
derlying relationships in data. Descriptive data mining tasks are often
exploratory in nature and frequently require postprocessingtechniques
to validate and explain the results.

Figure 1.3 illustrates four of the core data mining tasks that are described
in the remainder of this book.

Figure
1.3.Fourofthecoredatamining
tasks.
8 Chapter 1 Introduction

Predictive modeling refers to the task of building a model for the target
variable as a function of the explanatory variables. There are two types of
predictive modeling tasks: classification, which is used for discrete target
variables, and regression, which is used for continuous target variables. For
example, predicting whether a Web user will make a purchase at an online
bookstore is a classification task becausethe target variable is binary-valued.
On the other hand, forecasting the future price of a stock is a regression task
becauseprice is a continuous-valued attribute. The goal of both tasks is to
learn a model that minimizes the error between the predicted and true values
of the target variable. Predictive modeling can be used to identify customers
that will respond to a marketing campaign, predict disturbances in the Earth's
ecosystem,or judge whether a patient has a particular diseasebased on the
results of medical tests.
Example 1.1 (Predicting the Type of a Flower). Consider the task of
predicting a species of flower based on the characteristics of the flower. In
particular, consider classifying an Iris flower as to whether it belongs to one
of the following three Iris species: Setosa, Versicolour, or Virginica. To per-
form this task, we need a data set containing the characteristics of various
flowers of these three species. A data set with this type of information is
the well-known Iris data set from the UCI Machine Learning Repository at
http: /hrurw.ics.uci.edu/-mlearn. In addition to the speciesof a flower,
this data set contains four other attributes: sepal width, sepal length, petal
length, and petal width. (The Iris data set and its attributes are described
further in Section 3.1.) Figure 1.4 shows a plot of petal width versus petal
length for the 150 flowers in the Iris data set. Petal width is broken into the
categories low, med'ium, and hi'gh,which correspond to the intervals [0' 0.75),
[0.75,1.75), [1.75,oo), respectively.Also, petal length is broken into categories
low, med,'ium,and hi,gh,which correspondto the intervals [0' 2.5), [2.5,5), [5'
oo), respectively. Based on these categories of petal width and length, the
following rules can be derived:

Petal width low and petal length low implies Setosa.

Petal width medium and petal length medium implies Versicolour.
Petal width high and petal length high implies Virginica.

While these rules do not classify all the flowers, they do a good (but not
perfect) job of classifying most of the flowers. Note that flowers from the
Setosa speciesare well separated from the Versicolour and Virginica species
with respect to petal width and length, but the latter two species overlap
somewhat with respect to these attributes. I
 L.4 Data Mining Tasks I

l----a--fo-------i
r Setosa la o r
. Versicolour ,ftfo oto ai
: o o o I
o Virginica I
' t0f0a o
0?oo r a
arf I
?1.75 !0_l! _.! o__o
E
() .t
aa rO
r 1.5 . .4. a?o
E o aaaa
= a aaaaaaa
(t' aaaa a
() aa
(Ll aaa aa

I
I
lltll
llt I
I lllltt I
It!

122.5345(
PetalLength(cm)

Figure1.4. Petalwidthversus
petallength
for150lrisflowers,

Association analysis is used to discover patterns that describe strongly as-

sociated features in the data. The discoveredpatterns are typically represented
in the form of implication rules or feature subsets. Becauseof the exponential
size of its search space, the goal of association analysis is to extract the most
interesting patterns in an efficient manner. Useful applications of association
analysis include finding groups of genes that have related functionality, identi-
fying Web pages that are accessedtogether, or understanding the relationships
between different elements of Earth's climate system.
Example 1.2 (Market Basket Analysis). The transactions shown in Ta-
ble 1.1 illustrate point-of-sale data collected at the checkout counters of a
grocery store. Association analysis can be applied to find items that are fre-
quently bought together by customers. For example, we may discover the
rule {Diapers} -----*{lt:.ft}, which suggeststhat customers who buy diapers
also tend to buy milk. This type of rule can be used to identify potential
cross-sellingopportunities among related items. I

Cluster analysis seeksto find groups of closely related observationsso that

observations that belong to the same cluster are more similar to each other
10 Chapter 1 Introduction

Table1.1. Market data.

basket
Tlansaction ID Items
1 {Bread, Butter, Diapers, Milk}
2 {Coffee, Sugar, Cookies, Sakoon}
3 {Bread, Butter, Coffee, Diapers, Milk, Eggs}
4 {Bread, Butter, Salmon,Chicken}
r
{fgg", Bread, Butter}
o {Salmon, Diapers, Milk}
7 {Bread, Tea, Sugar, Eggs}
8 {Coffee, Sugar, Chicken, Eggs}
9 {Bread, Diapers, Mi1k, Salt}
10 {Tea, Eggs, Cookies, Diapers, Milk}

than observations that belong to other clusters. Clustering has been used to
group sets of related customers, find areas of the ocean that have a significant
impact on the Earth's climate, and compressdata.
Example 1.3 (Document Clustering). The collection of news articles
shown in Table 1.2 can be grouped based on their respective topics. Each
article is representedas a set of word-frequency pairs (r, where tu is a word
"),
and c is the number of times the word appears in the article. There are two
natural clusters in the data set. The first cluster consists of the first four ar-
ticles, which correspond to news about the economy,while the second cluster
contains the last four articles, which correspond to news about health care. A
good clustering algorithm should be able to identify these two clusters based
on the similarity between words that appear in the articles.

of newsarticles.
Table1.2.Collection
Article Words
I dollar: 1, industry: 4, country: 2, loan: 3, deal: 2, government: 2
2 machinery: 2, labor: 3, market: 4, industry: 2, work: 3, country: 1
.) job: 5, inflation: 3, rise: 2, jobless: 2, market: 3, country: 2, index: 3
A domestic: 3, forecast: 2, gain: 1, market: 2, sale: 3, price: 2
r
J patient: 4, symptom: 2, drug: 3, health: 2, clinic: 2, doctor: 2
o pharmaceutical:2, company: 3, drug: 2,vaccine:1, flu: 3
7 death: 2, cancer: 4, drug: 3, public: 4, health: 3, director: 2
8 medical: 2, cost: 3, increase: 2, patient: 2, health: 3, care: 1
 1.5 Scope and Organization of the Book 11

Anomaly detection is the task of identifying observationswhose character-

istics are significantly different from the rest of the data. Such observations
are known as anomalies or outliers. The goal of an anomaly detection al-
gorithm is to discover the real anomalies and avoid falsely labeling normal
objects as anomalous. In other words, a good anomaly detector must have
a high detection rate and a low false alarm rate. Applications of anomaly
detection include the detection of fraud, network intrusions, unusual patterns
of disease,and ecosystemdisturbances.
Example 1.4 (Credit Card trYaud Detection). A credit card company
records the transactions made by every credit card holder, along with personal
information such as credit limit, age, annual income, and address. since the
number of fraudulent cases is relatively small compared to the number of
legitimate transactions, anomaly detection techniques can be applied to build
a profile of legitimate transactions for the users. When a new transaction
arrives, it is compared against the profile of the user. If the characteristics of
the transaction are very different from the previously created profile, then the
transaction is flagged as potentially fraudulent. I

1.5 Scope and Organization of the Book

This book introduces the major principles and techniques used in data mining
from an algorithmic perspective. A study of these principles and techniques is
essentialfor developing a better understanding of how data mining technology
can be applied to various kinds of data. This book also serves as a starting
point for readers who are interested in doing researchin this field.
We begin the technical discussion of this book with a chapter on data
(Chapter 2), which discussesthe basic types of data, data quality, prepro-
cessing techniques, and measures of similarity and dissimilarity. Although
this material can be covered quickly, it provides an essential foundation for
data analysis. Chapter 3, on data exploration, discussessummary statistics,
visualization techniques, and On-Line Analytical Processing (OLAP). These
techniques provide the means for quickly gaining insight into a data set.
Chapters 4 and 5 cover classification. Chapter 4 provides a foundation
by discussing decision tree classifiers and several issues that are important
to all classification: overfitting, performance evaluation, and the comparison
of different classification models. Using this foundation, Chapter 5 describes
a number of other important classification techniques: rule-based systems,
nearest-neighborclassifiers,Bayesianclassifiers,artificial neural networks, sup-
port vector machines, and ensembleclassifiers,which are collections of classi-
!2 Chapter 1 lntroduction

fiers. The multiclass and imbalanced class problems are also discussed.These
topics can be covered independently.
Association analysis is explored in Chapters 6 and 7. Chapter 6 describes
the basics of association analysis: frequent itemsets, association rules, and
some of the algorithms used to generate them. Specific types of frequent
itemsets-maximal, closed,and hyperclique-that are important for data min-
ing are also discussed,and the chapter concludeswith a discussionof evalua-
tion measuresfor association analysis. Chapter 7 considers a variety of more
advancedtopics, including how association analysis can be applied to categor-
ical and continuous data or to data that has a concept hierarchy. (A concept
hierarchy is a hierarchical categorization of objects, e.g., store items, clothing,
shoes,sneakers.) This chapter also describeshow association analysis can be
extended to find sequential patterns (patterns involving order), patterns in
graphs, and negative relationships (if one item is present, then the other is
not).
Cluster analysis is discussedin Chapters 8 and 9. Chapter 8 first describes
the different types of clusters and then presentsthree specific clustering tech-
niques: K-means, agglomerative hierarchical clustering, and DBSCAN. This
is followed by a discussion of techniques for validating the results of a cluster-
ing algorithm. Additional clustering concepts and techniques are explored in
Chapter 9, including fiszzy and probabilistic clustering, Self-Organizing Maps
(SOM), graph-based clustering, and density-basedclustering. There is also a
discussion of scalability issues and factors to consider when selecting a clus-
tering algorithm.
The last chapter, Chapter 10, is on anomaly detection. After some basic
definitions, several different types of anomaly detection are considered: sta-
tistical, distance-based,density-based, and clustering-based. Appendices A
through E give a brief review of important topics that are used in portions of
the book: linear algebra, dimensionality reduction, statistics, regression,and
optimization.
The subject of data mining, while relatively young compared to statistics
or machine learning, is already too large to cover in a single book. Selected
references to topics that are only briefly covered, such as data quality' are
provided in the bibliographic notes of the appropriate chapter. Referencesto
topics not covered in this book, such as data mining for streams and privacy-
preserving data mining, are provided in the bibliographic notes of this chapter.
 1_.6 Bibliographic Notes 13

1.6 Bibliographic Notes

The topic of data mining has inspired many textbooks. Introductory text-
books include those by Dunham [10], Han and Kamber l2L), Hand et al. [23],
and Roiger and Geatz [36]. Data mining books with a stronger emphasis on
businessapplications include the works by Berry and Linoff [2], Pyle [34], and
Parr Rud [33]. Books with an emphasis on statistical learning include those
by Cherkassky and Mulier [6], and Hastie et al. 124]. Some books with an
emphasis on machine learning or pattern recognition are those by Duda et
al. [9], Kantardzic [25], Mitchell [31], Webb [41], and Witten and F]ank [42].
There are also some more specialized books: Chakrabarti [a] (web mining),
Fayyad et al. [13] (collection of early articles on data mining), Fayyad et al.
111](visualization), Grossman et al. [18] (scienceand engineering), Kargupta
and Chan [26] (distributed data mining), Wang et al. [a0] (bioinformatics),
and Zaki and Ho [44] (parallel data mining).
There are several conferencesrelated to data mining. Some of the main
conferencesdedicated to this field include the ACM SIGKDD International
Conferenceon Knowledge Discovery and Data Mining (KDD), the IEEE In-
ternational Conferenceon Data Mining (ICDM), the SIAM International Con-
ference on Data Mining (SDM), the European Conferenceon Principles and
Practice of Knowledge Discovery in Databases (PKDD), and the Pacific-Asia
Conferenceon Knowledge Discovery and Data Mining (PAKDD). Data min-
ing papers can also be found in other major conferencessuch as the ACM
SIGMOD/PODS conference,the International Conferenceon Very Large Data
Bases (VLDB), the Conferenceon Information and Knowledge Management
(CIKM), the International Conferenceon Data Engineering (ICDE), the In-
ternational Conferenceon Machine Learning (ICML), and the National Con-
ference on Artificial Intelligence (AAAI).
Journal publications on data mining include IEEE Transact'ionson Knowl-
edge and Data Engi,neering,Data Mi,ning and Knowledge Discouery, Knowl-
edge and Information Systems, Intelli,gent Data Analysi,s, Inforrnati,on Sys-
tems, and lhe Journal of Intelligent Informati,on Systems.
There have been a number of generalarticles on data mining that define the
field or its relationship to other fields, particularly statistics. Fayyad et al. [12]
describedata mining and how it fits into the total knowledgediscoveryprocess.
Chen et al. [5] give a database perspective on data mining. Ramakrishnan
and Grama [35] provide a general discussion of data mining and present several
viewpoints. Hand [22] describeshow data mining differs from statistics, as does
Fliedman lf 4]. Lambert [29] exploresthe use of statistics for large data sets and
provides some comments on the respective roles of data mining and statistics.
L4 Chapter 1 Introduction

Glymour et al. 116]consider the lessons that statistics may have for data
mining. Smyth et aL [38] describe how the evolution of data mining is being
driven by new types of data and applications, such as those involving streams,
graphs, and text. Emerging applications in data mining are consideredby Han
et al. [20] and Smyth [37] describessome researchchallengesin data mining.
A discussionof how developmentsin data mining researchcan be turned into
practical tools is given by Wu et al. [43]. Data mining standards are the
subject of a paper by Grossman et al. [17]. Bradley [3] discusseshow data
mining algorithms can be scaled to large data sets.
With the emergence of new data mining applications have come new chal-
lengesthat need to be addressed.For instance, concernsabout privacy breaches
as a result of data mining have escalated in recent years, particularly in ap-
plication domains such as Web commerce and health care. As a result, there
is growing interest in developing data mining algorithms that maintain user
privacy. Developing techniques for mining encrypted or randomized data is
known as privacy-preserving data mining. Some general referencesin this
area include papers by Agrawal and Srikant l1], Clifton et al. [7] and Kargupta
et al. [27]. Vassilios et al. [39] provide a survey.
Recent years have witnessed a growing number of applications that rapidly
generatecontinuous streams of data. Examples of stream data include network
traffic, multimedia streams, and stock prices. Severalissuesmust be considered
when mining data streams, such as the limited amount of memory available,
the need for online analysis, and the change of the data over time. Data
mining for stream data has become an important area in data mining. Some
selected publications are Domingos and Hulten [8] (classification), Giannella
et al. [15] (associationanalysis), Guha et al. [19] (clustering), Kifer et al. [28]
(changedetection), Papadimitriou et al. [32] (time series),and Law et al. [30]
(dimensionality reduction).

Bibliography
[1] R. Agrawal and R. Srikant. Privacy-preserving data mining. ln Proc. of 2000 ACM-
SIGMOD IntI. Conf. on Management of Data, pages 439-450, Dallas, Texas, 2000.
ACM Press.
12lM. J. A. Berry and G. Linofi. Data Mtni,ng Technr,ques: For Marketing, Sales, and'
Customer Relati,onship Management. Wiley Computer Publishing, 2nd edition, 2004.
[')] S. Bradley, J. Gehrke, R. Ramakrishnan, and R. Srikant. Scaling mining algorithms
lol
P.
to large databases. Communicati,ons of the ACM, 45(8):38 43,2002.
[4] S. Chakrabarti. Mini.ng the Web: Di.scoueri.ngKnouledge from Hypertert Data' Morgan
Kaufmann, San Flancisco, CA, 2003.
 Bibliography 15

[5] M.-s. chen, J. Han, and P. s. Yu. Data Mining: An overview from a Database
Perspective. IEEE Transact'ions on Knowled,ge abd Data Engineering, g(6):g66-gg3,
1996.
[6] v. cherkassky and F. Mulier. Learn'ing from Data: concepts, Theory, and, Method,s.
Wiley Interscience, 1g98.
[7] c. clifton, M. Kantarcioglu, and J. vaidya. Defining privacy for data mining. In
National Sc'ience Foundat'ion workshop on Nert Generation Data Mining, pages 126-
133, Baltimore, MD, November 2002.
f8] P' Domingos and G. Hulten. Mining high-speed data streams. In Proc. of the 6th Intt.
conf. on Knowled,ge Discouery and Data M'in'ing, pages z1-80, Boston, Massachusetts,
2000. ACM Press.
J9] R. o' Duda, P. E. Hart, and D. G. stork. Pattern classification John wiley & sons,
Inc., New York, 2nd edition, 2001.
[10] M. H. Dunham. Data Mini,ng: Introd,uctory and, Ad,uancedropics. prentice Hall, 2002.
f11] U. M. Fayyad, G. G. Grinstein, and A. Wierse, editors. Information Visualization in
Data Mining and, Knowled,ge Discouery. Morgan Kaufmann Publishers, San Ftancisco,
CA, September 200I.
112] u. M. Fayyad, G. Piatetsky-Shapiro, and P. smyth. Fyom Data Mining to Knowledge
Discovery: An overview. rn Ad,aances in Knowledge Discouery and Data M'ining, pages
1-34. AAAI Press, 1996.
[13] u. M. Fayyad, G. Piatetsky-shapiro, P. Smyth, and R. uthurusamy, editors. Aduances
'in Knowled,ge Discouery and Data Mini.ng. press, 1g96.
AAAI/MIT
[14] J. H. Friedman. Data Mining and Statistics: What's the Connection? Unpublished.
www-stat.stanford.edu/-jhf/ftp/dm-stat.ps, 1992.
[15] c. Giannella, J. Han, J. Pei, X. Yan, and P. s. Yu. Mining Fyequent patterns in Data
streams at Multiple Time Granularities. In H. Kargupta, A. Joshi, K. sivakumar, and
Y. Yesha, editors, Nert Generation Data M,ining, pages ISI-2I2. AAAI/MIT,2003.
116] c. Glymour, D. Madigan, D. Pregibon, and P. smyth. statistical rhemes and Lessons
for Data Mining. Data Mining and Knowleilge D,iscouerg,1(1):11-28, 1992.
[17] R. L. Grossman, M. F. Hornick, and G. Meyer. Data mining standards initiatives.
c omtnunications of the A c M, 45(g) :59-6I, 2002.
[18] R. L. Grossman, c. Kamath, P. Kegelmeyer, v. Kumar, and R. Namburu, editors. Data
Mini;ng for Sci,entific and Engineering Applicati,ons. Kluwer Academic Publishers, 2001.
119] s. Guha, A. Meyerson, N. Mishra, R. Motwani, and L. o'callaghan. clustering Data
Streams: Theory and Practice. IEEE Tbansact'ions on Knowledge and, Data Engineering,
15(3) :515-528, May/June 2003.
[20] J. Han, R. B. Altman, V. Kumar, H. Mannila, and D. pregibon. Emerging scientific
applications in data mining. Communications of the ACM, 4S(8):54-b8,2002.
[21] J. Han and M. Kamber. Data Mining: concepts and, Techniques. Morgan Kaufmann
Publishers, San Francisco, 2001.
[22] D. J. Hand. Data Mining: Statistics and More? The American Statistician, 52(2):
112-118,1998.
[23] D. J. Hand, H. Mannila, and P. smyth. Principles of Data Mining. MIT press, 2001.
l24l T. Hastie, R. Tibshirani, and J. H. Fliedman. The Elements of Stati.stical Learning:
Data Mini,ng, Inference, Pred,iction. Springer, New York, 2001.
[25] M. Kantardzic. Data Mini,ng: concepts, Mod,el.s,Method,s, and Algorithms. wiley-IEEE
Press, Piscataway NJ, 2003.
16 Chapter I Introduction

and Parallel Knowledge

[26] H. Kargupta and P. K. Chan, editors. Aduances in Di,stributed
Discouery. AAAI Press, September 2002.
Prop-
l27l H. Kargupta, s. Datta, Q. wang, and K. sivakumar. on the Privacy Preserving
erties of Random Data Perturbation Techniques. In Proc. of the 2003 IEEE IntI. Conf.
on Data Mi,n'ing, pages 99 106, Melbourne, Florida, December 2003. IEEE Computer
Society.
f28] D. Kifer, s. Ben-David, and J. Gehrke. Detecting change in Data streams. In Proc. of
the 30th VLDB Conf., pages 180 191, Toronto, Canada,2004. Morgan Kaufmann.
f29] D. Lambert. What Use is Statistics for Massive Data? In ACM SIGMOD Workshop
on Research Issues in Data Mini'ng and Knowledge Di'scouery,pages 54-62, 2000.
for Data
[30] M H. C. Law, N. Zhang, and A. K. Jain. Nonlinear Manifold Learning
Streams. In Proc. of the SIAM Intl. Conf. on Data Mi.ning, Lake Buena Vista, Florida,
April2004. SIAM.
[31] T. Mitchell. Mach'ine Learning. McGraw-Hill, Boston, MA, 1997.
stream min-
[32] S. Papadimitriou, A. Brockwell, and C. Faloutsos. Adaptive, unsupervised
ing. VLDB Journal, 13(3):222 239.2004.
f33l O. Parr Rud. Data Mi,ning Cookbook: Modeling Data for Marleet'ing, Risk and, Customer
Relationship Management John Wiley & Sons, New York, NY, 2001.
[34] D. Pyle. Business Modeling and, Data Mining. Morgan Kaufmann, san FYancisco, cA,
2003.
135] N. Ramakrishnan and A. Grama. Data Mining: From Serendipity to Science-Guest
Editors' Introduction. IEEE Computer, S2(8):34 37, 1999.
[36] R. Roiger and M. Geatz. Data Mzni,ng: A Tutorial Based, Primer. Addison-Wesley,
2002.
In
137] P. Smyth. Breaking out of the Black-Box: Research Challenges in Data Mining.
Proc. of the 2001 ACM SIGMOD Workshop on Research Issues i.n Data Mining and
Knowledg e Discouerg, 2OOL.
[38] P. Smyth, D. Pregibon, and C. Faloutsos. Data-driven evolution of data mining algo-
rithms. Commun'ications of the ACM, 45(8):33-37, 2002.
[39] V. S. Verykios, E. Bertino, I. N. Fovino, L. P. Provenza, Y. Saygin, and Y. Theodoridis.
State-of-the-art in privacy preserving data mining. SIGMOD Record,,33(1):50-57' 2004.
in
[40] J. T. L. Wang, M. J. Zaki, H. Toivonen, and D. tr. Shasha, editors. Data Mining
Bi,oi,nformatics. Springer, September 2004.
2002.
[41] A. R. Webb. Statistical Pattern Recogn'iti'on.John Wiley & Sons, 2nd edition,
and E. Frank. Data Mini,ng: Practzcal Machine Learn'ing Tools and Tech-
[42] I.H. Witten
niques with Jaaa Implernentat'ions. Morgan Kaufmann, 1999.
[43] X. Wu, P. S. Yu, and G. Piatetsky-Shapiro. Data Mining: How Research Meets Practical
Development ? Knowledg e and Inf ormati'on Systems, 5 (2) :248-261, 2003.
l44l M. J. Zaki and C.-T. Ho, editors. Large-ScaleParallel Data Mining. Springer, September
2002.

L.7 Exercises
1. Discuss whether or not each of the following activities is a data mining task.
 L.7 Exercises L7

(a) Dividing the customers of a company according to their gender.

(b) Dividing the customers of a company according to their profitability.
(c) Computing the total sales of a company.
(d) Sorting a student database based on student identification numbers.
(e) Predicting the outcomes of tossing a (fair) pair of dice.
(f) Predicting the future stock price of a company using historical records.
(g) Monitoring the heart rate of a patient for abnormalities.
(h) Monitoring seismic waves for earthquake activities.
(i) Extracting the frequencies of a sound wave.

2 . Suppose that you are employed as a data mining consultant for an Internet
search engine company. Describe how data mining can help the company by
giving specific examples of how techniques, such as clustering, classification,
association rule mining, and anomaly detection can be applied.

J. For each of the following data sets, explain whether or not data privacy is an
important issue.

(a) Census data collected from 1900-1950.

(b) IP addresses and visit times of Web users who visit your Website.
(c) Images from Earth-orbiting satellites.
(d) Names and addressesof people from the telephone book.
(e) Names and email addressescollected from the Web.
Data
This chapter discussesseveral data-related issues that are important for suc-
cessful data mining:

The Type of Data Data sets differ in a number of ways. For example, the
attributes used to describedata objects can be of different types-quantitative
or qualitative-and data sets may have special characteristics;e.g., some data
sets contain time series or objects with explicit relationships to one another.
Not surprisingly, the type of data determines which tools and techniques can
be used to analyze the data. F\rrthermore, new research in data mining is
often driven by the need to accommodate new application areas and their new
types of data.

The Quality of the Data Data is often far from perfect. while most data
mining techniques can tolerate some level of imperfection in the data, a focus
on understanding and improving data quality typically improves the quality
of the resulting analysis. Data quality issuesthat often need to be addressed
include the presenceof noise and outliers; missing, inconsistent, or duplicate
data; and data that is biased or, in some other way, unrepresentative of the
phenomenon or population that the data is supposed to describe.

Preprocessing Steps to Make the Data More suitable for Data Min-
ing often, the raw data must be processedin order to make it suitable for
analysis. While one objective may be to improve data quality, other goals
focus on modifying the data so that it better fits a specifieddata mining tech-
nique or tool. For example, a continuous attribute, e.g., length, m&y need to
be transformed into an attribute with discrete categories,e.g., short, med,ium,
or long, in order to apply a particular technique. As another example, the
20 Chapter 2 Data

number of attributes in a data set is often reduced becausemany techniques

are more effective when the data has a relatively small number of attributes.

Analyzing Data in Terms of Its Relationships One approach to data

analysis is to find relationships among the data objects and then perform
the remaining analysis using these relationships rather than the data objects
themselves. For instance, we can compute the similarity or distance between
pairs of objects and then perform the analysis-clustering, classification, or
anomaly detection-based on these similarities or distances. There are many
such similarity or distance measures)and the proper choice depends on the
type of data and the particular application.
Example 2.1 (An Illustration of Data-Related Issues). To further il-
Iustrate the importance of these issues,consider the following hypothetical sit-
uation. You receive an email from a medical researcher concerning a project
that you are eager to work on.

Hi,
I've attached the data file that I mentioned in my previous email.
Each line contains the information for a single patient and consists
of five fields. We want to predict the last field using the other fields.
I don't have time to provide any more information about the data
since I'm going out of town for a couple of days, but hopefully that
won't slow you down too much. And if you don't mind, could we
meet when I get back to discussyour preliminary results? I might
invite a few other members of mv team.

Thanks and see you in a couple of days.

Despite some misgivings, you proceed to analyze the data. The first few
rows of the fiIe are as follows:

0r2 232 33.5 0 10.7

020 72r 16.9 2 2L0.L
027 165 24.0 0 427.6

A brieflook at the data revealsnothing strange. You put your doubts aside
and start the analysis. There are only 1000 lines, a smaller data file than you
had hoped for, but two days later, you feel that you have made some progress.
You arrive for the meeting, and while waiting for others to arrive, you strike
 2L

up a conversationwith a statistician who is working on the project. When she

learns that you have also been analyzing the data from the project, she asks
if you would mind giving her a brief overview of your results.

Statistician: So, you got the data for all the patients?
Data Miner: Yes. I haven't had much time for analysis, but I
do have a few interesting results.
Statistician: Amazing. There were so many data issueswith
this set of patients that I couldn't do much.
Data Miner: Oh? I didn't hear about any possible problems.
Statistician: Well, first there is field 5, the variable we want to
predict. It's common knowledge among people who analyze
this type of data that results are better if you work with the
log of the values, but I didn't discover this until later. Was it
mentioned to you?
Data Miner: No.
Statistician: But surely you heard about what happened to field
4? It's supposedto be measured on a scale from 1 to 10, with
0 indicating a missing value, but becauseof a data entry
error, all 10's were changed into 0's. Unfortunately, since
some of the patients have missing values for this field, it's
impossible to say whether a 0 in this field is a real 0 or a 10.
Quite a few of the records have that problem.
Data Miner: Interesting. Were there any other problems?
Statistician: Yes, fields 2 and 3 are basically the same, but I
assumethat you probably noticed that.
Data Miner: Yes, but these fields were only weak predictors of
field 5.
Statistician: Anyway, given all those problems, I'm surprised
you were able to accomplish anything.
Data Miner: Thue, but my results are really quite good. Field 1
is a very strong predictor of field 5. I'm surprised that this
wasn't noticed before.
Statistician: What? Field 1 is just an identification number.
Data Miner: Nonetheless,my results speak for themselves.
Statistician: Oh, no! I just remembered. We assignedID
numbers after we sorted the records based on field 5. There is
a strong connection, but it's meaningless.Sorry.
22 Chapter 2 Data

Although this scenario representsan extreme situation, it emphasizesthe

importance of "knowing your data." To that end, this chapter will address
each of the four issues mentioned above, outlining some of the basic challenges
and standard approaches.

2.L Types of Data

A data set can often be viewed as a collection of data objects. Other
names for a data object are record, po'int, uector, pattern, euent, case,sample,
obseruat'ion,or ent'ity. In turn, data objects are described by a number of
attributes that capture the basic characteristics of an object, such as the
mass of a physical object or the time at which an event occurred. Other
names for an attribute are uariable, characteristi,c,field, feature, ot d'imens'ion.
Example 2.2 (Student Information). Often, a data set is a file, in which
the objects are records (or rows) in the file and each field (or column) corre-
sponds to an attribute. For example, Table 2.1 shows a data set that consists
of student information. Each row correspondsto a student and each column
is an attribute that describes some aspect of a student, such as grade point
average (GPA) or identification number (ID).

datasetcontaining
Table2,1.A sample information.
student
StudentID Year Grade Point Average (GPA)

1034262 Senior 3.24

1052663 Sophomore 3.51
1082246 Fleshman 3.62

Although record-based data sets are common, either in flat files or rela-
tional database systems, there are other important types of data sets and
systems for storing data. In Section 2.I.2,we will discusssome of the types of
data sets that are commonly encountered in data mining. However, we first
consider attributes.
 2.L Types of Data 23

2.L.t Attributes and Measurement

In this section we address the issue of describing data by considering what
types of attributes are used to describe data objects. We first define an at-
tribute, then consider what we mean by the type of an attribute, and finally
describe the types of attributes that are commonly encountered.

What Is an attribute?

We start with a more detailed definition of an attribute.

Definition 2.1. An attribute is a property or characteristic of an object
that may vary; either from one object to another or from one time to another.
For example, eye color varies from person to person, while the temperature
of an object varies over time. Note that eye color is a symbolic attribute with
a small number of possiblevalues{brown,black,blue,green, hazel, etc.}, while
temperature is a numerical attribute with a potentially unlimited number of
values.
At the most basic level, attributes are not about numbers or symbols.
However, to {iscuss and more precisely analyze the characteristics of objects,
we assign numbers or symbols to them. To do this in a well-defined way, we
need a measurement scale.
Definition 2.2. A measurement scale is a rule (function) that associates
a numerical or symbolic value with an attribute of an object.
Formally, the process of measurement is the application of a measure-
ment scale to associatea value with a particular attribute of a specific object.
While this may seem a bit abstract, we engagein the processof measurement
all the time. For instance, we step on a bathroom scale to determine our
weight, we classify someone as male or female, or we count the number of
chairs in a room to seeif there will be enough to seat all the people coming to
a meeting. In all these cases)the "physical value" of an attribute of an object
is mapped to a numerical or symbolic value.
With this background, we can now discuss the type of an attribute, a
concept that is important in determining if a particular data analysis technique
is consistent with a specific type of attribute.

The Type of an Attribute

It should be apparent from the previous discussion that the properties of an

attribute need not be the same as the properties of the values used to mea-
24 Chapter 2 Data

sure it. In other words, the values used to represent an attribute may have
properties that are not properties of the attribute itself, and vice versa. This
is illustrated with two examples.
Example 2.3 (Employee Age and ID Number). Two attributes that
might be associatedwith an employee are ID and age (in years). Both of these
attributes can be represented as integers. However, while it is reasonableto
talk about the average age of an employee, it makes no sense to talk about
the average employee ID. Indeed, the only aspect of employees that we want
to capture with the ID attribute is that they are distinct. Consequently,the
only valid operation for employeeIDs is to test whether they are equal. There
is no hint of this limitation, however, when integers are used to represent the
employee ID attribute. For the age attribute, the properties of the integers
used to represent age are very much the properties of the attribute. Even so,
the correspondence is not complete since, for example, ages have a maximum'
while integers do not.
Example 2.4 (Length of Line Segments). Consider Figure 2.1, which
shows some objects-line segments and how the length attribute of these
objects can be mapped to numbers in two different ways. Each successive
line segment, going from the top to the bottom, is formed by appending the
topmost line segment to itself. Thus, the second line segment from the top is
formed by appending the topmost line segment to itself twice, the third line
segment from the top is formed by appending the topmost line segment to
itself three times, and so forth. In a very real (physical) sense, all the line
segmentsare multiples of the first. This fact is captured by the measurements
on the right-hand side of the figure, but not by those on the left hand-side.
More specifically, the measurement scale on the left-hand side captures only
the ordering of the length attribute, while the scale on the right-hand side
captures both the ordering and additivity properties. Thus, an attribute can be
measuredin a way that does not capture all the properties of the attribute. t
The type of an attribute should tell us what properties of the attribute are
reflected in the values used to measure it. Knowing the type of an attribute
is important because it tells us which properties of the measured values are
consistent with the underlying properties of the attribute, and therefore, it
allows us to avoid foolish actions, such as computing the average employee ID.
Note that it is common to refer to the type of an attribute as the type of a
measurement scale.
 2.1 Types of Data 25

----> 1

----> 2

--> 3

--> 5
of lengthsto numbers
A mapping

propertiesof
rensth. nffii?;'fi"::ilin""till8in*o
Figure2.1.Themeasurement
ofthelength
of linesegments
ontwodifferent
scales
of measurement.

The Different Types of Attributes

A useful (and simple) way to specify the type of an attribute is to identify

the properties of numbers that correspond to underlying properties of the
attribute. For example, an attribute such as length has many of the properties
of numbers. It makes senseto compare and order objects by length, as well
as to talk about the differencesand ratios of length. The following properties
(operations) of numbers are typically used to describe attributes.

1. Distinctness : and *

2. Order <) <, >, and )

3. Addition * and -

4. Multiplication x and /

Given these properties, we can define four types of attributes: nominal,

ordinal, interval, and ratio. Table 2.2 gives the definitions of these types,
along with information about the statistical operations that are valid for each
type. Each attribute type possessesall of the properties and operations of the
attribute types above it. Consequently, any property or operation that is valid
for nominal, ordinal, and interval attributes is also valid for ratio attributes.
In other words, the definition of the attribute types is cumulative. However,
 26 Chapter 2 Data

Table2.2, Different types.

attribute
Attribute
Type Description Examples Operations
Nominal The values of a nominal zrp codes, mode, entropy,
attribute are just different employee ID numbers, contingency
names; i.e., nominal values eye color, gender correlation,
provide only enough y2 test
information to distinguish
one object from another.
t\ --) +
Tl
\
Ordinal The values of an ordinal hardness of minerals, median,
attribute provide enough {good,better,best}, percentiles,
information to order grades, rank correlation,
objects. street numbers run tests,
(<,>) siqn tests
lnterval For interval attributes, the calendar dates, mean,
differences between values temperature in Celsius standard deviation,
are meaningful, i.e., a unit or Fahrenheit Pearson's
of measurement exists. correlation,
(+,-) t and F tests
Katto For ratio variables, both temperature in Kelvin. geometric mean,
differences and ratios are monetary quantities, harmonic mean,
meaningful. counts, age, mass, percent
( +, l ) length, variation
electrical current

this does not mean that the operations appropriate for one attribute type are
appropriate for the attribute types above it.
Nominal and ordinal attributes are collectively referred to as categorical
or qualitative attributes. As the name suggests,qualitative attributes, such
as employeeID, lack most of the properties of numbers. Even if they are rep-
resented by numbers, i.e., integers, they should be treated more like symbols.
The remaining two types of attributes, interval and ratio, are collectively re-
ferred to as quantitative or numeric attributes. Quantitative attributes are
represented by numbers and have most of the properties of numbers. Note
that quantitative attributes can be integer-valued or continuous.
The types of attributes can also be described in terms of transformations
that do not changethe meaning of an attribute. Indeed, S. Smith Stevens,the
psychologist who originally defined the types of attributes shown in Table 2.2,
defined them in terms of these permissible transformations. For example,
 2.L Types of Data 27

Table2,3.Transformations
thatdefineattribute
levels,
Attribute
Typ" Tlansformation Comment
Nominal Any one-to-one mapping, €.g., & It all employee IIJ numbers are
permutation of values reassigned, it will not make any
differcnce
()rdinal An order-preserving change of An attribute encompassingthe
values. i.e.. notion of good, better, best can
new _ualue : f (old_ualue), be represented equally well by
where / is a monotonic function. the values {1,2,3} or by
{ 0 . 5 ,1 , 1 0 } .
Interval new -ualue : a * old-talue I b, The Fahrenheit and Celsius
o. and b constants. temperature scales differ in the
Iocation of their zero value and
the size of a degree (unit).
Ratio new -ualue : a * ol,d-ua|ue Length can be measured in
meters or feet.

the meaning of a length attribute is unchanged if it is measured in meters

instead of feet.
The statistical operations that make sensefor a particular type of attribute
are those that will yield the same results when the attribute is transformed us-
ing a transformation that preservesthe attribute's meaning. To illustrate, the
averagelength of a set of objects is different when measured in meters rather
than in feet, but both averagesrepresentthe same length. Table 2.3 shows the
permissible (meaning-preserving)transformations for the four attribute types
of Table 2.2.
Example 2.5 (Temperature Scales). Temperature provides a good illus-
tration of some of the concepts that have been described. First, temperature
can be either an interval or a ratio attribute, depending on its measurement
scale. When measured on the Kelvin scale, a temperature of 2o is, in a physi-
cally meaningful way, twice that of a temperature of 1o. This is not true when
temperature is measured on either the Celsius or Fahrenheit scales,because,
physically, a temperature of 1o Fahrenheit (Celsius) is not much different than
a temperature of 2" Fahrenheit (Celsius). The problem is that the zero points
of the Fahrenheit and Celsius scales are, in a physical sense, arbitrary, and
therefore, the ratio of two Celsius or Fahrenheit temperatures is not physi-
cally meaningful.
28 Chapter 2 Data

Describing Attributes by the Number of Values

An independent way of distinguishing between attributes is by the number of

values they can take.

Discrete A discrete attribute has a finite or countably infinite set of values.

Such attributes can be categorical, such as zip codes or ID numbers,
or numeric, such as counts. Discrete attributes are often represented
using integer variables. Binary attributes are a special case of dis-
crete attributes and assume only two values, e.g., true/false, yes/no,
male/female, or 0f 1. Binary attributes are often representedas Boolean
variables, or as integer variables that only take the values 0 or 1.

Continuous A continuous attribute is one whosevaluesare real numbers. Ex-

amples include attributes such as temperature, height, or weight. Con-
tinuous attributes are typically represented as floating-point variables.
Practically, real values can only be measured and representedwith lim-
ited precision.

In theory, any of the measurementscaletypes-nominal, ordinal, interval, and

ratio could be combined with any of the types based on the number of at-
tribute values-binary, discrete, and continuous. However, some combinations
occur only infrequently or do not make much sense. For instance, it is difficult
to think of a realistic data set that contains a continuous binary attribute.
Typically, nominal and ordinal attributes are binary or discrete, while interval
and ratio attributes are continuous. However, count attributes, which are
discrete, are also ratio attributes.

Asymmetric Attributes

For asymmetric attributes, only presence a non-zero attribute value-is re-

garded as important. Consider a data set where each object is a student and
each attribute records whether or not a student took a particular course at
a university. For a specific student, an attribute has a value of 1 if the stu-
dent took the course associatedwith that attribute and a value of 0 otherwise.
Because students take only a small fraction of all available courses, most of
the values in such a data set would be 0. Therefore, it is more meaningful
and more efficient to focus on the non-zero values. To illustrate, if students
are compared on the basis of the coursesthey don't take, then most students
would seem very similar, at least if the number of courses is large. Binary
attributes where only non-zero values are important are called asymmetric
 2.L Types of Data 29

binary attributes. This type of attribute is particularly important for as-

sociation analysis, which is discussedin Chapter 6. It is also possible to have
discrete or continuous asymmetric features. For instance, if the number of
credits associatedwith each course is recorded, then the resulting data set will
consist of asymmetric discrete or continuous attributes.

2.L.2 Types of Data Sets

There are many types of data sets, and as the field of data mining develops
and matures, a greater variety of data sets become available for analysis. In
this section, we describe some of the most common types. For convenience,
we have grouped the types of data sets into three groups: record data, graph-
based data, and ordered data. These categoriesdo not cover all possibilities
and other groupings are certainly possible.

General Characteristics of Data Sets

Before providing details of specific kinds of data sets, we discuss three char-
acteristics that apply to many data sets and have a significant impact on the
data mining techniquesthat are used: dimensionality, sparsity, and resolution.

Dimensionality The dimensionality of a data set is the number of attributes

that the objects in the data set possess.Data with a small number of dimen-
sions tends to be qualitatively different than moderate or high-dimensional
data. Indeed, the difficulties associatedwith analyzing high-dimensional data
are sometimes referred to as the curse of dimensionality. Becauseof this,
an important motivation in preprocessing the data is dimensionality reduc-
tion. These issues are discussedin more depth later in this chapter and in
Appendix B.

Sparsity For some data sets, such as those with asymmetric features, most
attributes of an object have values of 0; in many casesTfewer than 1% of
the entries are non-zero. In practical terms, sparsity is an advantage because
usually only the non-zero values need to be stored and manipulated. This
results in significant savings with respect to computation time and storage.
FurthermoreT some data mining algorithms work well only for sparse data.

Resolution It is frequently possible to obtain data at different levels of reso-

Iution, and often the properties ofthe data are different at different resolutions.
For instance, the surface of the Earth seems very uneven at a resolution of a
30 Chapter 2 Data

few meters, but is relatively smooth at a resolution of tens of kilometers. The

patterns in the data also depend on the level of resolution. If the resolution
is too fine, a pattern may not be visible or may be buried in noise; if the
resolution is too coarse, the pattern may disappear. For example, variations
in atmospheric pressure on a scale of hours reflect the movement of storms
and other weather systems. On a scale of months, such phenomena are not
detectable.

Record Data

Much data mining work assumesthat the data set is a collection of records
(data objects), each of which consists of a fixed set of data fields (attributes).
See Figure 2.2(a). For the most basic form of record data, there is no explicit
relationship among records or data fields, and every record (object) has the
same set of attributes. Record data is usually stored either in flat files or in
relational databases. Relational databasesare certainly more than a collection
of records, but data mining often does not use any of the additional information
available in a relational database. Rather, the database serves as a convenient
place to find records. Different types of record data are described below and
are illustrated in Figure 2.2.

Tbansaction or Market Basket Data Tbansaction data is a special type

of record data, where each record (transaction) involves a set of items. Con-
sider a grocery store. The set of products purchased by a customer during one
shopping trip constitutes a transaction, while the individual products that
were purchased are the items. This type of data is called market basket
data because the items in each record are the products in a person's "mar-
ket basket." Tlansaction data is a collection of sets of items, but it can be
viewed as a set of records whose fields are asymmetric attributes. Most often,
the attributes are binary, indicating whether or not an item was purchased,
but more generally, the attributes can be discrete or continuous, such as the
number of items purchased or the amount spent on those items. Figure 2.2(b)
shows a sample transaction data set. Each row representsthe purchasesof a
particular customer at a particular time.

The Data Matrix If the data objects in a collection of data all have the
same fixed set of numeric attributes, then the data objects can be thought of as
points (vectors) in a multidimensional space,where each dimension represents
a distinct attribute describing the object. A set of such data objects can be
interpreted as an n'Lby n matrix, where there are rn rows, one for each object,
 2.L Types of Data 31

(a) Record data. (b) Ttansaction data.

Document1 0 0 2 o 0 0 2

Document2 0 7 0 0 0 0 0

Document3 0 I 0 0 2 2 0 o 0

(c) Data matrix. (d) Document-term matrix.

Figure2.2, Different
variations
of record
data.

and n columns, one for each attribute. (A representationthat has data objects
as columns and attributes as rows is also fine.) This matrix is called a data
matrix or a pattern matrix. A data matrix is a variation of record data,
but becauseit consists of numeric attributes, standard matrix operation can
be applied to transform and manipulate the data. Therefore, the data matrix
is the standard data format for most statistical data. Figure 2.2(c) shows a
sample data matrix.

The Sparse Data Matrix A sparsedata matrix is a special caseof a data

matrix in which the attributes are of the same type and are asymmetric; i.e.,
only non-zero values are important. Transaction data is an example of a sparse
data matrix that has only 0 1 entries. Another common example is document
data. In particular, if the order of the terms (words) in a document is ignored,
32 Chapter 2 Data

then a document can be represented as a term vector, where each term is

a component (attribute) of the vector and the value of each component is
the number of times the corresponding term occurs in the document. This
representation of a collection of documents is often called a document-term
matrix. Figure 2.2(d) showsa sample document-term matrix. The documents
are the rows of this matrix, while the terms are the columns. In practice, only
the non-zero entries of sparse data matrices are stored.

Graph-Based Data

A graph can sometimes be a convenient and powerful representation for data.

We consider two specific cases: (1) the graph captures relationships among
data objects and (2) the data objects themselvesare representedas graphs.

Data with Relationships among Objects The relationships among ob-

jects frequently convey important information. In such cases,the data is often
representedas a graph. In particular, the data objects are mapped to nodes
of the graph, while the relationships among objects are captured by the links
between objects and link properties, such as direction and weight. Consider
Web pages on the World Wide Web, which contain both text and links to
other pages. In order to process search queries, Web search engines collect
and process Web pages to extract their contents. It is well known, however,
that the links to and from each page provide a great deal of information about
the relevance of a Web page to a query, and thus, must also be taken into
consideration. Figure 2.3(a) shows a set of linked Web pages.

Data with Objects That Are Graphs If objects have structure, that
is, the objects contain subobjects that have relationships, then such objects
are frequently represented as graphs. For example, the structure of chemical
compounds can be representedby a graph, where the nodes are atoms and the
links between nodes are chemical bonds. Figure 2.3(b) shows a ball-and-stick
diagram of the chemical compound benzene,which contains atoms of carbon
(black) and hydrogen (gray). A graph representation makes it possible to
determine which substructures occur frequently in a set of compounds and to
ascertain whether the presenceof any of these substructures is associatedwith
the presenceor absenceof certain chemical properties, such as melting point
or heat of formation. Substructure mining, which is a branch of data mining
that analyzes such data, is consideredin Section 7.5.
 2.1 Types of Data 33

UsefulLinks:
. Bbuoq@hv -
Knowledge Discovery and
. mer Useful Web sib
Data Mining Bibliography
(GeB up&td frequenily, so dsironenl)
o ACM SIGmD

o onuqqets

o fteDahh€

Bd Refereffi in Dab MilDg and

Knwled$ Dlsc@ry

Us@ Fayyad, cregory HateBky-Shapirc,

Usam Fayyad, "Mining Databiles: Towtrds
Ptrdc Smyfr, ed Rmmy ud|many,
Algdhms ftr howl€dge D$overy", Bulleun oI
"Advses in kowledge Dhcovery dd De
he EEE ConpuFr Smery Technical CoM@
Mining", MI hess/the Mnkss, 1996
on&BEnginedng,vol 2t,no I, Mmh 1998
J Ross Quinlm, "g 5i kogms ftr Mehne
hing", Mqil Kilfmmn hblishers, 1993 CMstopher kheus, Philip Chd, ild Gretory
Michael Bery ild ftdon Linon "Dau Mining Habbky-Shph, "Systems for knowledge
T€hniques (For kkdng, Sales, md Custom Dis@veqf itr dMbes", IEEE Trmsacdons on
Suppd). John Wiley & Sons, 197 howledge sd Data Engineedng, 5(6)1903-913,
tuembr 193

(a) Linked Web pages. (b) Benzene molecule.

Figure2.3. Different ofgraphdata.

variations

Ordered Data

For some types of data, the attributes have relationships that involve order
in time or space. Different types of ordered data are described next and are
shown in Figure 2.4.

Sequential Data Sequential data, also referred to as temporal data, can

be thought of as an extension of record data, where each record has a time
associatedwith it. Consider a retail transaction data set that also stores the
time at which the transaction took place. This time information makes it
possibleto find patterns such as "candy salespeak before Halloween." A time
can also be associated with each attribute. For example, each record could
be the purchase history of a customer, with a listing of items purchased at
different times. Using this information, it is possible to find patterns such as
"people who buy DVD players tend to buy DVDs in the period immediately
following the purchase."
Figure 2.a@) shows an example of sequential transaction data. There
are fi.vedifferent times-/7, t2, t3, tl, and t5; three different customers-Cl,
 34 Chapter 2 Data

Time Customer ItemsPurchased GGTTCCGCCTTCAGCCCCGCGCC

t1 c1 A,B CGCAGGGCCCGCCCCGCGCCGTC
12 c3 A,C
12 c1 c,D GAGAAGGGCCCGCCTGGCGGGCG
t3 c2 A,D GGGGGAGGCGGGGCCGCCCGAGC
t4 c2 E
CCAACCGAGTECGACCAGGTGCC
r5 c1 A,E
CCCTCTGCTCGGCCTAGACCTGA
Customer Time and ltems Purchased GCTCATTAGGCGGCAGCGGACAG
c1 (t1:A,B)(t2:C,D) (ts:A,E)
c2 (t3:A, D) (t4:E) GCCAAGTAGAAEAEGCGAAGCGC
c3 ( t 2 : AC
, ) TGGGCTGCCTGCTGCGACCAGGG

(a) Sequential transaction data. (b) Genomic sequencedata.

Minneapolis Av€rage Monthly Temperature (1S2-1993)

25

20

E
15

E
10

1983 19& 1985 1986 1987 1984 1989 1990 1991 lS2 1993 1994
y€r

(c) Temperature time series. (d) Spatial temperature data.

Figure2.4. Different
variations
ofordered
data.

C2, and C3; and five different items A, B, C, D, and E. In the top table,
each row corresponds to the items purchased at a particular time by each
customer. For instance, at time f3, customer C2 purchased items A and D. In
the bottom table, the same information is displayed, but each row corresponds
to a particular customer. Each row contains information on each transaction
involving the customer, where a transaction is consideredto be a set of items
and the time at which those items were purchased. For example, customer C3
bought items A and C at time t2.
 2.1 Types of Data 35

Sequence Data Sequencedata consists of a data set that is a sequenceof

individual entities, such as a sequenceof words or letters. It is quite similar to
sequential data, except that there are no time stamps; instead, there are posi-
tions in an ordered sequence. For example, the genetic information of plants
and animals can be represented in the form of sequencesof nucleotides that
are known as genes. Many of the problems associated with genetic sequence
data involve predicting similarities in the structure and function of genesfrom
similarities in nucleotide sequences.Figure 2.4(b) shows a section of the hu-
man genetic code expressedusing the four nucleotides from which all DNA is
constructed: A, T, G, and C.

Time Series Data Time series data is a special type of sequential data
in which each record is a time series, i.e., a series of measurementstaken
over time. For example, a financial data set might contain objects that are
time seriesof the daily prices of various stocks. As another example, consider
Figure 2.4(c), which shows a time series of the averagemonthly temperature
for Minneapolis during the years 1982 to 1994. When working with temporal
data, it is important to consider temporal autocorrelation; i.e., if two
measurements are close in time, then the values of those measurements are
often very similar.

Spatial Data Some objects have spatial attributes, such as positions or ar-
eas, as well as other types of attributes. An example of spatial data is weather
data (precipitation, temperature, pressure) that is collected for a variety of
geographical locations. An important aspect of spatial data is spatial auto-
correlation; i.e., objects that are physically close tend to be similar in other
ways as well. Thus, two points on the Earth that are close to each other
usually have similar values for temperature and rainfall.
Important examples of spatial data are the science and engineering data
sets that are the result of measurements or model output taken at regularly
or irregularly distributed points on a two- or three-dimensional grid or mesh.
For instance, Earth science data sets record the temperature or pressure mea-
sured at points (grid cells) on latitude-longitude spherical grids of various
resolutions,e.8., 1o by 1o. (See Figure 2.4(d).) As another example, in the
simulation of the flow of a gas, the speed and direction of flow can be recorded
for each grid point in the simulation.
36 Chapter 2 Data

Handling Non-Record Data

Most data mining algorithms are designed for record data or its variations,
such as transaction data and data matrices. Record-oriented techniques can
be applied to non-record data by extracting features from data objects and
using these features to create a record corresponding to each object. Consider
the chemical structure data that was described earlier. Given a set of common
substructures, each compound can be represented as a record with binary
attributes that indicate whether a compound contains a specific substructure.
Such a representation is actually a transaction data set, where the transactions
are the compounds and the items are the substructures.
In some cases, it is easy to represent the data in a record format, but
this type of representation does not capture all the information in the data.
Consider spatio-temporal data consisting of a time series from each point on
a spatial grid. This data is often stored in a data matrix, where each row
representsa location and each column represents a particular point in time.
However, such a representation does not explicitly capture the time relation-
ships that are present among attributes and the spatial relationships that
exist among objects. This does not mean that such a representation is inap-
propriate, but rather that these relationships must be taken into consideration
during the analysis. For example, it would not be a good idea to use a data
mining technique that assumesthe attributes are statistically independent of
one another.

2.2 Data Quality

Data mining applications are often applied to data that was collected for an-
other purpose, or for future, but unspecified applications. For that reasonT
data mining cannot usually take advantage of the significant benefits of "ad-
dressing quality issues at the source." In contrast, much of statistics deals
with the design of experiments or surveys that achieve a prespecified level of
data quality. Because preventing data quality problems is typically not an op-
tion, data mining focuseson (1) the detection and correction of data quality
problems and (2) the use of algorithms that can tolerate poor data quality.
The first step, detection and correction, is often called data cleaning.
The following sectionsdiscussspecific aspectsof data quality. The focus is
on measurement and data collection issues,although some application-related
issuesare also discussed.
 2.2 Data Quality 37

2.2.L Measurement and Data Collection Issues

It is unrealistic to expect that data will be perfect. There may be problems due
to human error, limitations of measuring devices,or flaws in the data collection
process. Values or even entire data objects may be missing. In other cases,
there may be spurious or duplicate objects; i.e., multiple data objects that all
correspond to a single "real" object. For example, there might be two different
records for a person who has recently lived at two different addresses. Even if
all the data is present and "looks fine," there may be inconsistencies-a person
has a height of 2 meters, but weighs only 2 kilograms.
In the next few sections,we focus on aspectsofdata quality that are related
to data measurement and collection. We begin with a definition of measure-
ment and data collection errors and then consider a variety of problems that
involve measurement error: noise, artifacts, bias, precision, and accuracy. We
conclude by discussing data quality issuesthat may involve both measurement
and data collection problems: outliers, missing and inconsistent values, and
duplicate data.

Measurement and Data Collection Errors

The term measurement error refers to any problem resulting from the mea-
surement process. A common problem is that the value recorded differs from
the true value to some extent. For continuous attributes, the numerical dif-
ference of the measured and true value is called the error. The term data
collection error refers to errors such as omitting data objects or attribute
values, or inappropriately including a data object. For example, a study of
animals of a certain species might include animals of a related speciesthat are
similar in appearanceto the speciesof interest. Both measurementerrors and
data collection errors can be either systematic or random.
We will only consider general types of errors. Within particular domains,
there are certain types of data errors that are commonplace, and there ofben
exist well-developed techniques for detecting and/or correcting these errors.
For example, keyboard errors are common when data is entered manually, and
as a result, many data entry programs have techniques for detecting and, with
human intervention, correcting such errors.

Noise and Artifacts

Noise is the random component of a measurement error. It may involve the

distortion of a value or the addition of spurious objects. Figure 2.5 shows a
time series before and after it has been disrupted by random noise. If a bit
38 Chapter 2 Data

(a) Time series. (b) Time series with noise.

2.5.Noise
Figure ina timeseries
context.

. i*.
i

^ +,
+
T+
T

iat

(a) Three groups of points. (b) With noise points (+) added.

Figure2.6. Noiseina spatial

context.

more noise were added to the time series,its shape would be lost. Figure 2.6
shows a set of data points before and after some noise points (indicated by
'+'s) have been added. Notice that some of the noise points are intermixed
with the non-noise points.
The term noise is often used in connection with data that has a spatial or
temporal component. In such cases,techniques from signal or image process-
ing can frequently be used to reduce noise and thus, help to discover patterns
(signals) that might be "lost in the noise." Nonetheless,the elimination of
noise is frequently difficult, and much work in data mining focuses on devis-
ing robust algorithms that produce acceptable results even when noise is
present.
 2.2 Data Quality 39

Data errors may be the result of a more deterministic phenomenon, such

as a streak in the same place on a set of photographs. Such deterministic
distortions of the data are often referred to as artifacts.

Precision, Bias, and Accuracy

In statistics and experimental science,the quality of the measurementprocess

and the resulting data are measured by precision and bias. We provide the
standard definitions, followed by a brief discussion. For the following defini-
tions, we assume that we make repeated measurements of the same underlying
quantity and use this set of values to calculate a mean (average) value that
servesas our estimate of the true value.
Definition 2.3 (Precision). The closenessof repeated measurements(of the
same quantity) to one another.
Deffnition 2.4 (Bias). A systematic of measurements from-the
quantity being measured.
Precision is often measured by the standard deviation of a set of values,
while bias is measured by taking the difference between the mean of the set
of values and the known value of the quantity being measured. Bias can
only be determined for objects whose measured quantity is known by means
external to the current situation. Supposethat we have a standard laboratory
weight with a mass of 1g and want to assessthe precision and bias of our new
Iaboratory scale. We weigh the mass five times, and obtain the following five
values: {1.015,0.990, 1.013,1.001,0.986}.The mean of thesevaluesis 1.001,
and hence, the bias is 0.001. The precision, as measured by the standard
deviation, is 0.013.
It is common to use the more general term, accuracy, to refer to the
degree of measurement error in data.
Definition 2.5 (Accuracy). The closenessof measurementsto the true value
of the quantity being measured.
Accuracy depends on precision and bias, but since it is a general concept,
there is no specific formula for accuracy in terms of these two quantities.
One important aspect of accuracy is the use of significant digits. The
goal is to use only as many digits to represent the result of a measurementor
calculation as are justified by the precision of the data. For example, if the
Iength of an object is measuredwith a meter stick whose smallest markings are
millimeters, then we should only record the length of data to the nearest mil-
limeter. The precision of such a measurement #ould be * 0.5mm. We do not
40 Chapter 2 Data

review the details of working with significant digits, as most readers will have
encountered them in previous courses, and they are covered in considerable
depth in science,engineering, and statistics textbooks.
Issuessuch as significant digits, precision, bias, and accuracy are sometimes
overlooked, but they are important for data mining as well as statistics and
science. Many times, data sets do not come with information on the precision
of the data, and furthermore, the programs used for analysis return results
without any such information. Nonetheless,without some understanding of
the accuracy of the data and the results, an analyst runs the risk of committing
serious data analysis blunders.

Outliers

Outliers are either (1) data objects that, in some sense,have characteristics
that are different from most of the other data objects in the data set, or
(2) values of an attribute that are unusual with respect to the typical values
for that attribute. Alternatively, we can speak of anomalous objects or
values. There is considerableleeway in the definition of an outlier, and many
different definitions have been proposed by the statistics and data mining
communities. Furthermore, it is important to distinguish between the notions
of noise and outliers. Outliers can be legitimate data objects or values. Thus,
unlike noise, outliers may sometimes be of interest. In fraud and network
intrusion detection, for example, the goal is to find unusual objects or events
from among a large number of normal ones. Chapter 10 discussesanomaly
detection in more detail.

Missing Values

It is not unusual for an object to be missing one or more attribute values.

In some cases,the information was not collected; e.g., some people decline to
give their age or weight. In other cases,some attributes are not applicable
to all objects; e.g., often, forms have conditional parts that are filled out only
when a person answers a previous question in a certain way, but for simplicity,
all fields are stored. Regardless,missing values should be taken into account
during the data analysis.
There are severalstrategies (and variations on these strategies) for dealing
with missing data, each of which may be appropriate in certain circumstances.
These strategies are listed next, along with an indication of their advantages
and disadvantages.
 2.2 Data Quality 4L

Eliminate Data Objects or Attributes A simple and effective strategy

is to eliminate objects with missing values. However, even a partially speci-
fied data object contains some information, and if many objects have missing
values, then a reliable analysis can be difficult or impossible. Nonetheless,if
a data set has only a few objects that have missing values, then it may be
expedient to omit them. A related strategy is to eliminate attributes that
have missing values. This should be done with caution, however, since the
eliminated attributes may be the ones that are critical to the analysis.

Estimate Missing Values Sometimes missing data can be reliably esti-

mated. For example, consider a time series that changes in a reasonably
smooth fashion, but has a few, widely scattered missing values. In such cases,
the missing values can be estimated (interpolated) bV using the remaining
values. As another example, consider a data set that has many similar data
points. In this situation, the attribute values of the points closestto the point
with the missing value are often used to estimate the missing value. If the
attribute is continuous, then the averageattribute value of the nearest neigh-
bors is used; if the attribute is categorical, then the most commonly occurring
attribute value can be taken. For a concrete illustration, consider precipitation
measurements that are recorded by ground stations. For areas not containing
a ground station, the precipitation can be estimated using values observed at
nearby ground stations.

Ignore the Missing Value during Analysis Many data mining approaches
can be modified to ignore missing values. For example, suppose that objects
are being clustered and the similarity between pairs of data objects needs to
be calculated. If one or both objects of a pair have missing values for some
attributes, then the similarity can be calculated by using only the attributes
that do not have missing values. It is true that the similarity will only be
approximate, but unless the total number of attributes is small or the num-
ber of missing values is high, this degree of inaccuracy may not matter much.
Likewise, many classification schemes can be modified to work with missing
values.

Inconsistent Values

Data can contain inconsistent values. Consider an addressfield, where both a

zip code and city are listed, but the specifiedzip code area is not contained in
that city. It may be that the individual entering this information transposed
two digits, or perhaps a digit was misread when the information was scanned
42 Chapter 2 Data

from a handwritten form. Regardless of the cause of the inconsistent values,

it is important to detect and, if possible, correct such problems.
Some types of inconsistencesare easy to detect. For instance, a person's
height should not be negative. In other cases,it can be necessaryto consult
an external source of information. For example, when an insurance company
processesclaims for reimbursement, it checks the names and addresseson the
reimbursement forms against a database of its customers.
Once an inconsistencyhas been detected, it is sometimespossibleto correct
the data. A product code may have "check" digits, or it may be possible to
double-checka product code against a list of known product codes, and then
correct the code if it is incorrect, but close to a known code. The correction
of an inconsistency requires additional or redundant information.
Example 2.6 (Inconsistent Sea Surface Temperature). This example
illustrates an inconsistency in actual time series data that measuresthe sea
surfacetemperature (SST) at various points on the ocean. SST data was origi-
nally collected using ocean-basedmeasurements from ships or buoys, but more
recently, satellites have been used to gather the data. To create a long-term
data set, both sourcesof data must be used. However, becausethe data comes
from different sources, the two parts of the data are subtly different. This
discrepancy is visually displayed in Figure 2.7, which shows the correlation of
SST values between pairs of years. If a pair of years has a positive correlation,
then the location correspondingto the pair of years is colored white; otherwise
it is colored black. (Seasonalvariations were removed from the data since, oth-
erwise, all the years would be highly correlated.) There is a distinct change in
behavior where the data has been put together in 1983. Years within each of
the two groups, 1958-1982 and 1983-1999,tend to have a positive correlation
with one another, but a negative correlation with years in the other group.
This does not mean that this data should not be used, only that the analyst
should consider the potential impact of such discrepancieson the data mining
analysis.

Duplicate Data

A data set may include data objects that are duplicates, or almost duplicates,
of one another. Many people receive duplicate mailings becausethey appear
in a database multiple times under slightly different names. To detect and
eliminate such duplicates, two main issuesmust be addressed. First, if there
are two objects that actually represent a single object, then the values of
corresponding attributes may differ, and these inconsistent values must be
 2.2 Data Quality 43

b
I

60 65 70 75 80 85 90 95
Year

Figure2.7, Conelation
of SSTdatabetween positive
pairsof years.Whiteareasindicate correlation.
Blackareasindicate
negative
correlation.

resolved. Second,care needsto be taken to avoid accidentally combining data

objects that are similar, but not duplicates, such as two distinct people with
identical names. The term deduplication is often used to refer to the process
of dealing with these issues.
In some cases,two or more objects are identical with respect to the at-
tributes measured by the database, but they still represent different objects.
Here, the duplicates are legitimate, but may still cause problems for some al-
gorithms if the possibility of identical objects is not specifically accounted for
in their design. An example of this is given in Exercise 13 on page 91.

2.2.2 Issues Related to Applications

Data quality issues can also be considered from an application viewpoint as
expressed by the statement "data is of high quality if it is suitable for its
intended use." This approach to data quality has proven quite useful, particu-
Iarly in businessand industry. A similar viewpoint is also present in statistics
and the experimental sciences,with their emphasison the careful design of ex-
periments to collect the data relevant to a specific hypothesis. As with quality
44 Chapter 2 Data

issuesat the measurementand data collection level, there are many issuesthat
are specific to particular applications and fields. Again, we consider only a few
of the general issues.

Timeliness Some data starts to age as soon as it has been collected. In

particular, if the data provides a snapshot of some ongoing phenomenon or
process, such as the purchasing behavior of customers or Web browsing pat-
terns, then this snapshot representsreality for only a limited time. If the data
is out of date, then so are the models and patterns that are based on it.

Relevance The available data must contain the information necessary for
the application. Consider the task of building a model that predicts the acci-
dent rate for drivers. If information about the age and gender of the driver is
omitted, then it is likely that the model will have limited accuracy unless this
information is indirectly available through other attributes.
Making sure that the objects in a data set are relevant is also challenging.
A common problem is sampling bias, which occurs when a sample does not
contain different types of objects in proportion to their actual occurrence in
the population. For example, survey data describesonly those who respond to
the survey. (Other aspects of sampling are discussedfurther in Section 2.3.2.)
Becausethe results of a data analysis can reflect only the data that is present,
sampling bias will typically result in an erroneous analysis.

Knowledge about the Data Ideally, data sets are accompanied by doc-
umentation that describes different aspects of the data; the quality of this
documentation can either aid or hinder the subsequentanalysis. For example,
if the documentation identifies several attributes as being strongly related,
these attributes are likely to provide highly redundant information, and we
may decide to keep just one. (Consider sales tax and purchase price.) If the
documentation is poor, however, and fails to tell us, for example, that the
missing values for a particular field are indicated with a -9999, then our analy-
sis of the data may be faulty. Other important characteristicsare the precision
of the data, the type of features (nominal, ordinal, interval, ratio), the scale
of measurement (e.g., meters or feet for length), and the origin of the data.

2.3 Data Preprocessing

In this section, we address the issue of which preprocessingsteps should be
applied to make the data more suitable for data mining. Data preprocessing
 2.3 Data Preprocessing 45

is a broad area and consists of a number of different strategies and techniques

that are interrelated in complex ways. We will present some of the most
important ideas and approaches,and try to point out the interrelationships
among them. Specifically,we will discussthe following topics:

o Aggregation
o Sampling
o Dimensionality reduction
o Feature subset selection
o Feature creation
o Discretization and binarization
o Variable transformation

Roughly speaking, these items fall into two categories: selecting data ob-
jects and attributes for the analysis or creating/changing the attributes. In
both casesthe goal is to improve the data mining analysis with respect to
time, cost, and quality. Details are provided in the following sections.
A quick note on terminology: In the following, we sometimes use synonyms
for attribute, such as feature or variable, in order to follow common usage.

2.3.L Aggregation

Sometimes "lessis more" and this is the casewith aggregation, the combining
of two or more objects into a single object. Consider a data set consisting of
transactions (data objects) recording the daily sales of products in various
store locations (Minneapolis, Chicago, Paris, ...) for different days over the
courseof a year. SeeTable 2.4. One way to aggregatetransactions for this data
set is to replace all the transactions of a single store with a single storewide
transaction. This reducesthe hundreds or thousands of transactions that occur
daily at a specific store to a single daily transaction, and the number of data
objects is reduced to the number of stores.
An obvious issue is how an aggregatetransaction is created; i.e., how the
values of each attribute are combined acrossall the records correspondingto a
particular location to create the aggregatetransaction that representsthe sales
of a single store or date. Quantitative attributes, such as price, are typically
aggregated by taking a sum or an average. A qualitative attribute, such as
item, can either be omitted or summarized as the set of all the items that were
sold at that location.
The data in Table 2.4 can also be viewed as a multidimensional array,
where each attribute is a dimension. FYom this viewpoint, aggregation is the
46 Chapter 2 Data

Table2.4.
Datasetcontaining
information
about purchases.
customer
Ttansaction ID Item I Store Location Date

: :
L01r23 Watch Chicago 0e106/04
r0rl23 Battery Chicago 0e/06104
t0rr24 Shoes Minneapolis 0s106104

process of eliminating attributes, such as the type of item, or reducing the

number of values for a particular attribute; e.g., reducing the possible values
for date from 365 days to 12 months. This type of aggregation is commonly
used in Online Analytical Processing (OLAP), which is discussedfurther in
Chapter 3.
There are several motivations for aggregation. First, the smaller data sets
resulting from data reduction require less memory and processing time, and
hence, aggregation may permit the use of more expensive data mining algo-
rithms. Second,aggregation can act as a change ofscope or scaleby providing
a high-level view of the data instead of a low-level view. In the previous ex-
ample, aggregating over store locations and months gives us a monthly, per
store view of the data instead of a daily, per item view. Finally, the behavior
of groups of objects or attributes is often more stable than that of individual
objects or attributes. This statement reflects the statistical fact that aggregate
quantities, such as averagesor totals, have less variability than the individ-
ual objects being aggregated. For totals, the actual amount of variation is
larger than that of individual objects (on average),but the percentageof the
variation is smaller, while for means, the actual amount of variation is less
than that of individual objects (on average). A disadvantage of aggregation is
the potential loss of interesting details. In the store example aggregating over
months loses information about which day of the week has the highest sales.
Example 2.7 (Australian Precipitation). This example is based on pre-
cipitation in Australia from the period 1982 to 1993. Figure 2.8(a) shows
a histogram for the standard deviation of average monthly precipitation for
3,030 0.5' by 0.5' grid cells in Australia, while Figure 2.8(b) showsa histogram
for the standard deviation of the averageyearly precipitation for the same lo-
cations. The average yearly precipitation has less variability than the average
monthly precipitation. All precipitation measurements (and their standard
deviations) are in centimeters.
I
 2.3 Data Preprocessing 47

;
.5
6

z
z

(a) Histogram of standard deviation of (b) Histogram of standard deviation of

average monthly precipitation average yearly precipitation

Figure2.8. Histograms
of standard
deviation andyearlyprecipitation
formonthly in Australia
forthe
oeriod1982to 1993.

2.3.2 Sampling

Sampling is a commonly used approach for selecting a subset of the data

objects to be analyzed. In statistics, it has long been used for both the pre-
Iiminary investigation of the data and the final data analysis. Sampling can
also be very useful in data mining. However, the motivations for sampling
in statistics and data mining are ofben different. Statisticians use sampling
because obtaining the entire set of data of interest is too expensive or time
consuming, while data miners sample becauseit is too expensiveor time con-
suming to processall the data. In some cases,using a sampling algorithm can
reduce the data size to the point where a better, but more expensivealgorithm
can be used.
The key principle for effective sampling is the following: Using a sample
will work almost as well as using the entire data set if the sample is repre-
sentative. fn turn, a sample is representative if it has approximately the
same property (of interest) as the original set of data. If the mean (average)
of the data objects is the property of interest, then a sample is representative
if it has a mean that is close to that of the original data. Becausesampling is
a statistical process, the representativeness of any particular sample will vary,
and the best that we can do is choose a sampling scheme that guarantees a
high probability of getting a representative sample. As discussednext, this
involves choosing the appropriate sample size and sampling techniques.
48 Chapter 2 Data

Sampling Approaches

There are many sampling techniques, but only a few of the most basic ones
and their variations will be covered here. The simplest type of sampling is
simple random sampling. For this type of sampling, there is an equal prob-
ability of selecting any particular item. There are two variations on random
sampling (and other sampling techniques as well): (1) sampling without re-
placement-as each item is selected, it is removed from the set of all objects
that together constitute the population, and (2) sampling with replace-
ment-objects are not removed from the population as they are selected for
the sample. In sampling with replacement,the sameobject can be picked more
than once. The samples produced by the two methods are not much different
when samples are relatively small compared to the data set size, but sampling
with replacement is simpler to analyze since the probability of selecting any
object remains constant during the sampling process.
When the population consists of different types of objects, with widely
different numbers of objects, simple random sampling can fail to adequately
represent those types of objects that are less frequent. This can cause prob-
lems when the analysis requires proper representation of all object types. For
example, when building classification models for rare classes,it is critical that
the rare classesbe adequately representedin the sample. Hence, a sampling
scheme that can accommodate differing frequencies for the items of interest is
needed. Stratified sampling, which starts with prespecified groups of ob-
jects, is such an approach. In the simplest version, equal numbers of objects
are drawn from each group even though the groups are ofdifferent sizes. In an-
other variation, the number of objects drawn from each group is proportional
to the size of that group.
Example 2.8 (Sampling and Loss of Information). Once a sampling
technique has been selected, it is still necessaryto choose the sample size.
Larger sample sizes increase the probability that a sample will be representa-
tive, but they also eliminate much of the advantage of sampling. Conversely,
with smaller sample sizes,patterns may be missedor erroneouspatterns can be
detected. Figure 2.9(a) shows a data set that contains 8000 two-dimensional
points, while Figures 2.9(b) and 2.9(c) show samplesfrom this data set of size
2000 and 500, respectively. Although most of the structure of this data set is
present in the sample of 2000 points, much of the structure is missing in the
sample of 500 points. r
 2.3 Data Preprocessing 49

ii

(a) 8000points (b) 2000points (c) 500 points

Figure2.9. Example
ofthelossofstructure
withsampling,

Example 2.9 (Determining the Proper Sample Size). To illustrate that

determining the proper sample size requires a methodical approach, consider
the following task.

Given a set of data that consistsof a small number of almost equal-

sized groups, find at least one representative point for each of the
groups. Assume that the objects in each group are highly similar
to each other, but not very similar to objects in different groups.
Also assume that there are a relatively small number of groups,
e.g., 10. Figure 2.10(a) shows an idealized set of clusters (groups)
from which these points might be drawn.

This problem can be efficiently solved using sampling. One approach is to

take a small sample of data points, compute the pairwise similarities between
points, and then form groups of points that are highly similar. The desired
set of representative points is then obtained by taking one point from each of
these groups. To follow this approach, however, we need to determine a sample
size that would guarantee, with a high probability, the desired outcome; that
is, that at least one point will be obtained from each cluster. Figure 2.10(b)
shows the probability of getting one object from each of the 10 groups as the
sample size runs from 10 to 60. Interestingly, with a sample size of 20, there is
little chance (20%) of getting a sample that includes all 10 clusters. Even with
a sample size of 30, there is still a moderate chance (almost a0%) of getting a
sample that doesn't contain objects from all 10 clusters. This issue is further
explored in the context of clustering by Exercise 4 on page 559.
I
50 Chapter 2 Data

oo
oo
(d

oo e
n

o o
o o Sample Size

(a) ren sroupsor points' containspoints

f,tl-T:i:1ti[ ;,""T"1"

Figure2.10.Finding points
representative from10groups.

Progressive Sampling

The proper sample size can be difficult to determine, so adaptive or progres-

sive sampling schemes are sometimes used. These approaches start with a
small sample, and then increase the sample size until a sample of sufficient
size has been obtained. While this technique eliminates the need to determine
the correct sample size initially, it requires that there be a way to evaluate the
sample to judge if it is large enough.
Suppose, for instance, that progressive sampling is used to learn a pre-
dictive model. Although the accuracy of predictive models increasesas the
sample size increases, at some point the increase in accuracy levels off. We
want to stop increasing the sample size at this leveling-off point. By keeping
track of the change in accuracy of the model as we take progressively larger
samples, and by taking other samples close to the size of the current one, we
can get an estimate as to how close we are to this leveling-off point, and thus,
stop sampling.

2.3.3 Dimensionality Reduction

Data sets can have a large number of features. Consider a set of documents,
where each document is represented by a vector whose components are the
frequencies with which each word occurs in the document. In such cases,
 2.3 Data Preprocessing 51

there are typically thousands or tens of thousands of attributes (components),

one for each word in the vocabulary. As another example, consider a set of
time seriesconsisting of the daily closing price of various stocks over a period
of 30 years. In this case,the attributes, which are the prices on specific days,
again number in the thousands.
There are a variety of benefits to dimensionality reduction. A key benefit
is that many data mining algorithms work better if the dimensionality the
number of attributes in the data-is lower. This is partly becausedimension-
ality reduction can eliminate irrelevant features and reduce noise and partly
becauseof the curse of dimensionality, which is explained below. Another ben-
efit is that a reduction of dimensionality can lead to a more understandable
model becausethe model may involve fewer attributes. Also, dimensionality
reduction may allow the data to be more easily visualized. Even if dimen-
sionality reduction doesn't reduce the data to two or three dimensions, data
is often visualized by looking at pairs or triplets of attributes, and the num-
ber of such combinations is greatly reduced. Finally, the amount of time and
memory required by the data mining algorithm is reduced with a reduction in
dimensionality.
The term dimensionality reduction is often reserved for those techniques
that reduce the dimensionality of a data set by creating new attributes that
are a combination of the old attributes. The reduction of dimensionality by
selecting new attributes that are a subset of the old is known as feature subset
selection or feature selection. It will be discussedin Section 2.3.4.
In the remainder of this section, we briefly introduce two important topics:
the curse of dimensionality and dimensionality reduction techniques based on
linear algebra approachessuch as principal components analysis (PCA). More
details on dimensionality reduction can be found in Appendix B.

The Curse of Dimensionality

llhe curse of dimensionality refers to the phenomenon that many types of

data analysis become significantly harder as the dimensionality of the data
increases. Specifically, as dimensionality increases,the data becomesincreas-
ingly sparse in the space that it occupies. For classification, this can mean
that there are not enough data objects to allow the creation of a model that
reliably assignsa class to all possible objects. For clustering, the definitions
of density and the distance between points, which are critical for clustering,
become less meaningful. (This is discussedfurther in Sectionsg.1.2, 9.4.5, and
9.4.7.) As a result) many clustering and classification algorithms (and other
52 Chapter 2 Data

data analysis algorithms) have trouble with high-dimensional data-reduced

classification accuracy and poor quality clusters.

Linear Algebra Techniques for Dimensionality Reduction

Some of the most common approaches for dimensionality reduction, partic-

ularly for continuous data, use techniques from linear algebra to project the
data from a high-dimensional spaceinto a lower-dimensionalspace. Principal
Components Analysis (PCA) is a linear algebra technique for continuous
attributes that finds new attributes (principal components) that (1) are linear
combinations of the original attributes, (2) are orthogonal (perpendicular) to
each other, and (3) capture the maximum amount of variation in the data. For
example, the first two principal components capture as much of the variation
in the data as is possiblewith two orthogonal attributes that are linear combi-
nations of the original attributes. Singular Value Decomposition (SVD)
is a linear algebra technique that is related to PCA and is also commonly used
for dimensionality reduction. For additional details, see Appendices A and B.

2.3.4 Feature Subset Selection

Another way to reduce the dimensionality is to use only a subset of the fea-
tures. While it might seemthat such an approach would lose information, this
is not the case if redundant and irrelevant features are present. Redundant
features duplicate much or all of the information contained in one or more
other attributes. For example, the purchaseprice of a product and the amount
of salestax paid contain much of the same information. Irrelevant features
Oontain almost no useful information for the data mining task at hand. For
instance, students' ID numbers are irrelevant to the task of predicting stu-
dents' grade point averages. Redundant and irrelevant features can reduce
classification accuracy and the quality of the clusters that are found.
While some irrelevant and redundant attributes can be eliminated imme-
diately by using common senseoI domain knowledge,selectingthe best subset
of features frequently requires a systematic approach. The ideal approach to
feature selection is to try all possible subsets of features as input to the data
mining aigorithm of interest, and then take the subset that produces the best
results. This method has the advantage of reflecting the objective and bias of
the data mining algorithm that will eventually be used. Unfortunately, since
the number of subsetsinvolving n attributes is 2n, such an approach is imprac-
tical in most situations and alternative strategies are needed. There are three
standard approachesto feature selection: embedded, filter, and wrapper.
 2.3 Data Preprocessing 53

Embedded approaches Feature selection occurs naturally as part of the

data mining algorithm. Specifically, during the operation of the data mining
algorithm, the algorithm itself decides which attributes to use and which to
ignore. Algorithms for building decision tree classifiers,which are discussedin
Chapter 4, often operate in this manner.

Filter approaches Features are selectedbefore the data mining algorithm

is run, using some approach that is independent of the data mining task. For
example, we might select sets of attributes whose pairwise correlation is as low
as possible.

Wrapper approaches These methods use the target data mining algorithm
as a black box to find the best subset of attributes, in a way similar to that
of the ideal algorithm described above, but typically without enumerating all
possible subsets.

Since the embedded approachesare algorithm-specific, only the filter and

wrapper approacheswill be discussedfurther here.

An Architecture for Feature Subset Selection

It is possible to encompassboth the filter and wrapper approacheswithin a

common architecture. The feature selection processis viewed as consisting of
four parts: a measure for evaluating a subset, a search strategy that controls
the generation of a new subset of features, a stopping criterion, and a valida-
tion procedure. Filter methods and wrapper methods differ only in the way
in which they evaluate a subset of features. For a wrapper method, subset
evaluation usesthe target data mining algorithm, while for a filter approach,
the evaluation technique is distinct from the target data mining algorithm.
The following discussionprovides some details of this approach, which is sum-
marized in Figure 2.11.
Conceptually, feature subset selection is a search over all possible subsets
of features. Many different types of search strategies can be used, but the
searchstrategy should be computationally inexpensiveand should find optimal
or near optimal sets of features. It is usually not possible to satisfy both
requirements, and thus, tradeoffs are necessary.
An integral part ofthe searchis an evaluation step tojudge how the current
subset of features comparesto others that have been considered. This requires
an evaluation measurethat attempts to determine the goodnessof a subset of
attributes with respect to a particular data mining task, such as classification
54 Chapter 2 Data

of a feature
Figure2,11.Flowchart subset process.
selection

or clustering. For the filter approach, such measuresattempt to predict how

well the actual data mining algorithm will perform on a given set of attributes.
For the wrapper approach, where evaluation consists of actually running the
target data mining application, the subset evaluation function is simply the
criterion normally used to measure the result of the data mining.
Because the number of subsets can be enormous and it is impractical to
examine them all, some sort of stopping criterion is necessary.This strategy is
usualiy based on one or more conditions involving the following: the number
of iterations, whether the value of the subset evaluation measure is optimal or
exceedsa certain threshold, whether a subset of a certain size has been ob-
tained, whether simultaneous size and evaluation criteria have been achieved,
and whether any improvement can be achieved by the options available to the
search strategy.
Finally, once a subset of features has been selected, the results of the
target data mining algorithm on the selected subset should be validated. A
straightforward evaluation approach is to run the algorithm with the full set
of features and compare the full results to results obtained using the subset of
features. Hopefully, the subset of features will produce results that are better
than or almost as good as those produced when using all features. Another
validation approach is to use a number of different feature selection algorithms
to obtain subsetsof features and then compare the results of running the data
mining algorithm on each subset.
 2.3 Data Preprocessing 55

Feature Weighting

Feature weighting is an alternative to keeping or eliminating features. More

important features are assigned a higher weight, while less important features
are given a lower weight. These weights are sometimes assigned based on do-
main knowledge about the relative importance of features. Alternatively, they
may be determined automatically. For example, some classification schemes,
such as support vector machines (Chapter 5), produce classification models in
which each feature is given a weight. Features with larger weights play a more
important role in the model. The normalization of objects that takes place
when computing the cosine similarity (Section 2.4.5) can also be regarded as
a type of feature weighting.

2.3.5 Feature Creation

It is frequently possible to create, from the original attributes, a new set of
attributes that captures the important information in a data set much more
effectively. Furthermore, the number of new attributes can be smaller than the
number of original attributes, allowing us to reap all the previously described
benefits of dimensionality reduction. Three related methodologiesfor creating
new attributes are described next: feature extraction, mapping the data to a
new space,and feature construction.

Feature Extraction

The creation of a new set of features from the original raw data is known as
feature extraction. Consider a set of photographs, where each photograph
is to be classified according to whether or not it contains a human face. The
raw data is a set of pixels, and as such, is not suitable for many types of
classification algorithms. However, if the data is processed to provide higher-
level features, such as the presenceor absenceof certain types of edges and
areasthat are highly correlated with the presenceof human faces,then a much
broader set of classification techniques can be applied to this problem.
Unfortunately, in the sense in which it is most commonly used, feature
extraction is highly domain-specific. For a particular field, such as image
processing, various features and the techniques to extract them have been
developed over a period of time, and often these techniques have limited ap-
plicability to other fields. Consequently,whenever data mining is applied to a
relatively new area, a key task is the development of new features and feature
extraction methods.
56 Chapter 2 Data

(a) Two time series. (b) Noisy time series. (c) Power spectrum

of theFourier
Figure2.12.Application transform
to identify
theunderlying in timeseries
frequencies
data.

Mapping the Data to a New Space

A totally different view of the data can reveal important and interesting fea-
tures. Consider, for example, time series data, which often contains periodic
patterns. If there is only a single periodic pattern and not much noise' then
the pattern is easily detected. If, on the other hand, there are a number of
periodic patterns and a significant amount of noise is present, then these pat-
terns are hard to detect. Such patterns can, nonetheless,often be detected
by applying a Fourier transform to the time seriesin order to change to a
representation in which frequency information is explicit. In the example that
follows, it will not be necessary to know the details of the Fourier transform.
It is enough to know that, for each time series,the Fourier transform produces
a new data object whose attributes are related to frequencies.
Example 2.10 (Fourier Analysis). The time series presented in Figure
2.I2(b) is the sum of three other time series,two of which are shown in Figure
2.12(a) and have frequenciesof 7 and 17 cycles per second, respectively. The
third time series is random noise. Figure 2.12(c) shows the power spectrum
that can be computed after applying a Fourier transform to the original time
series. (Informally, the power spectrum is proportional to the square of each
frequency attribute.) In spite ofthe noise, there are two peaks that correspond
to the periods of the two original, non-noisy time series. Again, the main point
is that better features can reveal important aspects of the data. I
 2.3 Data Preprocessing 57

Many other sorts of transformations are also possible. Besidesthe Fourier

transform, the wavelet transform has also proven very useful for time series
and other types of data.

Feature Construction

Sometimesthe features in the original data setshave the necessaryinformation,

but it is not in a form suitable for the data mining algorithm. In this situation,
one or more new features constructed out of the original features can be more
useful than the original features.
Example 2.11- (Density). To illustrate this, consider a data set consisting
of information about historical artifacts, which, along with other information,
contains the volume and mass of each artifact. For simplicity, assume that
these artifacts are made of a small number of materials (wood, clay, bronze,
gold) and that we want to classify the artifacts with respect to the material
of which they are made. In this case, a density feature constructed from the
mass and volume features, i.e., density : mass/uolume, would most directly
yield an accurate classification. Although there have been some attempts to
automatically perform feature construction by exploring simple mathematical
combinations of existing attributes, the most common approach is to construct
features using domain expertise.

2.3.6 Discretization and Binarization

Some data mining algorithms, especially certain classification algorithms, re-
quire that the data be in the form of categorical attributes. Algorithms that
find association patterns require that the data be in the form of binary at-
tributes. Thus, it is often necessaryto transform a continuous attribute into
a categorical attribute (discretization), and both continuous and discrete
attributes may need to be transformed into one or more binary attributes
(binarization). Additionally, if a categorical attribute has a large number of
values (categories),or some values occur infrequently, then it may be beneficial
for certain data mining tasks to reduce the number of categoriesby combining
some of the values.
As with feature selection,the best discretization and binarization approach
is the one that "produces the best result for the data mining algorithm that
will be used to analyze the data." It is typically not practical to apply such a
criterion directly. Consequently,discretization or binarization is performed in
58 Chapter 2 Data

ofa categorical
Table2.5.Conversion tothreebinaryattributes.
attribute
Catesorical Value Integer Value fi1 Ta

awful 0 0 0 0
poor I 0 0 1
OK 2 0 1 0
good 3 0 1 1
great 4 1 0 0

ofa categorical
Table2.6. Conversion attribute binaryattributes.
tofiveasymmetric
Categorical Value Integer Value :x7 u5 tr4
awtuL 0 1 U 0 0 0
poor I 0 1 0 0 0
OK 2 0 0 1 0 0
good 3 0 0 0 1 0
great 4 0 0 0 0 1

a way that satisfies a criterion that is thought to have a relationship to good

performance for the data mining task being considered.

Binarization

A simple technique to binarize a categorical attribute is the following: If there

are rn categorical values, then uniquely assign each original value to an integer
in the interval [0,rn - 1]. If the attribute is ordinal, then order must be
maintained by the assignment. (Note that even if the attribute is originally
representedusing integers,this processis necessaryifthe integers are not in the
interval 10,^ - 1].) Next, convert each of these m integers to a binary number.
Since n : flogr(m)l binary digits are required to represent these integers,
represent these binary numbers using n binary attributes. To illustrate, a
categorical variable with 5 values {awful, poor, OK, good,great} would require
three binary variables frrt fr2t and 23. The conversion is shown in Table 2.5.
Such a transformation can cause complications, such as creating unin-
tended relationships among the transformed attributes. For example, in Table
2.5, attributes 12 and 13 are correlated because information about the good
value is encoded using both attributes. Furthermore, association analysis re-
quires asymmetric binary attributes, where only the presenceof the attribute
(value : 1) is important. For associationproblems, it is therefore necessaryto
introduce one binary attribute for each categorical value) as in Table 2.6. Ifthe
 2.3 Data Preprocessing 59

number of resulting attributes is too large, then the techniquesdescribedbelow

can be used to reduce the number of categorical values before binarization.
Likewise, for association problems, it may be necessaryto replace a single
binary attribute with two asymmetric binary attributes. Consider a binary
attribute that records a person's gender, male or female. For traditional as-
sociation rule algorithms, this information needs to be transformed into two
asymmetric binary attributes, one that is a 1 only when the person is male
and one that is a 1 only when the person is female. (For asymmetric binary
attributes, the information representation is somewhat inefficient in that two
bits of storage are required to represent each bit of information.)

Discretization of Continuous Attributes

Discretization is typically applied to attributes that are used in classification
or associationanalysis. In general, the best discretization dependson the algo-
rithm being used, as well as the other attributes being considered. Typically,
however, the discretization of an attribute is consideredin isolation.
tansformation of a continuous attribute to a categorical attribute involves
two subtasks: deciding how many categories to have and determining how to
map the valuesofthe continuous attribute to these categories. In the first step,
after the values of the continuous attribute are sorted, they are then divided
into n intervals by specifying n - r split points. In the second,rather trivial
step, all the values in one interval are mapped to the same categorical value.
Therefore, the problem of discretization is one of deciding how many split
points to choose and where to place them. The result can be represented
either as a set of intervals {(ro, ,t], (*t, ,zl, . . . , (*n-t, r",) }, where ro and rn
may be *oo or -oo, respectively, or equivalently, as a series of inequalities
:tg 1 it 1 IIt ...t In-I 1r 1rn.

IJnsupervised Discretization A basic distinction between discretization

methods for classification is whether class information is used (supervised) or
not (unsupervised). If class information is not used, then relatively simple
approachesare common. For instance, the equal width approach divides the
range of the attribute into a user-specifiednumber of intervals each having the
same width. Such an approach can be badly affected by outliers, and for that
reason, an equal frequency (equal depth) approach, which tries to put
the same number of objects into each interval, is often preferred. As another
example of unsupervised discretization, a clustering method, such as K-means
(see Chapter 8), can also be used. Finally, visually inspecting the data can
sometimes be an effective approach.
60 Chapter 2 Data

Example 2.12 (Discretization Techniques). This example demonstrates

how these approacheswork on an actual data set. Figure 2.13(a) shows data
points belonging to four different groups, along with two outliers-the large
dots on either end. The techniques of the previous paragraph were applied
to discretize the r values of these data points into four categorical values.
(Points in the data set have a random g component to make it easy to see
how many points are in each group.) Visually inspecting the data works quite
well, but is not automatic, and thus, we focus on the other three approaches.
The split points produced by the techniquesequal width, equal frequency,and
K-means are shown in Figures 2.13(b), 2.13(c),and 2.13(d), respectively.The
split points are represented as dashed lines. If we measure the performance of
a discretization technique by the extent to which different objects in different
groups are assignedthe same categorical value, then K-means performs best,
followed by equal frequency, and finally, equal width. I

Supervised Discretization The discretization methods described above

are usually better than no discretization, but keeping the end purpose in mind
and using additional information (class labels) often produces better results.
This should not be surprising, since an interval constructed with no knowledge
of class labels often contains a mixture of class labels. A conceptually simple
approach is to place the splits in a way that maximizes the purity of the
intervals. In practice, however,such an approach requires potentially arbitrary
decisions about the purity of an interval and the minimum size of an interval.
To overcomesuch concerns,some statistically basedapproachesstart with each
attribute value as a separate interval and create larger intervals by merging
adjacent intervals that are similar according to a statistical test. Entropy-
based approachesare one of the most promising approachesto discretization,
and a simple approach based on entropy will be presented.
First, it is necessaryto define entropy. Let k be the number of different
class labels, mibe the number of values in the eth interval of a partition, and
mai be the number of values of classj in interval i. Then the entropy ei of the
'ith interval is given by the equation

k
er:l piilogzpij,
i:l

where pq : m4 fm2 is the probability (fraction of values) of class j in the i'th

interval. The total entropy, e, of the partition is the weighted average of the
individual interval entropies, i.e.,
 2.3 Data Preprocessing 61,

t :1.
tr.
.ti
i..
.ri
I i..
t r;.
!, .3
s- 3' l. 'l

.;.
! ?r
2.1

10 15 10 15 20

(a) Original data. (b) Equal width discretization.

i :-i
i| * ' . . .
I .i.
i .'T
i: i.',i.
:g

| ., '
I ...
!:.
i i,:,
10 15 20

(c) Equal frequency discretization. (d) K-means discretization.

Figure2.13. Different
discretization
techniques.

"" :- \ - / , = i t o ' o '

where rn is the number of values, wi : mif m is the fraction of values in the

ith interval. and n is the number of intervals. Intuitively, the entropy of an
interval is a measure of the purity of an interval. If an interval contains only
values ofone class (is perfectly pure), then the entropy is 0 and it contributes
62 Chapter 2 Data

nothing to the overall entropy. If the classes of values in an interval occur

equally often (the interval is as impure as possible), then the entropy is a
maximum.
A simple approach for partitioning a continuous attribute starts by bisect-
ing the initial values so that the resulting two intervals give minimum entropy.
This technique only needs to consider each value as a possible split point, be-
causeit is assumedthat intervals contain ordered sets of values. The splitting
processis then repeated with another interval, typically choosing the interval
with the worst (highest) entropy, until a user-specifiednumber of intervals is
reached, or a stopping criterion is satisfied.
Example 2.13 (Discretization of Two Attributes). This method was
used to independently discretize both the z and y attributes of the two-
dimensional data shown in Figure 2.I4. In the first discretization, shown in
Figure 2.14(a), the r and g attributes were both split into three intervals. (The
dashed lines indicate the split points.) In the second discretization, shown in
Figure 2.L4(b), the r and grattributes were both split into five intervals. I

This simple example illustrates two aspects of discretization. First, in two

dimensions,the classesof points are well separated,but in one dimension, this
is not so. In general, discretizing each attribute separately often guarantees
suboptimal results. Second, five intervals work better than three, but six
intervals do not improve the discretization much, at least in terms of entropy.
(Entropy values and results for six intervals are not shown.) Consequently,
it is desirable to have a stopping criterion that automatically finds the right
number of partitions.

Categorical Attributes with Too Many Values

Categorical attributes can sometimes have too many values. If the categorical
attribute is an ordinal attribute, then techniques similar to those for con-
tinuous attributes can be used to reduce the number of categories. If the
categorical attribute is nominal, however, then other approachesare needed.
Consider a university that has a large number of departments. Consequently,
a department name attribute might have dozens of different values. In this
situation, we could use our knowledge of the relationships among different
departments to combine departments into larger groups, such as eng'ineering,
soc'ialsciences,or biological sc'iences.If domain knowledge does not serve as
a useful guide or such an approach results in poor classification performance,
then it is necessaryto use a more empirical approach, such as grouping values
 2.3 Data Preprocessing 63

-----xg -----
::"""*S.;t-
:"-,$

(a) Three intervals

Figure2.14.Discretizing
r andy attributes
forfourgroups
(classes)
of points.

together only if such a grouping results in improved classification accuracy or

achievessome other data mining objective.

2.3.7 Variable Tlansformation

A variable transformation refers to a transformation that is applied to all

the values of a variable. (We use the term variable instead of attribute to ad-
here to common usage, although we will also refer to attribute transformation
on occasion.) In other words, for each object, the transformation is applied to
the value of the variable for that object. For example, if only the magnitude
of a variable is important, then the values of the variable can be transformed
by taking the absolute value. In the following section, we discusstwo impor-
tant types of variable transformations: simple functional transformations and
normalization.

Simple Functions

For this type of variable transformation, a simple mathematical function is

applied to each value individually. If r is a variable, then examples of such
transformationsinclude rk, log tr, e', yE,If n, sinr, or lrl. In statistics,vari-
able transformations, especiallysqrt, log , and 7f r , are often used to transform
data that does not have a Gaussian (normal) distribution into data that does.
While this can be important, other reasonsoften take precedencein data min-
64 Chapter 2 Data

ing. Supposethe variable of interest is the number of data bytes in a session)

and the number of bytes ranges from 1 to 1 billion. This is a huge range, and
it may be advantageousto compress it by using a lo916 transformation. In
this case, sessionsthat transferred 108 and 10e bytes would be more similar
to each other than sessionsthat transferred 10 and 1000 bytes (9 - 8 : 1
versus 3 - 1 : 2). For some applications, such as network intrusion detection,
this may be what is desired, since the first two sessionsmost likely represent
transfers of large files, while the latter two sessionscould be two quite distinct
types of sessions.
Variable transformations should be applied with caution since they change
the nature of the data. While this is what is desired, there can be problems
if the nature of the transformation is not fully appreciated. For instance, the
transformation If r reduces the magnitude of values that are 1 or larger, but
increasesthe magnitude of values between 0 and 1. To illustrate, the values
{ L , 2 , 3 }g o t o { 1 , + , + } ,b u t t h e v a l u e{s1 , ; , + } g o t o { 1 , 2 , 3 } . T h u s ,f o r
all sets of values, the transformation If r revercesthe order. To help clarify
the effect of a transformation, it is important to ask questions such as the
following: Does the order need to be maintained? Does the transformation
apply to all values, especially negative values and 0? What is the effect of
the transformation on the values between 0 and 1? Exercise 17 on page 92
explores other aspects of variable transformation.

Normalization or Standardization

Another common type of variable transformation is the standardization or

normalization of a variable. (In the data mining community the terms are
often used interchangeably. In statistics, however, the term normalization can
be confusedwith the transformations used for making a variable normal, i.e',
Gaussian.) The goal of standardization or normalization is to make an en-
tire set of values have a particular property. A traditional example is that
of "standardizing a variable" in statistics. If 7 is the mean (average) of the
attribute values and s, is their standard deviation, then the transformation
r' : (r -?)lt" creates a new variable that has a mean of 0 and a standard
deviation of 1. If different variables are to be combined in some way, then
such a transformation is often necessary to avoid having a variable with large
values dominate the results of the calculation. To illustrate, consider compar-
ing people based on two variables: age and income. For any two people, the
difference in income will likely be much higher in absolute terms (hundreds or
thousands of dollars) than the difference in age (less than 150). If the differ-
encesin the range ofvalues ofage and income are not taken into account, then
 2.4 Measuresof Similaritv and Dissimilaritv 65

the comparison between people will be dominated by differencesin income. In

particular, if the similarity or dissimilarity of two people is calculated using the
similarity or dissimilarity measuresdefined later in this chapter, then in many
cases,such as that of Euclidean distance, the income values will dominate the
calculation.
The mean and standard deviation are strongly affected by outliers, so the
above transformation is often modified. First, the mean is replaced by the
median, i.e., the middle value. Second,the standard deviation is replaced by
the absolute standard deviation. Specifically, if r is a variable, then the
absolute standard deviation of r is given by oa : Dlrl*n - ltl, where ri is
lhe 'ith value of the variable, rn is the number of objects, and.p, is either the
mean or median. Other approaches for computing estimates of the location
(center) and spread of a set of values in the presenceof outliers are described
in Sections 3.2.3 and 3.2.4, respectively. These measurescan also be used to
define a standardi zation transformation.

2.4 Measures of Similarity and Dissimilarity

Similarity and dissimilarity are important becausethey are used by a number
of data mining techniques, such as clustering, nearest neighbor classification,
and anomaly detection. In many cases)the initial data set is not neededonce
these similarities or dissimilarities have been computed. Such approachescan
be viewed as transforming the data to a similarity (dissimilarity) space and
then performing the analysis.
We begin with a discussionof the basics: high-level definitions of similarity
and dissimilarity, and a discussion of how they are related. For convenience,
the term proximity is used to refer to either similarity or dissimilarity. Since
the proximity between two objects is a function of the proximity between the
corresponding attributes of the two objects, we first describe how to measure
the proximity between objects having only one simple attribute, and then
consider proximity measures for objects with multiple attributes. This in-
cludes measuressuch as correlation and Euclidean distance, which are useful
for dense data such as time series or two-dimensional points, as well as the
Jaccard and cosine similarity measures, which are useful for sparse data like
documents. Next, we consider several important issuesconcerning proximity
measures. The section concludes with a brief discussion of how to select the
right proximity measure.
66 Chapter 2 Data

2.4.L Basics
Definitions

Informally, the similarity between two objects is a numerical measure of the

degreeto which the two objects are alike. Consequently,similarities are hi,gher
for pairs of objects that are more alike. Similarities are usually non-negative
and are often between 0 (no similarity) and 1 (complete similarity).
The dissimilarity between two objects is a numerical measure of the de-
gree to which the two objects are different. Dissimilarities are lower for more
similar pairs of objects. FYequently,the term distance is used as a synonym
for dissimilarity, although, as we shall see, distance is often used to refer to
a special class of dissimilarities. Dissimilarities sometimes fall in the interval
[0,1], but it is also common for them to range from 0 to oo.

TYansformations

Thansformations are often applied to convert a similarity to a dissimilarity,

or vice versa, or to transform a proximity measure to fall within a particular
range, such as [0,1]. For instance, we may have similarities that range from 1
to 10, but the particular algorithm or software packagethat we want to use
may be designed to only work with dissimilarities, or it may only work with
similarities in the interval 10,1].We discuss these issueshere becausewe will
employ such transformations later in our discussion of proximity. In addi-
tion, these issuesare relatively independent of the details of specific proximity
measures.
Frequently, proximity measures, especially similarities, are defined or trans-
formed to have values in the interval [0,1]. Informally, the motivation for this
is to use a scale in which a proximity value indicates the fraction of similarity
(or dissimilarity) between two objects. Such a transformation is often rela-
tively straightforward. For example, if the similarities between objects range
from 1 (not at all similar) to 10 (completely similar), we can make them fall
within the range 10,1] by using the transformation s' : (s - 1)/9, where s and
s/ are the original and new similarity values, respectively. In the more general
case, the transformation of similarities to the interval [0,1] is given by the
expression : (" - mi,n-s)I (mar -s - mi,n-s), where mar -s and m'in-s are the
"'
maximum and minimum similarity values, respectively. Likewise, dissimilarity
measureswith a finite range can be mapped to the interval [0,1] by using the
formula d' : (d - rni,n-d)l(mar-d - mi,n-d).
There can be various complications in mapping proximity measuresto the
interval 10,1],however. If, for example, the proximity measureoriginally takes
 2.4 Measuresof Similaritv and Dissimilaritv 67

values in the interval [0,-], then a non-linear transformation is needed and

values will not have the same relationship to one another on the new scale.
Consider the transformation d,' : dl(I* d) for a dissimilarity measure that
ranges from 0 to oo. The dissimilarities 0, 0.5, 2, 10, 100, and 1000 will be
transformed into the new dissimilarities0, 0.33, 0.67, 0.90, 0.99, and 0.999,
respectively. Larger values on the original dissimilarity scale are compressed
into the range of values near 1, but whether or not this is desirable dependson
the application. Another complication is that the meaning of the proximity
measure may be changed. For example, correlation, which is discussedlater,
is a measure of similarity that takes values in the interval [-1,1]. Mapping
these values to the interval [0,1] by taking the absolute value losesinformation
about the sign, which can be important in some applications. SeeExercise 22
on page 94.
Tlansforming similarities to dissimilarities and vice versa is also relatively
straightforward, although we again face the issues of preserving meaning and
changing a linear scale into a non-linear scale. If the similarity (or dissimilar-
ity) falls in the interval [0,1], then the dissimilarity can be defined as d : 1- s
(s : 1 - d). Another simple approach is to define similarity as the nega-
tive of the dissimilarity (or vice versa). To illustrate, the dissimilarities 0, 1,
10, and 100 can be transformed into the similarities 0, -1, -10, and -100,
respectively.
The similarities resulting from the negation transformation are not re-
stricted to the range [0,1], but if that is desired, then transformations such as
: : : 1-
" ;{1, s e-d, or s *mZ can be used. For the transformation
s :
7fi, the dissimilarities0, 1, 10, 100 are transformedinto 1, 0.5, 0.09, 0.01,
respectively. For s : e-d, they become 1.00, 0.37, 0.00, 0.00, respectively,
while for s: 1- *ffin they become 1.00, 0.99, 0.00, 0.00, respectively.
In this discussion,we have focusedon converting dissimilarities to similarities.
Conversion in the opposite direction is consideredin Exercise 23 on page 94.
In general, any monotonic decreasingfunction can be used to convert dis-
similarities to similarities, or vice versa. Of course, other factors also must
be consideredwhen transforming similarities to dissimilarities, or vice versa,
or when transforming the values of a proximity measure to a new scale. We
have mentioned issuesrelated to preserving meaning, distortion of scale, and
requirements of data analysis tools, but this list is certainly not exhaustive.

2.4.2 Similarity and Dissimilarity between Simple Attributes

The proximity of objects with a number of attributes is typically defined by
combining the proximities of individual attributes, and thus, we first discuss
68 Chapter 2 Data

proximity between objects having a single attribute. Consider objects de-

scribed by one nominal attribute. What would it mean for two such objects
to be similar? Since nominal attributes only convey information about the
distinctness of objects, all we can say is that two objects either have the same
value or they do not. Hence, in this casesimilarity is traditionally defined as 1
if attribute values match, and as 0 otherwise. A dissimilarity would be defined
in the opposite way: 0 if the attribute values match, and 1 if they do not.
For objects with a single ordinal attribute, the situation is more compli-
cated becauseinformation about order should be taken into account. Consider
an attribute that measuresthe quality of a product, €.8., a candy bar, on the
scale{poor, fai,r, OK, good,wonderful).It would seemreasonablethat a prod-
uct, P1, which is rated wonderful, would be closer to a produclP2, which is
rated good,than it would be to a product P3, which is rated OK. To make this
observation quantitative, the values of the ordinal attribute are often mapped
to successiveintegers, beginning at 0 or 1, e.g., {poor:O, fair:|, OK:2,
good:3, wonderful:4). Then, d(Pl,P2) - 3 - 2 : 7 or, if we want the dis-
similarity to fall between 0 and 1, d(P1, P2) : TZ :0.25. A similarity for
ordinal attributes can then be defined as s : 7 - d.
This definition of similarity (dissimilarity) for an ordinal attribute should
make the reader a bit uneasy since this assumesequal intervals, and this is not
so. Otherwise, we would have an interval or ratio attribute. Is the difference
between the values fair and good really the same as that between the values
OK and wonderful? Probably not, but in practice, our options are limited,
and in the absence of more information, this is the standard approach for
defining proximity between ordinal attributes.
For interval or ratio attributes, the natural measure of dissimilarity be-
tween two objects is the absolute difference of their values. For example, we
might compare our current weight and our weight a year ago by saying "I am
ten pounds heavier." In casessuch as these, the dissimilarities typically range
from 0 to oo, rather than from 0 to 1. The similarity of interval or ratio at-
tributes is typically expressedby transforming a similarity into a dissimilarity,
as previously described.
Table 2.7 summarizesthis discussion. In this table, r and g are two objects
that have one attribute of the indicated type. AIso, d(r,a) and s(r,gr) are the
dissimilarity and similarity between r and g, respectively. Other approaches
are possible; these are the most common ones.
The following two sections consider more complicated measuresof prox-
imity between objects that involve multiple attributes: (1) dissimilarities be-
tween data objects and (2) similarities between data objects. This division
 2.4 Measuresof Similaritv and Dissimilaritv 69

Table2.7. Similarity
anddissimilarity
forsimple
attributes
Attribute Dissimilarity Similaritv
T'ype
)_ 0 1fr:A I \fr:y
Nominal 11, - s:
I if rly 0 ifnly
d:lr-all("-t)
Ordinal (valuesmapped to integers 0 to n-1 s:I-d
where n is the number of values)
Interval or Ratio d:lr-al s:-d, ' s : ; i ,L t a s:e-o,
^ - 1 d-min-d
-

allows us to more naturally display the underlying motivations for employing

various proximity measures. We emphasize,however, that similarities can be
transformed into dissimilarities and vice versa using the approachesdescribed
earlier.

2.4.3 Dissimilarities between Data Objects

In this section, we discuss various kinds of dissimilarities. We begin with a
discussion of distances, which are dissimilarities with certain properties, and
then provide examples of more general kinds of dissimilarities.

Distances

We first present some examples, and then offer a more formal description of
distancesin terms of the properties common to all distances. The Euclidean
distance, d, between two points, x and y, in one-, two-, three-, or higher-
dimensional space, is given by the following familiar formula:

n
\-r - a k\)t ' ,
d(*, y) : ) \tn (2.1)
K=l

where n is the number of dimensions and rp and.Ak are) respectively,the kth

attributes (components) of r and g. We illustrate this formula with Figure
2.15 and Tables 2.8 and 2.9, which show a set of points, the e and grcoordinates
of these points, and the distance matrix containing the pairwise distances
of these points.
70 Chapter 2 Data

The Euclidean distance measure given in Equation 2.1 is generalized by

the Minkowski distance metric shown in Equation 2.2,

(, ,\
d(x, y) :
(Lor-orr)''"
where r is a parameter. The following are the three most common examples
of Minkowski distances.

. r :1. City block (Manhattan, taxicab, L1 norm) distance. A common

example is the Hamming distance, which is the number of bits that
are different between two objects that have only binary attributes, i.e.,
between two binary vectors.

o r :2. Euclidean distance (L2 norm).

. r : oo. Supremum (L*o, or L- norm) distance. This is the maximum

difference between any attribute of the objects. More formally, the L-
distance is defined by Equation 2.3

d(*, Y) : (2.3)
J* (U'wr-rrt')"'

The r parameter should not be confused with the number of dimensions (at-
tributes) n. The Euclidean, Manhattan, and supremum distances are defined
for all values of n: I,2,3,..., and specify different ways of combining the
differencesin each dimension (attribute) into an overall distance.
Tables 2.10 and 2.11, respectively, give the proximity matrices for the L1
and Loo distances using data from Table 2.8. Notice that all these distance
matrices are symmetric; i.e., the ijth entry is the same as the jith entry. In
Table 2.9, for instance, the fourth row of the first column and the fourth
column of the first row both contain the value 5.1.
Distances, such as the Euclidean distance, have some well-known proper-
ties. If d(*, y) is the distance between two points, x and y, then the following
properties hold.

1. Positivity

(a) d(x, x) > 0 for all x and y,

(b) d(x, Y) : 0 onlY if x : Y.
 2.4 Measuresof Similaritv and Dissimilaritv 7L

123456
X

Figure
2.15.Four points.
two-dimensional

2.8.r andy coordinates

Tabfe offourpoints.Table
2.9.Euclidean
distance
matrix
forTable
2.g.
point z coordinate y coordinate pl p2 p3 p4
p1 0 2 pl 0.0 2 . 8 3.2 5.1
p2 2 0 p2 2 . 8 0.0 t.4 3.2
p3 .) 1 p3 3.2 L.4 0.0 2.0
p4 1 p4 c . r 3.2 2.0 0.0

Table2.10.L1distance
matrix
forTable
2.8. Table
2,11.L- distance
matrix
forTable
2.8.
L1 p1 p2 p3 p4 L- pl p2 p3 p4
pl 0.0 4.0 4.0 6.0 p1 0.0 2.0 3.0 5.0
p2 4.0 0.0 2.0 4.0 p2 2.0 0.0 1 . 0 3.0
p3 4.0 2.0 0.0 2.0 p3 3.0 1.0 0.0 2.0
p4 6.0 4.0 2.0 0.0 p4 5.0 3.0 2.0 0 . 0

2. Symmetry
d(*,Y) : d(Y,x) for all x and Y.

3. T[iangle Inequality
d(x,z) < d(*, y) + d(y, z) for all points x, y,, and z.
Measures that satisfy all three properties are known as metrics. Some
people only use the term distance for dissimilarity measuresthat satisfy these
properties, but that practice is often violated. The three properties described
here are useful, as well as mathematically pleasing. AIso, if the triangle in-
equality holds, then this property can be used to increasethe efficiencyof tech-
niques (including clustering) that depend on distancespossessingthis property.
(SeeExercise 25.) Nonetheless,many dissimilarities do not satisfy one or more
of the metric properties. We give two examples of such measures.
72 Chapter 2 Data

Example 2.L4 (Non-metric Dissimilarities: Set Differences). This ex-

ample is based on the notion of the difference of two sets, as defined in set
theory. Given two sets -4 and B, A - B is the set of elements of A that are
n o t i n B . F o r e x a m p l e i,f A : { I , 2 , 3 , 4 } a n d B : { 2 , 3 , 4 } , t h e n A - B : { 1 }
and B - A - 0, the empty set. We can define the distance d between two
sets A and B as d(A,B): size(A- B), where s'ize ts a function returning
the number of elements in a set. This distance measure, which is an integer
value greater than or equal to 0, does not satisfy the second part of the pos-
itivity property the symmetry property, or the triangle inequality. However,
these properties can be made to hold if the dissimilarity measure is modified
as follows: d(A,B): size(A- B) + si,ze(B- A). SeeExercise21 on page
94. r
Example 2.15 (Non-metric Dissimilarities: Time). This example gives
a more everyday example of a dissimilarity measure that is not a metric, but
that is still useful. Define a measure of the distance between times of the day
as follows:
:
d,(t1,t2) _tr) lill i I:} (2.4)
{7^*t,az
To illustrate, d(lPM, 2PM) : t hour, while d(2PM, 1PM) : 23 hours.
Such a definition would make sense,for example, when answering the question:
"ff an event occurs at lPM every day, and it is now 2PM, how Iong do I have
to wait for that event to occur again?"

2.4.4 Similarities between Data Objects

For similarities, the triangle inequality (or the analogous property) typically
does not hold, but symmetry and positivity typically do. To be explicit, if
s(x, y) is the similarity between points x and y, then the typical properties of
similarities are the following:

1. s(x,y) : 1 only ifx : y. (0 < s S 1)

2. s(x,y) : s(y, x) for all x and y. (Symmetry)

There is no general analog of the triangle inequality for similarity mea-

sures. It is sometimes possible, however, to show that a similarity measure
can easily be converted to a metric distance. The cosineand Jaccard similarity
measures,which are discussedshortly, are two examples. Also, for specificsim-
ilarity measures,it is possibleto derive mathematical bounds on the similarity
between two objects that are similar in spirit to the triangle inequality.
 2.4 Measuresof Similarity and 73

Example 2.L6 (A Non-symmetric Similarity Measure). Consider an

experiment in which people are asked to classify a small set of characters as
they flash on a screen. The confusion matrix for this experiment records how
often each character is classified as itself, and how often each is classified as
another character. For instance, supposethat "0" appeared 200 times and was
classified as a "0" 160 times, but as an "o" 40 times. Likewise, suppose that
'o' appeared 200
times and was classifiedas an "o" 170 times, but as "0" only
30 times. If we take these counts as a measure of the similarity between two
characters, then we have a similarity measure,but one that is not symmetric.
In such situations, the similarity measure is often made symmetric by setting
s'(x, y) : s'(y, x) : (s(x, y)+ s(y, 12, where s/ indicatesthe new similarity
"D
measure,

2 . 4 . 5 Examples of Proximity Measures

This section provides specific examples of some similarity and dissimilarity
measures.

Similarity Measures for Binary Data

Similarity measuresbetween objects that contain only binary attributes are

called similarity coefficients, and typically have values between 0 and 1. A
value of 1 indicates that the two objects are completely similar, while a value
of 0 indicates that the objects are not at all similar. There are many rationales
for why one coefificientis better than another in specific instances.
Let x and y be two objects that consist of n binary attributes. The com-
parison of two such objects, i.e., two binary vectors, Ieadsto the following four
quantities (frequencies):

.foo: the number of attributes where x is 0 and y is 0

.for : the number of attributes where x is 0 and y is 1
,fio : the number of attributes where x is 1 and y is 0
: the number of attributes where x is 1 and y is 1
"fir

Simple Matching Coefficient One commonly used similarity coefficient is

the simple matching coefficient (SMC), which is defined as

number of matching attribute values ft * fss

SMC: -l (2.5)
number of attributes for fn * ,frr * ,foo'
74 Chapter 2 Data

This measure counts both presencesand absencesequally. Consequently, the

SMC could be used to find students who had answeredquestions similarly on
a test that consisted only of true/false questions.

Jaccard Coefficient Supposethat x and y are data objects that represent

two rows (two transactions) of a transaction matrix (seeSection 2.1.2). If each
asymmetric binary attribute correspondsto an item in a store, then a 1 indi-
cates that the item was purchased,while a 0 indicates that the product was not
purchased. Sincb the number of products not purchased by any customer far
outnumbers the number of products that were purchased,a similarity measure
such as SMC would say that all transactions are very similar. As a result, the
Jaccard coefficient is frequently used to handle objects consisting of asymmet-
ric binary attributes. The Jaccard coefficient, which is often symbolized by
,./,'is given by the following equation:

number of matching presences J7I

J: (2.6)
number of attributes not involved in 00 matches fot * fn -t ft

Example 2.17 (The SMC and Jaccard Similarity Coefficients). To

illustrate the difference between these two similarity measures, we calculate
SMC and -I for the following two binary vectors.

x : (1,0,0,0,0,0,0,0,0,0)
y : ( 0 , 0 , 0 , 0 , 0 ,10,,0 , 01, )

for :2 the numberof attributes wherex was 0 and y was 1

frc: I the numberof attributeswherex was 1 and y was 0
foo:7 the numberof attributeswherex was 0 and y was 0
/rr : 0 the number of attributes where x was 1 and y was 1

SMC: -;-,=--;t' -u.l

J:-T-,+----;-:--4-:g
JOI TJ IOTJ I I ZTITU

Cosine Similarity

Documents are often represented as vectors, where each attribute represents

the frequency with which a particular term (word) occurs in the document. It
is more complicated than this, of course, since certain common words are ig-
 2.4 Measuresof Similaritv and Dissimilaritv 75

nored and various processing techniques are used to account for different forms
of the same word, differing document lengths, and different word frequencies.
Even though documents have thousands or tens of thousands of attributes
(terms), each document is sparsesince it has relatively few non-zero attributes.
(The normalizations used for documents do not create a non-zero entry where
there was azero entry; i.e., they preservesparsity.) Thus, as with transaction
data, similarity should not depend on the number of shared 0 values since
any two documents are likely to "not contain" many of the same words, and
therefore, if 0-0 matches are counted, most documents will be highly similar to
most other documents. Therefore, a similarity measure for documents needs
to ignores 0-0 matches like the Jaccard measure, but also must be able to
handle non-binary vectors. The cosine similarity, defined next, is one of the
most common measure of document similarity. If x and y are two document
vectors, then

:
cos(x,y) (, 7\

ffi,
where. indicatesthe vectordot product,x .y : D[:trplp, andllxll is the
lengthof vectorx, ll*ll : 1fD|:rr2r: 1/x4.
Example 2.18 (Cosine Similarity of Two Document Vectors). This
example calculates the cosine similarity for the following two data objects,
which might represent document vectors:

* : ( 3 ,2 , 0 , 5 , 0 , 0 , 0
2, 0,0)
y : ( 1 , 0 , 0 , 0 , 0 , 0 ,10, ,0 , 2 )

x . y : 3 i .1 * 2 x 0 * 0 * 0 * 5 x 0 * 0 x 0 f 0 * 0 * 0 * 0 * 2 * l * 0 x 0 * 0 x 2 : 5
l l x l:l roxo *oxo *2x2*oxo *oxo:6.48
l l y l:l :2.24
cos(x,y) : 0.31
I

As indicated by Figure 2.16, cosine similarity really is a measure of the

(cosine of the) angle between x and y. Thus, if the cosine similarity is 1, the
angle between x and y is 0o, and x and y are the same except for magnitude
(length). If the cosine similarity is 0, then the angle between x and y is 90o,
and they do not share any terms (words).
76 Chapter 2 Data

Figule
2.16.Geometric
illustration measure.
ofthecosine

Equation2.7 canbe written as Equation 2.8.

c o s ( x , y: ) + . , , Y , , : x ' . y ' , (2.8)

llxll llvll

where x' : x/llxll and y/ : V lllyll. Dividing x and y by their lengths normal-
izesthem to have a length of 1. This means that cosinesimilarity does not take
the magnitude ofthe two data objects into account when computing similarity.
(Euclidean distance might be a better choice when magnitude is important.)
For vectors with a length of 1, the cosine measurecan be calculated by taking
a simple dot product. Consequently,when many cosine similarities between
objects are being computed, normalizing the objects to have unit length can
reduce the time required.

Extended Jaccard Coefficient (Tanimoto Coefficient)

The extended Jaccard coefficient can be used for document data and that re-
duces to the Jaccard coefficient in the caseof binary attributes. The extended
Jaccard coefficient is also known as the Tanimoto coefficient. (However, there
is another coefficient that is also known as the Tanimoto coefficient.) This co-
efficient, which we shall represent as E J , is defined by the following equation:

x.y
EJ(x,y): (2.e)
ll"ll'+llvll'-*.v

Correlation

The correlation between two data objects that have binary or continuous vari-
ables is a measure of the linear relationship between the attributes of the
objects. (The calculation of correlation between attributes, which is more
common, can be defined similarly.) More precisely, Pearson's correlation
 2 .4 Measures of Similaritv and Dissimilaritv 77

coefficient between two data objects, x and y, is defined by the following

equation:

covariance(x,y) StA
corr(x, y) : (2.10)
standard-deviation(x) xstandard-deviation(y) tr ta'

where we are using the following standard statistical notation and definitions:
1n
: s,s: :+ I("u
covariance(x,y) -z)(yx -9) (2.11)
n- r-. E:L

standard-deviation(x) : su :
fil@n-n)'
standard-deviation(y) : su:
;\l@*-vt'

1fl

i : I)-r*isthemeanofx
n4
k:1
1fl

g : 1fy*isthemeanofy
h:1

Example 2.19 (Perfect Correlation). Correlation is always in the range

-1 to 1. A correlation of 1 (-1) means that x and y have a perfect positive
(negative) linear relationship; that is, 16 - aAx -f b, where a, and b are con-
stants. The following two sets of values for x and y indicate cases where the
correlation is -1 and *1, respectively. In the first case,the means of x and y
were chosen to be 0, for simplicity.

x: ( - 3 , 6 , 0 , 3 ,- 6 )
y: ( 1 ,- 2 , 0 , - 7 , 2 )

x: (3,6,0,3,6)
y : (1,2,0,L,2)
I
78 Chapter 2 Data

-1.00 -.0.90 4.70 --0.60

-o.10 0.10 0.20 0.30

0.80 1.00

Figure plots
2.17.Scatter illustrating from-1 to1.
correlations

Example 2.20 (Non-linear Relationships). If the correlationis 0, then

there is no linear relationshipbetweenthe attributes of the two data objects.
However,non-linearrelationshipsmay still exist. In the following example,
n*: A7,but their correlationis 0.

*: (-3, -2,-1, 0, I, 2, 3)
Y:(9, 4,1,0,1,4,9)
I

Example 2.21 (Visualizing Correlation). It is also easy to judge the cor-

relation between two data objects x and y by plotting pairs of corresponding
attribute values. Figure 2.17 shows a number of these plots when x and y
have 30 attributes and the values of these attributes are randomly generated
(with a normal distribution) so that the correlation of x and y ranges from -1
to 1. Each circle in a plot representsone of the 30 attributes; its r coordinate
is the value of one of the attributes for x, while its 3r coordinate is the value
of the same attribute for y. I
If we transform x and y by subtracting off their means and then normaliz-
ing them so that their lengths are 1, then their correlation can be calculated by
 2.4 Measuresof Similaritv and Dissimilaritv 79

taking the dot product. Notice that this is not the same as the standardization
used in other contexts, where we make the transformations, r'* : (rp - ,) lt"
and y'r : (A* - T) I sa.

Bregman Divergence* This section provides a brief description of Breg-

man divergences, which are a family of proximity functions that share some
common properties. As a result, it is possible to construct general data min-
ing algorithms, such as clustering algorithms, that work with any Bregman
divergence. A concrete example is the K-means clustering algorithm (Section
8.2). Note that this section requires knowledge of vector calculus.
Bregman divergencesare loss or distortion functions. To understand the
idea of a loss function, considerthe following. Let x and y be two points, where
y is regarded as the original point and x is some distortion or approximation
of it. For example, x may be a point that was generated, for example, by
adding random noise to y. The goal is to measure the resulting distortion or
Ioss that results if y is approximated by x. Of course, the more similar x and
y are, the smaller the loss or distortion. Thus, Bregman divergencescan be
used as dissimilarity functions.
More formally, we have the following definition.
Definition 2.6 (Bregman Divergence). Given a strictly convex function
@ (with a few modest restrictions that are generally satisfied), the Bregman
divergence (loss function) D(x, y) generated by that function is given by the
following equation:

D(*,y) : d(x) - Q0) - (Vd(v),(* - y)) (2.r2)

where Vd(V) is the gradient of / evaluated at y, x - y, is the vector difference

between x and y, and (Vd(V), (" - V)) is the inner product between Vd(*)
and (x-y). For points in Euclidean space,the inner product is just the dot
product.

D(*,y) can be written as D(x,V) : d(x) - L(*), where.L(x) : d$)+

(Vd(V), (* - V)) and representsthe equation of a plane that is tangent to the
function Q at y. Using calculus terminology, L(x) is the linearization of. Q
around the point y and the Bregman divergenceis just the differencebetween
a function and a linear approximation to that function. Different Bregman
divergences are obtained by using different choices for S.
Example 2.22. We provide a concrete example using squared Euclidean dis-
tance, but restrict ourselvesto one dimension to simplify the mathematics. Let
80 Chapter 2 Data

r and y be real numbers and /(t) be the real valued function, d(t) : t2. In
that case, the gradient reduces to the derivative and the dot product reduces
to multiplication. Specifically,Equation 2.L2 becomesEquation 2.13.

D (*,i l : 1 2- A 2- 2A@- A): @ - il' ( 2.13)

The graph for this example, with 3r : 1, is shown in Figure 2.18. The
Bregman divergenceis shown for two values of r: r :2 and r :3. r

-1 01234
x

Figure
2.18.lllustration
ofBregman
divergence.

2.4.6 fssues in Proximity Calculation

This section discussesseveral important issuesrelated to proximity measures:
(1) how to handle the casein which attributes have different scalesand/or are
correlated, (2) how to calculate proximity between objects that are composed
of different types of attributes, e.g., quantitative and qualitative, (3) and how
to handle proximity calculation when attributes have different weights; i.e.,
when not all attributes contribute equally to the proximity of objects.
 2.4 Measuresof Similarity and Dissimilarity 81

Standardization and Correlation for Distance Measures

An important issue with distance measures is how to handle the situation

when attributes do not have the same range of values. (This situation is
often described by saying that "the variables have different scales.") Earlier,
Euclidean distance was used to measure the distance between people based on
two attributes: age and income. Unless these two attributes are standardized,
the distance between two people will be dominated by income.
A related issue is how to compute distance when there is correlation be-
tween some of the attributes, perhaps in addition to differencesin the rangesof
values. A generalization of Euclidean distance, the Mahalanobis distance,
is useful when attributes are correlated, have different ranges of values (dif-
ferent variances), and the distribution of the data is approximately Gaussian
(normal). Specifically,the Mahalanobis distance between two objects (vectors)
x and y is defined as

mahalanobis(x,y) : (x - y)>-1(x -y)r, (2.14)

where E-1 is the inverse of the covariancematrix of the data. Note that the
covariance matrix E is the matrix whose ijth entry is the covariance of the ith
and jth attributes as defined by Equation 2.II.
Example 2.23. In Figure 2.19, there are 1000 points, whose r and g at-
tributes have a correlation of 0.6. The distance between the two large points
at the opposite ends of the long axis of the ellipse is 14.7 in terms of Euclidean
distance, but only 6 with respect to Mahalanobis distance. In practice, com-
puting the Mahalanobis distance is expensive,but can be worthwhile for data
whose attributes are correlated. If the attributes are relatively uncorrelated,
but have different ranges, then standardizing the variables is sufficient.
I

Combining Similarities for fleterogeneous Attributes

The previous definitions of similarity were based on approachesthat assumed

all the attributes were of the sametype. A generalapproach is neededwhen the
attributes are of different types. One straightforward approach is to compute
the similarity between each attribute separately using Table 2.7, and then
combine these similarities using a method that results in a similarity between
0 and 1. Typically, the overall similarity is defined as the average of all the
individual attribute similarities.
82 Chapter 2 Data

Figure2.19.Setoftwo-dimensionalpoints.
TheMahalanobis
distance thetwopointsrepre-
between
sentedbylargedotsis6;theirEuclidean
distance
is 14.7.

Unfortunately, this approach does not work well if some of the attributes
are asymmetric attributes. For example, if all the attributes are asymmetric
binary attributes, then the similarity measuresuggestedpreviously reducesto
the simple matching coefficient, a measure that is not appropriate for asym-
metric binary attributes. The easiest way to fix this problem is to omit asym-
metric attributes from the similarity calculation when their values are 0 for
both of the objects whose similarity is being computed. A similar approach
also works well for handling missing values.
In summary, Algorithm 2.7 is effective for computing an overall similar-
ity between two objects, x and y, with different types of attributes. This
procedure can be easily modified to work with dissimilarities.

Using Weights

In much of the previous discussion, all attributes were treated equally when
computing proximity. This is not desirable when some attributes are more im-
portant to the definition of proximity than others. To addressthese situations,
 2.4 Measuresof Similaritv and Dissimilaritv 83

Algorithm 2.1 Similarities of heterogeneousobjects.

1-: For the kth attribute, compute a similarity, s,r(x,y), in the range [0, 1].
2 : Define an indicator variable, d'7.,for the kth attribute as follows:

3: Compute the overall similarity between the two objects using the following for-
mula:
:$ffi
similarity(x,") (2.15)

the formulas for proximity can be modified by weighting the contribution of

each attribute.
If the weights u.r6sum to L, bhen (2.15) becomes

similarity(x,y) : D?:tl|lns !(*' Y) (2.16)

DT:'6x
The definition of the Minkowski distance can also be modified as follows:

/ n \1/'
d(x, y) : ( D,'ol"r - akl'I (2.r7)
\t:r /

2.4.7 Selecting the Right Proximity Measure

The following are a few general observations that may be helpful. First, the
type of proximity measure should fit the type of data. For many types of dense,
continuous data, metric distance measures such as Euclidean distance are of-
ten used. Proximity between continuous attributes is most often expressed
in terms of differences, and distance measures provide a well-defined way of
combining these differences into an overall proximity measure. Although at-
tributes can have different scales and be of differing importance, these issues
can often be dealt with as described earlier.
For sparse data, which ofben consists of asymmetric attributes, we typi-
cally employ similarity measures that ignore 0-0 matches. Conceptually, this
reflects the fact that, for a pair of complex objects, similarity depends on the
number of characteristics they both share, rather than the number of charac-
teristics they both lack. More specifically, for sparse, asymmetric data, most
84 Chapter 2 Data

objects have only a few of the characteristics described by the attributes, and
thus, are highly similar in terms of the characteristics they do not have. The
cosine,Jaccard, and extended Jaccard measuresare appropriate for such data.
There are other characteristics of data vectors that may need to be consid-
ered. Suppose, for example, that we are interested in comparing time series.
If the magnitude of the time seriesis important (for example, each time series
represent total sales of the same organization for a different year), then we
could use Euclidean distance. If the time series represent different quantities
(for example, blood pressureand oxygen consumption), then we usually want
to determine if the time series have the same shape, not the same magnitude.
Correlation, which uses a built-in normalization that accounts for differences
in magnitude and level, would be more appropriate.
In some cases,transformation or normalization of the data is important
for obtaining a proper similarity measure since such transformations are not
always present in proximity measures. For instance, time series may have
trends or periodic patterns that significantly impact similarity. Also, a proper
computation of similarity may require that time lags be taken into account.
Finally, two time seriesmay only be similar over specific periods of time. For
example, there is a strong relationship between temperature and the use of
natural gas, but only during the heating season.
Practical consideration can also be important. Sometimes, a one or more
proximity measuresare already in use in a particular field, and thus, others
will have answered the question of which proximity measures should be used.
Other times, the software package or clustering algorithm being used may
drastically limit the choices. If efficiency is a concern, then we may want to
choosea proximity measure that has a property, such as the triangle inequality,
that can be used to reduce the number of proximity calculations. (SeeExercise
25.)
However, if common practice or practical restrictions do not dictate a
choice,then the proper choiceof a proximity measurecan be a time-consuming
task that requires careful consideration of both domain knowledge and the
purpose for which the measure is being used. A number of different similarity
measures may need to be evaluated to see which ones produce results that
make the most sense.

2.5 Bibliographic Notes

It is essential to understand the nature of the data that is being analyzed,
and at a fundamental level, this is the subject of measurement theory. In
 2.5 Bibliographic Notes 85

particular, one of the initial motivations for defining types of attributes was
to be precise about which statistical operations were valid for what sorts of
data. We have presented the view of measurement theory that was initially
described in a classic paper by S. S. Stevens 179]. (Tables 2.2 and 2.3 are
derived from those presentedby Stevens [80].) While this is the most common
view and is reasonably easy to understand and apply, there is, of course,
much more to measurementtheory. An authoritative discussioncan be found
in a three-volume series on the foundations of measurement theory [63, 69,
81]. AIso of interest is a wide-ranging article by Hand [55], which discusses
measurement theory and statistics, and is accompanied by comments from
other researchers in the field. Finally, there are many books and articles that
describe measurement issuesfor particular areas of scienceand engineering.
Data quality is a broad subject that spans every discipline that uses data.
Discussionsof precision, bias, accuracy, and significant figures can be found
in many introductory science,engineering,and statistics textbooks. The view
of data quality as "fitness for use" is explained in more detail in the book by
Redman [76]. Those interested in data quality may also be interested in MIT's
Total Data Quality Management program [70, 84]. However, the knowledge
neededto deal with specific data quality issuesin a particular domain is often
best obtained by investigating the data quality practices of researchersin that
field.
Aggregation is a less well-defined subject than many other preprocessing
tasks. However, aggregationis one of the main techniquesused by the database
area of Online Analytical Processing(OLAP), which is discussedin Chapter 3.
There has also been relevant work in the area of symbolic data analysis (Bock
and Diday [a7]). One of the goals in this area is to summarize traditional record
data in terms of symbolic data objects whose attributes are more complex than
traditional attributes. Specifically, these attributes can have values that are
setsof values (categories),intervals, or sets of valueswith weights (histograms).
Another goal of symbolic data analysis is to be able to perform clustering,
classification,and other kinds of data analysis on data that consistsof symbolic
data objects.
Sampling is a subject that has been well studied in statistics and related
fields. Many introductory statistics books, such as the one by Lindgren [65],
have some discussionon sampling, and there are entire books devoted to the
subject, such as the classic text by Cochran [49]. A survey of sampling for
data mining is provided by Gu and Liu [54], while a survey of sampling for
databases is provided by Olken and Rotem [ZZ]. There are a number of other
data mining and database-related sampling referencesthat may be of interest,
86 Chapter 2 Data

including papers by Palmer and Faloutsos [74], Provost et al. [75], Toivonen
[82],and Zakiet al. [85].
In statistics, the traditional techniques that have been used for dimension-
ality reduction are multidimensional scaling (MDS) (Borg and Groenen [48],
Kruskal and Uslaner [6a]) and principal component analysis (PCA) (Jolliffe
[58]), which is similar to singular value decomposition (SVD) (Demmel [50]).
Dimensionality reduction is discussedin more detail in Appendix B.
Discretization is a topic that has been extensively investigated in data
mining. Some classification algorithms only work with categorical data, and
association analysis requires binary data, and thus, there is a significant moti-
vation to investigate how to best binarize or discretize continuous attributes.
For association analysis, we refer the reader to work by Srikant and Agrawal
[78], while some useful referencesfor discretization in the area of classification
include work by Dougherty et al. [51], Elomaa and Rousu [SZ], Fayyad and
Irani [53], and Hussain et al. [56].
Feature selectionis another topic well investigated in data mining. A broad
coverageof this topic is provided in a survey by Molina et al. [71] and two
books by Liu and Motada [66, 67]. Other useful paperc include those by Blum
and Langley 1461,Kohavi and John [62], and Liu et al. [68].
It is difficult to provide referencesfor the subject of feature transformations
becausepractices vary from one discipline to another. Many statistics books
have a discussionof transformations, but typically the discussionis restricted
to a particular purpose, such as ensuring the normality of a variable or making
sure that variables have equal variance. We offer two references: Osborne [73]
and Ttrkey [83].
While we have covered some of the most commonly used distance and
similarity measures,there are hundreds of such measuresand more are being
created all the time. As with so many other topics in this chapter, many of
these measuresare specificto particular fields; e.g., in the area of time seriessee
papers by Kalpakis et al. [59] and Keogh and Pazzani [61]. Clustering books
provide the best general discussions.In particular, seethe books by Anderberg
[45], Jain and Dubes [57], Kaufman and Rousseeuw[00], and Sneath and Sokal
1771.

Bibliography
[45] M. R. Anderberg. Cluster Analysis for Appli,cati,ons. Academic Press, New York, De-
cember 1973.
[461A. BIum and P. Langley. Selection of Relevant Features and Examples in Machine
Learning. Artificial Intellig ence, 97 (l=2) :245-27 l, 1997.
 Bibliography 87

l47l H. H. Bock and E. Diday. Analysis of Sgmbolic Data: Exploratory Methods for Ertract-
ing Statistical Information from Complen Data (Studi,es in Classifi,cation, Data Analys'is,
and, Know ledge Org an'izat'ionl. Springer-Verlag Telos, January 2000.
[48] I. Borg and P. Groenen. Modern Multidimensional Scaling Theory and, Applications.
Springer-Verlag, February 1997.
[49] W. G. Cochran. Sampling Techniques. John Wiley & Sons, 3rd edition, JuJy 1977.
[50] J. W. Demmel. Applied, Numerical Linear Algebra. Society for Industrial & Applied
Mathematics, September 1997.
[51] J. Dougherty, R. Kohavi, and M. Sahami. Supervised and Unsupervised Discretization
of Continuous Features. In Proc. of the 12th Intl. Conf. on Machine Learni,ng, pages
L94-202, t995.
[52] T. Elomaa and J. Rousu. General and Efficient Multisplitting of Numerical Attributes.
M achine Learni,ng,36(3):201 244, 1999.
[53] U. M. Fayyad and K. B. Irani. Multi-interval discretization of continuousvalued at-
tributes for classification learning. In Proc. 13th Int. Joint Conf. on Arti;fi,cial Intelli,-
gence, pages lO22-L027. Morgan Kaufman, 1993.
154] F. H. Gaohua Gu and H. Liu. Sampling and Its Application in Data Mining: A Survey.
Technical Report TRA6/00, National University of Singapore, Singapore, 2000.
f55] D. J. Hand. Statistics and the Theory of Measurement. Jountal of the Rogal Statistical
Societg: Series A (Statistics in Societg),159(3):445-492, 1996.
[56] F. Hussain, H. Liu, C. L. Tan, and M. Dash. TRC6/99: Discretization: an enabling
technique. Technical report, National University of Singapore, Singapore, 1999.
[57j A. K. Jain and R. C. Dubes. Algorithrns for Clustering Data. Prentice Hall
Advanced Reference Series. Prentice Hall, March 1988. Book available online at
http: //www.cse.msu.edu/-jain/Clustering-Jain-Dubes.pdf.
[58] I. T. Jolliffe. Principal Cornponent Analys'is. Springer Verlag, 2nd edition, October
2002.
[59] K. Kalpakis, D. Gada, and V. Puttagunta. Distance Measures for Effective Clustering
of ARIMA Time-Series. In Proc. of the 2001 IEEE Intl. Conf. on Data Mini'ng, pages
273-280. IEEE Computer Society, 2001.
[60] L. Kaufman and P. J. Rousseeuw. Findi,ng Groups in Data: An Introduction to Cluster
Analysi.s. Wiley Series in Probability and Statistics. John Wiley and Sons, New York,
November 1990.
[61] E. J. Keogh and M. J. Pazzani. Scaling up dynamic time warping for datamining
applications. In KDD, pages 285-289, 2000.
[62] R. Kohavi and G. H. John. Wrappers for Feature Subset Selection. Artificial Intell'igence,
97 {I-2) :273-324, 1997.
f63] D. Krantz, R. D. Luce, P. Suppes, and A. Tversky. Foundations of Measurements:
Volume 1: Additiue and polgnomial representations. Academic Press, New York, 1971.
[64] J. B. Kruskal and E. M. Uslaner. Multiilimensional Scal'ing. Sage Publications, August
1978.
[65] B. W. Lindgren. Statistical Theory CRC Press, January 1993.
f66] H. Liu and H. Motoda, editors. Feature Ertract'ion, Constr-uction and Select'ion: A Data
Mini,ng Perspectiue. Kluwer International Series in Engineering and Computer Science,
453. Kluwer Academic Publishers, July 1998.
[67] H. Liu and H. Motoda. Feature Selection for Knowleilge Discouery and Data Mi'n-
ing. Kluwer International Series in Engineering and Computer Science, 454. Kluwer
Academic Publishers. Julv 1998.
88 Chapter 2 Data

168] H. Liu, H. Motoda, and L. Yu. Feature Extraction, Selection, and Construction. In
N. Ye, editor, The Handbook of Data Mi,ning, pages22 41. Lawrence Erlbaum Asso-
ciates, Inc., Mahwah, NJ, 2003.
[69] R. D. Luce, D. Krantz, P. Suppes, and A. Tversky. Foundat'ions of Measurements:
Volume 3: Representation, Ariomatizati,on, and Inuariance. Academic Press, New York,
1990.
[70] MIT Total Data Quality Management Program. web.mit.edu/tdqm/www/index.shtml,
2003.
171] L. C. Molina, L. Belanche, and A. Nebot. Feature Selection Algorithms: A Survey and
Experimental Evaluation. ln Proc. of the 2002 IEEE Intl. Conf. on Data M'ining,2OO2.
1721 F. Olken and D. Rotem. Random Sampling from Databases-A Survey. Stati,stics I
Comput'ing, 5(l) :25-42, March 1995.
[73] J. Osborne. Notes on the Use of Data TYansformations. Practical Assessment, Research
€i Eualuation, 28(6), 2002.
l74l C R. Palmer and C. Faloutsos. Density biased sampling: An improved method for data
mining and clusterin g. AC M SI G M O D Record, 29(2) :82-92, 2000.
[75] F. J. Provost, D. Jensen, and T. Oates. Efficient Progressive Sampling. In Proc. of the
5th IntI. Conf. on Knowled,ge Discouery and, Data Mining, pages 23-32, 1999.
[76] T. C. Redman. Data Qualitg: The Field Gui.d,e.Digital Press, January 2001.
l77l P H. A. Sneath and R. R. Sokal. Numerical Taronomy. FYeeman,San Francisco, 1971.
[78] R. Srikant and R. Agrawal. Mining Quantitative Association Rules in Large Relational
Tables. In Proc. of 1996 ACM-SIGMOD Intl. Conf. on Management of Data, pages
1-12, Montreal, Quebec, Canada, August 1996.
[79] S. S. Stevens. On the Theory of Scales of Measurement. Science, 103(2684):677-680,
June 1946.
[80] S. S. Stevens. Measurement. In G. M. Maranell, editor, Scali,ng: A Sourceboolefor
Behauioral Scientists, pages 22-4L Aldine Publishing Co., Chicago, 1974.
f81] P. Suppes, D. Krantz, R. D. Luce, and A. Tversky. Found,at'ionsof Measurements:
Volume 2: Geometrical, Threshold, and Probab'ilistic Representations. Academic Press,
New York, 1989.
f82] H. Toivonen. Sampling Large Databases for Association Rules. In VLDB96, pages
L34-I45. Morgan Kaufman, September 1996.
[83] J. W. Tukey. On the Comparative Anatomy of TYansformations. Annals of Mathematical
Stat'istics, 28(3):602-632, September 1957.
f84] R. Y. Wang, M. Ziad, Y. W. Lee, and Y. R. Wang. Data Quali,ty. The Kluwer In-
ternational Series on Advances in Database Systems, Volume 23. Kluwer Academic
Publishers, January 2001.
[85] M. J. Zaki, S. Parthasarathy, W. Li, and M. Ogihara. Evaluation of Sampling for Data
Mining of Association Rules. Technical Report TR617, Renssela,erPolytechnic Institute,
1996.

2.6 Exercises
1. In the initial example of Chapter 2, the statistician says, ((Yes,fields 2 and 3
are basically the same." Can you tell from the three lines of sample data that
are shown why she says that?
 2.6 Exercises 89

2. Classify the following attributes as binary, discrete, or continuous. Also classifu

them as qualitative (nominal or ordinal) or quantitative (interval or ratio).
Some cases may have more than one interpretation, so briefly indicate your
reasoning if you think there may be some ambiguity.
Example: Age in years. Answer: Discrete, quantitative, ratio

(u) Time in terms of AM or PM.

(b) Brightness as measured by a light meter.
(") Brightness as measured by people's judgments.
(d) Angles as measured in degrees between 0 and 360.
(e) Bronze, Silver, and Gold medals as awarded at the Olympics.
(f) Height above sea level.
(s)Number of patients in a hospital.
(h) ISBN numbers for books. (Look up the format on the Web.)
(i) Ability to pass light in terms of the following values: opaque, translucent'
transparent.

0) Military rank.
(k) Distance from the center of campus.
(l) Density of a substance in grams per cubic centimeter.
(-) Coat check number. (When you attend an event, you can often give your
coat to someone who, in turn, gives you a number that you can use to
claim your coat when you leave.)

3. You are approached by the marketing director of a local company, who believes
that he has devised a foolproof way to measure customer satisfaction' He
explains his scheme as follows: "It's so simple that I can't believe that no one
has thought of it before. I just keep track of the number of customer complaints
for each product. I read in a data mining book that counts are ratio attributes,
and so, my measure of product satisfaction must be a ratio attribute. But
when I rated the products based on my new customer satisfaction measure and
showed them to my boss, he told me that I had overlooked the obvious, and
that my measure was worthless. I think that he was just mad because our best-
selling product had the worst satisfaction since it had the most complaints.
Could you help me set him straight?"

(a) Who is right, the marketing director or his boss? If you answered, his
boss, what would you do to fix the meaaure of satisfaction?
(b) What can you say about the attribute type of the original product satis-
faction attribute?
90 Chapter 2 Data

I
A few months later, you are again approached by the same marketing director
as in Exercise 3. This time, he has devised a better approach to measure the
extent to which a customer prefers one product over other, similar products. He
explains, "When we develop new products, we typically create several variations
and evaluate which one customers prefer. Our standard procedure is to give
our test subjects all ofthe product variations at one time and then ask them to
rank the product variations in order of preference. However, our test subjects
are very indecisive, especially when there are more than two products. As a
result, testing takes forever. I suggested that we perform the comparisons in
pairs and then use these comparisons to get the rankings. Thus, if we have
three product variations, we have the customers compare variations I and 2,
then 2 and 3, and finally 3 and 1. Our testing time with my new procedure
is a third of what it was for the old procedure, but the employees conducting
the tests complain that they cannot come up with a consistent ranking from
the results. And my boss wants the latest product evaluations, yesterday. I
should also mention that he was the person who came up with the old product
evaluation approach. Can you help me?"

(a) Is the marketing director in trouble? Will his approach work for gener-
ating an ordinal ranking of the product variations in terms of customer
preference? Explain.
(b) Is there a way to fix the marketing director's approach? More generally,
what can you say about trying to create an ordinal measurement scale
based on pairwise comparisons?
(c) For the original product evaluation scheme, the overall rankings of each
product variation are found by computing its average over all test subjects.
Comment on whether you think that this is a reasonable approach. What
other approaches might you take?

5 . Can you think of a situation in which identification numbers would be useful

for prediction?

o . An educational psychologist wants to use association analysis to analyze test

results. The test consists of 100 questions with four possible answers each.

(a) How would you convert this data into a form suitable for association
analysis?
(b) In particular, what type of attributes would you have and how many of
them are there?

7 Which of the following quantities is likely to show more temporal autocorrela-

tion: daily rainfall or daily temperature? Why?

8. Discuss why a document-term matrix is an example of a data set that has

asymmetric discrete or asymmetric continuous features.
 2.6 Exercises 9L

9 . Many sciences rely on observation instead of (or in addition to) designed ex-
periments. Compare the data quality issues involved in observational science
with those of experimental science and data mining.

10.Discuss the difference between the precision of a measurement and the terms
single and double precision, as they are used in computer science, typically to
represent floating-point numbers that require 32 and 64 bits, respectively.

1 1 . Give at least two advantages to working with data stored in text files instead
of in a binary format.

12. Distinguish between noise and outliers. Be sure to consider the following ques-
tions.

(a) Is noise ever interesting or desirable? Outliers?

(b) Can noise objects be outliers?
(c) Are noise objects always outliers?
(d) Are outliers always noise objects?
(e) Can noise make a typical value into an unusual one, or vice versa?

13. Consider the problem of finding the K nearest neighbors of a data object. A
programmer designs Algorithm 2.2 for this task.

Algorithm 2.2 Algorithm for finding K nearest neighbors.

1: for ri : 1 to number of data objects do
2: Find the distances of the ith object to all other objects.
3: Sort these distances in decreasingorder.
(Keep track of which object is associated with each distance.)
4: return the objects associatedwith the first K distances of the sorted list
5: end for

(u) Describe the potential problems with this algorithm if there are duplicate
objects in the data set. Assume the distance function will only return a
distance of 0 for objects that are the same.
(b) How would you fix this problem?

14. The following attributes are measured for members of a herd of Asian ele-
phants: wei,ght, hei,ght, tusk length, trunk length, and ear area. Based on these
measurements, what sort of similarity mea"surefrom Section 2.4 would you use
to compare or group these elephants? Justify your answer and explain any
special circumstances.
92 Chapter 2 Data

1 5 . You are given a set of rn objects that is divided into K groups, where the ith
group is of size mi. If. the goal is to obtain a sample of size fl I ffi, what is
the difference between the following two sampling schemes? (Assume sampling
with replacement.)

(a) We randomly select n *mif m elements from each group.

(b) We randomly select n elements from the data set, without regard for the
group to which an object belongs.

16. Consider a document-term matrix, where tfii isthe frequency of the rith word
(term) in the jth document and m is the number of documents. Consider the
variable transformation that is defined by

tf'ti:tfti*nsffi, (2.18)

where dfi is the number of documents in which the i.th term appears, which
is known as the document frequency of the term. This transformation is
known as the inverse document frequency transformation.

(a) What is the effect of this transformation if a term occurs in one document?
In every document?
(b) What might be the purpose of this transformation?

17. Assume that we apply a square root transformation to a ratio attribute r to

obtain the new attribute r*. As part of your analysis, you identify an interval
(o, b) in which r* has a linear relationship to another attribute gr.

(a) What is the corresponding interval (o, b) in terms of r?

(b) Give an equation that relates y to r.

18. This exercise compares and contrasts some similarity and distance measures.
(a) For binary data, the Ll distance corresponds to the Hamming distance;
that is, the number of bits that are different between two binary vectors.
The Jaccard similarity is a measure of the similarity between two bina,ry
vectors. Compute the Hamming distance and the Jaccard similarity be-
tween the following two binary vectors.

x: 0101010001
y : 0100011000

(b) Which approach, Jaccard or Hamming distance, is more similar to the

Simple Matching Coefficient, and which approach is more similar to the
cosine measure? Explain. (Note: The Hamming mea,sure is a distance,
while the other three measures are similarities, but don't let this confuse
you.)
 2.6 Exercises 93

(c) Suppose that you are comparing how similar two organisms of different
species are in terms of the number of genes they share. Describe which
measure, Hamming or Jaccard, you think would be more appropriate for
comparing the genetic makeup of two organisms. Explain. (Assume that
each animal is represented as a binary vector, where each attribute is 1 if
a particular gene is present in the organism and 0 otherwise.)
(d) If you wanted to compare the genetic makeup of two organisms of the same
species, e.g., two human beings, would you use the Hamming distance,
the Jaccard coefficient, or a different measure of similarity or distance?
Explain. (Note that two human beings share > 99.9% of the same genes.)

1 9 . For the following vectors, x and y, calculate the indicated similarity or distance
measures.

(a) x : (1, 1, 1, 1), y : (2,2,2,2) cosine,correlation,Euclidean

(b) x : (0, 1,0, 1), y : (1,0, 1,0) cosine,correlation, Euclidean,Jaccard
( c ) x : ( 0 , - 1 , 0 , 1 ) , y : ( 1 , 0 , - 1 , 0 ) c o s i n e ,c o r r e l a t i o n E
, uclidean
( d ) x : ( 1 , 1 , 0 , 1 , 0 , 1 ) , y : ( 1 , 1 , 1 , 0 , 0 , 1 ) c o s i n e ,c o r r e l a t i o n J, a c c a r d
( e ) x : ( 2 , - 7 , 0 , 2 , 0 , - 3 ) , y : ( - 1 , 1 , - 1 , 0 , 0 , - 1 ) c o s i n ec, o r r e l a t i o n

20. Here, we further explore the cosine and correlation measures.

(a) What is the range of values that are possible for the cosine measure?
(b) If two objects have a cosine measure of 1, are they identical? Explain.
(c) What is the relationship of the cosine mea,sure to correlation, if any?
(Hint: Look at statistical measures such as mean and standard deviation
in cases where cosine and correlation are the same and different.)
(d) Figure 2.20(a) shows the relationship of the cosine measure to Euclidean
distance for 100,000 randomly generated points that have been normalized
to have an L2 length of 1. What general observation can you make about
the relationship between Euclidean distance and cosine similarity when
vectors have an L2 norm of 1?
(e) Figure 2.20(b) shows the relationship of correlation to Euclidean distance
for 100,000 randomly generated points that have been standardized to
have a mean of 0 and a standard deviation of 1. What general observa-
tion can you make about the relationship between Euclidean distance and
correlation when the vectors have been standardized to have a mean of 0
and a standard deviation of 1?
(f) Derive the mathematical relationship between cosine similarity and Eu-
clidean distance when each data object has an L2 length of 1.
(g) Derive the mathematical relationship between correlation and Euclidean
distance when each data point has been been standardized by subtracting
its mean and dividing by its standard deviation.
94 Chapter 2 Data

1.4 1.4
1.2 1.2
o o
o 1 1
q
o
o 0.8 o 0.8
c
G 0.6 (5 0.6
o o
o.4 o 0.4
l,rJ I.JJ
o.2 o.2

CosineSimilarity Correlation

(a) Relationship between Euclidean (b) Relationship between Euclidean

distance and the cosine measure. distance and correlation.

Figure
2.20.Graphs
forExercise
20.

2T, Show that the set difference metric given by

d(A, B) : size(A - B) + si,ze(B - A) (2.19)

satisfies the metric axioms given on page 70. ,4. and B are sets and A - B is
the set difference.

22. Discuss how you might map correlation values from the interval l-1,1] to the
interval [0,1]. Note that the type of transformation that you use might depend
on the application that you have in mind. Thus, consider two applications:
clustering time series and predicting the behavior of one time series given an-
other.

23. Given a similarity measure with values in the interval [0,1] describe two ways to
transform this similarity value into a dissimilarity value in the interval l0,oo].
,A Proximity is typically defined between a pair of objects.

(a) Define two ways in which you might define the proximity among a group
of objects.
(b) How might you define the distance between two sets of points in Euclidean
space?
(c) How might you define the proximity between two sets of data objects?
(Make no assumption about the data objects, except that a proximity
measure is defined between any pair of objects.)

25. You are given a set of points ,9 in Euclidean space, as well as the distance of
each point in ,S to a point x. (It does not matter if x e S.)
 2.6, Exercises 95

(a) If the goal is to find all points within a specified distance e of point y,
y * x, explain how you could use the triangle inequality and the already
calculated distances to x to potentially reduce the number of distance
calculations necessary?Hint: The triangle inequality, d(x,z) < d(x,y)*
d(y,*), can be rewritten as d(x,y) 2 d(x, z) - d(y,z).
(b) In general, how would the distance between x and y affect the number of
distance calculations?
(c) Suppose that you can find a small subset of points ,5', from the original
data set, such that every point in the data set is within a specified distance
e of at least one of the points in ^91and that you also have the pairwise
distance matrix for 51 Describe a technique that uses this information to
compute, with a minimum of distance calculations, the set of all points
within a distance of B of a specified point from the data set.

26. Show that 1 minus the Jaccard similarity is a distance measure between two data
objects, x and y, that satisfies the metric axioms given on page 70. Specifically,
d(*,y) : 1- J(x,y).

27. Show that the distance measure defined as the angle between two data vectors,
x and y, satisfies the metric axioms given on page 70. Specifically, d(*, y) :
arccos(cos(x,y)).

,9, Explain why computing the proximity between two attributes is often simpler
than computing the similarity between two objects.
ExploringData
The previous chapter addressed high-level data issues that are important in
the knowledge discovery process. This chapter provides an introduction to
data exploration, which is a preliminary investigation of the data in order
to better understand its specific characteristics. Data exploration can aid in
selecting the appropriate preprocessing and data analysis techniques. It can
even address some of the questions typically answered by data mining. For
example, patterns can sometimes be found by visually inspecting the data.
Also, some of the techniques used in data exploration, such as visualization,
can be used to understand and interpret data mining results.
This chapter covers three major topics: summary statistics, visualization,
and On-Line Analytical Processing (OLAP). Summary statistics, such as the
mean and standard deviation of a set of values, and visualization techniques,
such as histograms and scatter plots, are standard methods that are widely
employed for data exploration. OLAP, which is a more recent development,
consists of a set of techniques for exploring multidimensional arrays of values.
OlAP-related analysis functions focus on various ways to create summary
data tables from a multidimensional data array. These techniques include
aggregating data either across various dimensions or across various attribute
values. For instance, if we are given sales information reported according
to product, location, and date, OLAP techniques can be used to create a
summary that describes the sales activity at a particular location by month
and product category.
The topics covered in this chapter have considerable overlap with the area
known as Exploratory Data Analysis (EDA), which was created in the
1970s by the prominent statistician, John Tirkey. This chapter, like EDA,
places a heavy emphasis on visualization. Unlike EDA, this chapter does not
include topics such as cluster analysis or anomaly detection. There are two
98 Chapter 3 Exploring Data

reasonsfor this. First, data mining views descriptive data analysis techniques
as an end in themselves,whereasstatistics, from which EDA originated, tends
to view hypothesis-basedtesting as the final goal. Second, cluster analysis
and anomaly detection are large areas and require full chapters for an in-
depth discussion. Hence, cluster analysis is coveredin Chapters 8 and 9, while
anomaly detection is discussedin Chapter 10.

3.1- The Iris Data Set

In the following discussion, we will often refer to the Iris data set that is
available from the University of California at Irvine (UCI) Machine Learn-
ing Repository. It consists of information on 150 Iris flowers, 50 each from
one of three Iris species: Setosa, Versicolour, and Virginica. Each flower is
characterized by five attributes:

1. sepal length in centimeters

2. sepal width in centimeters

3. petal length in centimeters

4. petal width in centimeters

5. class (Setosa,Versicolour, Virginica)

The sepals of a flower are the outer structures that protect the more fragile
parts of the flower, such as the petals. In many flowers, the sepals are green,
and only the petals are colorful. For Irises, however, the sepals are also colorful.
As illustrated by the picture of a Virginica Iris in Figure 3.1, the sepals of an
Iris are larger than the petals and are drooping, while the petals are upright.

3.2 Summary Statistics

Summary statistics are quantities, such as the mean and standard deviation,
that capture various characteristics of a potentially large set of values with a
single number or a small set of numbers. Everyday examples of summary
statistics are the averagehousehold income or the fraction of collegestudents
who complete an undergraduate degree in four years. Indeed, for many people,
summary statistics are the most visible manifestation of statistics. We will
concentrate on summary statistics for the values of a single attribute, but will
provide a brief description of some multivariate summary statistics.
 3.2 Summary Statistics 99

Figure3.1. Picture
of lrisVirginica.
RobertH. Mohlenbrock@ USDA-NRCS PLANTS Database/
USDANRCS.1995.Northeast wetland guide
flora:Fieldoffice to plant
species.
Northeast
National
Technical
Center,
Chester,PA.Background removed.

This section considers only the descriptive nature of summary statistics.

However, as described in Appendix C, statistics views data as arising from an
underlying statistical processthat is characterizedby various parameters, and
some of the summary statistics discussedhere can be viewed as estimates of
statistical parameters of the underlying distribution that generated the data.

3.2.L Flequencies and the Mode

Given a set of unordered categorical values, there is not much that can be done
to further characterize the values except to compute the frequency with which
each value occurs for a particular set of data. Given a categorical attribute r,
which can take values {rt,. . . ,1ri,. .. u7r}and a set of rn objects, the frequency
of a value u; is defined as

number of objects with attribute value u;

frequency(u5): (3.1)

The mode of a categorical attribute is the value that has the highest frequency.
100 Chapter 3 Exploring Data

Example 3.1. Consider a set of students who have an attribute, class,which

can take values from the set {f reshman, sophomore, junior, senior}. Table
3.1 shows the number of students for each value of the class attribute. The
mode of the class attribute is f reshman, with a frequency of 0.33. This may
indicate dropouts due to attrition or a larger than usual freshman class.

Table sizeforstudents
3.1.Class ina hypothetical
college.
Class Size FYequency
freshman t40 0.33
sophomore 160 0.27
junior 130 0.22
senior 170 0.18

Categorical attributes often, but not always, have a small number of values,
and consequently, the mode and frequencies of these values can be interesting
and useful. Notice, though, that for the Iris data set and the class attribute,
the three types of flower all have the same frequency, and therefore, the notion
of a mode is not interesting.
For continuous data, the mode, as currently defined, is often not useful
becausea single value may not occur more than once. Nonetheless,in some
cases,the mode may indicate important information about the nature of the
values or the presenceof missing values. For example, the heights of 20 people
measuredto the nearest millimeter will typically not repeat, but if the heights
are measured to the nearest tenth of a meter, then some people may have the
same height. Also, if a unique value is used to indicate a missing value, then
this value will often show up as the mode.

3.2.2 Percentiles

For ordered data, it is more useful to consider the percentiles of a set of

values. In particular, given an ordinal or continuous attribute r and a number
p between 0 and 100, the pth percentile ro is a value of z such that pTo of the
observed values of r are less than ro. For instance, the 50th percentile is the
valte r5so/osuch that 50% of all values of r are less than r5sv. Table 3.2 shows
the percentiles for the four quantitative attributes of the Iris data set.
 3.2 SummaryStatistics 101-

Table3.2. Percentiles
forsepal
length,
sepal petallength,
width, andpetalwidth.(Allvalues
arein
centimeters.)
Percentile Sepal Length Sepal Width Petal Length Petal Width
0 4.3 2.0 1.0 0.1
10 4.8 2.5 t.4 0.2
20 5.0 2.7 r.o 0.2
30 5.2 2.8 t.7 0.4
40 D.r) 3.0 3.9 7.2
50 5.8 3.0 4.4 1.3
60 6.1 3.1 4.6 -t.l)
70 6.3 3.2 5.0 1.8
80 6.6 3.4 o.4 1.9
90 6.9 3.6 c.6 2.2
100 7.9 4.4 6.9 2.5

Example 3.2. The percentiles,ryyo,rro4o,... ,frgyyo,rno%of the integersfrom

1 t o 1 0 a r e ,i n o r d e r ,t h e f o l l o w i n g :1 . 0 ,7 . 5 , 2 . 5 , 3 . 5 , 4 . 5 , 5 . 56, . 5 ,7 . 5 , 8 . 5 ,9 . 5 ,
10.0. By tradition, min(r) : troyoand max(r) : *too%. r

3.2.3 Measures of Location: Mean and Median

For continuous data, two of the most widely used summary statistics are the
mean and median, which are measures of the locati,on of a set of values.
Considera set of nl objects and an attribute r. Let {r1,...,r^) be the
attribute values of r for these zn objects. As a concrete example, these values
might be the heights of rn children. Let {rg1,...,:xOd} representthe values
of z after they have been sorted in non-decreasingorder. Thus, ro): min(z)
and r1*1: max(r). Then, the mean and median are defined as follows:

* ^e^ a- /n- (\ r ) : I := I
m \-..
*L*' (3.2)
m,

_ ^r:^- / \ :
meoran(r/ f *,r*rl i f r n i s o d d , i . e . ,r n : 2 r * |
(3.3)
1 !.'i*,' * r 1 " + r 1 ) i f r n i s e v e n ,i ' e ' , m : 2 r
\ 2\*(r)

To summarize, the median is the middle value if there are an odd number
of values, and the average of the two middle values if the number of values
is even. Thus, for seven values, the median is 1141,while for ten values, the
median is |(r15; + rfol).
LO2 Chapter 3 Exploring Data

Although the mean is sometimesinterpreted as the middle of a set of values,

this is only correct if the values are distributed in a symmetric manner. If the
distribution of values is skewed, then the median is a better indicator of the
middle. AIso, the mean is sensitive to the presenceof outliers. For data with
outliers, the median again provides a more robust estimate of the middle of a
set of values.
To overcomeproblems with the traditional definition of a mean, the notion
of a trimmed mean is sometimes used. A percentagep between 0 and 100
is specified, the top and bottom (pl2)% of the data is thrown out, and the
mean is then calculated in the normal way. The median is a trimmed mean
with p -- L00yo,while the standard mean correspondsto p: go/o.
Example 3.3. Considerthe set of values {L,2,3,4, 5,90}. The mean of these
values is 17.5, while the median is 3.5. The trimmed mean with p : 40Tois
also 3.5. r

Example 3.4. The means, medians, and trimmed means (p : 20%) of the
four quantitative attributes of the Iris data are given in Table 3.3. The three
measuresof location have similar values except for the attribute petal length.

andmedians
Table3.3. Means forsepallength, andpetalwidth.(Allvalues
sepalwidth,petallength,
areincentimeters.)
Measure Sepal Length Sepal Width Petal Length Petal Width
mean 5.84 3.05 3.76 I 20
median 5.80 3.00 4.35 1.30
trimmed mean (20To) 5. (9 3.02 3.72 r.72

3.2.4 Measures of Spread: Range and Variance

Another set of commonly used summary statistics for continuous data are
those that measure the dispersion or spread of a set of values. Such measures
indicate if the attribute values are widely spread out or if they are relatively
concentrated around a single point such as the mean.
The simplest measure of spread is the range, which, given an attribute r
with a set of rn values {rr, . . . , r*}, is defined as

range(r) : max(r) - min(r) : r(^) - r(t). (3.4)

 3.2 SummaryStatistics 103

Table3.4.Range,
standard (std),
deviation absolute
average (AAD),
difference median
absolute
difier-
ence(MAD),andinterquartile
range (lQR)forsepal
length,
sepal petallength,
width, andpetal
width.
(Allvalues
areincentimeters.)
Measure Sepal Length Sepal Width Petal Length Petal Width
range 3.6 ,A 5.9 ,A

std 0.8 0.4 1.8 0.8

AAD 0.7 0.3 1.6 0.6
MAD 0.7 0.3 r.2 u.(
IQR 1.3 0.5 3.5 1.5

Although the range identifies the maximum spread, it can be misleading if

most of the values are concentrated in a narrow band of values, but there are
also a relatively small number of more extreme values. Hence, the variance
is preferred as a measure of spread. The variance of the (observed) values of
an attribute r is typically written as sl and is defined below. The standard
deviation, which is the square root of the variance, is written as su and has
the same units as r.

1m
variance(z)- s7: --- \-(2, - z)2 (3.5)
m,-lz-/''

The mean can be distorted by outliers, and since the variance is computed
using the mean, it is also sensitive to outliers. Indeed, the variance is particu-
larly sensitive to outliers since it uses the squared difference between the mean
and other values. As a result, more robust estimates of the spread of a set
of values are often used. Following are the definitions of three such measures:
the absolute average deviation (AAD), the median absolute deviation
(MAD), and the interquartile range(IQR). Table 3.4 shows these measures
for the Iris data set.

1ffi

AAD(z):'tl*i-nl (3.6)
m-

ft, - rl,.. ., l"- - rl))

MAD(z): med'ian( (3.7)
\/
interquartile range(r) : r1sTo- r2s% (3.8)
LO4 Chapter 3 Exploring Data

3.2.5 Multivariate Summary Statistics

Measures of location for data that consists of several attributes (multivariate
data) can be obtained by computing the mean or median separately for each
attribute. Thus, given a data set the mean of the data objects, x, is given by

(3.e)

where 4 is the mean of the i,th attribute r;.

For multivariate data, the spread of each attribute can be computed in-
dependently of the other attributes using any of the approachesdescribed in
Section 3.2.4. However, for data with continuous variables, the spread of the
data is most commonly captured. by the covariance matrix S, whose iith
entry sii is the covarianceof the i}h and jth attributes of the data. Thus, if ai
and ri are the ith and jth attributes, then

sij : covafiance(rrrt i). (3.10)

In turn, couariance(q,ri) is given by

1
r \n-
covariance(ri, r j) ., - ri)\rki - ri), (3.11)
v 2 - \ z - t l\rnt.
K:1

where rpi arrd,rkj arethe values of the ith andj'h attributes for the kth object.
Notice that covariance(r6,rt) : variance(r1). Thus, the covariancematrix has
the variances of the attributes along the diagonal.
The covariance of two attributes is a measure of the degree to which two
attributes vary together and depends on the magnitudes of the variables. A
value near 0 indicates that two attributes do not have a (linear) relationship,
but it is not possible to judge the degree of relationship between two variables
by looking only at the value of the covariance. Because the correlation of two
attributes immediately gives an indication of how strongly two attributes are
(linearly) related, correlation is preferred to covariance for data exploration.
(AIso see the discussionof correlation in Section 2.4.5.) The ijth entry of the
correlation matrix R, is the correlation between I'he ith and jth attributes
of the data. If rt arrd.rj are the i,th and jth attributes, then

,j) : **Xy-f,
ri.j : corcelntion(r6, (3.12)
 3.3 Visualization 105

where s2 and sy are the variances of r; and rjj respectively. The diagonal
entries of R are correlation(u,rt): 1, while the other entries are between
-1 and 1. It is also useful to consider correlation matrices that contain
the
pairwise correlations of objects instead of attributes.

3.2.6 Other Ways to Summarize the Data

There are, of course, other types of summary statistics. For instance, the
skewness of a set of values measures the degree to which the values are sym-
metrically distributed around the mean. There are also other characteristics
of the data that are not easy to measure quantitatively, such as whether the
distribution of values is multimodal; i.e., the data has multiple "bumps" where
most of the values are concentrated. In many cases, however, the most effec-
tive approach to understanding the more complicated or subtle aspectsof how
the values of an attribute are distributed, is to view the values graphically in
the form of a histogram. (Histograms are discussedin the next section.)

3.3 Visualization
Data visualization is the display of information in a graphic or tabular format.
Successfulvisualization requires that the data (information) be converted into
a visual format so that the characteristics of the data and the relationships
among data items or attributes can be analyzed or reported. The goal of
visualization is the interpretation of the visualized information by a person
and the formation of a mental model of the information.
In everyday life, visual techniques such as graphs and tables are often the
preferred approach used to explain the weather, the economy, and the results
of political elections. Likewise, while algorithmic or mathematical approaches
are often emphasized in most technical disciplines-data mining included-
visual techniques can play a key role in data analysis. In fact, sometimes the
use of visualization techniques in data mining is referred to as visual data
mining.

3.3.1 Motivations for Visualization

The overriding motivation for using visualization is that people can quickly
absorb large amounts of visual information and find patterns in it. Consider
Figure 3.2, which shows the Sea Surface Temperature (SST) in degreesCelsius
for July, 1982. This picture summarizes the information from approximately
250,000 numbers and is readily interpreted in a few seconds. For example, it
106 Chapter 3 Exploring Data

15

10

Temp
Longitude

(SST)forJuly,1982.
Temperature
Figure3,2. SeaSurface

is easy to see that the ocean temperature is highest at the equator and lowest
at the poles.
Another general motivation for visualization is to make use of the domain
knowledge that is "locked up in people's heads." While the use of domain
knowledgeis an important task in data mining, it is often difficult or impossible
to fully utilize such knowledgein statistical or algorithmic tools. In some cases,
an analysis can be performed using non-visual tools, and then the results
presentedvisually for evaluation by the domain expert. In other cases,having
a domain specialist examine visualizations of the data may be the best way
of finding patterns of interest since, by using domain knowledge, a person can
often quickly eliminate many uninteresting patterns and direct the focus to
the patterns that are important.

3.3.2 General Concepts

This section explores some of the general concepts related to visualization, in

particular, general approachesfor visualizing the data and its attributes. A
number of visualization techniquesare mentioned briefly and will be described
in more detail when we discuss specific approacheslater on. We assumethat
the reader is familiar with line graphs, bar charts, and scatter plots.
 3.3 Visualization 1.O7

Representation: Mapping Data to Graphical Elements

The first step in visualization is the mapping of information to a visual format;
i.e., mapping the objects, attributes, and relationships in a set of information
to visual objects, attributes, and relationships. That is, data objects, their at-
tributes, and the relationships among data objects are translated into graphical
elements such as points, lines, shapes,and colors.
Objects are usually represented in one of three ways. First, if only a
single categorical attribute of the object is being considered, then objects
are often lumped into categories based on the value of that attribute, and
these categories are displayed as an entry in a table or arr area on a screen.
(Examples shown later in this chapter are a cross-tabulation table and a bar
chart.) Second, if an object has multiple attributes, then the object can be
displayed as a row (or column) of a table or as a line on a graph. Finally,
an object is often interpreted as a point in two- or three-dimensional space,
where graphically, the point might be representedby a geometric figure, such
as a circle. cross. or box.
For attributes, the representation depends on the type of attribute, i.e.,
nominal, ordinal, or continuous (interval or ratio). Ordinal and continuous
attributes can be mapped to continuous, ordered graphical features such as
location along the x:) A) or z axes; intensity; color; or size (diameter, width,
height, etc.). For categorical attributes, each category can be mapped to
a distinct position, color, shape, orientation, embellishment, or column in
a table. However, for nominal attributes, whose values are unordered, care
should be taken when using graphical features, such as color and position that
have an inherent ordering associated with their values. In other words, the
graphical elements used to represent the ordinal values often have an order,
but ordinal values do not.
The representation of relationships via graphical elements occurs either
explicitly or implicitly. For graph data, the standard graph representation-
a set of nodes with links between the nodes-is normally used. If the nodes
(data objects) or links (relationships) have attributes or characteristicsoftheir
own, then this is representedgraphically. To illustrate, if the nodes are cities
and the links are highways, then the diameter of the nodes might represent
population, while the width of the links might represent the volume of traffic.
In most cases, though, mapping objects and attributes to graphical el-
ements implicitly maps the relationships in the data to relationships among
graphical elements. To illustrate, if the data object representsa physical object
that has a location, such as a city, then the relative positions of the graphical
objects correspondingto the data objects tend to naturally preservethe actual
1-08 Chapter 3 Exploring Data

relative positions of the objects. Likewise, if there are two or three continuous
attributes that are taken as the coordinates ofthe data points, then the result-
ing plot often gives considerable insight into the relationships of the attributes
and the data points becausedata points that are visually close to each other
have similar values for their attributes.
In general, it is difficult to ensure that a mapping of objects and attributes
will result in the relationships being mapped to easily observed relationships
among graphical elements. Indeed, this is one of the most challenging aspects
of visualization. In any given set of data, there are many implicit relationships,
and hence, a key challenge of visualization is to choose a technique that makes
the relationships of interest easily observable.

Arrangement

As discussedearlier, the proper choice of visual representation of objects and

attributes is essentialfor good visualization. The arrangement of items within
the visual display is also crucial. We illustrate this with two examples.
Example 3.5. This example illustrates the importance of rearranging a table
of data. In Table 3.5, which shows nine objects with six binary attributes,
there is no clear relationship between objects and attributes, at least at first
glance. If the rows and columns of this table are permuted, however, as shown
in Table 3.6, then it is clear that there are really only two types of objects in
the table-one that has all ones for the first three attributes and one that has
only ones for the last three attributes. I

(rows)
Table3.5, A tableof nineobjects with (rows)
Table3.6.Atableofnineobjects withsix
sixbinary (columns).
attributes binaryattributes permuted
(columns) sothatthe
oftherowsandcolumns
relationships areclear.
r23456 613254
1 010110 4 11100
2 101001 2 11100 0
J 010110 6 11100 0
4 101001 8 11100 0
5 010110 r
00011 1
o 101001 3 00011 I
,7
010110 9 00011 1
8 101001 1 00011 1
q
010110 7 00011 I
 3.3 Visualization 109

Example 3.6. Consider Figure 3.3(a), which showsa visualization of a graph.

If the connected components of the graph are separated, as in Figure 3.3(b),
then the relationships between nodes and graphs become much simpler to
understand.

(a) Original view of a graph.

V
(b) Uncoupled view ofconnected components
of the graph.

Figure ofa graph.

3.3.Twovisualizations

Selection

Another key concept in visualization is selection, which is the elimination

or the de-emphasisof certain objects and attributes. Specifically, while data
objects that only have a few dimensions can often be mapped to a two- or
three-dimensional graphical representation in a straightforward way, there is
no completely satisfactory and general approach to represent data with many
attributes. Likewise, if there are many data objects, then visualizing all the
objects can result in a display that is too crowded. If there are many attributes
and many objects, then the situation is even more challenging.
The most common approach to handling many attributes is to choose a
subset of attributes-usually two-for display. If the dimensionality is not too
high, a matrix of bivariate (two-attribute) plots can be constructed for simul-
taneous viewing. (Figure 3.16 shows a matrix of scatter plots for the pairs
of attributes of the Iris data set.) Alternatively, a visualization program can
automatically show a seriesof two-dimensional plots, in which the sequenceis
user directed or based on some predefined strategy. The hope is that visualiz-
ing a collection of two-dimensional plots will provide a more complete view of
the data.
L10 Chapter 3 Exploring Data

The technique of selecting a pair (or small number) of attributes is a type of

dimensionality reduction, and there are many more sophisticated dimension-
ality reduction techniques that can be employed, e.g., principal components
analysis (PCA). Consult Appendices A (Linear Algebra) and B (Dimension-
ality Reduction) for more information.
When the number of data points is high, e.9., more than a few hundred,
or the range of the data is large, it is difficult to display enough information
if
about each object. Some data points can obscure other data points, or a
data object may not occupy enough pixels to allow its features to be clearly
displayed. For example, the shape of an object cannot be used to encode a
characteristic of that object if there is only one pixel available to display it. In
these situations, it is useful to be able to eliminate some of the objects, either
by zooming in on a particular region of the data or by taking a sample of the
data points.

3.3.3 Techniques

Visualization techniques are often specialized to the type of data being ana-
lyzed. Indeed, new visualization techniques and approaches, as well as special-
ized variations ofexisting approaches,are being continuously created, typically
in responseto new kinds of data and visualization tasks.
Despite this specialization and the ad hoc nature of visualization, there are
some generic ways to classify visualization techniques. One such classification
is based on the number of attributes involved (1,2,3, or many) or whether the
data has some special characteristic, such as a hierarchical or graph structure.
Visualization methods can also be classifiedaccording to the type of attributes
involved. Yet another classification is based on the type of application: scien-
tific, statistical, or information visualization. The following discussionwill use
three categories: visualization of a small number of attributes, visualization of
data with spatial andf or temporal attributes, and visualization of data with
many attributes.
Most of the visualization techniques discussed here can be found in a wide
variety of mathematical and statistical packages, some of which are freely
available. There are also a number of data sets that are freely available on the
World Wide Web. Readers are encouraged to try these visualization techniques
as they proceed through the following sections.
 3.3 Visualization 111

Visualizing Small Numbers of Attributes

This section examines techniques for visualizing data with respect to a small
number of attributes. Some of these techniques, such as histograms, give
insight into the distribution of the observedvalues for a single attribute. Other
techniques, such as scatter plots, are intended to display the relationships
between the values of two attributes.

Stem and Leaf Plots Stem and leaf plots can be used to provide insight
into the distribution of one-dimensional integer or continuous data. (We will
assumeinteger data initially, and then explain how stem and leaf plots can be
applied to continuous data.) For the simplest type of stem and leaf plot, we
split the values into groups, where each group contains those values that are
the same except for the last digit. Each group becomesa stem, while the last
digits of a group are the leaves. Hence, if the values are two-digit integers,
e.g., 35, 36, 42, and 51, then the stems will be the high-order digits, e.g., 3,
4, and 5, while the leaves are the low-order digits, e.g., 1, 2, 5, and 6. By
plotting the stems vertically and leaves horizontally, we can provide a visual
representation of the distribution of the data.
Example 3.7. The set of integers shown in Figure 3.4 is the sepal length in
centimeters (multiplied by 10 to make the values integers) taken from the Iris
data set. For convenience,the values have also been sorted.
The stem and leaf plot for this data is shown in Figure 3.5. Each number in
Figure 3.4 is first put into one of the vertical groups-4, 5, 6, or 7-according
to its ten's digit. Its last digit is then placed to the right of the colon. Often,
especially if the amount of data is larger, it is desirable to split the stems.
For example, instead of placing all values whose ten's digit is 4 in the same
"bucket," the stem 4 is repeated twice; all values 40-44 are put in the bucket
corresponding to the first stem and all values 45-49 are put in the bucket
corresponding to the second stem. This approach is shown in the stem and
leaf plot of Figure 3.6. Other variations are also possible. I

Histograms Stem and leaf plots are a type of histogram, a plot that dis-
plays the distribution of values for attributes by dividing the possible values
into bins and showing the number of objects that fall into each bin. For cate-
gorical data, each value is a bin. If this results in too many values, then values
are combined in some way. For continuous attributes, the range of values is di-
vided into bins-typically, but not necessarily, of equal width-and the values
in each bin are counted.
LL2 Chapter 3 Exploring Data

43 44 44 44 45 46 46 46 46 47 47 48 48 48 48 48 49 49 49 49 49 49 50
50 50 50 50 50 50 50 50 50 51 51 51 51 51 51 51 51 51 52 52 52 52 53
54 54 54 54 54 54 55 55 55 55 55 55 55 56 56 56 56 56 56 57 57 57 57
57 57 57 57 58 58 58 58 58 58 58 59 59 59 60 60 60 60 60 60 61 61 61
61 61 61 62 62 62 62 63 63 63 63 63 63 63 63 63 64 64 64 64 64 64 64
65 65 65 65 65 66 66 67 67 67 67 67 67 67 67 68 68 68 69 69 69 69 70
7t 72 72 72 73 74 76 77 77 77 77 79
Figure datafromthelrisdataset.
length
3.4.Sepal

34444566667788888999999
0000000000Lt t l1tfl L222234444445555555666 66677 7 7 77 7 78888888999
A 000000 11 1 1 t1222233333333344444445555566777 77777 8889999
0t22234677779

Figure andleafplotforthesepal
3.5.Stem length
fromthelrisdataset.

3444
4: 566667788888999999
tr. 000000000011 1 111 1 1 122223444++4
tr. 555555566666677 777 7778888888999
A 000000 1 1 11 t t22223333333334444444
A 5 5 5 556 677 777 7 77 8889999
7. 0122234
677779

Figure3.6. Stemandleafplotforthesepallength
fromthelrisdatasetwhenbuckets
conesponding
todigitsaresplit.

Once the counts are available for each bin, a bar plot is constructed such
that each bin is representedby one bar and the area of each bar is proportional
to the number of values (objects) that fall into the corresponding range. If all
intervals are of equal width, then all bars are the same width and the height
of a bar is proportional to the number of values in the corresponding bin.
Exarnple 3.8. Figure 3.7 shows histograms (with 10 bins) for sepal length,
sepal width, petal length, and petal width. Since the shape of a histogram
can depend on the number of bins, histograms for the same data, but with 20
bins, are shown in Figure 3.8. I

There are variations of the histogram plot. A relative (frequency) his-

togram replaces the count by the relative frequency. However, this is just a
 3.3 Visualization 1-1-3

(a) Sepal length. (b) Sepal width. (c) Petal length (d) Petal width.

Figure
3.7.Histograms (10bins).
offourlrisattributes

) 05 I 15 2
Petwnh

(a) Sepal length. (b) Sepal width. (c) Petal length. (d) Petal width.

Figure3.8. Histograms (20bins).

offourlrisattributes

change in scale of the g axis, and the shape of the histogram does not change.
Another common variation, especially for unordered categorical data, is the
Pareto histogram, which is the same as a normal histogram except that the
categories are sorted by count so that the count is decreasing from left to right.

Two-Dimensional Histograms Two-dimensional histograms are also pos-

sible. Each attribute is divided into intervals and the two sets of intervals define
two-dimensional rectangles of values.
Example 3.9. Figure 3.9 shows a two-dimensional histogram of petal length
and petal width. Becauseeach attribute is split into three bins, there are nine
rectangular two-dimensional bins. The height of each rectangular bar indicates
the number of objects (flowers in this case) that fall into each bin. Most of
the flowers fall into only three of the bins-those along the diagonal. It is not
possible to see this by looking at the one-dimensionaldistributions. r
LL4 Chapter 3 Exploring Data

Figure3.9.Two-dimensional of petallength
histogram andwidthinthelrisdataset.

While two-dimensional histograms can be used to discover interesting facts

about how the values of two attributes co-occur, they are visually more com-
plicated. For instance, it is easy to imagine a situation in which some of the
columns are hidden bv others.

Box Plots Box plots are another method for showing the distribution of the
values of a single numerical attribute. Figure 3.10 shows a labeled box plot for
sepal length. The lower and upper ends of the box indicate the 25th and 75th
percentiles,respectively,while the line inside the box indicates the value of the
50th percentile. The top and bottom lines of the tails indicate the 10'h and
90th percentiles. Outliers are shown by "+" marks. Box plots are relatively
compact, and thus, many of them can be shown on the same plot. Simplified
versions of the box plot, which take less space,can also be used.
Example 3.1-0. The box plots for the first four attributes of the Iris data
set are shown in Figure 3.11. Box plots can also be used to compare how
attributes vary between different classesof objects, as shown in Figure 3.12.
T

Pie Chart A pie chart is similar to a histogram, but is typically used with
categorical attributes that have a relatively small number of values. Instead of
showing the relative frequency of different values with the area or height of a
bar, as in a histogram, a pie chart usesthe relative area of a circle to indicate
relative frequency. Although pie charts are common in popular articles, they
 3.3 Visualization 115

<- Outlier

<- 90thpercentile

g
o
o
<- 75rhpercentile E
o
<-- sOthpercentile o
<_ 25rhpercentile d

<- 1Othpercentile

Sepal Length Sepal Width Petal Length PetalWidth

Figure3.10. Description
of 3,11.Boxplotforlrisattributes.
Figure
boxplotforsepal
length,

=
LJ
l-

widh M L€ngth &lwidh Spalkngih Sepalwdlh told Length Peblwidh S6pal bngth S€palwdh bd bngth &l Wfr

(a) Setosa. (b) Versicolour. (c) Virginica.

3.12.Boxplots
Figure bylrisspecies.
ofattributes

are used less frequently in technical publications because the size of relative
areas can be hard to judge. Histograms are preferred for technical work.
Example 3.11. Figure 3.13 displays a pie chart that shows the distribution
of Iris speciesin the Iris data set. In this case, all three flower types have the
same freouencv. r

Percentile Plots and Empirical Cumulative Distribution Functions

A type of diagram that shows the distribution of the data more quantitatively
is the plot of an empirical cumulative distribution function. While this type of
plot may sound complicated, the concept is straightforward. For each value of
a statistical distribution, a cumulative distribution function (CDF) shows
116 Chapter 3 Exploring Data

Versicolour

ofthetypesof lrisflowers.
Figure3.13.Distribution

the probability that a point is lessthan that value. For each observedvalue, an
empirical cumulative distribution function (ECDF) shows the fraction
of points that are less than this value. Since the number of points is finite, the
empirical cumulative distribution function is a step function.
Example 3.12. Figure 3.14 shows the ECDFs of the Iris attributes. The
percentiles of an attribute provide similar information. Figure 3.15 shows the
percentile plots of the four continuous attributes of the Iris data set from
Table 3.2. The reader should compare these figures with the histograms given
in Figures 3.7 and 3.8. r

Scatter Plots Most people are familiar with scatter plots to some extent,
and they were used in Section 2.4.5 to illustrate linear correlation. Each data
object is plotted as a point in the plane using the values of the two attributes
as r and y coordinates. It is assumed that the attributes are either integer- or
real-valued.
Example 3.13. Figure 3.16 shows a scatter plot for each pair of attributes
of the Iris data set. The different species of Iris are indicated by different
markers. The arrangement of the scatter plots of pairs of attributes in this
type of tabular format, which is known as a scatter plot matrix, provides
an organized way to examine a number of scatter plots simultaneously. I
 3.3 Visualization LL7

(a) Sepal Length. (b) Sepal Width.

(c) Petal Length. (d) Petal Width.

Figure
3.14.Empirical
CDFsoffourlrisattributes.

0204608010

Figure plots
3.15.Percentile forsepal
length,
sepal petal
width, andpetal
length, width.
118 Chapter 3 Exploring Data

c\l

o
=
(d
q)
o-
X
XXX XX
X XXX Xg X
XffiX XXW
XX X XXg

o
-c v)
ct
c') (o
E
s9 at)
(U
o
o -c
o-
ol .9
(t
o
o_
x
(D
x x
x x (6
XX
x
X
X s <)
p x oo x <t>
X -c
X) f x o
x
x xg '= x
g XX
x (5
x
x
x x -E
X X o-
o <rt
o
Grt
o
+ ol =
@
ctt
o o u-
o goo ce9
t\ -C
o)
c
q)
@_
(d
o-
o
gr@

@F-@rOrQ\t|r)(f)lr|c\l (oSOltQc\l tQ-lOO

\f(r)ol NO
q$uagledes qprm ledes q$ue; le1ed qptrr le1ed
 3.3 Visualization 119

There are two main uses for scatter plots. First, they graphically show
the relationship between two attributes. In Section 2.4.5, we saw how scatter
plots could be used to judge the degreeof linear correlation. (SeeFigure 2.17.)
Scatter plots can also be used to detect non-linear relationships, either directly
or by using a scatter plot of the transformed attributes.
Second,when classlabels are available, they can be used to investigate the
degree to which two attributes separate the classes. If is possible to draw a
line (or a more complicated curve) that divides the plane defined by the two
attributes into separate regions that contain mostly objects of one class, then
it is possible to construct an accurate classifier based on the specified pair of
attributes. If not, then more attributes or more sophisticated methods are
neededto build a classifier. In Figure 3.16, many of the pairs of attributes (for
example, petal width and petal length) provide a moderate separation of the
Iris species.
Example 3.14. There are two separate approachesfor displaying three at-
tributes of a data set with a scatter plot. First, each object can be displayed
according to the values of three, instead of two attributes. F igure 3.17 shows a
three-dimensionalscatter plot for three attributes in the Iris data set. Second,
one of the attributes can be associatedwith some characteristic of the marker,
such as its size, color, or shape. Figure 3.18 shows a plot of three attributes
of the Iris data set, where one of the attributes, sepal width, is mapped to the
size of the marker. r

Extending Two- and Three-Dimensional Plots As illustrated by Fig-

ure 3.18, two- or three-dimensional plots can be extended to represent a few
additional attributes. For example, scatter plots can display up to three ad-
ditional attributes using color or shading, size, and shape, allowing five or six
dimensions to be represented. There is a need for caution, however. As the
complexity of a visual representation of the data increases,it becomesharder
for the intended audience to interpret the information. There is no benefit in
packing six dimensions' worth of information into a two- or three-dimensional
plot, if doing so makes it impossible to understand.

Visualizing Spatio-temporal Data

Data often has spatial or temporal attributes. For instance, the data may
consist of a set of observationson a spatial grid, such as observationsof pres-
sure on the surface of the Earth or the modeled temperature at various grid
points in the simulation of a physical object. These observations can also be
LzO Chapter 3 Exploring Data

c 1.5
c
o
J1

o
o
@ 0.5
*:;'H*:
"**-i,S#".*f
0

plotolsepal
3.17.Three-dimensional
Figure scatter width,
sepal andpetalwidth.
length,

4
Petal Length

plotofpetallength
Figure3.18,Scatter petalwidth,
versus withthesizeofthemarker
indicating
sepal
width.
 3.3 Visualization LzL

plotof SSTforDecember
Figure3.19,Contour 1998.

made at various points in time. In addition, data may have only a temporal
component, such as time seriesdata that gives the daily prices of stocks.

Contour Plots For some three-dimensional data, two attributes specify a

position in a plane, while the third has a continuous value, such as temper-
ature or elevation. A useful visualization for such data is a contour plot,
which breaks the plane into separate regions where the values of the third
attribute (temperature, elevation) are roughly the same. A common example
of a contour plot is a contour map that shows the elevation of land locations.
Example 3.15. Figure 3.19 shows a contour plot of the average sea surface
temperature (SST) for December 1998. The land is arbitrarily set to have a
temperature of 0oC. In many contour maps, such as that of Figure 3.19, the
contour lines that separate two regions are labeled with the value used to
separate the regions. For clarity, some of these labels have been deleted. r

Surface Plots Like contour plots, surface plots use two attributes for'the
r and 3l coordinates. The third attribute is used to indicate the height above
I22 Chapter 3 Exploring Data

(a) Set of 12 points. (b) Overall density function----surface

plot.

of a setof 12points.
Figure3,20.Density

the plane defined by the first two attributes. While such graphs can be useful,
they require that a value of the third attribute be defined for all combinations
of values for the first two attributes, at least over some range. AIso, if the
surface is too irregular, then it can be difficult to see all the information,
unless the plot is viewed interactively. Thus, surface plots are often used to
describe mathematical functions or physical surfaces that vary in a relatively
smooth manner.
Example 3.16. Figure 3.20 shows a surface plot of the density around a set
of 12 points. This example is further discussedin Section 9.3.3. r

Vector Field Plots In some data, a characteristic may have both a mag-
nitude and a direction associatedwith it. For example, consider the flow of a
substanceor the change of density with location. In these situations, it can be
useful to have a plot that displays both direction and magnitude. This type
of plot is known as a vector plot.
Example 3.17. Figure 3.2I shows a contour plot of the density of the two
smaller density peaks from Figure 3.20(b), annotated with the density gradient
vectors.

Lower-Dimensional Slices Consider a spatio-temporal data set that records

sornequantity, such as temperature or pressure)at various locations over time.
Such a data set has four dimensions and cannot be easily displayed by the types
 3.3 Visualization L23

\\\lllltrr\\\lll.t

ttlllrttlll\

plotofthegradient
Figure3.21,Vector (change) forthebottom
indensity peaksof Figure
twodensity
3.20.

of plots that we have described so far. However, separate "slices" of the data
can be displayed by showing a set of plots, one for each month. By examining
the change in a particular area from one month to another, it is possible to
notice changesthat occur, including those that may be due to seasonalfactors.
Example 3.18. The underlying data set for this example consists of the av-
erage monthly sea level pressure (SLP) from 1982 to 1999 on a 2.5o by 2.5'
Iatitude-longitude grid. The twelve monthly plots of pressurefor one year are
shown in Figure 3.22. In this example, we are interested in slices for a par-
ticular month in the year 1982. More generally, we can consider slices bf the
data along any arbitrary dimension.

Animation Another approach to dealing with slicesof data, whether or not

time is involved, is to employ animation. The idea is to display successive
two-dimensional slices of the data. The human visual system is well suited to
detecting visual changes and can often notice changes that might be difficult
to detect in another manner. Despite the visual appeal of animation, a set of
still plots, such as those of Figure 3.22, can be more useful since this type of
visualization allows the information to be studied in arbitrary order and for
arbitrary amounts of time.
L24 Chapter 3 Exploring Data

January February

April May

July August

plotsof sealevelpressure
Figure3,22.Monthly overthe12months
of 1982.

3.3.4 Visualizing Higher-Dimensional Data

This section considersvisualization techniques that can display more than the
handful of dimensionsthat can be observedwith the techniquesjust discussed.
However, even these techniques are somewhat limited in that they only show
some aspects of the data.

Matrices An image can be regarded as a rectangular array of pixels, where

each pixel is characterized by its color and brightness. A data matrix is a
rectangular array of values. Thus, a data matrix can be visualized as an image
by associating each entry of the data matrix with a pixel in the image. The
brightness or color of the pixel is determined by the value of the corresponding
entry of the matrix.
 W,8;, 3.3 Visualization I25

Ssrca V€cidour Virginl€

Figure3.23. Plotof thelrisdatamatrix

where Figure3.24.Plotofthelrisconelation
matrix.
columnshavebeenstandardized tohavea mean
of0 andstandard
deviationof1.

There are some important practical considerations when visualizing a data

matrix. If class labels are known, then it is useful to reorder the data matrix
so that all objects of a class are together. This makes it easier, for example, to
detect if all objects in a class have similar attribute values for some attributes.
If different attributes have different ranges, then the attributes are ofben stan-
dardized to have a mean of zero and a standard deviation of 1. This prevents
the attribute with the largest magnitude values from visually dominating the
plot.

Example 3.19. Figure 3.23 shows the standardized data matrix for the Iris
data set. The first 50 rows represent Iris flowers ofthe species Setosa, the next
50 Versicolour, and the last 50 Virginica. The Setosa flowers have petal width
and length well below the average, while the Versicolour flowers have petal
width and length around average. The Virginica flowers have petal width and
length above average. l
It can also be useful to look for structure in the plot of a proximity matrix
for a set of data objects. Again, it is useful to sort the rows and columns of
the similarity matrix (when class labels are known) so that all the objects of a
class are together. This allows a visual evaluation of the cohesivenessof each
class and its separation from other classes.
Example 3.20. Figure 3.24 shows the correlation matrix for the Iris data
set. Again, the rows and columns are organized so that all the flowers of a
particular species are together. The flowers in each group are most similar
L26 Chapter 3 Exploring Data

to each other, but Versicolour and Virginica are more similar to one another
than to Setosa. r
If class labels are not known, various techniques (matrix reordering and
seriation) can be used to rearrange the rows and columns of the similarity
matrix so that groups of highly similar objects and attributes are together
and can be visually identified. Effectively, this is a simple kind of clustering.
See Section 8.5.3 for a discussion of how a proximity matrix can be used to
investigate the cluster structure of data.

Parallel Coordinates Parallel coordinates have one coordinate axis for

each attribute, but the different axes are parallel to one other instead of per-
pendicular, as is traditional. Furthermore, an object is represented as a line
instead of as a point. Specifically, the value of each attribute of an object is
mapped to a point on the coordinate axis associated with that attribute, and
these points are then connected to form the line that representsthe object.
It might be feared that this would yield quite a mess. However, in many
cases,objects tend to fall into a small number of groups, where the points in
each group have similar values for their attributes. If so, and if the number of
data objects is not too large, then the resulting parallel coordinates plot can
reveal interesting patterns.
Example 3.2L. Figure 3.25 shows a parallel coordinates plot of the four nu-
merical attributes of the Iris data set. The lines representing objects of differ-
ent classesare distinguished by their shading and the use of three different line
styles-solid, dotted, and dashed. The parallel coordinates plot showsthat the
classesare reasonably well separated for petal width and petal length, but less
well separated for sepal length and sepal width. Figure 3.25 is another parallel
coordinates plot of the same data, but with a different ordering of the axes. r
One of the drawbacks of parallel coordinates is that the detection of pat-
terns in such a plot may depend on the order. For instance, if lines cross a
Iot, the picture can become confusing, and thus, it can be desirable to order
the coordinate axes to obtain sequencesof axes with less crossover. Compare
Figure 3.26, where sepal width (the attribute that is most mixed) is at the left
of the figure, to Figure 3.25, where this attribute is in the middle.

Star Coordinates and Chernoff Faces

Another approach to displaying multidimensional data is to encode objects

as glyphs or icons-symbols that impart information non-verbally. More
 3.3 Visualization L27

I
o
o
E
c
o
o
o
5

SepalWidth Petal Length PetalWidth

Figure3.25.A parallel plotofthefourlrisattributes.

coordinates

I
o
o
E
c
c)
o
o
f
(!

SepalWidth SepalLength PetalLength PetalWidth

Figure3.26. A parallel plotof thefourlrisattributes

coordinates withtheattributes to
reordered
emphasizesimilarities groups
anddissimilarities
of
1-28 Chapter 3 Exploring Data

specifically, each attribute of an object is mapped to a particular feature of a

glyph, so that the value of the attribute determines the exact nature of the
feature. Thus, at a glance, we can distinguish how two objects differ.
Star coordinates are one example of this approach. This technique uses
one axis for each attribute. These axes all radiate from a center point, like the
spokes of a wheel, and are evenly spaced. Typically, all the attribute values
are mapped to the range [0,1].
An object is mapped onto this star-shaped set of axes using the following
process: Each attribute value of the object is converted to a fraction that
represents its distance between the minimum and maximum values of the
attribute. This fraction is mapped to a point on the axis corresponding to
this attribute. Each point is connected with a line segment to the point on
the axis preceding or following its own axis; this forms a polygon. The size
and shape of this polygon gives a visual description of the attribute values of
the object. For ease of interpretation, a separate set of axes is used for each
object. In other words, each object is mapped to a polygon. An example of a
star coordinates plot of flower 150 is given in Figure 3.27(a).
It is also possible to map the values of features to those of more familiar
objects, such as faces. This technique is named Chernoff faces for its creator,
Herman Chernoff. In this technique, each attribute is associatedwith a specific
feature of a face, and the attribute value is used to determine the way that
the facial feature is expressed. Thus, the shape of the face may become more
elongated as the value of the correspondingdata feature increases.An example
of a Chernoff face for flower 150 is given in Figure 3.27(b).
The program that we used to make this face mapped the features to the
four features listed below. Other features of the face, such as width between
the eyes and length of the mouth, are given default values.

Data Feature Facial Feature

sepal length size of face
sepal width forehead/jaw relative arc length
petal length shape of forehead
petal width shape ofjaw

Example 3.22. A more extensiveillustration of thesetwo approachesto view-

ing multidimensional data is provided by Figures 3.28 and 3.29, which shows
the star and face plots, respectively, of 15 flowers from the Iris data set. The
first 5 flowers are of species Setosa, the second 5 are Versicolour, and the last
5 are Virginica. r
 3.3 Visualization L2g

:-;4"r6$h
fp+;fr;o[, oro

(a) Star graph of Iris 150. (b) Chernoff face of Iris 150.

Figure3.27.Starcoordinates
graphandChernoff
faceofthe150thflowerofthelrisdataset.

il\
4J A 04 I\
AJ
2 34 5

,4\
=vz /4\
€--F
,,f\
<vv <t>
v/ \,/
51 52 53 54 55

,,,,\
Y-T-7
\l/
+\t/ ,,'1t\
\ |
\Z
//
,-T\
\ l,/
\'/
-/t\
s\t/
J0l 102 '103 104 105

Figute
3.28.Plotof15lrisflowers
using
starcoordinates.

/Cit /-F\
vv

'I
2345

l^\ Z\
/oro\ Iore\ /oro\ rD /oro\
\_-/ \:-/ \_-/ \:./
51 52 53 54 55

/\
/eto) f\:/,1 /oro\ /oro) /oro)
\, tl \, \7
\7 '104
101 102 103 105

Figure3.29.A plotof 15lrisflowers faces.

usingChernoff
130 Chapter 3 Exploring Data

Despite the visual appeal of these sorts of diagrams, they do not scalewell,
and thus, they are of limited use for many data mining problems. Nonetheless,
they may still be of use as a means to quickly compare small sets of objects
that have been selectedby other techniques.

3.3.5 Do's and Don'ts

To conclude this section on visualization, we provide a short list of visualiza-

tion do's and don'ts. While these guidelines incorporate a lot of visualization
wisdom, they should not be followed blindly. As always, guidelines are no
substitute for thoughtful consideration of the problem at hand.

ACCENT Principles The follov,ring are the ACCEN? principles for ef-
fective graphical display put forth by D. A. Burn (as adapted by Michael
Friendlv):

Apprehension Ability to correctly perceive relations among variables. Does

the graph maximize apprehensionof the relations among variables?

Clarity Ability to visually distinguish all the elements of a graph. Are the
most important elements or relations visually most prominent?

Consistency Ability to interpret a graph based on similarity to previous

graphs. Are the elements, symbol shapes, and colors consistent with
their use in previous graphs?

Efficiency Ability to portray a possibly complex relation in as simple a way

as possible. Are the elements of the graph economically used? Is the
graph easy to interpret?

Necessity The need for the graph, and the graphical elements. Is the graph
a more useful way to represent the data than alternatives (table, text)?
Are all the graph elements necessary to convey the relations?

Tbuthfulness Ability to determine the true value representedby any graph-

ical element by its magnitude relative to the implicit or explicit scale.
Are the graph elements accurately positioned and scaled?

T\rfte's Guidelines Edward R. Tufte has also enumerated the following

principles for graphical excellence;
 3.4 OLAP and Multidimensional Data Analysis 131

Graphical excellenceis the well-designedpresentation of interesting data-

a matter of substance,of statistics, and of design.

Graphical excellenceconsists of complex ideas communicated with clar-

ity, precision, and efficiency.

Graphical excellence is that which gives to the viewer the greatest num-
ber of ideas in the shortest time with the least ink in the smallest space.

Graphical excellence is nearly always multivariate.

And graphical excellencerequires telling the truth about the data.

3.4 OLAP and Multidimensional Data Analysis

In this section, we investigate the techniques and insights that come from
viewing data sets as multidimensional arrays. A number of database sys-
tems support such a viewpoint, most notably, On-Line Analytical Processing
(OLAP) systems. Indeed, some of the terminology and capabilities of OLAP
systems have made their way into spreadsheet programs that are used by mil-
Iions of people. OLAP systems also have a strong focus on the interactive
analysis of data and typically provide extensive capabilities for visualizing the
data and generating summary statistics. For these reasons,our approach to
multidimensional data analysis will be based on the terminology and concepts
common to OLAP systems.

3.4.L Representing Iris Data as a Multidimensional Array

Most data sets can be representedas a table, where each row is an object and
each column is an attribute. In many cases,it is also possible to view the data
as a multidimensional array. We illustrate this approach by representing the
Iris data set as a multidimensional array.
Table 3.7 was created by discretizing the petal length and petal width
attributes to have values of low, med'ium, and hi,gh and then counting the
number of flowers from the Iris data set that have particular combinations
of petal width, petal length, and species type. (For petal width, the cat-
egories low, med'ium, and hi,gh correspond to the intervals [0, 0.75), [0.75,
1.75), [7.75, oo), respectively. For petal length, the categories low, med'ium,
and hi,gh correspond to the intervals 10, 2.5), 12.5,5), [5, m), respectively.)
I32 Chapter 3 Exploring Data

offlowers
Table3.7. Number a particular
having of petalwidth,petallength,
combination andspecies
rype.
Petal Length Petal Width SpeciesT Count
Iov,r low Setosa 40
low medium Setosa 2
medium low Setosa 2
medium medium Versicolour +L)

medium high Versicolour 3

medium high Virginica 3
high medium Versicolour 2
a
high medium Virginica
high high Versicolour 2
high high Virginica 44

Petal
widrh

Virginica
Versicolour
Setosa
high

medium

low

Petal -c E =
.9 .= o
width -c E
o
E
Figure3.30.A multidimensional
datarepresentation
forthelrisdataset.
 3.4 OLAP and Multidimensional Data Analysis 133

Table3.8.Crosstabulationofflowers
accord- offlowers
Table3.9,Cross-tabulation accord-
ingto petallengthandwidthforflowers
ofthe ingto petallengthandwidthforflowers
ofthe
Setosa species. Versicolour
species.
width width
Iow medium high
+l
low 000
b0 medium 0433
et

j high 022

Table3.10. Cross-tabulation
of flowersac-
cording
to petal
lengthandwidthforflowers
of
theVirginica
species.
width

Empty combinations-those combinations that do not correspond to at least

one flower-are not shown.
The data can be organized as a multidimensional array with three dimen-
sions cortesponding to petal width, petal length, and species type, as illus-
trated in Figure 3.30. For clarity, slices of this array are shown as a set of
three two-dimensional tables, one for each species-see Tables 3.8, 3.9, and
3.10. The information contained in both Table 3.7 and Figure 3.30 is the
same. However, in the multidimensional representation shown in Figure 3.30
(and Tables 3.8, 3.9, and 3.10), the values of the attributes-petal width, petal
length, and speciestype-are array indices.
What is important are the insights can be gained by looking at data from a
multidimensional viewpoint. Tables 3.8, 3.9, and 3.10 show that each species
of Iris is characterized by a different combination of values of petal length
and width. Setosa flowers have low width and length, Versicolour flowers have
medium width and length, and Virginica flowers have high width and length.

3.4.2 Multidimensional Data: The General Case

The previous section gave a specific example of using a multidimensional ap-

proach to represent and analyze a familiar data set. Here we describe the
general approach in more detail.
L34 Chapter 3 Exploring Data

The starting point is usually a tabular representation of the data, such

as that of Table 3.7, which is called a fact table. Two steps are necessary
in order to represent data as a multidimensional arrayi identification of the
dimensions and identification of an attribute that is the focus of the analy-
sis. The dimensions are categorical attributes or, as in the previous example,
continuous attributes that have been converted to categorical attributes. The
values of an attribute serve as indices into the array for the dimension corre-
sponding to the attribute, and the number of attribute values is the size of
that dimension. In the previous example, each attribute had three possible
values, and thus, each dimension was of size three and could be indexed by
threevalues. Thisproduceda3 x 3 x 3multidimensional array.
Each combination of attribute values (one value for each difierent attribute)
defines a cell of the multidimensional array. To illustrate using the previous
example, if petal length : lou),petal width : mediutr\ and species: Setosa,
a specific cell containing the value 2 is identified. That is, there are only two
flowers in the data set that have the specified attribute values. Notice that
each row (object) of the data set in Table 3.7 corresponds to a cell in the
multidimensional array.
The contents of each cell represents the value of a target quantity (target
variable or attribute) that we are interested in analyzing. In the Iris example,
the target quantity is the nurnber of flowers whose petal width and length
fall within certain limits. The target attribute is quantitative because a key
goal of multidimensional data analysis is to look aggregatequantities, such as
totals or averages.
The following summarizes the procedure for creating a multidimensional
data representation from a data set representedin tabular form. First, identify
the categorical attributes to be used as the dimensions and a quantitative
attribute to be used as the target of the analysis. Each row (object) in the
table is mapped to a cell of the multidimensional array. The indices of the cell
are specified by the values of the attributes that were selectedas dimensions,
while the value of the cell is the value of the target attribute. Cells not defined
by the data are assumedto have a value of 0.
Example 3.23. To further illustrate the ideas just discussed,we present a
more traditional example involving the sale of products.The fact table for this
example is given by Table 3.11. The dimensions of the multidimensional rep-
resentation are the product ID, locati,on,and date attributes, while the target
attribute is the reaenue. Figure 3.31 shows the multidimensional representa-
tion of this data set. This larger and more complicated data set will be used
to illustrate additional concepts of multidimensional data analysis. r
 3.4 OLAP and Multidimensional Data Analysis 135

3.4.3 Analyzing Multidimensional Data

In this section, we describe different multidimensional analysis techniques. In

particular, we discussthe creation of data cubes, and related operations, such
as slicing, dicing, dimensionality reduction, roll-up, and drill down.

Data Cubes: Computing Aggregate Quantities

A key motivation for taking a multidimensional viewpoint of data is the im-
portance of aggregating data in various ways. In the sales example, we might
wish to find the total sales revenue for a specific year and a specific product.
Or we might wish to see the yearly sales revenue for each location across all
products. Computing aggregate totals involves fixing specific values for some
of the attributes that are being used as dimensions and then summing over
all possible values for the attributes that make up the remaining dimensions.
There are other types of aggregatequantities that are also of interest, but for
simplicity, this discussionwill use totals (sums).
Table 3.12 shows the result of summing over all locations for various com-
binations of date and product. For simplicity, assume that all the dates are
within one year. Ifthere are 365 days in a year and 1000 products, then Table
3.12 has 365,000 entries (totals), one for each product-data pair. We could
also specify the store location and date and sum over products, or specify the
location and product and sum over all dates.
Table 3.13 shows the marginal totals of Table 3.12. These totals are the
result of further summing over either dates or products. In Table 3.13, the
total sales revenue due to product 1, which is obtained by summing across
row 1 (over all dates), is $370,000. The total sales revenue on January 1,
2004, which is obtained by summing down column 1 (over all products), is
$527,362. The total salesrevenue,which is obtained by summing over all rows
and columns (all times and products) is $227,352,127.All of these totals are
for all locations becausethe entries of Table 3.13 include all locations.
A key point of this example is that there are a number of different totals
(aggregates)that can be computed for a multidimensional array, depending on
how many attributes we sum over. Assume that there are n dimensions and
that the ith dimension (attribute) has si possible values. There are n different
ways to sum only over a single attribute. If we sum over dimension j, then we
obtain s1 x ... * sj-1 * tj+t * ... * s' totals, one for each possiblecombination
of attribute values of the n- l other attributes (dimensions). The totals that
result from summing over one attribute form a multidimensional array of n-I
dimensions and there are n such arrays of totals. In the sales example, there
136 Chapter 3 Exploring Data

Table
3.11,Sales ofproducts
revenue (indollars)
forvarious
locations
andtimes.
Product ID Location Date Revenue

:: ii
1 Minneapolis Oct. 18,2004 $250
1 Chicago Oct. 18,2004 $79

i p*i. Oct. 18, 2004 301

::
27 Minneapolis Oct. 18,2004 $2,321
27 Chicago Oct. 18, 2004 $3,278

n Paris Oct. 18,2004 $1,325

::

v{"'
ProductlD
Figure
3.31.Multidimensional
datarepresentation
forsales
data.
 3.4 OLAP and MultidimensionalData Analvsis L37

Table thatresult
3.12.Totals fromsumming timeandproduct.
fora fixed
overalllocations
date
JanI.2O04 Jan2,2004

tr

3.12withmarginaltotals.
3.13.Table
Table
date
Jan 1. 2004 Jan2,2004 Dec 31. 2004 | total

27 $3.800.020
tr

27,362 ,r27

are three sets of totals that result from summing over only one dimension and
each set of totals can be displayed as a two-dimensional table.
If we sum over two dimensions (perhaps starting with one of the arrays
of totals obtained by summing over one dimension), then we will obtain a
multidimensional array of totals with rz - 2 dimensions. There will be (!)
g
distinct anays of such totals. For the sales examples, there will be () :
arays of totals that result from summing over location and product, Iocation
and time, or product and time. In general, summing over ,k dimensions yields
([) arrays of totals, each with dimension n - k.
A multidimensional representation of the data, together with all possible
totals (aggregates),is known as a data cube. Despite the name, the size of
each dimension-the number of attribute values-does not need to be equal.
AIso, a data cube may have either more or fewer than three dimensions. More
importantly, a data cube is a generalization of what is known in statistical
terminology as a cross-tabulation. If marginal totals were added, Tables
3.8, 3.9, or 3.10 would be typical examples of cross tabulations.
138 Chapter 3 Exploring Data

Dimensionality Reduction and Pivoting

The aggregation described in the last section can be viewed as a form of

dimensionality reduction. Specifically, the jth dimension is eliminated by
summing over it. Conceptually, this collapseseach "column" of cells in the jth
dimension into a single cell. For both the sales and Iris examples, aggregating
over one dimension reduces the dimensionality of the data from 3 to 2. If si
is the number of possible values of the 7'h dimension, the number of cells is
reduced by a factor of sr. Exercise 17 on page 143 asks the reader to explore
the differencebetween this type of dimensionality reduction and that of PCA.
Pivoting refers to aggregating over all dimensions except two. The result
is a two-dimensional cross tabulation with the two specified dimensions as the
only remaining dimensions. Table 3.13 is an example of pivoting on date and
product.

Slicing and Dicing

These two colorful names refer to rather straightforward operations. Slicing is

selecting a group of cells from the entire multidimensional array by specifying
a specific value for one or more dimensions. Tables 3.8, 3.9, and 3.10 are
three slices from the Iris set that were obtained by specifying three separate
values for the speciesdimension. Dicing involves selecting a subset of cells by
specifying a range of attribute values. This is equivalent to defining a subarray
from the complete array. In practice, both operations can also be accompanied
by aggregation over some dimensions.

Roll-Up and Drill-Down

In Chapter 2, attribute values were regarded as being "atomic" in some sense.

However, this is not always the case. In particular, each date has a number
of properties associatedwith it such as the year, month, and week. The data
can also be identified as belonging to a particular businessquarter) or if the
application relates to education, a school quarter or semester. A location
also has various properties: continent, country, state (province, etc.), and
city. Products can also be divided into various categories, such as clothing,
electronics, and furniture.
Often these categories can be organized as a hierarchical tree or lattice.
For instance) years consist of months or weeks, both of which consist of days.
Locations can be divided into nations, which contain states (or other units
of local government), which in turn contain cities. Likewise, any category
 3.5 Bibliographic Notes 1-39

of products can be further subdivided. For example, the product category,

furniture, can be subdivided into the subcategories,chairs, tables, sofas, etc.
This hierarchical structure gives rise to the roll-up and drill-down opera-
tions. To illustrate, starting with the original sales data, which is a multidi-
mensional array with entries for each date, we can aggregate (roll up) the
sales across all the dates in a month. Conversely, given a representation of the
data where the time dimension is broken into months, we might want to split
the monthly sales totals (drill down) into daily sales totals. Of course, this
requires that the underlying sales data be available at a daily granularity.
Thus, roll-up and drill-down operations are related to aggregation. No'
tice, however, that they differ from the aggregation operations discusseduntil
now in that they aggregate cells within a dimension, not across the entire
dimension.

3.4.4 Final Comments on Multidimensional Data Analysis

Multidimensional data analysis, in the senseimplied by OLAP and related sys-

tems, consistsof viewing the data as a multidimensional array and aggregating
data in order to better analyze the structure of the data. For the Iris data,
the differences in petal width and length are clearly shown by such an anal-
ysis. The analysis of business data, such as sales data, can also reveal many
interesting patterns, such as profitable (or unprofitable) stores or products.
As mentioned, there are various types of database systems that support
the analysis of multidimensional data. Some of these systems are based on
relational databases and are known as ROLAP systems. More specialized
database systems that specifically employ a multidimensional data represen-
tation as their fundamental data model have also been designed. Such systems
are known as MOLAP systems. In addition to these types of systems,statisti-
cal databases (SDBs) have been developed to store and analyze various types
of statistical data, e.g., census and public health data, that are collected by
governmentsor other large organizations. Referencesto OLAP and SDBs are
provided in the bibliographic notes.

3.5 Bibliographic Notes

Summary statistics are discussed in detail in most introductory statistics
books, such as 192]. Referencesfor exploratory data analysis are the classic
text by Tirkey [104] and the book by Velleman and Hoaglin [105].
The basic visualization techniques are readily available, being an integral
part of most spreadsheets(Microsoft EXCEL [95]), statistics programs (sAS
1,4O Chapter 3 Exploring Data

[99], SPSS [102], R [96], and S-PLUS [98]), and mathematics software (MAT-
LAB [94] and Mathematica [93]). Most of the graphics in this chapter were
generated using MATLAB. The statistics package R is freely available as an
open source software packagefrom the R project.
The literature on visualization is extensive, covering many fields and many
decades. One of the classicsof the field is the book by Tufte [103]. The book
by Spence [tOt], which strongly influenced the visualization portion of this
chapter, is a useful referencefor information visualization-both principles and
techniques. This book also provides a thorough discussion of many dynamic
visualization techniques that were not covered in this chapter. Two other
books on visualization that may also be of interest are those by Card et al.
[87] and Fayyad et al. [S9].
Finally, there is a great deal of information available about data visualiza-
tion on the World Wide Web. Since Web sites come and go frequently, the best
strategy is a search using "information visualization," "data visualization," or
"statistical graphics." However, we do want to single out for attention "The
Gallery of Data Visualization," by Friendly [90]. The ACCENT Principles for
effective graphical display as stated in this chapter can be found there, or as
originally presented in the article by Burn [86].
There are a variety of graphical techniques that can be used to explore
whether the distribution of the data is Gaussian or some other specified dis-
tribution. Also, there are plots that display whether the observed values are
statistically significant in some sense. We have not covered any of these tech-
niques here and refer the reader to the previously mentioned statistical and
mathematical packages.
Multidimensional analysis has been around in a variety of forms for some
time. One of the original papers was a white paper by Codd [88], the father
of relational databases. The data cube was introduced by Gray et al. [91],
who described various operations for creating and manipulating data cubes
within a relational database framework. A comparison of statistical databases
and OLAP is given by Shoshani [100]. Specific information on OLAP can
be found in documentation from database vendors and many popular books.
Many database textbooks also have general discussionsof OLAP, often in the
context of data warehousing. For example, see the text by Ramakrishnan and
Gehrke [97].

Bibliography
[86] D. A. Burn. Designing Effective Statistical Graphs. In C. R. Rao, editor, Hand,bookof
Stati,stics 9. Elsevier/North-Holland, Amsterdam, The Netherlands, September 1993.
 3.6 Exercises L4L

,in Informat'ion
[87] S. K. Card, J. D. MacKinlay, and B. Shneiderman, editors. Read,ings
Visualization: Using Vision to Thznlc. Morgan Kaufmann Publishers, San Francisco,
CA, January 1999.
[88] E. F. Codd, S. B. Codd, and C. T. Smalley. Providing OLAP (On-line Analytical
Processing) to User- Analysts: An IT Mandate. White Paper, E.F. Codd and Associates,
1993.
f89] U. M. Fayyad, G. G. Grinstein, and A. Wierse, editors. Information V'isualization'in
Data Mining and, Knowled,ge Discouery. Morgan Kaufmann Publishers, San FYancisco,
CA, September 2001.
[90] M. F]iendly. Gallery of Data Visualization. http://www.math.yorku.ca/SCS/Gallery/,
2005.
[91] J. Gray, S. Chaudhuri, A. Bosworth, A. Layman, D. Reichart, M. Venkatrao, F. Pellow,
and H. Pirahesh. Data Cube: A Relational Aggregation Operator Generalizing Group-
By, Cross-Tab, and Sub-Totals. Journal Data Mining and Knouledge Discouerg, l(I):
29-53, 1997.
f92] B. W. Lindgren. Stat'istical Theory. CRC Press, January 1993.
[93] Mathematica 5.1. Wolfram Research, Inc. http://www.wolfram.comf ,2005.
[94] MATLAB 7.0. The MathWorks, Inc. http://www.mathworks.com, 2005.
[95] Microsoft Excel 2003. Microsoft, Inc. http://www.microsoft.comf ,2003.
[96] R: A language and environment for statistical computing and graphics. The R Project
for Statistical Computing. http: / /www.r-project.org/, 2005.
[97] R Ramakrishnan and J. Gehrke. Database Management Systems. McGraw-Hill, 3rd
edition, August 2002.
198] S-PLUS. Insightful Corporation. http: //www.insightful.com, 2005.
[99] SAS: Statistical Analysis System. SAS Institute Inc. http:f f www.sas.com/, 2005.
[100] A. Shoshani. OLAP and statistical databases: similarities and differences. In Proc.
of the Siateenth ACM SIGACT-SIGMOD-SIGART Symp. on Princ,i,ples of Database
Sgstems, pages 185-196. ACM Press, 1997.
[101] R. Spence. Inforrnation Visual'izati,on ACM Press, New York, December 2000.
[102] SPSS: Statistical Package for the Social Sciences.SPSS, lnc. http://www.spss.com/,
2005.
f103] E. R. Tufte. The Visual Di.splag of Quantitatiue Informat'ion. Graphics Press, Cheshire,
CT, March 1986.
[104] J. W. T\rkey. Exploratory data analys,is. Addison-Wesley, 1977.
[105] P. Velleman and D. Hoaglin. The ABC's of EDA: Applications, Basics, and Computing
of Exploratorg Data Analysis. Duxbury, 1981.

3.6 Exercises
1. Obtain one of the data sets available at the UCI Machine Learning Repository
and apply as many of the different visualization techniques described in the
chapter as possible. The bibliographic notes and book Web site provide pointers
to visualization software.
L42 Chapter 3 Exploring Data

2. Identify at least two advantages and two disadvantages ofusing color to visually
represent information.

3. What are the arrangement issues that arise with respect to three-dimensional
plots?

A Discuss the advantages and disadvantages of using sampling to reduce the num-
ber of data objects that need to be displayed. Would simple random sampling
(without replacement) be a good approach to sampling? Why or why not?
E
d. Describe how you would create visualizations to display information that de-
scribes the following types of systems.

(a) Computer networks. Be sure to include both the static aspects of the
network, such as connectivity, and the dynamic aspects, such as traffic.
(b) The distribution of specific plant and animal species around the world for
a specific moment in time.
(c) The use of computer resources, such as processor time, main memory, and
disk, for a set of benchmark database programs.
(d) The change in occupation of workers in a particular country over the last
thirty years. Assume that you have yearly information about each person
that also includes gender and level of education.

Be sure to address the following issues:

o Representation. How will you map objects, attributes, and relation-

ships to visual elements?
o Arrangement. Are there any special considerations that need to be
taken into account with respect to how visual elements are displayed? Spe-
cific examples might be the choice of viewpoint, the use of transparency,
or the separation of certain groups of objects.
o Selection. How will you handle a larqe number of attributes and data
objects?

6 . Describe one advantage and one disadvantage of a stem and leaf plot with
respect to a standard histogram.

7. How might you address the problem that a histogram depends on the number
and location of the bins?

8 . Describe how a box plot can give information about whether the value of an
attribute is symmetrically distributed. What can you say about the symmetry
of the distributions of the attributes shown in Fieure 3.11?

9. Compare sepal length, sepal width, petal length, and petal width, using Figure
3.r2.
 3.6 Exercises L43

10. Comment on the use of a box plot to explore a data set with four attributes:
age, weight, height, and income.

1 1 . Give a possible explanation as to why most of the values of petal length and
width fall in the buckets along the diagonal in Figure 3.9.

1 2 . Use Figures 3.14 and 3.15 to identify a characteristic shared by the petal width
and petal length attributes.

13. Simple line plots, such as that displayed in Figure 2.L2 on page 56, which
shows two time series, can be used to effectively display high-dimensional data.
For example, in Figure 2.72 iI is easy to tell that the frequencies of the two
time series are different. What characteristic of time series allows the effective
visualization of high-dimensional data?

r4. Describe the typesof situations that produce sparse or dense data cubes. Illus-
trate with examples other than those used in the book.

1 5 . How might you extend the notion of multidimensional data analysis so that the
target variable is a qualitative variable? In other words, what sorts of summary
statistics or data visualizations would be of interest?

16. Construct a data cube from Table 3.14. Is this a dense or sparse data cube? If
it is sparse, identify the cells that empty.

3.14.Fact
Table table 16.
forExercise
Product ID Location ID Number Sold
I I 10
1 3 6
2 1 5
2 2 22

17. Discuss the differences between dimensionality reduction based on aggregation

and dimensionality reduction based on techniques such as PCA and SVD.
Classification:
BasicConcepts,
DecisionTrees,and
ModelEvaluation
Classification,which is the task of assigningobjects to one of severalpredefined
categories,is a pervasiveproblem that encompassesmany diverse applications.
Examples include detecting spam email messages based upon the message
header and content, categorizing cells as malignant or benign based upon the
results of MRI scans, and classifying galaxies based upon their shapes (see
Figure 4.1).

(a) A spiral galaxy (b) An elliptical galaxy.

ofgalaxies.
Figure4.1.Classification arefromtheNASA
Theimages website.
L46 Chapter 4 Classification

Input Output
Classilication
onr.'l1,,"
set l---) model
---) Class label

Figure4.2. Classification
asthetaskof mapping setx intoitsclasslabelg.
aninputattribute

This chapter introduces the basic conceptsof classification,describessome

of the key issuessuch as model overfitting, and presents methods for evaluating
and comparing the performance of a classification technique. While it focuses
mainly on a technique known as decisiontree induction, most of the discussion
in this chapter is also applicable to other classification techniques, many of
which are covered in Chapter 5.

4.L Preliminaries
The input data for a classification task is a collection of records. Each record,
also known as an instance or example, is characterizedby a tuple (*,A), where
x is the attribute set and gris a special attribute, designated as the class label
(also known as category or target attribute). Table 4.1 showsa sample data set
used for classifying vertebrates into one of the following categories: mammal,
bird, fish, reptile, or amphibian. The attribute set includes properties of a
vertebrate such as its body temperature, skin cover, method of reproduction,
ability to fly, and ability to live in water. Although the attributes presented
in Table 4.L are mostly discrete, the attribute set can also contain continuous
features. The class label, on the other hand, must be a discrete attribute.
This is a key characteristic that distinguishes classification from regression,
a predictive modeling task in which g is a continuous attribute. Regression
techniques are covered in Appendix D.
Definition 4.1 (Classification). Classification is the task of learning a tar-
get function / that maps each attribute set x to one of the predefined class
Iabels g.
The target function is also known informally as a classification model.
A classification model is useful for the following purposes.

Descriptive Modeling A classification model can serve as an explanatory

tool to distinguish between objects of different classes.For example, it would
be useful-for both biologists and others-to have a descriptive model that
 4 . r Preliminaries 147

Table
4.1.Thevertebrate
dataset.
Name lJody Skin Gives Aquatic Aerral Has H be Ulass
Temperature Cover Birth Creature Creature Legs nates Label
human warm-blooded hair yes no no yes no mammal
python cold-blooded scales no no no no yes reptile
salmon cold-blooded scales no yes no no no fish
whale warm-blooded hair yes yes no no no mammal
frog cold-blooded none no seml no yes yes amphibian
komodo cold-blooded scales no no no yes no reptile
dragon
bat warm-blooded hair yes no yes yes yes mammal
plgeon warm-blooded feathers no no yes yes no bird
cat warm-blooded fur yes no no yes no mammal
leopard cold-blooded scales yes yes no no no fish
shark
turtle cold-blooded scales no semt no yes no reptile
penguln warm-blooded feathers no seml no yes no bird
porcuptne warm-blooded quills yes no no yes yes mammal
eel cold-blooded scales no yes no no no fish
salamander cold-blooded none no semr no yes yes amphibian

summarizes the data shown in Table 4.1 and explains what features define a
vertebrate as a mammal, reptile, bird, fish, or amphibian.

Predictive Modeling A classification model can also be used to predict

the class iabel of unknown records. As shown in Figure 4.2, a classification
model can be treated as a black box that automatically assignsa class label
when presented with the attribute set of an unknown record. Supposewe are
given the following characteristics of a creature known as a gila monster:

Name lJody Dlfln Gives Aquatic Aerial Has Hiber- Ulass

Iemperature Cover Birth Creature Creature Legs nates Label
gila monster cold-blooded scales no no no yes yes

We can use a classification model built from the data set shown in Table 4.1
to determine the class to which the creature belongs.

Classification techniques are most suited for predicting or describing data

sets with binary or nominal categories. They are less effective for ordinal
categories (e.g., to classify a person as a member of high-, medium-, or low-
income group) because they do not consider the implicit order among the
categories. Other forms of relationships, such as the subclass-superclassre-
lationships among categories (e.g., humans and apes are primates, which in
1,48 Chapter 4 Classification

turn, is a subclassof mammals) are also ignored. The remainder of this chapter
focusesonly on binary or nominal class labels.

4.2 General Approach to Solving a Classification

Problem
A classification technique (or classifier) is a systematic approach to building
classification models from an input data set. Examples include decision tree
classifiers, rule-based classifiers, neural networks, support vector machines,
and naive Bayes classifiers. Each technique employs a learning algorithm
to identify a model that best fits the relationship between the attribute set and
class label of the input data. The model generated by a learning algorithm
should both fit the input data well and correctly predict the class labels of
records it has never seen before. Therefore, a key objective of the learning
algorithm is to build models with good generalization capability; i.e., models
that accurately predict the class labels of previously unknown records.
Figure 4.3 shows a general approach for solving classification problems.
First, a training set consisting of records whose class labels are known must

t ;t'' I
ruooel
I I
K",':
Figure4.3. General
approach
forbuilding model.
a classification
 4.2 General Approach to Solving a Classification Problem L4g

Table4.2. Confusion
matrix problem.
fora 2-class
Predicted Class
C l a s s: 7 Ulass: U
Actual Class : I T
.t 77 JIO
Class U l a s s: 0 ./ 01 .Ioo

be provided. The training set is used to build a classification model, which is

subsequently applied to the test set, which consists of records with unknown
class labels.
Evaluation of the performance of a classification model is based on the
counts of test records correctly and incorrectly predicted by the model. These
counts are tabulated in a table known as a confusion matrix. TabIe 4.2
depicts the confusion matrix for a binary classification problem. Each entry
f4 in this table denotes the number of records from class e predicted to be
of class 7. For instance, /s1 is the number of records from class 0 incorrectly
predicted as class 1. Based on the entries in the confusion matrix, the total
number of correct predictions made by the model ir ("ftr + /oo) and the total
number of incorrect predictions ir (/ro + /or).
Although a confusion matrix provides the information neededto determine
how well a classification model performs, summarizing this information with
a single number would make it more convenient to compare the performance
of different models. This can be done using a performance metric such as
accuracy, which is defined as follows:

Number of correct oredictions fn* foo

Accuracy: (4.1)
Total number of predictions fr l- fn * ,for* ,foo'

Equivalently, the performance of a model can be expressed terms of its

error rate, which is given by the following equation:

Number of wrong predictions .trr -T

n__^_-_^r^ Jrc JO7
H:rrnr rof6
(4.2)
Total number of predictions f:-r*frc*,for*.foo

Most classification algorithms seek models that attain the highest accuracy, or
equivalently, the lowest error rate when applied to the test set. We will revisit
the topic of model evaluation in Section 4.5.
1-50 Chapter 4 Classification

4.3 Decision Tree Induction

This section introduces a decision tree classifier,which is a simple yet widely
used classification technique.

4.3.L How a Decision TYee Works

To illustrate how classification with a decision tree works, consider a simpler

version of the vertebrate classification problem described in the previous sec-
tion. Instead of classifying the vertebrates into five distinct groups of species,
we assign them to two categories: mammals and non-mammals.
Supposea new speciesis discoveredby scientists. How can we tell whether
it is a mammal or a non-mammal? One approach is to pose a seriesof questions
about the characteristics of the species. The first question we may ask is
whether the speciesis cold- or warm-blooded. If it is cold-blooded, then it is
definitely not a mammal. Otherwise, it is either a bird or a mammal. In the
latter case,we need to ask a follow-up question: Do the femalesof the species
give birth to their young? Those that do give birth are definitely mammals,
while those that do not are likely to be non-mammals (with the exception of
egg-laying mammals such as the platypus and spiny anteater).
The previous example illustrates how we can solve a classification problem
by asking a series of carefully crafted questions about the attributes of the
test record. Each time we receive an answer? a follow-up question is asked
until we reach a conclusion about the class label of the record. The seriesof
questions and their possible answerscan be organized in the form of a decision
tree, which is a hierarchical structure consisting of nodes and directed edges.
Figure 4.4 showsthe decision tree for the mammal classificationproblem. The
tree has three types of nodes:

o A root node that has no incoming edges and zero or more outgoing
edges.

o Internal nodes, each of which has exactly one incoming edge and two
or more outgoing edges.

o Leaf or terminal nodes, each of which has exactly one incoming edge
and no outgoing edges.

In a decision tree, each leaf node is assigned a class label. The non-
terminal nodes, which include the root and other internal nodes, contain
attribute test conditions to separate records that have different characteris-
tics. For example, the root node shown in Figure 4.4 uses the attribute Body
 4.3 Decision TheeInduction 151

Figure4.4. A decision
treeforthemammal problem.
classification

Temperature to separate warm-blooded from cold-blooded vertebrates. Since

all cold-blooded vertebrates are non-mammals, a leaf node labeled Non-mamma]s
is created as the right child of the root node. If the vertebrate is warm-blooded,
a subsequent attribute, Gives Birth, is used to distinguish mammals from
other warm-blooded creatures, which are mostly birds.
Classifying a test record is straightforward once a decision tree has been
constructed. Starting from the root node, we apply the test condition to the
record and follow the appropriate branch based on the outcome of the test.
This will lead us either to another internal node, for which a new test condition
is applied, or to a leaf node. The class label associatedwith the leaf node is
then assignedto the record. As an illustration, Figure 4.5 traces the path in
the decision tree that is used to predict the class label of a flamingo. The path
terminates at a leaf node labeled Non-rna-nmal-s.

4.3.2 How to Build a Decision Tbee

In principle, there are exponentially many decision trees that can be con-
structed from a given set of attributes. While some of the trees are more accu-
rate than others, finding the optimal tree is computationally infeasiblebecause
of the exponential size of the search space. Nevertheless,efficient algorithms
have been developed to induce a reasonably accurate, albeit suboptimal, de-
cision tree in a reasonable amount of time. These algorithms usually employ
a greedy strategy that grows a decision tree by making a series of locally op-
L52 Chapter 4 Classification

Unlabeled Name Body temperature Gives Birth Class

data Flamingo Warm No

Non-
mammats

4.5.Classifying
Figure anunlabeled lines
Thedashed
vertebrate. represent ofapplying
theoutcomes
various
attribute
testconditions The
vertebrate.
ontheunlabeled iseventually
vertebrate to
assigned
lIg lrf6a-aammalclass.

timum decisions about which attribute to use for partitioning the data. One
such algorithm is fluntts algorithm, which is the basis of many existing de-
cision tree induction algorithms, including ID3, C4.5, and CART. This section
presentsa high-level discussionof Hunt's algorithm and illustrates some of its
design issues.

flunt's Algorithm

In Hunt's algorithm, a decision tree is grown in a recursive fashion by parti-

tioning the training records into successivelypurer subsets. Let Dl be the set
of training records that are associatedwith node t and g : {At,U2, . . . ,A"} be
the class labels. The following is a recursive definition of Hunt's algorithm.

Step 1: If ail the records in Dt belong to the same class y1, then t is a leaf
node labeled as y.

Step 2: If D; contains records that belong to more than one class, an at-
tribute test condition is selectedto partition the records into smaller
subsets. A child node is created for each outcome of the test condi-
tion and the records in D1 are distributed to the children based on the
outcomes. The algorithm is then recursively applied to each child node.
 4.3 DecisionTYeeInduction 1-53

a.""
"""""""J"
Yes 125K
No 100K
No 70K
Yes 120K
No 95K
No 60K
Yes 220K
No 85K
No 75K
No 90K

Figure4.6.Training
setforpredicting
borrowers onloanpayments.
whowilldefault

To illustrate how the algorithm works, consider the problem of predicting

whether a loan applicant will repay her loan obligations or become delinquent,
subsequently defaulting on her loan. A training set for this problem can be
constructed by examining the records of previous borrowers. In the example
shown in Figure 4.6, each record contains the personal information of a bor-
rower along with a class label indicating whether the borrower has defaulted
on loan payments.
The initial tree for the classification problem contains a single node with
class label Defaulted = No (see Figure a.7@)), which means that most of
the borrowers successfullyrepaid their loans. The tree, however, needs to be
refined since the root node contains records from both classes.The records are
subsequently divided into smaller subsets based on the outcomes of the Hone
Ownertest condition) as shown in Figure 4.7(b). The justification for choosing
this attribute test condition will be discussedlater. For now, we will assume
that this is the best criterion for splitting the data at this point. Hunt's
algorithm is then applied recursively to each child of the root node. From
the training set given in Figure 4.6, notice that all borrowers who are home
owners successfully repaid their loans. The left child of the root is therefore a
leaf node labeled Def aulted = No (seeFigure 4.7(b)). For the right child, we
need to continue applying the recursive step of Hunt's algorithm until all the
records belong to the same class. The trees resulting from each recursive step
are shown in Figures a.7@) and (d).
L54 Chapter 4 Classification

Figure algorithm
4.7.Hunt's forinducing trees.
decision

Hunt's algorithm will work if every combination of attribute values is

present in the training data and each combination has a unique class label.
These assumptions are too stringent for use in most practical situations. Ad-
ditional conditions are neededto handle the following cases:

1. It is possible for some of the child nodes created in Step 2 to be empty;

i.e., there are no records associatedwith these nodes. This can happen
if none of the training records have the combination of attribute values
associated with such nodes. In this case the node is declared a leaf
node with the same class label as the majority class of training records
associatedwith its parent node.

2. In Step 2, if all the records associated with D; have identical attribute

values (except for the class label), then it is not possible to split these
records any further. In this case, the node is declared a leaf node with
the same class label as the majority class of training records associated
with this node.
 4.3 Decision TYeeInduction 155

Design Issues of Decision TYee Induction

A learning algorithm for inducing decision trees must address the following
two issues.

1. How should the training records be split? Each recursive step

of the tree-growing process must select an attribute test condition to
divide the records into smaller subsets. To implement this step, the
algorithm must provide a method for specifying the test condition for
different attribute types as well as an objective measure for evaluating
the goodnessof each test condition.

2. How should the splitting procedure stop? A stopping condition is

needed to terminate the tree-growing process. A possible strategy is to
continue expanding a node until either all the records belong to the same
class or all the records have identical attribute values. Although both
conditions are sufficient to stop any decision tree induction algorithm,
other criteria can be imposed to allow the tree-growing procedure to
terminate earlier. The advantagesof early termination will be discussed
later in Section4.4.5.

4.3.3 Methods for Expressing Attribute Test Conditions

Decision tree induction algorithms must provide a method for expressing an
attribute test condition and its correspondins outcomes for different attribute
types.

Binary Attributes The test condition for a binary attribute generatestwo

potential outcomes, as shown in Figure 4.8.

Warm- Cold-
blooded blooded

Figure4,8.Testcondition
forbinaryattributes.
156 Chapter 4 Classification

Single Divorced
(a) Multiwaysplit

OR

{Manied} {Single, {Single} {Married, {Single, {Divorced}

Divorced) Divorced) Married)

(b) Binarysplit{by groupingattributevalues}

fornominal
Figure4.9.Testconditions attributes.

Nominal Attributes Since a nominal attribute can have many values, its
test condition can be expressed in two ways, as shown in Figure 4.9. For
a multiway split (Figure 4.9(a)), the number of outcomes depends on the
number of distinct values for the corresponding attribute. For example, if
an attribute such as marital status has three distinct values-single, married,
or divorced-its test condition will produce a three-way split. On the other
hand, some decisiontree algorithms, such as CART, produce only binary splits
by considering all 2k-1 - 1 ways of creating a binary partition of k attribute
values. Figure 4.9(b) illustrates three different ways of grouping the attribute
values for marital status into two subsets.

Ordinal Attributes Ordinal attributes can also produce binary or multiway

splits. Ordinal attribute values can be grouped as long as the grouping does
not violate the order property of the attribute values. Figure 4.10 illustrates
various ways of splitting training records based on the Shirt Size attribute.
The groupings shown in Figures 4.10(a) and (b) preserve the order among
the attribute values, whereas the grouping shown in Figure a.10(c) violates
this property becauseit combines the attribute values Small and Large into
 4.3 Decision Tlee Induction L57

{Small, {Large, {Small} {Medium,Large, {Small, {Medium,

Medium) ExtraLarge) ExtraLarge) Large) ExtraLarge)
(a) (b) (c)

Figure4.10.Different
waysof grouping
ordinal values.
attribute

the same partition while Mediun and Extra Large are combined into another
partition.

Continuous Attributes For continuous attributes, the test condition can

be expressedas a comparison test (A < u) or (A 2 ,) with binary outcomes,or
a range query with outcomesof the form ui I A l ut+t, fot 'i: L,... ,k. The
difference between these approachesis shown in Figure 4.11. For the binary
case, the decision tree algorithm must consider all possible split positions u,
and it selects the one that produces the best partition. For the multiway
split, the algorithm must consider all possible ranges of continuous values.
One approach is to apply the discretization strategies described in Section
2.3.6 on page 57. After discretization, a new ordinal value will be assignedto
each discretized interval. Adjacent intervals can also be aggregated into wider
ranges as long as the order property is preserved.

{10K,25K} {25K,50K} {50K,80K}

(a) (b)

Figure4.11. Testcondition
forcontinuous
attributes.
158 Chapter 4 Classification

C0:1 C0:8 C0:1 C0:0 C0:0

C 1: 3 C 1 :0 C 1: 0 C1:1 C1:1

4.12.Multiway
Figure binary
versus splits.

4.3.4 Measures for Selecting the Best Split

There are many measuresthat can be used to determine the best way to split
the records. These measuresare defined in terms of the class distribution of
the records before and after splitting.
Let p(i.lt) denote the fraction of records belonging to classi at a given node
t. We sometimes omit the referenceto node I and expressthe fraction as p,;.
In a two-class problem, the class distribution at any node can be written as
(po,pt), where Pt:7 -Po. To illustrate, considerthe test conditions shown
in Figure 4.12. The class distribution before splitting is (0.5,0.5) because
there are an equal number of records from each class. If we split the data
using the Gender attribute, then the class distributions of the child nodes are
(0.6,0.4) and (0.4,0.6), respectively. Although the classesare no longer evenly
distributed, the child nodes still contain records from both classes. Splitting
on the second attribute, Car Type, will result in purer partitions.
The measuresdeveloped for selecting the best split are often based on the
degree of impurity of the child nodes. The smaller the degree of impurity, the
more skewed the class distribution. For example, a node with class distribu-
tion (0,1) has zero impurity, whereas a node with uniform class distribution
(0.5,0.5) has the highest impurity. Examples of impurity measuresinclude

c-l
Entropy(t) : -ln?.lt)rog2pllt), (4.3)
o1_,
Gini(r) : L -|,lp11t)12, (4.4)
i,:o
Classificationerror(t) : 1 -max[p(ilt)]' (4.5)

where c is the number of classesand 0log2 0 : O ," entropy calculations.

 4.3 Decision Tlee Induction 159

Entropy

0.7

eini -.'-'
-'-).' )':-'l'l- ... --.
..' -'\.\ -'aa
o.4 -aat .a,t -aa
-'a.
-at -.a' '\,
,' .'t
\'\.\

-r" -.a't '\.

,' ,'- Misclassification
error

0.1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

o

Figure4.13.Comparison
among
theimpurity
measures
forbinary problems.
classification

Figure 4.13 comparesthe values of the impurity measuresfor binary classi-

fication problems. p refers to the fraction of records that belong to one of the
two classes.Observethat all three measuresattain their maximum value when
the class distribution is uniform (i.e., when p : 0.5). The minimum values for
the measuresare attained when all the records belong to the same class (i.e.,
when p equals 0 or 1). We next provide several examples of computing the
different impuritv measures.

Node Nr Uount Gini : r - (016)2- (616)2: 0

Class:0 0 Entropy : -(016) Iog2(0/6)- (616)log2(6/6):0
Class:1 6 Error : 1 - maxl0/6,6/6]: 0

Node Ah Count Gini : | - Gl6)2 - Fl6)2 : 0.278

CIass:0 I Entropy: -(Il6)togr(716) - (516)ro92616): 0.650
Class:1 Error : 1 - maxfl/6,516]l: 9.167

Node l/a Uount Gini : r - (Jl6)2 - (3/6)2: 9.5

Class:0 .) Entropy: -(316) Iog2(3/6)- (3/6)logr(3/6): 1
Ulass:1 3 Error : 1 - maxlSI 6, 3I 6l : 0.b
160 Chapter 4 Classification

The preceding examples, along with Figure 4.13, illustrate the consistency
among different impurity measures. Based on these calculations, node lft has
the lowest impurity value, followed by l{z and lf3. Despite their consistency,
the attribute chosen as the test condition may vary depending on the choice
of impurity measure, as will be shown in Exercise 3 on page 198.
To determine how well a test condition performs, we need to compare the
degree of impurity of the parent node (before splitting) with the degree of
impurity of the child nodes (after splitting). The larger their difference, the
better the test condition. The gain, A, is a criterion that can be used to
determine the goodnessof a split:

k
-i
A:I(pare"t) (4.6)
j:t
Yttr),

where 1(.) is the impurity measure of a given node, ly' is the total number of
records at the parent node, k is the number of attribute values, and l/(u3)
is the number of records associated with the child node, u7. Decision tree
induction algorithms often choose a test condition that maximizes the gain
A. Since /(parent) is the same for all test conditions, maximizing the gain is
equivalent to minimizing the weighted average impurity measures of the child
nodes. Finally, when entropy is used as the impurity measurein Equation 4.6,
the difference in entropy is known as the information gain, 461o.

Splitting of Binary Attributes

Consider the diagram shown in Figure 4.14. Suppose there are two ways to
split the data into smaller subsets. Before splitting, the Gini index is 0.5 since
there are an equal number of records from both classes.If attribute A is chosen
to split the data, the Gini index for node N1 is 0.4898, and for node N2, it
is 0.480. The weighted averageof the Gini index for the descendentnodes is
(71L2) x 0.4898+ (5112) x 0.480 : 0.486. Similarly, we can show that the
weighted averageof the Gini index for attribute B is 0.375. Since the subsets
for attribute B have a smaller Gini index, it is preferred over attribute A.

Splitting of Nominal Attributes

As previously noted, a nominal attribute can produce either binary or multi-

way splits, as shown in Figure 4.15. The computation of the Gini index for a
binary split is similar to that shown for determining binary attributes. For the
first binary grouping of the Car Type attribute, the Gini index of {Sports,
 4.3 Decision Tree Induction L6L

Figure
4.14.Splitting
binary
attributes.

{Sports,
6''A
--:-sl,- 6;)
{Family,
tuxut).,/ \Tmrry) nxuryl.z \<{orts}

(a)Binarysplit (b) Multiwaysplit

Figure4.15.Splitting
nominal
attributes.

Luxury) is 0.4922 and the Gini index of {Fa:rily} is 0.3750. The weighted
average Gini index for the grouping is equal to

16120x 0.4922+ 4120x 0.3750: 0.468.

Similarly, for the secondbinary grouping of {Sports} and {Fanily, Luxury},

the weighted average Gini index is 0.167. The second grouping has a lower
Gini index becauseits corresponding subsets are much purer.
L62 Chapter 4 Classification

Figure4.16.Splitting attributes.
continuous

For the multiway split, the Gini index is computed for every attribute value.
Since Gini({raniry}) : 0.375, Gini({Sports}) : 0, and Gini({Luxury}) :
0.219, the overall Gini index for the multiway split is equal to

+ 8120x 0 + 8120x 0.219: 0.163.

4120x 0.375

The multiway split has a smaller Gini index compared to both two-way splits.
This result is not surprising becausethe two-way split actually merges some
of the outcomes of a multiway split, and thus, results in less pure subsets.

Splitting of Continuous Attributes

Consider the example shown in Figure 4.16, in which the test condition Annual
Income ( u is used to split the training records for the loan default classifica-
tion problem. A brute-force method for finding u is to consider every value of
the attribute in the ly' records as a candidate split position. For each candidate
u, the data set is scanned once to count the number of records with annual
income less than or greater than u. We then compute the Gini index for each
candidate and choose the one that gives the lowest value. This approach is
computationally expensive because it requires O(,nf) operations to compute
the Gini index at each candidate split position. Since there are -l[ candidates,
the overall complexity of this task is O(N\. To reduce the complexity, the
training records are sorted based on their annual income, a computation that
requires O(,n/logli) time. Candidate split positions are identified by taking
the midpoints between two adjacent sorted values: 55, 65, 72, and so on. How-
ever, unlike the brute-force approach, we do not have to examine all ly' records
when evaluating the Gini index of a candidate split position.
For the first candidate. u : 55. none of the records has annual income less
than $55K. As a result, the Gini index for the descendentnode with Annual
 4.3 Decision Tlee Induction 163

fncome < $55K is zero. On the other hand, the number of records with annual
income greater than or equal to $55K is 3 (for class Yes) and 7 (for class No),
respectively. Thus, the Gini index for this node is 0.420. The overall Gini
index for this candidate split position is equal to 0 x 0 + 1 x 0.420 :0.420.
For the second candidate. u : 65. we can determine its class distribution
by updating the distribution of the previous candidate. More specifically,the
new distribution is obtained by examining the class label of the record with
the lowest annual income (i.e., $60K). Since the class label for this record is
No, the count for classNo is increasedfrom 0 to 1 (for Annual Income < $65K)
and is decreasedfrom 7 to 6 (for Annual- Incone > $65K). The distribution
for class Yes remains unchanged. The new weighted-average Gini index for
this candidate split position is 0.400.
This procedure is repeated until the Gini index values for all candidates are
computed, as shown in Figure 4.16. The best split position correspondsto the
one that producesthe smallest Gini index, i.e., u:97. This procedureis less
expensive because it requires a constant amount of time to update the class
distribution at each candidate split position. It can be further optimized by
consideringonly candidate split positions located betweentwo adjacent records
with different class labels. For example, because the first three sorted records
(with annual incomes $60K, $70K, and $75K) have identical class labels, the
best split position should not reside between $60K and $75K. Therefore, the
candidatesplit positionsat a : $55K, $65K, $72K, $87K, $92K, $110K, $I22K,
$772K, and $230K are ignored becausethey are located between two adjacent
records with the same class labels. This approach allows us to reduce the
number of candidate split positions from 11 to 2.

Gain Ratio

Impurity measuressuch as entropy and Gini index tend to favor attributes that
have a large number of distinct values. Figure 4.12 shows three alternative
test conditions for partitioning the data set given in Exercise 2 on page 198.
Comparing the first test condition, Gender, with the second, Car Type, it
is easy to see that Car Type seems to provide a better way of splitting the
data since it produces purer descendentnodes. However, if we compare both
conditions with Customer ID, the latter appears to produce purer partitions.
Yet Custoner ID is not a predictive attribute becauseits value is unique for
each record. Even in a lessextreme situation, a test condition that results in a
large number of outcomes may not be desirable becausethe number of records
associated with each partition is too small to enable us to make anv reliable
predictions.
L64 Chapter 4 Classification

There are two strategies for overcoming this problem. The first strategy is
to restrict the test conditions to binary splits only. This strategy is employed
by decision tree algorithms such as CART. Another strategy is to modify the
splitting criterion to take into account the number of outcomes produced by
the attribute test condition. For example, in the C4.5 decision tree algorithm,
a splitting criterion known as gain ratio is used to deterrnine the goodness
of a split. This criterion is defined as follows:

Ai"fo
Ualn "
ratlo : (4.7)
;--;.,--*-.
5pt1t rnlo

Here, Split Info: -Df:rP(ui)logrP(u6) and /c is the total number of splits.

if
For example, each attribute value has the same number of records, then
Yi, : P(u,;) : Llk and the split information would be equal to log2 k. This
example suggeststhat if an attribute produces a large number of splits, its
split information will also be large, which in turn reduces its gain ratio.

4.3.5 Algorithm for Decision Tlee Induction

A skeleton decision tree induction algorithm called TreeGrowth is shown

in Algorithm 4.7. The input to this algorithm consists of the training records
E and the attribute set F. The algorithm works by recursively selecting the
best attribute to split the data (Step 7) and expanding the leaf nodes of the

Algorithm 4.L A skeleton decision tree induction algorithm.

TreeGrowth (8, F)
1: if stopping-cond(E,f') : true t}nen
2: leaf : createNode$.
3: leaf .label: Ctassify(E).
4: rcturn leaf .
5: else
6: root : createNode0.
7'. root.test-cond: f ind-best-split(E, F).
8: let V : {T.'lois a possible outcome of root.test-cond }.
9: for each u €V do
10: Eo : {e I root.test-cond(e) : u and e e E}.
11: chi,ld: TreeGrowth(8", F).
12: add chi,ld as descendentof root and Iabel the edge (root -- chi,ld) as u.
13: end for
14: end if
I5: return root.
 4.3 Decision Tlee Induction 165

tree (Steps 11 and 12) until the stopping criterion is met (Step 1). The details
of this algorithm are explained below:

1. The createNode$ function extends the decision tree by creating a new

node. A node in the decision tree has either a test condition, denoted as
node.test-cond, or a class label, denoted as node.label.

2. The f ind-best-split0 function determines which attribute should be

selected as the test condition for splitting the training records. As pre-
viously noted, the choice of test condition depends on which impurity
measure is used to determine the goodnessof a split. Some widely used
measuresinclude entropy, the Gini index, and the 12 statistic.

3. The Cl-assifyQ function determines the class label to be assignedto a

leaf node. For each leaf node t,let p(ilt) denote the fraction of training
records from class i associatedwith the node f. In most cases?the leaf
node is assigned to the class that has the majority number of training
records:
leaf .label: argmax p(i,lt), (4.8)

where the argmax operator returns the argument i that maximizes the
expressionp(i,lt). Besidesproviding the information neededto determine
the class label of a leaf node, the fraction p(i,lt) can also be used to es-
timate the probability that a record assignedto the leaf node t belongs
to class z. Sections 5.7.2 and 5.7.3 describe how such probability esti
mates can be used to determine the oerformance of a decisiontree under
different cost functions.

4. The stopping-cond0 function is used to terminate the tree-growing pro-

cess by testing whether all the records have either the same class label
or the same attribute values. Another way to terminate the recursive
function is to test whether the number of records have fallen below some
minimum threshold.

After building the decision tree, a tree-pruning step can be performed

to reduce the size of the decision tree. Decision trees that are too large are
susceptible to a phenomenon known as overfitting. Pruning helps by trim-
ming the branchesof the initial tree in a way that improves the generalization
capability of the decision tree. The issuesof overfitting and tree pruning are
discussedin more detail in Section 4.4.
 166 Chapter 4 Classification

1 6 01 11 11 1 08/Au9/2004 csumnedu/ HTTP/11 200

http://www u24 Mozilla/4.0
10:15:21 -kumar (compatible;
MSIE6.0;
Windows NT5.0)
1 6 0 . 1111 . 1 108/Aug/2004u E l http://www
cs.umn.edu/
HTTPN,1200 41378 Mozilla/4
0
10:15:34 -kumar/MINDS *umar (compatible;
MSIE6 0;
Windows NT5.0)
6 0 1 11 1 , 1 1uu/AUg/zuu4
GET nIIP://WWW.CS,Umn.eOU/HTTPN.l 200 1 0 1 8 5 1 crftp://www
csumneou/ MOZila/4.U
10:15:41 -kumar/MINDS/MINDS ,kumar/MINDS (compatible; MSIE6.0;
naners hlm Windows NT5.0)
6 0 1 11 1 , 1 1 JU/AUg/ZUU4 GET nttp//wwwuS,urln.euu/ t P t 1 . 1 200 7463 )ttp://wwwcs.umn edu/ M0zlila/4.u
10:16:11 -kumar/papers/papers .kumar (compatible;MSIE6 0;
html Windows NT5.0)
3 5 92 2 08/Aug/2004u E l csumnedu/ HTTPN( 200 3149
http://www Mozilla/s.0(Windows;
U;
10:16:15 -sieinbac Windows NT5.1;en-US;
rv:1.7)
Gecko/20040616

(a) Exampleof a Web serverlog

Totalnumberof pagesretrieved
in a Websession
umn.edu/-kumar
htto://www.cs. Totalnumberof imaqepaqesretrievedin a Websession
Totalamountol timesDentbv Websitevisitor
morethanoncein a Websession

madeusingHEADmethod
of requests

MlNDS/MlNDS_papers.htm

(b) Graphof a Web session. (c) Derivedattributesfor Web robotdetection.

4.17.InputdataforWebrobotdetection.
Figure

4.3.6 An Example: Web Robot Detection

Web usage mining is the task of applying data mining techniques to extract
useful patterns from Web accesslogs. These patterns can reveal interesting
characteristicsof site visitors; e.g., people who repeatedly visit a Web site and
view the same product description page are more likely to buy the product if
certain incentives such as rebates or free shipping are offered.
In Web usage mining, it is important to distinguish accessesmade by hu-
man users from those due to Web robots. A Web robot (also known as a Web
crawler) is a software program that automatically locates and retrieves infor-
mation from the Internet by following the hyperlinks embeddedin Web pages.
These programs are deployedby searchengine portals to gather the documents
necessaryfor indexing the Web. Web robot accessesmust be discarded before
applying Web mining techniques to analyze human browsing behavior.
 4.3 Decision Tree Induction 167

This section describeshow a decision tree classifier can be used to distin-

guish between accessesby human users and those by Web robots. The input
data was obtained from a Web server log, a sample of which is shown in Figure
a.I7(a). Each line correspondsto a single page request made by a Web client
(a user or a Web robot). The fields recorded in the Web log include the IP
address of the client, timestamp of the request, Web address of the requested
document, size of the document, and the client's identity (via the user agent
field). A Web sessionis a sequenceof requestsmade by a client during a single
visit to a Web site. Each Web sessioncan be modeled as a directed graph, in
which the nodes correspond to Web pages and the edgescorrespond to hyper-
links connecting one Web page to another. Figure 4.L7(b) shows a graphical
representation of the first Web sessiongiven in the Web server log.
To classify the Web sessions,features are constructed to describe the char-
acteristics of each session. Figure a.fi(c) shows some of the features used
for the Web robot detection task. Among the notable features include the
depth and breadth of the traversal. Depth determines the maximum dis-
tance of a requested page, where distance is measured in terms of the num-
ber of hyperlinks away from the entry point of the Web site. For example,
the home page http://vutw.cs.umn.edu/-lparar is assumedto be at depth
0, whereashttp : / /wuw. cs. unn. edu/kunar/MINDS/MINDS-papers.htnis lo-
cated at depth 2. Based on the Web graph shown in Figure 4.I7(b), the depth
attribute for the first sessionis equal to two. The breadth attribute measures
the width of the corresponding Web graph. For example, the breadth of the
Web sessionshown in Figure 4.17(b) is equal to two.
The data set for classification contains 2916 records, with equal numbers
of sessionsdue to Web robots (class 1) and human users (class 0). L0% of the
data were reservedfor training while the remaining 90% were used for testing.
The induced decision tree model is shown in Figure 4.18. The tree has an
error rate equal to 3.8% on the training set and 5.37oon the test set.
The model suggests that Web robots can be distinguished from human
users in the following way:

1 . Accessesby Web robots tend to be broad but shallow, whereas accesses

by human users tend to be more focused (narrow but deep).

2 . Unlike human users, Web robots seldom retrieve the image pages asso-
ciated with a Web document.

3 . Sessionsdue to Web robots tend to be long and contain a large number

of requested pages.
168 Chapter 4 Classification

DecisionTree:
depth= 1:
breadth>7: class 1
breadth<=7:
breadth<= 3:
lmagePages>0.375: class 0
lmagePages<=0.375:
totalPages<=6: class 1
totalPages>6:
I breadth<= 1: class 1
I breadth> 1: class 0
width > 3:
MultilP= 0:
lmagePages<=0.1333: class 1
lmagePages>0.1333:
breadth<= 6: class 0
breadth> 6: class 1
MultilP= 1:
I TotalTime<= 361: class 0
I TotalTime> 361: class 1
depth>1:
MultiAgent= 0:
deoth> 2: class 0
deoth< 2:
MultilP= 1: class 0
MultilP= 0:
breadth<= 6: class 0
breadth> 6:
I RepeatedAccess<= 0.322: class 0
I RepeatedAccess> 0.322: class 1
MultiAgent= 1:
I totalPages<= 81: class 0
I totalPages> 81: class 1

4.18.Decision
Figure treemodelfor
Webrobot
detection,

4. Web robots are more likely to make repeated requestsfor the same doc-
ument since the Web pages retrieved by human users are often cached
by the browser.

4.3.7 Characteristics of Decision Tbee Induction

The following is a summary of the important characteristics of decision tree

induction algorithms.

1. Decision tree induction is a nonparametric approach for building classifi-

cation models. In other words, it does not require any prior assumptions
regarding the type of probability distributions satisfied by the class and
other attributes (unlike some of the techniques described in Chapter 5).
 4.3 Decision Tbee Induction L69

2 . Finding an optimal decision tree is an NP-complete problem. Many de-

cision tree algorithms employ a heuristic-based approach to guide their
search in the vast hypothesis space. For example, the algorithm pre-
sented in Section 4.3.5 uses a greedy, top-down, recursive partitioning
strategy for growing a decision tree.

3 . Techniques developed for constructing decision trees are computationally

inexpensive, making it possible to quickly construct models even when
the training set size is very large. Furthermore, once a decision tree has
been built, classifying a test record is extremely fast, with a worst-case
complexity of O(to), where ,u.ris the maximum depth of the tree.

4. Decision trees, especially smaller-sizedtrees, are relatively easy to inter-

pret. The accuraciesof the trees are also comparable to other classifica-
tion techniques for many simple data sets.

5 . Decision trees provide an expressive representation for learning discrete-

valued functions. However, they do not generalize well to certain types
of Boolean problems. One notable example is the parity function, whose
value is 0 (1) when there is an odd (even) number of Boolean attributes
with the valueTrue. Accurate modeling of such a function requires a full
decision tree with 2d nodes, where d is the number of Boolean attributes
(seeExercise 1 on page 198).

6 . Decision tree algorithms are quite robust to the presenceof noise, espe-
cially when methods for avoiding overfitting, as described in Section 4.4,
are employed.

7 . The presence of redundant attributes does not adversely affect the ac-
curacy of decision trees. An attribute is redundant if it is strongly cor-
related with another attribute in the data. One of the two redundant
attributes will not be used for splitting once the other attribute has been
chosen. However, if the data set contains many irrelevant attributes, i.e.,
attributes that are not useful for the classificationtask, then some of the
irrelevant attributes may be accidently chosen during the tree-growing
process, which results in a decision tree that is larger than necessary.
Feature selection techniques can help to improve the accuracy of deci-
sion trees by eliminating the irrelevant attributes during preprocessing.
We will investigate the issue of too many irrelevant attributes in Section
4.4.3.
170 Chapter4 Classification

8 . Since most decision tree algorithms employ a top-down, recursive parti-

tioning approach, the number of records becomessmaller as we traverse
down the tree. At the leaf nodes, the number of records may be too
small to make a statistically significant decision about the class rep-
resentation of the nodes. This is known as the data fragmentation
problem. One possible solution is to disallow further splitting when the
number of records falls below a certain threshold.

9 . A subtree can be replicated multiple times in a decision tree, as illus-

trated in Figure 4.19. This makes the decision tree more complex than
necessary and perhaps more difficult to interpret. Such a situation can
arise from decision tree implementations that rely on a single attribute
test condition at each internal node. Since most of the decision tree al-
gorithms use a divide-and-conquer partitioning strategy, the same test
condition can be applied to different parts of the attribute space, thus
Ieading to the subtree replication problem.

problem.
4.19.Treereplication
Figure Thesame canappear
subtree atdifferent
branches.

10. The test conditions described so far in this chapter involve using only a
single attribute at a time. As a consequence,the tree-growing procedure
can be viewed as the process of partitioning the attribute space into
disjoint regions until each region contains records of the same class (see
Figure 4.20). The border between two neighboring regions of different
classesis known as a decision boundary. Since the test condition in-
volves only a single attribute, the decision boundaries are rectilinear; i.e.,
parallel to the "coordinate axes." This limits the expressivenessof the
 4.3 Decision Tree Induction L7L

1
0.9 oiv

0.8 V

0.7 oi
i

0.6 al
V
> 0.5
vi
0.4 V' i v
-----------d----
0.3
0.2 vi
0.1 vio

0r ,
0 0.1 0.3 0.4 0.5 0.6 0.7 0.8 0.9
x

Figure4.20.Example treeanditsdecision
ofa decision fora two-dimensional
boundaries dataset.

'I

i: .'l
.::... ..' '.' ' t
t
0.9
;\ ..1.'- ' t:.tt..t'
ata atat. 3

0.8 t
.t'
:rt
o.7 'ao-t
a

t. .
.t "t
u,o a
jltt'
r- a oo a a
to '

o.4
0
.l
o.2 + + ++ \ .

0.1 ..i..1;j- ;- \
oot
0.1 o.2 0.3 0.4 0.6 o.7 0.9

of datasetthatcannotbepartitioned
Figure4.21. Example involving
usingtestconditions
optimally
single
attributes.

decision tree representation for modeling complex relationships among

continuous attributes. Figure 4.21 illustrates a data set that cannot be
classified effectively by a decision tree algorithm that usestest conditions
involving only a single attribute at a time.
L72 Chapter 4 Classification

An oblique decision tree can be used to overcome this limitation

becauseit allows test conditions that involve more than one attribute.
The data set given in Figure 4.21can be easily representedby an oblique
decision tree containing a single node with test condition

r+A<I.

Although such techniques are more expressive and can produce more
compact trees, finding the optimal test condition for a given node can
be computationally expensive.
Constructive induction provides another way to partition the data
into homogeneous)nonrectangular regions (seeSection 2.3.5 on page 57).
This approach creates composite attributes representing an arithmetic
or logical combination of the existing attributes. The new attributes
provide a better discrimination of the classesand are augmented to the
data set prior to decisiontree induction. Unlike the oblique decision tree
approach, constructive induction is less expensive because it identifies all
the relevant combinations of attributes once, prior to constructing the
decision tree. In contrast, an oblique decision tree must determine the
right attribute combination dynamically, every time an internal node is
expanded. However, constructive induction can introduce attribute re-
dundancy in the data since the new attribute is a combination of several
existing attributes.

11. Studies have shown that the choice of impurity measure has little effect
on the performance of decisiontree induction algorithms. This is because
many impurity measuresare quite consistent with each other, as shown
in Figure 4.13 on page 159. Indeed, the strategy used to prune the
tree has a greater impact on the final tree than the choice of impurity
measure.

4.4 Model Overfitting

The errors committed by a classification model are generally divided into two
types: training errors and generalization errors. TYaining error, also
known as resubstitution error or apparent error, is the number of misclas-
sification errors committed on training records, whereas generalization error
is the expected error of the model on previously unseen records.
Recall from Section 4.2 that a good classification model must not only fit
the training data well, it must also accurately classify records it has never
 4.4 Model Overfitting 173

Trainingsel

:rr E..;* ";T 1i.'_

.'^i:#.$"-i."kii
;liS$.gJ' j:.,. : ";:'.-tsi.i
*i..j..r
18

16

,{.}tll.*fi :rliT"f,-:-g:'i;
14

-
(\l
't2

10
t .. *'-+ *
i **-
*;.**-*3S**o*l 1 * ++ +t*+
* ,** * + * * * +
* qoF oL-Y o
.^ r^**l * ** *
8
I .dr;
b

2 **** t
*l *aoi**3o-..*ia:..-i.^oo.i
0E
0 10 12 16 18
x1

Figure4.22.Example classes.
of a datasetwithbinary

seen before. In other words, a good model must have low training error as
well as low generalization error. This is important becausea model that fits
the training data too well can have a poorer generalization error than a model
with a higher training error. Such a situation is known as model overfitting.

Overfftting Example in Two-Dimensional Data For a more concrete

example of the overfitting problem, consider the two-dimensional data set
shown in Figure 4.22. The data set contains data points that belong to two
different classes,denoted as class o and class +, respectively. The data points
for the o class are generated from a mixture of three Gaussian distributions,
while a uniform distribution is used to generatethe data points for the + class.
There are altogether 1200 points belonging to the o class and 1800 points be-
Ionging to the + class. 30% of.the points are chosen for training, while the
remaining 70%oare used for testing. A decision tree classifier that uses the
Gini index as its impurity measure is then applied to the training set. To
investigate the effect of overfitting, different levels of pruning are applied to
the initial, fully-grown tree. Figure .23(b) shows the training and test error
rates of the decision tree.
L74 Chapter 4 Classification

o.4
- TrainingError
0.35 .- - TestError

0.25
o
G
r
h 0.2
Y
ul
0.1s

0.1

0.05

100 150 200 250 300

Numberof Nodes

Figure4.23.Training
andtesterrorrates.

Notice that the training and test error rates of the model are large when the
size of the tree is very small. This situation is known as model underfitting.
Underfitting occurs becausethe model has yet to learn the true structure of
the data. As a result, it performs poorly on both the training and the test
sets. As the number of nodes in the decision tree increases,the tree will have
fewer training and test enors. However, once the tree becomes too large, its
test error rate begins to increase even though its training error rate continues
to decrease.This phenomenon is known as model overfitting.
To understand the overfitting phenomenon, note that the training error of
a model can be reduced by increasing the model complexity. For example, the
Ieaf nodes of the tree can be expanded until it perfectly fits the training data.
Although the training error for such a complex tree is zero, the test error can
be large because the tree may contain nodes that accidently fit some of the
noise points in the training data. Such nodes can degrade the performance
of the tree becausethey do not generalize well to the test examples. Figure
4.24 shows the structure of two decision trees with different number of nodes.
The tree that contains the smaller number of nodes has a higher training error
rate, but a lower test error rate compared to the more complex tree.
Overfitting and underfitting are two pathologies that are related to the
model complexity. The remainder of this section examines some of the poten-
tial causes of model overfitting.
 4.4 Model Overfitting L75

2n
+

(a) Decision tree with 11 leaf (b) Decision tree with 24 leaf nodes.
nodes.

Figure
4.24.Decision withditferent
trees model
complexities.

4.4.L Overfitting Due to Presence of Noise

Consider the training and test sets shown in Tables 4.3 and 4.4 for the mammal
classification problem. Two of the ten training records are mislabeled: bats
and whales are classifi.edas non-mammals instead of mammals.
A decision tree that perfectly fits the training data is shown in Figure
a.25(a). Although the training error for the tree is zero, its error rate on

Table4.3. Anexampletraining
setforclassifying Classlabels
mammals. withasterisk repre-
symbols
sentmislabeled
records.

Name Body Grves Four- Hibernates Olass

Temperature Birth Iegged Label
porcupme warm-blooded yes yes yes yes
cat warm-blooded yes yes no yes
bat warm-blooded yes no yes no
whale warm-blooded yes no no no
salamander cold-blooded no yes yes no
komodo dragon cold-blooded no yes no no
python cold-blooded no no yes no
salmon cold-blooded no no no no
eagle warm-blooded no no no no
guppy cold-blooded yes no no no
L76 Chapter 4 Classification

Table4.4.Anexample
testsetforclassifying
mammals.
Name tsody Gl ves Four- Hibernates UIaSS
Temperature Birth legged Label
human warm-blooded yes no no yes
plgeon warm-blooded no no no no
elephant warm-blooded yes yes no yes
leopard shark coid-blooded yes no no no
turtle cold-blooded no yes no no
penguin cold-blooded no no no no
eel cold-blooded no no no no
dolphin warm-blooded yes no no yes
spiny anteater warm-blooded no yes yes yes
gila monster cold-blooded no yes yes no

(a) ModelMl (b) ModelM2

Figure4.25.Decision
treeinduced
fromthedatasetshowninTable
4.3.

the test set is 30%. Both humans and dolphins were misclassified as non-
mammals becausetheir attribute values for Body Tenperature, Gives Birth,
and Four-legged are identical to the mislabeled records in the training set.
Spiny anteaters, on the other hand, represent an exceptional casein which the
class label of a test record contradicts the class labels of other similar records
in the training set. Errors due to exceptional casesare often unavoidable and
establish the minimum error rate achievable bv anv classifier.
 4.4 Model Overfitting L77

In contrast, the decision tree M2 shown in Figure 4.25(b) has a lower test
error rate (I0%) even though its training error rate is somewhat higher (20%).
It is evident that the first decision tree, MI, has overfitted the training data
becausethere is a simpler model with lower error rate on the test set. The
Four-legged attribute test condition in model MI is spurious becauseit fits
the mislabeled training records, which leads to the misclassificationof records
in the test set.

4.4.2 Overfitting Due to Lack of Representative Samples

Models that make their classification decisions based on a small number of

training records are also susceptibleto overfitting. Such models can be gener-
ated becauseof lack of representative samples in the training data and learning
algorithms that continue to refine their models even when few training records
are available. We illustrate these effects in the example below.
Consider the five training records shown in Table 4.5. All of these training
records are labeled correctly and the corresponding decision tree is depicted
in Figure 4.26. Although its training error is zero, its error rate on the test
set is 30%.

Table4.5.Anexample
training
setforclassifying
mammals.
Name Body Lt lves Four- Hibernates Class
Temperature Birth Iegged Label
salamander cold- blooded no yes yes no
guppy cold-blooded yes no no no
eagle warm-blooded no no no no
poorwill warm-blooded no no yes no
platypus warm-blooded no yes Yes yes

Humans, elephants,and dolphins are misclassifiedbecausethe decisiontree

classifiesall warm-blooded vertebrates that do not hibernate as non-mammals.
The tree arrives at this classification decision becausethere is only one training
record, which is an eagle, with such characteristics. This example clearly
demonstrates the danger of making wrong predictions when there are not
enough representative examples at the leaf nodes of a decision tree.
I78 Chapter 4 Classification

Figure4.26.Decision
treeinduced
fromthedatasetshowninTable
4.5.

4.4.3 Overfitting and the Multiple Comparison Procedure

Model overfitting may arise in learning algorithms that employ a methodology

known as multiple comparison procedure. To understand multiple comparison
procedure, consider the task of predicting whether the stock market will rise
or fall in the next ten trading days. If a stock analyst simply makes random
guesses,the probability that her prediction is correct on any trading day is
0.5. However, the probability that she will predict correctly at least eight out
of the ten times is
( ' r '+
) ( ' r ) + ( 1 3 :)
2ro
o.ob4l,
which seems quite unlikely.
Suppose we are interested in choosing an investment advisor from a pool of
fifty stock analysts. Our strategy is to select the analyst who makes the most
correct predictions in the next ten trading days. The flaw in this strategy is
that even if all the analysts had made their predictions in a random fashion, the
probability that at least one of them makes at least eight correct predictions
rS
1 - (1 - 0.0547)50 : 0.9399,

which is very high. Although each analyst has a low probability of predicting
at least eight times correctly, putting them together, we have a high probability
of finding an analyst who can do so. Furthermore, there is no guarantee in the
 4.4 Model Overfitting LTg

future that such an analyst will continue to make accurate predictions through
random guessing.
How does the multiple comparison procedure relate to model overfitting?
Many learning algorithms explore a set of independent alternatives, {7a}, and
then choose an alternative, 'y*ur., that maximizes a given criterion function.
The algorithm will add 7*ur" to the current model in order to improve its
overall performance. This procedure is repeated until no further improvement
is observed. As an example, during decision tree growing, multiple tests are
performed to determine which attribute can best split the training data. The
attribute that leads to the best split is chosen to extend the tree as long as
the observed improvement is statistically significant.
Let ?o be the initial decision tree and Trbe the new tree after inserting an
internal node for attribute r. In principle, r can be added to the tree if the
observedgain, A(?s, T*), is greater than some predefined threshold a. If there
is only one attribute test condition to be evaluated, then we can avoid inserting
spurious nodes by choosing a Iarge enough value of a. However, in practice,
more than one test condition is available and the decision tree algorithm must
choosethe best attribute r,,,u* from a set of candidates,{*r,*2,. ..,rp}, to
partition the data. In this situation, the algorithm is actually using a multiple
comparison procedure to decide whether a decision tree should be extended.
More specifically, it is testing for A(?s, T*^u*) > a instead of A(?0, T") > o.
As the number of alternatives, k, increases, so does our chance of finding
A(fo, T*^u*) ) a. Unless the gain function A or threshold a is modified to
account for k, the algorithm may inadvertently add spurious nodes to the
model, which leads to model overfitting.
This effect becomesmore pronounced when the number of training records
from which z-ur. is chosenis small, becausethe variance of A(Tg, Tr^u*) is high
when fewer examples are available for training. As a result, the probability of
finding A(fo, Tr^u*) ) c increaseswhen there are very few training records.
This often happens when the decisiontree grows deeper, which in turn reduces
the number of records covered by the nodes and increases the likelihood of
adding unnecessarynodes into the tree. Failure to compensate for the large
number of alternatives or the small number of training records will therefore
lead to model overfitting.

4.4.4 Estimation of Generalization Errors

Although the primary reason for overfitting is still a subject of debate, it

is generally agreed that the complexity of a model has an impact on model
overfitting, as was illustrated in Figure 4.23. The question is, how do we
180 Chapter 4 Classification

determine the right model complexity? The ideal complexity is that of a

model that produces the lowest generalization error. The problem is that the
learning algorithm has accessonly to the training set during model building
(seeFigure 4.3). It has no knowledge of the test set, and thus, does not know
how well the tree will perform on records it has never seenbefore. The best it
can do is to estimate the generalization error of the induced tree. This section
presents several methods for doing the estimation.

Using Resubstitution Estimate

The resubstitution estimate approach assumesthat the training set is a good

representation of the overall data. Consequently,the training error, otherwise
known as resubstitution error, can be used to provide an optimistic estimate
for the generalization error. Under this assumption, a decision tree induction
algorithm simply selectsthe model that produces the lowest training error rate
as its final model. However, the training error is usually a poor estimate of
generalization error.
Example 4.1. Consider the binary decision trees shown in Figure 4.27. As-
sume that both trees are generated from the same training data and both
make their classification decisionsat each leaf node according to the majority
class. Note that the left tree, Ty, is more complex becauseit expands some
of the leaf nodes in the right tree, ?a. The training error rate for the left
tree is e(?:7):4124:0.167, while the training error rate for the right tree is

Decision
Tree,Ta DecisionTree, Ta

Figure4.27.Example treesgenerated
oftwodecision fromthesametraining
data.
 4.4 Model Overfitting L81

e(Tp) :6f24:0.25. Based on their resubstitution estimate, the left tree is

consideredbetter than the right tree. I

Incorporating Model Complexity

As previously noted, the chance for model overfitting increases as the model
becomes more complex. For this reason, we should prefer simpler models, a
strategy that agreeswith a well-known principle known as Occam's razor or
the principle of parsimony:
Definition 4.2. Occam's Razor: Given two models with the same general-
ization errors, the simpler model is preferred over the more complex model.
Occam's razor is intuitive becausethe additional components in a complex
model stand a greater chanceof being fitted purely by chance. In the words of
Einstein, "Everything should be made as simple as possible, but not simpler."
Next, we present two methods for incorporating model complexity into the
evaluation of classification models.

Pessimistic Error Estimate The first approach explicitly computes gener-

alization error as the sum of training error and a penalty term for model com-
plexity. The resulting generalization error can be considered its pessimistic
error estimate. For instance, let n(t) be the number of training records classi-
fied by node t and e(t) be the number of misclassifiedrecords. The pessimistic
error estimate of a decision tree 7, can be computed as follows:
"s(T),

- eQ)+ o(r)
If:, [".(t')+ cl(to)]
Y' :
e.(T\
l/,
Df:rn(tt)

where k is the number of leaf nodes, e(") is the overall training error of the
decision tree, N1 is the number of training records, and O(t6) is the penalty
term associatedwith each node l;.
Example 4.2. Consider the binary decision trees shown in Figure 4.27. If
the penalty term is equal to 0.5, then the pessimistic error estimate for the
left tree is
4-t7 x0.5 7.5
eg(Tr): : 0.3125
24 24
and the pessimistic error estimate for the right tree is

/m \ 6+4x0.5
e s \ tR ) : :
24 *,:0'3333'
182 Chapter 4 Classification

A B
.) .)
T T
Unlabeled

Figure
4.28.Theminimum (MDL)principle.
length
description

Thus, the left tree has a better pessimistic error rate than the right tree. For
binary trees, a penalty term of 0.5 means a node should always be expanded
into its two child nodes as long as it improves the classification of at least one
training record because expanding a node, which is equivalent to adding 0.5
to the overall error, is less costly than committing one training error.
If fr(r) : 1 for all the nodes t, the pessimistic error estimate for the left
tree is es(Tr) : 71124: 0.458, while the pessimisticerror estimate for the
right tree is en(Tp):10124:0.4L7. The right tree thereforehas a better
pessimistic error rate than the left tree. Thus, a node should not be expanded
into its child nodes unless it reduces the misclassification error for more than
one training record.

Minimum Description Length Principle Another way to incorporate

model complexity is based on an information-theoretic approach known as the
minimum description length or MDL principle. To illustrate this principle,
consider the example shown in Figure 4.28. ln this example, both A and B are
given a set of records with known attribute values x. In addition, person A
knows the exact class label for each record, while person B knows none of this
information. B can obtain the classification of each record by requesting that
A transmits the class labels sequentially. Such a messagewould require @(n)
bits of information, where n is the total number of records.
Alternatively, A may decide to build a classificationmodel that summarizes
the relationship between x and g. The model can be encoded in a compact
 4.4 Model Overfitting 183

form before being transmitted to B. If the model is 100% accurate, then the
cost of transmission is equivalent to the cost of encoding the model. Otherwise,
A must also transmit information about which record is classified incorrectly
by the model. Thus, the overall cost of transmission is

C ost(model,data) : C ost (model) + C ost (datalmodel), (4.e)

where the first term on the right-hand side is the cost of encoding the model,
while the second term representsthe cost of encoding the mislabeled records.
According to the MDL principle, we should seek a model that minimizes the
overall cost function. An example showing how to compute the total descrip-
tion length of a decision tree is given by Exercise 9 on page 202.

Estimating Statistical Bounds

The generalization error can also be estimated as a statistical correction to

the training error. Since generalization error tends to be larger than training
error, the statistical correction is usually computed as an upper bound to the
training error, taking into account the number of training records that reach
a particular leaf node. For instance, in the C4.5 decision tree algorithm, the
number of errors committed by each Ieaf node is assumed to follow a binomial
distribution. To compute its generalizationerror, we must determine the upper
bound limit to the observedtraining error, as illustrated in the next example.
Example 4.3. Consider the left-most branch of the binary decision trees
shown in Figure 4.27. Observe that the left-most leaf node of Tp has been
expanded into two child nodes in 72. Before splitting, the error rate of the
node is 217 :0.286. By approximating a binomial distribution with a normal
distribution, the following upper bound of the error rate e can be derived:

_2 T: .2
,+7#a",/2\,1t1+=/
+ffi
euw.r(Nrercr): , , (4.10)
-d/2
1 -l-
rrly'

where c is the confidence \evel, zo12is the standardized value from a standard
normal distribution, and -Ay'is the total number of training records used to
compute e. By replacing a : 25To,N : 7, and e : 217, the upper bound for
the error rate is eupp.r(7,217,0.25):0.503, which correspondsto 7 x 0.503:
3.521 errors. If we expand the node into its child nodes as shown in ft,, the
training error rates for the child nodes are Lf 4: 0.250 and 1/3 : 0.333,
L84 Chapter 4 Classification

respectively. Using Equation 4.10, the upper bounds of these error rates are
eupp.,(4,714,0.25):0.537 and e,,*",(3,1f 3,0.25) : 0.650,respectively.The
overall training error of the child nodesis 4 x 0.537+3 x 0.650:4.098, which
is larger than the estimated error for the corresponding node in 76. r

Using a Validation Set

In this approach, instead of using the training set to estimate the generalization
error, the original training data is divided into two smaller subsets. One of
the subsets is used for training, while the other, known as the validation set,
is used for estimating the generalization error. Typically, two-thirds of the
training set is reserved for model building, while the remaining one-third is
used for error estimation.
This approach is typically used with classification techniques that can be
parameterized to obtain models with different levels of complexity. The com-
plexity of the best model can be estimated by adjusting the parameter of the
learning algorithm (e.g., the pruning level of a decision tree) until the empir-
ical model produced by the learning algorithm attains the lowest error rate
on the validation set. Although this approach provides a better way for esti-
mating how well the model performs on previously unseen records, less data
is available for training.

4.4.5 Handling Overfitting in Decision Tree Induction

In the previous section, we described several methods for estimating the gen-
eralization error of a classification model. Having a reliable estimate of gener-
alization error allows the learning algorithm to search for an accurate model
without overfitting the training data. This section presents two strategies for
avoiding model overfitting in the context of decision tree induction.

Prepruning (Early Stopping Rule) In this approach, the tree-growing

algorithm is halted before generating a fully grown tree that perfectly fits the
entire training data. To do this, a more restrictive stopping condition must
be used; e.g., stop expanding a leaf node when the observed gain in impurity
measure (or improvement in the estimated generalization error) falls below a
certain threshold. The advantage of this approach is that it avoids generating
overly complex subtrees that overfit the training data. Nevertheless, it is
difficult to choose the right threshold for early termination. Too high of a
threshold will result in underfitted models, while a threshold that is set too low
may not be sufficient to overcomethe model overfitting problem. Furthermore,
 4.4 Model Overfitting L85

lmagePages>0.375: class 0
lmagePages<=0.375:
Simplified Decision Tree:
totalPages>6:
I breadth <= 1: class 1 depth= 1: -c-tass
I breadth > 1: class 0 leds:
t iimageP-a-ges-<---o T--l
I ilmagePages> 0.1333: I
1\Lul!ilP-=-o---- I breadth <= 6: class 0
tilmagePages:fo.i$3: Gtas-sT'i I i I breadth > 6: class I
I i lmagePages>0.1333: d e D t h> 1 :
I i breadth <= 6: class 0 i ------'l
uMuiriAsaT=oaEE;to
I Lbjqaltl jbgs_l_ _ _ _ _ _ _ _l
=
M u l t i l P 1"]j-
: T tr,tuniaGn-t-=l:
I TotalTime<= 361: class 0 | | t o t a l P a g e s < = 8 1 :c l a s s 0
I TotalTime> 361: class 1 | | totalPages>81: classl

NLul!i4s_e!t_=_ot
- _ ------------|
I ideoth"iz-:
^l
c-la-ss-il
| | deplh <= 2:
I i I MultilP= 1: class 0
lrlMultilP=0:
I I | | breadth <= 6: class 0
lrl I breadth>6:
lll | | RepeatedAccess<=0.322:classo i
liLIlEgp_eet9dAc9es1>_q jq2_2i_c!_a9s_1__l
MultiAgent= 1:
I t o t a l P a g e s < = 8 1 :c l a s s 0
I totalPages> 81: class I

Figure4,29. Post-pruning treeforWebrobotdetection,

of thedecision

even if no significant gain is obtained using one of the existing attribute test
conditions, subsequentsplitting may result in better subtrees.

Post-pruning In this approach, the decision tree is initially grown to its

maximum size. This is followed by a tree-pruning step, which proceeds to
trim the fully grown tree in a bottom-up fashion. Ttimming can be done by
replacing a subtree with (1) a new leaf node whose class label is determined
from the majority class of records affiliated with the subtree, or (2) the most
frequently used branch of the subtree. The tree-pruning step terminates when
no further improvement is observed. Post-pruning tends to give better results
than prepruning because it makes pruning decisions based on a fully grown
tree, unlike prepruning, which can suffer from premature termination of the
tree-growing process. However, for post-pruning, the additional computations
needed to grow the full tree may be wasted when the subtree is pruned.
Figure 4.29 illustrates the simplified decision tree model for the Web robot
detection example given in Section 4.3.6. Notice that the subtrees rooted at
186 Chapter 4 Classification

depth : t have been replaced by one of the branches involving the attribute
InagePages. This approach is also known as subtree raising. The depth )
1 and MultiAgent : 0 subtree has been replaced by a leaf node assignedto
class 0. This approach is known as subtree replacement. The subtree for
depth ) 1 and MultiAgent : 1 remains intact.

4.5 Evaluating the Performance of a Classifier

Section 4.4.4 describedseveralmethods for estimating the generalization error
of a model during training. The estimated error helps the learning algorithm
to do model selection; i.e., to find a model of the right complexity that is
not susceptibleto overfitting. Once the model has been constructed, it can be
applied to the test set to predict the class labels of previously unseen records.
It is often useful to measure the performance of the model on the test set
because such a measure provides an unbiased estimate of its generalization
error. The accuracy or error rate computed from the test set can also be
used to compare the relative performance of different classifiers on the same
domain. However, in order to do this, the class Iabels of the test records
must be known. This section reviews some of the methods commonly used to
evaluate the performance of a classifier.

4.5.L Holdout Method

In the holdout method, the original data with labeled examples is partitioned
into two disjoint sets, called the training and the test sets, respectively. A
classification model is then induced from the training set and its performance
is evaluated on the test set. The proportion of data reservedfor training and
for testing is typically at the discretion of the analysts (e.g., 50-50 or two-
thirds for training and one-third for testing). The accuracy of the classifier
can be estimated based on the accuracy of the induced model on the test set.
The holdout method has several well-known limitations. First, fewer la-
beled examples are available for training because some of the records are with-
held for testing. As a result, the induced model may not be as good as when all
the labeled examples are used for training. Second,the model may be highly
dependent on the composition of the training and test sets. The smaller the
training set size, the larger the variance of the model. On the other hand, if
the training set is too large, then the estimated accuracy computed from the
smaller test set is Iess reliable. Such an estimate is said to have a wide con-
fidence interval. Finally, the training and test sets are no longer independent
 4.5 Evaluating the Performance of a Classifier 187

of each other. Because the training and test sets are subsets of the original
data, a class that is overrepresentedin one subset will be underrepresentedin
the other, and vice versa.

4.5.2 Random Subsampling

The holdout method can be repeated several times to improve the estimation
of a classifier'sperformance. This approach is known as random subsampling.
Let acc.i be the model accuracy during the i,th iteration. The overall accuracy
is given by acq,,6 : Ditaccif k. Random subsampling still encounters some
of the problems associatedwith the holdout method becauseit does not utilize
as much data as possiblefor training. It also has no control over the number of
times each record is used for testing and training. Consequently,some records
might be used for training more often than others.

4.5.3 Cross-Validation
An alternative to random subsampling is cross-validation. In this approach,
each record is used the same number of times for training and exactly once
for testing. To illustrate this method, suppose we partition the data into two
equal-sizedsubsets. First, we choose one of the subsets for training and the
other for testing. We then swap the roles of the subsets so that the previous
training set becomesthe test set and vice versa. This approach is called a two-
fold cross-validation. The total error is obtained by summing up the errors for
both runs. In this example, each record is used exactly once for training and
once for testing. The k-fold cross-validation method generalizesthis approach
by segmenting the data into k equal-sizedpartitions. During each run, one of
the partitions is chosenfor testing, while the rest of them are used for training.
This procedure is repeated k times so that each partition is used for testing
exactly once. Again, the total error is found by summing up the errors for
all k runs. A special case of the k-fold cross-validation method sets k : N,
the size of the data set. In this so-called leave-one-out approach, each test
set contains only one record. This approach has the advantage of utilizing
as much data as possible for training. In addition, the test sets are mutually
exclusive and they effectively cover the entire data set. The drawback of this
approach is that it is computationally expensive to repeat the procedure ly'
times. Furthermore, since each test set contains only one record, the variance
of the estimated performance metric tends to be high.
188 Chapter 4 Classification

4.5.4 Bootstrap
The methods presented so far assume that the training records are sampled
without replacement. As a result, there are no duplicate records in the training
and test sets. In the bootstrap approach, the training records are sampled
with replacement; i.e., a record already chosen for training is put back into
the original pool of records so that it is equally likely to be redrawn. If the
original data has .ly' records, it can be shown that, on average, a bootstrap
sample of size ly' contains about 63.2%of the records in the original data. This
approximation follows from the fact that the probability a record is chosen by
a bootstrap sample is 1 - (1 - 1/,^f)^i. When l,r is sufficiently large, the
probability asymptotically approaches 1 - e-r :0.632. Records that are not
included in the bootstrap sample become part of the test set. The model
induced from the training set is then applied to the test set to obtain an
estimate of the accuracy of the bootstrap sample, e.;.The sampling procedure
is then repeated b times to generate b bootstrap samples.
There are several variations to the bootstrap sampling approach in terms
of how the overall accuracy of the classifier is computed. One of the more
widely used approaches is the .632 bootstrap, which computes the overall
accuracy by combining the accuraciesof each bootstrap sample (e;) with the
accuracy computed from a training set that contains all the labeled examples
in the original data (acc"):

1-a
Accuracy, (rcc6661
: x e6+ 0.368 x acc"). (4.11)
i ),{0.U32

4.6 Methods for Comparing Classifiers

It is often useful to compare the performance of different classifiers to deter-
mine which classifier works better on a given data set. However, depending
on the size of the data, the observed difference in accuracy between two clas-
sifiers may not be statistically significant. This section examines some of the
statistical tests available to compare the performance of different models and
classifiers.
For illustrative purposes, consider a pair of classification models, M4 and
M6. Suppose Mn achieves 85To accuracy when evaluated on a test set con-
taining 30 records, while Il[B achieves 75To accvacy on a different test set
containing 5000 records. Based on this information, is M4 a better model
than MB?
 4.6 Methods for Comparing Classifiers 189

The preceding example raises two key questions regarding the statistical
significance of the performance metrics:

1. Although Me has a higher accuracy than Ms, it was tested on a smaller

test set. How much confidence can we place on the accuracy for Ma?

2. Is it possibleto explain the differencein accuracy as a result of variations

in the composition of the test sets?

The first question relates to the issue of estimating the confidence interval of a
given model accuracy. The second question relates to the issue of testing the
statistical significanceof the observeddeviation. These issuesare investigated
in the remainder of this section.

4.6.1 Estimating a Confidence Interval for Accuracy

To determine the confidence interval, we need to establish the probability

distribution that governs the accuracy measure. This section describesan ap-
proach for deriving the confidence interval by modeling the classification task
as a binomial experiment. Following is a list of characteristics of a binomial
experiment:

1. The experiment consists of l/ independent trials, where each trial has

two possible outcomes: successor failure.

2. The probability of success,p, in each trial is constant.

An example of a binomial experiment is counting the number of heads that

turn up when a coin is flipped N times. If X is the number of successes
observed in l/ trials, then the probability that X takes a particular value is
given by a binomial distribution with mean l/p and variance l/p(l - p):

P(X:u):
On,' - p)N-"

For example, if the coin is fair (p : 0.5) and is flipped fifty times, then the
probability that the head shows up 20 times is

P(X :20) : (13;0.t",1- 0.5)30

: 0.041e.

If the experiment is repeated many times, then the average number of heads
expectedto show up is 50 x 0.5 : 25, while its varianceis 50 x 0.5 x 0.5 : I2.5.
190 Chapter 4 Classification

The task of predicting the class labels of test records can also be consid-
ered as a binomial experiment. Given a test set that contains l{ records, let
X be the number of records correctly predicted by a model and p be the true
accuracy of the model. By modeling the prediction task as a binomial experi-
ment, X has a binomial distribution with mean I/p and variance Np(L - p).
It can be shown that the empirical accuracy?acc : X lN , also has a binomial
distribution with mean p and variance p(t-p)lN (seeExercise 12). Although
the binomial distribution can be used to estimate the confidence interval for
acc, it is often approximated by a normal distribution when N is sufficiently
large. Based on the normal distribution, the following confidenceinterval for
acc car:'be derived:
acc-p
t(- z o t zI < zr_.p) :7 _ (r, (4.12)

where Zo12 and Zt-o/z are the upper and lower bounds obtained from a stan-
dard normal distribution at confidencelevel (1 - a). Since a standard normal
distribution is symmetric around Z:0, it follows that Zop: Zt-o/z.Rear-
ranging this inequality leads to the following confidence interval for p:

2xNxacc*Z',tr+Z*/r@
(4.13)
2(N + 22^,r) -
"t

The following table shows the values of Zo12 at different confidence levels:

l,-a 0.99 0 . 9 8 0.95 0 . 9 0 . 8 u . l 0.5

Zo/2 2 . 5 8 2 . 3 3 1 . 9 6 1 . 6 5 I . 2 8 r.04 0 . 6 7

Example 4.4. Consider a model that has an accuracy of 80% when evaluated
on 100 test records. What is the confidenceinterval for its true accuracy at a
95% confidencelevel? The confidencelevel of 95% correspondsto Zo12:1.96
according to the table given above. Inserting this term into Equation 4.13
yields a confidence interval between 71.1% and 86.7%. The following table
shows the confidenceinterval when the number of records, l/, increases:

N 20 50 100 500 1000 5000

Confidence 0.584 0.670 0.71.t 0.763 0.774 0.789
Interval - 0 . 9 1 9 - 0.888 - 0.867 - 0.833 - 0.824 - 0 . 8 1 1

Note that the confidenceinterval becomestiehter when N increases.

 4.6 Methods for Comparing Classifiers 1-91

4.6.2 Comparing the Performance of Two Models

Consider a pair of models, M1 and M2, that are evaluated on two independent
test sets, D1 and D2. Let n1 denote the number of records in D1 and n2 denote
the number of records in D2. In addition, suppose the error rate for M1 oL
Dl is el and the error rate for Mz on D2 is e2. Our goal is to test whether the
observed difference between e1 and e2 is statisticaliy significant.
Assuming that n1 and n2 are sufficiently large, the error rates e1 and e2
can be approximated using normal distributions. If the observed differencein
the error rate is denoted as d : er - €2t then d is also normally distributed
with mean d1,its true difference, and variance, o]. Tne variance of d can be
computed as follows:

n 2 .- - i ?" d -
e1(L-e1) , -e1 2 ( l - e 2 )
-7 (4.r4)
"d
nL n2

where - et)lu and e2(1 - ez)lnz are the variances of the error rates.
"t(I
Finally, at the (t - a)% confidencelevel, it can be shown that the confidence
interval for the true difference dl is given by the following equation:

dt:dLzo12G6. (4.15)

Example 4.5. Consider the problem described at the beginning of this sec-
tion. Model Ma has an error rate of er : 0.15 when applied to A1 : 39
test records, while model Ms has an error rate of ez : 0.25 when applied
to Ab : 5000 test records. The observed difference in their error rates is
d : 10.15- 0.251: 0.1. In this example, we are performing a two-sided test
to check whether dt : 0 or d,6I 0. The estimated variance of the observed
difference in error rates can be computed as follows:

0 . 1 5 ( 1- 0 . 1 5 )
^,
wd -
30
.qag#a:ooo43
or 64 :0.0655. Inserting this value into Equation 4.L5, we obtain the following
confidence interval for d+ at 95% confidence level:

d +: 0 . 1 + 1 . 9 6x 0 . 0 6 5 5: 0 . 1* 0 . 1 2 8 .

As the interval spans the value zero) we can conclude that the observed differ-
ence is not statistically significant at a g5% confidence level. I
L92 Chapter 4 Classification

At what confi.dencelevel can we reject the hypothesis that dt:0? To do

this, we need to determine the value of Zop such that the confidence interval
for d1 does not span the value zero. We can reverse the preceding computation
and look for the valte Zo12 such that d > Zo126a. Replacing the values of d
and 66 givesZo12 < 7.527. This value first occurs when (1 - a) S 0.936 (for a
two-sided test). The result suggeststhat the null hypothesis can be rejected
at confidence level of 93.6% or lower.

4.6.3 Comparing the Performance of Two Classifiers

Suppose we want to compare the performance of two classifiers using the /c-fold
cross-validation approach. Initially, the data set D is divided into /c equal-sized
partitions. We then apply each classifier to construct a model from k - 1 of
the partitions and test it on the remaining partition. This step is repeated /c
times, each time using a different partition as the test set.
Let Mii denote the model induced by classificationtechnique tr; during the
jth iteration. Note that each pair of models M1i and,M2i are tested on the
same partition j. Let e1i and e2i be their respectiveerror rates. The difference
between their error rates durin g the jth fold can be written as di - eU - e2j.
If k is sufficiently large, then d7 is normally distributed with mean dfu, which
is the true differencein their error rates, and variance a"'. Unlike the previous
approach, the overall variance in the observed differences is estimated using
the following formula:

(4.16)

where d is the average difference. For this approach, we need to use a f-

distribution to compute the confidence interval for df":

dt":dItg_.'1,t_r6a.".

The coefficient t1r-a;,r-1 is obtained from a probability table with two input
parameters, its confidencelevel (1 - and the number of degreesof freedom,
k - I. The probability table for the ")
t-distribution is shown in Table 4.6.
Example 4.6. Suppose the estimated difference in the accuracy of models
generated by two classification techniques has a mean equal to 0.05 and a
standard deviation equal to 0.002. If the accuracy is estimated using a 30-fold
cross-validation approach, then at a gbToconfidence level, the true accuracy
difference is
df" :0.05 +2.04x 0.002. (4.17)
 4.7 BibliographicNotes 193

Table4.6. Probability
tablefort-distribution.
(1-o
k-7 0.99 0.98 0.95 0.9 0.8
1
l- 3.08 6.31 72.7 31.8 63.7
2 1.89 2.92 4.30 6.96 9.92
A 1.53 2.L3 2.78 3.75 4.60
9 1.38 1.83 2.26 2.82 3.25
I4 7.34 r.76 2.L4 2.62 2.98
19 1.33 1 .73 2.09 2.54 2.86
24 r.32 7 . 7 r 2.06 2.49 2.80
29 1 . 3 1 7 .70 2.04 2.46 2.76

Since the confidence interval does not span the value zero, the observed dif-
ference between the techniques is statistically significant. r

4.7 Bibliographic Notes

Early classification systems were developed to organize a large collection of
objects. For example, the Dewey Decimal and Library of Congressclassifica-
tion systems were designed to catalog and index the vast number of library
books. The categories are typically identified in a manual fashion, with the
help of domain experts.
Automated classification has been a subject of intensive research for many
yearc. The study of classification in classical statistics is sometimes known as
discriminant analysis, where the objective is to predict the group member-
ship of an object based on a set of predictor variables. A well-known classical
method is Fisher's linear discriminant analysis [117], which seeksto find a lin-
ear projection of the data that produces the greatest discrimination between
objects that belong to different classes.
Many pattern recognition problems also require the discrimination of ob-
jects from different classes.Examples include speechrecognition, handwritten
character identification, and image classification. Readers who are interested
in the application of classification techniques for pattern recognition can refer
to the survey articles by Jain et al. 1722]and Kulkarni et al. [128] or classic
pattern recognition books by Bishop [107], Duda et al. [114], and Fukunaga
[118]. The subject of classificationis also a major researchtopic in the fields of
neural networks, statistical learning, and machine learning. An in-depth treat-
L94 Chapter 4 Classification

ment of various classification techniques is given in the books by Cherkassky

and Mulier [112],Hastie et al. [120],Michie et al. [133],and Mitchell [136].
An overview of decision tree induction algorithms can be found in the
survey articles by Buntine [110], Moret [137], Murthy [tSS], and Safavian et
aL l7a7l. Examples of somewell-known decisiontree algorithms include CART
[108],ID3 [143],C4.5 [ta5], and CHAID [125]. Both ID3 and C4.5 employ the
entropy measure as their splitting function. An in-depth discussion of the
C4.5 decision tree algorithm is given by Quinlan [145]. Besidesexplaining the
methodology for decision tree growing and tree pruning, Quinlan [145] also
described how the algorithm can be modified to handle data sets with missing
values. The CART algorithm was developed by Breiman et al. [108] and uses
the Gini index as its splitting function. CHAID [tZ5] uses the statistical y2
test to determine the best split during the tree-growing process.
The decision tree algorithm presented in this chapter assumes that the
splitting condition is specifiedone attribute at a time. An oblique decisiontree
can use multiple attributes to form the attribute test condition in the internal
nodes [121, 152]. Breiman et al. [108] provide an option for using linear
combinations of attributes in their CARf implementation. Other approaches
for inducing oblique decisiontrees were proposed by Heath et al. [121],Murthy
et al. [139], Cantil-Paz and Kamath 1111],and Utgoff and Brodley 11.52]1.
Although oblique decisiontrees help to improve the expressiveness of a decision
tree representation, learning the appropriate test condition at each node is
computationally challenging. Another way to improve the expressivenessof a
decision tree without using oblique decision trees is to apply a method known
as constructive induction [132]. This method simplifies the task of learning
complex splitting functions by creating compound features from the original
attributes.
Besidesthe top-down approach, other strategiesfor growing a decisiontree
include the bottom-up approach by Landeweerd et al. 1130]and Pattipati and
Alexandridis [142],as well as the bidirectional approach by Kim and Landgrebe
[126]. Schuermann and Doster [150] and Wang and Suen [154] proposed using
a soft splitting criterion to address the data fragmentation problem. In
this approach, each record is assigned to different branches of the decision tree
with different probabilities.
Model overfitting is an important issue that must be addressedto ensure
that a decision tree classifier performs equally well on previously unknown
records. The model overfitting problem has been investigated by many authors
including Breiman et al. [108], Schaffer [148], Mingers [135], and Jensen and
Cohen [123]. While the presence of noise is often regarded as one of the
 Bibliography 195

primary reasons for overfitting [t35, 140], Jensen and Cohen [123] argued
that overfitting is the result of using incorrect hypothesis tests in a multiple
comparison procedure.
Schapire 1149]defined generalization error as "the probability of misclas-
sifying a new example" and test error as "the fraction of mistakes on a newly
sampled test set." Generalization error can therefore be consideredas the ex-
pected test error of a classifier. Generalization error may sometimes refer to
the true error 1136]of a model, i.e., its expected error for randomly drawn
data points from the same population distribution where the training set is
sampled. These definitions are in fact equivalent if both the training and test
sets are gathered from the same population distribution, which is often the
casein many data mining and machine learning applications.
The Occam's razor principle is often attributed to the philosopher William
of Occam. Domingos [113] cautioned against the pitfall of misinterpreting
Occam's razor as comparing models with similar training errors, instead of
generalization errors. A survey on decision tree-pruning methods to avoid
overfitting is given by Breslow and Aha [109] and Esposito et al. [116]. Some
of the typical pruning methods include reduced error pruning [144],pessimistic
error pruninglL44], minimum error pruning [141], critical value pruning 1134],
cost-complexity pruning [108], and error-based pruning [1a5]. Quinlan and
Rivest proposed using the minimum description length principle for decision
tree pruning in [146].
Kohavi [127] had performed an extensive empirical study to compare the
performance metrics obtained using different estimation methods such as ran-
dom subsampling, bootstrapping, and k-fold cross-validation. Their results
suggest that the best estimation method is based on the ten-fold stratified
cross-validation. Efron and Tibshirani [115] provided a theoretical and empir-
ical comparison between cross-validation and a bootstrap method known as
the 632* rule.
Current techniquessuch as C4.5 require that the entire training data set fit
into main memory. There has been considerable effort to develop parallel and
scalable versions of decision tree induction algorithms. Some of the proposed
algorithms include SLIQ by Mehta et al. [131], SPRINT by Shafer et al. [151],
CMP by Wang and Zaniolo [153], CLOUDS by Alsabti et al. [106],RainForest
by Gehrke et al. [119], and ScalParC by Joshi et al. j2al. A general survey
of parallel algorithms for data mining is available in 1129].
196 Chapter 4 Classification

Bibliography
[106] K. Alsabti, S. Ranka, and V. Singh. CLOUDS: A Decision TYee Classifier for Large
Datasets. In Proc. of the lth Intl. Conf. on Knowledge D'iscouery and Data M'ining,
pages 2-8, New York. NY, August 1998.
11071 C. M. Bishop. Neural Networks for Pattern Recogniti,on. Oxford University Press,
Oxford, U.K., 1995.
l1o8lL. Breiman, J. H. Friedman, R. Olshen, and C. J. Stone. Classi,fi'cati'onand Regression
Trees. Chaprnan & Hall, New York, 1984.
l1OelL. A. Breslow and D. W. Aha. Simplifying Decision Tlees: A Survey. Knowledge
Engineering Reui,ew,L2(l):L-40, 1997.
[110] W. Buntine. Learning classification trees. In Artifici,al Intelligence Frontiers i;n Statis-
lics, pages I82 20I. Chapman & Hall, London, 1993.
1111] E. Cantri-Paz and C. Kamath. Using evolutionary algorithms to induce oblique decision
trees. In Proc. of the Genetic and Euoluti,onarg Computation Conf., pages 1053-1060,
San Flancisco, CA, 2000.
[112] V. Cherkassky and F. Mulier. Learning from Data: Concepts, Theory, and Method's.
Wiley Interscience, 1998.
[113] P. Domingos. The Role of Occam's Razor in Knowledge Discovery. Data Mi'ning and
Knouledg e Discouery, 3(4) :409-425, t999.
[114] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Class'ification John Wiley & Sons,
Inc., New York, 2nd edition, 2001.
[115] B. Efron and R. Tibshirani. Cross-validation and the Bootstrap: Estimating the Error
Rate of a Prediction Rule. Technical report, Stanford University, 1995.
[116] F. Esposito, D. Malerba, and G. Semeraro. A Comparative Analysis of Methods for
Pruning Decision Trees. IEEE Trans. Pattern Analysis and Machine Intelli,gence, 19
(5):476-491,May 1997.
[117] R. A. Fisher. The use of multiple measurements in taxonomic problems. Annals of
Eugenics,7:179 188,1936.
11181K. Fukunaga. Introduct'ion to Statist'icalPattern Recognit'i,on.Academic Press, New
York, 1990.
1119] J. Gehrke, R. Ramakrishnan, and V. Ganti. RainForest-A Framework for Fast De-
cision Tree Construction of Large Datasets. Data Mi,ning and Knowledge D'iscouerg, 4
(213):127-162,2000.
[120] T. Hastie, R. Tibshirani, and J. H. Friedman. The Elements of Stati.stical Leanting:
Data Mi,ning, Inference, Pred'icti,on. Springer, New York, 2001.
f121] D. Heath, S. Kasif, and S. Salzberg. Induction of Oblique Decision Trees. In Proc. of
the 13th IntI. Joint Conf. on Arti,fici,al Intelligence, pages 1002-1007, Chambery, France,
August 1993.
ll22] A. K. Jain, R. P. W. Duin, and J. Mao. Statistical Pattern Recognition: A Review.
IEEE Tfan. Patt. Anal. and,Mach. Intellig.,22(I):4 37,2000.
f123] D. Jensen and P. R. Cohen. Multiple Comparisons in Induction Algorithms. Mach'ine
Learning, 38(3) :309-338, March 2000.
ll24l M. V. Joshi, G. Karypis, and V. Kumar. ScalParC: A New Scalable and Efficient
Parallel Classification Algorithm for Mining Large Datasets. In Proc. of 12th IntI.
Parallel Processing Sgmp. (IPPS/SPDP), pages 573 579, Orlando, FL, April 1998.
[125] G. V. Kass. An Exploratory Technique for Investigating Large Quantities of Categor-
ical Data. Appli,ed Statistics, 29:Ll9-127, 1980.
 Bibtiography Lg7

[126]

lr27l

u28l
[12e]

[130j G. Landeweerd,T. Timmers, E. Gersema,M. Bins, and M. Halic. Binary

tree versus
single level tree classification of white blood cells. pattern
Recognzt,ion,L6:bTr-bTZ,
1983.
[131]-M' Mehta, R. Agrawal, and J. Rissanen. sLIQ: A Fast scalable classifier
for Data
Mining. ln Proc. of sth Intt. conf. on Extend,,ingDatabaseTechnorogy,pages
-the
Avignon, Flance,March 19g6. 1g-32,

[132] R' S' Michalski' A theory and methodologyof inductive learning. Arti,fi,cial
Intell,igence,
20:111-116,1988.
[133j D. Michie, D. J. Spiegelhalter, and C. C. Taylor. Machine Learning,
Neural and
Statistical Classi,fi,cati,oz.Ellis Horwood, Upper
. Saddle River, NJ, 1g94.
[134] -J' Mingers. Expert systems-Rule Induction with statisticar Data.
J Operational
Research Soc'ietg,Bg:89 47, 19g2.
[135] J' Mingers' An empirical comparisonof pruning methods for decisiontree
induction.
Machine Learni,ng,4:227_24J,Iggg.
[136] T. Mitchell. Machine Learn,ing.McGraw_Hill,Boston, MA, IggT.
1137]B. M. E' Moret. DecisionT}eesand Diagrams. computing suraegs, r4(4):893-62J,
t982.
I138]-S' K' Murthy. Automatic Construction of Decision TYeesfrom Data:
A
Disciplinarysurvey. Data Mi,ning and KnowredgeDiscouery,2(4):34b-3gg, Multi-
1ggg.
[139] S' K' Murthy, s. r.<3if, S.
_and salzberg. A system for induction of oblique decision
trees. ,.Iof Artificial Intelligence Research,2:I,JJ, Igg4.
[140] T' Niblett. constructing decisiontrees in noisy domains. In proc. of the
2nd.European
Working Sessionon Leaming, pages67_2g,Bled, yugoslavia,
May 19g7.
[141] T' Niblett and I. Bratko' LearningDecisionRulesin Noisy Domai ns.rn
Researchand,
Deuelopmentin Er,pert systems r11,cambridge, 1gg6.
cu-bridg" University press.
[142] K' R' Pattipati and M. G. Alexandri<Iis.Application of heuristic searchand
information
theory to sequentiarfault diagnosis. IEEE Trans. on
sgstems, Mon, ord, cgbernetics,
2o(4):872-s82,
rss}.
[143] J' R' Quinlan. Discovering rules by induction from large collection of
examples. In
D' Michie, editot, Eupert systems ,in the M,icro Erectronic
Ag". Edinburgh university
Press,Edinburgh,UK, 1g79.
Ir44) J- R' Quinlan. simplifying DecisionTtees. 1nrl. J. Man-Machinestud,ies,2T:22r_284,
1987.
[145] J. R. Quinlan. cl.s: progr.amsfor Machine Learni,ng.Morgan-Kaufmann publishers,
San Mateo, CA, 1gg3.
[146] J' R' Quinlan and R. L. Rivest. Inferring DecisionTleesusing the Minimum
Descrip-
tion Length Principle. Information and Computation,g0(B):2i7_24g,
19gg.
1-98 Chapter 4 Classification

[147] S. R. Safavian and D. Landgrebe. A Survey of Decision Ttee Classifier Methodology.

IEEE Trans. Sgstems, Man and Cyber"netics, 22:660-674, May/June 1998.
[148] C. Schaffer. Overfitting avoidence as bias. Machine Learn'ing, 10:153-178, 1993.
[149] R. E. Schapire. The Boosting Approach to Machine Learning: An Overview. In MSRI
Workshop on Nonlinear Estimati'on and Classificati'on, 2002.
[150] J. Schuermann and W. Doster. A decision-theoretic approach in hierarchical classifier
design. P attern Recogn'ition, 17:359-369, 1984.
[151] J. C. Shafer, R. Agrawal, and M. Mehta. SPRINT: A Scalable Parallel Classifier
for Data Mining. In Proc. of the 22nd VLDB Conf., pages 544-555, Bombay, India,
September 1996.
1152] P. E. Utgoff and C. E. Brodley. An incremental method for finding multivariate splits
for decision trees. In Proc. of the 7th IntI. Conf. on Mach'ine Learning, pages 58-65,
Austin. TX. June 1990.
1153] H. Wang and C. Zaniolo. CMP: A Fast Decision Tlee Classifier Using Multivariate
Predictions. In Proc. of the 16th IntI. Conf. on Data Engineering, pages 449-460, San
Diego, CA, March 2000.
1154] Q. R. Wang and C. Y. Suen. Large tree classifier with heuristic search and global
training. IEEE Trans. on Pattern Analys'is and Machine Intell'igence,9(1):91-102, 1987.

4.8 Exercises
1 . Draw the full decision tree for the parity function of four Boolean attributes,
A, B, C, and D. Is it possible to simplify the tree?

2 . Consider the training examples shown in Table 4.7 for a binary classification
problem.

(a) Compute the Gini index for the overall collection of training examples.
(b) Compute the Gini index for the Custoner ID attribute.
(c) Compute the Gini index for the Gender attribute.
(d) Compute the Gini index for the Car Type attribute using multiway split.
(e) Compute the Gini index for the Shirt Size attribute using multiway
split.
(f) Which attribute is better, Gender, Car Type, or Shirt Size?
(g) Explain why Custoner ID should not be used as the attribute test con-
dition even though it has the lowest Gini.

.f. Consider the training examples shown in Table 4.8 for a binary classification
problem.

(a) What is the entropy of this collection of training examples with respect
to the positive class?
 4.8 Exercises 199

Table
4.7.DatasetforExercise
2.
CustomerID Gender Car Type Shirt Size Class
1 M Family Small CO
2 M Sports Medium CO
3 M Sports Medium CO
4 M Sports Large CO
b M Sports Extra Large CO
6 M Sports Extra Large CO
F7
I F Sports Small CO
8 F Sports Small CO
9 F Sports Medium CO
10 F Luxury Large CO
11 M Family Large C1
t2 M Family Extra Large C1
13 M Family Medium C1
t4 M Luxury Extra Large C1
15 F Luxury Small C1
16 F Luxury Small C1
77 F Luxury Medium C1
18 F Luxury Medium C1
19 F Luxury Medium C1
20 F Luxury Larqe C1

Table
4,8.DatasetforExercise
3.
Instance aL a2 as Target Class
I TT1.O T

2 TT6.0 -T

3 TF5.O
4 FF4.O -T-

5 F T 7.0
6 FT3.O
a
T FF8.O
8 T F 7.0 -T-

o FT5.O

(b) What are the information gains of o1 and o2 relative to these training
examples?
(") For o3, which is a continuous attribute, compute the information gain for
every possible split.
200 Chapter 4 Classification

(d) What is the best split (among e7t a2t and o3) according to the information
gain?
(e) What is the best split (between 01 and o2) according to the classification
error rate?
(f) What is the best split (between 01 and o2) according to the Gini index?

4. Show that the entropy of a node never increases after splitting it into smaller
successornodes.

5. Consider the following data set for a binary class problem.

A B ClassLabel
I !' -r
T T -r
T T +
T F
T T +
F F
F F
F F
T T
T F

(a) Calculate the information gain when splitting on ,4 and B. Which at-
tribute would the decision tree induction algorithm choose?
(b) Calculate the gain in the Gini index when splitting on .A and B. Which
attribute would the decision tree induction algorithm choose?
(c) Figure 4.13 shows that entropy and the Gini index are both monotonously
increasing on the range [0, 0.5] and they are both monotonously decreasing
on the range [0.5, 1]. Is it possible that information gain and the gain in
the Gini index favor different attributes? Explain.

o . Consider the following set of training examples.

X Y Z No. of Class C1 Examples No. of ClassC2 Examples

0 0 0 5 40
0 0 1 0 15
0 1 0 10 5
0 I I 45 0
1 0 0 10
I 0 I 25 0
I 1
I 0 5 20
1 1 1
0 15
 4.8 Exercises zOL

(a) Compute a two-level decision tree using the greedy approach described in
this chapter. Use the classification error rate as the criterion for splitting.
What is the overall error rate of the induced tree?
(b) Repeat part (a) using X as the first splitting attribute and then choosethe
best remaining attribute for splitting at each of the two successor nodes.
What is the error rate of the induced tree?
(c) Compare the results of parts (a) and (b). Comment on the suitability of
the greedy heuristic used for splitting attribute selection.

7 The following table summarizes a data set with three attributes A, B. C and
two class labels *, -. Build a two-level decision tree.

Number of
A B C Instances
+
T T T ( 0
F T T 0 20
T F T 20 0
F F T 0 K

T T F 0 0
F T F 25 0
T F F 0 0
F F F 0 25

(a) According to the classification error rate, which attribute would be chosen
as the first splitting attribute? For each attribute, show the contingency
table and the gains in classification error rate.
(b) Repeat for the two children of the root node.
(c) How many instances are misclassified by the resulting decision tree?
(d) Repeat parts (a), (b), and (c) using C as the splitting attribute.
(e) Use the results in parts (c) and (d) to conclude about the greedy nature
of the decision tree induction algorithm.

8. Consider the decision tree shown in Figure 4.30.

(a) Compute the generalization error rate of the tree using the optimistic
approach.
(b) Compute the generalization error rate of the tree using the pessimistic
approach. (For simplicity, use the strategy of adding a factor of 0.5 to
each leaf node.)
(c) Compute the generalization error rate of the tree using the validation set
shown above. This approach is known as reduced error pruning.
2O2 Chapter 4 Classification

Instance A B c Class
+
2 0 0 +
3 1 +
4 1
5 1 0 0 +
6 1 0 0 +
7 1 1 0
8 1 0 +
I 1 1 0
10 1 1 0
Validation:
lnstance A B c Class
0 0 0 +
12 0 1 1 +
13 1 1 0 +
14 1 0 1
15 1 0 0 +

Figure treeanddatasetsforExercise
4.30.Decision 8.

9 . Consider the decision trees shown in Figure 4.31. Assume they are generated
from a data set that contains 16 binarv attributes and 3 classes,C1, C2, and
C3.

(a) Decisiontree with 7 errors (b) Decisiontree with 4 errors

4.31.Decision
Figure forExercise
trees 9.
 4.8 Exercises 2Og

Compute the total description length of each tree according to the

minimum description length principle.

o The total description length of a tree is given by:

C ost (tr ee, dat a) : C ost (tr ee) -f C ost (dat altr ee).

Each internal node of the tree is encoded by the ID of the splitting at-
tribute. If there are rn attributes, the cost of encodins each attribute is
Iog, rn bits.
Each leaf is encoded using the ID of the class it is associated with. If
there are k classes,the cost of encoding a class is log, k bits.
Cost(tree) is the cost of encoding all the nodes in the tree. To simplify the
computation, you can assume that the total cost of the tree is obtained
by adding up the costs of encoding each internal node and each leaf node.
Cost(dataltree) is encoded using the classificationerrors the tree commits
on the training set. Each error is encoded by log2 n bits, where n is the
total number of training instances.

Which decision tree is better, according to the MDL principle?

10. While the .632 bootstrap approach is useful for obtaining a reliable estimate of
model accuracy, it has a known limitation 1127]. Consider a two-class problem,
where there are equal number of positive and negative examples in the data.
Suppose the class labels for the examples are generated randomly. The classifier
used is an unpruned decision tree (i.e., a perfect memorizer). Determine the
accuracy of the classifier using each of the following methods.

(a) The holdout method, where two-thirds of the data are used for training
and the remaining one-third are used for testing.
(b) Ten-fold cross-validation.
(c) The .632 bootstrap method.
(d) Flom the results in parts (u), (b), and (c), which method provides a more
reliable evaluation of the classifier's accuracy?

11. Consider the following approach for testing whether a classifier A beats another
classifier B. Let l[ be the size of a given data set, pa be the accuracy of classifier
A, ps be the accuracy of classifier B, and p: (pt +pB)12 be the average
accuracy for both classifiers. To test whether classifier A is significantly better
than B, the following Z-statistic is used:

PA-PB

Classifier A is assumed to be better than classifier B if Z >

2O4 Chapter 4 Classification

Table 4.9 compares the accuracies of three different classifiers, decision tree
classifiers, naive Bayes classifiers, and support vector machines, on various data
sets. (The latter two classifiersare described in Chapter 5.)

4.9.Comparing
Table theaccuracy
ofvarious methods.
classification
Data Set Size Decision nalve Support vector
(r/) Tlee (%) Bayes (%) machine (%)
nneal 898 92.09 79.62 87.19
Australia 690 6b.bl 76.81 84.78
Auto 205 81.95 58.05 1 U .1 J

Breast 699 95.14 95.99 96.42

Cleve 303 76.24 83.50 84.49
Credit 690 85.80 77.54 85.07
Diabetes 768 72.40 75.91 76.82
German 1000 70.90 74.70 74.40
GIass 214 67.29 48.59 59.81
Heart 270 80.00 84.07 83.70
Hepatitis 155 81.94 83.23 87.10
Horse 368 85.33 78.80 82.6r
Ionosphere 351 89.17 82.34 88.89
Iris 150 94.67 95.33 96.00
Labor FA
78.95 94.74 92.98
LedT 3200 73.34 73.16 73.56
Lymphography L48 77.03 8 3 . 11 86.49
Pima /b6 14.;tO 76.04 76.95
Sonar 208 78.85 69.7L 76.92
Tic-tac-toe 958 83.72 70.04 98.33
Vehicle 846 7t.04 45.04 74.94
Wine 178 94.38 96.63 98.88
Zoo 101 93.07 93.07 96.04

Summarize the performance of the classifiers given in Table 4.9 using the fol-
lowing3 x 3table:

win-loss-draw Decision tree Naive Bayes Support vector

machine
Decision tree 0-0-23
NaiVetsayes 0-0-23
Support vector machine 0-0-23

Each cell in the table contains the number of wins. losses, and draws when
comparing the classifier in a given row to the classifier in a given column.
 4.8 Exercises 2O5

1 2 . Let X be a binomial random variable with mean lr'p and variance lfp(l -p).
Show that the ratio Xf N also has a binomial distribution with mean p and
v a r i a n c ep ( t - p ) l N .
Classification:
AlternativeTechniques
The previous chapter described a simple, yet quite effective, classification tech-
nique known as decision tree induction. Issues such as model overfitting and
classifier evaluation were also discussedin great detail. This chapter presents
alternative techniques for building classification models-from simple tech-
niques such as rule-based and nearest-neighbor classifiers to more advanced
techniques such as support vector machines and ensemble methods. Other
key issues such as the class imbalance and multiclass problems are also dis-
cussedat the end of the chapter.

5.1 Rule-Based Classifier

A rule-based classifier is a technique for classifying records using a collection
of "if . . .then. . ." rules. Table 5.1 shows an example of a model generated by a
rule-basedclassifierfor the vertebrate classificationproblem. The rules for the
model are representedin a disjunctivenormal form, .R : (r1Yr2V...rn), where
R is known as the rule set and r;'s are the classification rules or disjuncts.

Table
5.1.Example
ofa rulesetforthevertebrate problem.
classification
rri (Gives Birth : no) n (Aerial Creature : yes) ------+
Birds
r2i (Gives Birth : no) n (Aquatic Creature : yes) ------+Fishes
13: (Gives Birth : yes) n (Body Temperature : warm-blooded) ------+
Mammals
14: (Gives Birth : no) n (Aerial Creature : no) ----- Reptiles
rbi (Aquatic Creature : semi) ------+
Amphibians
208 Chapter 5 Classification: Alternative Techniques

Each classification rule can be expressedin the following way:

ri: (Condi,ti,ont)- gr. (5.1)

The left-hand side of the rule is called the rule antecedent or precondition.
It contains a conjunction of attribute tests:

Condi,tion,i : (At op ur) A (Az op u2) A . . .(An oP u*), (5.2)

where (Ai,ui) is an attribute-value pair and op is a logical operator chosen

from the set {:, 1,1,},<,>}. Each attribute test (Ai op u7) is known as
a conjunct. The right-hand side of the rule is called the rule consequent,
which contains the predicted class g;.
A rule r covers a record r if the precondition of r matches the attributes
of r. r is also said to be fired or triggered whenever it covers a given record.
For an illustration, consider the rule 11 given in Table 5.1 and the following
attributes for two vertebrates: hawk and grizzly bear.

Name Body SK1n Glves Aquatic Ae r1aI Has Hiber-

Temperature Cover Birth Creature Creature Legs nates
hawk warm-blooded feather no no yes yes no
qrizzly bear warm-blooded fur yes no no yes ves

11 coversthe first vertebrate becauseits precondition is satisfied by the hawk's

attributes. The rule does not cover the secondvertebrate becausegrizzly bears
give birth to their young and cannot fly, thus violating the precondition of 11.
The quality of a classificationrule can be evaluated using measuressuch as
coverageand accuracy. Given a data set D and a classification rule r : A -+ At
the coverage of the rule is defined as the fraction of records in D that trigger
the rule r. On the other hand, its accuracy or confidencefactor is defined as
the fraction of records triggered by r whose class labels are equal to gr. The
formal definitions of these measures are

Coverage(r
) !!l
tDl
. :
Accuracy(r) lAnal (5.3)
T
where lAl is the number of records that satisfy the rule antecedent, lA n gl is
the number of records that satisfy both the antecedent and consequent, and
lDl is the total number of records.
 5.1 Rule-Based Classifier 2Og

Table5,2.Thevertebrate
dataset.
l\ame lJody Skin Gives Aquatic Aerial -nas lllDer- C'lass Label
Temperature Cover Birth Creature Creature Legs nates
numan warm-blooded nalr yes no no yes no IVlammals
python cold-blooded scales no no no no yes Reptiles
salmon cold-blooded scales no yes no no no Fishes
whale warm-blooded hair yes yes no no no Mammals
froo cold-blooded none no semr no yes yes Amphibians
komodo cold-blooded scales no no no yes no Reptiles
dragon
bat warm-blooded hair yes no yes yes yes Mammals
prgeon warm-blooded feathers no no yes yes no Birds
cat warm-blooded fur yes no no yes no Mammals
guppv cold-blooded scales yes yes no no no Fishes
alligator cold-blooded scales no seml no yes no Reptiles
penguln warm-blooded feathers no semr no yes no Birds
porcuplne warm-blooded quills yes no no yes yes Mammals
eel cold-blooded scales no yes no no no Fishes
salamander cold-blooded none no semr no yes yes Amphibians

Example 5.1. Consider the data set shown in Table 5.2. The rule

(G:.ves Birth : yes) A (Body Temperature : warn-blooded) ------+

Mammals

has a coverage of 33% since five of the fifteen records support the rule an-
tecedent. The rule accuracy is 100% because all five vertebrates covered by
the rule are mammals. r

5.1.1 How a Rule-Based Classifier Works

A rule-based classifier classifiesa test record based on the rule triggered by
the record. To illustrate how a rule-based classifier works, consider the rule
set shown in Table 5.1 and the followins vertebrates:

Name lJody Skt n Glves Aquatic Aerial Has Hiber-

Temperature Cover Birth Creature Creature Legs nates
lemur warm-blooded fur yes no no yes yes
turtle cold-blooded scales no semr no yes no
dogfish shark cold-blooded scales yes yes no no no

o The first vertebrate, which is a lemur, is warm-blooded and gives birth

to its young. It triggers the rule 13, and thus, is classifiedas a mammal.
zLO Chapter 5 Classification: Alternative Techniques

o The second vertebrate, which is a turtle, triggers the rules 14 and rs.
Since the classespredicted by the rules are contradictory (reptiles versus
amphibians), their conflicting classesmust be resolved.

o None of the rules are applicable to a dogfish shark. In this case, we

need to ensure that the classifiercan still make a reliable prediction even
though a test record is not covered by any rule.

The previous example illustrates two important properties of the rule set gen-
erated by a rule-based classifier.

Mutually Exclusive Rules The rules in a rule set .R are mutually exclusive
if no two rules in .R are triggered by the same record. This property ensures
that every record is covered by at most one rule in R. An example of a
mutually exclusive rule set is shown in Table 5.3.

Exhaustive Rules A rule set -R has exhaustive coverage if there is a rule

for each combination of attribute values. This property ensures that every
record is coveredby at least one rule in -R. Assuming that Body Tenperature
and Gives Birth are binarv variables. the rule set shown in Table 5.3 has
exhaustive coverage.

ofa mutually
Table5.3. Example andexhaustive
exclusive ruleset.

11: (Body Temperature : cold-blooded) ---- Non-mammals

12: (Body Temperature : warm-blooded) A (Gives Birth : yes) .* Mammals
13: (Body Temperature : warm-blooded) A (Gives Birth : no) --' Non-mammals

Together, these properties ensure that every record is covered by exactly

one rule. Unfortunately, many rule-based classifiers,including the one shown
in Table 5.1, do not have such properties. If the rule set is not exhaustive,
then a default rt7e, r4: 0 .----+!4, must be added to cover the remaining
cases. A default rule has an empty antecedent and is triggered when all other
rules have failed. gr4is known as the default class and is typically assigned to
the majority class of training records not covered by the existing rules.
If the rule set is not mutually exclusive, then a record can be covered by
several rules, some of which may predict conflicting classes. There are two
ways to overcome this problem.
 5.1 Rule-BasedClassifier 2LI

Ordered Rules In this approach, the rules in a rule set are ordered in
decreasing order of their priority, which can be defined in many ways (e.g.,
based on accuracy, coverage,total description length, or the order in which
the rules are generated). An ordered rule set is also known as a decision
list. When a test record is presented,it is classifiedby the highest-ranked rule
that covers the record. This avoids the problem of having conflicting classes
predicted by multiple classification rules.

IJnordered Rules This approach allows a test record to trigger multiple

classification rules and considers the consequent of each rule as a vote for
a particular class. The votes are then tallied to determine the class label
of the test record. The record is usually assigned to the class that receives
the highest number of votes. In some cases, the vote may be weighted by
the rule's accuracy. Using unordered rules to build a rule-based classifier has
both advantages and disadvantages. Unordered rules are less susceptible to
errors caused by the wrong rule being selectedto classify a test record (unlike
classifiers based on ordered rules, which are sensitive to the choice of rule-
ordering criteria). Model building is also less expensive becausethe rules do
not have to be kept in sorted order. Nevertheless,classifying a test record can
be quite an expensive task becausethe attributes of the test record must be
compared against the precondition of every rule in the rule set.

In the remainder of this section, we will focus on rule-basedclassifiersthat

use ordered rules.

5.L.2 Rule-Ordering Schemes

Rule ordering can be implemented on a rule-by-rule basis or on a class-by-class
basis. The difference between these schemesis illustrated in Figure 5.1.

Rule-Based Ordering Scheme This approach orders the individual rules

by some rule quality measure. This ordering schemeensures that every test
record is classifiedby the "best" rule covering it. A potential drawback ofthis
scheme is that lower-ranked rules are much harder to interpret because they
assumethe negation of the rules preceding them. For example, the fourth rule
shown in Figure 5.1 for rule-based ordering,

Aquatic Creature : semi ------+

Amphibians,

has the following interpretation: If the vertebrate does not have any feathers
or cannot fly, and is cold-blooded and semi-aquatic, then it is an amphibian.
2L2 Chapter 5 Classification: Alternative Techniques

Rule-BasedOrdering Class-BasedOrdering
(SkinCover=feathers,
Aerial9r"31u1s=yes) Aerialgte2luvs=yes)
(SkinCover=feathers,
==> Birds ==> Birds

(Bodytemperature=warm-blooded, (Bodytemperature=warm-blooded,
==> 1446t";.
GivesBirth=yes) ==> Birds
GivesBirth=no)
(Bodytemperature=warm-blooded, (Bodytemperature=warm-blooded,
=-> Birds
GivesBirth=no) ==> J1l36P41t
GivesBirth=yes)
==>4606;6;"n.
(AquaticCreature=semi)) ==>4601'';6'"n.
(AquaticCreature=semi))
(SkinCover=scales,
AquaticCreature=no) ==>Amphibians
(SkinCover=none)
=-> Reptiles
Aquaticgr"s1u1e=no)
(SkinCover=scales,
Aquaticgt"u1u1s=yes)
(SkinCover=scales, ==> Reptiles
==> Fishes
Aquaticgtsslups=yes)
(SkinCover=scales,
==>Amphibians
(SkinCover=none) ==> Fishes

Figure5.1. Comparison
between
rule-based
andclass-based schemes.
ordering

The additional conditions (that the vertebrate does not have any feathers or
cannot fly, and is cold-blooded) are due to the fact that the vertebrate does
not satisfy the first three rules. If the number of rules is large, interpreting the
meaning of the rules residing near the bottom of the list can be a cumbersome
task.

Class-Based Ordering Scheme In this approach, rules that belong to the

same class appear together in the rule set R. The rules are then collectively
sorted on the basis of their classinformation. The relative ordering among the
rules from the same class is not important; as long as one of the rules fires,
the class will be assigned to the test record. This makes rule interpretation
slightly easier. However, it is possible for a high-quality rule to be overlooked
in favor of an inferior rule that happens to predict the higher-ranked class.

Since most of the well-known rule-based classifiers(such as C4.5rules and

RIPPER) employ the class-basedordering scheme, the discussion in the re-
mainder of this section focusesmainly on this type of ordering scheme.

5.1.3 How to Build a Rule-Based Classifier

To build a rule-based classifier, we need to extract a set of rules that identifies

key relationships between the attributes of a data set and the class label.
 5.1 Rule-BasedClassifier2I3

There are two broad classesof methods for extracting classification rules: (1)
direct methods, which extract classification rules directly from data, and (2)
indirect methods, which extract classification rules from other classification
models, such as decision trees and neural networks.
Direct methods partition the attribute spaceinto smaller subspacesso that
all the records that belong to a subspacecan be classifiedusing a single classi-
fication rule. Indirect methods use the classificationrules to provide a succinct
description of more complex classificationmodels. Detailed discussionsof these
methods are presented in Sections 5.1.4 and 5.1.5, respectively.

5.L.4 Direct Methods for Rule Extraction

The sequential covering algorithm is often used to extract rules directly
from data. Rules are grown in a greedy fashion based on a certain evaluation
measure. The algorithm extracts the rules one class at a time for data sets
that contain more than two classes.For the vertebrate classification problem,
the sequential covering algorithm may generate rules for classifying birds first,
followed by rules for classifying mammals, amphibians, reptiles, and finally,
fishes (see Figure 5.1). The criterion for deciding which class should be gen-
erated first depends on a number of factors, such as the class prevalence(i.e.,
fraction of training records that belong to a particular class) or the cost of
misclassifyirig records from a given class.
A summary of the sequential covering algorithm is given in Algorithm
5.1. The algorithm starts with an empty decision list, .R. The Learn-One-
Rule function is then used to extract the best rule for class y that covers the
current set of training records. During rule extraction, all training records
for class grare consideredto be positive examples, while those that belong to

Algorithm 5.1 Sequential covering algorithm.

t: Let E be the training records and ,4 be the set of attribute-value pairs, {(,4i, u7)}.
2: Let Y, be an ordered set of classes{yt,yz,. . . ,a*}.
3: Let R: { } be the initial rule list.
4: for each class U e Yo - {gr} do
5: while stopping condition is not met do
6: r +- Learn-One-Rule (E, A, y).
7: Remove training records from -E that are covered by r.
8: Addrtothebottomof therulelist: -R--' RVr.
9: end while
10: end for
11: Insert the default rule, {} + Uk, to the bottom of the rule list R.
2L4 Chapter 5 Classification: Alternative Techniques

other classesare consideredto be negative examples. A rule is desirable if it

covers most of the positive examples and none (or very few) of the negative
examples. Once such a rule is found, the training records covered by the rule
are eliminated. The new rule is added to the bottom of the decision list R.
This procedure is repeated until the stopping criterion is met. The algorithm
then proceedsto generate rules for the next class.
Figure 5.2 demonstrates how the sequential covering algorithm works for
a data set that contains a collection of positive and negative examples. The
rule ,R1, whose coverageis shown in Figure 5.2(b), is extracted first because
it covers the largest fraction of positive examples. All the training records
covered by .Rl are subsequently removed and the algorithm proceeds to Iook
for the next best rule. which is R2.

- :" " " " ':

:++:
:
i++ i iRl
-
:+
:-.-. --
-r:
. - 1 .- ;

-+
+,+ T

(a)OriginalData (b)Step1

:R1 i
- : .., .. ... .. i

-+
*+* :R2

(c) Step2 (d) Step3

Figure5.2.Anexample
ofthesequential algorithm.
covering
 5.1 Rule-BasedClassifier2L5

Learn-One-Rule F\rnction

The objective of the Learn-One-Rule function is to extract a classification

rule that covers many of the positive examples and none (or very few) of the
negative examples in the training set. However, finding an optimal rule is
computationally expensivegiven the exponential size of the searchspace. The
Learn-one-Rule function addressesthe exponential searchproblem by growing
the rules in a greedy fashion. It generates an initial rule r and keeps refining
the rule until a certain stopping criterion is met. The rule is then pruned to
improve its generalization error.

Rule-Growing Strategy There are two common strategies for growing a

classificationrule: general-to-specificor specific-to-general.Under the general-
to-specific strategy, an initial rule r , {} - 3ris created, where the left-hand
side is an empty set and the right-hand side contains the target class. The rule
has poor quality becauseit covers all the examples in the training set. New

Body Temperature= warm-blooded

Body Temperature= warm-blooded,

Body Temperature= warm-blooded,
Has Legs = yes => Mammals
- Gives Birth = yes => Mammals _

(a) General-to-specif
ic

Body Temperature=warm-blooded, Skin Cover=hair,

Gives Birth=yes,Aquatic creature=no,Aerial Creature=no
Has Legs=yes,Hjbernales=no=> Mammals

Skin Cover=hair,Gives Birth=yes Body Temperature=warm-blooded,

Aquatic Creature=no,Aerial Creature=no, Skin Cover=hair,Gives Birth=yes,
Has Legs=yes,Hibernates=no Aquatic creature=no, Aeilal Creature=no
=> Mammals Has Legs=yes=> Mammals

(b) Specificto-general

Figure
5.3.General{o-specific
andspecific-to-general
rule-growing
strategies.
2L6 Chapter 5 Classification: Alternative Techniques

conjuncts are subsequently added to improve the rule's quality. Figure 5.3(a)
shows the general-to-specific rule-growing strategy for the vertebrate classifi-
cation problem. The conjunct Body Tenperature=warn-blooded is initially
chosento form the rule antecedent. The algorithm then explores all the possi-
ble candidates and greedily choosesthe next conjunct, Gives Birth=yes, to
be added into the rule antecedent. This process continues until the stopping
criterion is met (e.g., when the added conjunct does not improve the quality
of the rule).
For the specific-to-general strategy, one of the positive examples is ran-
domly chosen as the initial seed for the rule-growing process. During the
refinement step, the rule is generalized by removing one of its conjuncts so
that it can cover more positive examples. Figure 5.3(b) shows the specific-to-
general approach for the vertebrate classification problem. Suppose a positive
example for mammals is chosen as the initial seed. The initial rule contains
the same conjuncts as the attribute values of the seed. To improve its cov-
erage, the rule is generalized by removing the conjunct Hibernate=no. The
refinement step is repeated until the stopping criterion is met, e.g., when the
rule starts covering negative examples.
The previous approachesmay produce suboptimal rules becausethe rules
are grown in a greedy fashion. To avoid this problem, a beam search may be
used, where k of the best candidate rules are maintained by the algorithm.
Each candidate rule is then grown separately by adding (or removing) a con-
junct from its antecedent. The quality ofthe candidates are evaluated and the
k best candidates are chosen for the next iteration.

Rule Evaluation An evaluation metric is needed to determine which con-

junct should be added (or removed) during the rule-growing process. Accu-
racy is an obvious choice becauseit explicitly measuresthe fraction of training
examplesclassifiedcorrectly by the rule. However, a potential limitation of ac-
curacy is that it does not take into account the rule's coverage. For example,
consider a training set that contains 60 positive examples and 100 negative
examples. Suppose we are given the following two candidate rules:

Rule 11: covers 50 positive examples and 5 negative examples,

Rule 12: covers 2 positive examples and no negative examples.

The accuracies for 11 and 12 are 90.9% and 100%, respectively. However,
11 is the better rule despite its lower accuracy. The high accuracy for 12 is
potentially spurious becausethe coverageof the rule is too low.
 5.1 Rule-BasedClassifier 2L7

The following approachescan be used to handle this problem.

1. A statistical test can be used to prune rules that have poor coverage.
For example, we may compute the following likelihood ratio statistic:

R: 2\ fitosff,;le),
i:I

where k is the number of classes,fi is the observed frequency of class e

examples that are covered by the rule, and el is the expected frequency
of a rule that makes random predictions. Note that ,R has a chi-square
distribution with k - 1 degrees of freedom. A large l? value suggests
that the number of correct predictions made by the rule is significantly
larger than that expected by random guessing. For example, since 11
covers 55 examples,the expected frequency for the positive classis ea :
55x601160:20.625, while the expected frequency for the negative class
is e- : 55 x 100/160 : 34.375. Thus, the likelihood ratio for 11 is

E(r1) : 2 x [50 x log2(50120.625)

f 5 x logr(5134.375)): 99.9.

Similarly, the expected frequenciesfor 12 a,ree+ : 2 x 60/160 : 0.75

and e- :2x 100/160:1.25. The likelihoodratio statisticfor re is

R(r2) : 2 x [2 x log2(210.75)+ 0 x log2(0/1.25)]: 5.66.

This statistic therefore suggeststhat rr is a better rule than 12.

2. At evaluation metric that takes into account the rule coveragecan be

used. Consider the following evaluation metrics:

f , -r1
Laplace : (5.4)
nl k'
f+ -t kp+
m-estimate : I (O.DJ
n-fk

where n is the number of examples covered by the rule, /-p is the number
of positive examplescoveredby the rule, k is the total number of classes,
and p1 is the prior probability for the positive class. Note that the m-
estimate is equivalent to the Laplace measure by choosing p+ : llk.
Depending on the rule coverage, these measures capture the trade-off
218 Chapter 5 Classification: Alternative Techniques

between rule accuracy and the prior probability of the positive class. If
the rule does not cover any training example, then the Laplace mea-
sure reduces to lf k, which is the prior probability of the positive class
assuming a uniform class distribution. The m-estimate also reduces to
the prior probability (p1) when n : 0. However, if the rule coverage
is large, then both measuresasymptotically approach the rule accuracy,
f+ln. Going back to the previous example, the Laplace measure for
11 is 51157 : 89.47To,which is quite close to its accuracy. Conversely,
the Laplace measure for 12 (75%) is significantly lower than its accuracy
because 12 has a much lower coverage.

3. An evaluation metric that takes into account the support count of the
rule can be used. One such metric is the FOILts information gain.
The support count of a rule correspondsto the number of positive exam-
ples covered by the rule. Suppose the rule r : A ------* covers ps positive
examples and ns negative examples. After adding a new conjunct B, the
extended rule r' : A tt,B ------+
* covers p1 positive examples and n1 neg-
ative examples. Given this information, the FOIL's information gain of
the extended rule is defined as follows:

FoIL's information gain: pt x ( bsr=!= - logz (5.6)

"'Pr +nr " - - 1o- ).
\ Polnsl

Since the measure is proportional to pr and ptl(n*nr), it prefers rules

that have high support count and accuracy. The FOIL's information
gains for rules 11 and 12 given in the preceding example are 43.12 and 2,
respectively. Therefore, 11 is a better rule than 12.

Rule Pruning The rules generated by the Learn-One-Rule function can be

pruned to improve their generalization errors. To determine whether pruning
is necessary,w€ may apply the methods described in Section 4.4 on page
172 to estimate the generalization error of a rule. For example, if the error
on validation set decreasesafter pruning, we should keep the simplified rule.
Another approach is to compare the pessimistic error of the rule before and
after pruning (see Section 4.4.4on page 179). The simplified rule is retained
in place of the original rule if the pessimistic error improves after pruning.
 5.1 Rule-Based Classifier 2I9

Rationale for Sequential Covering

After a rule is extracted, the sequential covering algorithm must eliminate

all the positive and negative examples covered by the rule. The rationale for
doing this is given in the next example.

R3 R2
T

+
class= + + +
+
++

class=

Figure5.4. Elimination
of training
records
bythesequential
covering R7,R2, andR3
algorithm.
represent
regions
coveredbythreedifferent
rules.

Figure 5.4 shows three possible rules, R7, R2, and R3, extracted from a
data set that contains 29 positive examples and 21 negative examples. The
accuraciesof .R1,R2, and E3 are I2115 (80%), 7lI0 (70%), and 8f L2 (66.7%),
respectively. .R1 is generated first because it has the highest accuracy. After
generating R1, it is clear that the positive examplescoveredby the rule must be
removed so that the next rule generated by the algorithm is different than .R1.
Next, supposethe algorithm is given the choice of generating either R2 or R3.
Even though R2 has higher accuracy than -R3, Rl and -R3 together cover 18
positive examples and 5 negative examples (resulting in an overail accuracy of
78.3%), whereasR1 and .R2together cover 19 positive examplesand 6 negative
examples (resulting in an overall accuracy of 76%). The incremental impact of
R2 or -R3on accuracy is more evident when the positive and negative exarnples
coveredby -Rl are removed before computing their accuracies.In particular, if
positive examples covered by R1 are not removed, then we may overestimate
the effective accuracy of ,R3, and if negative examples are not removed, then
we may underestimate the accuracy of R3. In the latter caseTwe might end up
preferring R2 over fi3 even though half of the false positive errors committed
by E3 have already been accounted for by the preceding rule, .R1.
22O Chapter 5 Classification: Alternative Techniques

RIPPER Algorithm

To illustrate the direct method, we consider a widely used rule induction algo-
rithm called RIPPER. This algorithm scalesalmost linearly with the number
of training examples and is particularly suited for building models from data
sets with imbalanced class distributions. RIPPER also works well with noisy
data sets becauseit uses a validation set to prevent model overfitting.
For two-class problems, RIPPER choosesthe majority class as its default
classand learns the rules for detecting the minority class. For multiclass prob-
lems, the classesare ordered according to their frequencies.Let (y,Az, . . . ,U")
be the ordered classes,where 91 is the least frequent class and g" is the most
frequent class. During the first iteration, instances that belong to 91 are Ia-
beled as positive examples,while those that belong to other classesare labeled
as negative examples. The sequentialcovering method is used to generaterules
that discriminate between the positive and negative examples. Next, RIPPER
extracts rules that distinguish y2 frorn other remaining classes. This process
is repeated until we are left with g., which is designated as the default class.

Rule Growing RIPPER employs a general-to-specificstrategy to grow a

rule and the FOIL's information gain measure to choose the best conjunct
to be added into the rule antecedent. It stops adding conjuncts when the
rule starts covering negative examples. The new rule is then pruned based
on its performance on the validation set. The following metric is computed to
determine whether pruning is needed: (p-") l@+n), where p (n) is the number
of positive (negative) examples in the validation set covered by the rule. This
metric is monotonically related to the rule's accuracy on the validation set. If
the metric improves after pruning, then the conjunct is removed. Pruning is
done starting from the last conjunct added to the rule. For example, given a
rde ABCD + a, RIPPER checkswhether D should be pruned first, followed
by CD, BCD, etc. While the original rule covers only positive examples, the
pruned rule may cover some of the negative examples in the training set.

Building the Rule Set After generating a rule, all the positive and negative
examples covered by the rule are eliminated. The rule is then added into the
rule set as long as it does not violate the stopping condition, which is based
on the minimum description length principle. If the new rule increasesthe
total description length of the rule set by at least d bits, then RIPPER stops
adding rules into its rule set (by default, d is chosento be 64 bits). Another
stopping condition used by RIPPER is that the error rate of the rule on the
validation set must not exceed 50%.
 5.1 Rule-BasedClassifier22L

RIPPER also performs additional optimization steps to determine whether

some of the existing rules in the rule set can be replaced by better alternative
rules. Readers who are interested in the details of the optimization method
may refer to the referencecited at the end of this chapter.

5.1.5 Indirect Methods for Rule Extraction

This section presents a method for generating a rule set from a decision tree.
In principle, every path from the root node to the Ieaf node of a decision tree
can be expressedas a classification rule. The test conditions encountered along
the path form the conjuncts ofthe rule antecedent,while the classlabel at the
leaf node is assignedto the rule consequent. Figure 5.5 shows an example of a
rule set generated from a decision tree. Notice that the rule set is exhaustive
and contains mutually exclusive rules. However, some of the rules can be
simplified as shown in the next example.

Rule Set

=-> -
rl: (P=No,Q=No)
12:(P=No,Q=Yes)==2 'i
==1 .'
r3; (P=Yes,Q=No)
14:(P=Yes,R=Yes,Q=No) ==> -
r5: (P=Yes,R=Yes,Q=Yes)==> a

Flgure5.5. Converting treeintoclassification

a decision rules.

Example 5.2. Consider the following three rules from Figure 5.5:
12: (P : No) A (Q : Yes) ------+
-1-
r3: (P : Yes) n (R: No) -----+
1
r5: (P : Yes) A (R: Yes) n (Q : Yes) ------+
f
Observe that the rule set always predicts a positive class when the value of Q
is Yes. Therefore, we may simplify the rules as follows:
r2t; (Q : Yes) ----+f
r3: (P : Yes) A (R: No) ------+
1.
222 Chapter 5 Classification: Alternative Techniques

Rul+Based
Classifier:
(GivesBirth=No, => Birds
AerialCreature=Yes)
(GivesBirth=No, => Fishes
AquaticCreature=Yes)
(GivesBirth=Yes)=> Mammals
(GivesBirth=No,
AerialCreature=No,
AquaticCreature=No)
=> Reptiles
( ) => Amphibians

Figure5.6. Classification
rulesextracted
froma decision
treeforthevertebrate problem.
classification

13 is retained to cover the remaining instances of the positive class. Although

the rules obtained after simplification are no longer mutually exclusive, they
are less complex and are easier to interpret. I
In the following, we describe an approach used by the C4.5rules algorithm
to generate a rule set from a decision tree. Figure 5.6 shows the decision tree
and resulting classification rules obtained for the data set given in Table 5.2.

Rule Generation Classification rules are extracted for every path from the
root to one of the leaf nodes in the decision tree. Given a classification rule
r : A -------+
gr, we consider a simplified rule, r' : A' + A, where A/ is obtained
by removing one of the conjuncts in A. The simplified rule with the lowest
pessimistic error rate is retained provided its error rate is lessthan that of the
original rule. The rule-pruning step is repeated until the pessimistic error of
the rule cannot be improved further. Becausesome of the rules may become
identical after pruning, the duplicate rules must be discarded.

Rule Ordering After generating the rule set, C4.5rules usesthe class-based
ordering schemeto order the extracted rules. Rules that predict the same class
are grouped together into the same subset. The total description length for
each subset is computed, and the classesare arranged in increasing order of
their total description length. The class that has the smallest description
 5.2 Nearest-Neighborclassifiers 223

length is given the highest priority becauseit is expected to contain the best
* gX
set of rules. The total description length for a class is given by tr"*ception
Imodel, where trexceptionis the number of bits neededto encodethe misclassified
examples, Zmodelis the number of bits neededto encode the model, and g is a
tuning parameter whose default value is 0.5. The tuning parameter depends
on the number of redundant attributes present in the model. The value of the
tuning parameter is small if the model contains many redundant attributes.

5.1.6 Characteristics of Rule-Based Classifiers

A rule-based classifier has the following characteristics:

o The expressivenessof a rule set is almost equivalent to that of a decision

tree becausea decision tree can be representedby a set of mutually ex-
clusive and exhaustive rules. Both rule-basedand decisiontree classifiers
create rectilinear partitions of the attribute space and assign a class to
each partition. Nevertheless,if the rule-based classifier allows multiple
rules to be triggered for a given record, then a more complex decision
boundary can be constructed.

o Rule-based classifiersare generally used to produce descriptive models

that are easier to interpret, but gives comparable performance to the
decision tree classifier.

o The class-based ordering approach adopted by many rule-based classi-

fiers (such as RIPPER) is well suited for handling data sets with imbal-
anced class distributions.

5.2 Nearest-Neighbor classifiers

The classification framework shown in Figure 4.3 involves a two-step process:
(1) an inductive step for constructing a classification model from data, and
(2) a deductive step for applying the model to test examples. Decision tree
and rule-based classifiers are examples of eager learners because they are
designedto learn a model that maps the input attributes to the class label as
soon as the training data becomesavailable. An opposite strategy would be to
delay the processof modeling the training data until it is neededto classify the
test examples. Techniques that employ this strategy are known as lazy learn-
ers. An example of alazy learner is the Rote classifier, which memorizesthe
entire training data and performs classification only if the attributes of a test
instance match one of the training examples exactly. An obvious drawback of
224 Chapter 5 Classification: Alternative Techniques

+ + + +
(a) 1-nearest
neighbor (b)2-nearestneighbor (c) 3-nearestneighbor

Figure5.7. The1-,2-,and3-nearest
neighbors
ofaninstance.

this approach is that some test records may not be classifiedbecausethey do

not match any training example.
One way to make this approach more flexible is to find all the training
examples that are relatively similar to the attributes of the test example.
These examples, which are known as nearest neighbors, can be used to
determine the classlabel of the test example. The justification for using nearest
neighbors is best exemplified by the following saying: "If i,t walks li,ke a duck,
quacksli,lcea duck, and looks li,kea duck, then i,t's probably a duck." A nearest-
neighbor classifier represents each example as a data point in a d-dimensional
space,where d is the number of attributes. Given a test example, we compute
its proximity to the rest of the data points in the training set, using one of
the proximity measures described in Section 2.4 on page 65. The k-nearest
neighbors of a given example z refer to the k points that are closest to z.
Figure 5.7 illustrates the L-, 2-, and 3-nearest neighbors of a data point
located at the center of each circle. The data point is classified based on
the class labels of its neighbors. In the case where the neighbors have more
than one label, the data point is assignedto the majority class of its nearest
neighbors. In Figure 5.7(a), the l-nearest neighbor of the data point is a
negative example. Therefore the data point is assigned to the negative class.
If the number of nearest neighbors is three, as shown in Figure 5.7(c), then
the neighborhood contains two positive examples and one negative example.
Using the majority voting scheme,the data point is assignedto the positive
class. In the case where there is a tie between the classes(see Figure 5.7(b)),
we may randomly chooseone of them to classify the data point.
The preceding discussionunderscoresthe importance of choosingthe right
value for k. If k is too small, then the nearest-neighbor classifier may be
 5.2 Nearest-Neighborclassifiers 225

neighbor
Figure5.8. k-nearest withlarge/c.
classification

susceptible to overfitting becauseof noise in the training data. On the other

hand, if k is too large, the nearest-neighborclassifier may misclassify the test
instance becauseits list of nearest neighbors may include data points that are
located far away from its neighborhood (seeFigure 5.8).

5.2.L Algorithm
A high-level summary of the nearest-neighbor classification method is given in
Algorithm 5.2. The algorithm computes the distance (or similarity) between
each test example , : (*',y') and all the training examples (x, g) e D to
determine its nearest-neighborlist, D". Such computation can be costly if the
number of training examples is large. However, efficient indexing techniques
are available to reduce the amount of comoutations needed to find the nearest
neighbors of a test example.

Algorithm 5.2 The k-nearest neighbor classification algorithm.

7: Let k be the number of nearest neighbors and D be the set of training examples.
2: fot each test example : (x' ,E') do
"
3: Compute d(x',x), the distance between z and every example, (x,y) e D.
4: Select D, ! D, the set of k closest training examples to z.
5: y' : argmax-D6u,ouy.n" I(u : yu1
6: end for
226 Chapter 5 Classification: Alternative Techniques

Once the nearest-neighbor list is obtained, the test example is classified

based on the majority class of its nearest neighbors:

Majority Voting: g' : argmax I(, : A), (5.7)

U
" (xi,g6)€D.

where u is a class label, 916

is the class label for one of the nearest neighbors,
and 1(.) is an indicator function that returns the value 1 if its argument is
true and 0 otherwise.
In the majority voting approach, every neighbor has the sameimpact on the
classification. This makes the algorithm sensitive to the choice of k, as shown
in Figure 5.7. One way to reduce the impact of k is to weight the influence
of each nearest neighbor x; according to its distance: wt.:7ld(x',x4)2. As
a result, training examples that are located far away from z have a weaker
impact on the classification compared to those that are located close to z.
Using the distance-weightedvoting scheme,the class label can be determined
as follows:

Distance-Weighted Voting: A' : argrnax u; x I(u : Ai). (5.8)

(x;,v6)€D.

5.2.2 Characteristics of Nearest-Neighbor Classifiers

The characteristics of the nearest-neighborclassifier are summarized below:

o Nearest-neighborclassificationis part of a more generaltechnique known
as instance-basedlearning, which usesspecifictraining instancesto make
predictions without having to maintain an abstraction (or model) de-
rived from data. Instance-basedlearning algorithms require a proximity
measureto determine the similarity or distance between instances and a
classification function that returns the predicted class of a test instance
based on its proximity to other instances.

o Lazy learners such as nearest-neighbor classifiersdo not require model

building. However, classifying a test example can be quite expensive
becausewe need to compute the proximity values individually between
the test and training examples. In contrast, eager learners often spend
the bulk of their computing resourcesfor model building. Once a model
has been built, classifying a test example is extremely fast.

o Nearest-neighborclassifiersmake their predictions based on local infor-

mation, whereas decision tree and rule-based classifiersattempt to find
 5.3 BayesianClassifiers 227

a global model that fits the entire input space. Becausethe classification
decisionsare made locally, nearest-neighborclassifiers(with small values
of /c) are quite susceptible to noise.

Nearest-neighbor classifiers can produce arbitrarily shaped decision bound-

aries. Such boundaries provide a more flexible model representation
compared to decision tree and rule-based classifiers that are often con-
strained to rectilinear decision boundaries. The decision boundaries of
nearest-neighbor classifiers also have high variability because they de-
pend on the composition of training examples. Increasing the number of
nearest neighbors may reduce such variability.

Nearest-neighbor classifiers can produce wrong predictions unless the

appropriate proximity measure and data preprocessing steps are taken.
For example, suppose we want to classify a group of people based on
attributes such as height (measured in meters) and weight (measured in
pounds). The height attribute has a low variability, ranging from 1.5 m
to 1.85 m, whereas the weight attribute may vary from 90 lb. to 250
lb. If the scale of the attributes are not taken into consideration, the
proximity measure may be dominated by differences in the weights of a
person.

5.3 Bayesian Classifiers

In many applications the relationship between the attribute set and the class
variable is non-deterministic. In other words, the class label of a test record
cannot be predicted with certainty even though its attribute set is identical
to some of the training examples. This situation may arise becauseof noisy
data or the presence of certain confounding factors that affect classification
but are not included in the analysis. For example, consider the task of pre-
dicting whether a person is at risk for heart diseasebased on the person's diet
and workout frequency. Although most people who eat healthily and exercise
regularly have less chance of developing heart disease,they may still do so be-
cause of other factors such as heredity excessivesmoking, and alcohol abuse.
Determining whether a person's diet is healthy or the workout frequency is
sufficient is also subject to interpretation, which in turn may introduce uncer-
tainties into the learning problem.
This section presents an approach for modeling probabilistic relationships
between the attribute set and the class variable. The section begins with an
introduction to the Bayes theorem, a statistical principle for combining prior
228 Chapter 5 Classification: Alternative Techniques

knowledge of the classeswith new evidence gathered from data. The use of the
Bayes theorem for solving classification problems will be explained, followed
by a description of two implementations of Bayesian classifiers: naiVe Bayes
and the Bayesian belief network.

5.3.1 Bayes Theorem

Cons'idera footballgame betweentwo riual teams: Team0 and Team1.

SupposeTeam 0 wins 65%oof the ti,me and Team 1 w,insthe remaining
matches. Among the gameswon by Team 0, only 30Toof them come
from playing on Team1's footballfield. On the other hand,,75Toof the
u'ictoriesfor Team1 are obta'inedwhi,leplayi.ngat home. If Teamf is to
host the nert match betweenthe two teams,wh,ichteam wi,ll most likelu
etnergeas the winner?

This question can be answered by using the well-known Bayes theorem. For
completeness, we begin with some basic definitions from probability theory.
Readerswho are unfamiliar with conceptsin probability may refer to Appendix
C for a brief review of this topic.
Let X and Y be a pair of random variables. Their joint probability, P(X :
r,Y : g), refers to the probability that variable X will take on the value
r and variable Y will take on the value g. A conditional probability is the
probability that a random variable will take on a particular value given that the
outcome for another random variable is known. For example, the conditional
probability P(Y :UlX: r) refers to the probability that the variable Y will
take on the value g, given that the variable X is observedto have the value r.
The joint and conditional probabilities for X and Y are related in the following
way:
P(x,Y) : P(Ylx) x P(X) : P(XIY) x P(Y). (5.e)
Rearranging the last two expressionsin Equation 5.9 leads to the following
formula, known as the Bayes theorem:

P(xlY)P(Y)
P(Ylx): (5.10)
P(X)

The Bayes theorem can be used to solve the prediction problem stated
at the beginning of this section. For notational convenience, let X be the
random variable that represents the team hosting the match and Y be the
random variable that representsthe winner of the match. Both X and Y can
 5.3 Bayesian Classifiers 229

take on values from the set {0,1}. We can summarize the information given
in the problem as follows:

Probability Team 0 wins is P(Y :0) : 0.65.

ProbabilityTeam1 wins is P(Y :1) : 1 - P(Y : 0) : 0.35.
Probability Team t hostedthe match it won is P(X : llY :1) : 0.75.
Probability Team t hostedthe match won by Team 0 is P(X : llY :0) : 0.9.

Our objective is to compute P(Y : llx : 1), which is the conditional

probability that Team 1 wins the next match it will be hosting, and compares
it against P(Y :OlX: 1). Using the Bayestheorem, we obtain

P (X : rl Y :1 ) x P( Y : 1)
P(Y:llx : 1) :
P(X :1)
P(X :LlY :1) x P(Y : 1)
P ( X : ! , Y : 1 )+ P ( X : 1 , Y : 0 )
P ( X : L I Y: 1 ) x P ( Y : 1 )
P (X : IIY :I) P( Y : 1)* P( X : r lY :O) P( Y : 0)
0.75x 0.35
0.75x0.35+0.3x0.65
: 0.5738.

where the law of total probability (seeEquation C.5 on page 722) was applied
in the secondline. Furthermore, P(Y :OlX : 1) : t - P(Y : llx - 1) :
0.4262. Since P(Y : llx : 1) > P(Y : OlX : 1), Team t has a better
chance than Team 0 of winning the next match.

5.3.2 Using the Bayes Theorem for Classification

Before describing how the Bayes theorem can be used for classification, let
us formalize the classification problem from a statistical perspective. Let X
denote the attribute set and Y denote the class variable. If the class variable
has a non-deterministic relationship with the attributes, then we can treat
X and Y as random variables and capture their relationship probabilistically
using P(YIX). This conditional probability is also known as the posterior
probability for Y, as opposed to its prior probability, P(Y).
During the training phase, we need to learn the posterior probabilities
P(ylX) for every combination of X and Y based on information gathered
from the training data. By knowing these probabilities, a test record X' can
be classified by finding the class Yt that maximizes the posterior probability,
230 Chapter 5 Classification: Alternative Techniques

P(Y'lX'). To illustrate this approach, consider the task of predicting whether

a loan borrower will default on their payments. Figure 5.9 shows a training
set with the following attributes: Hone Owner, Marital Status, and Annual
Income. Loan borrowers who defaulted on their payments are classified as
Yes, while those who repaid their loans are classifi.edas No.

a."'
"""""od"

Figure5.9.Training
setforpredicting problem.
theloandefault

Suppose we are given a test record with the following attribute set: X :
(HoneOwner: No, Marital Status : Married, Annual Income : $120K). To
classify the record, we need to compute the posterior probabilities P(YeslX)
and P(NolX) basedon information available in the training data. If P(YeslX) >
P(NolX), then the record is classified as Yes; otherwise, it is classifiedas No.
Estimating the posterior probabilities accurately for every possible combi-
nation of class labiel and attribute value is a difficult problem because it re-
quires a very large training set, even for a moderate number of attributes. The
Bayes theorem is useful becauseit allows us to expressthe posterior probabil-
ity in terms of the prior probability P(f), the class-conditional probability
P(X|Y), and the evidence,P(X):

P( xlY) x P( Y)
P(ylx): (5.11)
P(x)
When comparing the posterior probabilities for different values of Y, the de-
nominator term, P(X), is always constant, and thus, can be ignored. The
 5.3 BayesianClassifiers 23L

prior probability P(f) can be easily estimated from the training set by com-
puting the fraction of training records that belong to each class. To estimate
the class-conditionalprobabilities P(Xlf), we present two implementations of
Bayesian classification methods: the naiVe Bayes classifier and the Bayesian
belief network. These implementations are described in Sections 5.3.3 and
5.3.5,respectively.

5.3.3 NaiVe Bayes Classifier

A naive Bayes classifier estimates the class-conditional probability by assuming
that the attributes are conditionally independent, given the classlabel g. The
conditional independenceassumption can be formally stated as follows:

&
P ( X I Y: a ) : f r l x S v : 11, (5.12)
i:l

where each attribute set X : {Xr, X2,..., X4} consistsof d attributes.

Conditional Independence

Before delving into the details of how a naive Bayes classifier works, let us
examine the notion of conditional independence. Let X, Y, and Z denote
three sets of random variables. The variables in X are said to be conditionally
independent of Y, given Z, 1f the following condition holds:

P(xlY,z) : P(xlz). (5.13)

An example of conditional independenceis the relationship between a person's

arm length and his or her reading skills. One might observe that people with
longer arms tend to have higher levels of reading skills. This relationship can
be explained by the presenceof a confounding factor, which is age. A young
child tends to have short arms and lacks the reading skills of an adult. If the
age of a person is fixed, then the observed relationship between arm length
and reading skills disappears. Thus, we can conclude that arm length and
reading skills are conditionally independent when the age variable is fixed.
232 Chapter 5 Classification: Alternative Techniques

The conditional independencebetween X and Y can also be written into

a form that looks similar to Equation 5.12:

P\X,Y,Z)
P(x,Ylz) :
P(Z)
P (
-FEqX .Y , Z ) , . P ( Y . Z )
^
P(z)
P(xlY,z) x P(vlz)
P(xlz)x P(Ylz), (5.14)

where Equation 5.13 was used to obtain the last line of Equation 5.14.

How a Nai've Bayes Classifier Works

With the conditional independence assumption, instead of computing the

class-conditionalprobability for every combination of X, we only have to esti-
mate the conditional probability of each X4, given Y. The latter approach is
more practical becauseit does not require a very large training set to obtain
a good estimate of the probability.
To classify a test record, the naive Bayes classifier computes the posterior
probability for each class Y:

p(ytx)' - PV)n!:l P(xilY) (5.15)

P(x)

Since P(X) is fixed for every Y, it is sufficient to choosethe class that maxi-
mizes the numerator term, p(V)l[i:tP(X,lY). In the next two subsections,
we describe several approaches for estimating the conditional probabilities
P(X,lY) for categorical and continuous attributes.

Estimating Conditional Probabilities for Categorical Attributes

For a categorical attribute Xa, the conditional probability P(Xi : rilY : A)
is estimated according to the fraction of training instancesin class g that take
on a particular attribute value ri. For example, in the training set given in
Figure 5.9, three out of the seven people who repaid their loans also own a
home. As a result, the conditional probability for P(Home Owner:Yeslno) is
equal to 3/7. Similarly, the conditional probability for defaulted borrowers
who are single is given by P(Marital Status : SinglelYes) : 213.
 5.3 Bayesian Classifiers 233

Estimating Conditional Probabilities for Continuous Attributes

There are two ways to estimate the class-conditional probabilities for contin-
uous attributes in naive Bayes classifiers:

1. We can discretize each continuous attribute and then replace the con-
tinuous attribute value with its corresponding discrete interval. This
approach transforms the continuous attributes into ordinal attributes.
The conditional probability P(X,IY : U) is estimated by computing
the fraction of training records belonging to class g that falls within the
corresponding interval for Xi. The estimation error depends on the dis-
cretization strategy (as described in Section 2.3.6 on page 57), as well as
the number of discrete intervals. If the number of intervals is too large,
there are too few training records in each interval to provide a reliable
estimate for P(XrlY). On the other hand, if the number of intervals
is too small, then some intervals may aggregate records from different
classesand we may miss the correct decision boundary.

2. We can assume a certain form of probability distribution for the contin-

uous variable and estimate the parameters of the distribution using the
training data. A Gaussian distribution is usually chosento representthe
class-conditional probability for continuous attributes. The distribution
is characterized by two parameters, its mean, p,, and variance, o2. For
each class Aj, the class-conditional probability for attribute Xi is

., tt:j)2
_(tt_
zofi
P(Xi: r,ilY : y) : -)- exp (5.16)
1/2troii

The parameter p,ii can be estimated based on the sample mean of Xt

(z) for all training record.sthat belong to the class gt . Similarly, ol, can
be estimated from the sample variance (s2) of such training records. For
example, consider the annual income attribute shown in Figure 5.9. The
sample mean and variance for this attribute with respect to the class No
are

r 2 5 + 1 0 0 + 7 0 + . . . + 7 5:
110
(
, ( 1 2 5 1 1 0 ) 2+ ( 1 0 0- 1 1 0 ) 2+ . . . + ( 7 5- 1 1 0 ) 2
-
:2975
7(6)
s: t/2975:54.54.
234 Chapter 5 Classification: Alternative Techniques

Given a test record with taxable income equal to $120K, we can compute
its class-conditionalprobability as follows:

:
P(rncome=12olNo) : 0.0072.
6h.b4)"*p-95#f

Note that the preceding interpretation of class-conditional probability

is somewhat misleading. The right-hand side of Equation 5.16 corre-
sponds to a probability density function, f (X;pti,o;7). Since the
function is continuous, the probability that the random variable Xl takes
a particular value is zero. Instead, we should compute the conditional
probability that Xi lies within some interval, ri and rt t e, where e is a
small constant:

f rtle
P ( * o< X ; I r i * e l y : y r 1 : I fqo;ttij,oij)dxi
J:r'
= f (rt; ttti,o,ii) x e. (5.17)

Since e appears as a constant multiplicative factor for each class, it

cancels out when we normalize the posterior probability for P(flX).
Therefore, we can still apply Equation 5.16 to approximate the class-
conditional probability P (X,lY).

Example of the Naive Bayes Classifier

Consider the data set shown in Figure 5.10(a). We can compute the class-
conditional probability for each categorical attribute, along with the sample
mean and variance for the continuous attribute using the methodology de-
scribed in the previous subsections. These probabilities are summarized in
Figure 5.10(b).
To predict the class label of a test record ;q : (HomeOwner:No, Marital
Status : Married, Income : $120K), we need to compute the posterior prob-
abilities P(UolX) and P(YeslX). Recall from our earlier discussionthat these
posterior probabilities can be estimated by computing the product between
the prior probability P(Y) and the class-conditionalprobabilitiesll P(X,lY),
which corresponds to the numerator of the right-hand side term in Equation
5.15.
The prior probabilities of each class can be estimated by calculating the
fraction of training records that belong to each class. Since there are three
records that belong to the classYes and sevenrecords that belong to the class
 5.3 Bayesian Classifiers 235

P(HomeOwner=YeslNo) = 317
P(HomeOwner=NolNo) = 4fl
P(HomeOwner=YeslYes) =0
P(HomeOwner=NolYes) =1
P(Marital = 2n
Status=SinglelNo)
Yes 125K
No)= 1/7
P(MaritalStatus=Divorcedl
No 100K = 4t7
P(MaritalStatus=MarriedlNo)
No 70K = 2/3
P(MaritalStatus=SinglelYes)
Yes 120K = 1/3
P(MaritalStatus=DivorcedlYes)
=0
P(MaritalStatus=MarriedlYes)
No 95K
No 60K ForAnnualIncome:
Yes 220K lf class=No:samplemean=110
No 85K samplevariance=2975
No 75K samplemedn=90
lf class=Yes:
samplevariance=2S
No 90K

(a) (b)

Figure
5.10.Thenalve
Bayes
classifier problem.
fortheloanclassification

No, P(Yes) :0.3 and P(no) :0.7. Using the information provided in Figure
5.10(b), the class-conditionalprobabilities can be computed as follows:

P(Xluo) : P(Hone 0wner : NolNo)x P(status : MarriedlNo)

x P(Annual fncome : $120KlNo)
: 417 x 417 x 0.0072: 0.0024.

P(XlYes) : P(Home 0wner : IrtolYes)x P(Status : MarriedlYes)

x P(AnnuaI Income : $120KlYes)
: 1x0x1.2x10-e:0.

Putting them together, the posterior probability for class No is P(NolX) :

a x 7 l l 0 x 0 . 0 0 2 4 : 0 . 0 0 1 6 a , w h e r ea : l l P ( X ) i s a c o n s t a n t e r m . U s i n g
a similar approach, we can show that the posterior probability for class Yes
is zero because its class-conditional probability is zero. Since P(NolX) >
P(YeslX), the record is classifiedas No.
236 Chapter 5 Classification: Alternative Techniques

M-estimate of Conditional Probability

The preceding example illustrates a potential problem with estimating poste-

rior probabilities from training data. If the class-conditional probability for
one of the attributes is zero, then the overall posterior probability for the class
vanishes. This approach of estimating class-conditional probabilities using
simple fractions may seem too brittle, especially when there are few training
examples available and the number of attributes is large.
In a more extreme case, if the training examples do not cover many of
the attribute values, we may not be able to classify some of the test records.
For example, if P(Marital Status : DivorcedlNo) is zero instead of If7,
then a record with attribute set 1(: (HomeOwner- yes, Marital Status :
Divorced, Income : $120K) has the following class-conditionalprobabilities:

P(Xlno) : 3/7 x 0 x 0.0072: 0.

P ( X l v e s ) : 0 x 7 1 3x 7 . 2x 1 0 - e : 0 .

The naive Bayes classifier will not be able to classify the record. This prob-
lem can be addressed by using the m-estimate approach for estimating the
conditional probabilities :

!!
P (r,l a- ) : ?s! , (5.18)
n+Tn

where n is the total number of instances from class 3ry,n" is the number of
training examples from class gi that take on the value ri, rrl is a parameter
known as the equivalent sample size, and p is a user-specifiedparameter. If
there is no training set available (i.e., n:0), then P(rilyi) : p. Therefore
p can be regarded as the prior probability of observing the attribute value
ri among records with class 97. The equivalent sample size determines the
tradeoff between the prior probability p and the observed probability n.f n.
In the example given in the previous section, the conditional probability
P(Status : MarriedlYes) : 0 because none of the training records for the
class has the particular attribute value. Using the m-estimate approach with
m:3 and p :113, the conditional probability is no longer zero:

P(Marital Status:MarriedlYes) : (0+3 xtlS)/(J+3) :176.

 5.3 Bayesian Classifiers 237

If we assumep : If 3 for all attributes of class Yes and p : 213 for all
attributes of class No. then

P(Xluo) : P(Home Owner: NolNo)x P(status : MarriedlNo)

x P(Annual Incone : $120KlNo)
: : o.oo26.
6lto x 6/10x o.oo72

P(XlYes) : P(Home 0tmer : ttolYes) x P(status : MarriedlYes)

x P(AnnuaI Income: $120KlYes)
: 4 / 6 x 1 1 6x 7 . 2 x 1 0 - e : 1 . 3 x 1 0 - 1 0 .

The posterior probability for class No is P(llolx) : (t x 7110 x 0.0026 :

0.0018o, while the posterior probability for class Yes is P(YeslX) : o x
3/10 x 1.3 x 10-10 : 4.0 x 10-11a. Atthough the classificationdecisionhas
not changed, the m-estimate approach generally provides a more robust way
for estimating probabilities when the number of training examples is small.

Characteristics of Naive Bayes Classifiers

NaiVe Bayes classifiers generally have the following characteristics:

o They are robust to isolated noise points becausesuch points are averaged
out when estimating conditional probabilities from data. Naive Bayes
classifiers can also handle missing values by ignoring the example during
model building and classification.

o They are robust to irrelevant attributes. If Xi is an irrelevant at-

tribute, then P(XrlY) becomesalmost uniformly distributed. The class-
conditional probability for Xi has no impact on the overall computation
of the posterior probability.

o Correlated attributes can degrade the performance of naive Bayes clas-

sifiers becausethe conditional independenceassumption no longer holds
for such attributes. For example, consider the following probabilities:

P(A:0lY:0) :0.4, P(A:1lY:0) :0.6,

P ( A : 0 l Y : 1 ) : 0 . 6 , P ( A : L I Y: 1 ) : 0 . 4 ,

where A is a binary attribute and Y is a binary class variable. Suppose

there is another binary attribute B that is perfectly correlated with A
238 Chapter 5 Classification: Alternative Techniques

when Y : 0, but is independent of -4 when Y : I. For simplicity,

assumethat the class-conditionalprobabilities for B are the same as for
A. Given a record with attributes,4 :0.8:0. we can comoute its
posterior probabilities as follows:

P ( A : O l y : 0 ) P ( B : O l y : O ) P ( Y: 0 )
P(Y:0lA:0, B : 0) :
P(A:0, B : 0)
0 . 1 6x P ( Y : 0 )
P(A:0, B : 0)'

P ( A : O l y : I ) P ( B : O l y: l ) P ( Y : 1 )
P(Y : IlA:0,8 : 0) :
P(A:0, B : 0)
0 . 3 6x P ( Y 1 )
:
P(A:0, B : 0)'

If P(Y - 0) : P(Y : 1), then the naiVe Bayes classifier would assign
the record to class 1. However, the truth is,

P ( A : 0 ,B : O l Y : 0 ) : P ( A : 0l)' : 0) : 0.4,

becauseA and B are perfectly correlated when Y : 0. As a result, the

posterior probability for Y : 0 is

P ( A : 0 , 8 : O l Y : 0 ) P ( Y: 0 )
P(Y :0lA:0, B : 0) :
P(A:0,8:0)
0 . 4x P ( Y : 0 )
P(A :0,8 : 0 )'

which is larger than that for Y : 1. The record should have been
classifiedas class 0.

5.3.4 Bayes Error Rate

Supposewe know the true probability distribution that governs P(Xlf). The
Bayesianclassificationmethod allows us to determine the ideal decisionbound-
ary for the classification task, as illustrated in the following example.
Example 5.3. Consider the task of identifying alligators and crocodilesbased
on their respectivelengths. The averagelength of an adult crocodile is about 15
feet, while the averagelength of an adult alligator is about 12 feet. Assuming
 5.3 Bayesian Classifiers 239

\, Crocodile
\
\
\
\
\
\
\
\
\
\
\
\
\

5 10 tu
Length,*

Figure5.11.Comparing
thelikelihood of a crocodile
functions andanalligator.

that their length z follows a Gaussian distribution with a standard deviation

equal to 2 feet, we can express their class-conditional probabilities as follows:

:
P(Xlcrocodile) ( 5.1e)
#"""0 1
:
P(Xlnrri.gator) ;(ry)'l (5.20)
#""*o[ "(ry)')
Figure 5.11 shows a comparison between the class-conditionalprobabilities
for a crocodile and an alligator. Assuming that their prior probabilities are
the same, the ideal decision boundary is located at some length i such that

P(X: ilCrocodile): P(X: f lAlligator).

Using Equations 5.19 and 5.20, we obtain

(ft-rb\2
:\ /i-r2\2
\ , / , /'

which can be solved to yield f : 13.5. The decision boundary for this example
is located halfway between the two means. r
24O Chapter 5 Classification: Alternative Techniques

(a) (b) (c)

Figure probabilistic
5.12.Representing relationships
usingdirected graphs.
acyclic

When the prior probabilities are different, the decision boundary shifts
toward the class with lower prior probability (see Exercise 10 on page 319).
Furthermore, the minimum error rate attainable by any classifieron the given
data can also be computed. The ideal decision boundary in the preceding
example classifies all creatures whose lengths are less than ft as alligators and
those whose lengths are greater than 0 as crocodiles. The error rate of the
classifier is given by the sum of the area under the posterior probability curve
for crocodiles (from length 0 to i) and the area under the posterior probability
curve for alligators (from f to oo):

Error: et 'o.odirelX)d, * P(Alri-gatorlx)d,X.

fo" Ir*
The total error rate is known as the Bayes error rate.

5.3.5 Bayesian Belief Networks

The conditional independence assumption made by naive Bayes classifiers may

seem too rigid, especially for classification problems in which the attributes
are somewhat correlated. This section presents a more flexible approach for
modeling the class-conditional probabilities P(Xlf). Instead of requiring all
the attributes to be conditionally independent given the class, this approach
allows us to specify which pair of attributes are conditionally independent.
We begin with a discussionon how to represent and build such a probabilistic
model, followed by an example of how to make inferences from the model.
 5.3 Bayesian Classifiers 24L

Model Representation

A Bayesian belief network (BBN), or simply, Bayesian network, provides a

graphical representation of the probabilistic relationships among a set of ran-
dom variables. There are two key elements of a Bayesian network:

1. A directed acyclic graph (dag) encoding the dependence relationships

among a set of variables.

2. A probability table associatingeach node to its immediate parent nodes.

Consider three random variables, -4, B, and C, in which A and B are

independent variables and each has a direct influence on a third variable, C.
The relationships among the variables can be summarized into the directed
acyclic graph shown in Figure 5.12(a). Each node in the graph represents a
variable, and each arc asserts the dependencerelationship between the pair
of variables. If there is a directed arc from X to Y, then X is the parent of
Y and Y is the child of X. F\rrthermore, if there is a directed path in the
network from X to Z, then X is an ancestor of Z, whlle Z is a descendant
of X. For example, in the diagram shown in Figure 5.12(b), A is a descendant
of D and D is an ancestor of B. Both B and D arc also non-descendantsof
A. An important property of the Bayesian network can be stated as follows:

Property 1 (Conditional Independence). A node in a Bayesi'annetwork

i,s condi,tionally i,ndependentof its non-descendants,i'f i,tsparents are lcnown.

In the diagram shown in Figure 5.12(b), A is conditionally independent of

both B and D given C because the nodes for B and D are non-descendants
of node A. The conditional independenceassumption made by a naive Bayes
classifiercan also be representedusing a Bayesian network, as shown in Figure
5.12(c),where gris the target classand {Xt,Xz,...,Xa} is the attribute set.
Besides the conditional independenceconditions imposed by the network
topology, each node is also associatedwith a probability table.

1. If a node X does not have any parents, then the table contains only the
prior probability P(X).

2. If a node X has only one parent, Y, then the table contains the condi-
tional probability P(XIY).

3. If a node X has multiple parents, {Yt,Yz, . . . ,Yn}, then the table contains
the conditionalprobability P(XlY,Yz,. . ., Yr.).
242 Chapter 5 Classification: Alternative Techniques

Hb=Yes
HD=Yes
D=Healthy 0.2
E=Yes D=Unhealthy 0.85
D=Heatthy 0.25
E=Yes 0.45
D=Unhealthl
E=No
D=Healthy 0.55
E=No
0.75 CP=Yes
D=Unhealth!
HD=Yes
Hb=Yes
0.8
HD=Yes
Hh=Nn
0.6
HD=No
0.4
Hb=Yes
HD=No
Hb=No
0.1

Figure5.13.A Bayesian
beliefnetwork
fordetecting
heartdisease in patients.
andheartburn

Figure 5.13 shows an example of a Bayesian network for modeling patients

with heart diseaseor heartburn problems. Each variable in the diagram is
assumed to be binary-valued. The parent nodes for heart disease (HD) cor-
respond to risk factors that may affect the disease,such as exercise (E) and
diet (D). The child nodes for heart disease correspond to symptoms of the
disease,such as chest pain (CP) and high blood pressure (BP). For example,
the diagram shows that heartburn (Hb) may result from an unhealthy diet
and may lead to chest pain.
The nodes associated with the risk factors contain only the prior proba-
bilities, whereas the nodes for heart disease,heartburn, and their correspond-
ing symptoms contain the conditional probabilities. To save space, some of
the probabilities have been omitted from the diagram. The omitted prob-
abilities can be recovered by noting that P(X - *) : 1 - P(X : r) and
P(X : TIY) : 1 - P(X : rlY), where 7 denotes the opposite outcome of r.
For example, the conditional probability

P(Heart Disease: NolExercise: No,Diet : Healthy)

1 - P(Heart Disease : YeslExercise : No,Diet : Healthy)
1-0.55:0.45.
 5.3 BayesianClassifiers 243

Model Building

Model building in Bayesiannetworks involves two steps: (1) creating the struc-
ture of the network, and (2) estimating the probability values in the tables
associatedwith each node. The network topology can be obtained by encod-
ing the subjective knowledge of domain experts. Algorithm 5.3 presents a
systematic procedure for inducing the topology of a Bayesian network.

Algorithm 5.3 Algorithm for generating the topology of a Bayesian network.

1: Let T: (XuXz,...,X4) denote a total order of the variables.
2:forj:Ttoddo
3: Let X76 denote the jth highest order variable in ?.
4: Let r(X711) : {Xz(r), Xrp1, . . . , Xr1-t)} denote the set of variables preced-
ing X713y.
5: Remove the variables from r(Xrti>) that do not affect X1 (using prior knowl-
edge).
6: Create an arc between X71y; and the remaining variables in r(X711).
7: end for

Example 5.4. Consider the variables shown in Figure 5.13. After performing
Step 1, Iet us assume that the variables are ordered in the following way:
(E,D,HD,H\,CP,BP). From Steps 2 to 7, starting with variable D, we
obtain the following conditional probabilities:
. P(DIE) is simplified to P(D).

o P(HDlE, D) cannot be simplified.

o P(HblHD,E,D) is simplifiedto P(HblD).

o P(C PlHb, H D, E, D) is simplified to P(C PlHb, H D).

o P(BPICP,Hb,HD,E,D) is simplifiedto P(BPIHD).

Based on these conditional probabilities) we can create arcs between the nodes
(8, HD), (D, HD), (D, Hb), (HD, CP), (Hb, CP), and (HD, BP). These
arcs result in the network structure shown in Figure 5.13. r
Algorithm 5.3 guaranteesa topology that does not contain any cycles. The
proof for this is quite straightforward. If a cycle exists, then there must be at
least one arc connecting the lower-ordered nodes to the higher-ordered nodes,
and at least another arc connecting the higher-ordered nodes to the lower-
ordered nodes. Since Algorithm 5.3 prevents any arc from connecting the
244 Chapter 5 Classification: Alternative Techniques

lower-ordered nodes to the higher-ordered nodes, there cannot be any cycles

in the topology.
Nevertheless, the network topology may change if we apply a different or-
dering scheme to the variables. Some topology may be inferior because it
produces many arcs connecting between different pairs of nodes. In principle,
we may have to examine all d! possibleorderings to determine the most appro-
priate topology, a task that can be computationally expensive. An alternative
approach is to divide the variables into causal and effect variables, and then
draw the arcs from each causal variable to its corresponding effect variables.
This approach easesthe task of building the Bayesian network structure.
Once the right topology has been found, the probability table associated
with each node is determined. Estimating such probabilities is fairly straight-
forward and is similar to the approach used by naiVe Bayes classifiers.

Example of Inferencing Using BBN

Supposewe are interested in using the BBN shown in Figure 5.13 to diagnose
whether a person has heart disease. The following cases illustrate how the
diagnosis can be made under different scenarios.

Case 1: No Prior Information

Without any prior information, we can determine whether the person is likely
to have heart disease by computing the prior probabilities P(HD : Yes) and
P(HD: No). To simplify the notation, let a € {Yes,No} denote the binary
values of Exercise and B e {Healthy,Unhealthy} denote the binary values
of Diet.

P(HD:ves) : ttP(Hn
: Y e s l E : ( t , D : P ) P ( E: a , D : 0 )
d13

: ttp(uo : yesl,O
: (t,D: ilP(E : a)P(D: g)
aR

0 . 2 5x 0 . 7x 0 . 2 5+ 0 . 4 5x 0 . 7x 0 . 7 5 + 0 . 5 5x 0 . 3x 0 . 2 5
+ 0.75x 0.3 x 0.75
0.49.

Since P(HD - no) - 1 - P(ttO : yes) : 0.51, the person has a slightly higher
chance of not getting the disease.
 5.3 Bayesian Classifiers 245

Case 2: High Blood Pressure

If the person has high blood pressure)we can make a diagnosis about heart
disease by comparing the posterior probabilities, P(HD : YeslBP : High)
against P(ttO : Nolnt : High). To do this, we must compute P(Be : High):

P (n e:H i g h ) : frl n r
: HighlHD:7) p( HD
:7)

: 0.85 x 0.49+ 0.2 x 0.51: 0.5185.

where 7 € {Yes, No}. Therefore, the posterior probability the person has heart
diseaseis

P(ge : HighlHD: Yes)P(HD: Yes)

P(Ho : yeslBp : High) :
P(ne : High)
0'85 x 0.49
:
ffi:o'8033'
Similarly, P(ttO : NolBe: High) - 1 - 0.8033 : 0.1967. Therefore, when a
person has high blood pressure, it increasesthe risk of heart disease.

Case 3: High Blood Pressure, Healthy Diet, and Regular Exercise

Supposewe are told that the person exercisesregularly and eats a healthy diet.
How does the new information affect our diagnosis? With the new information,
the posterior probability that the person has heart diseaseis

P(Ho: YeslBP: Hlgh,D: Heal-thy,E: Yes)

P(BP : HighlHD: Yes,D : Healthy,E : Yes
P(ne : HighlD : HeaIthV,E : Yes)
)l
x P(HD: YeslD : Healthy,E : Yes)

P(ee : HighlHD: Yes)P(HD: YeslD : Healthy,E : Yes)

D, flBe : HighlHD
: ?)P(HD:.ylD: Healthy,E : Yes)
0.85x 0.25
0.85x0.25+0.2x0.75
: 0.5862,

while the probability that the persondoesnot haveheart diseaseis

P(tt0 : Nolee: High,D : Healthy,E : yes) - 1 - 0.5862: 0.4138.

246 Chapter 5 Classification: Alternative Techniques

The model therefore suggests that eating healthily and exercising regularly
may reduce a person's risk of getting heart disease.

Characteristics of BBN

Following are some of the general characteristics of the BBN method:

1. BBN provides an approach for capturing the prior knowledge of a par-

ticular domain using a graphical model. The network can also be used
to encode causal dependenciesamong variables.

2 . Constructing the network can be time consuming and requires a large

amount of effort. However, once the structure of the network has been
determined, adding a new variable is quite straightforward.

3 . Bayesian networks are well suited to dealing with incomplete data. In-
stanceswith missing attributes can be handled by summing or integrat-
ing the probabilities over all possible values of the attribute.

4 . Becausethe data is combined probabilistically with prior knowledge,the

method is quite robust to model overfitting.

5.4 Artificial Neural Network (ANN)

The study of artificial neural networks (ANN) was inspired by attempts to
simulate biological neural systems. The human brain consists primarily of
nerve cells called neurons) linked together with other neurons via strands
of fiber called axons. Axons are used to transmit nerve impulses from one
neuron to another wheneverthe neurons are stimulated. A neuron is connected
to the axons of other neurons via dendrites, which are extensions from the
cell body of the neuron. The contact point between a dendrite and an axon is
called a synapse. Neurologists have discovered that the human brain learns
by changing the strength of the synaptic connection between neurons upon
repeated stimulation by the same impulse.
Analogous to human brain structure, an ANN is composed of an inter-
connected assemblyof nodes and directed links. In this section, we will exam-
ine a family of ANN models, starting with the simplest model called percep-
tron, and show how the models can be trained to solve classificationproblems.
 5.4 Artificial Neural Network (ANN) Z4T

5.4.1 Perceptron

Consider the diagram shown in Figure 5.14. The table on the left shows a data
set containing three boolean variables (*t, ,r, 13) and an output variable, gr,
that takes on the value -1 if at least two of the three inputs arezero, and +1
if at least two of the inputs are greater than zero.

X1 X2 x3 v Input
-1 nodes
1 0 0
1 0 1 1
1 1 0 1
1 1 1 1
0 0 "l -1
0 1 0 -1
0 1 1 1
0 0 0 -1

(a)
(a) Data
Dataset.
set. (b) perceptron.

Figure5.14.Modeling
a boolean usinga perceptron.
function

Figure 5.14(b) illustrates a simple neural network architecture known as a

perceptron. The perceptron consistsof two types of nodes: input nodes, which
are used to represent the input attributes, and an output node, which is used
to represent the model output. The nodes in a neural network architecture
are commonly known as neurons or units. In a perceptron, each input node is
connectedvia a weighted link to the output node. The weighted link is used to
emulate the strength of synaptic connection between neurons. As in biological
neural systems, training a perceptron model amounts to adapting the weights
of the links until they fit the input-output relationships of the underlying data.
A perceptron computes its output value, Q, by performing a weighted sum
on its inputs, subtracting a bias factor t from the sum, and then examining
the sign of the result. The model shown in Figure 5.14(b) has three input
nodes, each of which has an identical weight of 0.3 to the output node and a
bias factor of f : 0.4. The output computed by the model is

I t, if 0.311* o.Jr2f 0.313- 0.4> o;

t"i - (5.21)
" if o.Jq * 0.Jr2 * 0.313- 0.4 < 0.
[-1,
248 Chapter 5 Classification: Alternative Techniques

For example, if 11 : \, fi2 : L, rz : 0, then 0 : +1 because0.32r * 0.312+

0 . 3 1 3- 0 . 4 i s p o s i t i v e . O n t h e o t h e r h a n d , i f r y : 0 , f r 2 : L,frB: 0, then
A: -t becausethe weighted sum subtracted by the bias factor is negative.
Note the difference between the input and output nodes of a perceptron.
An input node simply transmits the value it receivesto the outgoing link with-
out performing any transformation. The output node, on the other hand, is a
mathematical device that computes the weighted sum of its inputs, subtracts
the bias term, and then produces an output that depends on the sign of the
resulting sum. More specifically, the output of a perceptron model can be
expressed mathematicallv as follows:

- t),
0 : s i ' g n ( w d r d+ w d - t . x d - t + . . . + w z r z I w p 1 (5.22)

w h e r eu ) r t l r 2 t . . . , 1 1 )adr.e t h e w e i g h t so f t h e i n p u t l i n k s a n d 1 1 ,r 2 t . . . ) r d a r e
the input attribute values. The sign function, which acts as an activation
function for the output neuron, outputs a value *1 if its argument is positive
and -1 if its argument is negative. The perceptron model can be written in a
more compact form as follows:

0: s i ' g n l w a r a l w d , - L r d -+r . . . + w y r l l u t o r o f : s i g n ( w ' x ) , (5.23)

where u)0: -t, frl: I, and w'x is the dot product betweenthe weight vector
w and the input attribute vector x.

Learning Perceptron Model

During the training phase of a perceptron model, the weight parameters w

are adjusted until the outputs of the perceptron become consistent with the
true outputs of training examples. A summary of the perceptron learning
algorithm is given in Algorithm 5.4.
The key computation for this algorithm is the weight update formula given
in Step 7 of the algorithm:

.\k+r):.f, + \(ur - g[r))"ni, (5.24)

where tr(k) is the weight parameter associatedwith the i'h input link after the
kth ileration, ) is a parameter known as the learning rate, and rii is the
value of lhe jth attribute of the training example 4. The justification for the
weight update formula is rather intuitive. Equation 5.24 shows that the new
weight ?r(k+l) is a combination of the old weight tr(ft) and a term proportional
 5.4 Artificial Neural Network (ANN) 249

Algorithm 5.4 Perceptron learning algorithm.

t: Let D: {(xr, yt) | i : I,2,. . .,1[] be the set of training examples.
2: Initialize the weight vector with random values, w(0)
3: repeat
4: for each training example (xt,llt) € D do
5: Compute the predicted output yjk)
6: for each weight wi do
7: Update the weight, .\k+r) :.:*) + )(g, - g[r))*0,.
8: end for
9: end for
10: until stopping condition is met

to the prediction error, (g - i). If the prediction is correct, then the weight
remains unchanged. Otherwise, it is modified in the following ways:

o If y : +1 and f - -1, then the prediction error is (a - 0) : 2. To

compensatefor the error, we need to increasethe value of the predicted
output by increasing the weights of all links with positive inputs and
decreasingthe weights of all links with negative inputs.

o If at: -1 and 0: lI, then (g -0): -2. To compensatefor the error,

we need to decreasethe value of the predicted output by decreasingthe
weights of all links with positive inputs and increasing the weights of all
Iinks with negative inputs.

In the weight update formula, Iinks that contribute the most to the error term
are the ones that require the largest adjustment. However, the weights should
not be changed too drastically becausethe error term is computed only for
the current training example. Otherwise, the adjustments made in earlier
iterations will be undone. The learning rate ), a parameter whose value is
between 0 and 1, can be used to control the amount of adjustments made in
each iteration. If ,\ is close to 0, then the new weight is mostly influenced
by the value of the old weight. On the other hand, if ) is close to 1, then
the new weight is sensitive to the amount of adjustment performed in the
current iteration. In some cases)an adaptive ) value can be used; initially, ,\
is moderately large during the first few iterations and then gradually decreases
in subsequentiterations.
The perceptron model shown in Equation 5.23 is linear in its parameters
w and attributes x. Because of this, the decision boundary of a perceptron,
which is obtained by setting 0 : 0, is a linear hyperplane that separatesthe
data into two classes,-1 and *1. Figure 5.15 shows the decision boundary
25O Chapter 5 Classification: Alternative Techniques

0.5
x1

Figure5.15.Perceptron
decision forthedatagivenin Figure
boundary 5.14,

obtained by applying the perceptron learning algorithm to the data set given in
Figure 5.14. The perceptron learning algorithm is guaranteedto convergeto an
optimal solution (as long as the learning rate is sufficiently small) for linearly
separable classification problems. If the problem is not linearly separable,
the algorithm fails to converge. Figure 5.16 shows an example of nonlinearly
separable data given by the XOR function. Perceptron cannot find the right
solution for this data becausethere is no linear hyperplane that can perfectly
separate the training instances.

1.5

Xe 0.5

-0.5L
-0.5 0.5 1.5
x1

Figure problem.
5.16.XORclassification Nolinear
hyperplane thetwoclasses.
canseparate
 5.4 Artificial Neural Network (ANN) 251

5.4.2 Multilayer Artificial Neural Network

An artificial neural network has a more complex structure than that of a
perceptron model. The additional complexities may arise in a number of ways:

1. The network may contain several intermediary layers between its input
and output layers. Such intermediary layers are called hidden layers
and the nodes embedded in these layers are called hidden nodes. The
resulting structure is known as a multilayer neural network (see Fig-
ure 5.17). In a feed-forward neural network, the nodes in one layer

Figure5.17.Example
of a multilayer
feed{onrvard neural
artificial (ANN).
network

are connected only to the nodes in the next layer. The perceptron is a
single-layer, feed-forward neural network because it has only one layer
of nodes-the output layer that performs complex mathematical op-
erations. fn a recurrent neural network, the links may connect nodes
within the same layer or nodes from one layer to the previous layers.

2. The network may use types of activation functions other than the sign
function. Examples of other activation functions include linear, sigmoid
(logistic), and hyperbolic tangent functions, as shown in Figure 5.18.
These activation functions allow the hidden and output nodes to produce
output values that are nonlinear in their input parameters.

These additional complexities allow multilayer neural networks to model

more complex relationships between the input and output variables. For ex-
252 Chapter 5 Classification: Alternative Techniques

Linear function Sigmoid tunction

tun

Sign function

-1.5
-0.5 0 0.s 1 -1 -05 0 0.5

Figure5,18.Types functions
ofactivation inartificial networks.
neural

ample, consider the XOR problem described in the previous section. The in-
stancescan be classifiedusing two hyperplanes that partition the input space
into their respective classes,as shown in Figure 5.19(a). Because a percep-
tron can create only one hyperplane, it cannot find the optimal solution. This
problem can be addressed using a two-layer, feed-forward neural network, as
shown in Figure 5.19(b). Intuitively, we can think of each hidden node as a
perceptron that tries to construct one of the two hyperplanes, while the out-
put node simply combines the results of the perceptrons to yield the decision
boundary shown in Figure 5.19(a).
To learn the weights of an ANN model, we need an efficient algorithm
that converges to the right solution when a sufficient amount of training data
is provided. One approach is to treat each hidden node or output node in
the network as an independent perceptron unit and to apply the same weight
update formula as Equation 5.24. Obviously, this approach will not work
because we lack a priori, knowledge about the true outputs of the hidden
nodes. This makes it difficult to determine the error term, (gr- f), associated
 5.4 Artificial Neural Network (ANN) 253

1.5

X z0 5

0- 0 . 5
0.5
x1

(a) Decision boundary. (b) Neural network topology.

Figure5.19.A two-layer,
feed{orward
neural fortheXORproblem.
network

with each hidden node. A methodology for learning the weights of a neural
network based on the gradient descent approach is presented next.

Learning the ANN Model

The goal of the ANN learning algorithm is to determine a set of weights w

that minimize the total sum of souared errors:

1 'A{
:
E(w) (5.25)
il,rr,-o)2.
;-l

Note that the sum of squared errors depends on w becausethe predicted class
f is a function of the weights assignedto the hidden and output nodes. Figure
5.20 shows an example of the error surface as a function of its two parameters,
to1 and 'u2. This type of error surface is typically encountered when Qi is a
linear function of its parameters, w. If we replace Q : w 'x into Equation
5.25, then the error function becomesquadratic in its parameters and a global
minimum solution can be easily found.
In most cases)the output of an ANN is a nonlinear function of its param-
eters because of the choice of its activation functions (e.g., sigmoid or tanh
function). As a result, it is no longer straightforward to derive a solution for
w that is guaranteed to be globally optimal. Greedy algorithms such as those
based on the gradient descent method have been developed to efficiently solve
the optimization problem. The weight update formula used by the gradient
254 Chapter 5 Classification: Alternative Techniques

E(w1,w2)
1.8

1.6

1.4

1.2

1
I

0'l
w1

5,20.Enorsudace
Figure E(w1,w2)fora two-parameter
model.

descentmethod can be written as follows:

W r; + ' l I ; - .OE(w)
A---^ (5.26)
r
dw.i

where ) is the Iearning rate. The secondterm states that the weight should be
increasedin a direction that reducesthe overall error term. However, because
the error function is nonlinear, it is possible that the gradient descentmethod
may get trapped in a local minimum.
The gradient descent method can be used to learn the weights of the out-
put and hidden nodes ofa neural network. For hidden nodes, the computation
is not trivial becauseit is difficult to assesstheir error Lerm,0Ef 0t17, without
knowing what their output values should be. A technique known as back-
propagation has been developed to address this problem. There are two
phasesin each iteration of the algorithm: the forward phase and the backward
phase. During the forward phase, the weights obtained from the previous iter-
ation are used to compute the output value of each neuron in the network. The
computation progressesin the forward direction; i.e., outputs of the neurons
at level k are computed prior to computing the outputs at level /c + 1. Dur-
ing the backward phase, the weight update formula is applied in the reverse
direction. In other words, the weights at level k + 7 arc updated before the
weights at level k are updated. This back-propagation approach allows us to
use the errors for neurons at layer k + t to estimate the errors for neurons at
Iaver k.
 5.4 Artificial Neural Network (ANN) 255

Design Issues in ANN Learning

Beforewe train a neural network to Iearn a classification task, the following
designissuesmust be considered.
1. The number of nodes in the input layer should be determined. Assign an
input node to each numerical or binary input variable. If the input vari-
able is categorical, we could either create one node for each categorical
value or encode the k-ary variable using flog2 k-l input nodes.

2. The number of nodes in the output layer should be established. For

a two-class problem, it is sufficient to use a single output node. For a
k-class problem, there are k output nodes.

3. The network topology (e.g., the number of hidden layers and hidden
nodes, and feed-forward or recurrent network architecture) must be se-
lected. Note that the target function representation depends on the
weights of the links, the number of hidden nodes and hidden layers, bi-
ases in the nodes, and type of activation function. Finding the right
topology is not an easy task. One way to do this is to start from a fully
connected network with a sufficiently large number of nodes and hid-
den layers, and then repeat the model-building procedure with a smaller
number of nodes. This approach can be very time consuming. Alter-
natively, instead of repeating the model-building procedure, we could
remove some of the nodes and repeat the model evaluation procedure to
select the right model complexity.

4. The weights and biases need to be initialized. Random assignmentsare

usually acceptable.

5. Tlaining examples with missing values should be removed or replaced

with most likely values.

5.4.3 Characteristics of ANN

Following is a summary of the general characteristics of an artificial neural

network:

1. Multilayer neural networks with at least one hidden layer are univer-
sal approximators; i.e., they can be used to approximate any target
functions. Since an ANN has a very expressivehypothesis space,it is im-
portant to choosethe appropriate network topology for a given problem
to avoid model overfitting.
256 Chapter 5 Classification: Alternative Techniques

2 . ANN can handle redundant features becausethe weights are automat-

ically learned during the training step. The weights for redundant fea-
tures tend to be very small.

.t. Neural networks are quite sensitive to the presenceof noise in the train-
ing data. One approach to handling noise is to use a validation set to
determine the generalization error of the model. Another approach is to
decreasethe weight by some factor at each iteration.

4 . The gradient descent method used for learning the weights of an ANN
often convergesto somelocal minimum. One way to escapefrom the local
minimum is to add a momentum term to the weight update formula.
tr
ir. Training an ANN is a time consuming process,especiallywhen the num-
ber of hidden nodes is large. Nevertheless,test examplescan be classified
rapidly.

5.5 Support Vector Machine (SVM)

A classification technique that has received considerableattention is support
vector machine (SVM). This technique has its roots in statistical learning the-
ory and has shown promising empirical results in many practical applications,
from handwritten digit recognition to text categorization. SVM also works
very well with high-dimensional data and avoids the curse of dimensionality
problem. Another unique aspect of this approach is that it representsthe deci-
sion boundary using a subset of the training examples,known as the support
vectors.
To illustrate the basic idea behind SVM, we first introduce the concept of
a maximal margin hyperplane and explain the rationale of choosing such
a hyperplane. We then describe how a linear SVM can be trained to explicitly
look for this type of hyperplane in linearly separable data. We conclude by
showing how the SVM methodology can be extended to non-linearly separable
data.

5.5.1 Maximum Margin Hyperplanes

Figure 5.21 shows a plot of a data set containing examples that belong to
two different classes,represented as squares and circles. The data set is also
linearly separable; i.e., we can find a hyperplane such that all the squares
reside on one side of the hyperplane and all the circles reside on the other
 D.D Support Vector Machine (SVM) 257

oo
II o
o
I
oo
T
oo
T
I
TT ooo
o
T lr OO

T TI oo
I
oo
o
o
Figure5.21.Possible
decision
boundaries
fora linearly dataset.
separable

side. However, as shown in Figure 5.21, there are infinitely many such hyper-
planes possible. Although their training errors are zerol there is no guarantee
that the hyperplaneswill perform equally well on previously unseenexamples.
The classifier must choose one of these hyperplanes to represent its decision
boundary, based on how well they are expected to perform on test examples.
To get a clearer picture of how the different choicesof hyperplanes affect the
generalization errors, consider the two decision boundaries, ,B1and 82, shown
in Figure 5.22. Both decision boundaries can separate the training examples
into their respective classeswithout committing any misclassification errors.
Each decision boundary B; is associatedwith a pair of hyperplanes, denoted
as bi1 and bi2, respectively. b61is obtained by moving a parallel hyperplane
away from the decisionboundary until it touchesthe closestsquare(s),whereas
b;2 is obtained by moving the hyperplane until it touches the closest circle(s).
The distance between these two hyperplanes is known as the margin of the
classifier. From the diagram shown in Figure 5.22,notice that the margin for
.B1 is considerably larger than that for Bz. In this example, 81 turns out to
be the maximum margin hyperplane of the training instances.

Rationale for Maximum Margin

Decision boundaries with large margins tend to have better generalization

errors than those with small margins. Intuitively, if the margin is small, then
258 Chapter 5 Classification: Alternative Technioues

+---- - - - - - - - -- - -- t
brr marginfor 81 bp

Figure5.22.Margin
ofa decision
boundary.

any slight perturbations to the decision boundary can have quite a significant
impact on its classification. Classifiersthat produce decision boundaries with
small margins are therefore more susceptible to model overfitting and tend to
generalizepoorly on previously unseen examples.
A more formal explanation relating the margin of a linear classifier to its
generalization error is given by a statistical learning principle known as struc-
tural risk minimization (SRM). This principle provides an upper bound to
the generalization error of a classifier (R) in terms of its training error (,R"),
the number of training examples (l/), and the model complexity otherwise
known as its capacity (h). More specifically,with a probability of 1 - q, the
generalization error of the classifier can be at worst

R1R.+r(*, (5.27)

where cp is a monotone increasing function of the capacity h. The preced-

ing inequality may seem quite familiar to the readers because it resembles
the equation given in Section 4.4.4 (on page 179) for the minimum descrip-
tion length (MDL) principle. In this regard, SRM is another way to express
generalizationerror as a tradeoff between training error and model complexity.
 o.o Support Vector Machine(SVM) 259

The capacity of a linear model is inversely related to its margin. Models

with small margins have higher capacities becausethey are more flexible and
can fit many training sets, unlike models with large margins. However, accord-
ing to the SRM principle, as the capacity increases,the generalization error
bound will also increase. Therefore, it is desirable to design linear classifiers
that maximize the margins of their decision boundaries in order to ensure that
their worst-casegeneralization errors are minimized. One such classifieris the
linear SVM, which is explained in the next section.

5.5.2 Linear SVM: Separable Case

A linear SVM is a classifier that searches for a hyperplane with the largest
margin, which is why it is often known as a maximal margin classifier. To
understand how SVM learns such a boundary, we begin with some preliminary
discussionabout the decision boundary and margin of a linear classifier.

Linear Decision Boundary

Consider a binary classification problem consisting of l/ training examples.

Each example is denoted by a tuple (*i,At) (i, : I,2,. . . ,ly'), where x1 :
(*nt.,*t2,...,nu)r correspond.sto the attribute set for the i,th example. By
convention, Iet At € {-1, 1} denote its class label. The decision boundary of a
linear classifier can be written in the followins form:

w.x*b:0, (5.28)

where w and b are parameters of the model.

Figure 5.23 shows a two-dimensional training set consisting of squaresand
circles. A decision boundary that bisects the training examples into their
respective classesis illustrated with a solid line. Any example located along
the decision boundary must satisfy Equation 5.28. For example, if xo and x6
are two points located on the decision boundary, then

\n/.Xo*b:0,
w'xbfb:0.

Subtracting the two equations will yield the following:

w. (xa- xr) : 0,
260 Chapter 5 Classification: Alternative Techniques

Figure5.23.Decision
boundary
andmargin
ofSVM.

where Xb - Xo is a vector parallel to the decision boundary and is directed

from xo to x6. Since the dot product is zero, the direction for w must be
perpendicular to the decision boundary, as shown in Figure 5.23.
For any square x" located above the decision boundary, we can show that

W'Xs lb:k, (5.2e)

where k > 0. Similarly, for any circle x. located below the decision boundary,
we can show that
w.Xc *b:k" (5.30)
where k' < 0. If we label all the squares as class f 1 and all the circles as
class -1, then we can predict the class label gr for any test example z in the
following way:
I l, if w.z*b>0;
(5.31)
':t -t. ifw.z+b<0.

Margin of a Linear Classifier

Consider the square and the circle that are closest to the decision boundary.
Since the square is located above the decision boundary, it must satisfy Equa-
tion 5.29 for some positive value k, whereas the circle must satisfy Equation
 5.5 Support Vector Machine (SVM) 261

5.30 for some negative value kt. We can rescale the parameters w and b of
the decision boundary so that the two parallel hyperplanes b;1 and bi2 can be
expressed as follows:

b i 1: w . x * b : 1 , (5.32)
bn; w 'x*b--1. (5.33)

The margin of the decision boundary is given by the distance between these
two hyperplanes. To compute the margin, Iet x1 be a data point located on
br1and x2 be a data point on bi2, as shown in Figure 5.23. Upon substituting
these points into Equations 5.32 and 5.33, the margin d can be computed by
subtracting the second equation from the first equation:

w ' ( x r - x z ): 2
l l w lxl d : 2
,.
(5.34)
ll*ll'
Learning a Linear SVM Model

The training phase of SVM involves estimating the parameters w and b of the
decision boundary from the training data. The parameters must be chosenin
such a way that the following two conditions are met:

w.xi+b> lif y,;:1,

u/.xi+b< -7if yi --1. (5.35)

These conditions impose the requirements that all training instances from
class gr : 1 (i.e., the squares) must be located on or above the hyperplane
w.x* b: L, while those instancesfrom classU: -I (i.e.,the circles)must
be located on or below the hyperplane w .x * b - -1. Both inequalities can
be summarized in a more comoact form as follows:

gt(w. xi*b) ) 1, 'i:7,2,...,N. (5.36)

Although the preceding conditions are also applicable to any linear classi-
fiers (including perceptrons), SVM imposesan additional requirement that the
margin of its decision boundary must be maximal. Maximizing the margin,
however, is equivalent to minimizing the following objective function:

f r * r' 2: l l w l l 2 (5.37)
262 Chapter 5 Classification: Alternative Techniques

Definition 5.1 (Linear SVM: Separable Case). The learning task in SVM
can be formalized as the following constrained optimization problem:

mln
, -Il*ll'
subject
to ,it* i * ,, - ,, ,i: r,2,.. . ,N.

Since the objective function is quadratic and the constraints are linear in
the parameters w and b, this is known as a convex optimization problem,
which can be solved using the standard Lagrange multiplier method. Fol-
lowing is a brief sketch of the main ideas for solving the optimization problem.
A more detailed discussionis given in Appendix E.
First, we must rewrite the objective function in a form that takes into
account the constraints imposed on its solutions. The new objective function
is known as the Lagrangian for the optimization problem:

r, : ill*ll' - f ^,(r,t*'xi * b)- 1), (5.38)

where the parameters ); are called the Lagrange multipliers. The first term in
the Lagrangian is the same as the original objective function, while the second
term captures the inequality constraints. To understand why the objective
function must be modified, consider the original objective function given in
Equation 5.37. It is easy to show that the function is minimized when w : 0, a
null vector whose components are all zeros. Such a solution, however, violates
the constraints given in Definition 5.1 because there is no feasible solution
for b. The solutions for w and b are infeasible if they violate the inequality
constraints; i.e., if Ailw-xi+b) - 1 < 0. The Lagrangian given in Equation 5.38
incorporates this constraint by subtracting the term from its original objective
function. Assuming that .\; > 0, it is clear that any infeasible solution may
only increasethe value of the Lagrangian.
To minimize the Lagrangian, we must take the derivative of ,Lp with respect
to w and b and set them to zero:
l/
OL, - U^+ w : \-.
) Ai?Jtxi, (5.3e)
0w ^Lt
t-1

1V
oLo - U^+\ -
) A;tt;:U. (5.40)
ab ,Lt
i,:1
 5.5 Support Vector Machine (SVM) 263

Because the Lagrange multipliers are unknown, we still cannot solve for w and
b. If Definition 5.1 contains only equality instead of inequality constraints, then
we can use the ly' equations from equality constraints along with Equations
5.39 and 5.40 to find the feasible solutions for w, b, and ).;. Note that the
Lagrange multipliers for equality constraints are free parameters that can take
any values.
One way to handle the inequality constraints is to transform them into a
set of equality constraints. This is possible as long as the Lagrange multipliers
are restricted to be non-negative. Such transformation leads to the following
constraints on the Lagrange multipliers, which are known as the Karush-Kuhn-
T\rcker (KKT) conditions:

);)0, (5.41)
\u[sn(* 'xt * b) - 1] : O. (5.42)

At first glance, it may seem that there are as many Lagrange multipli-
ers as there are training instances. It turns out that many of the Lagrange
multipliers become zero after applying the constraint given in trquation 5.42.
The constraint states that the Lagrange multiplier Aamust be zero unless the
training instance x6 satisfies the equation At(w .xt I b) : t. Such training
instance, with )i ) 0, lies along the hyperplanes bi1 or b.i2and is known as a
support vector. taining instancesthat do not reside along these hyperplanes
have )6:0. Equations 5.39 and5.42 also suggestthat the parametersw and
b, which define the decision boundary, depend only on the support vectors.
Solving the preceding optimization problem is still quite a daunting task
becauseit involves a large number of parameters: w, b, and )a. The problem
can be simplified by transforming the Lagrangian into a function of the La-
grange multipliers only (this is known as the dual problem). To do this, we
first substitute Equations 5.39 and 5.40 into Equation 5.38. This will lead to
the following dual formulation of the optimization problem:
,A|I

L-n : 1 ,z-/^ ' - * Ls;\atuixi.xj. (5.43)

2 z-.J
i:1. i,i

The key differences between the dual and primary Lagrangians are as fol-
lows:
1. The dual Lagrangian involves only the Lagrange multipliers and the
training data, while the primary Lagrangian involves the Lagrange mul-
tipliers as well as parameters of the decision boundary. Nevertheless,the
solutions for both optimization problems are equivalent.
264 Chapter 5 Classification: Alternative Techniques

2. The quadratic term in Equation 5.43 has a negative sign, which means
that the original minimization problem involving the primary Lagrangian,
Lp, has turned into a maximization problem involving the dual La-
grangian, ,L2.
For large data sets, the dual optimization problem can be solved using
numerical techniques such as quadratic programming, a topic that is beyond
the scope of this book. Once the );'s are found, we can use Equations 5.39
and 5.42 to obtain the feasible solutions for w and b. The decision boundary
can be expressed as follows:

/N \
(f\yn*t'*)+b:0. (5.44)
\- /

b is obtained by solving Equation 5.42 for the support vectors. Becausethe )1's
are calculated numerically and can have numerical errors, the value computed
for b may not be unique. Instead it depends on the support vector used in
Equation 5.42. In practice, the averagevalue for b is chosento be the parameter
of the decision boundary.
Example 5.5. Consider the two-dimensional data set shown in Figure 5.24,
which contains eight training instances. Using quadratic programming, we can
solve the optimization problem stated in Equation 5.43 to obtain the Lagrange
multiplier .\6for each training instance. The Lagrange multipliers are depicted
in the last column of the table. Notice that only the fi.rst two instances have
non-zero Lagrange multipliers. These instances correspond to the support
vectors for this data set.
Let w : (1q,w2) and b denote the parameters of the decision boundary.
Using Equation 5.39, we can solve for w1 and w2 in the following way:

wr : -1
D \ n r o r o r : 6 5 . 5 6 2 L x 1 x 0 . 3 8 5 8 + 6 5 . 5 6 2x 1 x 0 ' 4 8 7 r : - 6 ' 6 4 .
z
\-, \;Apn:65.562L x 1x
w2 : ) 7 6 5 . 5 6 2 1 x- 1 x 0 . 6 1 1: - 9 ' 3 2 .
0 . 4 6 8+
2

The bias term b can be computed using Equation 5.42 for each support vector:

6(t) : 1-w.x1 :1- ( - 6 . 6 4 ) ( 0 . 3 8 5 3- ) ( - 9 . 3 2 ) ( 0 . 4 6 8 7 ) : 7 . 9 3 0 0 .

6(z) : - 1 - w . x 2 : - 1 - ( - 6 . 6 4 ) ( 0 . 4 8 7 1-) ( - 9 . 3 2 ) ( 0 . 6 1 1:) 7 . 9 2 8 9 .

Averaging these values, we obtain b : 7.93. The decision boundary corre-

sponding to these parameters is shown in Figure 5.24. r
 5.5 Support Vector Machine(SVM) 265

Lagrange
X1 X2 v Multiplier
0.3858 0.4687 1 65.5261
0.4871 0.611 -1 65.5261
0.9218 0.4103 -1 0
o.7382 0.8936 -1 0
0.1763 0.0579 1 0
o.4057 0.3529 1 0
0.9355 0.8132 -1 0
0.2146 0.0099 1 0

1
0.9
-6.64 x1 - 9.32 x2 + 7.93 = 0 tr
0.8 tr
0.7
0.6 tr
s 0.5
o
0.4 n
0.3
o
o.2
0.1
0
o
0 0.2 o.4 0.6 0.8 1
X1

Figure
5.24.Example
ofa linearly
separable
dataset.

Once the parameters of the decision boundary are found, a test instance z
is classified as follows:

/N \

f ( r ) : s i e n ( w ' z + b ) : s ' i s n ( f ^ u r u " tz + b ) .

\ d:l

It f (z): 1, then the test instance is classifiedas a positive class;otherwise, it

is classified as a negative class.
266 Chapter 5 Classification: Alternative Techniques

5.5.3 Linear SVM: Nonseparable Case

Figure 5.25 shows a data set that is similar to Figure 5.22, except it has two
new examples, P and Q. Although the decision boundary 81 misclassifiesthe
new examples,while 82 classifiesthem correctly, this does not mean that .B2is
a better decisionboundary than 81 becausethe new examplesmay correspond
to noise in the training data. .B1 should still be preferred over 82 becauseit
has a wider margin, and thus, is less susceptible to overfitting. However, the
SVM formulation presented in the previous section constructs only decision
boundaries that are mistake-free. This section examines how the formulation
can be modified to learn a decision boundary that is tolerable to small training
errors using a method known as the soft margin approach. More importantly,
the method presentedin this section allows SVM to construct a linear decision
boundary even in situations where the classesare not linearly separable. To
do this, the learning algorithm in SVM must consider the trade-off between
the width of the margin and the number of training errors committed by the
Iinear decision boundary.

n
o\,,o
margin
tore2'l o
r oo
r al'', o
r r l'\,, o
llf'. o OO

OO

o
!t.
o
['? o
f; ;;;;i" ro,,
i,--'o',
Figure
5.25,Decision
boundary
ofSVMforthenonseparable
case.
 5.5 Support Vector Machine (SVM) 267

w.x+b=0
1.2
trtrtr
tr
t1
LI
0.8
tr

N
X .w.x+b=-1+(

-0.5 0 0.5 1 1.5

x1

Figure
5,26.Slack fornonseparable
variables data.

While the original objective function given in Equation 5.37 is still appli-
cable, the decision boundary .B1 no longer satisfies all the constraints given
in Equation 5.36. The inequality constraints must therefore be relaxed to ac-
commodate the'nonlinearly separable data. This can be done by introducing
positive-valued slack variables ({) into the constraints of the optimization
problem, as shown in the following equations:

v/'xi+b>7-€n ify6-I,
w.xi+b<-1+€, if ya: -1, (5.45)

where Vz : {6 > 0.
To interpret the meaning of the slack variables {a, consider the diagram
shown in Figure 5.26. The circle P is one of the instances that violates the
constraintsgiven in Equation 5.35. Let w.x* b: -7 *{ denote a line that
is parallel to the decision boundary and passesthrough the point P. It can be
shown that the distance between this line and the hyperplane w'x * b: -L
is {/llwll. Thus, { provides an estimate of the error of the decision boundary
on the training example P.
In principle, we can apply the same objective function as before and impose
the conditions given in Equation 5.45 to find the decision boundary. However,
268 Chapter 5 Classification: Alternative Techniques

T\
,ll \
\
o
a\, o
ll
t.,, I
\,I I
I

I r-\' . r

I ll.
P

T I og\
I o
I
o
Figure5.27.A decision
boundary
thathasa widemargin
butlargetraining
error.

since there are no constraints on the number of mistakes the decisionboundary

can make, the learning algorithm may find a decision boundary with a very
wide margin but misclassifies many of the training examples, as shown in
Figure 5.27. To avoid this problem, the objective function must be modified
to penalize a decision boundary with large values of slack variables. The
modified objective function is given by the following equation:

N
.
llwll2
,f(*):#* c(D,to)*,
i,:L

where C and k are user-specified parameters representing the penalty of mis-

classifying the training instances. For the remainder of this section, we assume
k : I to simplify the problem. The parameter C can be chosenbased on the
model's performance on the validation set.
It follows that the Lagrangian for this constrained optimization problem
can be written as follows:

.l/ N ,^/
L p : | l lOw l l 2 +' c* t FL \,1 - S-. .
- r . -) - S-\
(5.46)
rl "ll LAt|at(w.xi +b) 1+€ri \u&,
i:7 i:I i:l

where the first two terms are the objective function to be minimized, the third
term representsthe inequality constraints associatedwith the slack variables,
 D.O Support Vector Machine (SVM) 269

and the last term is the result of the non-negativity requirements on the val-
ues of {,;'s. F\rrthermore, the inequality constraints can be transformed into
equality constraints using the following KKT conditions:

€r)0, )r20, h)0, (5.47)

l ;{sr(*. x; * b)- 1 + €,} : 0, (5.48)
ltt€t :0- (5.4e)

Note that the Lagrange multiplier ,\; given in Equation 5.48 is non-vanishing
only if the training instance resides along the lines w'x, * b : il or has
€r > O. On the other hand, the Lagrange multipliers;.ri given in Equation 5.49
arezero for any training instances that are misclassified(i.e., having {z > 0).
Setting the first-order derivative of ,L with respect to w, b, and {; to zero
would result in the following equations:

-^r .lr
AL s-.
: t \&r.ru. (5,50)
'' -
) AiU&ii:u 1uj
A u tr: /-t
i:7
N .lr
AL : - t \ r a r : Q
aI)nr,:0. (551)
0b i:t i:l
AL - (5"52)
C - \t - t-tt- Q 4 \tl l.tt: C.
otr,
Substituting Equations 5.50, 5.51, and 5.52 into the Lagrangian will pro-
duce the following dual Lagrangian:

1.----
Lp : iD,xoxtoiaixi.xi + Cl€t
z-
L,J z

-\ ^o{r,(\^tot*0. xi f b) - 1 + €,}

-\,-\;)€'

Nl

: Ir1 - : Ar\ilrlixi' xi, (5.53)

1/ 2,t
i:I i,i

which turns out to be identical to the dual Lagrangian for linearly separable
data (see Equation 5.40 on page 262). Nevertheless, the constraints imposed
27O Chapter 5 Classification: Alternative Techniques

on the Lagrange multipliers )1's are slightly different those in the linearly
separable case. In the linearly separable case, the Lagrange multipliers must
be non-negative, i.e., )e > 0. On the other hand, Equation 5.52 suggeststhat
)z should not exceed C (since both pri and )a are non-negative). Therefore,
the Lagrange multipliers for nonlinearly separable data are restricted to 0 (
\r<C.
The dual problem can then be solved numerically using quadratic pro-
gramming techniques to obtain the Lagrange multipliers .\a. These multipliers
can be replaced into Equation 5.50 and the KKT conditions to obtain the
parameters of the decision boundary.

5.5.4 Nonlinear SVM

The SVM formulations described in the previous sections construct a linear de-
cision boundary to separatethe training examplesinto their respectiveclasses.
This section presents a methodology for applying SVM to data sets that have
nonlinear decision boundaries. The trick here is to transform the data from its
original coordinate space in x into a new space O(x) so that a linear decision
boundary can be used to separate the instances in the transformed space. Af-
ter doing the transformation, we can apply the methodology presented in the
previous sections to find a linear decision boundary in the transformed space.

Attribute TYansformation

To illustrate how attribute transformation can lead to a linear decision bound-

ary, Figure 5.28(a) shows an example of a two-dimensional data set consisting
of squares (classifiedas A - 1) and circles (classifiedas A - -1). The data set
is generated in such a way that all the circles are clustered near the center of
the diagram and all the squares are distributed farther away from the center.
Instances of the data set can be classifiedusing the following equation:

lfM)0.2,
a ( r t , r z ): otherwise.
(5.54)
{1,
The decision boundary for the data can therefore be written as follows:

M-0.2,
which can be further simplified into the following quadratic equation:

*?-q+"3-rz:-0.46.
 o.o Support Vector Machine (SVM) 27L

tr

-{1 d
tr
tr
I -0 15

tr
tr

&\'
x?-x,

(a) Decision boundary in the original (b) Decision boundary in the

two-dimensional soace. formed soace.

Figure5.28.Classifying
datawitha nonlinear
decision
boundary.

A nonlinear transformation O is needed to map the data from its original

feature space into a new space where the decision boundary becomes linear.
Supposewe choosethe following transformation:

Q: (q,r2) - (r?,*Z,t[2q,rtr2,t\. Io.ooJ

In the transformed space, we can find the parameters w : (t10, lrrt ..., w+)
such that:
w4r21t .s*f + uz{2q + wrrtr2t tr.rs: fl.
For illustration purposes, let us plot the graph of r/ - t2 versus rl - q for
the previously given instances. Figure 5.28(b) shows that in the transformed
space,all the circles are located in the lower right-hand side of the diagram. A
Iinear decisionboundary can therefore be constructed to separatethe instances
into their respective classes.
One potential problem with this approach is that it may suffer from the
curse of dimensionality problem often associatedwith high-dimensional data.
We will show how nonlinear SVM avoids this problem (using a method known
as the kernel trick) later in this section.

Learning a Nonlinear SVM Model

Although the attribute transformation approach seems promising, it raises

several implementation issues. First, it is not clear what type of mapping
272 Chapter 5 Classification: Alternative Techniques

function should be used to ensure that a linear decision boundary can be

constructed in the transformed space. One possibility is to transform the data
into an infinite dimensional space,but such a high-dimensional spacemay not
be that easy to work with. Second,even if the appropriate mapping function is
known, solving the constrained optimization problem in the high-dimensional
feature space is a computationally expensive task.
To illustrate these issuesand examine the ways they can be addressed,let
us assume that there is a suitable function, O(x), to transform a given data
set. After the transformation, we need to construct a linear decision boundary
that will separatethe instancesinto their respectiveclasses.The linear decision
boundary in the transformed spacehas the following form: w.O(*) * b: 0.
Definition 5.2 (Nonlinear SVM). The learning task for a nonlinear SVM
can be formalized as the following optimization problem:

-,n ll:Yll'
w2

subjectto A i ( w . A ( x 6 )+ b ) > 1 , i:I,2,...,N.

Note the similarity between the learning task of a nonlinear SVM to that
of a linear SVM (seeDefinition 5.1 on page 262). The main difference is that,
instead of using the original attributes x, the learning task is performed on the
transformed attributes O(x). Following the approach taken in Sections 5.5.2
and 5.5.3 for linear SVM, we may derive the following dual Lagrangian for the
constrained optimization problem:

LD :1^, - . a(xr)
),;Aisiyie(xz) (5.56)
I f
zrJ

Once the );'s are found using quadratic programming techniques, the param-
eters w and b can be derived using the following equations:

-: )'iY';Q(x) l o . o/ /
T
+ b)- 1}:0,
r,;{s,,(I\1y1a(x).o(*r) (5.58)
J
 5.5 Support Vector Machine (SVM) 273

which are analogousto Equations 5.39 and 5.40 for linear SVM. Finally, a test
instance z canbe classifiedusing the following equation:

'o (z) + b ) : sisn(i ^,r,*(x1)' o(z).

f ( ") : si 'e n (w ( 5.5e)
,

Except for Equation 5.57, note that the rest of the computations (Equa-
tions 5.58 and 5.59) involve calculating the dot product (i.e., similarity) be-
tween pairs of vectors in the transformed space,O(*r) 'O("i). Such computa-
tion can be quite cumbersome and may suffer from the curse of dimensionality
problem. A breakthrough solution to this problem comes in the form ,rf a
method known as the kernel trick.

I(ernel TYick

The dot product is often regarded as a measure of similarity between two

input vectors. For example, the cosine similarity described in Section 2.4.5
on page 73 can be defined as the dot product between two vectors that are
normalized to unit length. Analogously, the dot product O(x6)'O(*;) can also
be regarded as a measure of similarity between two instances, x2 and x3, in
the transformed space.
The kernel trick is a method for computing similarity in the transformed
spaceusing the original attribute set. Consider the mapping function O given
in Equation 5.55. The dot product between two input vectors u and v in the
transformed space can be written as follows:

O(u). O(v) : t/2ut,Jiu2,L)'(u?,r|,

(u?,u\, rtrr, rtur,t1
: + + 2upt * 2uzuz
l7
"lul "|rl
: (u.v + 1)2. (5.60)

This analysis shows that the dot product in the transformed space can be
expressedin terms of a similarity function in the original space:

K ( u , v ) : O ( u ). O ( " ) : ( u . v + 1 ) 2 (5.61)

The similarity function, K, which is computed in the original attribute space'

is known as the kernel function. The kernel trick helps to address some
of the concelns about how to implement nonlinear SVM. First, we do not
have to know the exact form of the mapping function (D because the kernel
274 Chapter 5 Classification: Alternative Techniques

functions used in nonlinear SVM must satisfy a mathematical principle known

as Mercerts theorem. This principle ensuresthat the kernel functions can
always be expressed as the dot product between two input vectors in some
high-dimensional space. The transformed space of the SVM kernels is called
a reproducing kernel Hilbert space (RKHS). Second, computing the
dot products using kernel functions is considerably cheaper than using the
transformed attribute set O(x). Third, since the computations are performed
in the original space,issuesassociatedwith the curseof dimensionality problem
can be avoided.
Figure 5.29 showsthe nonlinear decisionboundary obtained by SVM using
the polynomial kernel function given in Equation 5.61. A test instance x is
classifiedaccording to the following equation:

rL
. ,\-, .a@) + b)
f (z) : sisn(\\siQ(xi)
t:I
n
: . ,\.-.
sign(\),igiK(xi.z) + b)
;-1

n
.
k A i \ x i ' z + I ) ' + 0" ) ,
z\- ' , .',
: slgn\) (5.62)
1,: I

where b is the parameter obtained using Equation 5.58. The decisionboundary

obtained by nonlinear SVM is quite closeto the true decision boundary shown
in Figure 5.28(a).

Mercerts Theorem

The main requirement for the kernel function used in nonlinear SVM is that
there must exist a corresponding transformation such that the kernel function
computed for a pair of vectors is equivalent to the dot product between the
vectors in the transformed space. This requirement can be formally stated in
the form of Mercer's theorem.

Theorem 5.1 (Mercer's Theorem). ,4 kernel functi,on K can be erpressed

AS
K(u,a): O(z) 'O(r)
i,f and only i,f, for any functi,on g(r) such that I g(n)2dris fini,te, then

*,*,s) s@)g@)dn da>- 0.

|
 5.5 Support Vector Machine(SVM) 275

1
tr
0.9
tr
tr
0.8

o.7 try
0.6

x u,c

0.4

0.3

0.2
tr
0.1 tr
tr
0
o.7
xl

Figure5.29.Decision produced
boundary SVMwithpolynomial
bya nonlinear kernel.

Kernel functions that satisfy Theorem 5.1 are called positive definite kernel
functions. Examples of such functions are listed below:

K(*,y) :(x.y+l)P (5.63)

K(*, y) : / {2"2) (5.64)
"-ll*-vll2
K(*,y) :tanh(kx'y-d) (5.65)

Example 5.6. Consider the polynomial kernel function given in Equation

5.63. Let 9(z) be a function that has a finite L2 rrorm, i.e., -[ g(x)2dx < a.

f
+ l)ee(x)
s(Y)dxdY
J 8'Y
P /\
"
: / t (l )t*.y)i g (*)g (y)d .xd,y
J 7_o\x/'

: (1)I (',J, ) [r""lo'(rzaz)o"('zas)o"

f ,-8,
276 Chapter 5 Classification: Alternative Techniques

:\-\-
LL (,)(-,:, '?"7' s(rt'r2' )d'r1d'"
i:O at,az,... )[ l ]'

Because the result of the integration is non-negative, the polynomial kernel

function therefore satisfies Mercer's theorem. r

5.5.5 Characteristics of SVM

SVM has many desirable qualities that make it one of the most widely used
classificationalgorithms. Following is a summary of the general characteristics
of SVM:

1. The SVM learning problem can be formulated as a convex optimization

problem, in which efficient algorithms are available to find the global
minimum of the objective function. Other classification methods, such
as rule-based classifiers and artificial neural networks, employ a greedy-
based strategy to search the hypothesis space. Such methods tend to
find only locally optimum solutions.

2. SVM performs capacity control by maximizing the margin of the decision

boundary. Nevertheless, the user must still provide other parameters
such as the type of kernel function to use and the cost function C for
introducing each slack variable.

3. SVM can be applied to categorical data by introducing dummy variables

for each categorical attribute value present in the data. For example, if
Marital Status has three values{Single, Married, Divorced}, we can
introduce a binary variable for each of the attribute values.

4. The SVM formulation presented in this chapter is for binary class prob-
lems. Some of the methods available to extend SVM to multiclass orob-
lems are presented in Section 5.8.

5.6 Ensemble Methods

The classification techniques we have seen so far in this chapter, with the ex-
ception of the nearest-neighbor method, predict the class labels of unknown
examples using a single classifier induced from training data. This section
presents techniques for improving classification accuracy by aggregating the
predictions of multiple classifiers. These techniques are known as the ensem-
ble or classifier combination methods. An ensemble method constructs a
 5.6 Ensemble Methods 277

set of base classifiers from training data and performs classificationby taking
a vote on the predictions made by each base classifier. This section explains
why ensemble methods tend to perform better than any single classifier and
presents techniques for constructing the classifier ensemble.

5.6.1 Rationale for Ensemble Method

The following example illustrates how an ensemble method can improve a

classifier'sperformance.
Example 5.7. Consider an ensembleof twenty-five binary classifiers,each of
which has an error rate of e : 0.35. The ensembleclassifier predicts the class
Iabel of a test example by taking a majority vote on the predictions made
by the base classifiers. If the base classifiersare identical, then the ensemble
will misclassifythe same examplespredicted incorrectly by the base classifiers.
Thus, the error rate of the ensembleremains 0.35. On the other hand, if the
base classifiersare independent-i.e., their errors are uncorrelated-then the
ensemblemakes a wrong prediction only if more than half of the base classifiers
predict incorrectly. In this case, the error rate of the ensemble classifier is

vensemDle
O a 1 -
- - e)25-t: 0'06, (5.66)

which is considerably lower than the error rate of the base classifiers. r
Figure 5.30 shows the error rate of an ensemble of twenty-five binary clas-
sifiers (e".,""-bre) for different base classifier error rates (e). The diagonal line
represents the case in which the base classifiersare identical, while the solid
line representsthe casein which the base classifiersare independent. Observe
that the ensemble classifier performs worse than the base classifiers when e is
larger than 0.5.
The preceding example illustrates two necessary conditions for an ensem-
ble classifier to perform better than a single classifier: (1) the base classifiers
should be independent ofeach other, and (2) the base classifiersshould do bet-
ter than a classifierthat performs random guessing. In practice, it is difficult to
ensure total independenceamong the base classifiers. Nevertheless,improve-
ments in classification accuracieshave been observed in ensemblemethods in
which the base classifiers are slightly correlated.
278 Chapter 5 Classification: Alternative Techniques

0.9

6
o
0.7
o u.o
a
o
o
o
o.4
E
o
@ UJ
c
uJ
0.2
0.1

Base classifiererror

Figure
5.30.Comparison
between
enors
ofbase
classifiers
anderrors
oftheensemble
classifier.

Step 1:
CreateMultiple
Data Sets

Step 2:
BuildMultiple
Classifiers

Step 3:
Combine
Classifiers

Figure5.31.A logicalview
oftheensemble
learning
method.

5.6.2 Methods for Constructing an Ensemble Classifier

A logical view of the ensemblemethod is presented in Figure 5.31. The basic
idea is to construct multiple classifiersfrom the original data and then aggre-
gate their predictions when classifying unknown examples. The ensemble of
classifierscan be constructed in many ways:
 5.6 Ensemble Methods 279

1 . By manipulating the training set. In this approach, multiple train-

ing sets are created by resampling the original data according to some
sampling distribution. The sampling distribution determines how lili<ely
it is that an example will be selected for training, and it may vary from
one trial to another. A classifieris then built from each training set using
a particular learning algorithm. Bagging and boosting are two exam-
ples of ensemble methods that manipulate their training sets. These
methods are described in further detail in Sections 5.6.4 and 5.6.5.

2 . By manipulating the input features. In this approach, a subset

of input features is chosen to form each training set. The subset can
be either chosen randomly or based on the recommendation of domain
experts. Some studies have shown that this approach works very well
with data sets that contain highly redundant features. Random forest,
which is described in Section 5.6.6, is an ensemblemethod that manip-
ulates its input features and uses decision trees as its base classifiers.

3 . By manipulating the class labels. This method can be used when the
number of classesis sufficiently large. The training data is transformed
into a binary class problem by randomly partitioning the class la,bels
into two disjoint subsets, Ag and A1. TYaining examples whose class
label belongs to the subset As are assignedto class 0, while those that
belong to the subset Al are assignedto class 1. The relabeled examples
are then used to train a base classifier. By repeating the class-relabeling
and model-building steps multiple times, an ensembleof base classifiers
is obtained. When a test example is presented,each base classifiet Ci is
used to predict its class label. If the test example is predicted as class
0, then all the classesthat belong to As will receive a vote. Conversely'
if it is predicted to be class 1, then all the classesthat belong to A1
will receive a vote. The votes are tallied and the class that receivesthe
highest vote is assignedto the test example. An example of this approach
is the error-correcting output coding method describedon page 307.

4.By manipulating the learning algorithm. Many learning algo-

rithms can be manipulated in such a way that applying the algoriithm
several times on the same training data may result in different models.
For example, an artificial neural network can produce different rnod-
els by changing its network topology or the initial weights of the links
between neurons. Similarly, an ensemble of decision trees can be con-
structed by injecting randomness into the tree-growing procedure. For
280 Chapter 5 Classification: Alternative Techniques

example, instead of choosing the best splitting attribute at each node,

we can randomly chooseone of the top k attributes for splitting.

The first three approachesare generic methods that are applicable to any
classifiers,whereasthe fourth approach depends on the type of classifierused.
The base classifiersfor most of these approachescan be generated sequentially
(one after another) or in parallel (all at once). Algorithm 5.5 shows the steps
needed to build an ensembleclassifier in a sequential manner. The first step
is to create a training set from the original data D. Depending on the type
of ensemble method used, the training sets are either identical to or slight
modifications of D. The size of the training set is often kept the same as the
original data, but the distribution of examples may not be identicall i.e., some
examples may appear multiple times in the training set, while others may not
appear even once. A base classifier Ci is then constructed from each training
set Da. Ensemble methods work better with unstable classifiers. i.e.. base
classifiersthat are sensitive to minor perturbations in the training set. Ex-
amples of unstable classifiersinclude decision trees, rule-based classifiers,and
artificial neural networks. As will be discussedin Section 5.6.3, the variability
among training examples is one of the primary sourcesof errors in a classifier.
By aggregating the base classifiersbuilt from different training sets, this may
help to reduce such types of errors.
Finally, a test example x is classified by combining the predictions made
by the base classifiersC;(x):

C. (*) : V ote(Ct(x), C2(x), . . ., Cn(*)).

The class can be obtained by taking a majority vote on the individual predic-
tions or by weighting each prediction with the accuracv of the base classifier.

Algorithm 5.5 General procedure for ensemblemethod.

l: Let D denote the original training data, k denote the number of base classifiers,
and 7 be the test data.
2: fori:ltokdo
3: Create training set, D4 ftom D.
4: Build a base classifier Ci from Di.
5: end for
6: for each test record r € T d,o
7 : C * ( r ) : V o t e ( C 1 ( x ) , C z ( * ) , . . . ,C r ( * ) )
8: end for
 5.6 Ensemble Methods 28'1.

5.6.3 Bias-Variance Decomposition

Bias-variance decomposition is a formal method for analyzing the prediction

error of a predictive model. The following example gives an intuitive explana-
tion for this method.
Figure 5.32 shows the trajectories of a projectile launched at a particular
angle. Suppose the projectile hits the floor surface at some location r, aL a
distance d away from the target position t. Depending on the force applied
to the projectile, the observed distance may vary from one trial to another.
The observed distance can be decomposed into several components. The first
component, which is known as bias, measures the average distance between
the target position and the Iocation where the projectile hits the floor. The
amount of bias depends on the angle of the projectile launcher. The second
component, which is known as variance, measutes the deviation between r
and the average position 7 where the projectile hits the floor. The variance
can be explained as a result of changes in the amount of force applied to the
projectile. Finally, if the target is not stationary, then the observed distance
is also affected by changes in the location of the target. This is considered the
noise component associatedwith variability in the target position. Putting
these components together, the averagedistance can be expressedas:

dy,e(a,t) : BiasB* VarianceTf Noiset, (5,67)

where / refers to the amount of force applied and 0 is the angle of the launcher.
The task of predicting the class label of a given example can be analyzed
using the same apploach. For a given classifier, some predictions may turn out
to be correct, while others may be completely off the mark. We can decompose
the expected error of a classifieras a sum of the three terms given in Equation
5.67, where expected error is the probability that the classifier misclassifiesa

{----} <-->
'Variance' 'Noise'
\_
'Bias'

Figure decomposition.
5.32.Bias-variance
282 Chapter 5 Classification: Alternative Techniques

given example. The remainder of this section examines the meaning of bias,
variance, and noise in the context of classification.
A classifier is usually trained to minimize its training error. However, to
be useful, the classifier must be abie to make an informed guess about the
class labels of examples it has never seen before. This requires the classifier to
generalizeits decision boundary to regions where there are no training exam-
ples available--a decision that depends on the design choice of the classifier.
For example, a key design issue in decision tree induction is the amount of
pruning needed to obtain a tree with low expected error. Figure 5.33 shows
two decision trees, ft and 72, that are generated from the same training data,
but have different complexities. 7z is obtained by pruning ?r until a tree with
maximum depth of two is obtained. Ty, on the other hand, performs very little
pruning on its decision tree. These design choices will introduce a bias into
the classifierthat is analogousto the bias of the projectile launcher described
in the previous example. In general, the stronger the assumptions made by
a classifier about the nature of its decision boundary, the larger the classi-
fier's bias will be. 72 therefore has a larger bias because it makes stronger
assumptions about its decision boundary (which is reflected by the size of the
tree) compared to ?r. other design choicesthat may introduce a bias into a
classifier include the network topology of an artificial neural network and the
number of neighbors consideredby a nearest-neighborclassifier.
The expected error of a classifier is also affected by variability in the train-
ing data becausedifferent compositions of the training set may lead to differ-
ent decision boundaries. This is analogous to the variance in r when different
amounts of force are applied to the projectile. The last component of the ex-
pected error is associatedwith the intrinsic noise in the target class. The target
class for some domains can be non-deterministic; i.e., instanceswith the same
attribute values can have different class labels. Such errors are unavoidable
even when the true decision boundary is known.
The amount of bias and variance contributing to the expected error depend
on the type of classifier used. Figure 5.34 compares the decision boundaries
produced by a decision tree and a 1-nearest neighbor classifier. For each
classifier, we plot the decision boundary obtained by "averaging" the models
induced from 100 training sets, each containing 100 examples. The true deci-
sion boundary from which the data is generated is also plotted using a dashed
line. The difference between the true decision boundary and the "averaged"
decision boundary reflects the bias of the classifier. After averaging the mod-
els, observe that the difference between the true decision boundary and the
decisionboundary produced by the l-nearest neighbor classifieris smaller than
 5.6 EnsembleMethods 283

15
d oo oo
o
x1 <-1.24 x1< 11.00 ooo
o oooo 3
o _of,;
)<2<9.25 o t f f i '*+
o o l *+ + +
o
o
-' + * + + *
o ++
*4
o + +*
+ + *****
O+ o
-5 10 15
(a) Decisiontree Tj

15
d o oo o oi
o
ooo
10 o oooo 3
o + ++
8 +
-^ oo + **oo
+ +
x1 <-1.24 x1< 11.00 o
o
+ +
o *a
* tl+*+ * +*
+O o t a
+ J**
o
(b) Decisiontree T2 -5 10 15

treeswithdifferent
Figure5.33.Twodecision complexities fromthesametraining
induced data.

the observed difference for a decision tree classifier. This result suggests that
the bias of a l-nearest neighbor classifier is lower than the bias of a decision
tree classifier.
On the other hand, the l-nearest neighbor classifier is more sensitive to
the composition of its training examples. If we examine the models induced
from different training sets, there is more variability in the decision boundary
of a l-nearest neighbor classifier than a decision tree classifier. Therefore, the
decision boundary of a decision tree classifier has a lower variance than the
l-nearest neighbor classifier.

5.6.4 Bagging

Bagging, which is also known as bootstrap aggregating, is a technique that

repeatedly samples (with replacement) from a data set according to a uniform
probability distribution. Each bootstrap sample has the same size as the origi-
nal data. Becausethe sampling is done with replacement,some instancesmay
appear several times in the same training set, while others may be omitted
from the training set. On average, a bootstrap sample D; contains appnoxi-
284 Chapter 5 Classification: Alternative Technioues

+ .
.,+ + r++
:.. {o
* ***** * #..*
T

' \*
... ** #"3 f ++, ' , ' oo
*H o T+
J+ r.,.

ii* ..
+ ji
o
- - - l )
,/ "":
"o
' ' I

+T
+
'
o

t ".. ,r+ -
?" ooo^
*Ja
o H ^ oo o o-
,-tjf".= .,
+
+ O
' 9O O OO
orifJe"" o".pdg o oo.p d
I
o o^o n o.'
a_'
o ^t o
oo(g ^
-30 L "O orc)
-30 +0L
-10

(a) Decision boundary for decision tree. (b) Decision boundary for l-nearest
neighbor.

Figure
5.34.Biasofdecision
treeand1-nearest
neighbor
classifiers.

Algorithm 5.6 Bagging algorithm.

1: Let k be the number of bootstrap samples.
2: fori,:Itokdo
3: Create a bootstrap sample of size N, Dt.
4: Tlain a base classifier Ca on the bootstrap sample Di.
5: end for
6: C. (r) : argmax 1,6(Ct1r1 : y) .
c
{d(') : 1 if its argument is true and 0 otherwise}.

mately 63% of the original training data becauseeach sample has a probability
1 - (1 - 1/l/)r' of being selected in each D,i.. If l/ is sufficiently large, this
probability convergesto 1- Ll"- 0.632. The basic procedure for bagging is
summarized in Algorithm 5.6. After training the k classifiers,a test instance
is assignedto the class that receivesthe highest number of votes.
To illustrate how bagging works, consider the data set shown in Table 5.4.
Let r denote a one-dimensionalattribute and y denote the classlabel. Suppose
we apply a classifier that induces only one-level binary decision trees, with a
test condition r ( k, where k is a split point chosen to minimize the entropy
of the leaf nodes. Such a tree is also known as a decision stump.
Without bagging, the best decisionstump we can produce splits the records
at either z < 0.35 or r I 0.75. Either way, the accuracy of the tree is at
 5.6 Ensemble Methods 285

Table5.4. Example anensemble

of datasetusedto construct classifiers.
of bagging
r 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 I
-1 -1 -1 -1 1 1
a I 1 1 1

most 70%. Suppose we apply the bagging procedure on the data set using
ten bootstrap samples. The examples chosen for training in each bagging
round are shown in Figure 5.35. On the right-hand side of each table, we also
illustrate the decision boundary produced by the classifier.
We classify the entire data set given in Table 5.4 by taking a majority
vote among the predictions made by each base classifier. The results of the
predictions are shown in Figure 5.36. Since the class labels are either -1 or
*1, taking the majority vote is equivalent to summing up the predicted values
of gr and examining the sign of the resulting sum (refer to the second to last
row in Figure 5.36). Notice that the ensembleclassifier perfectly classifiesall
ten examples in the original data.
The preceding example illustrates another advantage of using ensemble
methods in terms of enhancing the representation of the target function. Ilven
though each base classifier is a decision stump, combining the classifierscan
Iead to a decision tree of depth 2.
Bagging improves generalization error by reducing the variance of the base
classifiers. The performance of bagging depends on the stability of the base
classifier. If a base classifier is unstable, bagging helps to reduce the errors
associatedwith random fluctuations in the training data. If a base classifier
is stable, i.e., robust to minor perturbations in the training set, then the
error of the ensemble is primarily caused by bias in the base classifier. In
this situation, bagging may not be able to improve the performance of the
base classifiers significantly. It may even degrade the classifier's performance
becausethe effective size of each training set is about 37% smaller than the
original data.
Finally, since every sample has an equal probability of being selected, bag-
ging does not focus on any particular instance of the training data. It is
therefore less susceptible to model overfitting when applied to noisy data.

5.6.5 Boosting
Boosting is an iterative procedure used to adaptively change the distribution
of training examples so that the base classifierswill focus on examples that
are hard to classify. Unlike bagging, boosting assignsa weight to each training
286 Chapter 5 Classification: Alternative Techniques

Round1:
x 1 0 . 1 o.2 o.2 0.3 0.4 o.4 0.5 0.6 0.9 0.9 x<=0.35-=>y=1
v 1 1 1 1 -1 -1 -l 1 1 1 x>0.35==>y=-1

Round2:
x 1 0 . 1 o.2 0.3 0.4 0.5 0.8 0.9 1 1 1 x < = 0 . 6 5 = = 1y = |
v 1 1 1 -1 -1 1 1 1 1 1 x>0.65-=>y=1

Round3:
x I0.1 o.2 0.3 o.4 o.4 0.5 0.7 o.7 0.8 0.9 x < = 0 . 3 5 = = y y =|
v 1 1 1 -1 -1 -1 1 1 x>0.35==yy=-l

Round4:
x 1 0 . 1 0.1 o.2 0.4 o.4 0.5 0.5 0.7 0.8 0.9 x < = 0 . 3 = = 1y = l
v 1 1 1 -1 -1 1 -1 1 1 1 x v Q . t = = 1y = - 1

Bagging Round 5:
x 0.1 0.1 0.2 0.5 0.6 0.6 0.6 1 1 x<=0.35==>y=l
v 1 1 1 1 -1 I 1 1 1 x>0.35==>y=-1

Round6:
x o.2 0.4 0.5 0.6 0.7 o.7 o.7 0.8 0.9 1 x <= 0.75 ==> y - -1
v 1 1 1 1 1 1 1 1 1 x>0.75==>y=1

Round7:
x 0.1 o.4 o.4 0.6 0.7 0.8 0.9 0.9 0.9 1 x <= 0.75 ==> y = -1
v 1 -1 1 1 1 1 1 1 1 1 x>0.75==>y=1

Round8:
x 0.1 o.2 0.5 0.5 0.5 0.7 o.7 0.8 0.9 1 x <= 0.75 ==> y = -1
v 1 1 1 -1 -1 -t 1 1 1 1 x>0.75==2y=l

Round
x 0.1 0.3 0.4 o.4 0.6 o.7 o.7 0.8 1 1 x <= O.75==> y - -1
v 1 1 1 1 1 1 1 x>0.75==>y=-t

Round10:
x 0.1 0.1 0.1 0.1 0.3 0.3 0.8 x <= 0.05 ==> y = -1
0.8 0.9 0.9
v 1 x>0.05==>y=1
1 1 1 1 1 I 1 1

Figure5.35.Example
ofbagging.

example and may adaptively change the weight at the end of each boosting
round. The weights assigned to the training examples can be used in the
following ways:

1. They can be used as a sampling distribution to draw a set of bootstrap

samples from the original data.

2. They can be used by the base classifier to learn a model that is biased
toward higher-weight examples.
 5.6 Ensemble Methods 287

Round x=0,1x=0.2x=0.3x=0.4 x=0,5 x=0,6x=0.7x=0.8 x=0.9x=1.0

1 1 1 1 1 1 1 1 1 -1 1
2 1 1 1 1 1 1 1 1 1
3 1 1 1 -1 1 1 1 1 I

4 1 1 1 1 1 1 1 -1 1 -1
5 1 1 1 1 1 -1 1 1 1
-1 -1 -1 -1 ,l
6 1 1 1 1 1
'l -1 -1 1 1 1 1 1 1
I -1 -1 -1 1 1 1 -1 1 1 1
I 1 1 1 1 1 1 -1 1 1 1
'l
10 1 1 1 t
1 1 1 1 1

Sum 2 2 2 -o -6 -6 -6 2 2 2
Sign 1 1 1 1 1 1 1 1 1
TrueClass 1 1 1 1 1 1 1 1 1 1

of combining
Figure5.36.Example classifiers approach.
usingthebagging
constructed

This section describes an algorithm that uses weights of examples to de-

termine the sampling distribution of its training set. InitiallS the examples
are assigned equal weights, 1/N, so that they are equally likely to be chosen
for training. A sample is drawn according to the sampling distribution of the
training examples to obtain a new training set. Next, a classifier is induced
from the training set and used to classify all the examples in the original data.
The weights of the training examples are updated at the end of each boost-
ing round. Examples that are classified incorrectly will have their weights
increased, while those that are classified correctly will have their weights de-
creased. This forces the classifier to focus on examples that are difficult to
classify in subsequent iterations.
The following table shows the examples chosen during each boosting round.

Boosting (Round 1): 7 3 2 8 7 I 4 10 6 3

Boosting (Round 2): 5 4 I 4 2 5 1
F7
I 4 2
Boosting (Round 3): 4 4 8 10 4 5 4 6 3 4

Initially, all the examples are assigned the same weights. However, some ex-
amples may be chosen more than once, e.g., examples 3 and 7, becausethe
sampling is done with replacement. A classifier built from the data is then
used to classify all the examples. Suppose example 4 is difficult to classify.
The weight for this example will be increased in future iterations as it gets
misclassified repeatedly. Meanwhile, examples that were not chosen in the pre-
288 Chapter 5 Classification: Alternative Techniques

vious round, e.g., examples 1 and 5, also have a better chanceof being selected
in the next round since their predictions in the previous round were Iikely to
be wrong. As the boosting rounds proceed, examples that are the hardest to
classify tend to become even more prevalent. The final ensemble is obtained
by aggregating the base classifiersobtained from each boosting round.
Over the years, several implementations of the boosting algorithm have
been developed. These algorithms differ in terms of (1) how the weights of
the training examples are updated at the end of each boosting round, and (2)
how the predictions made by each classifierare combined. An implementation
called AdaBoost is explored in the next section.

AdaBoost

Let {(xy, yi) | j : 7,2,...,ltr} denote a set of N training examples.In the

AdaBoost algorithm, the importance of a base classifierCi dependson its error
rate, which is defined as

# tf '1r(co61t
+r,)], (5.68)

where I (p) : 1 if the predicate p is true, and 0 otherwise. The importance of

a classifier Ci is given by the following parameter,

1, /1- er\
ai:rt"\
* /

Note that a; has a large positive value if the error rate is closeto 0 and a large
negative value if the error rate is close to 1, as shown in Figure 5.37.
The a6 parameter is also used to update the weight of the training ex-
amples. To illustrate,let w[i) denote the weight assignedto example (*t,A)
during the jth boosting round. The weight update mechanism for AdaBoost
is given by the equation:

'[i) If Ci(x) : y,
.U+r1
' : * lexp-".r (5.6e)
z.i lexpo: i f C i @ ) I y i '

where Zi isthe normalization factor used to ensurethat !r.\l+t1 :1. The

weight update formula given in Equation 5.69 increasesthe weights of incor-
rectly classified examples and decreasesthe weights of those classified correctly.
 5.6 Ensemble Methods 289

el

E-1

-4

o.4
E

Figure5.37,Plotof o asa function errore.

oftraining

Instead of using a majority voting scheme, the prediction made by each

classifier C7 is weighted according to oi. This approach allows AdaBoost to
penalize models that have poor accuracy, e.g., those generated at the earlier
boosting rounds. In addition, if any intermediate rounds produce an error
rate higher than 50%, the weights are reverted back to their original uniform
values, ut: IlN, and the resampling procedure is repeated. The AdaBoost
algorithm is summarized in Algorithm 5.7.
Let us examine how the boosting approach works on the data set shown
in Table 5.4. Initially, all the examples have identical weights. Afber three
boosting rounds, the exampleschosenfor training are shown in Figure 5.38(a).
The weights for each example are updated at the end of each boosting round
using Equation 5.69.
Without boosting, the accuracy of the decision stump is, at best, 70%.
With AdaBoost, the results of the predictions are given in Figure 5.39(b).
The final prediction of the ensemble classifier is obtained by taking a weighted
average of the predictions made by each base classifi.er,which is shown in the
last row of Figure 5.39(b). Notice that AdaBoost perfectly classifiesall the
examples in the training data.
An important analytical result of boosting shows that the training error of
the ensembleis bounded bv the following expression:

€ensemble (5.70)
29O Chapter 5 Classification: Alternative Techniques

Algorithm 5.7 AdaBoost algorithm.

1: w : {r j : rlN I i : I,2,. . . ,,4f}. {Initialize the weights for all l/ examples.}
2: Let k be the number of boosting rounds.
3: for i,: I to k do
4'. Create training set Da by sampling (with replacement) from D according to w.
5: tain a base classifier Ci on Di.
6: Apply Ci to aIl examples in the original training set, D.
7: et : *lDrlu, 6(C{r) I Ai)l {Calculate the weighted error.}
8: if e; > 0.5 then
9: w : {wj : llN I j : t,2,...,N}. { R e s e tt h e w e i g h t sf o r a l l l / e x a m p l e s . }
10: Go back to Step 4.
11: end if
12: o.i: +h *.
13: Update the weight of each example according to Equation 5.69.
14: end for
15: C.(x) : argmax ll:ro,5(C1(*) : g))
a

where e,;is the error rate of each base classifier i. If the error rate of the base
classifier is less than 50%, we can write e; : 0.5 - li, where l; fir€asur€show
much better the classifieris than random guessing. The bound on the training
error of the ensemblebecomes

€ensemble (
T Uf - +":( exp(_ r."r) ( 5.7r )

If m < 7x for all i's, then the training error of the ensembledecreasesexpo-
nentially, which leads to the fast convergence of the algorithm. Nevertheless,
becauseof its tendency to focus on training examples that are wrongly classi-
fied, the boosting technique can be quite susceptible to overfitting.

5.6.6 Random Forests

Random forest is a classof ensemblemethods specifically designedfor decision

tree classifiers. It combines the predictions made by multiple decision trees,
where each tree is generated based on the values of an independent set of
random vectors, as shown in Figure 5.40. The random vectors are generated
from a fixed probability distribution, unlike the adaptive approach used in
AdaBoost, where the probability distribution is varied to focus on examples
that are hard to classify. Bagging using decision trees is a special case of
random forests, where randomnessis injected into the model-building process
 5.6 Ensemble Methods 29L

Boosting
Round1:
x 0.1 0.4 0.5 0.6 0.6 o.7 0.7 0.7 0.8 1
v 1 1 -1 -1 -1 -1 -1 -1 1 1

Boosting
Round2:
x 0.1 0.1 0.2 0.2 0.2 0.2 0.3 0.3 0.3 0.3
v 1 1 1 1 1 1 1 1 1 1

BoostingRound3:
x o.2 0.2 0.4 0.4 0.4 0.4 0.5 0.6 0.6 0.7
v 1 1 -1 1 -1 -1 -1 -1 -1 -1

(a)Training
recordschosenduringboosting

Round x=0.1 x=0.2 x=0,3 x=0,4 x=0.5 x=0.6 x=0.7 x=0,8 x=0.9 x=1,0
1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1 0.1
2 0 . 3 1 1 0.3't'l 0 . 3 t1 0 . 0 1 0 . 0 1 0 . 0 1 0.01 0 . 0 1 0.01 0.01
J 0.029 0.029 0.029 0.228 0.228 0.228 0.228 0.009 0.009 0.009
(b)Weightsof trainingrecords

Figure5.38.Example
of boosting.

by randomly choosing ly' samples,with replacement,from the original training

set. Bagging also uses the same uniform probability distribution to generate
its bootstrapped samples throughout the entire model-building process.
It was theoretically proven that the upper bound for generalization error
of random forests convergesto the following expression,when the number of
trees is sufficiently large.

G e n e r a l i z aetri roonr= - r y , (5,.72)

where p is the average correlation among the trees and s is a quantity that
measuresthe "strength" of the tree classifiers. The strength of a set of classi-
fiers refers to the averageperformance of the classifiers,where performance is
measured probabilistically in terms of the classifier'smargin:

margin,M(X,Y) : PlYo - Y)' - ry#P(Yo : Z), (5.73)

Z+Y

where YB is the predicted class of X according to a classifier built from some

random vector d. The higher the margin is, the more likely it is that the
292 Chapter 5 Classification: Alternative Techniques

Round Split Point Left Glass RightClass cx,

1 0.75 1 1 1.738
2 0.05 I 1 2.7784
.t 0.3 1 1 4.1195
(a)

Round x=0.1 x=0.2 x=0.3 x--0.4 x=0.5 x=0.6 x=0.7 x=0,8 x=0.9 x=l.0
1 1 1 1 1 1 -1 1 1 I

2 1 1 1 1 1 I 1 1
.t 1 1 1 1 1 1 1 1 1 1
Sum 5 . 1 6 5 . 1 6 5 . 1 6 -3.08 -3.08 -3.08 -3.08 0.397 0.397 0.397
Sign 1 1 1 1 1 1 1 1 1

(b)

5,39.Example
Figure olcombining
classifiers
constructed theAdaBoost
using approach.

Original
Trainingdata

Step2:
Userandom
vectorto
buildmultiple
ffi
+ +
decisiontrees
(c
Step3:
Combine
decisiontrees

Figure5.40.Random
forests.

classifiercorrectly predicts a given example X. Equation5.72 is quite intuitive;

as the trees become more correlated or the strength of the ensembledecreases,
the generalization error bound tends to increase. Randomization helps to
reduce the correlation among decision trees so that the generalization error of
the ensemblecan be improved.
 5.6 Ensemble Methods 293

Each decision tree uses a random vector that is generated from some fixed
probability distribution. A random vector can be incorporated into the t,ree-
growing process in many ways. The first approach is to randomly select F
input features to split at each node of the decision tree. As a result, instead of
examining all the available features, the decision to split a node is determined
from these selectedf. features. The tree is then grown to its entirety without
any pruning. This may help reduce the bias present in the resulting 1;ree.
Once the trees have been constructed, the predictions are combined using a
majority voting scheme. This approach is known as Forest-Rl, where RI refers
to random input selection. To increaserandomness,bagging can also be used
to generate bootstrap samples for Forest-Rl The strength and correlation of
random forests may depend on the size of F. If F is sufficiently small, then
the trees tend to become less correlated. On the other hand, the strength of
the tree classifier tends to improve with a larger number of features, F. As
a tradeoff, the number of features is commonly chosen to be F - logz d + L,
where d is the number of input features. Since only a subset of the features
needsto be examined at each node, this approach helps to significantly re<luce
the runtime of the algorithm.
If the number of original features d is too small, then it is difficult to choose
an independent set of random features for building the decision trees. One
way to increase the feature space is to create linear combinations of the input
features. Specifically, at each node, a new feature is generated by randomly
selecting .t of the input features. The input features are linearly combined
using coefficients generated from a uniform distribution in the range of [-1,
1]. At each node, F of such randomly combined new features are generated,
and the best of them is subsequentlyselectedto split the node. This approach
is known as Forest-RC.
A third approach for generating the random trees is to randomly select
one of the f' best splits at each node of the decision tree. This approach may
potentially generatetrees that are more correlated than Forest-Rl and Fonest-
RC, unless -F is sufficiently large. It also does not have the runtime savings of
Forest-Rl and Forest-RC because the algorithm must examine all the splitting
features at each node of the decision tree.
It has been shown empirically that the classification accuraciesof random
forests are quite comparable to the AdaBoost algorithm. It is also more robust
to noise and runs much faster than the AdaBoost algorithm. The classification
accuracies of various ensemble algorithms are compared in the next section.
294 Chapter 5 Classification: Alternative Techniques

Table
5.5.Comparing ofa decision
theaccuracy treeclassifier three
against methods.
ensemble
Data Set Number of Decision Bagging Boosting RF
(Attributes, Classes, Tlee (%) (%) (%) (%)
Records)
Anneal ( 3 9 ,6 , 8 9 8 ) 92.09 94.43 95.43 95.43
Australia ( 1 5 ,2 , 6 9 0 ) 85.51 87.10 85.22 85.80
Auto (26, 7, 205) 81.95 85.37 85.37 84.39
Breast ( 1 1 ,2 , 6 9 9 ) 95.r4 96.42 a7 2F, 96.14
Cleve (14,2, 303) 76.24 81.52 82.18 82.18
Credit (16,2, 690) 85.8 86.23 86.09 85.8
Diabetes (9,2,769) 72.40 76.30 73.18 ro.ro
German (21,2, 1000) 70.90 73.40 73.00 74.5
a, <n
Glass (r0, 7, 2L4) 67.29 76.r7 78.04
Heart \14,2,270) 80.00 81.48 80.74 83.33
Hepatitis (20,2, L55) 81.94 81.29 83.87 83.23
Horse (23, 2, 369) 85.33 85.87 8r.25 85.33
Ionosphere (35, 2, 351) 89.L7 92.02 93.73 93.45
Iris (5,3,150) 94.67 94.67 94,00 93.33
Labor (17, 2, 57) 78.95 84.2r 89,47 84.21
LedT (8, 10, 3200) 73.34 73.66 73.34 73.06
Lymphography ( 1 9 ,4 , 1 4 8 ) 77.03 79.05 85.14 82.43
Pima (9,2,769) 74.35 76.69 73,44 77.60
Sonar (6r, 2,208) 78.85 78.85 84.62 db.bd
Tic-tac-toe (10, 2, g5g) 83.72 93.84 98.54 95.82
Vehicle (r9, 4,846) 7r.04 74.rr 78.25 74.94
Waveform (22,3, 5000) 76.44 83.30 83.90 84.04
Wine (14, 3, 178) 94.38 96.07 97.75 97.75
Zoo (r7,7, Lol) 93.07 93.07 95.05 97.03

5.6.7 Empirical Comparison among Ensemble Methods

Table 5.5 shows the empirical results obtained when comparing the perfor-
mance of a decision tree classifier against bagging, boosting, and random for-
est. The base classifiersused in each ensemblemethod consist of fifty decision
trees. The classification accuraciesreported in this table are obtained from
ten-fold cross-validation. Notice that the ensemble cla,ssifi.ersgenerally out-
perform a single decision tree classifier on many of the data sets.

5.7 Class Imbalance Problem

Data sets with imbalanced class distributions are quite common in many real
applications. For example, an automated inspection system that monitors
products that come off a manufacturing assembly line may find that the num-
 5.7 Class Imbalance Problem 295

ber of defective products is significantly fewer than that of non-defectiveprod-

ucts. Similarly, in credit card fraud detection, fraudulent transactions are
outnumbered by legitimate transactions. In both of these examples, there is
a disproportionate number of instances that belong to different classes. The
degree of imbalance varies from one application to another-a manufacturing
plant operating under the six sigma principle may discover four defects in a
million products shipped to their customers, while the amount of credit card
fraud may be of the order of 1 in 100. Despite their infrequent occurrences,
a correct classification of the rare class in these applications ofben has greater
value than a correct classification of the majority class. However, becausethe
classdistribution is imbalanced, this presentsa number of problems to existing
classification algorithms.
The accuracy measure, which is used extensively to compare the perfor-
mance of classifiers, may not be well suited for evaluating models derived from
imbalanced data sets. For example, if lTo of the credit card transactions are
fraudulent, then a model that predicts every transaction as legitimate has an
accuracy of 99% even though it fails to detect any of the fraudulent activities.
Additionally, measuresthat are used to guide the learning algorithm (e.g., in-
formation gain for decision tree induction) may need to be modified to focus
on the rare class.
Detecting instancesof the rare classis akin to finding a needlein a haystack.
Becausetheir instancesoccur infrequently, models that describe the rare class
tend to be highly specialized. For example, in a rule-based classifier, the
rules extracted for the rare class typically involve a large number of attributes
and cannot be easily simplified into more general rules with broader coverage
(unlike the rules for the majority class). Such models are also susceptible
to the presenceof noise in training data. As a result, many of the existing
classification algorithms may not effectively detect instances of the rare class.
This section presentssome of the methods developedfor handling the class
imbalance problem. F irst, alternative metrics besidesaccuracy are introduced,
along with a graphical method called RoC analysis. we then describe how
cost-sensitiveIearning and sampling-basedmethods may be used to improve
the detection of rare classes.

5.7.t Alternative Metrics

Since the accuracy measure treats every class as equally important, it rnay
not be suitable for analyzing imbalanced data sets, where the rare class is
considered more interesting than the majority class. For binary classification,
the rare class is often denoted as the positive class,while the majority class is
296 Chapter 5 Classification:Alternative Techniques

Table matrix
5.6.A confusion fora binary problem
classification inwhich arenotequally
theclasses
imoortant.

Predicted Class
-f

Actual + /++ (TP) /+- (FN)

Class /-+ (FP) /-- (rN)

denoted as the negative class. A confusion matrix that summarizes the number
of instancespredicted correctly or incorrectly by a classificationmodel is shown
in Table 5.6.
The following terminology is often used when referring to the counts tab-
ulated in a confusion matrix:
o TYuepositive (TP) or fi1, which correspondsto the number of positive
examples correctly predicted by the classification model.

o Falsenegative (FN) or fi-, which correspondsto the number of positive

examples wrongly predicted as negative by the classification model.

o False positive (FP) or /-1, which correspondsto the number of negative

examples wrongly predicted as positive by the classification model.

o TYuenegative (TN) or /--, which correspondsto the number of negative

examples correctly predicted by the classification model.
The counts in a confusion matrix can also be expressedin terms of percentages.
The true positive rate (TPR) or sensitivity is defined as the fraction of
positive examples predicted correctly by the model, i.e.,

TPR:TPIQP+r'N).

Similarly, the true negative rate (?l/R) or specificity is defined as the

fraction of negative examples predicted correctly by the model, i.e.,

TNR: TN/(TN + Fp).

Finally, the false positive rate (FPR) is the fraction of negative examples
predicted as a positive class, i.e.,

FPR: FPIQN + FP),

 5.7 Class Imbalance Problem 297

while the false negative rate (F N R) is the fraction of positive examples

predicted as a negative class, i.e.,

FNR: FNI(TP + -F'N).

Recall and precision are two widely used metrics employed in applica-
tions where successfuldetection of one of the classesis consideredmore signif-
icant than detection of the other classes.A formal definition of these met,rics
is given below.

TP
Precision,p: (5.74)
TP+FP
TP
Recall, r : (5.75)
TP+FN

Precision determines the fraction of records that actually turns out to be

positive in the group the classifierhas declared as a positive class. The higher
the precision is, the lower the number of false positive errors committed by the
classifier. Recall measuresthe fraction of positive examplescorrectly predicted
by the classifier. Classifiers with large recall have very few positive examples
misclassified as the negative class. In fact, the value of recall is equivalent to
the true positive rate.
It is often possible to construct baseline models that maximize one metric
but not the other. For example, a model that declaresevery record to be the
positive class will have a perfect recall, but very poor precision. Conversely,
a model that assignsa positive class to every test record that matches one of
the positive records in the training set has very high precision, but low rer:all.
Building a model that maximizes both precision and recall is the key challenge
of classification algorithms.
Precision and recall can be summarized into another metric known as the
Fr meaSure.
2rp 2xTP
- (5.76)
r+p 2xTP+FP+f'l'r
In principle, .F'1representsa harmonic mean between recall and precision, i.e.,

,
al -
1 1.
III

i- p

The harmonic mean of two numbers z and grtends to be closer to the smaller
of the two numbers. Hence, a high value of F1-measure ensures that both
298 Chapter 5 Classification: Alternative Techniques

precision and recall are reasonably high. A comparison among harmonic, ge-
ometric, and arithmetic means is given in the next example.
Example 5.8. Consider two positive numbers a,: 1 and b: 5. Their arith-
m e t i c m e a ni s L r o : ( a + b ) 1 2 : 3 a n d t h e i r g e o m e t r i cm e a n i s F g : \ / o b :
2.236. Their harmonic mean is p,h: (2xlx5) 16 : L.667,which is closerto the
smaller value between o and b than the arithmetic and geometric means. r
More generally, the FB measure can be used to examine the tradeoff be-
tween recall and precision:

D
/ ^t

\lt- + tlrp
a\
(P'+r) xTP
,^q
(5.77)
r+p-p @2+t)rp+p2FP+Fr\/
Both precision and recall are special casesof FB by setting 0 :0 and B : 66,
respectively. Low values of B make Fp closer to precision, and high values
make it closer to recall.
A more general metric that captures .F-Bas well as accuracy is the weighted
accuracy measure,which is defined by the following equation:

wtTP -f utTN
Weighted &ccltro,c/: (5.78)
utTP * utzFP + unF N + u)4TN'

The relationship between weighted accuracy and other performance metrics is

summarized in the following table:

Measure lal 1.D2 wJ w4

Recall 1 I 0 0
Precision 1 0 1 0
FB 9 2+ t 13' 1 0
Accuracy 1 1 1 1
I

5.7.2 The Receiver Operating Characteristic Curve

A receiver operating characteristic (ROC) curve is a graphical approach for

displaying the tradeoll between true positive rate and false positive rate of a
classifier. In an ROC curve, the true positive rate (TPR) is plotted along the
g axis and the false positive rate (FPR) is shown on the r axis. Each point
along the curve corresponds to one of the models induced by the classifier.
Figure 5.41 shows the ROC curves for a pair of classifiers,M1 and M2.
 5.7 Class Imbalance Problem 299

0.9 Mz.'

4
o.7
o
(d
E u.o
c) M1
o u.c
o
(L
c)
5 0.4
[,/
I

,/, ,,
,/ .'
0.1
l'
0
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9
FalsePositiveRate

Figure
5.41.ROCcurves
fortwodifferent
classifiers.

There are several critical points along an ROC curve that have well-known
interpretations:
(TPR:O, FPR:0): Model predicts every instance to be a negative class.
(TPR:l, FPR:I): Model predicts every instance to be a positive class.
(TPR:l, FPR:O): The ideal model.

A good classificationmodel should be located as closeas possibleto the up-

per left corner of the diagram, while a model that makes random guessesshould
reside along the main diagonal, connectingthe points (TPR:0,FPR:0)
and (TPR: I,FPR:1). Random guessingmeans that a record is classi-
fied as a positive class with a fixed probability p, irrespective of its attribute
set. For example, consider a data set that contains na positive instances
and n- negative instances. The random classifier is expected to correctly
classify pna of the positive instances and to misclassify pn- of the negative
instances. Therefore, the TPR of the classifier is (pn)lnt : pt while its
FPRis (pn,)/p - p. Since theTPR and FPR are identical, the ROC curve
for a random classifier always reside along the main diagonal.
An ROC curve is useful for comparing the relative performance among
different classifiers. In Figure 5.4I, Ml is better than M2 when f'PE is less
300 Chapter 5 Classification: Alternative Techniques

than 0.36, while Mz is superior when FPIB is greater than 0.36. Clearly,
neither of these two classifiersdominates the other.
The area under the ROC curve (AUC) provides another approach for eval-
uating which model is better on average. If the model is perfect, then its area
under the ROC curve would equal 1. If the model simply performs random
guessing,then its area under the ROC curve would equal 0.5. A model that
is strictly better than another would have a larger area under the ROC curve.

Generating an ROC curve

To draw an ROC curve, the classifiershould be able to produce a continuous-

valued output that can be used to rank its predictions, from the most likely
record to be classified as a positive class to the least likely record. These out-
puts may correspond to the posterior probabilities generated by a Bayesian
classifier or the numeric-valued outputs produced by an artificial neural net-
work. The following procedure can then be used to generate an ROC curve:

1 . Assuming that the continuous-valuedoutputs are defined for the positive

class, sort the test records in increasing order of their output values.

2 . Select the lowest ranked test record (i.e., the record with lowest output
value). Assign the selected record and those ranked above it to the
positive class. This approach is equivalent to classifying all the test
records as positive class. Because all the positive examples are classified
correctly and the negative examples are misclassified,TPR: FPR: I.

3 . Select the next test record from the sorted list. Classify the selected
record and those ranked above it as positive, while those ranked below it
as negative. Update the counts of TP and FP by examining the actual
class label of the previously selected record. If the previously selected
record is a positive class, the TP count is decremented and the FP
count remains the same as before. If the previously selectedrecord is a
negative class, the FP count is decremented and TP cotnt remains the
same as before.

4. Repeat Step 3 and update theTP and FP counts accordingly until the
highest ranked test record is selected.

5 . PIot the TPR against FPR of the classifier.

Figure 5.42 shows an example of how to compute the ROC curve. There
are five positive examples and five negative examples in the test set. The class
 5.7 ClassImbalanceProblem 301-

5 4 4 o J 3 2 2 1 0
'I
5 c 4 4 2 0 0 0
TN 0 0 1 1 J 4 4 5 5
FN 0 1 1 2 2 2 3 3 4 5
TPR I 0.8 0.8 u.o 0.6 0.6 0.6 o.4 o.4 0.2 0
'I
FPR 1 0.8 0.8 06 o.4 o.2 o.2 0 0 0

Figure5.42.
Constructing
anROCcurve.

Figure5,43.BOCcurveforthedatashownin Figure
5.42.

Iabels of the test records are shown in the first row of the table. The second row
corresponds to the sorted output values for each record. For example, they
may correspond to the posterior probabilities P(*lx) generated by a naive
Bayes classifier. The next six rows contain the counts of.TP, FP,TN, and
f.ly', along with their corresponding TPR and FPR. The table is then filled
from left to right. Initially, all the records are predicted to be positive. Thus,
TP : FP :5 and TPR : FPR : 1. Next, we assign the test record with
the lowest output value as the negative class. Becausethe selected record is
actually a positive example, the TP count reduces from 5 to 4 and the FP
count is the same as before. The FPR and TPR are updated accordingly.
This process is repeated until we reach the end of the list, where TPR : 0
and FPR:0. The ROC curve for this example is shown in Figure 5.43.
3O2 Chapter 5 Classification: Alternative Techniques

5.7.3 Cost-Sensitive Learning

A cost matrix encodes the penalty of classifying records from one class as
another. LeL C(i,,j) denote the cost of predicting a record from class i as class
j. With this notation, C(1, -) is the cost of committing a false negative error,
while C(-, +) is the cost of generating a false alarm. A negative entry in the
cost matrix represents the reward for making correct classification. Given a
collection of ly' test records, the overall cost of a model M is

C{M): T P x C (+,+) + FP x C( - ,+) + F,^rx C( *, - )

+ T,^/x C(-, -). (5.79)

U n d e rt h e 0 / 1 c o s t m a t r i x , i . e . , C ( + , * ) : C ( - , - ) : 0 and C(+,,-1 :
C(-, +) : 1, it can be shownthat the overallcostis equivalentto the number
of misclassification errors.

Ct(M) : 0 x gP +7,^/) + 1 x (FP+f.lr) : N x Err, (5.80)

where Err is the error rate of the classiher.

Example 5.9. Consider the cost matrix shown in Table 5.7: The cost of
committing a false negative error is a hundred times larger than the cost
of committing a false alarm. In other words, failure to detect any positive
example is just as bad as committing a hundred false alarms. Given the
classificationmodels with the confusion matrices shown in Table 5.8, the total
cost for each model is

C t ( M ) : 1 5 0x ( - 1 ) * 6 0 x 1 + 4 0 x 1 0 0: 3 9 1 0 ,
Ct(Mz) : 250x (-1) * 5 x 1 -f 45 x 100: 4255.

Table5,7.Costmatrix
forExample
5.9.
Predicted Class
Ulass : * Class : -
Actual Class : f -l 100
Class UIaSS : - I (.)
 5.7 Class Imbalance Problem 3OB

Table5,8. Confusion
matrix
fortwoclassification
models.
Model M1 Predicted Olass Model M2 Predicted Class
UIASS + UIASS - Ulass + Ulass -
Actual Class + 150 40 Actual Class + 250 45
Class UIaSS - 60 250 Class Class- 200

Notice that despite improving both of its true positive and false positive counts,
model Mz is still inferior since the improvement comes at the expense of in-
creasing the more costly false negative errors. A standard accuracy measure
would have preferred model M2 over M1. I
A cost-sensitiveclassification technique takes the cost matrix into consid-
eration during model building and generatesa model that has the lowest cost.
For example, if false negative errors are the most costly, the learning algorithm
will try to reduce these errors by extending its decision boundary toward the
negative class, as shown in Figure 5.44. In this way, the generated model can
cover more positive examples, although at the expense of generating additional
false alarms.

Figure5.44.Modifying
thedecision (from81 to F2)toreduce
boundary thefalsenegative
errorsofa
classifier.

There are various ways to incorporate cost information into classification

algorithms. For example,in the context of decisiontree induction, the cost
3O4 Chapter 5 Classification: Alternative Techniques

information can be used to: (1) choosethe best attribute to use for splitting
the data, (2) determine whether a subtree should be pruned, (3) manipulate
the weights ofthe training records so that the learning algorithm convergesto
a decision tree that has the lowest cost, and (4) modify the decision rule at
each leaf node. To illustrate the last approach, let p(ilt) denote the fraction of
training records from class i that belong to the leaf node t. A typical decision
rule for a binary classification problem assignsthe positive class to node t if
the following condition holds.

p(+lt) > p(-lt)

:=+ e1lt)> (1-e(+lr))
==+ zp(+lt) > 1
+ p(+lt) > 0.5. (5.81)

The preceding decision rule suggeststhat the classlabel of a leaf node depends
on the majority class of the training records that reach the particular node.
Note that this rule assumesthat the misclassification costs are identical for
both positive and negative examples. This decision rule is equivalent to the
expressiongiven in Equation 4.8 on page 165.
Instead of taking a majority vote, a cost-sensitive algorithm assigns the
class label e to node t if it minimizes the following expression:

c (i,lt)-- D e(j lt)c (j, i.). (5.82)

J

In the case where C(+,+) : C(-,-) : 0, a leaf node f is assignedto the

positive class if:

-) >' p( - lt) C( - ,+)

p(+lt)C(+,
+ -) > (1- e(+lr))c(-,
p(+lt)C(+, +)
+ p (+l t)>
c(-,+) (5.83)
C ( - , + ) + C ( + ,- ) '
This expression suggeststhat we can modify the threshold of the decision rule
from 0.5 to C(-,+)lQe, +)+ C(+, -)) to obtain a cost-sensitiveclassifier.
If C(-,+) < C(+,-), then the threshold will be less than 0.5. This result
makes sensebecausethe cost of making a false negative error is more expensive
than that for generating a false alarm. Lowering the threshold will expand the
decision boundary toward the negative class, as shown in Figure 5.44.
 D.J Class Imbalance Problem 305

X1 X

(a) Without oversampling (b) With oversampling

Figute5.45.lllustrating
theeffectofoversampling
oftherareclass.

5.7.4 Sampling-Based Approaches

Sampling is another widely used approach for handling the class imbalance
problem. The idea of sampling is to modify the distribution of instances so
that the rare classis well representedin the training set. Some of the available
techniques for sampling include undersampling, oversampling, and a hybrid
of both approaches. To illustrate these techniques, consider a data set that
contains 100 positive examples and 1000 negative examples.
In the case of undersampling, a random sample of 100 negative examples
is chosen to form the training set along with all the positive examples. One
potential problem with this approach is that some of the useful negative exam-
ples may not be chosenfor training, therefore, resulting in a less than optimal
model. A potential method to overcomethis problem is to perform undersam-
pling multiple times and to induce multiple classifierssimilar to the ensemble
Iearning approach. Focused undersampling methods may also be used, where
the sampling procedure makes an informed choice with regard to the nega-
tive examples that should be eliminated, e.g., those located far away from the
decision boundary.
Oversampling replicates the positive examplesuntil the training set has an
equal number of positive and negative examples. Figure 5.45 illustrates the
effect of oversamplingon the construction of a decisionboundary using a classi-
fier such as a decision tree. Without oversampling, only the positive examples
at the bottom right-hand side of Figure 5.45(a) are classified correctly. The
positive example in the middle of the diagram is misclassifiedbecause there
306 Chapter 5 Classification: Alternative Techniques

are not enough examples to justify the creation of a new decision boundary
to separate the positive and negative instances. Oversampling provides the
additional examples neededto ensure that the decision boundary surrounding
the positive example is not pruned, as illustrated in Figure 5.45(b).
However, for noisy data, oversampling may causemodel overfitting because
some of the noise examples may be replicated many times. In principle, over*
sampling does not add any new information into the training set. Replication
of positive examplesonly prevents the learning algorithm from pruning certain
parts of the model that describe regions that contain very few training exam-
ples (i.e., the small disjuncts). The additional positive examples also tend to
increasethe computation time for model building.
The hybrid approach uses a combination of undersampling the majority
class and oversampling the rare class to achieve uniform class distribution.
Undersampling can be performed using random or focusedsubsampling. Over-
sampling, on the other hand, can be done by replicating the existing positive
examples or generating new positive examples in the neighborhood of the ex-
isting positive examples. In the latter approach, we must first determine the
k-nearest neighbors for each existing positive example. A new positive ex-
ample is then generated at some random point along the line segment that
joins the positive example to one of its k-nearest neighbors. This process is
repeated until the desired number of positive examples is reached. Unlike the
data replication approach, the new examples allow us to extend the decision
boundary for the positive classoutward, similar to the approach shown in Fig'
ure 5.44. Nevertheless,this approach may still be quite susceptible to model
overfitting.

5.8 Multiclass Problem

Some of the classificationtechniquesdescribed in this chapter, such as support
vector machines and AdaBoost, are originally designed for binary classification
problems. Yet there are many real-world problems, such as character recogni-
tion, face identification, and text classification,where the input data is divided
into more than two categories. This section presents several approaches for
extending the binary classifiers to handle multiclass problems. To illustrate
these approaches,let Y : {Ar,yz,...,AK} be the set of classesof the input
data.
The first approach decomposesthe multiclass problem into K binary prob-
lems. For each class At e Y , a binary problem is created where aII instances
that belong to 96 are considered positive examples, while the remaining in-
 5.8 MulticlassProblem 307

stances are considered negative examples. A binary classifier is then con-

structed to separate instances of class y6 from the rest of the classes.This is
known as the one-against-rest(1-r) approach.
The second approach, which is known as the one-against-one (1-1) ap-
proach, constructs K(K - L)/2binary classifiers,where each classifieris used
to distinguish between a pair of classes,(Ao,Ai). Instances that do not belong
to either Ai or yj are ignored when constructing the binary classifierfor (gi,yi).
In both 1-r and 1-1 approaches,a test instance is classified by combining the
predictions made by the binary classifiers. A voting scheme is typically em-
ployed to combine the predictions, where the class that receives the highest
number of votes is assigned to the test instance. In the 1-r approach, if an
instance is classified as negative, then all classesexcept for the positive class
receive a vote. This approach, however, may lead to ties among the different
classes.Another possibility is to transform the outputs of the binary classifiers
into probability estimates and then assign the test instance to the class that
has the highest probability.
Example 5.10. Consider a multiclass problem where Y : {Ar,,A2,Az,A4}.
Suppose a test instance is classified as (+, -, -, -) according to the 1-r ap-
proach. In other words, it is classified as positive when 91 is used as the
positive class and negative when 92, 93, and !4 are used as the positive class.
Using a simple majority vote, notice that gfl receivesthe highest number of
votes, which is four, while the remaining classesreceive only three votes. The
test instance is therefore classifiedas gr.
Suppose the test instance is classified as follows using the 1-1 approach:

Binary pair *i Ut l, at l: at +i A2 +" a2 *: at

of classes -i
Az
-i
Az
-i
U+
-i us - i 'y4
U+
Classification -T-
+ + +

The first two rows in this table correspond to the pair of classes(AtAi) chosen
to build the classifierand the last row representsthe predicted classfor the test
instance. After combining the predictions, 91 and 94 each receive two votes,
while 92 and gr3each receives only one vote. The test instance is therefore
classifiedas either At or A+, depending on the tie-breaking procedure. I

Error-Correcting Output Coding

A potential problem with the previous two approaches is that they are sensitive
to the binary classificationerrors. For the 1-r approach given in Example 5.10,
308 Chapter 5 Classification: Alternative Techniques

if at least of one of the binary classifiersmakes a mistake in its prediction, then

the ensemblemay end up declaring a tie between classesor making a wrong
prediction. For example, supposethe test instance is classifiedas (+, -, *, -)
due to misclassificationby the third classifier. In this case,it will be difficult to
tell whether the instance should be classifiedas 91 or 93, unlessthe probability
associatedwith each class prediction is taken into account.
The error-correcting output coding (ECOC) method provides a more ro-
bust way for handling multiclass problems. The method is inspired by an
information-theoretic approach for sending messagesacross noisy channels.
The idea behind this approach is to add redundancy into the transmitted
messageby means of a codeword, so that the receiver may detect errors in the
received message and perhaps recover the original message if the number of
erro s is small.
For multiclass learning, each class gais represented by a unique bit string of
length n known as its codeword. We then train n binary classifiersto predict
each bit of the codeword string. The predicted class of a test instance is given
by the codeword whose Hamming distance is closestto the codeword produced
by the binary classifiers. Recall that the Hamming distance between a pair of
bit strings is given by the number of bits that differ.
Example 5.11. Consider a multiclass problem where Y : {At,Az,ys,Aa}.
Supposewe encode the classesusing the following 7-bit codewords:

Class Codeword
At 1 1 1 1 1 1 1
Ut 0 0 0 0 1 1 1
UI 0 0 1 1 0 0 I
U+ 0 1 0 1 0 I 0

Each bit of the codeword is used to train a binary classifier. If a test instance
is classifiedas (0,1,1,1,1,1,1)by the binary classifiers,then the Hamming dis-
tance between the codeword and gr1is 1, while the Hamming distance to the
remaining classesis 3. The test instance is therefore classifiedas 91. I

An interesting property of an error-correcting code is that if the minimum

Hamming distance between any pair of codewords is d, then any t@ - L)12))
errors in the output code can be corrected using its nearest codeword. In
Example 5.11, becausethe minimum Hamming distance between any pair of
codewords is 4, the ensemble may tolerate errors made by one of the seven
 5.9 Bibliographic Notes 309

binary classifiers. If there is more than one classifier that makes a mistake,
then the ensemblemay not be able to compensatefor the error.
An important issue is how to design the appropriate set of codewords for
different classes. FYom coding theory, a vast number of algorithms have been
developed for generating n-bit codewords with bounded Hamming distance.
However, the discussionof these algorithms is beyond the scope of this book.
It is worthwhile mentioning that there is a significant difference between the
design of error-correcting codes for communication tasks compared to those
used for multiclass learning. For communication, the codewords should max-
imize the Hamming distance between the rows so that error correction can
be performed. Multiclass learning, however, requires that the row-wise and
column-wise distances of the codewords must be well separated. A larger
column-wise distance ensuresthat the binary classifiersare mutually indepen-
dent, which is an important requirement for ensemblelearning methods.

5.9 Bibliographic Notes

Mitchell [208j provides an excellent coverageon many classificationtechniques
from a machine learning perspective. Extensive coverage on classification can
also be found in Duda et al. [180],Webb [219],Fukunaga [187],Bishop [159],
Hastie et al. [192],Cherkasskyand Mulier [167],Witten and F]ank [221],Hand
et al. [190],Han and Kamber [189],and Dunham [181].
Direct methods for rule-based classifiers typically employ the sequential
covering schemefor inducing classification rules. Holte's 1R [195] is the sim-
plest form of a rule-based classifier because its rule set contains only a single
rule. Despite its simplicity, Holte found that for some data sets that exhibit
a strong one-to-one relationship between the attributes and the class label,
lR performs just as well as other classifiers. Other examples of rule-based
classifiersinclude IREP [184],RIPPER [170],CN2 1168,1691,AQ [207],RISE
[176], and ITRULE [214]. Table 5.9 shows a comparison of the characteristics
of four of these classifiers.
For rule-basedclassifiers,the rule antecedentcan be generalizedto include
any propositional or first-order logical expression (e.g., Horn clauses). Read-
ers who are interested in first-order logic rule-based classifiers may refer to
references such as [208] or the vast literature on inductive logic programming
1209]. Quinlan [211] proposed the C4.5rules algorithm for extracting classifi-
cation rules from decision trees. An indirect method for extracting rules from
artificial neural networks was given by Andrews et al. in [157].
310 Chapter 5 Classification: Alternative Techniques

Table ofvarious
5,9.Comparison rule-based
classifiers,
RIPPER u1\ z L]N2 AQR
(unordered) (ordered)
ttule-growlng Generai-to- L;eneral-to- General-tG. General-to-specific
strategy specific specific specific (seeded by a
oositive examole)
Evaluation FOIL's Infb gain Laplace Itntropy ancl Number of
Metric likelihood ratio true positives
Stopping All examples No performance No perfbrmance Kules cover only
condition for belong to the galn galn positive class
rule-erowinq same class
Rule Pruning Keducect None None None
error Drunlnq
lnstance Positive and Positive only Positive only Positive and
Elimination negative negatlve
btopprng tlrlor > bUTo oI I\o perlormance No perlormance All positive
condition for based on MDL garn garn examples are
adding rules covered
Kule uet tfeplace or :itatrstrcal None None
Pruning modifv rules tests
Search stratesv Greedv Beam search Beam search Beam sea,rch

Cover and Hart [t72] presented an overview of the nearest-neighborclassi-

fication method from a Bayesian perspective. Aha provided both theoretical
and empirical evaluations for instance-basedmethods in [155]. PEBLS, which
was developed by Cost and Salzberg [171], is a nearest-neighborclassification
algorithm that can handle data sets containing nominal attributes. Each train-
ing example in PEBLS is also assigned a weight factor that depends on the
number of times the example helps make a correct prediction. Han et al. [188]
developed a weight-adjusted nearest-neighboralgorithm, in which the feature
weights are learned using a greedy, hill-climbing optimization algorithm.
Naive Bayes classifiers have been investigated by many authors, including
Langley et al. [203], Ramoni and Sebastiani l2I2), Lewis [204], and Domingos
andPazzani [178]. Although the independenceassumption used in naive Bayes
classifiers may seemrather unrealistic, the method has worked surpdsingly well
for applications such as text classification. Bayesian belief networks provide a
more flexible approach by allowing someof the attributes to be interdependent.
An excellent tutorial on Bayesian belief networks is given by Heckerman in
[1e4].
Vapnik [2I7, 218] had written two authoritative books on Support Vector
Machines (SVM). Other useful resourceson SVM and kernel methods include
the books by Cristianini and Shawe-Taylor [173] and Scholkopf and Smola
 5.9 Bibliographic Notes 311

[213]. There are several survey articles on SVM, including those written by
Burges [164],Bennet et al. [158],Hearst 1193],and Mangasarian[205].
A survey of ensemble methods in machine learning was given by Diet-
terich [174]. The bagging method was proposed by Breiman [161]. Reund
and Schapire [186] developed the AdaBoost algorithm. Arcing, which stands
for adaptive resampling and combining, is a variant of the boosting algorithm
proposed by Breiman [162]. It usesthe non-uniform weights assignedto train-
ing examples to resample the data for building an ensembleof training sets.
Unlike AdaBoost, the votes of the base classifiersare not weighted when de-
termining the class label of test examples. The random forest method was
introduced by Breiman in [163].
Related work on mining rare and imbalanced data sets can be found in the
survey papers written by Chawla et al. [166] and Weiss 1220].Sampling-based
methods for mining imbalanced data sets have been investigated by many au-
thors, such as Kubat and Matwin 1202),Japkowitz [196], and Drummond and
Holte [179]. Joshi et al. [199] discussedthe limitations of boosting algorithms
for rare class modeling. Other algorithms developed for mining rare classes
include SMOTE [165],PNrule [198],and CREDOS [200].
Various alternative metrics that are well-suited for class imbalanced prob-
lems are available. The precision, recall, and F1-measureare widely used met-
rics in information retrieval 1216].ROC analysis was originally used in signal
detection theory. Bradley [160] investigated the use of area under the ROC
curve as a performance metric for machine learning algorithms. A method
for comparing classifier performance using the convex hull of ROC curves was
suggested by Provost and Fawcett in [210]. Ferri et al. lf85] developed a
methodology for performing ROC analysis on decision tree classifiers. They
had also proposed a methodology for incorporating area under the ROC curve
(AUC) as the splitting criterion during the tree-growing process. Joshi [197]
examined the performance of these measuresfrom the perspectiveof analyzing
rare classes.
A vast amount of literature on cost-sensitive learning can be found in
the online proceedings of the ICML'2000 Workshop on cost-sensitive learn-
ittg. The properties of a cost matrix had been studied by Elkan in [182].
Margineantu and Dietterich [206] examined various methods for incorporating
cost information into the C4.5 learning algorithm, including wrapper meth-
ods, class distribution-based methods, and loss-basedmethods. Other cost-
sensitive learning methods that are algorithm-independent include AdaCost
[t83], Metacost [177],and costing [222].
3L2 Chapter 5 Classification: Alternative Techniques

Extensive literature is also available on the subject of multiclass learning.

This includes the works of Hastie and Tibshirani [191], Allwein et al. 1156],
Kong and Dietterich [201], and Tax and Duin [215]. The error-correcting
output coding (ECOC) method was proposed by Dietterich and Bakiri [175].
They had also investigated techniques for designing codesthat are suitable for
solving multiclass problems.

Bibliography
[155] D. W. Aha. A studg of instance-based algorithms for superuised learning tasks: mathe-
rnatical, empirical, and, psgchological eualuat'ions. PhD thesis, University of California,
Irvine, 1990.
[156] E. L. Allwein, R. E. Schapire, and Y. Singer. Reducing Multiclass to Binary: A
Unifying Approach to Margin Classifiers. Journal of Machine Learn'ing Research, I:
113-141, 2000.
[157] R, Andrews, J. Diederich, and A. Tickle. A Survey and Critique of Techniques For
Extracting Rules Flom Tbained Artificial Neural Networks. Knowledge Based, Sgstems,
8(6):373-389, 1995.
[158] K. Bennett and C. Campbell. Support Vector Machines: Hype or Hallelujah. SIGKDD
Erp lorat'i,on s, 2 (2) : L-13, 2000.
[159] C. M. Bishop. Neural Networks for Pattern Recognit'ion. Oxford University Press,
Oxford. U.K.. 1995.
[160] A. P. Bradley. The use of the area under the ROC curve in the Evaluation of Machine
Learning Algorithms. P attern Recogni,tion,30(7) : 1145-1 1 49, 1997.
[161] L. Breiman. Bagging Predictors. Mach,ine Lear"ning,24(.2):123 140, 1996.
[162] L. Breiman. Bias, Variance, and Arcing Classifiers. Technical Report 486, University
of California, Berkeley, CA, 1996.
f163] L. Breiman. Random Forests. Mach'ine Learn'ing, 45(I):5-32,2001.
[164] C. J. C. Burges. A Ttrtorial on Support Vector Machines for Pattern Recognition.
D ata Mining and, Knowled,ge Discox erg, 2(2) :t21-167, 1998.
[165] N. V. Chawla, K. W. Bowyer, L. O. Hall, and W. P. Kegelmeyer. SMOTE: Synthetic
Minority Over-sampling Technique. Journal of Artifi,cial Intelli,gence Research, 16:321-
357, 2002.
[166] N. V. Chawla, N. Japkowicz, and A. Kolcz. Editorial: Special Issue on Learning from
Imbalanced Data Sets. SIGKDD Erplorat'ions, 6(1):1-6, 2004.
[167] V. Cherkassky and F. Mulier. Learn'ing frorn Data: Concepts, Theory, and Method,s.
Wiley Interscience, 1998.
[168] P. Clark and R. Boswell. Rule Induction with CN2: Some Recent Improvements. In
Machine Lear"ning: Proc. of the Sth European Conf. (EWSL-97/, pages 151-163, 1991.
[169] P. Clark and T. Niblett. The CN2 Induction Algorithm. Machine Learning, 3(4):
26L-283, 1989.
[170] W. W. Cohen. Fast Effective Rule Induction. In Proc. of the 12th Intl. Conf. on
Mach'ine Leamting, pages 115-123, Tahoe City, CA, July 1995.
[171] S. Cost and S. Salzberg. A Weighted Nearest Neighbor Algorithm for Learning with
Symbolic Features. Mach'ine Learn'ing, 10:57-78, 1993.
ll72l T. M. Cover and P. E. Hart. Nearest Neighbor Pattern Classification. Knowleilge
Baseil Sgstems, 8(6):373-389, 1995.
 Bibliography 313

[173] N. Cristianini and J. Shawe-Taylor. An Introduction to Support Vector Machines and

Other Kernel-baseil Learning Method,s. Cambridge University Press, 2000.
f174] T. G. Dietterich. Ensemble Methods in Machine Learning. In Fi,rst IntI. Workshop on
Multiple Classifi,er Sgsterns, Cagliari, Italy, 2000.
[175] T. G. Dietterich and G. Bakiri. Solving Multiclass Learning Problems via Error-
Correcting Output Codes. Joum,al of Arti,ficial Intelligence Research,2:263-286, Lggl.
[176] P. Domingos. The RISE system: Conquering without separating. In Proc. of the 6th
IEEE Intl. Conf. on Tools wi,th Artificial Intelligence, pages 704-707, New Orleans, LA,
1994.
lL77l P. Domingos. MetaCost: A General Method for Making Classifiers Cost-Sensitive. In
Proc. of the Sth Intl. Conf. on Knowledge Discouery and. Data Mi,ning, pages 155-164,
San Diego, CA, August 1999.
f1781 P. Domingos and M. Pazzani. On the Optimality of the Simple Bayesian Classifier
under Zero-One Loss. Machine Leanri,ng,29(2-3):103 130, 1997.
[179] C. Drummond and R. C. Holte. C4.5, Class imbalance, and Cost sensitivity: Why
under-sampling beats over-sampling. In ICML'2001 Workshop on Learning from Im-
balancedData Sets 1d Washington, DC, August 2003.
[180] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classi,fication John Wiley & Sons,
Inc., New York, 2nd edition, 2001.
f1811 M. H. Dunham. Data Mi.ni,ng: Introductory and Aduanced,Topics. Prentice Hall, 2002.
[182] C. Elkan. The Foundations of Cost-Sensitive Learning. ln Proc. of the 17th IntL Joint
Conf. on Arti,ficial Intelligence, pages 973-978, Seattle, WA, August 2001.
[183] W. Fan, S. J. Stolfo, J. Zhang, and P. K. Chan. AdaCost: misclassification cost-
sensitive boosting. In Proc. of the 16th Intl. Conf. on Mach,ine Learning, pages 97-105,
Bled, Slovenia, June 1999.
[184] J. Fiirnkranz and G. Widmer. Incremental reduced error pruning. In Proc. of the 11th
IntI. Conf . on Mach'ine Learning, pages 70-77, New Brunswick, NJ, July 1994.
[185] C. Ferri, P. Flach, and J. Hernandez-Orallo. Learning Decision TYeesUsing the Area
Under the ROC Curve. In Proc. of the 19th IntL Conf. on Machine Leanting, pages
739-L46, Sydney, Australia, July 2002.
1186] Y. Freund and R. E. Schapire. A decision-theoretic generalization of on-line learning
and an application to boosting. Journal of Computer and Sgstern Sciences,55(1):119-
139, 1997.
[187] K. Fukunaga. Introd"uction to Stati,sti.cal Pattern Recognition. Academic Press, New
York, 1990.
U88] E.-H. Han, G. Karypis, and V. Kumar. Text Categorization Using Weight Adjusted
k-Nearest Neighbor Classification. In Proc. of the Sth Pacifi,c-Asia Conf. on Knowled,ge
Discouerg and, Data Mining, Lyon, Fbance,2001.
f1891 J. Han and M. Kamber. Data Mining: Concepts and Techniques. Morgan Kaufmann
Publishers, San FYancisco,2001.
[190] D. J. Hand, H. Mannila, and P. Smyth. Pri,nci,plesof Data Mi,ning. MIT Press,2001.
[191] T. Hastie and R. Tibshirani. Classification by pairwise coupling. Annals of Statisti,cs,
26(2):451-a71, 1998.
f192] T. Hastie, R. Tibshirani, and J. H. Fbiedman. The Elements of Statistical Learning:
Data Mining, Inference, PreiLiction. Springer, New York, 2001.
[193] M. Hearst. Ttends & Controversies: Support Vector Machines. IEEE Intetti,gent
Systems, 13(4) :18-28, 1998.
3L4 Chapter 5 Classification: Alternative Techniques

[194] D. Heckerman. Bayesian Networks for Data Mining. Data Mi,ni,ng and Knouledge
Discouerg,1(1):79 LI9, L997.
1195] R. C. Holte. Very Simple Classification Rules Perform Well on Most Commonly Used
Data sets. Machine Learn'ing, 11:63-91, 1993.
[196] N. Japkowicz. The Class Imbalance Problem: Significance and Strategies. In Proc.
of the 2000 Intl. Conf. on Arti,ficial Intelli,gence: Special Tracle on Ind,uct'iue Learning,
volume 1, pages lll-117, Las Vegas, NV, June 2000.
[197] M. V. Joshi. On Evaluating Performance of Classifiers for Rare Classes. In Proc. of
the 2002 IEEE IntI. Conf. on Data Mining, Maebashi City, Japan, December 2002.
1198] M. V. Joshi, R. C. Agarwal, and V. Kumar. Mining Needles in a Haystack: Classifying
Rare Classes via Two-Phase Rule Induction. In Proc. of 2001 ACM-SIGMOD IntI. Conf.
on Managernent of Data, pages 91-102, Santa Barbara, CA, June 2001.
[199] M. V. Joshi, R. C. Agarwal, and V. Kumar. Predicting rare classes: can boosting
make any weak learner strcing? In Proc. of the 8th IntI. Conf. on Knowledge Discouery
and Data Mining, pages 297 306, Edmonton, Canada, JuIy 2002.
1200] M. V. Joshi and V. Kumar. CREDOS: Classification Using Ripple Down Structure
(A Case for Rare Classes). ln Proc. of the SIAM IntI. Conf. on Data Min'ing, pages
32L-332, Orlando, FL, April 2004.
[201] E. B. Kong and T. G. Dietterich. Error-Correcting Output Coding Corrects Bias and
Variance. In Proc. of the 12th IntI. Conf. on Machine Lear"ning, pages 313 321, Tahoe
City, CA, July 1995.
[202] M. Kubat and S. Matwin. Addressing the Curse of Imbalanced Ttaining Sets: One
Sided Selection. In Proc. of the l/+th IntI. Conf. on Machine Lear"n'ing,pages 179-186,
Nashville, TN, July 1997.
[203] P. Langley, W. Iba, and K. Thompson. An analysis of Bayesian classifiers. In Proc. oJ
the 10th National Conf. on Artif,cial Intelligence, pages 223-228, 1992.
[204] D. D. Lewis. Naive Bayes at Forty: The Independence Assumption in Information
Retrieval. In Proc. of the 10th European Conf. on Mach'i,ne Learning (ECML 1998),
pages 4-15, 1998.
[205] O. Mangasarian. Data Mining via Support Vector Machines. Technical Report Tech-
nical Report 01-05, Data Mining Institute, May 2001.
[206] D. D. Margineantu and T. G. Dietterich. Learning Decision Tlees for Loss Minimization
in Multi-Class Problems. Technical Report 99-30-03, Oregon State University, 1999.
1207] R. S. Michalski, I. Mozetic, J. Hong, and N. Lavrac. The Multi-Purpose Incremental
Learning System AQ15 and Its Testing Application to Three Medical Domains. In Proc.
of sth National Conf. on Arti,fici,al Intell'igence, Orlando, August 1986.
[208] T. Mitchell. Machine Learning. McGraw-Hill, Boston, MA, 1997.
f209] S. Muggleton. Found,ati,ons of Inducti,ue Log,ic Programm'ing. Prentice Hall, Englewood
Cliffs. NJ, 1995.
[210] F. J. Provost and T. Fawcett. Analysis and Visualization of Classifier Performance:
Comparison under Imprecise Class and Cost Distributions. In Proc. of the Srd, Intl.
Conf. on Knowled,ge D'iscouery and Data M'in'ing, pages 43-48, Newport Beach, CA,
August 1997.
[211] J. R. Quinlan. CI.S: Programs for Machi,ne Learn'ing. Morgan-Kaufmann Publishers,
San Mateo. CA. 1993.
l2L2l M. Ramoni and P. Sebastiani. Robust Bayes classifierc. Arti.fi,cial Intelligence, I25:
209-226,200r.
 5.10 Exercises 31-5

f213] B. Scholkopf and A. J. Smola. Learning with Kernels: Support Vector Machines,
Regularizati,on, Opt'i,mizat'ion, and, Begond. MIT Press, 2001.
[2I4) P. Smyth and R. M. Goodman. An Information Theoretic Approach to Rule Induction
from Databases. IEEE Trans. on Knowled,geand"Data Engi'neering,4(4):301-316, 1992.
[215] D. M. J. Tax and R. P. W. Duin. Using Two-Class Classifiers for Multiclass Classi-
fication. In Proc. of the 16th IntI. Conf. on Pattern Recogn'ition (ICPR 2002), pages
124-127, Quebec, Canada, August 2002.
f216] C. J. van Rijsbergen. Inforrnat'ion Retrieual. Butterworth-Heinemann, Newton, MA,
1978.
[217] V. Vapnik. The Nature of Statistical Learn'ing Theorg. Springer Verlag, New York,
1995.
[218] V. Vapnik. Statistical Learn'ing Theorg. John Wiley & Sons, New York, 1998.
[219] A. R. Webb. Statistical Pattern Recogni.tion.John Wiley & Sons, 2nd edition, 2002.
1220] G. M. Weiss. Mining with Rarity: A Unifying Flamework. SIGKDD Erplorations,6
(l):7-L9,2004.
l22l] I. H. Witten and E. FYank. Data Mining: Pract'ical Machine Learning Tools and
Techn'iqueswith Jaua Implementations. Morgan Kaufmann, 1999.
[222) B. Zadrozny, J. C. Langford, and N. Abe. Cost-Sensitive Learning by Cost-
Proportionate Example Weighting. In Proc. of the 2003 IEEE IntI. Conf. on Data
M'i,n'ing,pages 435-442, Melbourne, FL, August 2003.

5.10 Exercises
1. Consider a binary classification problem with the following set of attributes and
attribute values:

o Air Conditioner : {Working, Broken}

o Engine : {Good, Bad}
o Mileage : {High, Medium, Low}
o Rust : {yes, No}

Suppose a rule-based classifier produces the following rule set:

Mileage : HiSh _- Value : Low

Mileage : Low ------+
Value : High
Air Conditioner : Working, Engine : Good ----- Value : High
Air Conditioner : Working, Engine : Bad ----+ Value : Low
Air Conditioner : Brokel ---+ Value : Low

(a) Are the rules mutually exclustive?

316 Chapter 5 Classification: Alternative Techniques

(b) Is the rule set exhaustive?

(c) Is ordering needed for this set of rules?
(d) Do you need a default class for the rule set?

2. The RIPPER algorithm (by Cohen [170]) is an extension of an earlier algorithm

called IREP (by Fiirnkranz and Widmer 1184]). Both algorithms apply the
reduced-error pruning method to determine whether a rule needs to be
pruned. The reduced error pruning method uses a validation set to estimate
the generalization error of a classifier. Consider the following pair of rules:

Rt: A ----- C
Rz: AnB------+C

-R2is obtained by adding a new conjunct, B, to the left-hand side of R1. For
this question, you will be asked to determine whether -R2 is preferred over fi1
from the perspectives of rule-growing and rule-pruning. To determine whether
a rule should be pruned, IREP computes the following measure:

urREp:-_fr1,, ,
n-L(N -n)

where P is the total number of positive examples in the validation set, Iy' is
the total number of negative examples in the validation set, p is the number of
positive examples in the validation set covered by the rule, and n is the number
of negative examples in the validation set covered by the rtIe. ulppp is actually
similar to classification accuracy for the validation set. IREP favors rules that
have higher values of u1pBp. On the other hand, RIPPER applies the following
measure to determine whether a rule should be pruned:

u n t p p n n- : ' - =
p+n

(a) Suppose -R1 is covered by 350 positive examples and 150 negative ex-
amples, while R2 is covered by 300 positive examples and 50 negative
examples. Compute the FOIL's information gain for the rule Rz with
respectto R1.
(b) Consider a validation set that contains 500 positive examples and 500
negative examples. For R1, suppose the number of positive examples
covered by the rule is 200, and the number of negative examples covered
by the rule is 50. For rR2,suppose the number of positive examples covered
by the rule is 100 and the number of negative examples is 5. Compute
urREp for both rules. Which rule does IREP prefer?
(c) Compute uRrppEn for the previous problem. Which rule does RIPPER
prefer?
 5.10 Exercises 3L7

C4.5rules is an implementation of an indirect method for generating rules from

a decision tree. RIPPER is an implementation of a direct method for generating
rules directly from data.

(a) Discuss the strengths and weaknessesof both methods.

(b) Consider a data set that has a large difference in the class size (i.e., some
classes are much bigger than others). Which method (between C4.5rules
and RIPPER) is better in terms of finding high accuracy rules for the
small classes?

4. Consider a training set that contains 100 positive examples and 400 negative
examples. For each of the following candidate rules,

R1 A .-----+* (covers 4 positive and I negative examples),

Rz: B .-----+
* (covers 30 positive and 10 negative examples),
R3: C ------+
* (covers 100 positive and 90 negative examples),

determine which is the best and worst candidate rule according to:

(a) Rule accuracy.

(b) FOIL's information gain.
(c) The likelihood ratio statistic.
(d) The Laplace measure.
(e) The m-estimate measure (with k :2 and p+ :0.2).

5 . Figure 5.4 illustrates the coverage of the classification rules R1, ,R2, and R3.
Determine which is the best and worst rule according to:

(a) The likelihood ratio statistic.

(b) The Laplace measure.
(c) The m-estimate measure (with k :2 and P+ :0.58).
(d) The rule accuracy after R1 has been discovered, where none of the exam-
ples covered by R1 are discarded).
(e) The rule accuracy after rRl has been discovered, where only the positive
examples covered by iBl are discarded).
(f) The rule accuracy after Rl has been discovered, where both positive and
negative examples covered by -Rl are discarded.

6. (a) Suppose the fraction of undergraduate students who smoke is 15% and
the fraction of graduate students who smoke is 23%. If one-fifth of the
college students are graduate students and the rest are undergraduates,
what is the probability that a student who smokes is a graduate student?
318 Chapter 5 Classification: Alternative Techniques

(b) Given the information in part (a), is a randomly chosen college student
more likely to be a graduate or undergraduate student?
(c) Repeat part (b) assuming that the student is a smoker.
(d) Suppose 30% of the graduate students live in a dorm but only l0% of
the undergraduate students live in a dorm. If a student smokes and lives
in the dorm, is he or she more likely to be a graduate or undergraduate
student? You can assume independence between students who live in a
dorm and those who smoke.

7. Consider the data set shown in Table 5.10

Table
5.10.DatasetforExercise
7.
Record A B C Class
1 0 0 0 +
2 0 0 1
e 0 I 1
t
0 1 I
0 0 1 +
o 1 0 1 +
7 1 0 1
8 1 0 I

9 1 1 1 +
10 I 0 1 +

(a) Estimatethe conditionalprobabilitiesfor P(Al-l_),P(Bi+), P(Cl+), P(Al-),

P(Bl-), and P(Cl-).
(b) Usethe estimateof conditional probabilitiesgiven in the previousquestion
to predictthe classlabel for a test sample(A:0,8 - I,C :0) using
the naive Bayes approach.
(c) Estimate the conditional probabilities using the m-estimate approach,
with p : I/2 and m:4.
(d) Repeat part (b) using the conditional probabilities given in part (c).
(e) Compare the two methods for estimating probabilities. Which method is
better and why?

8. Consider the data set shown in Table 5.11.

(a) Estimate the conditional probabilities for P(A : 1l+), P(B : 11a),
P(C : 1l+), P(.4 : 1l-), P(B : 1l-), and P(C : 1l-) using the
same approach as in the previous problem.
 5.10 Exercises 319

5.11.DatasetforExercise
Table 8.
lnstance A B C Olass
I 0 0 1
2 1 0 1 +
J 0 1 0
4 1 0 0
5 1 0 I +
t 0 0 1 +
7 1 I 0
8 0 0 0
q
0 1 0 +
10 1 1 I +

(b) Use the conditional probabilities in part (a) to predict the class label for
a test sample (A : l, B : l,C :1) using the naive Bayes approach.
( c ) C o m p a r eP ( A : 1 ) , P(B: 1), and P(A: !,8 :1). Statethe relation-
ships between A and B.
(d) Repeat the analysis in part (c) using P(A : l), P(B : 0), and P(A:
1 , , B: 0 ) .
( e ) C o m p a r eP ( A : I , B : I l C l a s s : * ) a g a i n s tP ( A : l l C l a s s : * ) a n d
P(B : llClass: 1). Are the variables conditionally independent given
the class?

9. (a) Explain how naive Bayes performs on the data set shown in Figure 5.46.
(b) If each class is further divided such that there are four classes (AI, 42,
,B1, and B2), will naive Bayes perform better?
(c) How will a decision tree perform on this data set (for the two-class prob-
lem)? What if there are four classes?

10. Repeat the analysis shown in Example 5.3 for finding the location of a decision
boundary using the following information:

(a) The prior probabilities are P(Crocodile):2 x P(lrrigator).

(b) The prior probabilities are P(Alligator):2 x P(Crocodile).
(c) The prior probabilities are the same, but their standard deviations are
different; i.e., o(Crocodile) :4 and o(Alligator) : 2.

11. Figure 5.47 illustrates the Bayesian belief network for the data set shown in
Table 5.12. (Assume that all the attributes are binary).

(a) Draw the probability table for each node in the network.
32O Chapter 5 Classification: Alternative Techniques

Attributes

Figure
5.46,DatasetforExercise
9.

Figure
5.47,Bayesian
belief
network.

(b) Use the Bayesian network to compute P(Engine : Bad, Air Conditioner
: Broken).

12. Given the Bayesian network shown in Figure 5.48, compute the following prob-
abilities:

(a) P(B : good, F : empty, G : empty, 5 : yes).

(b) P(B - bad, F : empty? G : not empty, S : no).
(c) Given that the battery is bad, compute the probability that the car will
start.

13. Consider the one-dimensional data set shown in Table 5.13.

 5.1-0 Exercises 32L

5,12.DatasetforExercise
Table 11.
Mileage Engine Air Conditioncr Number of Records Number of Records
with Car Value:Hi with Car Value:Lo
H Good Working J 4
Hi Good Broken 1 z
Hi Bad Working 1 K

Hi Bad Broken 0 A

Lo Good Working I 0
Lo Good Broken 1
Lo Bad Working I 2
Lo Bad Broken 0 z

P(B=bad)=0.1 P ( F = e m p t y=)0 . 2

P(G= emptyI B = good,F = notempV)= 0.1

P(G= emptyI g = good,F = emPtV)= 0.8
P(G= emptyI B = bad,F = notempty)= 0.2
P(G= emptyI B = bad,F = empty)= 0.9

P(S= no I B - good,F = notempty)= 0.1

P(S= no I 3 = good,F = emPtY)
= 0.8
P(S= no I B = bad,F = notemPty)= 0.9
P(S= no I B = bad,F = emPty)
= 1.0

Figure
5.48,Bayesian network
belief 12.
forExercise

(u) Classify the data point r : 5.0 according to its 1-, 3-, 5-, and 9-nearest
neighbors (using majority vote).
(b) Repeat the previous analysis using the distance-weighted voting approach
describedin Section 5.2.1.

14. The nearest-neighbor algorithm described in Section 5.2 can be extended to

handle nominal attributes. A variant of the algorithm called PEBLS (Parallel
Examplar-Based Learning System) by Cost and Salzberg ll7ll measures the
distance between two values of a nominal attribute using the modified value
difference metric (MVDM). Given a pair of nominal attribute values' V1 and
322 Chapter 5 Classification: Alternative Techniques

Table
5.13.DatasetforExercise
13.
(t
x U.D 3.0 4.O 4.6 4.9 5 . 2 5 . 3 7.0 9.5
v + + -t- +

V2,lhe distance between them is defined as follows:

k1

d . ( V t , v )t: l L L - ' ' ' 1 , (5.84)

- | nr
t- I I
nz I

where nii is the number of examples from class i with attribute value Vi and
n, is the number of examples with attribute value [.
Consider the training set for the loan classification problem shown in Figure
5.9. Use the MVDM measure to compute the distance between every pair of
attribute values for the Home 0wner and Marital Status attributes.

1 5 . For each of the Boolean functions given below, state whether the problem is
linearly separable.

(a) A AND B AND C

(b) NOT A AND B
(") (,.+oR B) AND (,4 oR C)
(d) (,4 xoR B) AND (A OR B)

lo. (a) Demonstrate how the perceptron model can be used to representthe AND
and OR functions between a pair of Boolean variables.
(b) Comment on the disadvantage of using linear functions as activation func-
tions for multilayer neural networks.

1 7 . You are asked to evaluate the performance of two classification models, M1 and
M2. The test set you have chosen contains 26 binary attributes, labeled as ,4
throtgh Z.
Table 5.14 shows the posterior probabilities obtained by applying the models
to the test set. (Only the posterior probabilities for the positive class are
shown). As this is a two-classproblem, P(-) : 1- P(+) and P(-lA, . . . , Z) :
I - P(+lA, . . . , Z). Assume that we are mostly interested in detecting instances
from the positive class.

(a) PIot the ROC curve for both M1 and M2. (You should plot them on the
same graph.) Which model do you think is better? Explain your reasons.
(b) For model M1, suppose you choosethe cutoffthreshold to be f :0.5. In
other words, any test instances whose posterior probability is greater than
C will be classified as a positive example. Compute the precision, recall,
and F-measure for the model at this threshold value.
 5.1-0 Exercises 323

Table5.14.Posterior
orobabilities 17.
forExercise
Instance Tlue Class P ( + \ A , . . , Z ,M r ) P ( + 1 , 4 , . . . , 2 , M 2 )
1 + 0. 0.61
2 + 0.69 0.03
3 0.44 0.68
4 0.55 0.31
5 + 0.67 0.45
6 + 0.47 0.09
7 0.08 0.38
8 0.15 0.05
q
+ 0.45 0.01
10 0.35 0.04

(c) Repeat the analysis for part (c) using the same cutoff threshold on model
M2. Compare the F-measure results for both models. Which model is
better? Are the results consistent with what you expect from the ROC
curve?
(d) Repeat part (c) for model M1 using the threshold t : 0.1. Which thresh-
old do you prefer, t : 0.5 or f : 0.1? Are the results consistent with what
you expect from the ROC curve?

18. Following is a data set that contains two attributes, X and Y, and two class
labels. "+" and "-". Each attribute can take three different values: 0' 1, or 2.

Number of
X Y Instances
+
0 0 0 100
1 0 0 U
2 0 0 100
0 1 10 100
I 1 10 0
2 I 10 100
0 2 0 100
1 2 0 0
2 2 0 100

The concept for the "+" class is Y : 1 and the concept for the "-" class is
X:0YX:2.
((1"
(a) Build a decision tree on the data set. Does the tree capture ffts 3,nd
tt-tt concepts?
324 Chapter 5 Classification: Alternative Techniques

(b) What are the accuracy, precision, recall, and F1-measure of the decision
tree? (Note that precision, recall, and F1-measureare defined with respect
to the "+" class.)
(") Build a new decision tree with the following cost function:

f 0, if i:i;
c(i,j):11, ifi:*,j:-:
, ir'i: -, j : +.
l ,*N'#B:ifii+fi#*tr
(Hint: only the leaves of the old decision tree need to be changed.) Does
the decision tree capture the "+" concept?
(d) What are the accuracy) precision, recall, and f'1-measure of the new deci-
sion tree?

19. (a) Consider the cost matrix for a two-classproblem. Let C(*, *) : C(-, -) :
p, C(+,-): C(-,1): q, and q ) p. Show that minimizingthe cost
function is equivalent to maximizing the classifier's accuracy.
(b) Show that a cost matrix is scale-invariant. For example, if the cost matrix
is rescaled from C(i, j) ------PC(i,j), where B is the scaling factor, the
decision threshold (Equation 5.82) will remain unchanged.
(c) Show that a cost matrix is translation-invariant. In other words, adding a
constant factor to all entries in the cost matrix will not affect the decision
threshold (Equation 5.82).

20. Consider the task of building a classifier from random data, where the attribute
values are generated randomly irrespective of the class labels. Assume the data
set contains records from two classes,"+" and "-." Half of the data set is used
for training while the remaining half is used for testing.

(a) Suppose there are an equal number of positive and negative records in
the data and the decision tree classifier predicts every test record to be
positive. What is the expected error rate of the classifieron the test data?
(b) Repeat the previous analysis assuming that the classifier predicts each
test record to be positive class with probability 0.8 and negative class
with probability 0.2.
(c) Suppose two-thirds of the data belong to the positive class and the re-
maining one-third belong to the negative class. What is the expected
error of a classifier that predicts every test record to be positive?
(d) Repeat the previous analysis assuming that the classifier predicts each
test record to be positive class with probabllity 213 and negative class
with orobabilitv 1/3.
 5.10 Exercises 325

21. Derive the dual Lagrangian for the linear SVM with nonseparable data where
the objective function is

r,---\ l l*ll' , cr $ r .i2

"f(w): , +u\Let) .

22. Consider the XOR problem where there are four training points:

( 1 ,1 ,- ) , ( 1 , 0+, ) ,( 0 ,1 ,+ ) , ( 0 , 0-,) .

Tlansform the data into the following feature space:

iD: (1,Jirr, r/i*r, Jirrrr, *7).

"?,
Find the maximum margin linear decision boundary in the transformed space.

23. Given the data sets shown in Figures 5.49, explain how the decision tree, naive
Bayes, and k-nearest neighbor classifiers would perform on these data sets.
326 Chapter 5 Classification: Alternative Techniques

Attrlbub Afrribuie6
Distinguishing
Anribule6 Noise Aflributes Anributes
Dislinguishing NoiseAtribut€s

lrl

tl

1l
I ClassA i:iii;:i;iii ClassA

lrrl
"'i
tt
ii '
ll
'' tr
,,
I Clss B
rrii' ClassB

llll I
,,i,':'il
rl

(a) Synthetic data set 1. (b) Synthetic data set 2.

Attributes
Oistinguishing Oislinguishing
Attribute ret 1 Attributeset 2 Noise Attribules
ClassA ClassB ClassA ClassB ClassA

60% lilled 40% filled ClassB ClassA ClassB ClassA ClassB

with 1 with 1 Class

ClassA ClassB Cla$ A ClassB ClassA

40ol. filled 60% filled ClassB ClassA ClassB ClassA ClassB

wilh 1 with I
Clss

(c) Synthetic data set 3. (d) Synthetic data set 4

(e) Synthetic data set 5. (f) Synthetic data set 6.

Figure
5.49.DatasetforExercise
23.
AssociationAnalysis:
BasicConceptsand
Algorithms
Many businessenterprisesaccumulate large quantities of data from their day-
to-day operations. For example, huge amounts of customer purchase data are
collected daily at the checkout counters of grocery stores. Table 6.1 illustrates
an example of such data, commonly known as market basket transactions.
Each row in this table correspondsto a transaction, which contains a unique
identifier labeled TID and a set of items bought by a given customer. Retail-
ers are interested in analyzing the data to learn about the purchasing behavior
of their customers. Such valuable information can be used to support a vari-
ety of business-related applications such as marketing promotions, inventory
management, and customer relationship management.
This chapter presents a methodology known as association analysis,
which is useful for discovering interesting relationships hidden in large data
sets. The uncovered relationships can be representedin the form of associa-

Table6.1. Anexample
of market transactions.
basket
TID Items
1 {Bread, Milk}
2 {Bread, Diapers, Beer, Eggs}
3 {Milk, Diapers, Beer, Cola}
4 {Bread, Milk, Diapers, Beer}
tr
{Bread, Milk, Diapers, Cola}
328 Chapter 6 Association Analvsis

tion rules or sets of frequent items. For example, the following rule can be
extracted from the data set shown in Table 6.1:

--'
{liapers} {eeer}.

The rule suggeststhat a strong relationship exists between the sale of diapers
and beer becausemany customers who buy diapers also buy beer. Retailers
can use this type of rules to help them identify new opportunities for cross-
selling their products to the customers.
Besidesmarket basket data, association analysis is also applicable to other
application domains such as bioinformatics, medical diagnosis, Web mining,
and scientific data analysis. In the analysis of Earth sciencedata, for example,
the association patterns may reveal interesting connections among the ocean,
land, and atmospheric processes.Such information may help Earth scientists
develop a better understanding of how the different elements of the Earth
system interact with each other. Even though the techniques presented here
are generally applicable to a wider variety of data sets,for illustrative purposes)
our discussionwill focus mainly on market basket data.
There are two key issuesthat need to be addressedwhen applying associ-
ation analysis to market basket data. First, discovering patterns from a large
transaction data set can be computationally expensive. Second, some of the
discoveredpatterns are potentially spurious becausethey may happen simply
by chance. The remainder of this chapter is organized around these two is-
sues. The first part of the chapter is devoted to explaining the basic concepts
of association analysis and the algorithms used to efficiently mine such pat-
terns. The second part of the chapter deals with the issue of evaluating the
discoveredpatterns in order to prevent the generation of spurious results.

6.1 Problem Definition

This section reviews the basic terminology used in association analysis and
presents a formal description of the task.

Binary Representation Market basket data can be representedin a binary

format as shown in Table 6.2, where each row corresponds to a transaction
and each column corresponds to an item. An item can be treated as a binary
variable whose value is one if the item is present in a transaction and zero
otherwise. Becausethe presenceof an item in a transaction is often considered
more important than its absence,an item is an asymmetric binary variable.
 \
6.L Problem Definition 329

ofmarket
0/1 representation
Table6.2.A binary data,
basket
TID Bread Milk Diapers Beer Eggs Cola
-l
I 1 0 0 0 0
2 1 0 1 I 1 0
3 0 1 1 1 0 1
4 1 1 1 I 0 0
5 1 1 1 0 0 I

This representationis perhaps a very simplistic view of real market basket data
becauseit ignores certain important aspects of the data such as the quantity
of items sold or the price paid to purchase them. Methods for handling such
non-binary data will be explained in Chapter 7.

Itemset and Support Count Let I : {h,i2,...,i'a} be the set of all items
in a market basket data and T : {h,t2,..-,t1"} be the set of all transactions'
Each transaction ti contains a subset of items chosen from 1. In association
analysis, a collection of zero or more items is termed an itemset. If an itemset
contains /c items, it is called a k-itemset. For instance, {Beer, Diapers, Mj-Ik}
is an example of a 3-itemset. The null (or empty) set is an itemset that does
not contain any items.
The transaction width is defined as the number of items present in a trans-
action. A transaction fy is said to contain an itemset X if X is a subset of
t7. For example, the secondtransaction shown in Table 6.2 contains the item-
set {Bread, Diapers} but not {Bread, Milk}. An important property of an
itemset is its support count, which refers to the number of transactions that
contain a particular itemset. Mathematically, the support count, o(X), for an
itemset X can be stated as follows:

o(X) : l{t,lx C ti, t,i

where the symbol | . I denote the number of elements in a set. In the data set
shown in Table 6.2, the support count for {Beer, Diapers, Milk} is equal to
two because there are only two transactions that contain all three items.

Association Rule An association rule is an implication expression of the

form X , Y, where X and Y are disjoint itemsets, i.e., X )Y : @. The
strength of an association rule can be measured in terms of its support and
confidence. Support determines how often a rule is applicable to a given
330 Chapter 6 Association Analysis

data set, while confidence determines how frequently items in Y appear in

transactions that contain X. The formal definitions of these metrics are

o(XuY) .
Support, s(X ------+
I/) : 1 (6.1)
.Atr
o(X uY)
Confidence, c(X --+ Y) : (6.2)
"(x)
Example 6.1. Consider the rule {Uitt, Diapers} -----*{eeer}. Since the
support count for {ltitt<, Diapers, Beer} is 2 and the total number of trans-
actions is 5, the rule's support is 2f 5 :0.4. The rule's confidenceis obtained
by dividing the support count for {ttitt<, Diapers, Beer} by the support count
for {Uitt<, Diapers}. Since there are 3 transactions that contain milk and di-
apers,the confidencefor this rule is 213: 0.67. I

Why Use Support and Confidence? Support is an important measure

because a rule that has very low support may occur simply by chance. A
low support rule is also likely to be uninteresting from a businessperspective
becauseit may not be profitable to promote items that customers seldom buy
together (with the exception of the situation described in Section 6.8). For
these reasons,support is often used to eliminate uninteresting rules. As will
be shown in Section 6.2.1, support also has a desirable property that can be
exploited for the efficient discovery of association rules.
Confidence, on the other hand, measures the reliability of the inference
made by a rule. For a given rule X > Y, the higher the confidence,the more
Iikely it is for Y to be present in transactions that contain X. Confidencealso
provides an estimate of the conditional probability of Y given X.
Association analysis results should be interpreted with caution. The infer-
ence made by an associationrule does not necessarilyimply causality. Instead,
it suggestsa strong co-occurrencerelationship between items in the antecedent
and consequentof the rule. Causality, on the other hand, requires knowledge
about the causal and effect attributes in the data and typically involves rela-
tionships occurring over time (e.g., ozone depletion leads to global warming).

Formulation of Association Rule Mining Problem The association

rule mining problem can be formally stated as follows:
Definition 6.1 (Association Rule Discovery). Given a set of transactions
7, find all the rules having support ) minsup and confidence ) minconf ,
where m'insup and m'inconf are the corresponding support and confidence
thresholds.
 6.1 ProblemDefinition 331

A brute-force approach for mining associationrules is to compute the sup-

port and confidence for every possible rule. This approach is prohibitively
expensive because there are exponentially many rules that can be extracted
from a data set. More specifically,the total number of possiblerules extracted
from a data set that contains d items is

R:Jd -2d+r +I. (6.3)

The proof for this equation is left as an exerciseto the readers (seeExercise 5
on page 405). Even for the small data set shown in Table 6.1, this approach
requires us to compute the support and confidencefor 36 - 27 * 1 : 602 rules.
More than 80% of the rules are discarded after applying mi,nsup : 20Vo and
minconf : 5070,thus making most of the computations become wasted. To
avoid performing needlesscomputations, it would be useful to prune the rules
early without having to compute their support and confidencevalues.
An initial step toward improving the performance of association rule min-
ing algorithms is to decouple the support and confidencerequirements. From
Equation 6.2, notice that the support of a rule X -----+Y depends only on
the support of its corresponding itemset, X U Y. For example, the following
rules have identical support because they involve items from the same itemset,
{Beer, Diapers, MiIk}:
----* ------
{Beer, Diapers} {t'li.ft}, {Beer, Milk} {Diapers},
-----r ----*
{Diapers, Milk} {eeer}, {eeer} {Diapers, Milk},
------ ------+
{uirt} {Beer,Diapers}, {oiapers} {Beer,Milk}.
If the itemset is infrequent, then all six candidate rules can be pruned imme-
diately without our having to compute their confidencevalues.
Therefore, a common strategy adopted by many association rule mining
algorithms is to decomposethe problem into two major subtasks:
1. FYequent Itemset Generation, whose objective is to find all the item-
sets that satisfy Lhemi,nsup threshold. These itemsets are called frequent
itemsets.

2. Rule Generation, whose objective is to extract all the high-confidence

rules from the frequent itemsets found in the previous step. These rules
are called strong rules.
The computational requirements for frequent itemset generation are gen-
erally more expensive than those of rule generation. Efficient techniques for
generating frequent itemsets and associationrules are discussedin Sections6.2
and 6.3, respectively.
332 Chapter 6 Association Analysis

Figure
6.1.Anitemset
lattice.

6.2 Frequent Itemset Generation

A lattice structure can be used to enumerate the list of all possible itemsets.
Figure 6.1 showsan itemset lattice for 1: {a,b,c.,d,e}.In general,a data set
that contains k items can potentially generate up to 2k - 7 frequent itemsets,
excluding the null set. Because k can be very large in many practical appli-
cations, the search space of itemsets that need to be explored is exponentially
Iarge.
A brute-force approach for finding frequent itemsets is to determine the
support count for every candidate itemset in the lattice structure. To do
this, we need to compare each candidate against every transaction, an opera-
tion that is shown in Figure 6.2. If the candidate is contained in a transaction,
its support count will be incremented. For example, the support for {Bread,
Milk) is incremented three times becausethe itemset is contained in transac-
tions 1, 4, and 5. Such an approach can be very expensivebecauseit requires
O(N Mw) comparisons,where l/ is the number of transacti ons, M : 2k - | is
the number of candidate itemsets, and tl is the maximum transaction width.
 6.2 Frequent Itemset Generation 333

t
N
M

thesupport
Figure6.2. Counting itemsets.
ofcandidate

There are several ways to reduce the computational complexity of frequent

itemset generation.

1. Reduce the number of candidate itemsets (M). The Apri'ori' prin'

ciple, described in the next section, is an effectiveway to eliminate some
of the candidate itemsets without counting their support values.

2. Reduce the number of comparisons. Instead of matching each can-

didate itemset against every transaction, we can reduce the number of
comparisons by using more advanced data structures, either to store the
candidate itemsets or to compress the data set. We will discuss these
strategies in Sections 6.2.4 and 6.6.

6.2.L The Apriori Principle

This section describes how the support measure helps to reduce the number
of candidate itemsets explored during frequent itemset generation. The use of
support for pruning candidate itemsets is guided by the following principle.

Theorem 6.I (Apriori Principle). If an'itemset'is frequent, then all of its

subsetsmust also be frequent.

To illustrate the idea behind the Apri,ore principle, consider the itemset
Iattice shown in Figure 6.3. Suppose {c, d, e} is a frequent itemset. Clearly,
any transaction that contains {c,d,e} must also contain its subsets, {",d},
{","}, {d,e}, {"}, {d}, and {e}. As a result, if {c,d,e} is frequent, then
all subsets of {c, d,e} (i.e., the shaded itemsets in this figure) must also be
frequent.
/
334 Chapter 6 Association Analysis

Figure6,3. An illustration
ol lhe Aprioriprinciple.
lt {c,d,e} is frequent,
thenall subsets
of this
itemset
arefrequent.

Conversely,if an itemset such as {a, b} is infrequent, then all of its supersets

must be infrequent too. As illustrated in Figure 6.4, the entire subgraph
containing the supersets of {o, b} can be pruned immediately once {a, b} is
found to be infrequent. This strategy of trimming the exponential search
space based on the support measure is known as support-based pruning.
Such a pruning strategy is made possible by a key property of the support
measure, namely, that the support for an itemset never exceedsthe support
for its subsets. This property is also known as the anti-monotone property
of the support measure.
Definition 6.2 (Monotonicity Property). Let 1 be a set of items, and
J :21 be the pov/er set of 1. A measure / is monotone (or upward closed) if

YX,Y e J : (X eY) ------,

f(X) < f(Y),
 6.2 Frequent Itemset Generation 335

\r
Pruned
\\--_
Supersets

Figure6.4.Anillustration pruning.
ofsupport-based lt {a,b} isinfrequent, of {a,b}
thenallsupersets
areinfrequent.

which means that if X is a subset of Y, then /(X) must not exceed/(f). On

the other hand, / is anti-monotone (or downward closed) if

vx,YeJ: (X qY) - f(Y)<f(x),

which means that if X is a subset of Y, then /(Y) must not exceed /(X).
Any measure that possessesan anti-monotone property can be incorpo-
rated directly into the mining algorithm to effectively prune the exponential
search space of candidate itemsets, as will be shown in the next section.

6.2.2 Fbequent Itemset Generation in the Apri'ori Algorithm

Apri,ori, is the first association rule mining algorithm that pioneered the use
of support-based pruning to systematically control the exponential growth of
candidate itemsets. Figure 6.5 provides a high-level illustration of the frequent
itemset generation part of the Apriori algorithm for the transactions shown in
336 Chapter 6 Association Analysis

Candidate
1-ltemsets

Minimum count= 3
support

Itemsetsremoved
becauseof low
support
Candidate
3-ltemsets
Itemset Count
{Bread,Diapers,Milk} 3

Figure6.5. llluskation
of frequent generation
itemset usingtheApriorialgorithm.

Table 6.1. We assumethat the support threshold is60To, which is equivalent

to a minimum support count equal to 3.
Initially, every item is considered as a candidate l-itemset. Afber count-
ing their supports, the candidate itemsets {Co1a} and {Eggs} are discarded
becausethey appear in fewer than three transactions. In the next iteration,
candidate 2-itemsets are generated using only the frequent 1-itemsetsbecause
the Apri,orz principle ensures that all supersets of the infrequent 1-itemsets
must be infrequent. Because there are only four frequent 1-itemsets, the num-
ber of candidate 2-itemsets generated by the algorithm is ( f ) : O. Two
of these six candidates, {Beer, Bread} and {Beer, Milk}, are subsequently
found to be infrequent after computing their support values. The remain-
ing four candidates are frequent, and thus will be used to generate candidate
3-itemsets. Without support-based.pruning, there are ( ! ) : 2O candidate
3-itemsets that can be formed using the six items given in this example. With
the Apri,ori principle, we only need to keep candidate 3-itemsetswhose subsets
are frequent. The only candidate that has this property is {Bread, Diapers,
Mirk).
The effectiveness of the Apri,ore pruning strategy can be shown by count-
ing the number of candidate itemsets generated. A brute-force strategy of
 6.2 FreouentItemset Generation 337

enumeratingall itemsets(up to size3) as candidateswill produce

(?). (!) . (:) :6 + 15t20:4r

candidates. With the Apri,ori principle, this number decreasesto

(l).(i).1:6*6*1:13
candidates, which represents a 68% reduction in the number of candidate
itemsets even in this simple example.
The pseudocode for the frequent itemset generation part of the Apri,ori,
algorithm is shown in Algorithm 6.1. Let Cp denote the set of candidate
k-itemsets and Fr denote the set of frequent k-itemsets:

o The algorithm initially makes a single passover the data set to determine
the support of each item. Upon completion of this step, the set of all
frequent 1-itemsets, f'r, will be known (steps 1 and 2).

o Next, the algorithm will iteratively generate new candidate k-itemsets

using the frequent (k - 1)-itemsets found in the previous iteration (step
5). Candidate generation is implemented using a function called apriori-
gen, which is described in Section 6.2.3.

Algorithm 6.1 Flequent itemset generation of the Apriori, algorithm.

I: k :1.
2: F1"- { i I i e I no({t}) > lr' x minsup}. {Find all frequent l-itemsets}
3: repeat
4: k:k+7.
5: Cr : apriori-gen(Fr-1). {Generate candidate itemsets}
6: for each transaction t eT do
7: Ct : subset(Cn, t). {Identify all candidates that belong to l}
8: for each candidate itemset c € C1 do
9: o(c) : o(c) + 1. {Increment support count}
10: end for
11: end for
12: Fr: { cl c€Cp Ao(c) ) N x mi.nsup}. {Extract the frequent /c-itemsets}
13: until F* : A
14: Result : UF*.
338 Chapter 6 Association Analysis

To count the support of the candidates, the algorithm needs to make an

additional pass over the data set (steps 6 10). The subset function is
used to determine all the candidate itemsets in Ca that are contained in
each transaction f. The implementation of this function is described in
Section 6.2.4.

After counting their supports, the algorithm eliminates all candidate

itemsets whose support counts are less than mi,nsup (step 12).

The algorithm terminates when there are no new frequent itemsets gen-
erated,i.e., Fp: 0 (step 13).

The frequent itemset generation part of the Apri,orz algorithm has two im-
portant characteristics. First, it is a level-wise algorithm; i.e., it traversesthe
itemset lattice one level at a time, from frequent 1-itemsets to the maximum
size of frequent itemsets. Second, it employs a generate-and-test strategy
for finding frequent itemsets. At each iteration, new candidate itemsets are
generated from the frequent itemsets found in the previous iteration. The
support for each candidate is then counted and tested against the minsup
threshold. The total number of iterations neededby the algorithm is k-.* a 1,
where k-r* is the maximum size of the frequent itemsets.

6.2.3 Candidate Generation and Pruning

The apriori-gen function shown in Step 5 of Algorithm 6.1 generatescandidate
itemsets by performing the following two operations:

1. Candidate Generation. This operation generates new candidate k-

itemsets based on the frequent (k - l)-itemsets found in the previous
iteration.

2. Candidate Pruning. This operation eliminates some of the candidate

k-itemsets using the support-based pruning strategy.
To illustrate the candidate pruning operation, consider a candidate /c-itemset,
X: {h,i2,...,ip}. The algorithm must determinewhether all of its proper
subsets,X - {li} (Vl : 1,2,...,k), are frequent. If one of them is infre-
quent, then X is immediately pruned. This approach can effectively reduce
the number of candidate itemsets considered during support counting. The
complexity of this operation is O(k) for each candidate k-itemset. However,
as will be shown later, we do not have to examine all k subsets of a given
candidate itemset. If m of the k subsets were used to generate a candidate,
we only need to check the remaining k - rn subsetsduring candidate pruning.
 6.2 Frequent Itemset Generation 339

In principle, there are many ways to generate candidate itemsets. The fol-
Iowing is a list of requirements for an effectivecandidate generation procedure:

1. It should avoid generating too many unnecessarycandidates. A candi-

date itemset is unnecessaryif at least one of its subsets is infrequent.
Such a candidate is guaranteed to be infrequent according to the anti-
monotone property of support.

2. It must ensure that the candidate set is complete, i.e., no frequent item-
sets are left out by the candidate generation procedure. To ensure com-
pleteness,the set of candidate itemsets must subsume the set of all fre-
quent itemsets, i.e., Vk : fi, C Cp.

3. It should not generate the same candidate itemset more than once. For
example, the candidate itemset {a,b,c,d} can be generated in many
ways-by merging {a,b, c} with {d}, {b, d} with {o, {c} with {a,b, d},
"},
etc. Generation of duplicate candidates leads to wasted computations
and thus should be avoided for efficiency reasons.

Next, we will briefly describe several candidate generation procedures, in-

cluding the one used by the apriori-gen function.

Brute-Force Method The brute-force method considersevery k-itemset as

a potential candidate and then applies the candidate pruning step to remove
any unnecessarycandidates (see Figure 6.6). The number of candidate item-
sets generated at level k is equal to ( i, ), where d is the total number of items.
Although candidate generation is rather trivial, candidate pruning becomes
extremely expensive because a large number of itemsets must be examined.
Given that the amount of computations needed for each candidate is O(k),
the overall complexity of this method k O(Dfl:, tt (1")) : O(d .2d-t).
"

Fr-r x F1 Method An alternative method for candidate generation is to

extend each frequent (k - 1)-itemset with other frequent items. Figure 6.7
illustrates how a frequent 2-itemset such as {Beer, Diapers} can be aug-
mented with a frequent item such as Bread to produce a candidate 3-itemset
{Beer, Diapers, Bread}. This method will produce O(lFn_tl x lF1l) candi-
date,k-itemsets, where l4l ir the number of frequent j-itemsets. The overall
complexity of this step is O(DnklFn_tllF l)
The procedure is complete becauseevery frequent k-itemset is composed
of a frequent (k - 1)-itemset and a frequent 1-itemset. Therefore, all frequent
k-itemsets are part of the candidate k-itemsets generated by this procedure.
34O Chapter 6 Association Analysis

CandidateGeneration
Itemset
Itreer, trreao, uora
iBeer,Bread,Diapers)
{Beer,Bread,Milk}
{Beer,Bread,Eqqs}
{Beer,Cola,Diapers}
{Beer,Cola,Milk}
itseer, uola, Eggs'
{Beer,Diaoers.Milk}
{Beer,Diapers,Eggs}
{Beer, Milk, Eqqs}
{Bread,Cola,Diapers}
{Bread, Cola, Milk}
{Dreao, uora, trqqsl
{Bread,Diapers,Milk}
{Breao,Drapers,
Eqqsl
{Bread, Milk, Eqqs}
lola, Diapers, Milk)
{Cola,Diapers,Eqqs)
{Cola,Milk,Eggs}
{Diapers,Milk,Eqqs)

Figure6.6.A brute-force forgenerating

method candidate
3-itemsets.

Freouent
2-itemset

Candidate
CandidateGeneration
Pruning
Itemset
Itemset
{Beer,Diapers.Bread)
{Bread,Diapers,
Milk}
Frequent {Beer, Diapers.Milk}
1-itemset {Bread. Diapers,Milk}
{Bread, Milk, Beer}

Figure6,7.Generatingandpruning
candidate
k-itemsets
bymerging
a frequent(k - l)-itemsetwitha
item.Notethatsomeofthecandidates
frequent because
areunnecessary theirsubsets are infrequent,

This approach, howevet, does not prevent the same candidate itemset from
being generated more than once. For instance, {Bread, Diapers, Mitk} can
be generatedby merging {Bread, Diapers} with {Milk}, {Bread, MiIk} with
{niapers}, or {Diapers, MiIk} with {Bread}. One way to avoid generating
 6.2 Requent Itemset Generation 34L

duplicate candidates is by ensuring that the items in each frequent itemset are
kept sorted in their lexicographic order. Each frequent (/c-l)-itemset X is then
extended with frequent items that are lexicographically larger than the items in
X. For example, the itemset {Bread, Diapers} can be augmented with {Milk}
since Mil-k is lexicographically larger than Bread and Diapers. However, we
should not augment {Diapers, MiIk} with {Bread} nor {Bread, Milk} with
{Oi-apers} becausethey violate the lexicographic ordering condition.
While this procedure is a substantial improvement over the brute-force
method, it can still produce a large number of unnecessarycandidates. For
example, the candidate itemset obtained by merging {Beer, Diapers} with
{Ui-ft} is unnecessarybecauseone of its subsets, {Beer, Milk}, is infrequent.
There are several heuristics available to reduce the number of unnecessary
candidates. For example, note that, for every candidate k-itemset that survives
the pruning step, every item in the candidate must be contained in at least
k - L of the frequent (k - 1)-itemsets. Otherwise, the candidate is guaranteed
to be infrequent. For example, {Beer, Diapers, Milk} is a viable candidate
3-itemset only if every item in the candidate, including Beer, is contained in
at least two frequent 2-itemsets. Since there is only one frequent 2-itemset
containing Beer, all candidate itemsets involving Beer must be infrequent.

Fr-r x F6-1 Method The candidate generation procedure in the apriori-gen

function mergesa pair of frequent (k - l)-itemsets only if their first k - 2 items
a r e i d e n t i c a l L. e t A : { o t , a 2 t . . . , a k _ r } a n d B : { b t , b z , . . . , b n _ t }b e a p a i r
of frequent (/c- l)-itemsets. A and B are merged if they satisfy the following
conditions:

ai: b i ( f o r z : 1 , 2 , . . . , k - 2 ) a n d a 7 r - 1I b n - t .

In Figure 6.8, the frequent itemsets {Bread, Diapers} and {Bread, Milk} are
merged to form a candidate 3-itemset {Bread, Diapers, Milk}. The algorithm
does not have to merge {Beer, Di-apers} with {Diapers, Milk} becausethe
first item in both itemsets is different. Indeed, if {Beer, Diapers, Milk} is a
viable candidate, it would have been obtained by merging {Beer, Diapers}
with {Beer, MiIk} instead. This example illustrates both the completenessof
the candidate generation procedure and the advantagesof using lexicographic
ordering to prevent duplicate candidates. However, becauseeach candidate is
obtained by merging a pair of frequent (k-1)-itemsets, an additional candidate
pruning step is needed to ensure that the remaining k - 2 subsets of the
candidate are frequent.
342 Chapter 6 Association Analysis

Frequent
2-itemset

Candidate Candidate
Generation Pruning

Frequent
2-itemsel

andpruning
Figure6.8,Generating candidate
k-itemsets pairsoffrequent
bymerging (k- l)-itemsets.

6.2.4 Support Counting

Support counting is the process of determining the frequency of occurrence

for every candidate itemset that survives the candidate pruning step of the
apriori-gen function. Support counting is implemented in steps 6 through 11
of Algorithm 6.1. One approach for doing this is to compare each transaction
against every candidate itemset (see Figure 6.2) and to update the support
counts of candidates contained in the transaction. This approach is computa-
tionally expensive,especiallywhen the numbers of transactions and candidate
itemsets are large.
An alternative approach is to enumerate the itemsets contained in each
transaction and use them to update the support counts oftheir respectivecan-
didate itemsets. To illustrate, consider a transaction t that contains five items,
{I,2,3,5,6}. There are ( 3 ) : tO itemsetsof size 3 containedin this transac-
tion. Some of the itemsets may correspond to the candidate 3-itemsets under
investigation, in which case, their support counts are incremented. Other
subsets of t that do not correspond to any candidates can be ignored.
Figure 6.9 showsa systematic way for enumerating the 3-itemsetscontained
in l. Assuming that each itemset keeps its items in increasing lexicographic
order, an itemset can be enumerated by specifying the smallest item first,
followed by the larger items. For instance, given t : {L,2,3,5,6}, all the 3-
itemsets contained in f must begin with item 1, 2, or 3. It is not possible to
construct a 3-itemset that begins with items 5 or 6 becausethere are only two
 6.2 Frequent Itemset Generation 343

Transaction,
t
12356

Level 2

12

Level 3 Subsetsof 3 items

Figure6.9. Enumerating
subsets
ofthreeitemsfroma transaction
r.

items in f whose labels are greater than or equal to 5. The number of ways to
specify the first item of a 3-itemset contained in t is illustrated by the Level
1 prefix structures depicted in Figure 6.9. For instance, 1 [2J 5 6 I represents
a 3-itemset that begins with item 1, followed by two more items chosen from
t h e s e t{ 2 , 3 , 5 , 6 } .
After fixing the first item, the prefix structures at Level 2 represent the
number of ways to select the seconditem. For example, 1 2 F 5 6l corresponds
to itemsets that begin with prefix (1 2) and are followed by items 3, 5, or 6.
Finally, the prefix structures at Level 3 representthe complete set of 3-itemsets
contained in t. For example, the 3-itemsets that begin with prefix {1 2} are
{1,2,3}, {7,2,5}, and {1,2,6}, while those that begin with prefix {2 3} are
{2,3,5} and {2,3,6}.
The prefix structures shown in Figure 6.9 demonstrate how itemsets con-
tained in a transaction can be systematically enumerated, i.e., by specifying
their items one by one, from the leftmost item to the rightmost item. We
still have to determine whether each enumerated 3-itemset correspondsto an
existing candidate itemset. If it matches one of the candidates, then the sup-
port count of the corresponding candidate is incremented. In the next section,
we illustrate how this matching operation can be performed efficiently using a
hash tree structure.
344 Chapter 6 Association Analysis

Leaf nodes
containing
candidate
2-itemsets

Transactions

Bread,Milk,Diapers,Cola

Figure6.10.Counting
thesupport
of itemsets
usinghashstructure.

Support Counting Using a Hash Tlee

In the Apriori, algorithm, candidate itemsets are partitioned into different

buckets and stored in a hash tree. During support counting, itemsets contained
in each transaction are also hashed into their appropriate buckets. That way,
instead of comparing each itemset in the transaction with every candidate
itemset, it is matched only against candidate itemsets that belong to the same
bucket, as shown in Figure 6.10.
Figure 6.11 shows an example of a hash tree structure. Each internal node
of the tree uses the following hash function, h(p) : p mod 3, to determine
which branch of the current node should be followed next. For example, items
1, 4, and 7 are hashed to the same branch (i.e., the leftmost branch) because
they have the same remainder after dividing the number by 3. All candidate
itemsets are stored at the leaf nodes of the hash tree. The hash tree shown in
Figure 6.11 contains 15 candidate 3-itemsets, distributed across9 leaf nodes.
Consider a transaction, f, : {1,2,3,5,6}. To update the support counts
of the candidate itemsets, the hash tree must be traversed in such a way
that all the leaf nodes containing candidate 3-itemsets belonging to f must be
visited at least once. Recall that the 3-itemsets contained in t must begin with
items 1, 2,or 3, as indicated by the Level 1prefix structures shown in Figure
6.9. Therefore, at the root node of the hash tree, the items 1, 2, and 3 of the
transaction are hashed separately. Item 1 is hashed to the left child ofthe root
node, item 2 is hashed to the middle child, and item 3 is hashed to the right
child. At the next level of the tree, the transaction is hashed on the second
 6.2 FYeouentItemset Generation 345

Hash Function

-'-:,@
Transaction
F,*;]
,-.w

Figure6.11. Hashing attherootnodeof a hashtree.

a transaction

item listed in the Level 2 structures shown in Figure 6.9. For example, after
hashing on item 1 at the root node, items 2, 3, and 5 of the transaction are
hashed. Items 2 and 5 are hashed to the middle child, while item 3 is hashed
to the right child, as shown in Figure 6.12. This process continues until the
leaf nodes of the hash tree are reached. The candidate itemsets stored at the
visited leaf nodes are compared against the transaction. If a candidate is a
subset of the transaction, its support count is incremented. In this example, 5
out of the 9 leaf nodes are visited and 9 out of the 15 itemsets are compared
against the transaction.

6.2.5 Computational Complexity

The computational complexity of the Apri,ori, algorithm can be affected by the
following factors.

Support Threshold Lowering the support threshold often results in more

itemsets being declared as frequent. This has an adverse effect on the com-
346 Chapter 6 Association Analysis

1+ l23s6l
---

Transaction
t;;;-:-:r
lrzrcol
z+
-@
3+

Figure6.12.Subset ontheleftmost
operation oftherootof a candidate
subtree hashtree.

putational complexity of the algorithm becausemore candidate itemsets must

be generated and counted, as shown in Figure 6.13. The maximum size of
frequent itemsets also tends to increasewith lower support thresholds. As the
maximum size of the frequent itemsets increases,the algorithm will need to
make more passesover the data set.

Number of Items (Dimensionality) As the number of items increases,

more spacewill be neededto store the support counts of items. If the number of
frequent items also grows with the dimensionality of the data, the computation
and I/O costs will increasebecauseof the larger number of candidate itemsets
generated by the algorithm.

Number of Tbansactions Since the Apri,ori, algorithm makes repeated

passesover the data set, its run time increaseswith a larger number of trans-
actions.

Average TYansaction Width For densedata sets, the averagetransaction

width can be very large. This affectsthe complexity of the Apriori algorithm in
two ways. First, the maximum sizeof frequent itemsets tends to increaseas the
 6.2 Freouent Itemset Generation 347

x105
4

3.5

f;3
E
I zs
a
6

E r.s
3
E
z1 iil

0.5
\:, Y
il
0 It=--**-*-
10
Size of ltemsel

(a) Number of candidate itemsets.

, x105

E
o

E
;
d
E
z

-+J-f-+-+-+
0
0 10
Size of ltemsel

(b) Number of frequent itemsets.

Figure6.13. Effectof support onthenumber

threshold of candidate
andfrequent
itemsets.

averagetransaction width increases.As a result, more candidate itemsets must

be examined during candidate generation and support counting, as illustrated
in Figure 6.14. Second, as the transaction width increases, more itemsets
348 Chapter 6 Association Analysis

9 v
t
8 il
il
o_ il
o il .F.
o6 il vv
6
il
il
() il
YI
6 ill v
E3 t:
z
A\y
1
'+#t*.
0
10 15
Size ol ltemset

(a) Number of candidate itemsets.

x105
10

E
o

o
v
I
I
o4

z
2 \b
1 ;
4*+-. v
0
10 15
Size of ltemset

(b) Number of Frequent ltemsets.

Figure6.14.Effect
of average
transaction
widthonthenumber
ofcandidate itemsets.
andfrequent

are contained in the transaction. This will increase the number of hash tree
traversals performed during support counting.
A detailed analysis of the time complexity for the Apri,ori, algorithm is
presented next.
 6.3 Rule Generation 349

Generation of frequent l-itemsets For each transaction, we need to up-

date the support count for every item present in the transaction. Assuming
that tr is the averagetransaction width, this operation requires O(l/tr) time,
where .A[is the total number of transactions.

Candidate generation To generate candidate k-itemsets, pairs of frequent

(k - l)-itemsets are merged to determine whether they have at least k - 2
items in common. Each merging operation requires at most k - 2 equality
comparisons. In the best-casescenario, every merging step produces a viable
candidate k-itemset. In the worst-casescenario,the algorithm must merge ev-
ery pair of frequent (k - 1)-itemsetsfound in the previous iteration. Therefore,
the overall cost of merging frequent itemsets is

- z)lcnl( costormerging
.irr - 2)lLn-r12.
iU
k:2 k:2

A hash tree is also constructed during candidate generation to store the can-
didate itemsets. Because the maximum depth of the tree is k, the cost for
populating the hash tree with candidate itemsets rs O(l[:2klcrl). During
candidate pruning, we need to verify that the ,k- 2 subsetsof every candidate
k-itemset are frequent. Since the cost for looking up a candidate in a hash
tree is O(k), the candidate pruning step require" O(D|:rk(k - z)lckl) time.

Support counting Each transaction of length ltl produces (lll) itemsets of

size k. This is also the effective number of hash tree traversals performed for
each transaction. The cost for support counting is O(,n/Dr(T)"r), where tr
is the maximum transaction width and a7,is the cost for updating the support
count of a candidate k-itemset in the hash tree.

6.3 Rule Generation

This section describes how to extract association rules efficiently from a given
frequent itemset. Each frequent k-itemse t, Y , can produce up to 2k - 2 associa-
tion rules, ignoring rules that have empty antecedentsor consequents(0 ---- Y
orY ------+
0). An association rule can be extracted by partitioning the itemset
Y into two non-empty subsets,X and Y -X, suchthat X ------+Y -X satisfies
the confidencethreshold. Note that all such rules must have already met the
support threshold becausethey are generated from a frequent itemset.
350 Chapter 6 Association Analysis

Example 6.2. Let X : {I,2,3} be a frequent itemset. There are six candi-
date associationrules that can be generatedfrom X: {1,2} - i3}, {1,3} -
.----* ----- ._--> ----*
{2}, {2,3} {1}, {U {2,3}, {2} {1,3}, and {3} {1,2}. As
each of their support is identical to the support for X, the rules must satisfy
the support threshold. r
Computing the confidenceof an associationrule does not require additional
scansof the transaction data set. Consider the rule {1,2} - {3}, which is
generatedfrom the frequent itemset X : {1,2,3}. The confi.dencefor this rule
is o({1, 2,3}) lo({1, 2}). Because{1, 2, 3} is frequent,the anti-monotoneprop-
erty of support ensuresthat {1,2} must be frequent, too. Since the support
counts for both itemsets were already found during frequent itemset genera-
tion, there is no need to read the entire data set again.

6.3.1 Confidence-Based Pruning

Unlike the support measure) confidence does not have any monotone property.
For example, the confidence for X ------. Y can be larger, smaller, or equal to the
c o n f i d e n c e f o r a n o t h erru l e * ,t,where *gX andf e Y (seeExercise
3 on page 405). Nevertheless,if we compare rules generated from the same
frequent itemset Y, the following theorem holds for the confidencemeasure.

Theorem 6.2. If a rule X ------+

Y - X doesnot sati,sfythe confidencethreshold,
then any rule Xt --+ Y - Xt , where X' ,is a subset of X, must not sati,sfythe
confidencethreshold as well.

To prove this theorem, consider the following two rules: Xt ---', Y - Xt and
X +Y-X,where
- XtcX. T h e c o n f i d e n c e otfh e r u l e sa r e o ( Y ) l o ( X / ) a n d
o(V) lo(X), respectively.SinceX/ is a subsetof X , o(Xt) > Therefore,
"(X).
the former rule cannot have a higher confidencethan the latter rule.

6.3.2 Rule Generation in Apriori Algorithm

The Apri,orz algorithm uses a level-wise approach for generating association

rules, where each level correspondsto the number of items that belong to the
rule consequent. Initially, all the high-confidencerules that have only one item
in the rule consequent are extracted. These rules are then used to generate
new candidate rules. For example, if {acd}------- {b} and {abd} ------{c} are
high-confidencerules, then the candidate rule {a,d} ------, {bc} is generated by
merging the consequentsof both rules. Figure 6.15 shows a lattice structure
for the associationrules generated from the frequent itemset {a,b,c,d}. If any
node in the lattice has low confidence, then according to Theorem 6.2, the
 6.3 Rule Generation 35L

I
I
\
\
\
\

aa Pruned
'..- --.'
Rules

Figure6.15,Pruning
ofassociation
rulesusingtheconfidence
measure.

entire subgraph spanned by the node can be pruned immediately. Suppose

the confidence lbr {bcd} ------'{o} is low. All the rules containing item a in
its consequent, including {cd} ----- {ab}, {bd} -----+{ac}, {bc) ------{od}, and
{d} - {abc} can be discarded.
A pseudocodefor the rule generation step is shown in Algorithms 6.2 and
6.3. Note the similarity between the ap-genrules procedure given in Algo-
rithm 6.3 and the frequent itemset generation procedure given in Algorithm
6.1. The only difference is that, in rule generation, we do not have to make
additional passesover the data set to compute the confidenceofthe candidate
rules. Instead, we determine the confidenceof each rule by using the support
counts computed during frequent itemset generation.

Algorithm 6.2 Rule generation of the Apri,ori algorithm.

1: for each frequent k-itemset f p, k > 2 do
2: H1:{iliefn} {l-item consequentsof the rule.}
3: call ap-genrules(/6,fI1.)
4: end for
352 Chapter 6 Association Analysis

Algorithm 6.3 Procedure ap-genrules(fp, H*).

t: k : lfpl {size of frequent itemset.}
Z, m: lH^l {size of rule consequent.}
3:ifk>mllthen
4: Hm+t : aPriori-gen(f1-).
5: for each h^+r € H*a1 do
6: conf : - h^+t).
")( f n ) l " U n
7: if conf m'inconf tl:.en
8: output the rule (f* - h*+t) ------+hm*r.
9: else
10: delete h*q1 from Hm+r.
11: end if
12: end for
13: call ap-genrules(/p,H-+r.)
14: end if

6.3.3 An Example: Congressional Voting Records

This section demonstrates the results of applying association analysis to the

voting records of members of the United States House of Representatives.The
data is obtained from the 1984 CongressionalVoting Records Database, which
is available at the UCI machine learning data repository. Each transaction
contains information about the party affiIiation for a representative along with
his or her voting record on 16 key issues. There are 435 transactions and 34
items in the data set. The set of items are listed in Table 6.3.
The Apri,ore algorithm is then applied to the data set with mi,nsup : 30To
and minconf : 90T0. Some of the high-confidence rules extracted by the
algorithm are shown in Table 6.4. The first two rules suggest that most of the
members who voted yes for aid to El Salvador and no for budget resolution and
MX missile are Republicans; while those who voted no for aid to EI Salvador
and yes for budget resolution and MX missile are Democrats. These high-
confidencerules show the key issuesthat divide members from both political
parties. If mi,nconf is reduced, we may find rules that contain issuesthat cut
across the party lines. For example, with mi.nconf : 40Vo,the rules suggest
that corporation cutbacks is an issue that receives almost equal number of
votes from both parties-52.3% of the members who voted no are Republicans,
while the remaining 47.7% of them who voted no are Democrats.
 6.4 CompactRepresentation
of FrequentItemsets 353

Table6.3.Listofbinaryattributes
fromthe1984United Congressional
States Voting
Records.
Source:
TheUCImachine learningrepository.

1. Republican 1 8 .aid to Nicaragua : no

2. Democrat 1 9 . MX-missile : yes
3. handicapped-infants -- yes 20. MX-missile : no
4. handicapped-infants : no 21. immigration : yes
5. water project cost sharing : yes 22. immigration : no
6. water project cost sharing : pe 23. synfuel corporation cutback : yes
7. budget-resolution - yes ,L synfuel corporation cutback : no
8. budget-resolution : no 25. education spending : yes
9. physician L," 1r"s2s - yes 26. education spending : 1e
10. physician fee freeze : no 27. right-to-sue : yes
11. aid to EI Salvador : yes 28. right-to-sue : no
12. aid to EI Salvador : no 29. crrme - yes
13. religious groups in schools : yes 30. crime : no
14. religious groups in schools : no 3 1 . duty-free-exports : yes
15. anti-satellite test f61 - 1les 32. duty-free-exports : no
16. anti-satellite test barr: no 33. export administration act : yes
17. aid to Nicaragua : yes 34. export administration act : no

Table
6.4.Association
rules
extracted
fromthe1984
United
States Voting
Congressional Records.
Association Rule Confidence
{budget resolution : no, Mx-missile:no, aid to El Salvador : yes } 9r.0%
------+
{Republican}
{budget resolution : y€s, MX-missile:yes, aid to El Salvador : no } 97.5%
-----+{Democrat}
{crime: y€s, right-to-sue : y€s, physician fee freeze : yes} 93.5To
------+
{Republican}
{crime : no, right-to-sue : no, physician fee freeze : no} l00Yo
------+
{Democrat}

6.4 Compact Representation of Flequent Itemsets

In practice, the number of frequent itemsets produced from a transaction data
set can be very large. It is useful to identify a small representative set of
itemsets from which all other frequent itemsets can be derived. Two such
representationsare presentedin this section in the form of maximal and closed
frequent itemset;s.
354 Chapter 6 Association Analysis

Figure6.16,Maximal
frequent
itemset.

6.4.L Maximal Flequent Itemsets

Definition 6.3 (Maximal Flequent ltemset). A maximal frequent item-

set is defined as a frequent itemset for which none of its immediate supersets
are frequent.
To illustrate this concept, consider the itemset lattice shown in Figure
6.16. The itemsets in the lattice are divided into two groups: those that are
frequent and those that are infrequent. A frequent itemset border, which is
representedby a dashed line, is also illustrated in the diagram. Every itemset
located above the border is frequent, while those located below the border (the
shaded nodes) are infrequent. Among the itemsets residing near the border,
{o, d}, {a, c, e), and {b, c, d, e} are consideredto be maximal frequent itemsets
becausetheir immediate supersets are infrequent. An itemset such as {o,d}
is maximal frequent becauseall of its immediate supersets,{a,b,d}, {a,c,d},
and {a, d,e}, are infrequent. In contrast, {o,"} is non-maximal becauseone
of its immediate supersets,{a, c, e}, is frequent.
Maximal frequent itemsets effectively provide a compact representation of
frequent itemsets. In other words, they form the smallest set of itemsets from
 6.4 Compact Representation of Frequent Itemsets 355

which all frequerrt itemsets can be derived. For example, the frequent itemsets
shown in Figure 6.16 can be divided into two groups:

o Frequent il;emsetsthat begin with item a and that may contain items c,
d, or e. This group includes itemsets such as {o), {o.,c), {a,d}, {a,e},
and{a,c,e}.

o Flequent it,emsetsthat begin with items b, c, d, or e. This group includes

itemsetssuch as {b}, {b, c}, {c,d},{b,c,d,e}, etc.

Requent itemsel;sthat belong in the first group are subsets of either {a,c,e}
or {4, d}, while those that belong in the secondgroup are subsetsof {b, c, d, e}.
Hence, the maximal frequent itemsets {a, c, e}, {o, d}, and {b, c, d, e} provide
a compact representation of the frequent itemsets shown in Figure 6.16.
Maximal frequent itemsets provide a valuable representation for data sets
that can produce very long, frequent itemsets, as there are exponentially many
frequent itemsets in such data. Nevertheless,this approach is practical only
if an efficient algorithm exists to explicitly find the maximal frequent itemsets
without havingto enumerate all their subsets. We briefly describe one such
approach in Secl;ion6.5.
Despite providing a compact representation, maximal frequent itemsets do
not contain the support information of their subsets. For example, the support
of the maximal fiequent itemsets {a,c,e}, {o,d}, and {b,c,d,e} do not provide
any hint about the support of their subsets. An additional pass over the data
set is therefore needed to determine the support counts of the non-maximal
frequent itemsets. In some cases, it might be desirable to have a minimal
representation of frequent itemsets that preservesthe support information.
We illustrate such a representation in the next section.

6.4.2 Closed Flequent Itemsets

Closed itemsets provide a minimal representation of itemsets without losing

their support infbrmation. A formal definition of a closed itemset is presented
below.
Definition 6.4 (Closed Itemset). An itemset X is closed if none of its
immediate supersetshas exactly the same support count as X.
Put another way, X is not closed if at least one of its immediate supersets
has the same support count as X. Examples of closed itemsets are shown in
Figure 6.17. To better illustrate the support count of each itemset, we have
associatedeach node (itemset) in the lattice with a list of its corresponding
356 Chapter 6 Association Analysis

1 aoc

abcd
oce
1,2,4
4 acde
de

Figure6.17.Anexample
oftheclosed
frequent (withminimum
itemsets countequalto40%).
support

transaction IDs. For example, since the node {b, c} is associatedwith transac-
tion IDs 1,2, and 3, its support count is equal to three. Flom the transactions
given in this diagram, notice that every transaction that contains b also con-
tains c. Consequently,the support for {b} is identical to {b, c} and {b} should
not be considered a closed itemset. Similarly, since c occurs in every transac-
tion that contains both a and d, the itemset {o,d} is not closed. On the other
hand, {b, c} is a closed itemset because it does not have the same support
count as any of its supersets.
Definition 6.5 (Closed Flequent Itemset). An itemset is a closed fre-
quent itemset if it is closedand its support is greater than or equal to m'insup.
In the previous example, assumingthat the support threshold is 40%, {b,c}
is a closed frequent itemset becauseits support is 60%. The rest of the closed
frequent itemsets are indicated by the shaded nodes.
Algorithms are available to explicitly extract closed frequent itemsets from
a given data set. Interested readers may refer to the bibliographic notes at the
end of this chapter for further discussionsof these algorithms. We can use the
closed frequent itemsets to determine the support counts for the non-closed
 6.4 Compact Representation of Requent Itemsets 357

Algorithm 6.4 Support counting using closed frequent itemsets.

1: Let C denote the set of closed frequent itemsets
2: Let k-.* denote the maximum size of closed frequent itemsets
3: F6-"* : {flf e C, lfl: k-.*} {Find all frequent itemsets of size k-.*.}
4: for k : k^u*- 1 downto 1 do
5: Fn : {f lf ( Fp',1, lf l: k} {Find all frequent itemsetsof size k.}
6: for each f e F* do
7: if f f C then
8: f.support:max{ft.suryortlfte Fn+r, f Cf'}
9: end if
10: end for
11: end for

frequent itemsets. For example, consider the frequent itemset {o,d} shown
in Figure 6.17. Becausethe itemset is not closed, its support count must be
identical to one of its immediate supersets. The key is to determine which
superset(amon6;{a,b,d}, {a,c,d}, or {a,d,e}) has exactly the same support
count as {a, d}. The Apri,ori,principle states that any transaction that contains
the superset of .l-a,d) must also contain {o,d}. However, any transaction that
contains {o,d} does not have to contain the supersets of {a,d}. For this
reason, the support for {a, d} must be equal to the largest support among its
supersets.Since {a, c, d} has a larger support than both {a,b,d} and {a,d,e},
the support for .[a,d] must be identical to the support for {4, c, d}. Using this
methodology, an algorithm can be developed to compute the support for the
non-closed frequent itemsets. The pseudocode for this algorithm is shown in
Algorithm 6.4. The algorithm proceeds in a specific-to-generalfashion, i.e.,
from the largest to the smallest frequent itemsets. This is because,in order
to find the support for a non-closedfrequent itemset; the support for all of its
supersetsmust be known.
To illustrate the advairtageof using closed frequent itemsets, consider the
data set shown in Table 6.5, which contains ten transactions and fifteen items.
The items can be divided into three groups: (1) Group -4, which contains
items ar through as; Q) Group B, which contains items b1 through b5i and
(3) Group C, which contains items c1 through c5. Note that items within each
group are perfectly associatedwith each other and they do not appear with
items from another group. Assuming the support threshold is 20To,the total
number of frequent itemsets is 3 x (25- 1) : 93. However, there are only three
closedfrequentitemsetsin the data: ({41,a2tayta4lab},{bt,bz,bz,b+,b5},and
{"t,"z,cs,ca,c5}). It is often sufficient to present only the closed frequent
itemsets to the analysts instead of the entire set of frequent itemsets.
358 Chapter 6 Association Analysis

Table
6.5.Atransaction
datasetforminino itemsets.
closed
TID A7 A2 AJ A4 As O1 bz bs ba b5 C1 C2 ca C4 Cb

I I 1 I I 1 0 0 0 0 0 0 U U 0 0
2 I 1 1_ 1_ 1 0 0 0 0 0 0 0 0 0 0
J 1 I 1. l. I 0 0 0 0 0 0 0 0 0 0
4 0 0 0 0 0 I 1 1 1 1 0 0 0 0 0
( 0 0 0 0 0 I 1 1 1 1 0 0 0 0 0
t) 0 0 0 0 0 I 1 1 1 1 0 0 0 0 0
7 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1
e 0 0 0 0 0 0 0 0 0 0 1 L I 1 1
rl 0 0 0 0 0 0 0 0 0 0 1 1 1 I I
10 0 0 0 0 0 0 0 0 0 0 1 1 I 1 1

Figure6.18.Relationships frequent,
among maximal
frequent,
andclosed itemsets.
frequent

Closed frequent itemsets are useful for removing some of the redundant
association rules. An association rule X ------Y is redundant if there exists
another rule X/ ------Y' , where X is a subset of X/ and Y is a subset of Y/, such
that the support and confidence for both rules are identical. In the example
shown in Figure 6.L7, {b} is not a closedfrequent itemset while {b, c} is closed.
The association rule {b} - {d, e} is therefore redundant becauseit has the
sarne support and confidence as {b, c} - {d,"}. Such redundant rules are
not generated if closed frequent itemsets are used for rule generation.
Finally, note that all maximal frequent itemsets are closed because none
of the maximal frequent itemsets can have the same support count as their
immediate supersets. The relationships among frequent, maximal frequent,
and closed frequent itemsets are shown in Figure 6.18.
 6.5 Alternative Methods for Generating Frequent Itemsets 359

6.5 Alternative Methods for Generating Fhequent

Itemsets
Apri,ori, is one of the earliest algorithms to have successfully addressed the
combinatorial explosion of frequent itemset generation. It achieves this by ap-
plying the Apii,ori principle to prune the exponential searchspace. Despite its
significant performance improvement, the algorithm still incurs considerable
I/O overhead since it requires making severalpassesover the transaction data
set. In addition, as noted in Section 6.2.5, the performance of the Apri,ori
algorithm may degrade significantly for densedata sets becauseof the increas-
ing width of transactions. Several alternative methods have been developed
to overcome these limitations and improve upon the efficiency of the Apri,ori,
algorithm. The following is a high-level description of these methods.

Tlaversal of Itemset Lattice A search for frequent itemsets can be con-

ceptually viewecl as a traversal on the itemset lattice shown in Figure 6.1.
The search strategy employed by an algorithm dictates how the lattice struc-
ture is traversed during the frequent itemset generation process. Some search
strategies are better than others, depending on the configuration of frequent
itemsets in the lattice. An overview of these strategies is presented next.

o General-t;o-Specific versus Specific-to-General: The Apriori, al-

gorithm usesa general-to-specificsearchstrategy, where pairs offrequent
(k- l)-itemsets are merged to obtain candidate k-itemsets. This general-
to-specific search strategy is effective, provided the maximum length of
a frequent itemset is not too long. The configuration of frequent item-
sets that works best with this strategy is shown in Figure 6.19(a), where
the darker nodes represent infrequent itemsets. Alternatively, a specific-
to-general search strategy looks for more specific frequent itemsets first,
before finding the more general frequent itemsets. This strategy is use-
ful to discover maximal frequent itemsets in dense transactions, where
the frequent itemset border is located near the bottom of the lattice,
as shown in Figure 6.19(b). The Apri,ori, principle can be applied to
prune all subsets of maximal frequent itemsets. Specifically, if a candi-
date k-itemset is maximal frequent, we do not have to examine any of its
subsets of size k - l. However, if the candidate k-itemset is infrequent,
we need to check all of its k - 1 subsets in the next iteration. Another
approach is to combine both general-to-specificand specific-to-general
search strategies. This bidirectional approach requires more space to
360 Chapter 6 Association Analysis

Frequent
Itemset Frequent
Border null Itemset null

\
{a1,a2,...,an} {a1,a2,...,an} ltemsel
Border
(a) General-to-specific (b) Specific-to-general (c) Bidirectional

Figure6.19.General-to-specific,
specific{o-general,
andbidirectional
search.

store the candidate itemsets, but it can help to rapidly identify the fre-
quent itemset border, given the configuration shown in Figure 6.19(c).

Equivalence Classes: Another way to envision the traversal is to first

partition the lattice into disjoint groups of nodes (or equivalenceclasses).
A frequent itemset generation algorithm searches for frequent itemsets
within a particular equivalence class first before moving to another equiv-
alence class. As an example, the level-wise strategy used in the Apri,ori,
algorithm can be considered to be partitioning the lattice on the basis
of itemset sizes;i.e., the algorithm discoversall frequent l-itemsets first
before proceeding to larger-sized itemsets. Equivalence classescan also
be defined according to the prefix or suffix labels of an itemset. In this
case, two itemsets belong to the same equivalence class if they share
a common prefix or suffix of length k. In the prefix-based approach,
the algorithm can search for frequent itemsets starting with the prefix
a before looking for those starting with prefixes b) c) and so on. Both
prefix-based and suffix-based equivalence classes can be demonstrated
using the tree-like structure shown in Figure 6.20.

Breadth-First versus Depth-First: The Apriori, algorithm traverses

the lattice in a breadth-first manner) as shown in Figure 6.2L(a). It first
discovers all the frequent 1-itemsets, followed by the frequent 2-itemsets,
and so on, until no new frequent itemsets are generated. The itemset
 6.5 Alternative Methods for Generating Flequent Itemsets 361

\.--/

(a) Prefixtree.

Figure
6,20"Equivalence
classes ontheprefix
based andsuffix
labels
ofitemsets.

(a) Breadthfirst (b) Depthfirst

Figure
6,21.Breadth{irst
anddepth-first
traversals.

lattice can also be traversed in a depth-first manner, as shown in Figures

6.21(b) and 6.22. The algorithm can start from, say, node a, in Figure
6.22, and count its support to determine whether it is frequent. If so, the
algorithm progressivelyexpands the next level of nodes, i.e., ab, abc, and
so on, until an infrequent node is reached, say, abcd. It then backtracks
to another branch, say, abce, and continues the search from there.
The deprth-first approach is often used by algorithms designed to find
maximal frequent itemsets. This approach allows the frequent itemset
border to be detected more quickly than using a breadth-first approach.
Once a merximal frequent itemset is found, substantial pruning can be
362 Chapter 6 Association Analysis

I \
I o )l
I I- t,
I
I I
I
qc I
I de I
\
tt
)r bcd
ade bce bde cde r/
//
--------t'

abce abde

abcde

Figure6.22.Generating
candidate
itemsets
usingthedepth{irst
approach.

performed on its subsets. For example, if the node bcde shown in Figure
6.22 is maximal frequent, then the algorithm does not have to visit the
subtrees rooted at bd,,be, c, d, and e becausethey will not contain any
maximal frequent itemsets. However, if abcis maximal frequent, only the
nodes such as ac and bc arc not maximal frequent (but the subtrees of
ac and bc may still contain maximal frequent itemsets). The depth-first
approach also allows a different kind of pruning based on the support
of itemsets. For example, suppose the support for {a,b,c} is identical
to the support for {a, b}. The subtrees rooted at abd and abe can be
skipped because they are guaranteed not to have any maximal frequent
itemsets. The oroof of this is left as an exerciseto the readers.

Representation of Transaction Data Set There are many ways to rep-

resent a transaction data set. The choice of representation can affect the I/O
costs incurred when computing the support of candidate itemsets. Figure 6.23
shows two different ways of representing market basket transactions. The rep-
resentation on the left is called a horizontal data layout, which is adopted
by many association rule mining algorithms, including Apri,ori,. Another pos-
sibility is to store the list of transaction identifiers (TID-list) associatedwith
each item. Such a representation is known as the vertical data layout. The
support for each candidate itemset is obtained by intersecting the TlD-lists of
its subset items. The length of the TlD-lists shrinks as we progress to larger
 6.6 FP-Growth Algorithm 363

Horizontal
Data Layout VerticalData Layout

'l 2 1
1 2
4 2 3 4 3
5 5 4 5 o
6 7 8 I
7 I I
8 10
9

dataformat.
andvertical
Figure6.23,Horizontal

sized itemsets. [fowever, one problem with this approach is that the initial
set of TlD-lists may be too large to fit into main memory, thus requiring
more sophisticated techniquesto compressthe TlD-lists. We describe another
effective approach to represent the data in the next section.

6.6 FP-Growth Algorithm

This section presents an alternative algorithm called FP-growth that takes
a radically different approach to discovering frequent itemsets. The algorithm
does not subscribe to the generate-and-testparadigm of Apriori^ Instead, it
encodesthe data set using a compact data structure called an FP-tree and
extracts frequent itemsets directly from this structure. The details of this
approach are presented next.

6.6.1 FP-Tbee Representation

An FP-tree is a compressedrepresentation of the input data. It is constructed

by reading the data set one transaction at a time and mapping eachtransaction
onto a path in the FP-tree. As different transactions can have several items
in common, their paths may overlap. The more the paths overlap with one
another, the more compressionwe can achieve using the FP-tree structure. If
the size of the F.P-tree is small enough to fit into main memory, this will allow
us to extract frequent itemsets directly from the structure in memory instead
of making repeated passesover the data stored on disk.
364 Chapter 6 Association Analysis

Transaction
DataSet

b:1

(i) AfterreadingTID=1 (ii)AfterreadingTID=2

(iv)Afterreading
TID=10

Figure6.24,Construction
of anFP-tree.

Figure 6.24 shows a data set that contains ten transactions and five items.
The structures ofthe FP-tree after reading the first three transactions are also
depicted in the diagram. Each node in the tree contains the label of an item
along with a counter that shows the number of transactions mapped onto the
given path. Initially, the FP-tree contains only the root node representedby
the nulf symbol. The FP-tree is subsequently extended in the following way:

1. The data set is scanned once to determine the support count of each
item. Infrequent items are discarded, while the frequent items are sorted
in decreasingsupport counts. For the data set shown in Figure 6.24, a
is the most frequent item, followed by b, c, d, and e.
 6.6 FP-GrowthAlgorithm 365

2. The algoril;hm makes a second pass over the data to construct the FP-
tree. After reading the first transaction, {o,b), the nodes labeled as a
and b are created. A path is then formed from nulI --+a '--+b to encode
the transaction. Every node along the path has a frequency count of 1.

3. After reading the second transaction, {b,cd}, a new set of nodes is cre-
ated for items b, c, arrd d. A path is then formed to represent the
transaction by connecting the nodes null ---+b ---+c -* d. Every node
along this path also has a frequency count equal to one. Although the
first two transactions have an item in common, which is b, their paths
are disjoint becausethe transactions do not share a common prefix.

4. The third transaction, {a,cd,e}, shares a common prefix item (which

is a) with the first transaction. As a result, the path for the third
transaction, null ) a ---+c ---+d --+ e, overlaps with the path for the
first transaction, nuII ---+a -- b. Becauseof their overlapping path, the
frequency count for node o is incremented to two, while the frequency
counts for the newly created nodes, c, d, and e) are equal to one.

5. This processcontinues until every transaction has been mapped onto one
of the paths given in the FP-tree. The resulting FP-tree after reading
all the transactions is shown at the bottom of Figure 6.24.

The sizeof an FP-tree is typically smaller than the sizeof the uncompressed
data because ma,ny transactions in market basket data often share a few items
in common. In the best-case scenario, where all the transactions have the
same set of iterns, the FP-tree contains only a single branch of nodes. The
worst-casescenario happens when every transaction has a unique set of items.
As none of the transactions have any items in common, the size of the FP-tree
is effectively the same as the size of the original data. However, the physical
storage requirement for the FP-tree is higher because it requires additional
space to store pointers between nodes and counters for each item.
The size of an FP-tree also depends on how the items are ordered. If
the ordering scheme in the preceding example is reversed, i.e., from lowest
to highest support item, the resulting FP-tree is shown in Figure 6.25. The
tree appears to be denser because the branching factor at the root node has
increased from 2 to 5 and the number of nodes containing the high support
items such as a and b has increased from 3 to 12. Nevertheless, ordering
by decreasing support counts does not always lead to the smallest tree. For
exanlple, suppose we augment the data set given in Figure 6.24 with 100
transactions that contain {e}, 80 transactions that contain {d}, 60 transactions
366 Chapter 6 Association Analysis

..-. c:2
d:2
b:1
"'it

"-a:l

Figure6.25. An FP-treerepresentation
for thedataset shownin Figure6.24witha different
item
ordering
scheme.

that contain {"}, and 40 transactions that contain {b}. Item e is now most
frequent, followed by d, c, b, and a. With the augmented transactions, ordering
by decreasingsupport counts will result in an FP-tree similar to Figure 6.25,
while a schemebased on increasing support counts produces a smaller FP-tree
similar to Figure 6.2a$v).
An FP-tree also contains a list of pointers connecting between nodes that
have the same items. These pointers, represented as dashed lines in Figures
6.24 and 6.25, help to facilitate the rapid accessof individual items in the tree.
We explain how to use the FP-tree and its corresponding pointers for frequent
itemset generation in the next section.

6.6.2 FYequent Itemset Generation in FP-Growth Algorithm

FP-growth is an algorithm that generatesfrequent itemsets from an FP-tree
by exploring the tree in a bottom-up fashion. Given the example tree shown in
Figure 6.24, the algorithm looks for frequent itemsets ending in e first, followed
by d, c, b, and finally, a. This bottom-up strategy for finding frequent item-
sets ending with a particular item is equivalent to the suffix-based approach
described in Section 6.5. Since every transaction is mapped onto a path in the
FP-tree, we can derive the frequent itemsets ending with a particular item,
sayTe, by examining only the paths containing node e. These paths can be
accessedrapidly using the pointers associated with node e. The extracted
paths are shown in Figure 6.26(a). The details on how to processthe paths to
obtain frequent itemsets will be explained later.
 6.6 FP-Growth Algorithm 367

c:3

d:1

(a) Pathscontainingnode e (b) Pathscontainingnode d

null null null

o
,\
)b:2 a:8
a:8
c:2
b:5
c:3
(c) Pathscontainingnode c (d) Pathscontainingnode b (e) Pathscontainingnode a

Figure6.26.Decomposing
thefrequent generation
itemset problemintomultiple where
subproblems,
eachsubproblem
involves
findingfrequent
itemsets ine,d,c,b,anda.
ending

Table6.6.Thelistoffrequent
itemsets
ordered suffixes.
bytheircorresponding
Suffix FbequentItemsets
e {e}, {d,e}, {a,d,e}, {c,e},{a,e}
d {d}, {c,d}, {b,c,d}, {a,c,d}, {b,d}, {a,b,d}, {a,d}
c {c}, { b , c } , { a , b , c } ,{ a , c }
b {b}, {a,b}
a {a}

After finding the frequent itemsets ending in e, the algorithm proceedsto

look for frequent itemsets ending in d by processingthe paths associatedwith
node d. The corresponding paths are shown in Figure 6.26(b). This process
continues until all the paths associated with nodes c, b, and finally a) are
processed. The paths for these items are shown in Figures 6.26(c), (d), and
(e), while their corresponding frequent itemsets are summarized in Table 6.6.
FP-growth finds all the frequent itemsets ending with a particular suffix
by employing a divide-and-conquer strategy to split the problem into smaller
subproblems. For example, suppose we are interested in finding all frequent
368 Chapter 6 Association Analysis

d:1 d:1
(b) ConditionalFP-treefor e

(c) Prefixpathsendingin de (d) ConditionalFP-treefor de

n u l l .)
I
d
c:1 c:1 a:2
(e) Prefixpathsendingin ce (f) Prefixpathsending
in ae

Figure6.27.Example
ofapplying
theFP-growth tofindfrequent
algorithm itemsets ine.
ending

itemsets ending in e. To do this, we must first check whether the itemset

{e} itself is frequent. If it is frequent, we consider the subproblem of finding
frequent itemsets ending in de, followed by ce, be, and ae. In turn, each
of these subproblems are further decomposed into smaller subproblems. By
merging the solutions obtained from the subproblems,all the frequent itemsets
ending in e can be found. This divide-and-conquer approach is the key strategy
employed by the FP-growth algorithm.
For a more concrete example on how to solve the subproblems, consider
the task of finding frequent itemsets ending with e.

1. The first step is to gather all the paths containing node e. These initial
paths are called prefix paths and are shown in Figure 6.27(a).

2. Fbom the prefix paths shown in Figure 6.27(a), the support count for e is
obtained by adding the support counts associatedwith node e. Assuming
that the minimum support count is 2, {e} is declared a frequent itemset
becauseits support count is 3.
 6.6 FP-GrowthAlgorithm 369

3 . Because {e} is frequent, the algorithm has to solve the subproblems of

finding frequent itemsets ending in de, ce, be, and ae. Before solving
these subproblems, it must first convert the prefix paths into a con-
ditional FP-tree, which is structurally similar to an FP-tree, except
it is used to find frequent itemsets ending with a particular suffix. A
conditional FP-tree is obtained in the following way:

(a) First, the support counts along the prefix paths must be updated
becausesome ofthe counts include transactions that do not contain
item e. For example, the rightmost path shown in Figure 6.27(a),
nufl ------+
b:2 -------+
c:2 ------+
e:1, includes a transaction {b,c} that
does not contain item e. The counts along the prefix path must
therefore be adjusted to 1 to reflect the actual number of transac-
tions containing {b, c, e}.
(b) The prefix paths are truncated by removing the nodes for e. These
nodes can be removed becausethe support counts along the prefix
pathsrhave been updated to reflect only transactions that contain e
and the subproblems of finding frequent itemsets ending in de, ce,
be, anrdae no longer need information about node e.
(c) After updating the support counts along the prefix paths, some
of the items may no longer be frequent. For example, the node b
appears only once and has a support count equal to 1, which means
that there is only one transaction that contains both b and e. Item b
can be safely ignored from subsequent analysis because all itemsets
ending in be must be infrequent.

The conditional FP-tree for e is shown in Figure 6.27(b). The tree looks
different than the original prefix paths becausethe frequency counts have
been updated and the nodes b and e have been eliminated.
A FP-growth usesthe conditional FP-tree for e to solve the subproblems of
finding frequent itemsets ending in de, ce, and ae. To find the frequent
itemsets ending in de, the prefix paths for d are gathered from the con-
ditional FP-tree for e (Figure 6.27(c)). By adding the frequency counts
associatedwith node d, we obtain the support count for {d,e}. Since
the support count is equal to 2, {d,e} is declared a frequent itemset.
Next, the algorithm constructs the conditional FP-tree for de using the
approach described in step 3. After updating the support counts and
removing the infrequent item c, the conditional FP-tree for de is shown
in Figure 6.27(d). Since the conditional FP-tree contains only one item,
37O Chapter 6 Association Analysis

o, whose support is equal to minsup, the algorithm extracts the fre-

quent itemset {a,d,e} and moves on to the next subproblem, which is
to generate frequent itemsets ending in ce. After processing the prefix
paths for c, only {c, e} is found to be frequent. The algorithm proceeds
to solve the next subprogram and found {a,e} to be the only frequent
itemset remaining.

This example illustrates the divide-and-conquer approach used in the FP-

growth algorithm. At each recursive step, a conditional FP-tree is constructed
by updating the frequency counts along the prefix paths and removing all
infrequent items. Because the subproblems are disjoint, FP-growth will not
generate any duplicate itemsets. In addition, the counts associatedwith the
nodes allow the algorithm to perform support counting while generating the
common suffix itemsets.
FP-growth is an interesting algorithm becauseit illustrates how a compact
representation of the transaction data set helps to efficiently generate frequent
itemsets. In addition, for certain transaction data sets, FP-growth outperforms
the standard Apriori, algorithm by severalorders of magnitude. The run-time
performance of FP-growth depends on the compaction factor of the data
set. If the resulting conditional FP-trees are very bushy (in the worst case, a
full prefix tree), then the performance of the algorithm degradessignificantly
becauseit has to generatea large number of subproblemsand merge the results
returned by each subproblem.

6.7 Evaluation of Association Patterns

Association analysis algorithms have the potential to generate a large number
of patterns. For example, although the data set shown in Table 6.1 contains
only six items, it can produce up to hundreds of association rules at certain
support and confidence thresholds. As the size and dimensionality of real
commercial databasescan be very large, we could easily end up with thousands
or even millions of patterns, many of which might not be interesting. Sifting
through the patterns to identify the most interesting ones is not a trivial task
because"one person's trash might be another person'streasure." It is therefore
important to establish a set of well-acceptedcriteria for evaluating the quality
of association patterns.
The first set of criteria can be established through statistical arguments.
Patterns that involve a set of mutually independent items or cover very few
transactions are considered uninteresting becausethey may capture spurious
relationships in the data. Such patterns can be eliminated by applying an
 6.7 Evaluation of Association Patterns 371

objective intenestingness measure that uses statistics derived from data

to determine whether a pattern is interesting. Examples of objective interest-
ingnessmeasuresinclude support, confidence,and correlation.
The secondset of criteria can be establishedthrough subjective arguments.
A pattern is consideredsubjectively uninteresting unless it reveals unexpected
information about the data or provides useful knowledge that can lead to
profitable actions. For example, the rrle {Butter} - {Bread} may not be
interesting, despite having high support and confidence values, because the
relationship represented by the rule may seem rather obvious. On the other
hand, the rule {Di,apers} ------{Beer} is interesting becausethe relationship is
quite unexpected and may suggest a new cross-selling opportunity for retailers.
Incorporating subjective knowledge into pattern evaluation is a difficult task
becauseit requires a considerableamount of prior information from the domain
experts.
The following are some of the approaches for incorporating subjective
knowledge into the pattern discovery task.

Visualization This approach requires a user-friendly environment to keep

the human user in the loop. It also allows the domain experts to interact with
the data mining system by interpreting and verifying the discoveredpatterns.

Template-based approach This approach allows the users to constrain

the type of patterns extracted by the mining algorithm. Instead of reporting
all the extracted rules, only rules that satisfy a user-specifiedtemplate are
returned to the users.

Subjective interestingness measure A subjective measurecan be defined

based on domain information such as concept hierarchy (to be discussedin
Section 7.3) or profit margin of items. The measure can then be used to filter
patterns that are obvious and non-actionable.

Readers interested in subjective interestingnessmeasuresmay refer to re-

sourceslisted in the bibliography at the end of this chapter.

6.7.t Objective Measures of Interestingness

An objective measure is a data-driven approach for evaluating the quality

of association patterns. It is domain-independent and requires minimal in-
put from the users, other than to specify a threshold for filtering low-quality
patterns. An objective measure is usually computed based on the frequency
372 Chapter 6 Association Analysis

Tabfe6.7.A 2-way
contingency
tableforvariables
A andB.
B B
A Jtl JLO fr+
T
A J01 ./ 00 fo+
J+O t/

counts tabulated in a contingency table. Table 6.7 shows an example of a

contingency table for a pair of binary variables, ,4 and B. We use the notation
A (B) to indicate that ,4 (B) is absent from a transaction. Each entry fii in
this 2 x 2 table denotes a frequency count. For example, fi1 is the number of
times A and B appear together in the same transaction, while /e1 is the num-
ber of transactions that contain B but not -4. The row sum fi-. represents
the support count for A, while the column sum /a1 represents the support
count for B. Finally, even though our discussion focuses mainly on asymmet-
ric binary variables, note that contingency tables are also applicable to other
attribute types such as symmetric binary, nominal, and ordinal variables.

Limitations of the support-confidence Flamework Existing associa-

tion rule mining formulation relies on the support and confidencemeasures,to
eliminate uninteresting patterns. The drawback of support was previously de-
scribed in Section 6.8, in which many potentially interesting patterns involving
low support items might be eliminated by the support threshold. The dra6-
back of confidenceis more subtle and is best demonstrated with the following
example.
Example 6.3. Suppose we are interested in analyzing the relationship be-
tween people who drink tea and coffee. We may gather information about the
beverage preferences among a group of people and summarize their responses
into a table such as the one shown in Table 6.8.

preferences
Table6.8.Beverage a group
among of1000people.
Cof f ee Cof f ee
Tea 150 50 200
Tea 650 150 800
800 200 1000
 6.7 Evaluation of Association Patterns 373

The information given in this table can be used to evaluate the association
rule {?ea,} ------,
{Cof f ee}. At fi.rstglance,it may appear that people who drink
tea also tend to drink coffee becausethe rule's support (15%) and confidence
(75%) values are reasonably high. This argument would have been acceptable
except that the fraction of people who drink coffee, regardless of whether they
drink tea, is 80%, while the fraction of tea drinkers who drink coffee is only
75%. Thus knowing that a person is a tea drinker actually decreasesher
probability of being a coffee drinker from 80% to 75Tol The rule {Tea) -,
{Cof f ee} is therefore misleading despite its high confidencevalue. r

The pitfall of confidence can be traced to the fact that the measure ignores
the support of the itemset in the rule consequent. Indeed, if the support of
coffee drinkers is taken into account, we would not be surprised to find that
many of the people who drink tea also drink coffee. What is more surprising is
that the fraction of tea drinkers who drink coffee is actually less than the overall
fraction of people who drink coffee, which points to an inverse relationship
between tea drinkers and coffeedrinkers.
Because of rbhelimitations in the support-confidence framework, various
objective measures have been used to evaluate the quality of association pat-
terns. Below, we provide a brief description of these measures and explain
some of their strengths and limitations.

Interest Factor The tea-coffee example shows that high-confidence rules

can sometimes be misleading becausethe confidencemeasureignores the sup-
port of the itemset appearing in the rule consequent. One way to addressthis
problem is by applying a metric known as lift:

(6.4)

which computes the ratio between the rule's confidence and the support of
the itemset in the rule consequent. For binary variables, Iift is equivalent to
another objective measure called interest factor, which is defined as follows:

I(4, B) :
s ( A ,B ) _ N"frr (6.5)
s(,4)x s(B) f+f +t
Interest factor compares the frequency of a pattern against a baseline fre-
quency computed under the statistical independence assumption. The baseline
frequency for a pair of mutually independent variables is

fn :1fr
f+ h+f+t
t?,f+t ir
or equivalently,"fn:t:;:: (6.6)
f
 Chapter Association Analysis

Table
6.9.Contingency forthewordpairs
tables ({p,q}and{r,s}.
p p r r
q 880 bt, 930 s 20 50 70
q 50 20 70 5 CU 880 930
930 70 1000 70 930 r000

This equation follows from the standard approach of using simple fractions
as estimates for probabilities. The fraction fnlN is an estimate for the joint
probability P(A,B), while fia/,n/ and fyf N are the estimates for P(A) and
P(B), respectively.lt A and B are statistically independent,then P(A,B):
P(A) x P(B), thus leading to the formula shown in Equation 6.6. Using
Equations 6.5 and 6.6, we can interpret the measure as follows:

1, if ,4 and B arc independent;

I(A, B) 1, if A and B are positively correlated; (6.7)
{. l, if A and B are negatively correlated.

For the tea-coffeeexample shown in Table 6.8, 1: O.H3_8- :0.9375, thus sug-
gesting a slight negative correlation between tea drinkers and coffee drinkers.

Limitations of Interest Factor We illustrate the limitation of interest

factor with an example from the text mining domain. In the text domain, it
is reasonableto assumethat the association between a pair of words depends
on the number of documents that contain both words. For example, because
of their stronger association,we expect the words data and mining to appear
together more frequently than the words compiler and mining in a collection
of computer sciencearticles.
Table 6.9 shows the frequency of occurrencesbetween two pairs of words,
{p,q} and {","}. Using the formula given in Equation 6.5, the interest factor
for {p,q} is 1.02 and for {r, s} is 4.08. These results are somewhat troubling
for the following reasons. Although p and q appear together in 88% of the
documents, their interest factor is close to 1, which is the value when p and q
are statistically independent. On the other hand, the interest factor for {r, s}
is higher than {p, q} even though r and s seldom appear together in the same
document. Confidence is perhaps the better choice in this situation becauseit
considersthe association between p and q (9a.6%) to be much stronger than
that between r and s (28.6%).
 6.7 Evaluation of Association Patterns 375

Correlation Analysis Correlation analysis is a statistical-based technique

for analyzing relationships between a pair of variables. For continuous vari-
ables, correl-ationis defined using Pearson's correlation coefficient (see Equa-
tion 2.10 on page 77). For binary variables, correlation can be measuredusing
the d-coefficient. which is defined as

fnfoo- fotfn (6.8)

The value of correlation ranges from -1 (perfect negative correlation) to *1

(perfect positive correlation). If the variables are statistically independent,
then @: 0. For example, the correlation between the tea and coffee drinkers
given in Table 6.8 is -0.0625.

Limitations of Correlation Analysis The drawback of using correlation

can be seen from the word association example given in Table 6.9. Although
the words p and g appear together more often than r and s, their /-coefficients
are identical, i.e., Q(p,q): Q(r,s) :0.232. This is becausethe @-coefficient
gives equal importance to both co-presenceand co-absenceof items in a trans-
action. It is therefore more suitable for analyzing symmetric binary variables.
Another limitation of this measure is that it does not remain invariant when
there are proportional changesto the sample size. This issue will be discussed
in greater detail when we describethe properties of objective measureson page
377.

IS Measure .I^9is an alternative measure that has been proposed for han-
dling asymmetric binary variables. The measure is defined as follows:

rs(A,B) : (6.e)

Note that .LS is large when the interest factor and support of the pattern
are large. For example, the value of 1^9for the word pairs {p, q} and {r, s}
shown in Table 6.9 are 0.946 and 0.286, respectively. Contrary to the results
given by interest factor and the @-coefficient, the 15 measure suggests that
the association between {p, q} i. stronger than {r, s}, which agreeswith what
we expect from word associationsin documents.
It is possible to show that 15 is mathematically equivalent to the cosine
measurefor binary variables (seeEquation2.T on page 75). In this regard, we
376 Chapter 6 Association Analysis

Table6.10.Example tableforitemsp andq.

of a contingency

q q

p 800 100 900

p 100 0 100
900 100 1000

consider A and B as a pair of bit vectors, A o B : s(A,B) the dot product

between the vectors, and lAl : \f(A) the magnitude of vector A. Therefore:

I-S\ (- Ar,-s' : - $ : ,
-a l' : cosi,ne(A,B). (6.10)
J4A) AB) l a l x "I*n,l
The 1,S measurecan also be expressedas the geometric mean between the
confidenceof association rules extracted from a pair of binary variables:

IS(4, B) : c(A-- B) x c(B -- A). (6.11)

Because the geometric mean between any two numbers is always closer to the
smaller number, the 1^9value of an itemset {p, q} is low whenever one of its
rules, p ---+ Q or e + p, has low confidence.

Limitations of rs Measure The 1,9 value for a pair of independent item-

sets, A and B. is

s ( A ,B ) s(A) x s(B)
B) :
IS1"a"o(,4, s(.4)x
s(A) x s(B)

since the value depends on s(A) and s(B), IS shares a similar problem as
the confidence measure-that the value of the measure can be quite large,
even for uncorrelated and negatively correlated patterns. For example, despite
the large 1^9 value between items p and q given in Table 6.10 (0.889), it is
still less than the expected value when the items are statistically independent
(ISi'auo : 0.9).
 6.7 Evaluation of Association Patterns 377

Alternative Objective Interestingness Measures

Besides the measureswe have described so far, there are other alternative mea-
sures proposed for analyzing relationships between pairs of binary variables.
These measurescan be divided into two categories, symmetric and asym-
metric measures. A measureM is symmetric if M(A --'- B): M(B ------A).
For example, interest factor is a symmetric measure becauseits value is iden-
tical for the rules A ------+
B and B ---- A. In contrast, confidence is an
asymmetric measure since the confidence for A ---. B and B -'----+,4. may not
be the same. Symmetric measuresare generally used for evaluating itemsets,
while asymmetric measures are more suitable for analyzing association rules.
Tables 6.11 and 6.12 provide the definitions for some of these measures in
terms of the frequency counts of a 2 x 2 contingency table.

Consistency among Objective Measures

Given the wide variety of measures available, it is reasonable to question

whether the measures can produce similar ordering results when applied to
a set of association patterns. If the measures are consistent, then we can
choose any one of them as our evaluation metric. Otherwise, it is important
to understand what their differences are in order to determine which measure
is more suitable for analyzing certain types of patterns.

Tabfe
6.11.Examples objective
ofsymmetric fortheitemset
measures {A, B}.
Measure(Symbol) Definition
N"frr -"fr+f+r
Correlation (@) t;_-i--
v Jr+J+1J0+J+o
/ . ? \ l l o a \
Odds ratio (a) \IttIoo) / (,1'to"rot/
N"frr +Nloq lrr.ffl:/ot_lt_q
--Nt=1+
Kappa (rc)
f-+r=j;+j+"
/^rr \ llr r \
Interest (1) \1\ Jtl)/ U1+J+1/
/ r \ l/ /-.-.a'-- r-\
Cosine (1S) \Irt) / \t/ It+J+r)
Piatetsky-Shapiro (P,9) .flt - --T-
JL+J+1
M
Collective strength (S)

Jaccard (o fu f( fr + + f+t
All-confidence (h) -i"[#,#]
378 Chapter 6 Association Analysis

Tabfe6.12.Examples
ofasymmetric
objective fortheruleA ------+
measures B.
Measure (Symbol) Definition

Goodman-Kruskal (.\) (Di *** f 1x - rnarxf +*)/(N - max;,fup)

Mutual Information (M) (D,;D,# to*#tl t t- D,+ t"g#)
J-Measure (J) #t"g#+ffrog#
Gini index (G) # * (#)'+(#)'l- (+)'
+#'t(#)'+(#)'l-(#)'
Laplace (.L) ( / " + t )1 f f p+ z )
Conviction (V) (fr+f +o)l(w f,o)
Certainty factor (F) (# - #)t0-+)
Added Value (AV) J71 _ J+1

Table
6.13.Example
ofcontingency
tables.
Example . t1 7 .t10 J01 foo
E1 8 123 83 424 1370
E2 8330 2 622 r046
Es 3954 3080 5 296r
Ea 2886 1363 1320 4431
E5 1500 2000 500 6000
E6 4000 2000 1000 3000
R. 9481 298 \27 94
Es 4000 2000 2000 2000
Es 7450 2483 4 63
Erc 61 2483 /l 7452

Supposethe symmetric and asymmetric measuresare applied to rank the

ten contingency tables shown in Table 6.13. These contingency tables are cho-
sen to illustrate the differences among the existing measures. The ordering
produced by these measuresare shown in Tables 6.14 and 6.15, respectively
(with 1 as the most interesting and 10 as the least interesting table). Although
someof the measuresappear to be consistent with each other, there are certain
measures that produce quite different ordering results. For example, the rank-
ings given by the fcoefficient agree with those provided by rc and collective
strength, but are somewhat different than the rankings produced by interest
 6.7 Evaluation of Association Patterns 379

Table6.14.Rankings tablesusingthesymmetric
of contingency giveninTable
measures 6.11.
o a K I IS PS ^9 C h
n1 1 3 I tt 2 2 1 2 2
r 2 3 3
E2 2 1 2 7 3
E3 3 2 A ^ 5 1 3 o 8
F7
Ea i
8 3 3 I 3 4 7 5
n q q
E5 5 I 0 2 6 6 9
E6 6 9 5 5 o 4 E
J l) 7
,7 ,7
E7 6. I 1 8 7 1 1
F7
E8 8 10 8 8 8 8 8 1
I

q
Es 9 4 I 10 4 I 4 4
Erc 10 5 10 1 10 10 10 10 10

Table6.15.Rankings tablesusingtheasymmetric
ofcontingency giveninTable
measures 6.12.
) M J G L V F AV
r
E1 I 1
r 1 1 4 2 2 J

tr
E2 2 2 2 .) J 1 I 6
tr r 4
E3 3 2 2 6 o
q I
Ea 4 o 3 4 3 3
q F7
4 8 tr
2
E5 6 D d
r F7 n
3
E6 3 8 6 4
.7 q
E7 7 5 9 8 3 7
,7
Eg 8 9 7 10 8 8 7
'| q
Es 6 4 10 I I 10
Eto 10 10 8 10 h 1 0 10 8

factor and odds ratio. Furthermore, a contingency table such as -E1sis ranked
lowest according to the @-coefficient,but highest according to interest factor.

Properties of Objective Measures

The results shown in Table 6.14 suggestthat a significant number of the mea-
sures provide conflicting information about the quality of a pattern. To under-
stand their differences,we need to examine the properties of these measures.

Inversion Property Consider the bit vectors shown in Figure 6.28. The
0/1 bit in each column vector indicates whether a transaction (row) contains
a particular item (column). For example, the vector A indicates that item a
380 Chapter 6 Association Analysis

A B CD EF

( ) (b) (c)
Figure6,28. Effectof theinversionoperation.
ThevectorsC andE areinversions
of vectorA, while
thevectorD is aninversion of vectors
B andF.

belongs to the first and last transactions, whereas the vector B indicates that
item b is contained only in the fifth transaction. The vectors C and E are in
fact related to the vector A-their bits have been inverted from 0's (absence)
to l's (presence),and vice versa. Similarly, D is related to vectors B and F by
inverting their bits. The process of flipping a bit vector is called inversion.
If a measure is invariant under the inversion operation, then its value for the
vector pair (C, D) should be identical to its value for (A, B). The inversion
property of a measure can be tested as follows.

Definition 6.6 (Inversion Property). An objective measureM is invariant

under the inversion operation if its value remains the same when exchanging
the frequency counts fi1 with /66 and fis with /61.
Among the measuresthat remain invariant under this operation include
the @-coefficient,odds ratio, n, and collective strength. These measuresmay
not be suitable for analyzing asymmetric binary data. For example, the /-
coefficient between C and D is identical to the @-coefficientbetween A and
B, even though items c and d appear together more frequently than a and b.
Fbrthermore, the d-coefficient between C and D is less than that between E
and F even though items e and / appear together only once! We had previously
raised this issue when discussing the limitations of the fcoefficient on page
375. For asymmetric binary data, measuresthat do not remain invariant under
the inversion operation are preferred. Some of the non-invariant measures
include interest factor, IS, PS, and the Jaccard coefficient.
 6,7 Evaluation of Association Patterns 381

NuIl Addition Property Suppose we are interested in analyzing the re-

lationship between a pair of words, such as data and rnining, in a set of
documents. If a collection of articles about ice fishing is added to the data set,
should the association between data and mining be affected? This processof
adding unrelated data (in this case, documents) to a given data set is known
as the null addition operation.
Deffnition 6.7 (Null Addition Property). An objective measure M is
invariant under the null addition operation if it is not affected by increasing
/es, while all other frequenciesin the contingency table stay the same.
For applications such as document analysis or market basket analysis, the
measure is expected to remain invariant under the null addition operation.
Otherwise, the relationship between words may disappear simply by adding
enough documents that do not contain both words! Examples of measures
that satisfy this property include cosine (19) and Jaccard ({) measures,while
those that viqlate this property include interest factor, PS, odds ratio, and
the fcoefficient.

Scaling Property Table 6.16 shows the contingency tables for gender and
the grades achieved by students enrolled in a particular course in 1993 and
2004. The data in these tables showed that the number of male students has
doubled since 1993, while the number of female students has increased by a
factor of 3. However, the male students in 2004 are not performing any better
than those in 1993 because the ratio of male students who achieve a high
grade to those who achieve a low grade is still the same, i.e., 3:4. Similarly,
the female students in 2004 are performing no better than those in 1993. The
associationbetween grade and gender is expected to remain unchangeddespite
changesin the sampling distribution.

6.16.Thegrade-gender
Table example.
Male Female
High High 60 60 720
Low Low 80 30 t1
740 90 230

(a) Sample data from 1993. (b) Sample data from 2004.
382 Chapter 6 Association Anal.ysis

Table
6.17.Properties
ofsymmetric
measures.
Svmbol Measure Inversion Null Addition Scaling
0 @-coefficient Yes No L\O
a odds ratio Yes No Yes
K Cohen's Yes No No
r Interest No No No
IS Cosine No Yes No
PS Piatetsky-Shapiro's Yes No No
,9 Collective strength Yes No No
e Jaccard No Yes No
h All-confidence No No No
Support No No No

Definition 6.8 (Scaling Invariance Property). An objective measure M

is invariant under the row/column scaling operation if Mg) : M(T'), where
7 is a contingency table with frequency counts lfn; frc; ,for; ,foo], Tt is a
contingency table with scaled frequency counts [k*sfn; kzksfn; kft+fof
kzk+foo), and k1, kz, ks, k4 are positive constants.

From Table 6.17, notice that only the odds ratio (a) is invariant under
the row and column scaling operations. All other measures such as the f
coefficient, n, IS, interest factor, and collective strength (,9) change their val-
ues when the rows and columns of the contingency table are rescaled. Although
we do not discussthe properties of asymmetric measures(such as confidence,
J-measure, Gini index, and conviction), it is clear that such measuresdo not
preservetheir values under inversion and row/column scaling operations, but
are invariant under the null addition oneration.

6.7.2 Measures beyond Pairs of Binary Variables

The measuresshown in Tables 6.11 and 6.72 aredefined for pairs of binary vari-
ables (e.g.,2-itemsets or association rules). However, many of them, such as
support and all-confidence,are also applicable to larger-sizeditemsets. Other
measures,such as interest factor, IS, PS, and Jaccard coefficient, can be ex-
tended to more than two variables using the frequency tables tabulated in a
multidimensional contingency table. An example of a three-dimensional con-
tingency table for a, b, and c is shown in Table 6.18. Each entry fiip in this
table representsthe number of transactions that contain a particular combi-
nation of items a, b, and c. For example, frct is the number of transactions
that contain a and c, but not b. On the other hand, a marginal frequency
 6.7 Evaluation of Association Patterns 383

Table6.18.Example
of a three-dimensional table.
contingency

such as ,fi+r is the number of transactions that contain a and c, irrespective

of whether b is present in the transaction.
Given a k-itemset {h,iz, . . . ,in}, the condition for statistical independence
can be stated as follows:

t. -
f o . r + . . . +fx+ b . . . +x ' . . x f + + . . . t o (6.12)
J L\t2...Itc
AIk-1

With this definition, we can extend objective measures such as interest factor
and P,S, which are based on deviations from statistical independence'to more
than two variables:

Iy'ft-1 x ftrb...tr
T_
f , . r + . . . +fx+ b . . . +x . . . x f + + . . t u

ps : I+-- lvrlt

Another approach is to define the objective measure as the maximum, min-

imum, or average value for the associations between pairs of items in a pat-
tern. For example,given a k-itemsetX: {h,i2,...,ip},we may definethe
/-coefficient for X as the average @-coefficient between every pair of items
(io,i) in X. However, becausethe measure considers only pairwise associa-
tions, it may not capture all the underlying relationships within a pattern.
Analysis of multidimensional contingency tables is more complicated be-
cause of the presenceof partial associationsin the data. For example, some
associationsmay appear or disappear when conditioned upon the value of cer-
tain variables. This problem is known as Simpson's paradox and is described
in the next section. More sophisticated statistical techniques are available to
analyze such relationships, e.g., loglinear models, but these techniques are
beyond the scope of this book.
384 Chapter 6 AssociationAnalysis

Table
6.19.A two-way
contingency
tablebetween
thesaleofhigh-definition
television
andexercise
machine.
Buy Buv Exercise Machine
HDTV Yes No
YES 99 81 180
No o4 66 t20
153 L47 300

Table6,20.Example
ofa three-way
contingency
table.
Customer lJuy Buy Exercise Machine Total
Group HDTV YCS No
College Students Yes I 9 10
No A 30 34
Working Adult YES 98 72 770
No 50 36 86

6.7.3 Simpson's Paradox

It is important to exercisecaution when interpreting the association between
variables becausethe observedrelationship may be influenced by the presence
of other confounding factors, i.e., hidden variables that are not included in
the analysis. In some cases, the hidden variables may cause the observed
relationship between a pair of variables to disappear or reverseits direction, a
phenomenon that is known as Simpson's paradox. We illustrate the nature of
this paradox with the following example.
Consider the relationship between the sale of high-definition television
(HDTV) and exercisemachine, as shown in Table 6.19. The rule {HDTV:Yes}
------+
{Exercise machine:Yes} has a confidence of 99/180:557o and the rule
-_.-+
{HDTV:No} {Exercise machine:Yes} has a confidenceof 541720: 45%o.
Together, these rules suggest that customers who buy high-definition televi-
sions are more likely to buy exercise machines than'those who do not buy
high-definition televisions.
However, a deeper analysis reveals that the sales of these items depend
on whether the customer is a college student or a working adult. Table 6.20
summarizesthe relationship between the sale of HDTVs and exercisemachines
among college students and working adults. Notice that the support counts
given in the table for college students and working adults sum up to the fre-
quencies shown in Table 6.19. Furthermore, there are more working adults
 6.7 Evaluation of Association Patterns 385

than college students who buy these items. For college students:

---' Ill0:10To,
c({UOfV:Yes} {Exercise machine:Yes})
-----* 4134: 7I.8To,
c({HOtV:No} {Exercise machine:Yes})

while for working adults:

-----* : 981170: 57.7Vo,

c({HnfV:Yes} {Exercise machine:Yes})
----- : 50/86 :58.IVo.
c({UOfV:No} {Exercise machine:Yes})

The rules suggest that, for each group, customers who do not buy high-
definition televisions are more likely to buy exercise machines, which contradict
the previous conclusion when data from the two customer groups are pooled
together. Even if alternative measures such as correlation, odds ratio' or
interest are applied, we still find that the sale of HDTV and exercise machine
is positively correlated in the combined data but is negatively correlated in
the stratified data (seeExercise 20 on page 414). The reversal in the direction
of association is known as Simpson's paradox.
The paradox can be explained in the following way. Notice that most
customers who buy HDTVs are working adults. Working adults are also the
largest group of customers who buy exercise machines. Because nearly 85% of
the customers are working adults, the observed relationship between HDTV
and exercise machine turns out to be stronger in the combined data than
what it would have been if the data is stratified. This can also be illustrated
mathematically as follows. Suppose

o,lb < cf d and plq < rls,

where afb andplqmay represent the confidence of the rule A ---, B in two
different strata, while cld and rf s may represent the confidence of the rule
A ------+
B in the two strata. When the data is pooled together' the confidence
valuesof the rules in the combineddata are (a+dl@+q) and (c+r)l@+s),
respectively. Simpson's paradox occurs when

a*P clr
b+q> d+r'

thus leading to the wrong conclusion about the relationship between the vari-
ables. The lesson here is that proper stratification is needed to avoid generat-
ing spurious patterns resulting from Simpson's paradox. For example, market
386 Chapter 6 Association Analysis

1000 1500
Itemssortedby support

Figure6.29.Support
distribution
of itemsinthecensus
dataset.

basket data from a major supermarket chain should be stratified according to

store locations, while medical records from various patients should be stratified
according to confounding factors such as age and gender.

6.8 Effect of Skewed Support Distribution

The performances of many association analysis algorithms are influenced by
properties of their input data. For example, the computational complexity of
the Apri,ori algorithm depends on properties such as the number of items in
the data and averagetransaction width. This section examinesanother impor-
tant property that has significant influence on the performance of association
analysis algorithms as well as the quality of extracted patterns. More specifi-
cally, we focus on data sets with skewed support distributions, where most of
the items have relatively low to moderate frequencies,but a small number of
them have very high frequencies.
An example of a real data set that exhibits such a distribution is shown in
Figure 6.29. The data, taken from the PUMS (Public Use Microdata Sample)
census data, contains 49;046 records and 2113 asymmetric binary variables.
We shall treat the asymmetric binary variables as items and records as trans-
actions in the remainder of this section. While more than 80% of the items
have support lessthan 1%, a handfuI of them have support greater than 90%.
 6.8 Effect of Skewed Support Distribution 387

Table6.21.Grouping
theitemsinthecensus values.
datasetbasedontheirsupport
Group G1 G2 Gs
Support <t% r % - 9 0 % > gUYa
1-Dr
Number of Items I'dU 358 20

To illustrate the effect of skewed support distribution on frequent itemset min-

ing, we divide the items into three groups, Gt, Gz, and G3, according to their
support levels. The number of items that belong to each group is shown in
Table 6.21.
Choosing the right support threshold for mining this data set can be quite
tricky. If we set the threshold too high (e.g., 20%), then we may miss many
interesting patterns involving the low support items from Gr. In market bas-
ket analysis, such low support items may correspond to expensive products
(such as jewelry) that are seldom bought by customers, but whose patterns
are still interesting to retailers. Conversely, when the threshold is set too
low, it becomesdifficult to find the association patterns due to the following
reasons. First, the computational and memory requirements of existing asso-
ciation analysis algorithms increaseconsiderably with low support thresholds.
Second,the number of extracted patterns also increasessubstantially with low
support thresholds. Third, we may extract many spurious patterns that relate
a high-frequency item such as milk to a low-frequency item such as caviar.
Such patterns, which are called cross-support patterns, are likely to be spu-
rious because their correlations tend to be weak. For example, at a support
threshold equal to 0.05yo,there are 18,847frequent pairs involving items from
Gr and G3. Out of these, 93% of them are cross-supportpatterns; i.e., the pat-
terns contain items from both Gr and G3. The maximum correlation obtained
from the cross-support patterns is 0.029, which is much lower than the max-
imum correlation obtained from frequent patterns involving items from the
same group (which is as high as 1.0). Similar statement can be made about
many other interestingnessmeasuresdiscussedin the previous section. This
example shows that a large number of weakly correlated cross-support pat-
terns can be generated when the support threshold is sufficiently low. Before
presenting a methodology for eliminating such patterns, we formally define the
concept of cross-support patterns.
388 Chapter 6 Association Analysis

Definition 6.9 (Cross-Support Pattern). A cross-support pattern is an

itemset X : {ir,,i2,. . . ,i6} whose support ratio

(6.13)

is less than a user-specifiedthreshold h".

Example 6.4. Suppose the support for milk is 70To,while the support for
sugar is 10% and caviar is 0.04%. Given h" : 0.01, the frequent itemset
{milk, sugar) caviar} is a cross-support pattern becauseits support ratio is

m i n [ 0 . 7 , 0 . 1 , 0 . 0 0 0 4 ]0.0004
: 0.00058
< 0.01.
m a x 1 0 . 7 , 0 . 1 , 0 . 0 0 0 4 10 . 7
I

Existing measures such as support and confidence may not be suffi.cient

to eliminate cross-support patterns, as illustrated by the data set shown in
Figure 6.30. Assumingthat h.:0.3, the itemsets{p,q}, {p,r}, and {p,q,r}
are cross-support patterns because their support ratios, which are equal to
0.2, are less than the threshold h". Although we can apply a high support
threshold, say, 20Vo,to eliminate the cross-support patterns, this may come
at the expense of discarding other interesting patterns such as the strongly
correlated itemset, {q, r} that has support equal to L6.7To.
Confidence pruning also does not help becausethe confidenceof the rules
extracted from cross-support patterns can be very high. For example, the
confidence for {q} - {p} is 80% even though {p,S} is a cross-support pat-
tern. The fact that the cross-support pattern can produce a high-confidence
rule should not come as a surprise becauseone of its items (p) appears very
frequently in the data. Therefore, p is expected to appear in many of the
transactions that contain q. Meanwhile, the rule {q} - {r} also has high
confidence even though {q,r} is not a cross-support pattern. This example
demonstrates the difficulty of using the confidencemeasure to distinguish be-
tween rules extracted from cross-support and non-cross-supportpatterns.
Returning to the previous example, notice that the rule {p} -----+{q} has
very low confi.dencebecause most of the transactions that contain p do not
contain q. In contrast, the rule {r} - {q}, which is derived from the pattern
{q,r}, has very high confidence. This observation suggeststhat cross-support
patterns can be detected by examining the lowest confidencerule that can be
extracted from a given itemset. The proof of this statement can be understood
as follows.
 6.8 Effect of Skewed Support Distribution 389

Figure6.30.A transaction
datasetcontaining p, q,?fidr, where
threeitems, p isa highsupport
item
andq andr arelowsupportitems.

1. Recall the following anti-monotone property of confidence:

conf({ij4} - { i 2 , , i 4 , . . ., i n } ) < c o n f ( { i , j , 2 i s }- {in,is,...,i*}).

This property suggests that confi.dencenever increases as we shift more

items from the left- to the right-hand side of an associationrule. Because
of this property, the lowest confidence rule extracted from a frequent
itemset contains only one item on its left-hand side. We denote the set
of all rules with only one item on its left-hand side as -Rr.

2. Given a frequent itemset {it,,ir,.. . ,in}, the rule

{ii} - { i r , i , r , .. . , i i - r , i i + r , . . . , i n }

has the lowest confidencein l?1 if s(i): max l"(ir), s(iz),...,s(ir)].

This follows directly from the definition of confidence as the ratio be,
tween the rule's support and the support of the rule antecedent.
390 Chapter 6 Association Analysis

3. Summarizing the previous points, the lowest confidenceattainable from

a f r e q u e n ti t e m s e t{ h , i 2 , . . . , 2 6 } i s

s({i,1,i.2,...,i,n))
'
m a x [ s ( i 1 ) ,s ( i z ) , . . . , s ( i r ) ]

This expression is also known as the h-confidence or all-confidence

measure. Becauseof the anti-monotone property of support, the numer-
ator of the h-confidence measure is bounded by the minimum support
of any item that appears in the frequent itemset. In other words, the
h-confidenceof an itemset X : {h,,i2,.. . ,i7r} must not exceedthe fol-
lowing expression:

ty),'' 't!':ll
< gt" l,-t!'.tl'
h-confidence(x)
m a x i s ( e 1) , s \ z z ) , .. . , s \ L k ) )

Note the equivalencebetween the upper bound of h-confidenceand the

support ratio (r) given in Equation 6.13. Becausethe support ratio for
a cross-support pattern is always less than h", the h-confidence of the
pattern is also guaranteed to be less than h..

Therefore, cross-support patterns can be eliminated by ensuring that the

h-confidence values for the patterns exceed h". As a final note, it is worth
mentioning that the advantages of using h-confidence go beyond eliminating
cross-support patterns. The measure is also anti-monotone, i.e.,

e({i,1,i2,. . ., i,n}) ) h-confidence({21,

h-confidenc i2, . . ., in+r}),

and thus can be incorporated directly into the mining algorithm. Furthermore,
h-confidenceensuresthat the items contained in an itemset are strongly asso-
ciated with each other. For example, suppose the h-confidenceof an itemset
X is 80%. If one of the items in X is present in a transaction, there is at least
an 80% chance that the rest of the items in X also belong to the same trans-
action. Such strongly associatedpatterns are called hyperclique patterns.

6.9 Bibliographic Notes

The association rule mining task was first introduced by Agrawal et al. in
1228,2291to discover interesting relationships among items in market basket
 6.9 Bibliographic Notes 391

transactions. Since its inception, extensive studies have been conducted to

address the various conceptual, implementation, and application issues per-
taining to the association analysis task. A summary of the various research
activities in this area is shown in Figure 6.31.

Conceptual Issues

Researchin conceptual issuesis focused primarily on (1) developing a frame-

work to describe the theoretical underpinnings of association analysis, (2) ex-
tending the formulation to handle new types of patterns, and (3) extending the
formulation to incorporate attribute types beyond asymmetric binary data.
Following the pioneering work by Agrawal et al., there has been a vast
amount of researchon developinga theory for the associationanalysisproblem.
In 1254],Gunopoulos et al. showed a relation between the problem of finding
maximal frequent itemsets and the hypergraph transversal problem. An upper
bound on the complexity of associationanalysis task was also derived. Zaki et
al. [334, 336] and Pasquier et al. [294] have applied formal concept analysis to
study the frequent itemset generation problem. The work by Zaki et al. have
subsequentlyled them to introduce the notion of closedfrequent itemsets 1336].
Friedman et al. have studied the association analysis problem in the context
of bump hunting in multidimensional space [252]. More specifically, they
consider frequent itemset generation as the task of finding high probability
density regions in multidimensional space.
Over the years, new types of patterns have been defined, such as profile
association rules [225], cyclic association rules [290], finzy association rules
[273], exception rules [316], negative association rules 1238, 304], weighted
association rules [240, 300], dependencerules [308], peculiar rules[340], inter-
transaction association rules [250, 323], and partial classification rules 1231,
285]. Other types of patterns include closed itemsets [294, 336], maximal
itemsets 1234],hyperclique patterns 1330],support envelopes [314], emerging
patterns [246], and contrast sets [233]. Association analysis has also been
successfully applied to sequential 1230,312], spatial [266], and graph-based
1268,274,293, 331, 335] data. The concept of cross-support pattern was first
introduced by Hui et al. in [330]. An efficient algorithm (called Hyperclique
Miner) that automatically eliminates cross-supportpatterns was also proposed
by the authors.
Substantial researchhas been conducted to extend the original association
rule formulation to nominal [311], ordinal [281], interval [2S+],and ratio [253,
255, 31L, 325, 339] attributes. One of the key issuesis how to define the support
measure for these attributes. A methodology was proposed by Steinbach et
392 Chapter 6 Association Analysis

al. [315] to extend the traditional notion of support to more general patterns
and attribute types.
 6.9 Bibliographic Notes 393

o
(t)
(tt
c
(u
o
(E
C)
o
U'
.Jt
(E
ct) E
'c= o o
U)
9"0 o< <D

=g
'-G
c -v= x
Yd

.=
(J
C'

' = K E e -. g=:
f/'tL .c
o- id
C)
CE
.:..;,+-FF< (D
6.9, =v xA
,*F st-+A"x^
E = =? aJ>
<D
-(E i?f?i?-iY
.l)

EB o
::
FE
6< oo
(5

o
o -c
E -o
o
oo
(o
E
E
5
C"

-a?
(o
o
5
.9
IJ-

,9.9 a

S :EB :
gE€
'EecEgE
394 Chapter 6 Association Analysis

Implementation Issues

Researchactivities in this area revolve around (1) integrating the mining ca-
pability into existing databasetechnology, (2) developing efficient and scalable
mining algorithms, (3) handling user-specifiedor domain-specific constraints,
and ( ) post-processingthe extracted patterns.
There are several advantages to integrating association analysis into ex-
isting database technology. First, it can make use of the indexing and query
processingcapabilities of the database system. Second,it can also exploit the
DBMS support for scalability, check-pointing, and parallelization [301]. The
SETM algorithm developed by Houtsma et al. [265] was one of the earliest
algorithms to support association rule discovery via SQL queries. Since then,
numerous methods have been developedto provide capabilities for mining as-
sociation rules in databasesystems. For example, the DMQL [258]and M-SQL
[267] query languagesextend the basic SQL with new operators for mining as-
sociation rules. The Mine Rule operator [283] is an expressiveSQL operator
that can handle both clustered attributes and item hierarchies. Tsur et al.
[322] developed a generate-and-testapproach called query flocks for mining
associationrules. A distributed OLAP-based infrastructure was developed by
Chen et al. l2aL] for mining multilevel association rules.
Dunkel and Soparkar l2a8] investigated the time and storage complexity
of the Apri,ori algorithm. The FP-growth algorithm was developedby Han et
al. in [259]. Other algorithms for mining frequent itemsets include the DHP
(dynamic hashing and pruning) algorithm proposed by Park et al. [292] and
the Partition algorithm developed by Savasereet al [303]. A sampling-based
frequent itemset generation algorithm was proposed by Toivonen [320]. The
algorithm requires only a single pass over the data, but it can produce more
candidate itemsets than necessary. The Dynamic Itemset Counting (DIC)
algorithm [239] makes only 1.5 passesover the data and generatesless candi-
date itemsets than the sampling-based algorithm. Other notable algorithms
include the tree-projection algorithm [223] and H-Mine [295]. Survey articles
on frequent itemset generation algorithms can be found in 1226,262]. A repos-
itory of data sets and algorithms is available at the Frequent Itemset Mining
Implementations (FIMI) repository (http:llfirni.cs.helsinki.fi). Parallel algo-
rithms for mining association patterns have been developed by various authors
[224,256,287,306,337]. A survey of such algorithms can be found in [333].
Online and incremental versionsof associationrule mining algorithms had also
been proposed by Hidber [260] and Cheung et aL. 12421.
Srikant et al. [313] have consideredthe problem of mining associationrules
in the presenceof boolean constraints such as the following:
 6.9 Bibliographic Notes 395

(Cookies A Milk) V (descendents(Cookies)A -ancestors(Wheat Bread))

Given such a constraint, the algorithm looks for rules that contain both cook-
ies and milk, or rules that contain the descendent items of cookies but not
ancestor items of wheat bread. Singh et al. [310] and Ng et al. [288] had also
developed alternative techniques for constrained-based association rule min-
ing. Constraints can also be imposed on the support for different itemsets.
This problem was investigated by Wang et al. [324], Liu et al. in [279], and
Seno et al. [305].
One potential problem with association analysis is the large number of
patterns that can be generated by current algorithms. To overcomethis prob-
lem, methods to rank, summarize, and filter patterns have been developed.
Toivonen et al. [321] proposed the idea of eliminating redundant rules using
structural rule covers and to group the remaining rules using clustering.
Liu et al. [280]applied the statistical chi-squaretest to prune spurious patterns
and summarized the remaining patterns using a subset of the patterns called
direction setting rules. The use of objective measures to filter patterns
has been investigated by many authors, including Brin et al. [238], Bayardo
and Agrawal [235], Aggarwal and Yu 12271,and DuMouchel and Pregibon[2a7].
The properties for many of these measureswere analyzed by Piatetsky-Shapiro
[297], Kamber and Singhal [270], Hilderman and Hamilton [261], and Tan et
al. [318]. The grade-genderexample used to highlight the importance of the
row and column scaling invariance property was heavily influenced by the
discussion given in [286] by Mosteller. Meanwhile, the tea-coffeeexample il-
lustrating the limitation of confidence was motivated by an example given in
[238] by Brin et al. Becauseof the limitation of confidence,Brin et al. [238]
had proposed the idea of using interest factor as a measure of interesting-
ness. The all-confidence measure was proposed by Omiecinski [289]. Xiong
et al. [330] introduced the cross-support property and showed that the all-
confi.dencemeasure can be used to eliminate cross-support patterns. A key
difficulty in using alternative objective measuresbesidessupport is their lack
of a monotonicity property, which makes it difficult to incorporate the mea-
sures directly into the mining algorithms. Xiong et al. [328] have proposed
an efficient method for mining correlations by introducing an upper bound
function to the fcoefficient. Although the measure is non-monotone, it has
an upper bound expressign that can be exploited for the efficient mining of
strongly correlated itempairs.
Fabris and Fleitas [249] have proposed a method for discovering inter-
esting associationsby detecting the occurrencesof Simpson's paradox [309].
Megiddo and Srikant [282] described an approach for validating the extracted
396 Chapter 6 Association Analysis

patterns using hypothesis testing methods. A resampling-basedtechnique was

also developed to avoid generating spurious patterns becauseof the multiple
comparison problem. Bolton et al. [237]have applied the Benjamini-Hochberg
[236] and Bonferroni correction methods to adjust the p-values of discovered
patterns in market basket data. Alternative methods for handling the multiple
comparison problem were suggestedby Webb [326] and Zhang et al. [338].
Application of subjective measuresto association analysis has been inves-
tigated by many authors. Silberschatz and Tuzhilin [307] presented two prin-
ciples in which a rule can be consideredinteresting from a subjective point of
view. The concept of unexpected condition rules was introduced by Liu et al.
in 12771.Cooley et al. [243] analyzed the idea of combining soft belief sets
using the Dempster-Shafertheory and applied this approach to identify contra-
dictory and novel association patterns in Web data. Alternative approaches
include using Bayesian networks [269] and neighborhood-basedinformation
[2a5] to identify subjectively interesting patterns.
Visualization also helps the user to quickly grasp the underlying struc-
ture of the discovered patterns. Many commercial data mining tools display
the complete set of rules (which satisfy both support and confidence thresh-
old criteria) as a two-dimensional plot, with each axis corresponding to the
antecedent or consequentitemsets of the rule. Hofmann et al. [263] proposed
using Mosaic plots and Double Decker plots to visualize associationrules. This
approach can visualize not only a particular rule, but also the overall contin-
gency table between itemsets in the antecedent and consequent parts of the
rule. Nevertheless,this technique assumesthat the rule consequentconsistsof
only a single attribute.

Application fssues

Association analysis has been applied to a variety of application domains such

as Web mining 1296,3L71,document analysis 1264],telecommunication alarm
diagnosis [271],network intrusion detection 1232,244,275], and bioinformatics
1302, 3271. Applications of association and correlation pattern analysis to
Earth Sciencestudies have been investigated in [298, 299, 319].
Association patterns have also been applied to other learning problems
such as classification1276,278],regression[291],and clustering1257,329,332].
A comparison between classification and association rule mining was made
by Freitas in his position paper [251]. The use of association patterns for
clustering has been studied by many authors including Han et al.l257l, Kosters
Yang et al. [332] and Xiong et al. [329].
et al. 12721,
 Bibliography 397

Bibliography
[223] R. C. Agarwal, C. C. Aggarwal, and V. V. V. Prasad. A Ttee Projection Algorithm
for Generation of Flequent ltemsets. Journal of Parallel and Distri,buted Computing
(Speci,alIssue on Hi,gh Performance Data Mining),61(3):350-371, 2001.
12241 R. C. Agarwal and J. C. Shafer. Parallel Mining of Association Rules. IEEE Transac-
t'ions on Knowledge and, Data Eng'ineering,8(6):962-969, March 1998.
12251C. C. Aggarwal, Z. Sun, and P. S. Yu. Online Generation of Profile Association Rules.
In Proc. of the lth IntI. Conf. on Knowled,ge D'iscouerg and, Data Mining, pages 129-
133, New York, NY, August 1996.
[226] C. C. Aggarwal and P. S. Yu. Mining Large Itemsets for Association Rules. Dafa
Engineering B ullet'in, 2l (l) :23-31, March 1998.
12271 C. C. Aggarwal and P. S. Yu. Mining Associations with the Collective Strength
Approach. IEEE Tfans. on Knowled,ge and Data Eng,ineer'ing,13(6):863-873, Jan-
uary/February 2001.
[228] R. Agrawal, T. Imielinski, and A. Swami. Database mining: A performance perspec-
tive. IEEE Transactions on Knowledge and Data Eng'ineering, S:9L4 925, 1993.
1229] R. Agrawal, T. Imielinski, and A. Swami. Mining association rules between sets of
items in large databases. In Proc. ACM SIGMOD IntI. Conf. Management of Data,
pages 207-216, Washington, DC, 1993.
f230] R. Agrawal and R. Srikant. Mining Sequential Patterns. ln Proc. of Intl. Conf. on
Data Engineering, pages 3-14, Taipei, Taiwan, 1995.
1231] K. Ali, S. Manganaris, and R. Srikant. Partial Classification using Association Rules.
In Proc. of the ?rd Intl. Conf. on Knowledge Discouery and, Data M'ining, pages 115
118, Newport Beach, CA, August 1997.
12321 D. Barbarii, J. Couto, S. Jajodia, and N. Wu. ADAM: A Testbed for Exploring the
Use of Data Mining in Intrusion Detection. SIGMOD Record,,30(4):15 24,2001.
[233] S. D. Bay and M. Pazzani. Detecting Group Differences: Mining Contrast Sets. Dota
Min'ing and Know ledge Dis couery, 5 (3) :2L3-246, 200I.
[234] R. Bayardo. Efficiently Mining Long Patterns from Databases. In Proc. of 1998 ACM-
SIGMOD Intl. Conf. on Management of Data, pages 85-93, Seattle, WA, June 1998.
[235] R. Bayardo and R. Agrawal. Mining the Most Interesting Rules. In Proc. of the Sth
Intl. Conf. on Knowledge Discouerg and Data Min'ing, pages 145-153, San Diego, CA,
August 1999.
[236] Y. Benjamini and Y. Hochberg. Controlling the False Discovery Rate: A Practical
and Powerful Approach to Multiple Testing. Journal Rogal Statistical Society 8,57
(1):289-300, 1995.
1237] R. J. Bolton, D. J. Hand, and N. M. Adams. Determining Hit Rate in Pattern Search.
In Proc. of the ESF Etploratory Workshop on Pattern Detect'i,on and Discouery in
Data Mini,ng, pages 36-48, London, UK, September 2002.
[238] S. Brin, R. Motwani, and C. Silverstein. Beyond market baskets: Generalizing associ-
ation rules to correlations. In Proc. ACM SIGMOD IntI. Conf. Management of Data,
pages265-276, Tucson, AZ, 1997.
[239] S. Brin, R. Motwani, J. Ullman, and S. Tsur. Dynamic Itemset Counting and Impli-
cation Rules for market basket data. In Proc. of 1997 ACM-SIGMOD IntI. Conf. on
Management of Data, pages 255 264, T\rcson, AZ, J:lrre L997.
1240] C. H. Cai, A. tr\r, C. H. Cheng, and W. W. Kwong. Mining Association Rules with
Weighted Items. In Proc. of IEEE Intl. Database Eng'ineering and Appli,cations Sgmp.,
pages 68-77, Cardiff, Wales, 1998.
398 Chapter 6 Association Analysis

[241] Q. Chen, U. Dayal, and M. Hsu. A Distributed OLAP infrastructure for E-Commerce.
In Proc. of the lth IFCIS IntI. Conf. on Cooperatiue Information Systems, pages 209-
220, Edinburgh, Scotland, 1999.
12421 D. C. Cheung, S. D. Lee, and B. Kao. A General Incremental Technique for Maintaining
Discovered Association Rules. In Proc. of the Sth IntI. Conf. on Database Systems for
Aduanced Appl'ications, pages 185-194, Melbourne, Australia, 1997.
[243] R. Cooley, P. N. Tan, and J. Srivastava. Discovery of Interesting Usage Patterns
from Web Data. In M. Spiliopoulou and B. Masand, editors, Aduances in Web Usage
AnalEsis and User ProfiIing, volume 1836, pages 163-182. Lecture Notes in Computer
Science, 2000.
12441P. Dokas, L.Ertoz, V. Kumar, A. Lazarevic, J. Srivastava, and P. N. Tan. Data Mining
for Network Intrusion Detection. In Proc. NSF Workshop on Nert Generation Data
M'ini,ng, Baltimore, MD, 2002.
1245] G. Dong and J. Li. Interestingness of discovered association rules in terms of
neighborhood-based unexpectedness. In Proc. of the 2nd, Paci,fi,c-Asia Conf. on Knowl-
ed,geDiscouery and Data Min'i,ng, pages 72-86, Melbourne, Australia, April 1998.
[246] G. Dong and J. Li. Efficient Mining of Emerging Patterns: Discovering Tbends and
Differences. In Proc. of the 5th Intl. Conf. on Knowledge Discouery and Data M'ining,
pages 43-52, San Diego, CA, August 1999.
12471 W. DuMouchel and D. Pregibon. Empirical Bayes Screening for Multi-Item Associa-
tions. In Proc. of the 7th IntI. Conf. on Knowledge D'iscouerg and, Data Mining, pages
67-76, San Flancisco, CA, August 2001.
[248] B. Dunkel and N. Soparkar. Data Organization and Access for Efficient Data Mining.
In Proc. of the 15th Intl. Conf. on Data Engineering, pages 522-529, Sydney, Australia,
March 1999.
12491 C. C. Fabris and A. A. Fleitas. Discovering surprising patterns by detecting occurrences
of Simpson's paradox. In Proc. of the 19th SGES Intl. Conf. on Knowledge-Based,
Systems and"Applied Artificial Intelligence), pages 1,48-160, Cambridge, UK, December
1999.
[250] L. Feng, H. J. Lu, J. X. Yu, and J. Han. Mining inter-transaction associations with
templates. In Proc, of the 8th IntI. Conf. on Inforrnation and Knowled,ge Managemept,
pages 225-233, Kansas City Missouri, Nov 1999.
[251] A. A. Freitas. Understanding the crucial differences between classification and discov-
ery of association rules a position paper. SIGKDD Erplorations,2(l):6549, 2000.
12521 J. H. Friedman and N. L Fisher. Bump hunting in high-dimensional data. Statisti.cs
aniL Computing, 9(2):123-143, April 1999.
[253] T. Fukuda, Y. Morimoto, S. Morishita, and T. Tokuyama. Mining Optimized Asso-
ciation Rules for Numeric Attributes. In Proc. of the 15th Symp. on Principles of
Database Systems, pages 182 191, Montreal, Canada, June 1996.
12541D. Gunopulos, R. Khardon, H. Mannila, and H. Toivonen. Data Mining, Hypergraph
TYansversals, and Machine Learning. In Proc. of the 16th Sgmp. on Princi,ples of
Database Sgstems, pages 209-216, T\rcson, AZ,May 1997.
[255] E.-H. Han, G. Karypis, and V. Kumar. Min-Apriori: An Algorithm for Finding As-
sociation Rules in Data with Continuous Attributes. http://www.cs.umn.edu/-han,
L997.
[256] E.-H. Han, G. Karypis, and V. Kumar. Scalable Parallel Data Mining for Association
Rules. In Proc. of 1997 ACM-SIGMOD IntI. Conf. on Management of Data, pages
277-288. T\rcson. AZ.Mav 1997.
 Bibliography 399

12571E.-IJ. Han, G. Karypis, V. Kumar, and B. Mobasher. Clustering Based on Association

Rule Hypergraphs. In Proc. of the 1997 ACM SIGMOD Workshop on Research Issues
in Data Mi,ning and, Knowledge D'iscoaery,Tucson, AZ, 1997.
[258] J. Han, Y. Fu, K. Koperski, W. Wang, and O. R. Zaiane. DMQL: A data mining query
language for relational databases. In Proc. of the 1996 ACM SIGMOD Workshop on
Research Issues in Data Mi,ni,ng and Knowledge Discouery, Montreal, Canada, June
1996.
[259] J. Han, J. Pei, and Y. Yin. Mining Frequent Patterns without Candidate Generation.
In Proc. ACM-SIGMOD Int. Conf. on Management of Data (SIGMOD'00), pages
1-12, Dallas, TX, May 2000.
[260] C. Hidber. Online Association Rule Mining. ln Proc. of 1999 ACM-SIGMOD IntL
Conf. on Managernent of Data, pages 145-156, Philadelphia, PA, 1999.
[261] R. J. Hilderman and H. J. Hamilton. Knouledge Discouery and Measures of Interest.
Kluwer Academic Publishers, 2001
12621 J. Hipp, U. Guntzer, and G. Nakhaeizadeh. Algorithms for Association Rule Mining-
A General Survey. Si,gKDD Erplorations,2(1):58-64, June 2000.
[263] H. Hofmann, A. P. J. M. Siebes, and A. F. X. Wilhelm. Visualizing Association Rules
with Interactive Mosaic Plots. In Proc. of the 6th IntI. Conf. on Knowledge Discouerg
- and, Data Mining, pages 227-235, Boston, MA, August 2000.
12641 D. Holt and S. M. Chung. Efficient Mining of Association Rules in Text Databases.
J.
In Proc. of the 8th IntI. Conf. on Inforrnation and, Knowledge Management, pages
234-242, Kansas City Missouri, 1999.
[265] M. Houtsma and A. Swami. Set-oriented Mining for Association Rules in Relational
Databases. In Proc. of the 11th IntI. Conf. on Data Eng,ineering, pages 25 33, Taipei,
Taiwan, 1995.
[266] Y. Huang, S. Shekhar, and H. Xiong. Discovering Co-location Patterns from Spatial
Datasets: A General Approach. IEEE Tfans. on Knowledge and, Data Engineering, L6
(12) :1472-1485, December 2004.
12671 T.Imielinski, A. Virmani, and A. Abdulghani. DataMine: Application Programming
Interface and Query Language for Database Mining. In Proc. of the 2nd Intl. Conf.
on Knowledge D'iscouerg and Data Mi,n'ing,pages 256-262, Portland, Oregon, 1996.
[268] A. Inokuchi, T. Washio, and H. Motoda. An Apriori-based Algorithm for Mining
Frequent Substructures from Graph Data. In Proc. of the lth European Conf. of Prin-
ci,ples and Practice of Knowledge Discouery i,n Databases, pages 13 23, Lyon, Fbance,
2000.
f269] S. Jaroszewicz and D. Simovici. Interestingness of Flequent Itemsets Using Bayesian
Networks as Background Knowledge. In Proc. of the 10th Intl. Conf. on Knowled"ge
Discouerg and Data Min'ing, pages 178-186, Seattle, WA, August 2004.
1270] M. Kamber and R. Shinghal. Evaluating the Interestingness of Characteristic Rules. In
Proc. of the Znd Intl. Conf. on Knowledge Di,scouerE and Data Min'ing, pages 263-266,
Portland, Oregon, 1996.
l27ll M. Klemettinen. A Knowleilge Di,scoaerg Methodologg for Telecornrnunicat'ion Network
Alarm Databases. PhD thesis, University of Helsinki, 1999.
1272) W. A. Kosters, E. Marchiori, and A. Oerlemans. Mining Clusters with Association
Rules. In The 9rd, SEmp. on Intelligent Data AnalEsis (IDA99), pages 39-50, Amster-
dam, August 1999.
12731 C. M. Kuok, A. Fu, and M. H. Wong. Mining Fuzzy Association Rules in Databases.
ACM SIGMOD Record,27(l):47-46, March 1998.
4OO Chapter 6 Association Analysis

1274] M. Kuramochi and G. Karypis. Frequent Subgraph Discovery. In Proc. of the 2001
IEEE Intl. Conf. on Data Mi,ning, pages 313-320, San Jose, CA, November 2001.
1275] W. Lee, S. J. Stolfo, and K. W. Mok. Adaptive Intrusion Detection: A Data Mining
Approach. Artificial Intelligence Reu'iew,14(6) :533-567, 2000.
1276] W. Li, J. Han, and J. Pei. CMAR: Accurate and Efficient Classification Based on
Multiple Class-association Rules. In Proc. of the 2001 IEEE IntI. Conf. on Data
M'ining, pages 369 376, San Jose, CA, 2001.
12771B. Liu, W. Hsu, and S. Chen. Using General Impressions to Analyze Discovered
Classification Rules. In Proc. of the Srd Intl. Conf. on Knowledge Discouery and Data
Mining, pages 31-36, Newport Beach, CA, August 1997.
12781B. Liu, W. Hsu, and Y. Ma. Integrating Classification and Association Rule Mining.
In Proc. of the lth IntI. Conf. on Knowledge D'iscouery and, Data M'ini,ng, pages 80-86,
New York, NY, August 1998.
1279] B. Liu, W. Hsu, and Y. Ma. Mining association rules with multiple minimum supports.
In Proc. of the Sth Intl. Conf. on Knowledge Discouerg and Data Mining, pages 125
134, San Diego, CA, August 1999.
1280] B. Liu, W. Hsu, and Y. Ma. Pruning and Summarizing the Discovered Associations. In
Proc. of theSthIntI. Conf. onKnowledgeDiscoueryandDataMining, pages125 134,
San Diego, CA, August 1999.
1281] A. Marcus, J. L Maletic, and K.-I. Lin. Ordinal association rules for error identifi-
cation in data sets. In Proc. of the 10th Intl. Conf. on Inforrnation and, Knowledge
Management, pages 589-591, Atlanta, GA, October 2001.
[282] N. Megiddo and R. Srikant. Discovering Predictive Association Rules. In Proc. of the
Ith Intl. Conf. on Knowled,ge Discouery and Data Min'ing, pages 274-278, New York,
August 1998.
[283] R. Meo, G. Psaila, and S. Ceri. A New SQL-like Operator for Mining Association
Rules. In Proc. of the 22nd VLDB Conf., pages I22 133, Bombay, India, 1-996.
[284j R. J. Miller and Y. Yang. Association Rules over Interval Data. In Proc. of 1997
ACM-SIGMOD Intl. Conf. on Management of Data, pages 452-461, T\rcson, LZ,May
1997.
[285] Y. Morimoto, T. Fukuda, H. Matsuzawa, T. Tokuyama, and K. Yoda. Algorithms for
mining association rules for binary segmentations of huge categorical databases. In
Proc. of the 2lth VLDB Conf., pages 380-391, New York, August 1998.
[286] F. Mosteller. Association and Estimation in Contingency Tables. Journal of the Amer-
ican Statistical Association 63:1-28. 1968,
12871 A. Mueller. Fast sequential and parallel algorithms for association rule mining: A
comparison. Technical Report CS-TR-3515, University of Maryland, August 1995.
[288] R. T. Ng, L. V. S. Lakshmanan, J. Han, and A. Pang. Exploratory Mining and Pruning
Optimizations of Constrained Association Rules. In Proc. of 1998 ACM-SIGMOD IntI.
Conf. on Management of Data, pages 13-24, Seattle, WA, June 1998.
1289] E. Omiecinski. Alternative Interest Measures for Mining Associations in Databases.
IEEE Ttans. on Knowledge and" Data Engineering, 15(1):57-69, January/February
2003.
[290] B. Ozden, S. Ramaswamy, and A. Silberschatz. Cyclic Association Rules. In Proc. of
the llth IntI. Conf. on Data Eng., pages 412 42I, Orlando, FL, February 1998.
[291] A. Ozgur, P. N. Tan, and V. Kumar. RBA: An Integrated Framework for Regression
based on Association Rules. In Proc. of the SIAM IntI. Conf. on Data M'ining, pages
2 M 2 7 , O r l a n d o ,F L , A p r i l 2 0 0 4 .
 Bibliography 4OL

1292] J. S. Park, M.-S. Chen, and P. S. Yu. An efiective hash-based algorithm for mining
association rrles. SIGMOD Record,25(2):175 186, 1995.
[293] S. Parthasarathy and M. Coatney. Efficient Discovery of Common Substructures in
Macromolecules. In Proc. of the 2002 IEEE IntI. Conf. on Data M'ining, pages 362-369,
Maebashi City, Japan, December 2002.
[294] N. Pasquier, Y. Bastide, R. Taouil, and L. Lakhal. Discovering frequent closed itemsets
for association rules. In Proc. of the 7th Intl. Conf. on Database Theory 0CDT'99),
pages 398 416, Jerusalem, Israel, January 1999.
[295] J. Pei, J. Han, H. J. Lu, S. Nishio, and S. Tang. H-Mine: Hyper-Structure Mining of
Frequent Patterns in Large Databases. In Proc. of the 2001 IEEE Intl. Conf. on Data
M'ining, pages 441-448, San Jose, CA, November 2001.
[296] J. Pei, J. Han, B. Mortazavi-Asl, and H. Zhtt. Mining Access Patterns Efficiently from
Web Logs. In Proc. of the lth Pacific-Asia Conf. on Knowledge Discouery and" Data
Mining, pages 396-407, Kyoto, Japan, April 2000.
1297] G. Piatetsky-Shapiro. Discovery, Analysis and Presentation of Strong Rules. In
G. Piatetsky-Shapiro and W. Frawley, editors, Knowledge Discouery in Databases,
pages 229-248. MIT Press, Cambridge, MA, 1991.
[298] C. Potter, S. Klooster, M. Steinbach, P. N. Tan, V. Kumar, S. Shekhar, and C. Car-
valho. Understanding Global Teleconnections of Climate to Regional Model Estimates
of Amazon Ecosystem Carbon Fluxes. Global Change Biology, 70(5):693-703, 20A4.
[299] C. Potter, S. Klooster, M. Steinbach, P. N. Tan, V. Kumar, S. Shekhar, R. Myneni,
and R. Nemani. Global Teleconnections of Ocean Climate to Terrestrial Carbon Flux.
J. Geophysical Research, 108(D17), 2003.
1300] G. D. Ramkumar, S. Ranka, and S. Tsur. Weighted Association Rules: Model and
Algorithm. http: //www.cs.ucla.edu/ "c zdemof tsw f , 1997.
13o1lS . Sarawagi, S. Thomas, and R. Agrawal. Integrating Mining with Relational Database
Systems: Alternatives and Implications. In Proc. of 1998 ACM-SIGMOD IntI. Conf.
on Management of Data, pages 343-354, Seattle, WA, 1998.
[302] K. Satou, G. Shibayama, T, Ono, Y. Yamamura, E. Furuichi, S. Kuhara, and T. Takagi.
Finding Association Rules on Heterogeneous Genome Data. In Proc. of the Pacific
Symp. on Biocomputing, pages 397-408, Hawaii, January 1997.
[303] A. Savasere, E. Omiecinski, and S. Navathe. An efficient algorithm for mining associ-
ation rules in large databases. In Proc. of the 21st Int. Conf. on Very Large Databases
(VLDB'gs), pages 432-444, Z:uicln, Switzerland, September 1995.
1304] A. Savasere, E. Omiecinski, and S. Navathe. Mining for Strong Negative Associations
in a Large Database of Customer Tbansactions. In Proc. of the llth Intl. Conf. on
Data Engineering, pages 494 502, Orlando, Florida, February 1998.
[305] M. Seno and G. Karypis. LPMiner: An Algorithm for Finding FYequent Itemsets Using
Length-Decreasing Support Constraint. In Proc. of the 2001 IEEE Intl. Conf. on Data
Min'ing, pages 505-512, San Jose, CA, November 2001.
[306] T. Shintani and M. Kitsuregawa. Hash based parallel algorithms for mining association
rules. In Proc of the lth IntI. Conf . on Parallel and, Di;stributed, Info. Sgstems, pages
19-30, Miami Beach, FL, December 1996.
[307] A. Silberschatz and A. Tirzhilin. What makes patterns interesting in knowledge discov-
ery systems. IEEE Trans. on Knowledge and Data Engineering,8(6):970-974, 1996.
[308] C. Silverstein, S. Brin, and R. Motwani. Beyond market baskets: Generalizing associ-
ation rules to dependence rules. Data Mining and Knowledge D'iscouery, 2(1):39-68,
1998.
4O2 Chapter 6 Association Analysis

f309] E.-H. Simpson. The Interpretation of Interaction in Contingency Tables. Journal of

the Rogal Stati,stical Societg, B(13):238-241, 1951.
1310] L. Singh, B. Chen, R. Haight, and P. Scheuermann. An Algorithm for Constrained
Association Rule Mining in Semi-structured Data. In Proc. of the ?rd Pacific-Asi,a
Conf. on Knouled.ge Di,scoueryand Data M'ining, pages 148-158, Beijing, China, April
1999.
1311] R. Srikant and R. Agrawal. Mining Quantitative Association Rules in Large Relational
Tables. In Proc. of 1996 ACM-SIGMOD IntI. Conf. on Management of Data, pages
1-12, Montreal, Canada, 1996.
[312] R. Srikant and R. Agrawal. Mining Sequential Patterns: Generalizations and Perfor-
mance Improvements. In Proc. of the 5th IntI Conf. on Ertend'ing Database Technologg
(EDBT'96), pages 18 32, Avignon, France, 1996.
[313] R. Srikant, Q. Vu, and R. Agrawal. Mining Association Rules with Item Constraints.
In Proc. of the 9rd IntI. Conf. on Knowledge D'iscouery and Data Mining, pages 67-73,
Newport Beach, CA, August 1997.
1314] M. Steinbach, P. N. Tan, and V. Kumar. Support Envelopes: A Technique for Ex-
ploring the Structure of Association Patterns. In Proc. of the 10th Intl. Conf. on
Knowled,ge D'iscouery and, Data Min'ing, pages 296 305, Seattle, WA, August 2004.
1315] M. Steinbach, P. N. Tan, H. Xiong, and V. Kumar. Extending the Notion of Support.
In Proc. of the 10th IntI. Conf. on Knowledge Discoaerg and Data Mining, pages 689-
694, Seattle, WA, August 2004.
[316] E. Suzuki. Autonomous Discovery of Reliable Exception Rules. In Proc. of the ?rd
Intl. Conf. on Knowled,ge Discouery and Data Mi,ning, pages 259-262, Newport Beach,
CA, August 1997.
[317] P. N. Tan and V. Kumar. Mining Association Patterns in Web Usage Data. In Proc.
of the IntI. Conf. on Ad"uances'in Infrastructure for e-Business, e-Ed"ucation, e-Science
and e-Medi,ci,ne on the Internet, L'Aquila, Italy, January 2002.
1318] P. N. Tan, V. Kumar, and J. Srivastava. Selecting the Right Interestingness Measure
for Association Patterns. In Proc. of the Sth Intl. Conf. on Knowledge D'iscouery and,
Data Mining, pages 32-41, Edmonton, Canada, JuJy 2002.
[319] P. N. Tan, M. Steinbach, V. Kumar, S. Klooster, C. Potter, and A. Torregrosa. Finding
Spatio.Temporal Patterns in Earth Science Data. In KDD 2001 Workshop on Temporal
Data Mi,ni,ng,San Francisco, CA, 2001.
[320] H. Toivonen. Sampling Large Databases for Association Rules. In Proc. of the 22nd
VLDB Conf., pages 134-145, Bombay, India, 1996.
1321] H. Toivonen, M. Klemettinen, P. Ronkainen, K. Hatonen, and H. Mannila. Pruning
and Grouping Discovered Association Rules. In ECML-95 Workshop on Statist'ics,
Machine Learning and, Knowledge D'iscouery 'in Databases, pages 47 - 52, Heraklion,
Greece, April 1995.
[322] S. Tsur, J. Ullman, S. Abiteboul, C. Clifton, R. Motwani, S. Nestorov, and A. Rosen-
thal. Query Flocks: A Generalization of Association Rule Mining. In Proc. of 1998
ACM-SIGMOD Intl. Conf. on Management of Data, pages 1-12, Seattle, WA, June
1998.
[323] A. Tung, H. J. Lu, J. Han, and L. Feng. Breaking the Barrier of TYansactions: Mining
Inter-TYansaction Association Rules. In Proc. of the Sth Intl. Conf. on Knowledge
Discouery and, Data Mining, pages 297-301, San Diego, CA, August 1999.
[324] K. Wang, Y. He, and J. Han. Mining Frequent Itemsets Using Support Constraints.
In Proc. of the 26th VLDB Conf., pages 43 52, Cairo, Egypt, September 2000.
 BIBLIOGRAPHY 4O3

[325] K. Wang, S. H. Tay, and B. Liu. Interestingness-Based Interval Merger for Numeric
Association Rules. In Proc. of the lth IntL Conf. on Knowledge Discouerg and Data
Min'ing, pages 121-128, New York, NY, August 1998.
[326] G. I. Webb. Preliminary investigations into statistically valid exploratory rule dis-
covery. In Proc. of the Australasian Data Mi,ni,ng Workshop (AusDM1?), Canberra,
Australia, December 2003.
[327] H. Xiong, X. He, C. Ding, Y.Zhang, V. Kumar, and S. R. Holbrook. Identification
of Functional Modules in Protein Complexes via Hyperclique Pattern Discovery. In
Proc. of the Pacif,c Sgmposium on Biocomputing, (PSB 2005),Mad, January 2005.
13281H. Xiong, S. Shekhar, P. N. Tan, and V. Kumar. Exploiting a Support-based Upper
Bound of Pearson's Correlation Coefficient for Efficiently Identifying Strongly Corre-
lated Pairs. In Proc. of the 10th IntI. Conf. on Knowled,ge Di,scouery and Data Mining,
pages 334 343, Seattle, WA, August 2004.
[329] H. Xiong, M. Steinbach, P. N. Tan, and V. Kumar. HICAP: Hierarchial Clustering
with Pattern Preservation. In Proc. of the SIAM Intl. Conf. on Data Mining, pages
279 290, Orlando, FL, April 2004.
[330] H. Xiong, P. N. Tan, and V. Kumar. Mining Strong Affinity Association Patterns in
Data Sets with Skewed Support Distribution. In Proc. of the 2003 IEEE IntI. Conf.
on Data M'in'ing, pages 387 394, Melbourne, FL,2OO3.
[331] X. Yan and J. Han. gSpan: Graph-based Substructure Pattern Mining. ln Proc. of
the 2002 IEEE Intl. Conf. on Data Mining, pages 72I 724, Maebashi City, Japan,
December 2002.
1332] C. Yang, U. M. Fayyad, and P. S. Bradley. Efficient discovery of error-tolerant frequent
itemsets in high dimensions. In Proc. of the 7th Intl. Conf. on Knowled,ge Discouery
and, Data M'in'ing, pages 194 203, San Francisco, CA, August 2001.
f333] M. J. Zaki. Parallel and Distributed Association Mining: A Survey. IEEE Concurrency,
special issue on Parallel Mechanisrns for Data Min'ing,7(4):14-25, December 1999.
[334] M. J. Zaki. Generating Non-Redundant Association Rules. In Proc. of the 6th IntI.
Conf. on Knowledge Discouery and Data Min'ing, pages 34-43, Boston, MA, August
2000.
[335] M. J. Zaki. Efficiently mining frequent trees in a forest. In Proc. of the 8th IntI.
Conf. on Knowledge Di,scouery and Data Mining, pages 71-80, Edmonton, Canada,
July 2002.
f336] M. J. Zaki and M. Orihara. Theoretical foundations of association rules. In Proc. of
the 1998 ACM SIGMOD Workshop on Research Issues in Data Mining and, Knowledge
Discouery, Seattle, WA, June 1998.
[337] M. J. Zaki, S. Parthasarathy, M. Ogihara, and W. Li. New Algorithms for Fast
Discovery of Association Rules. In Proc. of the Srd Intl. Conf . on Knowled,ge Discouery
and"Data Mining, pages 283-286, Newport Beach, CA, August 1997.
[338] H. Zhang, B. Padmanabhan, and A. T\rzhilin. On the Discovery of Significant Statis-
tical Quantitative Rules. In Proc. of the 10th IntL Conf. on Knowled,ge D'iscouery and
Data Mi.ning, pages 374-383, Seattle, WA, August 2004.
13391 Z. Zhang, Y. Lu, and B. Zhang. An Effective Partioning-Combining Algorithm for
Discovering Quantitative Association Rules. In Proc. of the 1st Paci,fic-Asia Conf. on
Knowled"geDiscouery and, Data Mi,ning, Singapore, 1997.
[340] N. Zhong, Y. Y. Yao, and S. Ohsuga. Peculiarity Oriented Multi-database Mining. In
Proc. of the ?rd European Conf. of Principles and Practice of Knowledge Discouery in
Databases,pages 136-146, Prague, Czech Republic, 1999.
4O4 Chapter 6 Association Analysis

6.10 Exercises
1. For each of the following questions, provide an example of an association rule
from the market basket domain that satisfies the following conditions. Also,
describe whether such rules are subjectively interesting.

(a) A rule that has high support and high confidence.

(b) A rule that has reasonably high support but low confidence.
(c) A rule that has low support and low confidence.
(d) A rule that has low support and high confidence.

2. Consider the data set shown in Table 6.22.

6.22.Example
Table ofmarket transactions.
basket
CustomerID Tlansaction ID Items ljouqht
I {a,d,e}
1 0024 {a,b,c,e}
2 0012 {a,b,d,e}
2 0031 {a, c, d, e}
3 0015 {b,c,e}
3 0022 {b,d,e}
4 0029 {",d}
4 0040 {a,b,c}
5 0033 {a, d,e}
tr
0038 { a , b ,e }

(a) Compute the support for itemsets {e}, {b,d}, and {b,d,e} by treating
each transaction ID as a market basket.
(b) Use the results in part (a) to compute the confidence for the associa-
tion rules {b,d} ---, {e} and {"} ------+
{b,d}. Is confidence a symmetric
measure?
(") Repeat part (a) by treating each customer ID as a market basket. Each
item should be treated as a binary variable (1 if an item appears in at
Ieast one transaction bought by the customer, and 0 otherwise.)
(d) Use the results in part (c) to compute the confidence for the association
rules {b, d} - {e} and {"}------ {b,d,}.
( e ) Suppose s1 and c1 are the support and confidence values ofan association
rule r when treating each transd,ction ID as a market basket. Also, let s2
and c2 be the support and confidence values of r when treating each cus-
tomer ID as a market basket. Discuss whether there are any relationships
between sr and s2 or c1 and,c2.
 6.10 Exercises 4O5

3. (a) What is the confidence for the rules A ------A and A ---- A?
(b) Let crt c2t and ca be the confidence values of the rules {p} ------+ {q},
{p} - {q,r}, and {p, r} - {g}, respectively. If we assume that c1, c2,
and ca have different values, what are the possible relationships that may
exist among cI, c2, and ca? Which rule has the lowest confidence?
(c) Repeat the analysis in part (b) assuming that the rules have identical
support. Which rule has the highest confidence?
(d) Tiansitivity: Suppose the confidence of the rules A - B and B -------,
C
are larger than some threshold, m'inconf . Is it possible that A -----*C has
a confidence less than mi,nconf?

4 . For each of the following measures, determine whether it is monotone, anti-

monotone, or non-monotone (i.e., neither monotone nor anti-monotone).

Example: Support, t : "iFl is anti-monotone because s(X) )

s ( Y ) w h e n e v e rX C Y .

(a) A characteristic rule is a rule of the form {p) - {qt,qr,. . . , g,}, where
the rule antecedent contains only a single item. An itemset of size k can
produce up to k characteristic rules. Let ( be the minimum confidence of
all characteristic rules generated from a given itemset:

C ( { p r , p z ,.. . , p k } ) m i n I c ( { p 1 }- { p z , p s , .. . , p k } ) , - - -
'({Pn} ' {Pr'Ps" ' 'Pl"-1}) ]

Is ( monotone, anti-monotone, or non-monotone?

(b) A discriminant rule is a rule of the form {pt,pr,. . .,pn} ------{q}, where
the rule consequent contains only a single item. An itemset of size k can
produce up to k discriminant rules. Let 4 be the minimum confidence of
all discriminant rules generated from a given itemset:

\({pr,pz, -. -,pk}) : m i n l c ( { p 2 , P z-, . - , P n } - - - - -{-p r } ) , . . .

c({n,nz,. . .pn-t}------ {p*}) ]

Is 4 rhonotone, anti-monotone, or non-monotone?

(c) Repeat the analysis in parts (a) and (b) bV replacing the min function
with a max function.

5. Prove Equation 6.3. (Hint: First, count the number of ways to create an itemset
that forms the left hand side of the rule. Next, for each size k itemset selected
for the left-hand side, count the number of ways to choosethe remainin1d-k
items to form the right-hand side of the rule.)
4OG Chapter 6 Association Analysis

6.23.Market
Table basket
transactions,
Tlansaction ID Items Boueht
1 tMilk, Beer, Diapers]
2 {Bread, Butter, Milk}
3 {Milk, Diapers, Cookies}
4 {Bread, Butter, Cookies}
{Beer, Cookies, Diapers}
o {Milk, Diapers, Bread, Butter}
7 {Bread, Butter, Diapers}
8 {Beer, Diapers}
9 {Milk, Diapers, Bread, Butter}
10 {Beer, Cookies}

6. Consider the market basket transactions shown in Table 6.23.

(a) What is the maximum number of association rules that can be extracted
from this data (including rules that have zero support)?
(b) What is the maximum size of frequent itemsets that can be extracted
(assuming minsup > 0)?
(c) Write an expression for the maximum number of size-3 itemsets that can
be derived from this data set.
(d) Find an itemset (of size 2 or larger) that has the largest support.
(e) Find a pair of items, a and b, such that the rules {o} -, {b} and {b} -----+
{a} have the same confidence.

7. Consider the following set of frequent 3-itemsets:

{ 1 , 2 , 3 } , { 1 , 2 , 4 ) , { r , 2 , 5 } , { r , 3 , 4 } , { 1 ,3 , 5 } , { 2 , 3 , 4 } , { 2 , 3 , 5 } , { 3 , 4 , 5 } .

Assume that there are only five items in the data set.

(a) List all candidate 4-itemsets obtained by a candidate generation procedure

using the Fn-t x ?'1 merging strategy.
(b) List all candidate 4-itemsets obtained by the candidate generation proce-
dure in Apriori.
(c) List all candidate 4-itemsets that survive the candidate pruning step of
the Apriori, algorithm.

8. The Apri,ori algorithm uses a generate-and-count strategy for deriving frequent

itemsets. Candidate itemsets of size k + I are created by joining a pair of
frequent itemsets of size k (this is known as the candidate generation step). A
candidate is discarded if-any one of its subsets is found to be infrequent during
the candidate pruning step. Suppose the Apriori algorithm is applied to the
 6.10 Exercises 4O7

Table6.24.Example
of market
basket
transactions.
Tlansaction ID Items Bought
1 { a , b ,d , e }
2 {b,c,d}
3 {a,b,d,e}
4 {a, c, d, e}
r
J
{b, c, d, e}
0 {b,d,e}
7 {", d}
8 { a , b ,c }
I {a, d,e}
10 {b, d}

data set shown in Table 6.24 with m'insup : 30yo, i.e., any itemset occurring
in less than 3 transactions is considered to be infrequent.

(a) Draw an itemset lattice representing the data set given in Table 6.24.
Label each node in the Iattice with the following letter(s):
o N: If the itemset is not considered to be a candidate itemset by
the Apriori. algorithm. The-re are two reasons for an itemset not to
be considered as a candidate itemset: (1) it is not generated at all
during the candidate generation step, or (2) it is generated during
the candidate generation step but is subsequently removed during
the candidate pruning step becauseone of its subsets is found to be
infrequent.
o F: If the candidate itemset is found to be frequent by the Apri'ori
algorithm.
o I: If the candidate itemset is found to be infrequent after support
counting.
(b) What is the percentage of frequent itemsets (with respect to all itemsets
in the lattice)?
(c) What is the pruning ratio of the Apri,ori algorithm on this data set?
(Pruning ratio is defined as the percentage of itemsets not considered
to be a candidate because (1) they are not generated during candidate
generation or (2) they are pruned during the candidate pruning step.)
(d) What is the false alarm rate (i.e, percentage of candidate itemsets that
are found to be infrequent after performing support counting)?

9. The Apriori algorithm uses a hash tree data structure to effrciently count the
support of candidate itemsets. Consider the hash tree for candidate 3-itemsets
shown in Figure 6.32.
408 Chapter 6 Association Analysis

Figure6,32,Anexample
ofa hashtreestructure.

(a) Given a transaction that contains items {1,3,4,5,8}, which of the hash
tree leaf nodes will be visited when fi.nding the candidates of the transac-
tion?
(b) Use the visited leaf nodes in part (b) to determine the candidate itemsets
that are contained in the transaction {1,3,4,5,8}.

10. Consider the following set of candidate 3-itemsets:

{ t , 2 , 3 } , { r , 2 , 6 } , { 1 , 3 , 4 } ,{ 2 , 3 , 4 ) ,{ 2 , 4 , 5 } ,{ 3 , 4 , 6 } ,{ 4 , 5 , 6 }

(a) Construct a hash tree for the above candidate 3-itemsets. Assume the
tree uses a hash function where all odd-numbered items are hashed to
the left child of a node, while the even-numbered items are hashed to the
right child. A candidate k-itemset is inserted into the tree by hashing on
each successiveitem in the candidate and then following the appropriate
branch of the tree according to the hash value. Once a leaf node is reached,
the candidate is inserted based on one of the following conditions:
Condition 1: If the depth of the leaf node is equal to k (the root is
assumed to be at depth 0), then the candidate is inserted regardless
of the number of itemsets already stored at the node.
Condition 2: If the depth of the leaf node is less than k, then the candi-
date can be inserted as long as the number of itemsets stored at the
node is less than mars'ize. Assume mars'ize:2 for this question.
Condition 3: If the depth of the leaf node is less than k and the number
of itemsets stored at the node is equal to mars'ize, then the leaf
node is converted into an internal node. New leaf nodes are created
as children of the old leaf node. Candidate itemsets previouslv stored
 6.10 Exercises 4Og

Figure
6.33.Anitemset
lattice

in the old leaf node are distributed to the children based on their hash
values. The new candidate is also hashed to its appropriate leafnode.
(b) How many leaf nodes are there in the candidate hash tree? How many
internal nodes are there?
(c) Consider a transaction that contains the following items: {1,2,3,5,6}.
Using the hash tree constructed in part (a), which leaf nodes will be
checked against the transaction? What are the candidate 3-itemsets con-
tained in the transaction?

11. Given the lattice structure shown in Figure 6.33 and the transactions given in
Table 6.24,label each node with the following letter(s):

a M if the node is a maximal frequent itemset,

a C if it is a closed frequent itemset,
a ,A'/if it is frequent but neither maximal nor closed, and
a 1 if it is infreouent.

Assume that the support threshold is equal to 30T0.

12. The original association rule mining formulation uses the support and confi-
dence measures to prune uninteresting rules.
ALO Chapter 6 Association Analysis

(a) Draw a contingency table for each of the following rules using the trans-
actions shown in Table 6.25.

Table6,25.Example
ofmarket transactions.
basket
Tlansaction ID Items Bought
I { a , b ,d , e }
2 {b,c,d}
3 { a , b ,d , e }
4 {a, c, d, e}
E
{b,c,d,e}
6 {b,d,e}
7 {",d}
8 {a,b,c}
I {a,d,e}
10 {b,d}

Rules: {b} * {"}, {r} - {d,}, {b} - {d}, {"} - {"}, {c}------ ia}.

(b) Use the contingency tables in part (a) to compute and rank the rules in
decreasing order according to the following measures.
i. Support.
ii. Confidence.
iii. Interest(X----+Y): t##PV).
iv. IS(X ------Y) --

X ------+
v. Klosgen( Y) : \IF@fi x (P(Y lX) - P (Y)), whereP(Y lX) :
P(X\

Y)t -:1+++EF).
vi. Oddsratio(X------ e6.Y1e1X.v1'

13. Given the rankings you had obtained in Exercise 12, compute the correlation
between the rankings of confidence and the other five measures. Which measure
is most highly correlated with confidence? Which measure is least correlated
with confidence?

14. Answer the following questions using the data sets shown in Figure 6.34. Note
that each data set contains 1000 items and 10,000 transactions. Dark cells
indicate the presence of items and white cells indicate the absence of items. We
will apply l}ae Apriori, algorithm to extract frequent itemsets with m'insup :
I0% (i.e., itemsets must be contained in at least 1000 transactions)?

(a) Which data set(s) will produce the most number of frequent itemsets?
 6.L0 Exercises 4LL

(b) Which data set(s) will produce the fewest number of frequent itemsets?
(c) Which data set(s) will produce the longest frequent itemset?
(d) Which data set(s) will produce frequent itemsets with highest maximum
support?
(e) Which data set(s) will produce frequent itemsets containing items with
wide-va,rying support levels (i.e., items with mixed support, ranging from
less than 20% to more than 70%).

15. (u) Prove that the / coefficientis equal to 1 if and only if fn : fr+ : f +r.
(b) Showthat if A andB areindependent, thenP(A, B)xP(A, B) : P(A, B)x
P(A, B).
(") Showthat Yule's Q and Y coefficients
-
n
\4 : f/rrloo /ro/otl
L/,r/00T /./rl
r, lrfitrf*- tfit;I'1
t'ff 'J* + 'If.of")
are normalized versions of the odds ratio.
(d) Write a simplified expression for the value of each measure shown in Tables
6.11 and 6.12 when the variables are statistically independent.

16. Consider the interestingness mea,sure, m : !-(W#4, for an association

rule ,4, -----+B.

(a) What is the range of this measure? When does the measure attain its
maximum and minimum values?
(b) How does M behave when P(,4, B) is increased while P(,a) and P(B)
remain unchanged?
(c) How does M behave when P(A) is increased while P(,4, B) and P(B)
remain unchanged?
(d) How does M behave when P(B) is increased while P(A, B) and P(A)
remain unchanged?
(e) Is the measure symmetric under variable permutation?
(f) What is the value of the measure when A and B are statistically indepen-
dent?
(g) Is the measure null-invariant?
(h) Does the measure remain inva.riant under row or column scaling opera-
tions?
(i) How does the measure behave under the inversion operation?
412 Chapter 6 Association Analysis

Items
I

rr
2000 2000
o
-t I
4000 () 4000
(U T
o
6000 $u I
6000
F
8000
l. It

I
8000

200 400 600 800

(b)

Items
l-
I
I
2000 2000
I
o o

4000 - I 4000
I
o
qt I
U'
6000 rr 6000
F
C'
lI r - tr
I rl
I. 8000 8000
-
II

200 400 600 800 200 400 600 800

(c) (d)

Items Items

2000 2000
o o
4000 o 1 0 %a r e 1 s 4000
o o
(U (U 90% are 0s
at, at,
(I' 6000 (U
(uniformlydistributed) 6000
F F
8000 8000

200 400 600 800 200 400 600 800

(e) (f)

Figute
6.34.Figures
forExercise
14.
 6.10 Exercises 4Lg

t7. Suppose we have market basket data consisting of 100 transactions and 20
items. If the support for item ais25To, the support for item bis90% and the
support for itemset {a, b} is 20%. Let the support and confidence thresholds
be L0% and 60%, respectively.

(a) Compute the confidence of the association rule {o} * {b}. Is the rule
interesting according to the confidence measure?
(b) Compute the interest measure for the association pattern {4, b}. Describe
the nature of the relationship between item a and item b in terms of the
interest measure.
(c) What conclusions can you draw from the results of parts (a) and (b)?
(d) Prove that if the confidence of the rule {o} * {b} is less than the support
of {b}, then:
i. c({a}------'
{bi) > c({di------{bi),
ii. c({a} ---' {b}) > .s({b}),
where c(.) denote the rule confidence and s(') denote the support of an
itemset.

18. Table 6.26 shows a 2 x 2 x 2 contingency table for the binary variables A and
B at different values of the control variable C.

Table.
Table6.26.A Contingency
A
1 0
1 0 15
C=0 B 0 15 30
1 5 0
C=1 B
0 0 15

(a) Compute the @coefficientfor A and B when C : O,C : I, and C : 0 or

1 . N o t e t h a t r6\(( { At . 8)}/ ) : W
\/P(A)P(B)(1-P(A))(I-P(B))
(b) What conclusions can you draw from the above result?

19. Consider the contingency tables shown in Table 6.27.

(a) For table I, compute support, the interest measure, and the / correla-
tion coeffi.cient for the association pattern {A, B}. Also, compute the
confidence of rules A -- B and B --- A.
4L4 Chapter 6 AssociationAnalysis

Table6.27.Contingency 19,
forExercise
tables

A A
A A

(b) Table II.

(b) For table II, compute support, the interest measure, and the @ correla-
tion coefficient for the association pattern {A, B}. Also, compute the
confidence of rules A ---, B and,B - A.
(c) What conclusions can you draw from the results of (a) and (b)?

20. Consider the relationship between customers who buy high-definition televisions
and exercise machines as shown in Tables 6.19 and 6.20.

(a) Compute the odds ratios for both tables.

(b) Compute the fcoefficient for both tables.
(c) Compute the interest factor for both tables.

For each of the measures given above, describe how the direction of association
changes when data is pooled together instead of being stratified.
AssociationAnalysis:
AdvancedConcepts
The association rule mining formulation described in the previous chapter
assumes that the input data consists of binary attributes called items. The
presenceof an item in a transaction is also assumed to be more important than
its absence. As a result, an item is treated as an asymmetric binary attribute
and only frequent patterns are considered interesting.
This chapter extends the formulation to data sets with symmetric binary,
categorical, and continuous attributes. The formulation will also be extended
to incorporate more complex entities such as sequencesand graphs. Although
the overall structure of association analysis algorithms remains unchanged, cer-
tain aspects of the algorithms must be modified to handle the non-traditional
entities.

7.1 Handling Categorical Attributes

There are many applications that contain symmetric binary and nominal at-
tributes. The Internet survey data shown in Table 7.1 contains symmetric
binary attributes such as Gender, Computer at Home, Chat Online, Shop
On1ine, and Privacy Concerns; as well as nominal attributes such as Leve1
of Education and State. Using association analysis, we may uncover inter-
esting information about the characteristics of Internet users such as:

------+ Concerns = Yes}.

{Suop On1ine = Yes} {Privacy

This rule suggests that most Internet users who shop online are concerned
about their personal privacy.
4LG Chapter 7 Association Analysis: Advanced Concepts

Table7.1.Internet
survey
datawithcategorical
attributes.
Gender Level o t State Computer Uhat Shop Privacy
Education at l{ome Online Online Concerns
Female Graduate Illinois Yes Yes Yes Yes
Male College California No No No No
Male Graduate Michigan Yes Yes Yes Yes
Female College Virginia No No Yes Yes
Female Graduate California Yes No No Yes
Male College Minnesota Yes Yes Yes Yes
Male College Alaska Yes Yes Yes No
Male High School Oregon Yes No No No
Female *: No
Graduate Texas
l:: ):

To extract such patterns, the categorical and symmetric binary attributes

are transformed into "items" first, so that existing association rule mining
algorithms can be applied. This type of transformation can be performed
by creating a new item for each distinct attribute-value pair. For example,
the nominal attribute Level of Education can be replaced by three binary
items: Education = Co1lege, Education = Graduate, and Education = High
School. Similarly, symmetric binary attributes such as Gender can be con-
verted into a pair of binary items, MaIe and Fenale. Table 7.2 shows the
result of binarizing the Internet survey data.

Table7.2.Internet
survey
dataafterbinarizing
categorical
andsymmetric
binary
attributes.
Male Female Education Education Privacy Privacy
: Graduate : College : Yes :NO
0 t 1 0 I 0
I 0 0 1 0 1
I 0 I 0 1 0
0 I 0 1
I I 0
0 1 L 0 1 0
I 0 0 I 1 0
1 0 0 I 0 1
1 0 0 0 0 1
0 1 0 0 I
i
 7.L HandlingCategoricalAttributes 41,7

There are several issues to consider when applying association analysis to

the binarized data:
1. Some attribute values may not be frequent enough to be part of a fre-
quent pattern. This problem is more evident for nominal attributes
that have many possible values, e.g., state names. Lowering the support
threshold does not help because it exponentially increases the number
of frequent patterns found (many of which may be spurious) and makes
the computation more expensive. A more practical solution is to group
related attribute values into a small number of categories. For exam-
ple, each state name can be replaced by its corresponding geographi-
cal region, such as Midwest, Pacific Northwest, Southwest, and East
Coast. Another possibility is to aggregate the less frequent attribute
values into a single category called Others, as shown in Figure 7.1.

NewYork

Michigan

California
Minnesota

Massachusetts

Figure7.1. A piechartwitha merged

category
called0thers.

2. Some attribute values may have considerably higher frequencies than

others. For example, suppose 85% of the survey participants own a
home computer. By creating a binary item for each attribute value
that appears frequently in the data, we may potentially generate many
redundant patterns, as illustrated by the following example:

= Yes, Shop Online = Yes}

{Computer at hone
------- = Yes}.
{Privacy Concerns
4LB Chapter 7 Association Analysis: Advanced Concepts

The rule is redundant because it is subsumed by the more general rule

given at the beginning of this section. Becausethe high-frequency items
correspond to the typical values of an attribute, they seldom carry any
new information that can help us to better understand the pattern. It
may therefore be useful to remove such items before applying standard
association analysis algorithms. Another possibility is to apply the tech-
niques presented in Section 6.8 for handling data sets with a wide range
of support values.

3. Although the width of every transaction is the same as the number of

attributes in the original data, the computation time may increase es-
pecially when many of the newly created items become frequent. This
is because more time is needed to deal with the additional candidate
itemsets generated by these items (see Exercise 1 on page 473). One
way to reduce the computation time is to avoid generating candidate
itemsets that contain more than one item from the same attribute. For
example, we do not have to generate a candidate itemset such as {State
= X, State = Y, . . .) becausethe support count of the itemset is zero.

7.2 Handling Continuous Attributes

The Internet survey data described in the previous section may also contain
continuous attributes such as the ones shown in Table 7.3. Mining the con-
tinuous attributes may reveal useful insights about the data such as "users
whose annual income is more than $120K belong to the 45 60 age group" or
"users who have more than 3 email accounts and spend more than 15 hours
online per week are often concernedabout their personal privacy." Association
rules that contain continuous attributes are commonly known as quantitative
association rules.
This section describesthe various methodologies for applying association
analysis to continuous data. We will specifically discuss three types of meth-
ods: (i) discretization-based methods, (2) statistics-based methods, and (3)
non-discretization methods. The quantitative association rules derived using
these methods are quite different in nature.

7.2.I Discretization-Based Methods

Discretization is the most common approach for handling continuous attributes.

This approach groups the adjacent values of a continuous attribute into a finite
number of intervals. For example, the Age attribute can be divided into the
 7.2 Handling ContinuousAttributes ALg

Table7.3. Internet
survey attributes.
datawithcontinuous
Gender Age Annual No. of Hours Spent No. of Email Privacy
Income Online per Week Accounts Concern
Female 26 90K 20 4 Yes
Male 51 135K 10 2 No
Male 29 80K 10 3 Yes
Female 45 120K 15 3 Yes
Female 31 95K 20 D Yes
Male 25 55K 25 D Yes
Male 37 100K 10 I No
2 No
Male
t"i1"
47
26 *1
65K 8
12 I
I:
following intervals:

A g e € [ 1 2 , 1 6 ) ,A g e € [ 1 6 , 2 0 ) ,A g e e 1 2 0 , 2 4 ) , . . . , A g e€ [ 5 6 , 6 0 ) ,

where [a, b) represents an interval that includes a but not b. Discretization

can be performed using any of the techniquesdescribed in Section 2.3.6 (equal
interval width, equal frequen,cy, entropy-based, or clustering). The discrete
intervals are then mapped into asymmetric binary attributes so that existing
association analysis algorithms can be applied. Table 7.4 shows the Internet
survey data after discretization and binarization.

Table7.4. Internet
survey
dataafterbinarizing andcontinuous
categorical attributes.
Male Female Age Age Age Privacy Privacy
< 1 3 € [13,21) € [21,30) : Yes :NO
0 1 0 0 1 I U
1 0 0 0 0 0 I
1 0 0 0 1 I 0
0 1 0 0 0 1 0
0 I U 0 0 1 0
1 0 0 0 I 1 0
1 0 0 0 0 0 1
1 0 0 0 0 0 I
0 I I
: i : :
42O Chapter 7 AssociationAnalysis: AdvancedConcepts

Table7.5.A breakdown userswhoparticipated

of Internet totheiragegroup.
inonlinechataccording
Age Group Chat Online : Yes Chat Online: No
12,16) 13
16,20) 11 2
.20,24) 11 3
24,28) 12 13
28,32) T4 t2
32,36) 15 L2
36,40) 16 t4
40,44) 16 t4
44,48) 10
48,52) 11
52,56) 10
56.60) 11

A key parameter in attribute discretization is the number of intervals used

to partition each attribute. This parameter is typically provided by the users
and can be expressed in terms of the interval width (for the equal interval
width approach), the average number of transactions per interval (for the equal
frequency approach), or the number of desired clusters (for the clustering-
based approach). The difficulty in determining the right number of intervals
can be illustrated using the data set shown in Table 7.5, which summarizes the
responses of 250 users who participated in the survey. There are two strong
rules embedded in the data:
Er: Age € 116,24) --r Chat Online : Yes (s : 8.8%, c : 81.5%).
Rzt Age € [44,60) ------+
Chat Online : No (s : t6.8To,c : 70To).

These rules suggest that most of the users from the age group of 16-24 often
participate in online chatting, while those from the age group of. 44-60 are less
likely to chat online. In this example, we consider a rule to be interesting only
if its support (s) exceeds5% and its confidence (c) exceeds65%. One of the
problems encountered when discretizing the Age attribute is how to determine
the interval width.

1. If the interval is too wide, then we may lose some patterns because of
their lack of confidence. For example, when the interval width is 24
years, .Rr and R2 arc replaced by the following rules:

R ' r , A g e € [ 1 2 , 3 6 )- - + C h a t O n l i n e : Y e s ( s : 3 0 % , c : 5 7 . 7 V o ) .
RL, Age € [36,60) ---+ Chat Online : No (s : 28y0,c: 58.3Vo).
 7.2 Handling Continuous Attributes 42I

Despite their higher supports, the wider intervals have caused the con-
fidence for both rules to drop below the minimum confidence threshold.
As a result, both patterns are lost after discretization.

If the interval is too narrow, then we may lose some patterns because of
their lack of support. For example, if the interval width is 4 years, then
.Rr is broken up into the following two subrules:

Rftn), Age € [16,20) ------+

Chat Online: Yes (s:4.4T0, c:84.6%).
li\
RYl, Age € 120,,24)--+ Chat Online : No (s:4.4To, c:78.6T0).

Since the supports for the subrules are less than the minimum support
threshold, -R1is lost after discretization. Similarly, the rule -Bz, which
is broken up into four subrules, will also be lost because the support of
each subrule is less than the minimum support threshold.

3 . If the interval width is 8 years, then the rule R2 is broken up into the
following two subrules:

Rt?,Age € 144,52)------+
Chat Online : No (s:8.4To,c:70To).
Rf), Age € [52,60) ------+ :
Chat Online No (s:8.4To,c:70T0).

Since El!) and ,t?l| have sufficient support and confidence,R2 can be
recoveredby aggregatingboth subrules. Meanwhile, E1 is broken up
into the following two subrules:

nl?)' Age € 112,20)----- Chat Online : Yes (s:9.2To,c:60.5%).

R[?' Age € [20,28)------+
Chat Online: Yes (s:9.2T0,c:60.0%).

Unlike R2, we cannot recoverthe rule ftr by aggregatingthe subrules

becauseboth subrulesfail the confidencethreshold.
One way to addressthese issuesis to considerevery possiblegrouping of
adjacentintervals. For example,we can start with an interval width of 4 years
and then merge the adjacentintervals into wider intervals,Age € [12,16),
Age € [L2,20),..., Age € [12,60),Age € [16,20),Age € [16,24),etc. This
approachenablesthe detection of both -Br and R2 as strong rules. However,
it also leadsto the following computational issues:
1. The computation becomes extremely expensive. If the rangeis
initially divided into,k intervals,then k(k -I)12 binary items must be
422 Chapter 7 Association Analysis: Advanced Concepts

generated to represent all possible intervals. F\rrthermore, if an item

corresponding to the interval [a,b) is frequent, then all other items cor-
responding to intervals that subsume [a,b) must be frequent too. This
approach can therefore generate far too many candidate and frequent
itemsets. To addressthese problems, a maximum support threshold can
be applied to prevent the creation of items corresponding to very wide
intervals and to reduce the number of itemsets.

2. Many redundant rules are extracted. For example, consider the

following pair of rules:
R s , {Age e [16,20), Gender: MaIe] -----*{CUat Onl-ine = Yes},
R a , : {Age e 116,24),Gender: Male} ----* {Cnat Onl-ine = Yes}.
lB+ is a generalization of ,?3 (and R3 is a specialization of -Ra) because
Ba has a wider interval for the Age attribute. If the confidence values
for both rules are the same, then .E+ should be more interesting be-
cause it covers more examples-including those for R3. fi3 is therefore
a redundant rule.

7.2.2 Statistics-Based Methods

Quantitative associationrules can be used to infer the statistical properties of a
population. For example, suppose we are interested in finding the average age
ofcertain groups oflnternet usersbasedon the data provided in Tables 7.1 and
7.3. Using the statistics-basedmethod described in this section, quantitative
association rules such as the following can be extracted:

{Amual Incone > $100K, Shop Online = Yes} ----+Age: Mear : 38.

The rule states that the average age of Internet users whose annual income
exceeds $100K and who shop online regularly is 38 years old.

Rule Generation

To generate the statistics-based quantitative association rules, the target at-

tribute used to characterize interesting segments of the population must be
specified. By withholding the target attribute, the remaining categorical and
continuous attributes in the data are binarized using the methods described
in the previous section. Existing algorithms such as Apri,ori, or FP-growth
are then applied to extract frequent itemsets from the binarized data. Each
 7.2 Handling Continuous Attributes 423

frequent itemset identifies an interesting segment of the population. The dis-

tribution of the target attribute in each segment can be summarized using
descriptive statistics such as mean, median, variance, or absolute deviation.
For example, the preceding rule is obtained by averaging the age of Inter-
net users who support the frequent itemset {Annual Incorne > $100K, Snop
Online = Yes).
The number of quantitative associationrules discoveredusing this method
is the same as the number of extracted frequent itemsets. Becauseof the way
the quantitative association rules are defined, the notion of confidenceis not
applicable to such rules. An alternative method for validating the quantitative
association rules is presented next.

Rule Validation

A quantitative association rule is interesting only if the statistics computed

from transactions covered by the rule are different than those computed from
transactions not covered by the rule. For example, the rule given at the be-
ginning of this section is interesting only if the average age of Internet users
who do not support the frequent itemset {Annua1 Income > 100K, Shop
Online = Yes) is significantly higher or lower than 38 years old. To deter-
mine whether the difference in their average ages is statistically significant,
statistical hypothesis testing methods should be applied.
Consider the quantitative association rule, -4. ------+ t : p, where A is a
frequent itemset, t is the continuous target attribute, and p, is the averagevalue
of f among transactions covered by A. Furthermore, let p/ denote the average
value of f among transactions not covered by A. The goal is to test whether
the difference between p and p/ is greater than some user-specified threshold,
A. In statistical hypothesis testing, two opposite propositions, known as the
null hypothesis and the alternative hypothesis, are given. A hypothesis test
is performed to determine which of these two hypothesesshould be accepted,
based on evidence gathered from the data (seeAppendix C).
In this case, assuming that F I lt' , the null hypothesis is ,FIs: pt : p,l L,
while the alternative hypothesis is Ifi : Lt' > p * L. To determine which
hypothesis should be accepted, the following Z-statistic is computed:

l-t'- t-t- L
^2 ^2
(7.1)
rlt n2

where n1 is the number of transactions support ing A, nz is the number of trans-

actions not supporting A, s1 is the standard deviation for f among transactions
424 Chapter 7 Association Analysis: Advanced Concepts

that support A, and s2 is the standard deviation for t among transactions that
do not support A. Under the null hypothesis, Z has a standard normal distri-
bution with mean 0 and variance 1. The value of Z comptfted using Equation
7.1 is then compared against a critical value, Zo, which is a threshold that
depends on the desired confidence level. If / ) Za, then the null hypothesis
is rejected and we may conclude that the quantitative association rule is in-
teresting. Otherwise, there is not enough evidence in the data to show that
the difference in mean is statistically significant.
Example 7.1-. Consider the quantitative association rule

------
Age:F:38.
{Income > 100K, Shop Online:Yes}

Supposethere are 50 Internet users who supported the rule antecedent. The
standard deviation of their agesis 3.5. On the other hand, the averageage of
the 200 users who do not support the rule antecedent is 30 and their standard
deviation is 6.5. Assume that a quantitative association rule is considered
interesting only if the difference between p and ;.1/is more than 5 years. Using
Equation 7.1 we obtain

38-30-5
:4.4414.

For a one-sided hypothesis test at a g5% confidence level, the critical value
for rejecting the null hypothesis is 1.64. Since Z > 1.64, the null hypothesis
can be rejected. We therefore conclude that the quantitative association rule
is interesting because the difference between the average ages of users who
support and do not support the rule antecedent is more than 5 years. t

7.2.3 Non-discretization Methods

There are certain applications in which analysts are more interested in find-
ing associations among the continuous attributes, rather than associations
among discrete intervals of the continuous attributes. For example, consider
the problem of finding word associations in text documents, as shown in Ta-
ble 7.6. Each entry in the document-word matrix representsthe normalized
frequency count of a word appearing in a given document. The data is normal-
ized by dividing the frequency of each word by the sum of the word frequency
across all documents. One reason for this normalization is to make sure that
the resulting support value is a number between 0 and L. However, a more
 7.2 Handling ContinuousAttributes 425

Table7,6. Normalized matrix.

document-word
Document wordl word2 wordg worda word5 word6
d1 0.3 0.6 0 0 0 0.2
d2 0.1 0.2 0 0 0 0.2
ds 0.4 0.2 u.( 0 0 0.2
da 0.2 0 0.3 0 0 0.1
ds 0 0 0 1.0 1.0 0.3

important reason is to ensure that the data is on the same scale so that sets
of words that vary in the same way have similar support values.
In text mining, analysts are more interested in finding associations between
words (e.g., data and nining) instead of associationsbetween ranges of word
frequencies(e.g., data € [1,4] and mining € [2,3]). One way to do this is
to transform the data into a 0/1 matrix, where the entry is 1 if the normal-
ized frequency count exceeds some threshold t, and 0 otherwise. While this
approach allows analysts to apply existing frequent itemset generation algo-
rithms to the binarized data set, finding the right threshold for binarization
can be quite tricky. If the threshold is set too high, it is possible to miss some
interesting associations. Conversely,if the threshold is set too low, there is a
potential for generating a large number of spurious associations.
This section presents another methodology for finding word associations
known as min-Apri,ora. Analogous to traditional association analysis, an item-
set is considered to be a collection of words, while its support measuresthe
degree of association among the words. The support of an itemset can be
computed based on the normalized frequency of its corresponding words. For
example, consider the document d1 shown in Table 7.6. The normalized fre-
quencies for uordl and word2 in this document are 0.3 and 0.6, respectively.
One might think that a reasonableapproach to compute the association be-
tween both words is to take the average value of their normalized frequencies,
i.e., (0.3+0.6)12:0.45. The support of an itemset can then be computed by
summing up the averaged normalized frequencies across all the documents:

q=!f *gry
s({word,1,word,2}): *y# *W#:t.
This result is by no means an accident. Because every word frequency is
normalized to 1, averaging the normalized frequencies makes the support for
every itemset equal to 1. All itemsets are therefore frequent using this ap-
proach, making it uselessfor identifying interesting patterns.
426 Chapter 7 Association Analysis: Advanced Concepts

In min-Apriori, the association among words in a given document is ob-

tained by taking the minimum value of their normalized frequencies, i.e.,
min(word1,word2) : min(0.3,0.6) : 0.3. The support of an itemset is com-
puted by aggregating its association over all the documents.

s({word1,word2}): min(0.3,0.6) + min(0.4,0.2)

+ min(0.1,0'2)
* min(0.2,0)
: 0.6.

The support measure defined in min-Apriori has the following desired prop-
erties, which makes it suitable for finding word associationsin documents:

1. Support increases monotonically as the normalized frequency of a word

increases.

2. Support increasesmonotonically as the number of documents that con-

tain the word increases.

3. Support has an anti-monotone property. For example, consider a pair of

itemsets{A,B} and {,4, B,C}. Sincemin({A,B}) > min({A, B,C}),
s({A,B}) > B,C}). Therefore, support decreasesmonotonically
"({A,
as the number of words in an itemset increases.

The standard Apriori, algorithm can be modified to find associations among

words using the new support definition.

7.3 Handling a Concept Hierarchy

A concept hierarchy is a multilevel organization of the various entities or con-
cepts defined in a particular domain. For example, in market basket analysis,
a concept hierarchy has the form of an item taxonomy describing the "is-a"
relationships among items sold at a grocery store----e.g.,milk is a kind of food
and DVD is a kind of home electronics equipment (see Figure 7.2). Concept
hierarchies are often defined according to domain knowledge or based on a
standard classification scheme defined by certain organizations (e.g., the Li-
brary of Congress classification scheme is used to organize library materials
based on their subject categories).
A concept hierarchy can be representedusing a directed acyclic graph,
as shown in Figure 7.2. If there is an edge in the graph from a node p to
another node c, we call p the parent of c and c the child of p. For example,
 7.3 Handling a Concept Hierarchy 427

Food Electronics

??"ff:
"olSto,
Figure7.2. Example
of anitemtaxonomy.

nilk is the parent of skin milk because there is a directed edge from the
node milk to the node skim milk. * is called an ancestor of X (and X a
descendent of *) if there is a path from node * to node X in the directed
acyclic graph. In the diagram shown in Figure 7.2, f ood is an ancestor of skim
rnil-k and AC adaptor is a descendentof electronics.
The main advantages of incorporating concept hierarchies into association
analysis are as follows:

1. Items at the lower levels of a hierarchy may not have enough support to
appear in any frequent itemsets. For example, although the sale of AC
adaptors and docking stations may be low, the sale of laptop accessories,
which is their parent node in the concept hierarchy, may be high. Unless
the concept hierarchy is used, there is a potential to miss interesting
patterns involving the laptop accessories.

2. Rules found at the lower levels of a concept hierarchy tend to be overly

specific and may not be as interesting as rules at the higher levels. For
example, staple items such as milk and bread tend to produce many low-
level rules such as skim nilk ------+ wheat bread, 2"/" niJ-k ------+
wheat
bread, and skin milk -----+white bread. Using a concept hierarchy,
they can be summarized into a single rule, milk ------+bread. Considering
only items residing at the top level of their hierarchies may not be good
enough because such rules may not be of any practical use. For example,
although the rule electronics ----+food may satisfy the support and
428 Chapter 7 Association Analysis: Advanced Concepts

confidence thresholds, it is not informative because the combination of

electronics and food items that are frequently purchased by customers
are unknown. If milk and batteries are the only items sold together
frequently, then the pattern {food, electronics} may have overgener-
alized the situation.
Standard association analysis can be extended to incorporate concept hi-
erarchies in the following way. Each transaction t is initially replaced with its
extended transaction //, which contains all the items in t along with their
corresponding ancestors. For example, the transaction {DVD, wheat bread}
can be extended to {DVD,wheat bread, hone electronics, electronics,
bread, food), where hone electronics and electronics are the ancestors
of DVD,while bread and food are the ancestors of wheat bread. With this
approach, existing algorithms such as Apri,ori can be applied to the extended
database to find rules that span different levels of the concept hierarchy. This
approach has several obvious limitations:
1. Items residing at the higher levels tend to have higher support counts
than those residing at the lower levels of a concept hierarchy. As a result,
if the support threshold is set too high, then only patterns involving the
high-level items are extracted. On the other hand, if the threshold is set
too low, then the algorithm generates far too many patterns (most of
which may be spurious) and becomes computationally inefficient.

2. Introduction of a concept hierarchy tends to increase the computation

time of association analysis algorithms because of the larger number of
items and wider transactions. The number of candidate patterns and
frequent patterns generated by these algorithms may also grow expo-
nentially with wider transactions.

3. Introduction of a concept hierarchy may produce redundant rules. A

rule X ------+
Y is redundant if there exists a more general rule * , t,
where * is an ancestor of. X, i ir un ancestor of Y, and both rules
have very similar confidence. For example, suppose{bread} ------+{nilk},
------ -'-
{white bread} {2%ni1k}, {rheat bread} {2% milk}, {white
bread) ---+ {skim milk}, and {wheat bread} ------+ {skin nitk} have
very similar confidence. The rules involving items from the lower level of
the hierarchy are considered redundant becausethey can be summarized
by a rule involving the ancestor items. An itemset such as {skin nilk,
milk, food) is also redundant becausefood and milk are ancestors of
skim nilk. Fortunately, it is easy to eliminate such redundant itemsets
during frequent itemset generation, given the knowledge of the hierarchy.
 7.4 Sequential Patterns 429

Timeline
l rbt tlt bt l z r ' a r ' S s b
rttl
Sequencefor ttlltl

ObjectA: tttrrtl
rt,rtl
? i 61i i i
3i1irl

Sequencefor
ObjectB:

Sequencefor
ObjectC:
1
7
8

Figure
7.3.Example
ofa sequence
database,

7.4 Sequential Patterns

Market basket data often contains temporal information about when an item
was purchased by customers. Such information can be used to piece together
the sequenceof transactions made by a customer over a certain period of time.
Similarly, event-based data collected from scientific experiments or the mon-
itoring of physical systems such as telecommunications networks, computer
networks, and wireless sensor networks, have an inherent sequential nature
to them. This means that an ordinal relation, usually based on temporal or
spatial precedence,exists among events occurring in such data. However, the
conceptsof associationpatterns discussedso far emphasizeonly co-occurrence
relationships and disregard the sequential information of the data. The latter
information may be valuable for identifying recurring features of a dynamic
system or predicting future occurrencesof certain events. This section presents
the basic concept of sequential patterns and the algorithms developed to dis-
cover them.

7.4.L Problem Formulation

The input to the problem of discovering sequential patterns is a sequencedata

set, which is shown on the left-hand side of Figure 7.3. Each row records the
occurrencesof events associatedwith a particular object at a given time. For
example, the first row contains the set of events occurring at timestamp t : 10
43O Chapter 7 Association Analysis: Advanced Concepts

for object A. By sorting all the events associated with object A in increasing
order of their timestamps, a sequence for object A is obtained, as shown on
the right-hand side of Figure 7.3.
Generally speaking, a sequenceis an ordered list of elements. A sequence
can be denoted as s : (ep2es . . .en),, where each element e3 is a collection of
one or more events,i.e., ej : {h,'i2, . . . ,26}. The following is a list of examples
of sequences:

o Sequenceof Web pages viewed by a Web site visitor:

( {Homepage} {Electronics} {Cameras and Camcorders} {Digital Cam-
eras) {Shopping Cart} {Order Confirmation} {Return to Shoppi"e} )

o Sequenceof events leading to the nuclear accident at Three-Mile Island:

( {clogged resin} {outlet valve closure} {loss of feedwater} {condenser
polisher outlet valve shut) {booster pumps trip} {main waterpump trips}
{main turbine trips} {reactor pressure increases} )
o Sequenceof classestaken by a computer sciencemajor:
( {Algorithms and Data Structures, Introduction to Operating Systems}
{Database Systems,Computer Architecture} {Computer Networks, Sofb-
ware Engineering) {Computer Graphics, Parallel Programming} )

A sequence can be characterized by its length and the number of occur-

ring events. The length of a sequencecorrespondsto the number of elements
present in the sequence, while a k-sequence is a sequence that contains k
events. The Web sequence in the previous example contains 7 elements and
7 eventsl the event sequence at Three-Mile Island contains 8 elements and 8
events; and the class sequencecontains 4 elements and 8 events.
Figure 7.4 provides examples of sequences,elements, and events defined for
a variety of application domains. Except for the last row, the ordinal attribute
associatedwith each of the first three domains correspondsto calendar time.
For the last row, the ordinal attribute correspondsto the location of the bases
(A, C, G, T) in the gene sequence. Although the discussion on sequential
patterns is primarily focused on temporal events, it can be extended to the
case where the events have spatial ordering.

Subsequences

A sequencef is a subsequence of another sequences if each ordered element in

f is a subset of an ordered element in s. Formally, the sequence1 : (tfi2 . . .t^)
 7.4 SequentialPatterns 431

Sequence Sequence Element Event

Database (Transaction) (ltem)
Customer Purchasehistoryof a given A set of itemsboughtby Books,diaryproducts,
customer a customerat timet CDs.etc
Web Data Browsingactivityof a Thecollectionof files Homepage,index
particular
Webvisitor viewedby a Webvisitor page,contactinfo,etc
aftera singlemouseclick
Eventdata Historyof eventsgenerated Eventstriggeredby a Typesof alarms
by a givensensor sensorat timet generatedby sensors
Genome DNAsequenceof a An elementof the DNA BasesA,T,G,C
sequences particular
species sequence

OrdinalAttribute

Figure7.4. Examples
of elements
andeventsin sequence
datasets,

is a subsequence of s : ("r"2...s,) if there exist integersL a jt < jz <... a

jrn 1n such that h e sj,tz e sj",...,t^ e sj*. If t is a subsequence of
s, then we say that t is contained in s. The following table gives examples
illustrating the idea of subsequencesfor various sequences.

Sequence, s Sequence,f Is f a subsequenceof s?

< 1 2 , 4 13,5,6 8 2 | { 3 , 6 1{ 8 } > Yes
< 1 2 , 4 13'5,6 8 2[.{8 Yes
< { 1 , 2 } 3,41> tt 12 No
< { 2 , 4 12 , 4 1 2 , 5 2 11 4 Yes

7.4.2 Sequential Pattern Discovery

Let D be a data set that contains one or more data sequences. The term
data sequence refers to an ordered list of events associated with a single data
object. For example, the data set shown in Figure 7.3 contains three data
sequences,one for each object A, B, and C.
The support of a sequence s is the fraction of all data sequencesthat
contain s. If the support for s is greater than or equal to a user-specified
432 Chapter 7 Association Analysis: Advanced Concepts

Obiect Timestamp Events

A 1 1 . 2 ,4 Minsup = 50o/o
A 2 2,3
A 3 5 Examples of Sequential Patterns:
B 1 1,2
B 2 2,3,4 <{1,2}> s=607o
<{2,3}> s=607o
c 1 1,2
<{2,4\> s=807o
c 2 2,3,4 <{3}{5}> s=807"
c 3 2,4,5 <{1}{2}> s=807"
D 1 2 <{2){2}> s=607"
D 2 3,4 <{1i{2,3}> s=607o
D 3 4,5 <{2) {2,3)> s=607o
E 1 1,3 <{1,2} {2,3}> s=607o
E 2 2,4,5

pafterns
Figure7.5. Sequential derived fivedatasequences.
froma datasetthatcontains

threshold m'insup, then s is declared to be a sequential pattern (or frequent

sequence).
Definition 7.1 (Sequential Pattern Discovery). Given a sequencedata
set D and a user-specifiedminimum support threshold minsup, the task of
sequential pattern discovery is to find all sequenceswith support ) minsup.
Figure 7.5 illustrates an example of a data set that contains five data
sequences.The support for the sequence< {1}{2} ) is equal to 80% becauseit
occurs in four of the five data sequences(every object except for D). Assuming
that the minimum support threshold is 50%, any sequencethat appears in at
least three data sequencesis considered to be a sequential pattern. Examples
of sequential patterns extracted from the given data set include <{1}{2}>,
< { 1 , 2 } >,<{2 ,3 }>,<u ,2 }{2 ,3 }>,etc.
Sequential pattern discovery is a computationally challenging task because
there are exponentially many sequencescontained in a given data sequence.
For example, the data sequence<{a,b} {c,d,e} {f} {g,h,i}> contains sequences
such as <{a} {c,d} {f} {S}>, <{c,d,e}>, <{b} {s}>, etc. It can be easily
shown that the total number of k-sequencespresent in a data sequence with
n events ir (?) A data sequencewith nine events therefore contains

0.(3) (3) ++ :29-1:511

distinct sequences.
 7.4 Sequential Patterns 433

A brute-force approach for generating sequential patterns is to enumerate

all possible sequencesand count their respective supports. Given a collection
of n events, candidate l-sequences are generated first, followed by candidate
2-sequences,candidate 3-sequences,and so on:

l-sequences:1ir ), f i2 ), . . ., 1 xn )
2-sequences:< {h,iz} >, < {rir,ze}), . . ., 1 {in_t,'in} },
< {zr}{ir} >, < {it}{iz} ),..., < {i-_t}{i-} >
3-sequences:I {i1,'i2,is}>, < {ir,,iz,iq}}, ..., 1 {,h,iz}{ir} ), ...,
< { i t } { z r ,i z } > , . . . , < { z t } { i r } { r t } > , . . . , < { i " } { i . } { i ^ } >

Notice that the number of candidate sequencesis substantially larger than

the number of candidate itemsets. There are two reasons for the additional
number of candidates:

1. An item can appear at most once in an itemset, but an event can appear
more than once in a sequence.Given a pair of items, ir and i2, only one
candidate 2-itemset, {h,iz}, can be generated. On the other hand, there
are many candidate 2-sequences, such as ( {i1,iz} >, < {it}{iz} >,
< {iz}{it} ), and 1{h,it} >, that can be generated.

2. Order matters in sequences,but not for itemsets. For example, {1, 2} and
{2,1} refersto the sameitemset,whereas< {it}{iz} ) and < {iz}{i} >
correspond to different sequences,and thus must be generated separately.

The Apriori principle holds for sequential data becauseany data sequence
that contains a particular k-sequencemust also contain all of its (k - 1)-
subsequences.An Apri,ori,-Iike algorithm can be developed to extract sequen-
tial patterns from a sequencedata set. The basic structure of the algorithm
is shown in Algorithm 7.1.
Notice that the structure of the algorithm is almost identical to Algorithm
6.1 presented in the previous chapter. The algorithm would iteratively gen-
erate new candidate k-sequences,prune candidates whose (k - l)-sequences
are infrequent, and then count the supports of the remaining candidates to
identify the sequential patterns. The detailed aspects of these steps are given
next.

Candidate Generation A pair of frequent (k - 1)-sequencesare merged to

produce a candidate k-sequence.To avoid generating duplicate candidates, re-
call that the traditional Apriori algorithm merges a pair of frequent k-itemsets
only if their first k - 1 items are identical. A similar approach can be used
434 Chapter 7 Association Analysis: Advanced Concepts

Algorithm 7.L Apriora-like algorithm for sequential pattern discovery.

I: k: l.
gI#D } mi,nsupl.
2: F1":{ilielA {Find all frequent1-subsequences.}
3: repeat
4: k:k+I.
5: Cn : apriori-gen(Fs-1). {Generatecandidatek-subsequences.}
6: for eachdata sequencet €T do
7: Ct : subsequence(C6, t). {Identify all candidatescontainedin t.}
8: for each candidate k-subsequencec € Ct do
9: o(c) : o(c) + 1. {Increment the support count'}
10: end for
11: end for
t2: Fn: {cl c€CuA sfP > mi'nsup}. {Extract the frequentlc-subsequences'}
13: until Fn:A
14:Answer:UFn.

for sequences.The criteria for merging sequencesare stated in the form of the
following procedure.

Sequence Merging Procedure

A sequence s(1) is merged with another sequence s(2) only if the subsequence
obtained by dropping the first event in s(1) is identical to the subsequence
obtained by dropping the last event in s(2). The resulting candidate is the
sequence5(1), concatenated with the last event from s(2). The last event from
s(2) can either be merged into the same element as the last event in s(1) or
different elements depending on the following conditions:

1. If the last two events in s(2) belong to the same element, then the last event
in s(2) is part of the last element in s(1) in the merged sequence.

2. If.the last two events in s(2) belong to different elements, then the last event
in s(2) becomes a separate element appended to the end of s(1) in the
merged sequence.

Figure 7.6 illustrates examplesof candidate 4-sequencesobtained by merg-

ing pairs of frequent 3-sequences.The first candidate ({t}{Z}{3}{4}) is ob'
tained by merging ((1X2X3)) with ((2)(3)(a)). Since events 3 and 4 belong
to different elements of the second sequence,they also belong to separate ele-
ments in the merged sequence.On the other hand, merging ({1}t5}{3}) with
({5}{3,4}) produces the candidate 4-sequence({1}{5i{3,4}). In this case,
 7.4 Sequential Patterns 435

Frequent
3-sequences

< (1)(2) (3)> Candidate

< ( 1 ) ( 25 ) > Generation
< ( 1 ) ( 5 ) ( 3 )>
< (2) (3)(4)> < (1)(2)(3)(4)> Candidate
<(25)(3)> < (1)(2 5) (3)> Pruning
< (3) (4) (5) > < (1)(5)(3 4) >
<(5)(34)> . (2)(3)(4)(5)>
. (1) (2 5) (3) >
<(25)(34)'

Figure7,6, Example
of thecandidate andpruning
generation pattern
stepsof a sequential mining
algorithm.

since events 3 and 4 belong to the same element of the second sequence,they
are combined into the same element in the merged sequence.Finally, the se-
quences({1}{2}{3}) and ({t}{2,5}) do not have to be mergedbecauseremov-
ing the fi.rst event from the first sequencedoes not give the same subsequence
as removing the last event from the secondsequence.Although ({1}{2,5}t3})
is a viable candidate, it is generated by merging a different pair of sequences,
({1}{2,5}) and ({2,5}{3}). This example shows that the sequencemerging
procedure is complete; i.e., it will not miss any viable candidate, while at the
same time, it avoids generating duplicate candidate sequences.

Candidate Pruning A candidate k-sequenceis pruned if at least one of its

-
(k l)-sequencesis infrequent. For example,suppose({1}{2}{3}{+}) is a can-
didate 4-sequence.We needto checkwhether ({1}{2}{4}) and ({t}{3}{a}) are
frequent 3-sequences.Sinceboth are infrequent, the candidate ({t}{Z}{3}{4})
can be eliminated. Readers should be able to verify that the only candi-
date 4-sequencethat survives the candidate pruning step in Figure 7.6 is
({1}{2 5}{3})

Support Counting During support counting, the algorithm will enumer-

ate all candidate k-sequencesbelonging to a particular data sequence. The
support of these candidates will be incremented. After counting their sup-
ports, the algorithm may identify the frequent k-sequences and may discard
all candidates whose support counts are less than the m'insup threshold.
436 Chapter 7 Association Analysis: Advanced Concepts

u(s;*r)-l(s;)<= maxgap
l(s;+r)-u(b;)>mingap windowsize
WS
e

Sequence:

u(sn) - l(s1) <= tttdXSP?tl

Timewindow(w) for eachelementis characterized by [,u]
where| : earliesttime of occurrenceof an eventin w
u : latesttime of occurrenceof an eventin w

Figure7.7.Timing
constraints pattern.
of a sequential

7.4.3 Timing Constraints

This section presentsa sequentialpattern formulation where timing constraints
are imposed on the events and elements of a pattern. To motivate the need
for timing constraints, consider the following sequenceof coursestaken by two
students who enrolled in a data mining class:

Student A: ( {Statistics} {Database Systems} {Data Mining} ).

Student B: ( {Database Systems} {Statistics} {Data Mining} ).

The sequentialpattern of interest is ( {Statistics, Database Systems} {Data

Mining) ), which means that students who are enrolled in the data mining
class must have previously taken a course in statistics and database systems.
Clearly, the pattern is supported by both students even though they do not
take statistics and database systems at the same time. In contrast, a student
who took a statistics course ten years earlier should not be considered as
supporting the pattern becausethe time gap between the coursesis too long.
Becausethe formulation presentedin the previous section does not incorporate
these timing constraints, a new sequential pattern definition is needed.
Figure 7.7 illustrates some of the timing constraints that can be imposed
on a pattern. The definition of these constraints and the impact they have on
sequential pattern discovery algorithms will be discussedin the next sections.
Note that each element of the sequential pattern is associated with a time
window [l,z], where I is the earliest occurrence of an event within the time
window and u is the latest occurrence of an event within the time window.
 7.4 Sequential Patterns 437

The maxspan Constraint

The marspan constraint specifies the maximum allowed time difference be-
tween the latest and the earliest occurrencesof events in the entire sequence.
For example, suppose the following data sequences contain events that oc-
cur at consecutive time stamps (1, 2, 3, . ..). Assuming that marspan : 3,
the following table contains sequential patterns that are supported and not
supported by a given data sequence.

Data Secuence.s Sequential Pattern. t Doess support f'l

1 , 3 1 3,4 4l t5 6,7 8 < { e {4 fes
1 , 3 ) 3,4 4 11 5 6,7 8 <{3 {6 fes
1 , 3 1 3,4 4j {5i {6,7 8 < { 1,3} 6l> No

In general, the longer the marspan, the more likely it is to detect a pattern
in a data sequence.However, a longer marspan can also capture spurious pat-
terns becauseit increasesthe chancefor two unrelated events to be temporally
related. In addition, the pattern may involve events that are already obsolete.
The marspan constraint affects the support counting step of sequential
pattern discovery algorithms. As shown in the preceding examples,some data
sequencesno longer support a candidate pattern when the marspan constraint
is imposed. If we simply apply Algorithm 7.7., the support counts for some
patterns may be overestimated. To avoid this problem, the algorithm must be
modified to ignore cases where the interval between the first and last occur-
rences of events in a given pattern is greater t'han marspan.

The mingap and maxgap Constraints

Timing constraints can also be specified to restrict the time difference be-
tween two consecutive elements of a sequence. If the maximum time difference
(margap) is one week, then events in one element must occur within a week's
time of the events occurring in the previous element. If the minimum time dif-
ference (mi,ngap) is zero, then events in one element must occur immediately
after the events occurring in the previous element. The following table shows
examples of patterns that pass or fail the margap and mingap constraints,
assuming that margap: 3 and mi,ngap: t.

Data Sequence,s Sequential Pattern, f rnargap rnl.ngap

1 , 3 3,4 4l l5 6,7 8 3 o Pass Pass
1 , 3 3,4 4l {5 6,7 8 o 8 Pass Fail
1 , 3 314 4t 15 6,7 8 1,3 6l> Fail Pass
1 , 3 314 4 f t 5 6,7 8 1 3 {8}> Fail Fail
438 Chapter 7 Association Analysis: Advanced Concepts

As with marspan) these constraints will affect the support counting step
of sequential pattern discovery algorithms because some data sequencesno
longer support a candidate pattern when mingap and rnargap constraints are
present. These algorithms must be modified to ensure that the timing con-
straints are not violated when counting the support of a pattern. Otherwise,
some infrequent sequencesmay mistakenly be declared as frequent patterns.
A side effect of using the margap constraint is that the Apri,ori principle
might be violated. To illustrate this, consider the data set shown in Figure
7.5. Without mingap or rnargap constraints, the support for ({Z}{S}) and
({2}{3}{5}) are both equal to 60%. However,if mi,ngap: 0 and margap: L,
then the support for ({Z}{5}) reducesto 40To,while the support for ({Z}{a}{S})
is still 60%. In other words, support has increasedwhen the number of events
in a sequenceincreases-which contradicts the Apri,ori, principle. The viola-
tion occurs becausethe object D does not support the pattern ({2}{5}) since
the time gap between events 2 and 5 is greater than rnargap. This problem
can be avoided by using the concept of a contiguous subsequence.
Definition 7.2 (Corftiguous Subsequence). A sequences is a contiguous
subsequenceof w - \e1e2...ek) if any one of the following conditions hold:

1. s is obtained from u-rafter deleting an event from either €1 or ep,

2. s is obtained from tr after deleting an event from any element ei e w

that contains at least two events, or

3. s is a contiguous subsequenceof f and t is a contiguous subsequenceof

w.

The following examples illustrate the concept of a contiguous subsequence:

Data Sequence,s Sequential Pattern, I Is f a contiguous

subsequenceof s?
1) {2,3}> rt 12 Yes
1,2){2} {3} > 7t 12 YES
< t 3 , 4 ]t 1 , 2 ){ 2 , 3 }t 4 } > 1l 12 fes
<{1} {3} {2} > r\ { 2 No
< { 1 , 2 }{ 1 } { 3 } { 2 } > rl {2 No

Using the concept of contiguous subsequences)the Apri.ori, principle can

be modified to handle n'Largap constraints in the following way.
Definition 7.3 (Modified Apri.orz Principle). If a k-sequenceis frequent,
then all of its contiguous k - l-subsequencesmust also be frequent.
 7.4 Sequential Patterns 439

The modified Apri,ori, principle can be applied to the sequential pattern

discovery algorithm with minor modifications. During candidate pruning, not
all k-sequencesneed to be verified since some of them may violate the margap
constraint. For example,if margap:1, it is not necessaryto check whether
the subsequence({1}{2,3}{5}) of the candidate ({1}{2,3}{4}{5}) is frequent
since the time difference between elements {2,3} and {5} is greater than one
time unit. Instead,only the contiguoussubsequences of ({1}{2,3}{a}{5}) need
to be examined. These subsequences include ({1}{2,3}{4}), ({2,3}{4}{5}),
({1}{2}{4}{5}), and (t1}{3}{4}{5}).

The Window Size Constraint

Finally, events within an element s7 do not have to occur at the same time. A
window size threshold (tr.'s)can be defined to specify the maximum allowed
time difference between the latest and earliest occurrences of events in any
element of a sequential pattern. A window size of 0 means all events in the
same element of a pattern must occur simultaneously.
The following example uses u)s : 2 to determine whether a data se-
quence supports a given sequence(assuming mingap : 0, margap : 3, and
marspan: -).

Data Sequence,s Sequential Pattern, f Does s support f'l

1,3 3,4 4l {5 6,7 8 3,4 o Yes
1,3 3,4 4I l5 6,7 8 4,6 8 Yes
1,3 3,4 4 f l o 6 , 7 8 3,4,6 l8t > No
1,3 3,4 4 1 . { 5 6 , 7 8 1,3,4) 6,7,8)> No

In the last example, although the pattern ({1,3,4} {6,7,8}) satisfiesthe win-
dow size constraint, it violates the margap constraint becausethe maximum
time difference between events in the two elements is 5 units. The window
size constraint also affects the support counting step of sequential pattern dis-
covery algorithms. If Algorithm 7.I is applied without imposing the window
size constraint, the support counts for some of the candidate patterns might
be underestimated, and thus some interesting patterns may be lost.

7.4.4 Alternative Counting Schemes

There are several methods available for counting the support of a candidate
k-sequencefrom a database of sequences.For illustrative purposes, consider
the problem of counting the support for sequenc" ({p}{q}), as shown in Figure
7.8. Assume that ?rs : 0, mingap : 0, margap: 1, and marspan :2.
44O Chapter 7 Association Analysis: Advanced Concepts

Object'sTimeline
ppp (p)(q)
Sequence:
ppqqqqq
t_-_t-_______.1__ ]_
, ---'i---------'t-- (Method,Count)
i1 i2 i3 i4 i5 i6 i7
#iiii COBJ

J cwlN

Hi
itH | "r'**'*

CDISTO 8

CDIST 5

Figure7.8. Comparing
different
support methods.
counting

COBJ: One occurrenceper object.

This method looks for at least one occurrence of a given sequence in
an object's timeline. In Figure 7.8, even though the sequence((p)(q))
appears several times in the object's timeline, it is counted only once-
with p occurring at t:1 and q occuring at f : 3.

CWIN: One occurrence per sliding window.

In this approach, a sliding time window of fixed length (marspan) is
moved across an object's timeline, one unit at a time. The support
count is incremented each time the sequenceis encounteredin the sliding
window. In Figure 7.8, the sequence({p}{q}) is observedsix times using
this method.

CMINWIN: Number of minimal windows of occurrence.

A minimal window of occurrence is the smallest window in which the
sequenceoccurs given the timing constraints. In other words, a minimal
 7.4 Sequential Patterns 44'l'

window is the time interval such that the sequenceoccurs in that time
interval, but it does not occur in any of the proper subintervals of it. This
definition can be considered as a restrictive version of CWIN, because
its effect is to shrink and collapsesome of the windows that are counted
by CWIN. For example, sequenc" ({p}{q}) has four minimal window
-- 3, q:
occurrences:(1) the pair (p: t :2, q'. t :3)' (2) the pair (p'. t
t:4), (3) the pair (p: t:5, Q:t:6), and (a) the pair (p: t:6, q:
t:7). The occurrenceof eventp at t:1 and event q at t :3 is not a
minimal window occurrence because it contains a smaller window with
(p: t:2, g: t: 3), which is indeed a minimal window of occurrence'

o CDIST-O: Distinct occurrences with possibility of event-timestamp

overlap.
A distinct occurrence of a sequenceis defined to be the set of event-
timestamp pairs such that there has to be at least one new event-
timestamp pair that is different from a previously counted occurrence.
Counting all such distinct occurrencesresults in the CDIST-O method.
If the occurrencetime of eventsp and q is denoted as a tuple (t(p),t(q)),
then this method yields eight distinct occurrencesof sequence({p}{q})
at times (1,3), (2,3), (2,4), (3,4), (3,5), (5,6), (5,7), and (6,7).

o CDIST: Distinct occurrenceswith no event-timestamp overlap allowed.

In CDIST-O above, two occurrences of a sequencewere allowed to have
overlapping event-timestamp pairs, e.g., the overlap between (1,3) and
(2,3). In the CDIST method, no overlap is allowed. Effectively, when an
event-timestamp pair is considered for counting, it is marked as used and
is never used again for subsequent counting of the same sequence. As
an example, there are five distinct, non-overlapping occurrencesof the
sequence({p}tq}) in the diagram shown in Figure 7.8. These occurrences
happen at times (7,3), (2,4), (3,5), (5,6), and (6,7). Observethat these
occurrencesare subsets of the occurrencesobserved in CDIST-O.

One final point regarding the counting methods is the need to determine the
baselinefor computing the support measufe. For frequent itemset mining, the
baseline is given by the total number of transactions. For sequential pattern
mining, the baseline depends on the counting method used. For the COBJ
method, the total number of objects in the input data can be used as the
baseline. For the CWIN and CMINWIN methods, the baselineis given by the
sum of the number of time windows possible in all objects. For methods such
as CDIST and CDIST_O, the baseline is given by the sum of the number of
distinct timestamps present in the input data of each object.
442 Chapter 7 Association Analysis: Advanced Concepts

7.5 Subgraph Patterns

This section describesthe application of associationanalysis methods to more
complex entities beyond itemsets and sequences.Examples include chemical
compounds, 3-D protein structures, network topologies, and tree structured
XML documents. These entities can be modeled using a graph representation,
as shown in Table 7.7.

Table
7.7.Graph
representation
ofentities
invarious
application
domains.

Application Graphs Vertices trdges

Web mining Web browsing patterns Web paees Hyperlink between pages
Computational Structure of chemical Atoms or Bond between atoms or
chemistry compounds tons rons
Network computing Computer networks Computers and Interconnection between
servers machines
Semantic Web Collection of XML XML elements Parent-child relationship
documents between elements
Bioinformatics Protein structures Amino acids Contact residue

A useful data mining task to perform on this type of data is to derive a

set of common substructures among the collection of graphs. Such a task is
known as frequent subgraph mining. A potential application of frequent
subgraph mining can be seenin the context of computational chemistry. Each
year, new chemical compounds are designedfor the developmentof pharmaceu-
tical drugs, pesticides, fertilizers, etc. Although the structure of a compound
is known to play a major role in determining its chemical properties, it is dif-
ficult to establish their exact relationship. Flequent subgraph mining can aid
this undertaking by identifying the substructures commonly associatedwith
certain properties of known compounds. Such information can help scientists
to develop new chemical compounds that have certain desired properties.
This section presents a methodology for applying association analysis to
graph-based data. The section begins with a review of some of the basic
graph-related conceptsand definitions. The frequent subgraph mining problem
is then introduced, followed by a description of how the traditional Apri,ori,
algorithm can be extended to discover such patterns.
 7.5 Subgraph Patterns 443

7.5J Graphs and Subgraphs

A graph is a data structure that can be used to represent the relationships

among a set of entities. Mathematically, a graph is composedof a vertex set V
and a set of edges,E connecting between pairs of vertices. Each edge is denoted
by a vertex pair (r'i, u7), where ui,ui € I/. A label l(u,;) can be assignedto each
vertex u, representing the name of an entity. Similarly each edge (ut,ui) can
also be associatedwith a label l(ua,u7) describing the relationship between a
pair of entities. Table 7.7 shows the vertices and edgesassociated with different
types of graphs. For example, in a Web graph, the vertices correspond to Web
pages and the edges represent the hyperlinks between Web pages.

Definition 7.4 (Subgraph). A graph G' : (V',,E/) is a subgraph of another

graph G : (V,E) if its vertex set Vt is a subset of V and its edge set .E' is a
subset of ,8. The subgraph relationship is denoted as Gt es G.
Figure 7.9 shows a graph that contains 6 vertices and 11 edgesalong with
one of its possible subgraphs. The subgraph, which is shown in Figure 7.9(b),
contains only 4 of the 6 vertices and 4 of the 11 edgesin the original graph.

(a) Labeledgraph. (b)Subgraph.

Figure
7.9,Example
ofa subgraph.

Definition 7.5 (Support). Given a collection of graphs Q, the support for

a subgraph g is defined as the fraction of all graphs that contain g as its
subgraph,i.e.:
s(g) : l { G , l g e s G eG
, ,i,e8}l (7.2)
t0l
444 Chapter 7 Association Analysis: Advanced Concepts

Subgraph9.,
aO'-{'e
= 80%
support

Subgraphg,
af,J___ad
/t
Oe
= 60%
support

Graph Data Set support= 40%

Figure7.10.Computing
thesupport froma setofgraphs.
of a subgraph

Example 7.2. Consider the five graphs, G1 through Gr, shown in Figure
7.10. The graph pr shown on the top right-hand diagram is a subgraph of G1,
Gs, G+, and Gs. Therefore, s(gr) : 415 : 80%. Similarly, we can show that
s(gz) :60% because92 is a subgraph of G1, G2, and G3, while s(gs) : lO%
because93 is a subgraph of G1 and G3. I

7.5.2 FYequent Subgraph Mining

This section presentsa formal definition of the frequent subgraph mining prob-
Iem and illustrates the complexity of this task.
Definition 7.6 (Flequent Subgraph Mining). Given a set of graphs f
and a support threshold, m'insup, the goal of frequent subgraph mining is to
find all subgraphs g such that s(9) > rnxnsup.
While this formulation is generally applicable to any type of graph, the
discussion presented in this chapter focuses primarily on undirected, con-
nected graphs. The definitions of these graphs are given below:
1. A graph is connected ifthere exists a path between every pair ofvertices
in the graph, in which a path is a sequenceof vertic€s 1u1u2...ute )
 7.5 Subgraph Patterns 445

such that there is an edge connecting between every pair of adjacent

vertices (ro,r*r) in the sequence.

2. A graph is undirected if it contains only undirected edges. An edge

(ut,ui) is undirected if it is indistinguishablefrom (ui,ut).

Methods for handling other types of subgraphs (directed or disconnected) are

left as an exerciseto the readers (seeExercise 15 on page 482).
Mining frequent subgraphs is a computationally expensivetask becauseof
the exponential scale of the search space. To illustrate the complexity of this
task, consider a data set that contains d entities. In frequent itemset mining,
each entity is an item and the size of the search space to be explored is 2d,
which is the number of candidate itemsets that can be generated. In frequent
subgraph mining, each entity is a vertex and can have up to d - 1 edges to
other vertices. Assuming that the vertex labels are unique, the total number
of subgraphs is
d ,,u

I(;) x 2i(i-r)/2 )

where (f) is tne number of ways to choose i vertices to form a subgraph and
2i(i-L)12is the maximum number of edgesbetween vertices. Table 7.8 compares
the number of itemsets and subgraphs for different values of d.

Table7,8.A comparison number

between ofitemsets fordifferent
andsubgraphs d.
dimensionality,
Number of entities, d I 2 J
A ( f) 7 8
Number of itemsets 2 ,4 8 16 32 o4t28 256
Number of subgraphs 2 18 113 1,450 40,069 2,350,602 28,619,25L3

The number of candidate subgraphs is actually much smaller becausethe

numbers given in Table 7.8 include subgraphs that are disconnected. Discon-
nected subgraphs are usually ignored because they are not as interesting as
connected subgraphs.
A brute-force method for doing this is to generate all connected subgraphs
as candidates and count their respective supports. For example, consider the
graphs shown in Figure 7.11(a). Assuming that the vertex labels are chosen
from the set {a, b} and the edge labels are chosen from the set {p, q}, the list
of connectedsubgraphs with one vertex up to three vertices is shown in Figure
7.11(b). The number of candidate subgraphs is considerably larger than the
446 Chapter 7 Association Analysis: Advanced Concepts

G1 G2 G4

(a) Exampleof a graphdataset

k=1 @ @
Dn Q^
€_fi9 €H9
D^ o^
€HD €H9
o^ o^
QHD (9HU
k=3

c+.4
\I' €J

(b) Listof connected

subgraphs.

Figure7.'11,Brute{orce
method
formining
frequent
subgraphs.

number of candidate itemsets in traditional association rule minine for the

following reasons:

1. An item can appear at most once in an itemset, whereas a vertex label

can appear more than once in a graph.

2. The same pair of vertex labels can have multiple choicesof edge labels.

Given the large number of candidate subgraphs, a brute-force method may

break down even for moderately sized graphs.
 7.5 Subgraph Patterns 447

G1 G2 G3 G4

(a,b,p) (a,b,q) (a,b,r) (b,c,p) ( b , c , q ) (b,c,r) (d,e,0

G1 1 0 0 0 0 1 0
G2 1 0 0 0 0 0 0
G3 0 0 1 1 0 0 0
G4 0 0 0 0 0 0 0

Figure
7,12,Mapping ofgraph
acollection intomarket
structures basket
transactions.

7.5.3 Apriori-Iike Method

This section examines how an Apri.ori,-like algorithm can be developed for
finding frequent subgraphs.

Data Tlansformation

One possible approach is to transform each graph into a transaction-like for-

mat so that existing algorithms such as Apri,ori, can be applied. Figure 7.12
illustrates how to transform a collection of graphs into its equivalent market
basket representation. In this representation, each combination of edge Ia-
bel l(e) with its correspondingvertex labels, (I(ut),l(ui)), is mapped into an
"item." The width of the "transaction" is given by the number of edgesin the
graph. Despite its simplicity, this approach works only if every edge in a graph
has a unique combination of vertex and edge labels. Otherwise, such graphs
cannot be accurately modeled using this representation.

General Structure of the fYequent Subgraph Mining Algorithm

An Apriori,-like algorithm for mining frequent subgraphs consists of the fol-

lowing steps:

1. Candidate generation, which is the process of merging pairs of fre-

quent (k - l)-subgraphs to obtain a candidate k-subgraph.
448 Chapter 7 Association Analysis: Advanced Concepts

2. Candidate pruning, which is the process of discarding all candidate

k-subgraphs that contain infrequent (k - 1)-subgraphs.

3. Support counting, which is the process of counting the number of

graphs in Q that contain each candidate.

4. Candidate elimination, which discards all candidate subgraphs whose

support counts are less than minsup.

The specificdetails of these steps are discussedin the remainder of this section.

7.5.4 Candidate Generation

During candidate generation, a pair of frequent (k - l)-subgraphs are merged
to form a candidate k-subgraph. The first question is how to define k, the size
of a subgraph. In the example shown in Figure 7.7I, k refers to the number
of vertices in the graph. This approach of iteratively expanding a subgraph
by adding an extra vertex is known as vertex growing. Alternatively, k may
refer to the number of edges in the graph. This approach of adding an extra
edge to the existing subgraphs is known as edge growing.
To avoid generating duplicate candidates, we may impose an additional
condition for merging, that the two (k - 1)-subgraphs must share a common
(k-2)-subgraph. The common (k-2)-subgraph is known as their core. Below,
we briefly describethe candidate generation procedure for both vertex-growing
and edge-growingstrategies.

Candidate Generation via Vertex Growing

Vertex growing is the processof generating a new candidate by adding a new

vertex into an existing frequent subgraph. Before describing this approach,
let us first consider the adjacency matrix representation of a graph. Each
entry M(i,j) in the matrix contains either the label of the edge connecting
between the vertices ui and uji or zero, if there is no edge between them.
The vertex-growing approach can be viewed as the process of generating a
k x k adjacency matrix by combining a pair of (k - 1) x (k - 1) adjacency
matrices, as illustrated in Figure 7.13. G1 and G2 are two graphs whose
adjacency matrices are given by M(GI) and M(G2), respectively. The core
for the graphs is indicated by dashed lines in the diagram. The procedure for
generating candidate subgraphs via vertex growing is presented next.
 7.5 Subgraph Patterns 449

t----__--_----l

+
G3 = merge(G1,G2)

opp q0
por 00
MGg= pr0 Or
900 0?
00r ?0

strategy.
Figure7.13. Vertex-growing

Subgraph Merging Procedure via Vertex Growing

An adjacency matrix 141G)i" merged with another matrix llit\z) il the submatrices
identical
obtained by removing the last row and last column of MG) and MQ) are
with the Iast
to each other. The resulting matrix is the matrix ry'(l), appended
matrix are
row and last column of matrix M(2). T]he remaining entries of the new
vertices'
either zero or replaced by all valid edge labels connecting the pair of

The resulting graph contains one or two edges more than the original
graphs. In Figure 7.13, both G1 and G2 contain four vertices and four edges'
Aft", -"rging, the resulting graph G3 has flve vertices. The number of edges
e
in G3 depends on whether the vertices d and € are connected. If d and
are disconnected,then G3 has five edgesand the corresponding matrix entry
(d, e)
for (d, e) \s zero. Otherwise, G3 has six edges and the matrix entry for
is
corresponds to the label for the newly created edge. since the edge label
unknown, we need to consid.erall possibleedge labels fot (d,e), thus increasing
the number of candidate subgraphs substantially'

Candidate Generation via Edge Growing

Edge growing inserts a new edge to an existing frequent subgraph during

not
candidate generation. unlike vertex growing, the resulting subgraph does
45O Chapter 7 Association Analysis: Advanced Concepts

,/
G 3 = m e r g e( G l , G 2 )

G4 = mslgs (G1' G2)

Figure7.14.Edge-growing
strategy.

necessarilyincreasethe number of vertices in the original graphs. Figure 7.14

shows two possible candidate subgraphs obtained by merging G1 and G2 via
the edge-growing strategy. The first candidate subgraph, G3, has one extra
vertex, while the second candidate subgraph, G4, has the same number of
vertices as the original graphs. The core for the graphs is indicated by dashed
lines in the diagram.
The procedure for generating candidate subgraphs via edge growing can
be summarized as follows.

Subgraph Merging Procedure via Edge Growing

A frequent subgraph 9(1) is merged with another frequent subgraph g(2) only if
the subgraph obtained by removing an edge from 9(r) is topologically equivalent
to the subgraph obtained by removing an edge from gQ). After merging, the
resulting candidate is the subgraph 9(r), appended with the extra edge from gQ)

The graphs to be merged may contain several vertices that are topolog-
ically equivalent to each other. To illustrate the concept of topologically
equivalent vertices, consider the graphs shown in Figure 7.15. The graph G1
contains four vertices with identical vertex labels, "a." lf a new edge is at-
 7.5 Subgraph Patterns 451

G1 G2 G3

Figure7.15.lllustration
oftopologically vertices.
equivalent

tached to any one of the four vertices, the resulting graph will look the same.
The vertices in G1 are therefore topologically equivalent to each other.
The graph G2 has two pairs of topologically equivalent vertices, u1 with
u4 and uz with u3, even though the vertex and edge labels are identical. It is
easy to seethat u1 is not topologically equivalent to u2 becausethe number of
edgesincident on the vertices is different. Therefore, attaching a new edge to
u1 results in a different graph than attaching the same edge to u2. Meanwhile,
the graph G3 does not have any topologically equivalent vertices. While a1
andu4 have the same vertex labels and number of incident edges,attaching a
new edge to u1 results in a different graph than attaching the same edge to u4.
The notion of topologically equivalent vertices can help us understand why
multiple candidate subgraphs can be generatedduring edge growing. Consider
the (k - 1)-subgraphs G1 and G2 shown in Figure 7.16. To simplify the
notation, their core, which contains k - 2 common edges between the two
graphs, is drawn as a rectangular box. The remaining edge in Gl that is not
included in the core is shown as a dangling edge connecting the vertices a and
b. Similarly, the remaining edge in G2 that is not part of the core is shown as
a dangling edge connecting vertices c and d. Although the cores for G1 and
G2 are identical, a and c may or may not be topologically equivalent to each

Figure7.16.General formerging
approach viaedgegrowing.
a pairofsubgraphs
452 Chapter 7 Association Analysis: Advanced Concepts

G3= Merge(c1, c2) G3 = Merge(G1,G2) G3 = Merge(G1,c2)

(a)a+candbtd (b)a=candb*d

G3 = Merge(G1,G2) G3 = Merge(G1,G2)

(c)a+candb=d

G3 = Merge(c1, c2)

G3 = Merge(G1,G2) G3 = Merge(G1,G2)

(d)a=candb=d

Figure7.17.Candidate generated
subgraphs viaedgegrowing.

other. If a and c are topologically equivalent, we denote them as a : c. For

vertices outside the core, we denote them as b: dif their labels are identical.
The following rule of thumb can be used to determine the candidate sub-
graphs obtained during candidate generation:

I. If a I c and b + d, then there is only one possible resulting subgraph,

as shown in Figure 7.17(a).

2. If a: c but b + d, then there are two possible resulting subgraphs, as

shown in Figure 7.17(b).
 7.5 Subgraph Patterns 453

Figure7.18.Multiplicity
of candidates
during generation.
candidate

3. If a I c btft b : d, then there are two possible resulting subgraphs, as

shown in Figure 7.17(c).

4. If a: c and b: d, then there are three possible resulting subgraphs, as

shown in Figure 7.L7(d).

Multiple candidate subgraphs can also be generated when there is more

than one core associatedwith the pair of (k - 1)-subgraphs,as shown in Figure
7.18. The shaded vertices correspond to those vertices whose edges form a
core during the merging operation. Each core may lead to a different set of
candidate subgraphs. In principle, if a pair of frequent (k - l)-subgraphs is
merged, there can be at most k-2 cores,each of which is obtained by removing
an edge from one of the merged graphs. Although the edge-growingprocedure
can produce multiple candidate subgraphs,the number of candidate subgraphs
tends to be smaller than those produced by the vertex-growing strategy.

7.5.5 Candidate Pruning

After the candidate k-subgraphs are generated, the candidates whose (k -

1)-subgraphs are infrequent need to be pruned. The pruning step can be
performed by successivelyremoving an edge from the candidate k-subgraph
and checking whether the corresponding (k - l)-subgraph is connected and
frequent. If not, the candidate k-subgraph can be discarded.
To check whether the (k - l)-subgraph is frequent, it should be matched
against other frequent (k - 1)-subgraphs. Determining whether two graphs are
topologically equivalent (or isomorphic) is known as the graph isomorphism
problem. To illustrate the difficulty of solving the graph isomorphism problem,
454 Chapter 7 Association Analysis: Advanced Concepts

Figure7.19.Graphisomorphism

consider the two graphs shown in Figure 7.19. Even though both graphs look
different, they are actually isomorphic to each other becausethere is a one-to-
one mapping between vertices in both graphs.

Handling Graph Isomorphism

A standard approach for handling the graph isomorphism problem is to map

each graph into a unique string representation known as its code or canonical
label. A canonical label has the property that if two graphs are isomorphic,
then their codes must be the same. This property allows us to test for graph
isomorphism by comparing the canonical labels of the graphs.
The first step toward constructing the canonical label of a graph is to
find an adjacency matrix representation for the graph. Figure 7.20 shows an

0p pq
p0 r0
M=
pr 00
q0 00

Figure7.20.Adjacency of a graph.
matrixrepresentation
 7.5 Subgraph Patterns 455

example of such a matrix for the given graph. In principle, a graph can have
more than one adjacency matrix representation because there are multiple
ways to order the vertices in the adjacency matrix. In the example shown in
Figure 7.20, lhe first row and column correspond to the vertex a that has 3
edges, the second row and column correspond to another vertex a that has
2 edges, and so on. To derive all the adjacency matrix representations for
a graph, we need to consider all possible permutations of rows (and their
corresponding columns) of the matrix.
Mathematically, each permutation corresponds to a multiplication of the
initial adjacency matrix with a corresponding permutation matrix, as illus-
trated in the following example.
Example 7.3. Consider the following matrix:

23
67
M_
10 11
L4 15

The following permutation matrix can be used to exchange the first row (and
column) with the third row (and column\ of M:

Prs:

where Prs is obtained by swapping the first and third row of the identity
matrix. To exchangethe first and third rows (and columns), the permutation
matrix is multiplied wilh M:

M' : P{sx M x

Note that multiplying M from the right with P13 exchangesthe first and third
columns of.M, while multiplying M from the left with Pfi exchanges the first
and third rows of M. If all three matrices are multiplied, this will produce a
matrix M'whose first and third rows and columns have been swapped. r
456 Chapter 7 Association Analysis: Advanced Concepts

A(3) A(4)
code = 1100111000010010010100001011

A(2) A(1)

A(3) A(4)
code = 101101001010000010011
0001110

Figure
7.21.String
representation matrices.
ofadjacency

The secondstep is to determine the string representationfor each adjacency

matrix. Since the adjacency matrix is symmetric, it is suffrcient to construct
the string representation based on the upper triangular part of the matrix. In
the example shown in Figure 7.2I,the code is obtained by concatenating the
entries of the upper triangular matrix in a column-wise fashion. The final step
is to compare all the string representations of the graph and choose the one
that has the lowest (or highest) lexicographic value.
The preceding approach seemsexpensivebecauseit requires us to examine
all possible adjacency matrices of a graph and to compute each of their string
representation in order to find the canonical label. More specifically, there
are kl permutations that must be considered for every graph that contains k
vertices. Some of the methods developedto reduce the complexity of this task
include caching the previously computed canonical label (so that we do not
have to recompute it again when performing an isomorphism test on the same
graph) and reducing the number of permutations needed to determine the
canonical label by incorporating additional information such as vertex labels
and the degree of a vertex. The latter approach is beyond the scope of this
 7.6 InfreouentPatterns 457

book, but interested readers may consult the bibliographic notes at the end of
this chapter.

7.5.6 Support Counting

Support counting is also a potentially costly operation because all the can-
didate subgraphs contained in each graph G e g must be determined. One
way to speed up this operation is to maintain a Iist of graph IDs associated
with each frequent (k - l)-subgraph. Whenever a new candidate k-subgraph
is generatedby merging a pair of frequent (k - 1)-subgraphs,their correspond-
ing lists of graph IDs are intersected. Finally, the subgraph isomorphism tests
are performed on the graphs in the intersected list to determine whether they
contain a particular candidate subgraph.

7.6 Infrequent Patterns

The association analysis formulation described so far is based on the premise
that the presenceof an item in a transaction is more important than its ab-
sence. As a consequence,patterns that are rarely found in a databaseare often
consideredto be uninteresting and are eliminated using the support measure.
Such patterns are known as infrequent patterns.
Definition 7.7 (Infrequent Pattern). An infrequent pattern is an itemset
or a rule whose support is less than the minsup threshold.
Although a vast majority of infrequent patterns are uninteresting, some
of them might be useful to the analysts, particularly those that correspond
to negative correlations in the data. For example, the sale of DVDsand VCRs
together is low becauseany customer who buys a DVDwill most likely not buy
a VCR,and vice versa. Such negative-correlated patterns are useful to help
identify competing items, which are items that can be substituted for one
another. Examples of competing items include tea versus coffee,butter versus
margarine, regular versus diet soda, and desktop versus laptop computers.
Some infrequent patterns may also suggest the occurrence of interesting
rare eventsor exceptional situations in the data. For example, if {f :-re = Yes}
is frequent but {Fire = Yes, Alarm = 0"} is infrequent, then the latter is an
interesting infrequent pattern because it may indicate faulty alarm systems.
To detect such unusual situations, the expected support of a pattern must
be determined, so that, if a pattern turns out to have a considerably lower
support than expected, it is declared as an interesting infrequent pattern.
458 Chapter 7 Association Analysis: Advanced Concepts

Mining infrequent patterns is a challenging endeavor because there is an

enormous number of such patterns that can be derived from a given data set.
More specifically, the key issues in mining infrequent patterns are: (1) how
to identify interesting infrequent patterns, and (2) how to efficiently discover
them in large data sets. To get a different perspective on various types of
interesting infrequent patterns, two related concepts-negative patterns and
negatively correlated patterns-are introduced in Sections 7.6.1 and 7.6.2, re-
spectively. The relationships among these patterns are elucidated in Section
7.6.3. Finally, two classesof techniques developed for mining interesting in-
frequent patterns are presented in Sections 7.6.5 and 7.6.6.

7.6.L Negative Patterns

Let 1 : {h,iz,. ..,ia} be a set of items. A negative itern, ti4,denotesthe

absenceof item i7,from a given transaction. For example, cof f ee is a negative
item whose value is 1 if a transaction does not contain cof f ee.
Definition 7.8 (Negative Itemset). A negative itemset X is an itemset
that has the following properties: (1) X : AU B, where .4 is a set of positive
items, B is a set of negativeitems, lBl > 1, and (2) s(X) > minsup.
Definition 7.9 (Negative Association Rule). A negative association rule
is an associationrule that has the following properties: (1) the rule is extracted
from a negative itemset, (2) the support of the rule is greater than or equal to
minsup, and (3) the confidenceof the rule is greater than or equal to minconf.
The negative itemsets and negative associationrules are collectively known
as negative patterns throughout this chapter. An example of a negative
association rule is tea ------+
E6f f ee, which may suggestthat people who drink
tea tend to not drink coffee.

7.6.2 Negatively Correlated Patterns

Section 6.7.1 on page 371 described how correlation analysis can be used to
analyze the relationship between a pair of categorical variables. Measures
such as interest factor (Equation 6.5) and the /-coefficient (Equation 6.8)
were shown to be useful for discovering itemsets that are positively correlated.
This section extends the discussionto negatively correlated patterns.
Let X : {rt,r2t...,r7,} denotea k-itemsetand P(X) denotethe proba-
bility that a transaction contains X. In association analysis, the probability
is often estimated using the itemset support, s(X).
 7.6 Infrequent Patterns 459

Definition 7.10 (Negatively Correlated Itemset). Atr itemset X is neg-

ativelv correlated if

k
< : s ( r r ) x s ( r 2 ) x. . . x s ( r k ) , (7.3)
" ( x ) jf:lr" ( " i )

where s(ri) is the support of an item ri.

The right-hand side of the preceding expression,lI!:tt@), representsan

estimate of the probability that all the items in X are statistically independent.
Definition 7.10 suggeststhat an itemset is negatively correlated if its support
is below the expected support computed using the statistical independence
assumption. The smaller s(X), the more negatively correlated is the pattern.
Definition 7.1L (Negatively Correlated Association Rule). An asso-
ciation rule X ---+ Y is negatively correlated if

s(XuY) < s(X)s()'), (7.4)

where X and Y are disjoint itemsets;i.e., XUY :4.

The preceding definition provides only a partial condition for negative cor-
relation between items in X and items in Y. A full condition for negative
correlation can be stated as follows:

s(Xuv).I s@;)fs(s), (7.5)

where ri e X and gi e Y. Because the items in X (and in Y) are often

positively correlated, it is more practical to use the partial condition to de-
fine a negatively correlated association rule instead of the full condition. For
example, although the rule

{eyeg}ass, lens cleaner} -----r{contact lens, saline solution}

is negatively correlated according to Inequality 7.4, eyeglass is positively

correlated with lens cleaner and contact l-ens is positively correlated with
saline solution. If Inequality 7.5 is applied instead, such a rule could be
missed becauseit may not satisfy the full condition for negative correlation.
460 Chapter 7 Association Analysis: Advanced Concepts

The condition for negative correlation can also be expressedin terms of

the support for positive and negative itemsets. Let X and Y denote the
corresponding negative itemsets for X and Y, respectively. Since

s(xur)-s(x)s(Y)
r- _'tf I
: s(XU)') - uY)+ s(xuY)l uY)+ s(xur)l
fs(x f(x
f.l
: s(xu")lt s ( X u Y )- s ( x u v ) - s ( x u v ) l s(XuY)s(xuY)
: s(X u Y)s(X u 7) - s(X u Y)s(X r-tY),
the condition for negative correlation can be stated as follows:

s(x u v)s(X u 7) < s(Xu Y)s(Xuv). (7.6)

The negatively correlated itemsets and association rules are known as nega-
tively correlated patterns throughout this chapter.

7.6.3 Comparisons among Infrequent Patterns, Negative Pat-

terns, and Negatively Correlated Patterns

Infrequent patterns, negative patterns, and negatively correlated patterns are

three closely related concepts. Although infrequent patterns and negatively
correlated patterns refer only to itemsets or rules that contain positive items,
while negative patterns refer to itemsets or rules that contain both positive
and negative items, there are certain commonalities among these concepts, as
illustrated in Figure 7.22.
First, note that many infrequent patterns have correspondingnegative pat-
terns. To understand why this is the case, consider the contingency table
shown in Table 7.9. If XUY is infrequent, then it is likely to have a corre-
sponding negative itemset unless rn'insup is too high. For example, assuming
that mi,nsup < 0.25, if XUY is infrequent, then the support for at least one of
the following itemsets, X UY , X UY , or X U Y, must be higher than mi,nsup
since the sum of the supports in a contingency table is 1.
Second, note that many negatively correlated patterns also have corre-
sponding negative patterns. Consider the contingency table shown in Table
7.9 and the condition for negative correlation stated in Inequality 7.6. If X
and Y have strong negative correlation, then

s(XuZ) x s(x u Y) >>s(Xu r) x s(XuY).

 7.6 InfrequentPatterns 461

FrequentPatterns

Figwe7.22.Comparisons
among patterns,
infrequent patterns,
negative andnegatively
correlated
patterns.

Tabfe7.9.Atwo-way
contingency ruleX --.-+Y.
tablefortheassociation

Y Y
X s(X u Y) s ( Xu Y ) s(x)
X s(X u Y ) s ( Xu Y ) s(x)
s(v) s(v) 1

Therefore, either X UY or X U Y, or both, must have relatively high support

when X and Y are negatively correlated. These itemsets correspond to the
negative patterns.
Finally, becausethe lower the support of X U Y, the more negatively cor-
related is the pattern, negatively correlated patterns that are infrequent tend
to be more interesting than negatively correlated patterns that are frequent.
The infrequent, negatively correlated patterns are illustrated by the overlap-
ping region in Figure 7.22 between both types of patterns.

7.6.4 Techniques for Mining Interesting Infrequent Patterns

In principle, infrequent itemsets are given by all itemsets that are not extracted
by standard frequent itemset generation algorithms such as Apri,ori, and FP-
462 Chapter 7 Association Analysis: Advanced Concepts

Maximal Frequent
Itemset

'/'
Frequent
Itemset
Border
Infrequent

Figure7.23,Frequent itemsets.
andinfrequent

growth. These itemsets correspondto those located below the frequent itemset
border shown in Figure 7.23.
Since the number of infrequent patterns can be exponentially large, es-
pecially for sparse, high-dimensional data, techniques developed for mining
infrequent patterns focus on finding only interesting infrequent patterns. An
example of such patterns includes the negatively correlated patterns discussed
in Section 7.6.2. These patterns are obtained by eliminating all infrequent
itemsets that fail the negative correlation condition provided in Inequality
7.3. This approach can be computationally intensive because the supports
for all infrequent itemsets must be computed in order to determine whether
they are negatively correlated. Unlike the support measure used for mining
frequent itemsets, correlation-basedmeasuresused for mining negatively corre-
lated itemsets do not possessan anti-monotone property that can be exploited
for pruning the exponential search space. Although an efficient solution re-
mains elusive, several innovative methods have been developed, as mentioned
in the bibliographic notes provided at the end of this chapter.
The remainder of this chapter presentstwo classesof techniquesfor mining
interesting infrequent patterns. Section 7.6.5 describes methods for mining
 7.6 InfrequentPatterns 463

TID Items TID A A B B c c D D

1 {A,B} 1 1 0 1 0 0 1 0 1
2 {A,B,C} 2 1 0 1 0 1 0 0 1
3 tc) 3 0 1 0 1 1 0 0 1
4 {B,C} 4 0 1 1 0 1 0 0 1
5 {B,D} 5 0 1 1 0 0 1 1 0

OriginalTransactions Transactions
with Negativeltems

Figure7.24.
Augmenting
a datasetwithnegative
items.

negative patterns in data, while Section 7.6.6 describes methods for finding
interesting infrequent patterns based on support expectation.

7.6.5 Techniques Based on Mining Negative Patterns

The first class of techniques developed for mining infrequent patterns treats
every item as a symmetric binary variable. Using the approach described in
Section 7.1, the transaction data can be binarized by augmenting it with neg-
ative items. Figure 7.24 shows an example of transforming the original data
into transactions having both positive and negative items. By applying exist-
ing frequent itemset generation algorithms such as Apriori on the augmented
transactions, all the negative itemsets can be derived.
Such an approach is feasible only if a few variables are treated as symmetric
binary (i.e., we look for negative patterns involving the negation of only a
small number of items). If every item must be treated as symmetric binary,
the problem becomescomputationally intractable due to the following reasons.

1. The number of items doubles when every item is augmented with its
corresponding negative item. Instead of exploring an itemset lattice of
size2d, where d is the number of items in the original data set, the lattice
becomesconsiderably larger, as shown in Exercise 21 on page 485.

2. Support-based pruning is no longer effective when negative items are

augmented. For each variable r, either r or r has support greater than
or equal to 50%. Hence, even if the support threshold is as high as
50Vo,half of the items will remain frequent. For lower thresholds, many
more items and possibly itemsets containing them will be frequent. The
support-based pruning strategy employed by Apriori, is effective only
464 Chapter 7 Association Analysis: Advanced Concepts

when the support for most itemsets is low; otherwise, the number of
frequent itemsets grows exponentially.

3. The width of each transaction increaseswhen negative items are aug-

mented. Supposethere are d items available in the original data set. For
sparse data sets such as market basket transactions, the width of each
transaction tends to be much smaller than d. As a result, the maximum
size of a frequent itemset, which is bounded by the maximum transac-
tion width, rumax,tends to be relatively small. When negative items are
included, the width of the transactions increasesto d becausean item is
either present in the transaction or absent from the transaction, but not
both. Since the maximum transaction width has grown from'u.r,r.ur. to
d, this will increase the number of frequent itemsets exponentially. As
a result, many existing algorithms tend to break down when they are
applied to the extended data set.

The previous brute-force approach is computationally expensivebecauseit

forces us to determine the support for a large number of positive and negative
patterns. Instead of augmenting the data set with negative items, another
approach is to determine the support of the negative itemsets based on the
support of their corresponding positive items. For example, the support for
{p,Q,r} can be computed in the following way:

s ( { p , Q , r } ): - q } ) - ' ( { p , r } ) + s ( { p ,q , r } ) .
"({p}) "({p,
More generally,the support for any itemsetX UY can be obtainedas follows:

s(XuY):s(X)+t {(-t)uxs(Xoz)}. (7.7)

i:r Zcy,lzl:i

To apply Equation 7.7, s(X U Z) mtxt be determined for every Z that is a

subset of Y. The support for any combination of X and Z that exceedsthe
m'insup threshold can be found using the Apri.ori, algorithm. For all other
combinations, the supports must be determined explicitly, e.8., by scanning
the entire set of transactions. Another possible approach is to either ignore
the support for any infrequent itemset X U Z or to approximate it with the
m'insup threshold.
Severaloptimization strategies are available to further improve the perfor-
mance of the mining algorithms. First, the number of variables consideredas
 7.6 Infrequent Patterns 465

symmetric binary can be restricted. More specifically,a negative item E is con-

sidered interesting only if y is a frequent item. The rationale for this strategy
is that rare items tend to produce a large number of infrequent patterns and
many of which are uninteresting. By restricting the set Y given in Equation 7.7
to variables whose positive items are frequent, the number of candidate nega-
tive itemsets consideredby the mining algorithm can be substantially reduced.
Another strategy is to restrict the type of negative patterns. For example, the
algorithm may consider only a negative pattern X U Y if it contains at least
one positive item (i.e., lxl > 1). The rationale for this strategy is that if the
data set contains very few positive items with support greater than 50%, then
most of the negative patterns of the form X U Y will become frequent, thus
degrading the performance of the mining algorithm.

7.6.6 Techniques Based on Support Expectation

Another class of techniques considers an infrequent pattern to be interesting
only if its actual support is considerably smaller than its expected support. For
negatively correlated patterns, the expected support is computed based on the
statistical independence assumption. This section describes two alternative
approaches for determining the expected support of a pattern using (1) a
concept hierarchy and (2) a neighborhood-basedapproach known as indirect
association.

Support Expectation Based on Concept Hierarchy

Objective measuresalone may not be sufficient to eliminate uninteresting in-

frequent patterns. For example, suppose bread and laptop computer are
frequent items. Even though the itemset {bread, Iaptop conputer} is in-
frequent and perhaps negatively correlated, it is not interesting becausetheir
lack of support seemsobvious to domain experts. Therefore, a subjective ap-
proach for determining expected support is needed to avoid generating such
infrequent patterns.
In the preceding example, bread and laptop computer belong to two
completely different product categories, which is why it is not surprising to
find that their support is low. This example also illustrates the advantage of
using domain knowledge to prune uninteresting patterns. For market basket
data, the domain knowledge can be inferred from a concept hierarchy such
as the one shown in Figure 7.25. The basic assumption of this approach is
that items from the same product family are expected to have similar types of
interaction with other items. For example, since ham and bacon belong to the
466 Chapter 7 Association Analysis: Advanced Concepts

Taco Oatmeal Chocolate Ham Bacon Boneless

Chio

Figure7.25.Example hierarchy.
of a concept

same product family, we expect the association between ham and chips to be
somewhat similar to the association between bacon and chips. If the actual
support for any one ofthese pairs is lessthan their expected support, then the
infrequent pattern is interesting.
To illustrate how to compute the expected support, consider the diagram
shown in Figure 7.26. Supposethe itemset {C,G} is frequent. Let s(') denote
the actual support of a pattern and e(.) denote its expected support. The
expected support for any children or siblings of C and G can be computed
using the formula shown below.

e(s(E,J)):
s(C,G)rffi"8 (7.8)

e(s(C,J)): s(C,G)>< (7.e)

#
e(s(C,H)) : s(C,G)"
ffi (7.10)

For example, if soda and snack food are frequent, then the expected
support between diet soda and chips can be computed using Equation 7.8
because these items are children of soda and snack food, respectively. If
the actual support for diet soda and chips is considerably lower than their
expected value, then diet soda and chips form an interesting infrequent
pattern.
 7.6 InfrequentPatterns 467

DEJK

Figure7.26.Mining
interesting patterns
negative usinga concept
hierarchy.

Support Expectation Based on Indirect Association

Consider a pair of items, (a, b), that are rarely bought together by customers.
Ifa and b are unrelated items, such as bread and DVOplayer, then their support
is expected to be low. On the other hand, if a and b are related items, then
their support is expected to be high. The expected support was previously
computed using a concept hierarchy. This section presents an approach for
determining the expected support between a pair of items by looking at other
items commonly purchased together with these two items.
For example, suppose customers who buy a sleeping bag also tend to
buy other camping equipment, whereas those who buy a desktop computer
also tend to buy other computer accessoriessuch as an optical mouse or a
printer. Assuming there is no other item frequently bought together with both
a sleeping bag and a desktop computer, the support for these unrelated items
is expected to be low. On the other hand, supposediet and regular soda are
often bought together with chips and cookies. Even without using a concept
hierarchy, both items are expected to be somewhat related and their support
should be high. Becausetheir actual support is low, diet and regular soda
form an interesting infrequent pattern. Such patterns are known as indirect
association patterns.
A high-level illustration of indirect association is shown in Figure 7.27.
Items a and b correspond to diet soda and regular soda, while Y, which is
known as the mediator set, contains items such as chips and cookies. A
formal definition of indirect association is presented next.
468 Chapter 7 Association Analysis: Advanced Concepts

a .-v
'.,,^
a\' tl
.\_-/

Figwe7.27. Anindirect
association a pairof items.
between

Definition 7.L2 (Indirect Association). A pair of items a,b is indirectly

associatedvia a mediator set Y if the following conditions hold:

1. s({a,b}) < t" (Itempair support condition).

2. 2Y I 0 such that:

(a) s({a} U y) > t7 and s({b} u Y) 2 tt (Mediator support condition).

(b) d({a},Y) > ta,d({b},Y) > ta, where d(X,Z) is an objectivemea-
sure of the association between X and Z (Mediator dependence
condition).

Note that the mediator support and dependence conditions are used to
ensure that items in Y form a close neighborhood to both a and b. Some
of the dependencemeasuresthat can be used include interest, cosine or IS,
Jaccard, and other measurespreviously described in Section 6.7.1 on page 371.
Indirect associationhas many potential applications. In the market basket
domain, a and b may refer to competing items such as desktop and laptop
computers. In text mining, indirect association can be used to identify syn-
onyms, antonyms, or words that are used in different contexts. For example,
given a collection of documents, the word data may be indirectly associated
with gold via the mediator mining. This pattern suggests that the word
mining can be used in two different contexts-data mining versus gold min-
itrg.
Indirect associationscan be generated in the following way. First, the set
of frequent itemsets is generated using standard algorithms such as Apri'ori'
or FP-growth. Each pair of frequent k-itemsets are then merged to obtain
a candidate indirect association (a,b,Y), where a and b are a pair of items
and Y is their common mediator. For example, if {p,q,r} and {p,q,s} are
 7.7 Bibliographic Notes 469

Algorithm 7.2 Algorithm for mining indirect associations.

1: Generate Fa,the set of frequent itemsets.
2: fot k :2 to k-ur. do
3: Cn: {(a,b,Y)l{a}Uy € Fn,{b}Uy € Fp,alb}
4: for each candidate (a,b,Y) € Cp do
b: if s({o,,b}) < r" A d({a},y) Z ta A d({b}, y) > ta then
6: In : In U {(o, b,Y)}
7: end if
8: end for
9: end for
1o: Result : UIr.

frequent 3-itemsets, then the candidate indirect association (r,t,{p,q}) is ob-

tained by merging the pair of frequent itemsets. Once the candidates have
been generated,it is necessaryto verify that they satisfy the itempair support
and mediator dependenceconditions provided in Definition 7.12. However,
the mediator support condition does not have to be verified becausethe can-
didate indirect association is obtained by merging a pair of frequent itemsets.
A summary of the algorithm is shown in Algorithm 7.2.

7.7 Bibliographic Notes

The problem of mining associationrules from categorical and continuous data
was introduced by Srikant and Agrawal in 1363].Their strategy was to binarize
the categorical attributes and to apply equal-frequency discretization to the
continuous attributes. A partial completeness measure was also proposed
to determine the amount of information loss as a result of discretization. This
measure was then used to determine the number of discrete intervals needed
to ensure that the amount of information loss can be kept at a certain desired
level. Following this work, numerous other formulations have been proposed
for mining quantitative association rules. The statistics-based approach was
developedby Aumann and Lindell [343] to identify segmentsof the population
who exhibit interesting behavior characterizedby some quantitative attributes.
This formulation was later extended by other authors including Webb [363]and
Zhang et al. [372]. The min-Apri,ori algorithm was developed by Han et al.
[349] for finding association rules in continuous data without discretization.
The problem of mining association rules in continuous data has also been
 Chapter 7 Association Analysis: Advanced Concepts

investigated by numerous other researchers including Fukuda et al' 1347)'

Lent et al. [355], Wang et al. [367], and Miller and Yang [357].
The method described in Section 7.3 for handling concept hierarchy using
extended transactions was developed by Srikant and Agrawal 1362].An alter-
native algorithm was proposed by Han and Ib [350], where frequent itemsets
are generated one level at a time. More specifically, their algorithm initially
generatesall the frequent l-itemsets at the top level of the concept hierarchy.
The set of frequent 1-itemsets is denoted as .L(1,1). Using the frequent 1-
itemsets in L(7,1), the algorithm proceedsto generate all frequent 2-itemsets
at level 7, L(I,2). This procedure is repeated until all the frequent itemsets
involving items from the highest level of the hierarchy, ,L(1,k) (k > 1), are
extracted. The algorithm then continues to extract frequent itemsets at the
next level of the hierarchy,L(2,I), based on the frequent itemsets in.L(1,1).
The procedure is repeated until it terminates at the lowest level of the concept
hierarchy requested by the user.
The sequential pattern formulation and algorithm described in Section 7.4
was proposed by Agrawal and Srikant in [341, 364]. Similarly, Mannila et
al. [356] introduced the concept of frequent episode, which is useful for min-
ing sequential patterns from a long stream of events. Another formulation of
sequentialpattern mining based on regular expressionswas proposed by Garo-
falakis et al. in [348]. Joshi et al. have attempted to reconcile the differences
between various sequential pattern formulations [352]. The result was a uni-
versal formulation of sequential pattern with the different counting schemes
described in Section 7.4.4. Alternative algorithms for mining sequential pat-
terns were also proposed by Pei et aI. [359], Ayres et al. [344], Cheng et al.
1346],and Seno et al. [361].
The frequent subgraph mining problem was initially introduced by Inokuchi
et al. in [351]. They used a vertex-growing approach for generating frequent
induced subgraphs from a graph data set. The edge-growing strategy was
developed by Kuramochi and Karypis in 1353],where they also presented an
Apri,ori,-Iike algorithm called FSG that addressesissues such as multiplicity
of candidates, canonical labeling, and vertex invariant schemes. Another fre-
quent subgraph mining algorithm known as gSpan was developed by Yan and
Han in [370]. The authors proposed using a minimum DFS code for encoding
the various subgraphs. Other variants of the frequent subgraph mining prob-
Iems were proposed by Zaki in 1371],Parthasarathy and Coatney in 1358],and
Kuramochi and Karypis in [354].
The problem of mining infrequent patterns has been investigated by many
authors. Savasereet al. [360] examined the problem of mining negative asso-
 Bibliography 47L

ciation rules using a concept hierarchy. Tan et al. [365] proposed the idea of
mining indirect associationsfor sequential and non-sequential data. Efficient
algorithms for mining negative patterns have also been proposed by Boulicaut
et al. [345], Teng et al. [366], Wu et al. [369], and Antonie and Za'iane 13421.

Bibliography
f341] R. Agrawal and R. Srikant. Mining Sequential Patterns. In Proc. of IntI. Conf. on
Data Engi,neeri:ng, pages 3 14, Taipei, Taiwan, 1995.
1342] M.-L. Antonie and O. R. Zaiane. Mining Positive and Negative Association Rules:
An Approach for Confined Rules. In Proc. of the 8th European Conf of Princi,ples
and Practice of Knowledge D,i,scoueryi,n Databases, pages 27-38, Pisa, Italy, September
2004.
[343] Y. Aumann and Y. Lindell. A Statistical Theory for Quantitative Association Rules.
In KDD99, pages 261-270, San Diego, CA, August 1999.
13441 J . Ayres, J. Flannick, J. Gehrke, and T. Yiu. Sequential Pattern mining using a bitmap
representation. In Proc. of the 9th IntI. Conf. on Knowledge D,iscouerg and Data Mining,
pages 429 435, trdmonton, Canada, JuJy 2002.
[345] J.-F. Boulicaut, A. Bykowski, and B. Jeudy. Towards the Tlactable Discovery of
Association Rules with Negations. In Proc. of the lth Intl. Conf on Flerible Query
Answering Sgstems FQAS'1}, pages 425-434, Warsaw, Poland, October 2000.
[346] H. Cheng, X. Yan, and J. Han. IncSpan: incremental mining of sequential patterns
in large database. In Proc. of the 10th Intl. Conf. on Knowledge Discouerg and, Data
M'in'ing, pages 527 532, Seattle, WA, August 2004.
[347] T. Fukuda, Y. Morimoto, S. Morishita, and T. Tokuyama. Mining Optimized Associa-
tion Rules for Numeric Attributes. In Proc. of the 15th SErnp. on Principles of Database
Sgstems, pages 182-191, Montreal, Canada, June 1996.
1348] M. N. Garofalakis, R. Rastogi, and K. Shim. SPIRIT: Sequential Pattern Mining with
Regular Expression Constraints. In Proc. of the 25th VLDB ConJ., pages 223-234,
Edinburgh, Scotland, 1999.
1349] E.-H. Han, G. Karypis, and V. Kumar. Min-Apriori: An Algorithm for Finding As-
sociation Rules in Data with Continuous Attributes. http://www.cs.umn.edu/-han,
1997.
[350] J. Han and Y. Fu. Mining Multiple-Level Association Rules in Large Databases. IEEE
Trans. on Knowledge and Data Eng'ineering, 11(5):798-804, 1999.
[351] A. Inokuchi, T. Washio, and H. Motoda. An Apriori-based Algorithm for Mining F]e-
quent Substructures from Graph Data. In Proc. of the /1th European Conf. of Principles
and, Practice of Knowledge D'iscouery in Databases,pages 13-23, Lyon, France, 2000.
[352] M. V. Joshi, G. Karypis, and V. Kumar. A Universal Formulation of Sequential
Patterns. In Proc. of the KDD'2001 workshop on Temporal Data Mi,ning, San Ftancisco,
CA, August 2001.
[353] M. Kuramochi and G. Karypis. Frequent Subgraph Discovery. In Proc. of the 2001
IEEE Intl. Conf. on Data Mi,ning, pages 313-320, San Jose, CA, November 2001.
[354] M. Kuramochi and G. Karypis. Discovering Frequent Geometric Subgraphs. In Proc.
of the 2002 IEEE Intl. Conf. on Data Mini,ng, pages 258-265, Maebashi City, Japan,
December 2002.
472 Chapter 7 Association Analysis: Advanced Concepts

[355] B. Lent, A. Swami, and J. Widom. Clustering Association Rules. In Proc. of the 13th
IntI. Conf. on Data Engineering, pages 220 231, Birmingham, U.K, April 1997.
[356] H. Mannila, H. Toivonen, and A. I. Verkamo. Discovery of F]equent Episodes in Event
Sequences. Data Mi,ni.ng and Knowledge Di,scouery,1(3):259-289, November 1997.
[357] R. J. Miller and Y. Yang. Association Rules over Interval Data. In Proc. of 1997
ACM-SIGMOD IntL Conf. on Management of Data, pages 452 461, T\rcson, AZ, May
1997.
1358] S. Parthasarathy and M. Coatney. Efficient Discovery of Common Substructures in
Macromolecules. In Proc. of the 2002 IEEE Intl. Conf. on Data M'ining, pages 362 369,
Maebashi City, Japan, December 2002.
1359] J. Pei, J. Han, B. Mortazavi-Asl, Q. Chen, U. Dayal, and M. Hsu. PrefixSpan: Mining
Sequential Patterns efficiently by prefix-projected pattern growth. In Proc of the 17th
IntI. Conf. on Data Engi,neering, Heidelberg, Germany, April 2001.
[360] A. Savasere,E. Omiecinski, and S. Navathe. Mining for Strong Negative Associations
in a Large Database of Customer TYansactions. In Proc. of the llth IntI. Conf. on Data
Eng'ineering, pages 494-502, Orlando, Florida, February 1998.
[361] M. Seno and G. Karypis. SLPMiner: An Algorithm for Finding Frequent Sequential
Patterns Using Length-Decreasing Support Constraint. In Proc. of the 2002 IEEE Intl.
Conf. on Data Mzni,ng, pages 418-425, Maebashi City, Japan, December 2002.
f362] R. Srikant and R. Agrawal. Mining Generalized Association Rules. In Proc. of the
21st VLDB Conf., pages 407-419, Ztxich, Switzerland, 1995.
1363] R. Srikant and R. Agrawal. Mining Quantitative Association Rules in Large Relational
Tables. In Proc. of 1996 ACM-SIGMOD Intl. Conf. on Managementof Data, pages 1
12, Montreal, Canada, 1996.
1364] R. Srikant and R. Agrawal. Mining Sequential Patterns: Generalizations and Perfor-
mance Improvements. In Proc. of the Sth Intl Conf. on Ertending Database Technologg
(EDBT'96), pages 18 32, Avignon, France, 1996.
1365] P. N. Tan, V. Kumar, and J. Srivastava. Indirect Association: Mining Higher Order
Dependencies in Data. In Proc. of the lth European Conf. of Princ'iples and Practice
of Knouledge D'iscoueryin Databases, pages 632-637, Lyon, FYance,2000.
[366] W. G. Teng, M. J. Hsieh, and M.-S. Chen. On the Mining of Substitution Rules for
Statistically Dependent Items. In Proc. of the 2002 IEEE Intl. Conf. on Data Mining,
pages 442-449, Maebashi City, Japan, December 2002.
1367] K. Wang, S. H. Tay, and B. Liu. Interestingness-Based Interval Merger for Numeric
Association Rules. In Proc. of the lth Intl. Conf. on Knowledge Discouery and Data
Mr,ruing,pages 121 128, New York, NY, August 1998.
f368] G. I. Webb. Discovering associations with numeric variables. In Proc. of the 7th IntI.
Conf. on Knowledge Discouery and, Data M'ining, pages 383-388, San Francisco, CA,
August 2001.
f369] X. Wu, C. Zhang, and S. Zhang. Mining Both Positive and Negative Association Rules.
ACM Trans. on Informat'ion Sgstems,22(3):381 405,2004.
[370] X. Yan and J. Han. gSpan: Graph-based Substructure Pattern Mining. In Proc.
of the 2002 IEEE IntI. Conf. on Data Mi,ni.ng, pages 72L-724, Maebashi City, Japan,
December 2002.
f3711 M. J. Zakl. Efficiently mining frequent trees in a forest. In Proc. of the 8th Intl. Conf.
on Knowledge Discouery and Data Mining, pages 71-80, Edmonton, Canada, Jdy 2002.
1372] H. Zlnang, B. Padmanabhan, and A. Tuzhilin. On the Discovery of Significant Statis-
tical Quantitative Rules. In Proc. of the 10th Intl. Conf. on Knowled,ge D'i,scouery and'
Data Mi,ni,ng,pages 374-383, Seattle, WA, August 2004.
 7.8 Exercises 473

7.8 Exercises
1. Consider the traffic accident data set shown in Table 7.10.

Table7,10.Traffic
accident
dataset.
Weat her Driver's Tlaffic Seat belt Urash
Condition Condition Violation Severity
Good Alcohol-impaired Exceed speed limit No Major
Bad Sober None Yes Minor
Good Sober Disobey stop sign Yes Minor
Good Sober Exceed speed limit Yes Major
Bad Sober Disobey traffic signal No Major
Good Alcohol-impaired Disobey stop sign Yes Minor
Bad Alcohol-impaired None Yes Major
Good Sober Disobey trafrc signal Yes Major
Good Alcohol-impaired None No Major
Bad Sober Disobey traffic signal No Major
Good Alcohol-impaired Exceed speed limit Yes Major
Bad Sober Disobey stop sign Yes Minor

(a) Show a binarized version of the data set.

(b) What is the maximum width of each transaction in the binarized data?
(c) Assuming that support threshold is 30%, how many candidate and fre-
quent itemsets will be generated?
(d) Create a data set that contains only the following asymmetric binary
attributes: (LJeather: Bad, Driver's condition : Alcohol-impaired,
Traffic violation: Y e s , S e a t B e l t : N o ,C r a s h S e v e r i t y : t ' t a j o r ) .
For Traffic violation, only None has a value of 0. The rest of the
attribute values are assigned to 1. Assuming that support threshold is
30%, how many candidate and frequent itemsets will be generated?
(e) Compare the number of candidate and frequent itemsets generated in
parts (c) and (d).

2. (a) Consider the data set shown in Table 7.11. Supposewe apply the following
discretization strategies to the continuous attributes of the data set.

Dl: Partition the range of each continuous attribute into 3 equal-sized

bins.
D2: Partition the range of each continuous attribute into 3 bins; where
each bin contains an eoual number of transactions
474 Chapter 7 Association Analysis: Advanced Concepts

2.
7.11,DatasetforExercise
Table
TID Temperature Pressure Alarm 1 Alarm 2 Alarm 3
I 9l) 1105 0 0 1
2 6D 1040 I 1
I 0
3 103 1090 I 1 1
4 97 1084 1 0 0
o 80 1038 0 1 1
r) 100 1080 1 1 0
7 83 1025 1 0 1
8 86 1030 1 0 0
o 101 1100 1 1 I

For each strategy, answer the following questions:

i. Construct a binarized version of the data set.
ii. Derive all the frequent itemsets having support > 30%.
(b) The continuous attribute can also be discretized using a clustering ap-
proach.
i. PIot a graph of temperature versus pressure for the data points shown
in Table 7.11.
ii. How many natural clusters do you observe from the graph? Assign
a label (Cr, Cr, etc.) to each cluster in the graph.
iii. What type of clustering algorithm do you think can be used to iden-
tify the clusters? State your reasons clearly.
iv. Replace the temperature and pressure attributes in Table 7.11 with
asymmetric binary attributes C1, C2, etc. Construct a transac-
tion matrix using the new attributes (along with attributes Alarml,
Alarm2, and Alarm3).
v. Derive all the frequent itemsets having support > 30% from the bi-
narized data.

,). Consider the data set shown in Table 7.I2. The first attribute is continuous,
while the remaining two attributes are asymmetric binary. A rule is considered
to be strong if its support exceeds 15% and its confidence exceeds60%. The
data given in Table 7.12 supports the following two strong rules:

( i ) { ( 1 < A < 2 ) , 8 : 1 } - - - +{ C : 1 }
( i i ) { ( 5 < A < 8 ) , 8 : 1 } - - +{ C : 1 }

(a) Compute the support and confidence for both rules.

(b) To find the rules using the traditional Apriori algorithm, we need to
discretize the continuous attribute A. Supposewe apply the equal width
 7.8 Exercises 475

Table7.12.
DatasetforExercise
3.
A B C
1 1 I
2 1 I
,1 1 0
4 1 0
o 1 1
o 0 1
7 0 0
8 1 1
q
0 0
10 0 0
11 0 0
12 0 1

binning approach to discretize the data, with bin-wi,dth : 2,3,4. For

each b'in-w'idth, state whether the above two rules are discovered by the
Apriori, algorithm. (Note that the rules may not be in the same exact
form as before because it may contain wider or narrower intervals for A.)
For each rule that corresponds to one of the above two rules, compute its
support and confidence.
(c) Comment on the effectiveness of using the equal width approach for clas-
sifying the above data set. Is there a bin-width that allows you to find
both rules satisfactorily? If not, what alternative approach can you take
to ensure that vou will find both rules?

4. Consider the data set shown in Table 7.13.

Table
7.13.DatasetforExercise
4.
Age Number of Hours Online per Week (B)
(A) 0-5 5-10 10-20 20-30 30-40
10-15 2 .f
( 2
15-25 2 ( 10 10 3
25-35 10 15 r .J 2
35-50 4 o K
3 2

(a) For each combination of rules given below, specify the rule that has the
highestconfidence.
i . 1 5< A < 2 5 - - - - +
1 0< B < 2 0 , I 0 < A < 2 5 - - - - 1
+ 0 < B < 2 0 ,a n d
75 < A ( 35-------+ 70 < B <20.
476 Chapter 7 Association Analysis: Advanced Concepts

i i . 1 5 < A < 2 5 - - - - -1- +

0 <B < 20, 15 <A<25'-----+ 5 < B (20, and
I5<4125-----+5<B<30.
iii. 15<A<25--+ I 0 < B < 2 0 a n d 1 0 < A ( 3 5 - - - - -5+< B < 3 0 .
(b) Suppose we are interested in finding the average number of hours spent
online per week by Internet users between the age of 15 and 35. Write the
corresponding statistics-based association rule to characterizethe segment
of users. To compute the average number of hours spent online, approx-
imate each interval by its midpoint value (e.g., use B:7.5 to represent
theinterval5<B<10).
( c ) Test whether the quantitative association rule given in part (b) is statis-
tically significant by comparing its mean against the average number of
hours spent online by other users who do not belong to the age group.

5. For the data set with the attributes given below, describe how you would con-
vert it into a binary transaction data set appropriate for association analysis.
Specifically, indicate for each attribute in the original data set

(a) how many binary attributes it would correspond to in the transaction

data set,
(b) how the values of the original attribute would be mapped to values of the
binary attributes, and
(c) if there is any hierarchical structure in the data values of an attribute that
could be useful for grouping the data into fewer binary attributes.

The following is a list of attributes for the data set along with their possible
values. Assume that all attributes are collected on a per-student basis:

o Year : FYeshman,Sophomore, Junior, Senior, Graduate:Masters, Gradu-

ate:PhD, Professional
o Zip code : zip code for the home address of a U.S. student, zip code for
the local address of a non-U.S. studenr
o College : Agriculture, Architecture, Continuing Education, Education,
Liberal Arts, Engineering, Natural Sciences,Business,Law, Medical, Den-
tistry, Pharmacy, Nursing, Veterinary Medicine
o On Campus : 1 if the student lives on campus, 0 otherwise
o Each of the following is a separate attribute that has a value of 1 if the
person speaks the language and a value of 0, otherwise.
- Arabic
- Bengali
- Chinese Mandarin
- English
- Portuguese
 7.8 Exercises 477

- Russian
- Spanish

6. Consider the data set shown in Table 7.14. Suppose we are interested in ex-
tracting the following association rule:
( Age l ctz,Play Piano : Yes} ------'
{41 {Enjoy Classical Music : Yes}

Table
7.14.Data
setforExercise
6.
Age Play Piano Eniov Classical Music
a Yes Yes
11 Yes Yes
1A
I4 Yes No
17 Yes No
19 Yes Yes
2l No No
25 No No
29 Yes Yes
33 No No
39 No Yes
4I No No
4.7 No Yes

To handle the continuous attribute, we apply the equal-frequency approach

with 3, 4, and 6 intervals. Categorical attributes are handled by introducing as
many new asymmetric binary attributes as the number of categorical values.
Assume that the support threshold is 10% and the confidencethreshold is 70%.

(a) Suppose we discretize the Age attribute into 3 equal-frequency intervals.

Find a pair of values for a1 and a2 that satisfy the minimum support and
minimum confidence requirements.
(b) Repeat part (a) by discretizing the Age attribute into 4 equal-frequency
intervals. Compare the extracted rules against the ones you had obtained
in part (a).
(c) Repeat part (a) by discretizing the Age attribute into 6 equal-frequency
intervals. Compare the extracted rules against the ones you had obtained
in part (a).
(d) From the results in part (u), (b), and (c), discuss how the choice of dis-
cretization intervals will affect the rules extracted by association rule min-
ing algorithms.

7. Consider the transactions shown in Table 7.15, with an item taxonomy given
in Figure 7.25.
478 Chapter 7 Association Analysis: Advanced Concepts

7.15.Example
Table olmarket
basket
transactions.
Tbansaction ID Items tsought
1 Chips, Cookies, Regular Soda, Ham
z Chips, Ham, Boneless Chicken, Diet Soda
2 Ham, Bacon, Whole Chicken, Regular Soda
4 Chips, Ham, BonelessChicken, Diet Soda
r
Chips, Bacon, BonelessChicken
tl Chips, Ham, Bacon, Whole Chicken, Regular Soda
7 Chips, Cookies, BonelessChicken, Diet Soda

(a) What are the main challenges of mining association rules with item tax-
onomy?
(b) Consider the approach where each transaction t is replaced by an extended
transaction tt that contains all the items in f as well as their respective
ancestors. For example, the transaction t : { Chips, Cookies} will be
replaced by tt : {Ctrips, Cookies, Snack Food, Food}. Use this approach
to derive all frequent itemsets (up to size 4) with support > 70%.
(c) Consider an alternative approach where the frequent itemsets are gener-
ated one level at a time. Initially, all the frequent itemsets involving items
at the highest level of the hierarchy are generated. Next, we use the fre-
quent itemsets discovered at the higher level of the hierarchy to generate
candidate itemsets involving items at the lower levels of the hierarchy. For
example, we generate the candidate itemset {Cnips, Diet Soda} only if
{Snack Food, Soda} is frequent. Use this approach to derive all frequent
itemsets (up to size 4) with support > 70%.
(d) Compare the frequent itemsets found in parts (b) and (c). Comment on
the efficiency and completeness of the algorithms.

8 . The following questions examine how the support and confi.denceof an associ-
ation rule may vary in the presence of a concept hierarchy.

(a) Consider an item r in a given concept hierarchy. Let 11, fr2, . . .,7r denote
the k children of r in the concept hierarchy. Show that < Li:r t(T'),
"(r)
where s(.) is the support of an item. Under what conditions will the
inequality become an equality?
(b) Let p and q denote a pair of items, while p and Q are their corresponding
parents in the concept hierarchy. If s({p, q}) > mi,nsup, which of the fol-
Iowing itemsets are guaranteed to be frequent? (i) q}), (ii) s({p,4}),
"({p,
and (iii) s({p,,?})
(c) Consider the association rule {p} ------+
{q}. Supposethe confidence of the
rule exceeds mi,nconf . Which of the following rules are guaranteed to
 7.8 Exercises 479

have confidence higher than m'inconfZ (i) {p} ------{,?}, (ii) {p} ------+
{q},
and (iii) {p} ----.+{4}.

9. (a) List all the 4-subsequencescontained in the following data sequence:

< { 1 , 3 }{ 2 } { 2 , 3 }{ 4 } > ,
assuming no timing constraints.
(b) List all the 3-element subsequences contained in the data sequence for
part (a) assuming that no timing constraints are imposed.
(c) List all the 4-subsequencescontained in the data sequence for part (a)
(assuming the timing constraints are flexible).
(d) List all the 3-element subsequences contained in the data sequence for
part (a) (assuming the timing constraints are flexible).

10. Find all the frequent subsequences with support > 50% given the sequence
database shown in Table 7.16. Assume that there are no timing constraints
imposed on the sequences.

Table
7.16, Example
ofevent generated
sequences byvarious
sensors.
Sensor Timestamp Events
S1 1 A,B
2 C
3 D,E
^ C
S2 1 A,B
2 C,D
.) E
S3 1 B
2 A
3 B
4 D,E
S4 1 C
2 D,E
3 C
4 E
S5 1 B
2 A
.) B,C
4 A,D
480 Chapter 7 Association Analysis: Advanced Concepts

11. ( a ) F o r e a c ho f t h e s e q u e n c er sr : < € r e z . . . e t . . . e t + t . . . € t a s t ) g i v e nb e l o w ,
determine whether they are subsequencesof the sequence

< { 7 , 2 ,3 } { 2 , 4 } { 2 , 4 ,5 } { 3 , 5 } i 6 } >

subjected to the following timing constraints:

mingap : 6 (interval between last event in ea and first event
in e6'r1is > 0)
maxgap : 3 (interval between first event in e4 and last event
in eial is < 3)
maxspan : 5 (interval between first event in e1 and last event
in e1o"1is < 5)
ws:1- (time between first and last events in ei is ( 1)
o ,u):1{1}{2}{3}>
o w :1 {t,2,J,4}{5,6> }
o r.t):1 {2, 4}{2,4}{Gi >
o , u ) : 1 { 1 } { 2 , 4 } { 6 }>
e u :(
{ 1 , 2 } { 3 , 4 } { 5 , 6>}
(b) Determinewhether eachof the subsequences to given in the previousques-
tion are contiguoussubsequences of the following sequencess.
o s : { { I , 2 , 3 , 4 , 5 , 6 } { 1 , 2 , 3 , 4 , 5 , 6 } { 1 , 25, 3, 6, 4} >,
. s : < { 1 , 2 , 3 , 4 } { I , 2 , 3 , 4 , 5 , 6 } { 35,,46,} >
o s : ( { 1 , 2 } { 12, , 3 , 4 } { 3 , 4 , 5 , 6 } { 5 , >6 }
o s : ( { 1 , 2 , 3 } { 2 , 3 , 4 , 5 } { 4 , 5 ,>6 }

12. For eachof the sequence w : \"t,. . .,eh"t) below,determinewhetherthey are

subsequences of the following data sequence:

({A, B}{C, D}{A, B}{C, D}{A, B}{C, D}>

subjectedto the followingtiming constraints:

mingap : Q (interval betweenlast event in eaand first event
in e,a1is > 0)
maxgap : 2 (interval betweenfirst event in ei and last event
in e;a1is S 2)
maxspan: 6 (interval betweenfirst event in er and last event
in e1o"1is < 6)
UJ.9: 1 (time between first and last events in ei is < 1)

(a) u,: ({A){B}{C}{D})

(b) .: ({A}{B,C,D}{A))
(c) tr: ({A}{8, C,D}{A})
( d ) t n: ( { B , C } { A , D } { B , C h
 7.8 Exercises 48L

("), : ({A,B,C, D}{A, B,C, D})

13. Consider the following frequent 3-sequences:

< { r , 2 , 3 }> , < { 1 , 2 } { 3 >
} , < {1}{2,3> } , < { r , 2 } { 4 }> ,
< {1,3}{4> } , < { 1 ,2 , 4 }> , < { 2 , 3 } { 3 >
} , < { 2 , 3 } { 4> },
< {2}{3}{3}), and< {2}{3}{4}>.

(a) List all the candidate 4-sequencesproduced by the candidate generation

step of the GSP algorithm.
(b) List all the candidate 4-sequencespruned during the candidate pruning
step of the GSP algorithm (assuming no timing constraints).
( c ) List all the candidate 4-sequencespruned during the candidate pruning
step of the GSP algorithm (assuming margap : l).

14. Consider the data sequenceshown in Table 7.17 for a given object. Count the
number of occurrencesfor the sequence({p}{q}{"}) according to the following
counting methods:

(a) COBJ (one occurrence per object).

(b) CWIN (one occurrence per sliding window).
(") CMINWIN (number of minimal windows of occurrence).
(d) CDIST-O (distinct occurrenceswith possibility of event-timestamp over-
Iap).
(e) CDIST (distinct occurrences with no event timestamp overlap allowed).

Table7.17.Example
ofevent 14.
dataforExercise
sequence
Timestamp Events
1 P'Q
2 r
t) S
4 P,Q
tr frs
6 p
7 Qtr
8 QrS
9 p
10 QrrtS
482 Chapter 7 Association Analysis: Advanced Concepts

15. Describe the types of modifications necessary to adapt the frequent subgraph
mining algorithm to handle:

(a) Directed graphs

(b) Unlabeled graphs
(c) Acyclic graphs
(d) Disconnected graphs

For each type of graph given above, describe which step of the algorithm will be
affected (candidate generation, candidate pruning, and support counting), and
any further optimization that can help improve the efEciency of the algorithm.

16. Draw all candidate subgraphs obtained from joining the pair of graphs shown in
Figure 7.28. Assume the edge-growingmethod is used to expand the subgraphs.

(a)

(b)

Figure7.28.Graphs 16.
forExercise

17. Draw all the candidate subgraphs obtained by joining the pair of graphs shown
in Figure 7.29. Assume the edge-growing method is used to expand the sub-
graphs.
 7.8 Exercises 483

(a)

(b)

Figure forExercise
7.29,Graphs 17.

18. (a) If support is defined in terms of induced subgraph relationship, show that
the confidenceofthe rule 91 - gz can be greater than 1 ifgl and !2 are
allowed to have overlapping vertex sets.
(b) What is the time complexity needed to determine the canonical label of
a graph that contains lVl vertices?
(c) The core of a subgraph can have multiple automorphisms. This will in-
crease the number of candidate subgraphs obtained after merging two
frequent subgraphs that share the same core. Determine the maximum
number of candidate subgraphs obtained due to automorphism of a core
of size k.
(d) Two frequent subgraphs of size k may share multiple cores. Determine
the maximum number of cores that can be shared by the two frequent
subgraphs.

le. (a) Consider a graph mining algorithm that uses the edge-growing method to
join the two undirected and unweighted subgraphs shown in Figure 19a.

+
484 Chapter 7 Association Analysis: Advanced Concepts

I Draw all the distinct cores obtained when meiging the two subgraphs.
II How many candidates are generated using the following core?

20. The original association rule mining framework considers only presence of items
together in the same transaction. There are situations in which itemsets that
are infrequent may also be informative. For instance, the itemset TV, DVD, -
VCR suggeststhat many customerswho buy TVs and DVDs do not buy VCRs.
In this problem, you are asked to extend the association rule framework to neg-
ative itemsets (i.e., itemsets that contain both presence and absence of items).
We will use the negation symbol (-) to refer to absenceof items.

(a) A naive way for deriving negative itemsets is to extend each transaction
to include absenceof items as shown in Table 7.18.

Table
7.18,Example dataset.
ofnumeric
TID TV -TV DVD -DVD VCR -VCR
1 I 0 0 1 0 I
2 1
0 0 I 0 I

i. Supposethe transaction database contains 1000 distinct items. What

is the total number of positive itemsets that can be generated from
these items? (Note: A positive itemset does not contain any negated
items).
ii. What is the maximum number of frequent itemsets that can be gen-
erated from these transactions? (Assume that a frequent itemset may
contain positive, negative, or both types of items)
iii. Explain why such a naive method of extending each transaction with
negative items is not practical for deriving negative itemsets.
(b) Consider the database shown in Table 7.15. What are the support and
confidence values for the following negative association rules involving
regular and diet soda?
i. -Regular ------+
Diet.
 7.8 Exercises 485

iii. -Diet ------+

Regular.
iv. Diet ---+ -Regular.

27. Suppose we would like to extract positive and negative itemsets from a data
set that contains d items.

(a) Consider an approach where we introduce a new variable to represent each

negative item. With this approach, the number of items grows from d to
2d. What is the total size of the itemset lattice, assuming that an itemset
may contain both positive and negative items of the same variable?
(b) Assume that an itemset must contain positive or negative items of different
variables. For example, the itemset {a, a,b,Z} is invalid becauseit contains
both positive and negative items for variable o. What is the total size of
the itemset lattice?

22. For each type of pattern defined below, determine whether the support measure
is monotone, anti-monotone, or non-monotone (i.e., neither monotone nor anti-
monotone) with respect to increasing itemset size.

(a) Itemsets that contain both positive and negative items such as {a,b,Z,d}.
Is the support measure monotone, anti-monotone, or non-monotone when
applied to such patterns?
(b) Boolean logical patterns such as {(a v b V c), d, e}, which may con-
tain both disjunctions and conjunctions of items. Is the support measure
monotone, anti-monotone, or non-monotone when applied to such pat-
terns?

23. Many association analysis algorithms rely on an Apriori-like approach for find-
ing frequent patterns. The overall structure of the algorithm is given below.

Algorithm 7.3 Apri,ori,-like algorithm.

I: k:I.
2: F6 : { i I i eIA <#D 2 minsup}. {Find frequent l-patterns.}
3: repeat
4: k: k+I.
5: Cr - genCandidate(Fp-1). {Candidate Generation}
6: 6o : pruneCandidate(Cr, Fr-r). {Candidate Pruning}
7: Cx: cotnt(Cu, D). {Support Counting}
8: Fu: {cl c€Cp A # > rninsup]1. {Extract frequent patterns}
9: until Fx : A
10: Answer : lJFr.
486 Chapter 7 Association Analysis: Advanced Concepts

Suppose we are interested in finding boolean logical rules such as

------
{o v b} {c,d},

which may contain both disjunctions and conjunctions of items. The corre-
sponding itemset can be written as {(o Y b),c,d}.

(a) Does the Apriori. principle still hold for such itemsets?
(b) How should the candidate generation step be modified to find such pat-
terns?
(c) How should the candidate pruning step be modified to find such patterns?
(d) How should the support counting step be modified to find such patterns?
ClusterAnalysis:
BasicConceptsand
Algorithms
Cluster analysis divides data into groups (clusters) that are meaningful, useful,
or both. Ifmeaningful groups are the goal, then the clusters should capture the
natural structure of the data. In some cases,however,cluster analysis is only a
useful starting point for other purposes,such as data summarization. Whether
for understanding or utility, cluster analysis has long played an important
role in a wide variety of fields: psychology and other social sciences, biology,
statistics, pattern recognition, information retrieval, machine learning, and
data mining.
There have been many applications of cluster analysis to practical prob-
lems. We provide some specific examples, organized by whether the purpose
of the clustering is understanding or utility.

Clustering for Understanding Classes,or conceptually meaningful groups

of objects that share common characteristics, play an important role in how
people analyze and describe the world. Indeed, human beings are skilled at
dividing objects into groups (clustering) and assigning particular objects to
these groups (classification). For example, even relatively young children can
quickly label the objects in a photograph as buildings, vehicles, people, ani-
mals, plants, etc. In the context of understanding data, clusters are potential
classesand cluster analysis is the study of techniques for automatically finding
classes. The following are some examples:
488 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

Biology. Biologists have spent many years creating a taxonomy (hi-

erarchical classification) of all living things: kingdom, phylum, class,
order, family, genus)and species.Thus, it is perhaps not surprising that
much of the early work in cluster analysis sought to create a discipline
of mathematical taxonomy that could automatically find such classifi-
cation structures. More recently, biologists have applied clustering to
analyze the large amounts of genetic information that are now available.
For example, clustering has been used to find groups of genesthat have
similar functions.

Information Retrieval. The World Wide Web consists of billions of

Web pages, and the results of a query to a search engine can return
thousands of pages. Clustering can be used to group these search re-
sults into a small number of clusters, each of which captures a particular
aspect of the query. For instance, a query of "movie" might return
Web pages grouped into categories such as reviews, trailers, stars, and
theaters. Each category (cluster) can be broken into subcategories(sub-
clusters), producing a hierarchical structure that further assistsa user's
exploration of the query results.

Climate. Understanding the Earth's climate requires finding patterns

in the atmosphere and ocean. To that end, cluster analysis has been
applied to find patterns in the atmospheric pressureof polar regions and
areas of the ocean that have a significant impact on land climate.

Psychology and Medicine. An illness or condition frequently has a

number of variations, and cluster analysis can be used to identify these
different subcategories.For example, clustering has been used to identify
different types of depression. Cluster analysis can also be used to detect
patterns in the spatial or temporal distribution of a disease.

Business. Businessescollect large amounts of information on current

and potential customers. Clustering can be used to segment customers
into a small number of groups for additional analysis and marketing
activities.

Clustering for Utility Cluster analysis provides an abstraction from in-

dividual data objects to the clusters in which those data objects reside. Ad-
ditionally, some clustering techniques characterize each cluster in terms of a
cluster prototype; i.e., a data object that is representative of the other ob-
jects in the cluster. These cluster prototypes can be used as the basis for a
 489

number of data analysis or data processingtechniques. Therefore, in the con-

text of utility, cluster analysis is the study of techniques for finding the most
representativecluster prototypes.

o Summarization. Many data analysis techniques,such as regressionor

PCA, have a time or space complexity of O(m2) or higher (where rn is
the number of objects), and thus, are not practical for large data sets.
However, instead of applying the algorithm to the entire data set, it can
be applied to a reduced data set consisting only of cluster prototypes.
Depending on the type of analysis, the number of prototypes, and the
accuracy with which the prototypes represent the data, the results can
be comparable to those that would have been obtained if all the data
could have been used.

o Compression. Cluster prototypes can also be used for data compres-

sion. In particular, a table is created that consists of the prototypes for
each clusterl i.e., each prototype is assignedan integer value that is its
position (index) in the table. Each object is represented by the index
of the prototype associatedwith its cluster. This type of compressionis
known as vector quantization and is often applied to image, sound,
and video data, where (1) many of the data objects are highly similar
to one another, (2) some loss of information is acceptable, and (3) a
substantial reduction in the data size is desired.

o Efficiently Finding Nearest Neighbors. Finding nearest neighbors

can require computing the pairwise distance between all points. Often
clusters and their cluster prototypes can be found much more efficiently.
Ifobjects are relatively closeto the prototype oftheir cluster, then we can
use the prototypes to reduce the number of distance computations that
are necessaryto find the nearest neighbors of an object. Intuitively, if two
cluster prototypes are far apart, then the objects in the corresponding
clusters cannot be nearest neighbors of each other. Consequently, to
find an object's nearest neighbors it is only necessaryto compute the
distance to objects in nearby clusters, where the nearnessof two clusters
is measuredby the distance between their prototypes. This idea is made
more precise in Exercise 25 on page 94.

This chapter provides an introduction to cluster analysis. We begin with

a high-level overview of clustering, including a discussion of the various ap-
proaches to dividing objects into sets of clusters and the different types of
clusters. We then describe three specific clustering techniques that represent
49O Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

broad categoriesof algorithms and illustrate a variety of concepts: K-means,

agglomerative hierarchical clustering, and DBSCAN. The final section of this
chapter is devoted to cluster validity-methods for evaluating the goodness
of the clusters produced by a clustering algorithm. More advanced clustering
concepts and algorithms will be discussedin Chapter 9. Whenever possible,
we discuss the strengths and weaknessesof different schemes. In addition,
the bibliographic notes provide referencesto relevant books and papers that
explore cluster analysis in greater depth.

8.1 Overview
Before discussing specific clustering techniques, we provide some necessary
background. First, we further define cluster analysis, illustrating why it is
difficult and explaining its relationship to other techniques that group data.
Then we explore two important topics: (1) different ways to group a set of
objects into a set of clusters, and (2) types of clusters.

8.1.1 What Is Cluster Analysis?

Cluster analysis groups data objects based only on information found in the
data that describesthe objects and their relationships. The goal is that the
objects within a group be similar (or related) to one another and different from
(or unrelated to) the objects in other groups. The greater the similarity (or
homogeneity) within a group and the greater the difference between groups,
the better or more distinct the clustering.
In many applications, the notion of a cluster is not well defined. To better
understand the difficulty of deciding what constitutes a cluster, considerFigure
8.1, which shows twenty points and three different ways of dividing them into
clusters. The shapes of the markers indicate cluster membership. Figures
8.1(b) and 8.1(d) divide the data into two and six parts, respectively. However,
the apparent division of each of the two larger clusters into three subclusters
may simply be an artifact of the human visual system. AIso, it may not be
unreasonable to say that the points form four clusters, as shown in Figure
8.1(c). This figure illustrates that the definition of a cluster is imprecise and
that the best definition depends on the nature of data and the desired results.
Cluster analysis is related to other techniques that are used to divide data
objects into groups. For instance, clustering can be regarded as a form of
classification in that it creates a labeling of objects with class (cluster) labels.
However, it derives these labels only from the data. In contrast, classification
 8.1 Overview 491

aa a
aa a aa - tt.- ^:^
oo lr a
aa aa !o rr ^^i^
a aa I

(a) Original points. (b) Two clusters,

++
'++ * ++*
+i + ** -]rl.
vv
V
?
.l' !o ,,**!.

(c) Four clusters. (d) Six clusters.

Figure8.1. Different thesamesetof points.

waysof clustering

in the senseof Chapter 4 is supervised classification; i.e., new, unlabeled

objects are assigneda class label using a model developed from objects with
known class labels. For this reason, cluster analysis is sometimes referred
to as unsupervised classification. When the term classification is used
without any qualification within data mining, it typically refers to supervised
classification.
Also, while the terms segmentation and partitioning are sometimes
used as synonyms for clustering, these terms are frequently used for approaches
outside the traditional bounds of cluster analysis. For example, the term
partitioning is often used in connection with techniquesthat divide graphs into
subgraphs and that are not strongly connected to clustering. Segmentation
often refers to the division of data into groups using simple techniques; e.9.,
an image can be split into segments based only on pixel intensity and color, or
people can be divided into groups based on their income. Nonetheless,some
work in graph partitioning and in image and market segmentation is related
to cluster analysis.

8.1.2 Different Types of Clusterings

An entire collection of clusters is commonly referred to as a clustering, and in

this section, we distinguish various types of clusterings: hierarchical (nested)
versus partitional (unnested), exclusive versus overlapping versus finzy, and
complete versus partial.

Hierarchical versus Partitional The most commonly discusseddistinc-

tion among different types of clusterings is whether the set of clusters is nested
492 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

or unnested, or in more traditional terminology, hierarchical or partitional. A

partitional clustering is simply a division of the set of data objects into
non-overlapping subsets (clusters) such that each data object is in exactly one
subset. Taken individually, each collection of clusters in Figures 8.1 (b-d) is
a partitional clustering.
If we permit clusters to have subclusters, then we obtain a hierarchical
clustering, which is a set of nested clusters that are organized as a tree. Each
node (cluster) in the tree (except for the leaf nodes) is the union of its children
(subclusters), and the root ofthe tree is the cluster containing all the objects.
Often, but not always, the leavesof the tree are singleton clusters of individual
data objects. If we allow clusters to be nested, then one interpretation of
Figure 8.1(a) is that it has two subclusters(Figure 8.1(b)), each of which, in
turn, has three subclusters(Figure 8.1(d)). The clustersshown in Figures 8.1
(u-d), when taken in that order, also form a hierarchical (nested) clustering
with, respectively, I, 2, 4, and 6 clusters on each level. Finally, note that a
hierarchical clustering can be viewed as a sequence of partitional ciusterings
and a partitional clustering can be obtained by taking any member of that
sequence;i.e., by cutting the hierarchical tree at a particular level.

Exclusive versus Overlapping versus Ftzzy The clusterings shown in

Figure 8.1 are all exclusive, as they assign each object to a single cluster.
There are many situations in which a point could reasonablybe placed in more
than one cluster, and these situations are better addressedby non-exclusive
clustering. In the most general sense, an overlapping or non-exclusive
clustering is used to reflect the fact that an object can si,multaneouslybelong
to more than one group (class). For instance, a person at a university can be
both an enrolled student and an employee of the university. A non-exclusive
clustering is also often used when, for example, an object is "between" two
or more clusters and could reasonably be assigned to any of these clusters.
Imagine a point halfway between two of the clusters of Figure 8.1. Rather
than make a somewhat arbitrary assignment of the object to a single cluster,
it is placed in all of the "equally good" clusters.
In a fizzy clustering, every object belongs to every cluster with a mem-
bership weight that is between 0 (absolutely doesn't belong) and 1 (absolutely
belongs). In other words, clusters are treated as finzy sets. (Mathematically,
a fizzy set is one in which an object belongs to any set with a weight that
is between 0 and t. In fiizzy clustering, we often impose the additional con-
straint that the sum of the weights for each object must equal 1.) Similarly,
probabilistic clustering techniques compute the probability with which each
 8.1 Overview 493

point belongs to each cluster, and these probabilities must also sum to 1. Be-
causethe membership weights or probabilities for any object sum to 1, a finzy
or probabilistic clustering does not addresstrue multiclass situations, such as
the case of a student employee, where an object belongs to multiple classes.
Instead, these approachesare most appropriate for avoiding the arbitrariness
of assigning an object to only one cluster when it may be close to several. In
practice, a htzzy or probabilistic clustering is often converted to an exclusive
clustering by assigning each object to the cluster in which its membership
weight or probability is highest.

Complete versus Partial A complete clustering assignsevery object to

a cluster, whereasa partial clustering does not. The motivation for a partial
clustering is that some objects in a data set may not belong to well-defined
groups. Many times objects in the data set may represent noise, outliers, or
"uninteresting background." For example, some newspaper stories may share
a common theme, such as global warming, while other stories are more generic
or one-of-a-kind. Thus, to find the important topics in last month's stories, we
may want to search only for clusters of documents that are tightly related by a
common theme. In other cases,a complete clustering of the objects is desired.
For example, an application that uses clustering to organize documents for
browsing needs to guarantee that all documents can be browsed.

8.1.3 Different Types of Clusters

Clustering aims to find useful groups of objects (clusters), where usefulnessis

defined by the goals of the data analysis. Not surprisingly, there are several
different notions of a cluster that prove useful in practice. In order to visually
illustrate the differencesamong these types of clusters, we use two-dimensional
points, as shown in Figure 8.2, as our data objects. We stress, however, that
the types of clusters described here are equally valid for other kinds of data.

Well-Separated A cluster is a set of objects in which each object is closer

(or more similar) to every other object in the cluster than to any object not
in the cluster. Sometimes a threshold is used to specify that all the objects in
a cluster must be sufficiently close (or similar) to one another. This idealistic
definition of a cluster is satisfied only when the data contains natural clusters
that are quite far from each other. Figure 8.2(a) gives an example of well-
separated clusters that consists of two groups of points in a two-dimensional
space. The distance between any two points in different groups is larger than
494 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

the distance between any two points within a group. Well-separated clusters
do not need to be globular, but can have any shape.

Prototype-Based A cluster is a set of objects in which each object is closer

(more similar) to the prototype that defines the cluster than to the prototype
of any other cluster. For data with continuous attributes, the prototype of a
cluster is often a centroid, i.e., the average(mean) of all the points in the clus-
ter. When a centroid is not meaningful, such as when the data has categorical
attributes, the prototype is often a medoid, i.e., the most representativepoint
of a cluster. For many types of data, the prototype can be regarded as the
most central point, and in such instances, we commonly refer to prototype-
based clusters as center-based clusters. Not surprisingly, such clusters tend
to be globular. Figure 8.2(b) shows an example of center-basedclusters.

Graph-Based If the data is represented as a graph, where the nodes are

objects and the links represent connections among objects (seeSection 2.7.2),
then a cluster can be defined as a connected component; i.e., a group of
objects that are connected to one another, but that have no connection to
objects outside the group. An important example of graph-based clusters are
contiguity-based clusters, where two objects are connected only if they are
within a specified distance of each other. This implies that each object in a
contiguity-based cluster is closer to some other object in the cluster than to
any point in a different cluster. Figure 8.2(c) showsan example of such clusters
for two-dimensional points. This definition of a cluster is useful when clusters
are irregular or intertwined, but can have trouble when noise is present since,
as illustrated by the two spherical clusters of Figure 8.2(c), a small bridge of
points can merge two distinct clusters.
Other types of graph-based clusters are also possible. One such approach
(Section 8.3.2) defines a cluster as a cliquel i.e., a set of nodes in a graph that
are completely connected to each other. Specifically, if we add connections
between objects in the order of their distance from one another, a cluster is
formed when a set of objects forms a clique. Like prototype-based clusters,
such clusters tend to be elobular.

Density-Based A cluster is a dense region of objects that is surrounded by

a region of low density. Figure 8.2(d) shows some density-based clusters for
data created by adding noise to the data of Figure 8.2(c). The two circular
clusters are not merged, as in Figure 8.2(c), becausethe bridge between them
fades into the noise. Likewise, the curve that is present in Figure 8.2(c) also
 8.1 Overview 495

fades into the noise and does not form a cluster in Figure 8.2(d). A density-
baseddefinition of a cluster is often employed when the clusters are irregular or
intertwined, and when noise and outliers are present. By contrast, a contiguity-
based definition of a cluster would not work well for the data of Figure 8.2(d)
since the noise would tend to form bridses between clusters.

Shared-Property (Conceptual Clusters) More generally, we can define

a cluster as a set of objects that share some property. This definition encom-
passesall the previous definitions of a cluster; e.g., objects in a center-based
cluster share the property that they are all closest to the same centroid or
medoid. However, the shared-property approach also includes new types of
clusters. Consider the clusters shown in Figure 8.2(e). A triangular area
(cluster) is adjacent to a rectangular one, and there are two intertwined circles
(clusters). In both cases, a clustering algorithm would need a very specific
concept of a cluster to successfullydetect these clusters. The processof find-
ing such clusters is called conceptual clustering. However, too sophisticated
a notion of a cluster would take us into the area of pattern recognition, and
thus, we only consider simpler types of clusters in this book.

Road Map

In this chapter, we use the following three simple, but important techniques
to introduce many of the concepts involved in cluster analysis.

o K-means. This is a prototype-based, partitional clustering technique

that attempts to find a user-specifiednumber of clusters (K), which are
representedby their centroids.

o Agglomerative Hierarchical Clustering. This clustering approach

refers to a collection of closely related clustering techniquesthat produce
a hierarchical clustering by starting with each point as a singleton cluster
and then repeatedly merging the two closest clusters until a single, all-
encompassingcluster remains. Some of these techniques have a natural
interpretation in terms of graph-based clustering, while others have an
interpretation in terms of a prototype-based approach.

o DBSCAN. This is a density-basedclustering algorithm that produces

a partitional clustering, in which the number of clusters is automatically
determined by the algorithm. Points in low-density regions are classi-
fied as noise and omitted; thus, DBSCAN does not produce a complete
clustering.
496 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

(a) Well-separated clusters. Each (b) Center-based clusters. Each

point is closer to all of the points in its point is closer to the center of its
cluster than to any point in another cluster than to the center of anv
cluster. other cluster.

:: : :: ; : : : : : : : : ::iiill,'lN: : : :: : : :
: : : : : : : : ; : : : ; : : / l i i i l i i i i i i i i i:i:N: ; : :
: ::::i :.:::i :: :i l i i i i i : i i i i i i i i ,i i i r i i l i , i ,
:' : : : : : : : \ , i ' l , i , i , i , i ,: i: .: l: t: :
:,:,,
:,i':'i'ii:'i,ilil.rir4:,

(c) Contiguity-based clusters. Each (d) Density-based clusters. Clus-

point is closer to at least one point ters are regions of high density sep-
in its cluster than to anv point in arated by regions of low density.
another cluster.

(e) Conceptual clusters. Points in a cluster share some general

property that derives from the entire set of points. (Points in the
intersection of the circles belong to both.)

Figure
8.2.Different
types
ofclusters
asillustrated points.
bysetsoftwo-dimensional

8.2 K-means
Prototype-based clustering techniques create a one-level partitioning of the
data objects. There are a number of such techniques, but two of the most
prominent are K-means and K-medoid. K-means defines a prototype in terms
of a centroid, which is usually the mean of a group of points, and is typically
 8.2 K-means 497

applied to objects in a continuous n-dimensional space. K-medoid defines a

prototype in terms of a medoid, which is the most representative point for a
group of points, and can be applied to a wide range of data since it requires
only a proximity measure for a pair of objects. While a centroid almost never
corresponds to an actual data point, a medoid, by its definition, must be an
actual data point. In this section, we will focus solely on K-means, which is
one of the oldest and most widely used clustering algorithms.

8.2.1 The Basic K-means Algorithm

The K-means clustering technique is simple, and we begin with a description

of the basic algorithm. We first chooseK initial centroids, where l( is a user-
specified parameter, namely, the number of clusters desired. Each point is
then assignedto the closest centroid, and each collection of points assignedto
a centroid is a cluster. The centroid of each cluster is then updated based on
the points assignedto the cluster. We repeat the assignmentand update steps
until no point changes clusters, or equivalently, until the centroids remain the
same.
K-means is formally describedby Algorithm 8.1. The operation of K-means
is illustrated in Figure 8.3, which showshow, starting from three centroids, the
final clusters are found in four assignment-update steps. In these and other
figures displaying K-means clustering, each subfigure shows (1) the centroids
at the start of the iteration and (2) the assignment of the points to those
centroids. The centroids are indicated by the "*" symbol; all points belonging
to the same cluster have the same marker shape.

Algorithm 8.1 Basic K-means algorithm.

1: Select K points as initial centroids.
2: repeat
3: Form K clusters by assigning each point to its closest centroid.
4: Recompute the centroid of each cluster.
5: until Centroids do not change.

In the first step, shown in Figure 8.3(a), points are assignedto the initial
centroids, which are all in the larger group of points. For this example, we use
the mean as the centroid. After points are assignedto a centroid, the centroid
is then updated. Again, the figure for each step shows the centroid at the
beginning of the step and the assignment of points to those centroids. In the
second step, points are assignedto the updated centroids, and the centroids
498 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

AAAAAA AA

trDA A
tr o + o+ o o tr
otr ^ tro ^ otr | trQ
o-oo 8- :.coP9 o--oo 8" b3 P
s o-{oo 8- :.cTPS o-dP o
tr"o- tr"o- tr'o-
o "oo o "oo o "oo tr

(a) Iteration 1. (b) Iteration 2. (c) Iteration 3. (d) Iteration 4.

Figure8.3. UsingtheK-means
algorithm insample
to findthreeclusters data.

are updated again. In steps 2, 3, and 4, which are shown in Figures 8.3 (b),
(c), and (d), respectively,two of the centroids move to the two small groups of
points at the bottom of the figures. When the K-means algorithm terminates
in Figure 8.3(d), becauseno more changesoccur, the centroids have identified
the natural groupings of points.
For some combinations of proximity functions and types of centroids, K-
means always convergesto a solution; i.e., K-means reachesa state in which no
points are shifting from one cluster to another, and hence, the centroids don't
change. Becausemost of the convergenceoccurs in the early steps, however,
the condition on line 5 of Algorithm 8.1 is often replacedby a weaker condition,
e.g., repeat until only l% of the points change clusters.
We considereach of the steps in the basic K-means algorithm in more detail
and then provide an analysis of the algorithm's space and time complexity.

Assigning Points to the Closest Centroid

To assign a point to the closest centroid, we need a proximity measure that

quantifies the notion of "closest" for the specific data under consideration.
Euclidean (L2) distance is often used for data points in Euclidean space,while
cosine similarity is more appropriate for documents. However, there may be
several types of proximity measures that are appropriate for a given type of
data. For example, Manhattan (L1) distance can be used for Euclidean data,
while the Jaccard measure is often employed for documents.
Usually, the similarity measures used for K-means are relatively simple
since the algorithm repeatedly calculates the similarity of each point to each
centroid. In some cases.however. such as when the data is in low-dimensional
 8.2 K-means 499

Table8.1.Table
of notation.
Svmbol Description
x An object.
Ci The zth cluster.
Ci The centroid of cluster C6.
c The centroid of all points.
T11,4 The number of objects in the ith cluster.
TI-L The number of objects in the data set.
K The number of clusters.

Euclidean space, it is possible to avoid computing many of the similarities,

thus significantly speeding up the K-means algorithm. Bisecting K-means
(described in Section 8.2.3) is another approach that speeds up K-means by
reducing the number of similarities computed.

Centroids and Objective F\rnctions

Step 4 of the K-means algorithm was stated rather generally as "recompute

the centroid of each cluster," since the centroid can vary, depending on the
proximity measure for the data and the goal of the clustering. The goal of
the clustering is typically expressedby an objective function that depends on
the proximities of the points to one another or to the cluster centroids; e.9.,
minimize the squared distance of each point to its closest centroid. We illus-
trate this with two examples. However, the key point is this: once we have
specified a proximity measure and an objective function, the centroid that we
should choose can often be determined mathematically. We provide mathe-
matical details in Section 8.2.6, and provide a non-mathematical discussionof
this observation here.

Data in Euclidean Space Consider data whose proximity measure is Eu-

clidean distance. For our objective function, which measuresthe quality of a
clustering, we use the sum of the squared error (SSE), which is also known
as scatter. In other words, we calculate the error of each data point, i.e., its
Euclidean distance to the closest centroid, and then compute the total sum
of the squared errors. Given two different sets of clusters that are produced
by two different runs of K-means, we prefer the one with the smallest squared
error since this means that the prototypes (centroids) of this clustering are
a better representation of the points in their cluster. Using the notation in
Table 8.1, the SSE is formally defined as follows:
500 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

K
SSE: t t d,i,st( ci,r ) 2 (8.1)
i:t r€Ci

where di,st is the standard Euclidean (Lz) distance between two objects in
Euclidean space.
Given these assumptions, it can be shown (see Section 8.2.6) that the
centroid that minimizes the SSE of the cluster is the mean. Using the notation
in Table 8.1, the centroid (mean) of the ith cluster is defined by Equation8.2.

I
Ci:
mi t*
xeC,i.
(8.2)

To illustrate, the centroid of a cluster containing the three two-dimensional

p o i n t s ,( 1 , 1 ) ,( 2 , 3 ) ,a n d ( 6 , 2 ) ,i s ( ( 1 + 2 + 6 ) 1 3 , ( ( 1+ 3 + 2 ) 1 3 ): ( 3 , 2 ) .
Steps 3 and 4 of the K-means algorithm directly attempt to minimize
the SSE (or more generally, the objective function). Step 3 forms clusters
by assigning points to their nearest centroid, which minimizes the SSE for
the given set of centroids. Step 4 recomputes the centroids so as to further
minimize the SSE. However, the actions of K-means in Steps 3 and 4 are only
guaranteed to find a local minimum with respect to the SSE since they are
based on optimizing the SSE for specific choicesof the centroids and clusters,
rather than for all possible choices. We will later seean example in which this
Ieads to a suboptimal clustering.

Document Data To illustrate that K-means is not restricted to data in

Euclidean space,we consider document data and the cosinesimilarity measure.
Here we assume that the document data is represented as a document-term
matrix as described on page 31. Our objective is to maximizethe similarity
of the documents in a cluster to the cluster centroid; this quantity is known
as the cohesion of the cluster. For this objective it can be shown that the
cluster centroid is, as for Euclidean data, the mean. The analogous quantity
to the total SSE is the total cohesion,which is given by Equation 8.3.

h-
\-a \-
Iotaluoneslon: )
ZJ ZJ
) cosxnelx,c;.) (8.3)
i:I x€C,i

The General Case There are a number of choicesfor the proximity func-
tion, centroid, and objective function that can be used in the basic K-means
 8.2 K-means 501

Table8.2. K-means:
Common forproximity,
choices andobjective
centroids, functions.
Proximity Fbnction Centroid Obiective F\rnction
Manhattan (L1) median Minimize sum of the Lr distance of an ob-
iect to its cluster centroid
Squared Euclidean (L!) mean Minimize sum of the squared L2 distance
of an obiect to its cluster centroid
coslne mean Maximize sum of the cosine similarity of
an object to its cluster centroid
Bregman divergence mean Minimize sum of the Bregman divergence
of an obiect to its cluster centroid

algorithm and that are guaranteed to converge. Table 8.2 shows some possible
choices,including the two that we have just discussed. Notice that for Man-
hattan (L1) distance and the objective of minimizing the sum of the distances,
the appropriate centroid is the median of the points in a cluster.
The last entry in the table, Bregman divergence(Section 2.4.5), is actually
a classof proximity measuresthat includes the squared Euclidean distance, Ll,
the Mahalanobis distance, and cosine similarity. The importance of Bregman
divergence functions is that any such function can be used as the basis of a K-
means style clustering algorithm with the mean as the centroid. Specifically,
if we use a Bregman divergence as our proximity function, then the result-
ing clustering algorithm has the usual properties of K-means with respect to
convergence,local minima, etc. Furthermore) the properties of such a cluster-
ing algorithm can be developed for all possible Bregman divergences.Indeed,
K-means algorithms that use cosine similarity or squared Euclidean distance
are particular instances of a general clustering algorithm based on Bregman
divergences.
For the rest our K-means discussion, we use two-dimensional data since
it is easy to explain K-means and its properties for this type of data. But,
as suggested by the last few paragraphs, K-means is a very general clustering
algorithm and can be used with a wide variety of data types, such as documents
and time series.

Choosing Initial Centroids

When random initialization of centroids is used, different runs of K-means

typically produce different total SSEs. We illustrate this with the set of two-
dimensional points shown in Figure 8.3, which has three natural clusters of
points. Figure 8.4(a) shows a clustering solution that is the global minimum of
5O2 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

AA AA
rA n
AAN A
^aA
a
o +1o!loP
o
a
"9UvOvn
^o"o n"
"oo
A o
o --
trtr trtr tr on
u nu O

*trtr
o tr

(a) Optimal clustering (b) Suboptimal clustering.

Figure8.4.Threeoptimal clusters.
andnon-optimal

the SSE for three clusters, while Figure 8.4(b) shows a suboptimal clustering
that is only a local minimum.
Choosing the proper initial centroids is the key step of the basic K-means
procedure. A common approach is to choose the initial centroids randomly,
but the resulting clusters are often poor.
Exarnple 8.1 (Poor Initial Centroids). Randomly selected initial cen-
troids may be poor. We provide an example of this using the same data set
used in Figures 8.3 and 8.4. Figures 8.3 and 8.5 show the clusters that re-
sult from two particular choices of initial centroids. (For both figures, the
positions of the cluster centroids in the various iterations are indicated by
crosses.)In Figure 8.3, even though all the initial centroids are from one natu-
ral cluster, the minimum SSE clustering is still found. In Figure 8.5, however,
even though the initial centroids seem to be better distributed, we obtain a
suboptimal clustering, with higher squared error. r
Example 8.2 (Lirnits of Random Initialization). One technique that
is commonly used to address the problem of choosing initial centroids is to
perform multiple runs, each with a different set of randomly chosen initial
centroids, and then select the set of clusters with the minimum SSE. While
simple, this strategy may not work very well, depending on the data set and
the number of clusters sought. We demonstrate this using the sample data set
shown in Figure 8.6(a). The data consists of two pairs of clusters, where the
clusters in each (top-bottom) pair are closer to each other than to the clusters
in the other pair. Figure 8.6 (b d) shows that if we start with two initial
centroids per pair of clusters, then even when both centroids are in a single
 8.2 K-means 503

AA
r
a ^aa ^-a
^A
A A "AAA ^
^61 ofo.

" s"."&":4. ' q:':f".4

a o

vo"vO t o
-OO o"o-
o
tr tr Dtr tr
otr ntr tro oo
v06 o on
o -So oooi "oo "- o* Utr- tr -tro tr -oo ?_oo
oo o oo Eo o
o
tr o tr o

(a) Iteration 1. (b) Iteration 2, (c) Iteration 3. (d) Iteration 4.

Figure8.5. Poorstarting forK-means.

centroids

cluster, the centroids will redistribute themselves so that the "true" clusters
are found. However, Figure 8.7 shows that if a pair of clusters has only one
initial centroid and the other pair has three, then two of the true clusters will
be combined and one true cluster will be split.
Note that an optimal clustering will be obtained as long as two initial
centroids fall anywhere in a pair of clusters, since the centroids will redistribute
themselves, one to each cluster. Unfortunately, as the number of clusters
becomes larger, it is increasingly likely that at least one pair of clusters will
have only one initial centroid. (See Exercise 4 on page 559.) In this case,
becausethe pairs of clusters are farther apart than clusters within a pair, the
K-means algorithm will not redistribute the centroids between pairs of clusters,
and thus, only a local minimum will be achieved. r
Because of the problems with using randomly selected initial centroids,
which even repeated runs may not overcome, other techniques are often em-
ployed for initialization. One effective approach is to take a sample of points
and cluster them using a hierarchical clustering technique. K clusters are ex-
tracted from the hierarchical clustering, and the centroids of those clusters are
used as the initial centroids. This approach often works well, but is practical
only if (1) the sample is relatively small, €.9., a few hundred to a few thousand
(hierarchical clustering is expensive), and (2) K is relatively small compared
to the sample size.
The following procedure is another approach to selecting initial centroids.
Select the first point at random or take the centroid of all points. Then, for
each successiveinitial centroid, select the point that is farthest from any of
the initial centroids already selected. In this way, we obtain a set of initial
5O4 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

A A o A A o
aoa aOO AAA aOO
A -aaAA A AA^ o-^oo
a
aa^ o"o
aaa a ooo AAA A oov
A vo A O9
o V

tr A
tr trtr tr {atr
trn tr a- tr ^oo
- v _9
:
_ Bu
nn
o "o ifo oo {Fot-
trtr trtr V\/
v

(a) Initial points. (b) Iteration 1.

A A A A
A^
AAA AA
A AA^ A

aaA ooo
f;Sa
AAA
a A
t Oln
-o A

tr tr
trtr tr tr otr tr
trn tr
oo{o trd-oo
D'tr trtr
otr trtr

(c) Iteration 2. (d) Iteration 3.

Figure8.6,Twopairsof clusters
witha pairofinitial withineachpairofclusters.
centroids

centroids that is guaranteed to be not only randomly selected but also well
separated. Unfortunately, such an approach can select outliers, rather than
points in denseregions (clusters). AIso, it is expensiveto compute the farthest
point from the current set of initial centroids. To overcome these problems?
this approach is often applied to a sample of the points. Since outliers are
rare, they tend not to show up in a random sample. In contrast, points
from every dense region are likely to be included unless the sample size is very
small. Also, the computation involved in finding the initial centroids is greatly
reduced becausethe sample size is typically much smaller than the number of
points.
Later on, we will discusstwo other approachesthat are useful for produc-
ing better-quality (lower SSE) clusterings: using a variant of K-means that
 8.2 K-means 505

A A A
A A^
AA AA ^Oo
A A oot o o-^o o
aA^ o "{o
AA.
aaA aaA a ooo
a
oo
+ A
o
+
A A
AAA AAA
Aa A Ar A
A AA ^
aa?n
AA. AA-
A4 Aa

(a) Iteration 1. (b) Iteration 2

A A A o
A A^
A^
AA A.
A o-^o o A o"^oo
AA^
AA^ o"+o AA^
aa^ o "+o
AAA A ooo aaA a ooo
A vo A
o o
+ +
A A
AAAA AAAA
An A A -trv
u
u.
^A A vrV
aa-An o-+'-w
AA_ AA_ -uv
AA AA otrv
o

(c) Iteration 3. (d) Iteration 4.

Figure8.7.Twopairsofclusters withina pairofclusters.

withmoreorfewerthantwoinitialcentroids

is less susceptible to initialization problems (bisecting K-means) and using

postprocessingto "fixup" the set of clusters produced.

Time and Space Complexity

The space requirements for K-means are modest because only the data points
and centroids are stored. Specifically, the storage required is O((m-l K)n),
where rn is the number of points and n is the number of attributes. The time
requirements for K-means are also modest-basically linear in the number of
data points. In particular, the time required is O(1x K +mxn), where 1is the
number of iterations required for convergence. As mentioned, 1 is often small
and can usually be safely bounded, as most changes typically occur in the
506 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

first few iterations. Therefore, K-means is linear in rn, the number of points,
and is efficient as well as simple provided that K, the number of clusters, is
significantly less than m.

8.2.2 K-means: Additional Issues

Handling Empty Clusters

One of the problems with the basic K-means algorithm given earlier is that
empty clusters can be obtained if no points are allocated to a cluster during
the assignment step. If this happens, then a strategy is needed to choose a
replacement centroid, since otherwise, the squared error will be larger than
necessary. One approach is to choose the point that is farthest away from
any current centroid. If nothing else, this eliminates the point that currently
contributes most to the total squared error. Another approach is to choose
the replacement centroid from the cluster that has the highest SSE. This will
typically split the cluster and reduce the overall SSE of the clustering. If there
are several empty clusters, then this processcan be repeated several times.

Outliers

When the squared error criterion is used, outliers can unduly influence the
clusters that are found. In particular, when outliers are present, the resulting
cluster centroids (prototypes) -uy not be as representative as they otherwise
would be and thus, the SSE will be higher as well. Becauseof this, it is often
useful to discover outliers and eliminate them beforehand. It is important,
however, to appreciate that there are certain clustering applications for which
outliers should not be eliminated. When clustering is used for data com-
pression, every point must be clustered, and in some cases,such as financial
analysis, apparent outliers, e.g., unusually profitable customers, can be the
most interesting points.
An obvious issue is how to identify outliers. A number of techniques for
identifying outliers will be discussedin Chapter 10. If we use approachesthat
remove outliers before clustering, we avoid clustering points that will not clus-
ter well. Alternatively, outliers can also be identified in a postprocessingstep.
For instance) we can keep track of the SSE contributed by each point, and
eliminate those points with unusually high contributions, especially over mul-
tiple runs. Also, we may want to eliminate small clusters since they frequently
represent groups of outliers.
 8.2 K-means 5O7

Reducing the SSE with Postprocessing

An obvious way to reduce the SSE is to find more clusters, i.e., to use a larger
K. However, in many cases, we would like to improve the SSE, but don't
want to increase the number of clusters. This is often possible because K-
means typically convergesto a local minimum. Various techniques are used
to "fix up" the resulting clusters in order to produce a clustering that has
lower SSE. The strategy is to focus on individual clusters since the total SSE
is simply the sum of the SSE contributed by each cluster. (We will use the
terminology total SSE and cluster ,9^9E,respectively, to avoid any potential
confusion.) We can change the total SSE by performing various operations
on the clusters, such as splitting or merging clusters. One commonly used
approach is to use alternate cluster splitting and merging phases. During a
splitting phase, clusters are divided, while during a merging phase, clusters
are combined. In this way, it is often possibleto escapelocal SSE minima and
still produce a clustering solution with the desired number of clusters. The
following are some techniques used in the splitting and merging phases.
Two strategies that decreasethe total SSE by increasing the number of
clusters are the following:

Split a cluster: The cluster with the largest SSE is usually chosen, but we
could also split the cluster with the largest standard deviation for one
particular attribute.

Introduce a new cluster centroid: Often the point that is farthest from
any cluster center is chosen. We can easily determine this if we keep
track of the SSE contributed by each point. Another approach is to
choose randomly from all points or from the points with the highest
SSE.

Two strategies that decreasethe number of clusters, while trying to mini-

mize the increasein total SSE, are the following:

Disperse a cluster: This is accomplishedby removing the centroid that cor-

responds to the cluster and reassigningthe points to other clusters. Ide-
ally, the cluster that is dispersed should be the one that increasesthe
total SSE the Ieast.

Merge two clusters: The clusters with the closest centroids are typically
chosen,although another, perhaps better, approach is to merge the two
clusters that result in the smallest increase in total SSE. These two
merging strategies are the same ones that are used in the hierarchical
508 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

clustering techniquesknown as the centroid method and Ward's method,

respectively. Both methods are discussedin Section 8.3.

Updating Centroids Incrementally

Instead of updating cluster centroids after all points have been assignedto a
cluster, the centroids can be updated incrementally, after each assignment of
a point to a cluster. Notice that this requires either zero or two updates to
cluster centroids at each step, since a point either moves to a new cluster (two
updates) or stays in its current cluster (zero updates). Using an incremental
update strategy guarantees that empty clusters are not produced since all
clusters start with a single point, and if a cluster ever has only one point, then
that point will always be reassignedto the same cluster.
In addition, if incremental updating is used, the relative weight of the point
being added may be adjusted; e.g., the weight of points is often decreasedas
the clustering proceeds. While this can result in better accuracy and faster
convergence, it can be difficult to make a good choice for the relative weight,
especially in a wide variety of situations. These update issuesare similar to
those involved in updating weights for artificial neural networks.
Yet another benefit of incremental updates has to do with using objectives
other than "minimize SSE." Supposethat we are given an arbitrary objective
function to measure the goodness of a set of clusters. When we process an
individual point, we can compute the value of the objective function for each
possiblecluster assignment,and then choosethe one that optimizes the objec-
tive. Specific examples of alternative objective functions are given in Section
8.5.2.
On the negative side, updating centroids incrementally introduces an or-
der dependency. In other wotds, the clusters produced may depend on the
order in which the points are processed. Although this can be addressedby
randomizing the order in which the points are processed,the basic K-means
approach of updating the centroids after all points have been assignedto clus-
ters has no order dependency. Also, incremental updates are slightly more
expensive. However, K-means converges rather quickly, and therefore, the
number of points switching clusters quickly becomesrelatively small.

8.2.3 BisectingK-means

The bisecting K-means algorithm is a straightforward extension of the basic

K-means algorithm that is based on a simple idea: to obtain K clusters, split
the set of all points into two clusters, select one of these clusters to split, and
 8.2 K-means 509

so on, until K clusters have been produced. The details of bisecting K-means
are given by Algorithm 8.2.

Algorithm 8.2 Bisecting K-means algorithm.

1: Initialize the list of clusters to contain the cluster consisting of all points.
2: repeat
3: Remove a cluster from the list of clusters.
4: {Perform several "trial" bisections of the chosen cluster.}
5: for i : 7 to number of tri.als do
6: Bisect the selected cluster usins basic K-means.
7: end for
8: Select the two clusters from the bisection with the lowest total SSE.
9: Add these two clusters to the list of clusters.
10: until Until the list of clusters contains 1( clusters.

There are a number of different ways to choosewhich cluster to split. We

can choose the largest cluster at each step, choose the one with the largest
SSE, or use a criterion based on both size and SSE. Different choicesresult in
different clusters.
We often refine the resulting clusters by using their centroids as the initial
centroids for the basic K-means algorithm. This is necessarybecause,although
the K-means algorithm is guaranteedto find a clustering that representsa local
minimum with respect to the SSE, in bisecting K-means we are using the K-
means algorithm "locally," i.e., to bisect individual clusters. Therefore, the
final set of clusters does not represent a clustering that is a Iocal minimum
with respect to the total SSE.
Example 8.3 (Bisecting K-means and Initialization). To illustrate that
bisecting K-means is less susceptible to initialization problems, we show, in
Figure 8.8, how bisecting K-means finds four clusters in the data set originally
shown in Figure 8.6(a). In iteration 1, two pairs of clusters are found; in
iteration 2,the rightmost pair of clusters is split; and in iteration 3, the leftmost
pair of clusters is split. Bisecting K-means has less trouble with initialization
becauseit performs several trial bisections and takes the one with the lowest
SSE, and becausethere are only two centroids at each step. r
Finally, by recording the sequenceof clusterings produced as K-means
bisects clusters, we can also use bisecting K-means to produce a hierarchical
clustering.
51_0 Chapter Cluster Analysis: Basic Concepts and Algorithms

A o o o B
o
ooo aAA
o
ooo NE tr nE E
o oo8o oo8o oft E "-+E
ooo o AXA ooo o E'E B E'E E

A ^- A o EE trtr
o tr tr
+ + +
o o
OoOo ^rF a ooo o oo o an
ooo%?
ooo%3
o o-
OO
A, A-A
A; AA
O6
o o' "q%^:
o 6 a
"fr"4
a^
A A

(a) Iteration 1. (b) Iteration 2. (c) Iteration 3.

Figure8.8. Bisecting onthefourclusters

K-means example.

8.2.4 K-means and Different Types of Clusters

K-means and its variations have a number of limitations with respect to finding
different types of clusters. In particular, K-means has difficulty detecting the
"natural" clusters, when clusters have non-spherical shapesor widely different
sizesor densities. This is illustrated by Figures 8.9, 8.10, and 8.11. In Figure
8.9, K-means cannot find the three natural clusters becauseone of the clusters
is much larger than the other two, and hence,the larger cluster is broken, while
one of the smaller clusters is combined with a portion of the larger cluster. In
Figure 8.10, K-means fails to find the three natural clusters becausethe two
smaller clusters are much denser than the Iarger cluster. Finally, in Figure
8.11, K-means finds two clusters that mix portions of the two natural clusters
becausethe shape of the natural clusters is not globular.
The difficulty in these three situations is that the K-means objective func-
tion is a mismatch for the kinds of clusters we are trying to find since it is
minimized by globular clusters of equal size and density or by clusters that are
well separated. However) these limitations can be overcome, in some sense,if
the user is willing to accept a clustering that breaks the natural clusters into a
number of subclusters. Figure 8.12 shows what happens to the three previous
data sets if we find six clusters instead of two or three. Each smaller cluster is
pure in the sensethat it contains only points from one of the natural clusters.

8.2.5 Strengths and Weaknesses

K-means is simple and can be used for a wide variety of data types. It is also
quite efficient, even though multiple runs are often performed. Some variants,
including bisecting K-means, are even more efficient, and are less suscepti-
ble to initialization problems. K-means is not suitable for all types of data,
 8.2 K-means 51-1

ooo
o oo
ooo
o
oooooooo
o9 oo O9 Oo

(a) Original points. (b) Three K-meansclusters.

Figure8.9. K-means
withclusters
ofdifferent
size.

V
V V
A
ooo L
V Vv O^
A ^ O
too O
-E-
VVV 'vv -n
v
VV
"4+"\
^AAa o "o*o u -
-crtr-tr
rilif
n-
V w. V ooo oooo
VV V V V o +
-l
fi-
-,Untr
VVVV oooo
Vv
VVV ooo*.'o " u_uqa u
V V' 6 O-
vy -tr
v v o oo o
VV oo

(a) Original points. (b) Three K-means clusters.

Figure8.10.K-means
withclusters
ofdifferent
density.

-
_
LJ
tr T'I
tr" ntr tr

oo
ooo
uoO
a'\-

ro ooq
o oo o o" o
o ouo o
oo o

(a) Original points. (b) Two K-means clusters.

Figure8.11. K-means
withnon-globular
clusters.
5L2 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

oo o
opo
o o a\ o

og trtr

(a) Unequal sizes.

A
nA
o o o " " ,*' - 4 A a -ll
ln-
U-

-E-tr-l-Enn"
oio u nlf'l-
oFf,"
o ^ lA
ge O V v
v
vY7
o oo "
+ 17
na\ -
"# v v
o og v
OV

(b) Unequal densities

I
trDtr
o oo(

!+
N A^
n-D
v.
"* +A < '

zro
" o oooq$
^^ o-o
oo o
o o- ou o o n
\J

(c) Non-spherical shapes.

Figure8.12.UsingK-means
tofindclusters ofthenatural
thataresubclusters clusters.
 8.2 K-means 513

however. It cannot handle non-globular clusters or clusters of different sizes

and densities, although it can typically find pure subclusters if a large enough
number of clusters is specified. K-means also has trouble clustering data that
contains outliers. Outlier detection and removal can help significantly in such
situations. Finally, K-means is restricted to data for which there is a notion of
a center (centroid). A related technique, K-medoid clustering, does not have
this restriction, but is more expensive.

8.2.6 K-means as an Optimization Problem

Here, we delve into the mathematics behind K-means. This section, which can
be skipped without loss of continuity, requires knowledge of calculus through
partial derivatives. Familiarity with optimization techniques, especially those
based on gradient descent, may also be helpful.
As mentioned earlier, given an objective function such as "minimize SSE,'
clustering can be treated as an optimization problem. One way to solve this
problem-to find a global optimum is to enumerate all possible ways of di-
viding the points into clusters and then choose the set of clusters that best
satisfiesthe objective function, e.g., that minimizes the total SSE. Of course,
this exhaustive strategy is computationally infeasible and as a result, a more
practical approach is needed,even if such an approach finds solutions that are
not guaranteed to be optimal. One technique, which is known as gradient
descent, is based on picking an initial solution and then repeating the fol-
lowing two steps: compute the change to the solution that best optimizes the
objective function and then update the solution.
We assumethat the data is one-dimensional,i.e., di.st(r,A) : @ - A)2.
This does not change anything essential,but greatly simplifies the notation.

Derivation of K-means as an Algorithm to Minimize the SSE

In this section, we show how the centroid for the K-means algorithm can be
mathematically derived when the proximity function is Euclidean distance
and the objective is to minimize the SSE. Specifically,we investigate how we
can best update a cluster centroid so that the cluster SSE is minimized. In
mathematical terms, we seek to minimize Equation 8.1, which we repeat here,
specializedfor one-dimensionaldata.

K
sSE:ttk,-*)' (8.4)
i:l reCt
5L4 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

Herc, Ci is the ith cluster, r is a point in C,i, and c,;is the mean of the ith
cluster. SeeTable 8.1 for a complete list of notation.
We can solve for Lhe kth centroid c4, which minimizes Equation 8.4, by
differentiating the SSE, setting it equal to 0, and solving, as indicated below.

nK

9stu : o \ - \ - ("0 - ,)'

dcn orr?^fu,
K
rD
\-\- -L)
/ ,/'
i:I n€C.i.

\./r- r2- *+ ( q , - r n ) : 0
iL tU t1

1
* @n- rn): 0 + rrl1ac1,
:
D"r)(1,- D"r
D, mlc
reC6 r€Cp

Thus, as previously indicated, the best centroid for minimizing the SSE of
a cluster is the mean of the points in the cluster.

Derivation of K-means for SAE

To demonstrate that the K-means algorithm can be applied to a variety of

different objective functions, we consider how to partition the data into K
clusters such that the sum of the Manhattan (L1) distances of points from the
center of their clusters is minimized. We are seeking to minimize the sum of
the L1 absolute errors (SAE) as given by the following equation, where d,i.sty,
is the L1 distance. Again, for notational simplicity, we use one-dimensional
data, i.e., di,sfi., : lci - nl.

S A E :tt d,isty,( "t,r ) ( 8.5)

i:I r€Cl.

We can solve for the kth centroid c7r,which minimizes Equation 8.5, by
differentiating the SAE, setting it equal to 0, and solving.
 8.3 Agglomerative Hierarchical Clustering 5L5

3tou :
ocx

: t 3p1,-rl:s
,6ro"*

a\
a
T fiV- - rl :o *
c/1,
e
- cn)
sisn(r : o

If we solve for q", we find that qr: med'ian{r e Cp}, the median of the
points in the cluster. The median of a group of points is straightforward to
compute and less susceptibleto distortion by outliers.

8.3 Agglomerative Hierarchical Clustering

Hierarchical clustering techniques are a second important category of cluster-
ing methods. As with K-means, these approachesare relatively old compared
to many clustering algorithms, but they still enjoy widespread use. There are
two basic approachesfor generating a hierarchical clustering:

Agglomerative: Start with the points as individual clusters and, at each

step, merge the closest pair of clusters. This requires defining a notion
of cluster proximity.

Divisive: Start with one, all-inclusive cluster and, at each step, split a cluster
until only singleton clusters of individual points remain. In this case)we
need to decide which cluster to split at each step and how to do the
splitting.

Agglomerative hierarchical clustering techniques are by far the most common)

and, in this section, we will focus exclusively on these methods. A divisive
hierarchical clustering technique is described in Section 9.4.2.
A hierarchical clustering is often displayed graphically using a tree-like
diagram called a dendrogram, which displays both the cluster-subcluster
516 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

pl p2 p3 p4

(a) Dendrogram. (b) Nested cluster diagram.

Figure8.13.A hierarchical offourpoints

clustering andasnested
shownasa dendrogram clusters.

relationships and the order in which the clusters were merged (agglomerative
view) or split (divisive view). For sets of two-dimensional points, such as those
that we will use as examples, a hierarchical clustering can also be graphically
representedusing a nested cluster diagram. Figure 8.13 shows an example of
these two types of figures for a set of four two-dimensional points. These points
were clustered using the single-link technique that is describedin Section 8.3.2.

8.3.1 Basic Agglomerative Hierarchical Clustering Algorithm

Many agglomerative hierarchical clustering techniques are variations on a sin-

gle approach: starting with individual points as clusters, successivelymerge
the two closest clusters until only one cluster remains. This approach is ex-
pressedmore formally in Algorithm 8.3.

Algorithm 8.3 Basic agglomerative hierarchical clustering algorithm.

1: Compute the proximity matrix, if necessary.
2: repeat
3: Merge the closest two clusters.
4: Update the proximity matrix to reflect the proximity between the new
cluster and the original clusters.
5: until Only one cluster remains.
 8.3 AgglomerativeHierarchical Clustering 5L7

Defining Proximity between Clusters

The key operation of Algorithm 8.3 is the computation of the proximity be-
tween two clusters, and it is the definition of cluster proximity that differ-
entiates the various agglomerative hierarchical techniques that we will dis-
cuss. Cluster proximity is typically defined with a particular type of cluster
in mind-see Section 8.1.2. For example, many agglomerative hierarchical
clustering techniques, such as MIN, MAX, and Group Average, come from
a graph-based view of clusters. MIN defines cluster proximity as the prox-
imity between the closest two points that are in different clusters, or using
graph terms, the shortest edge between two nodes in different subsetsof nodes.
This yields contiguity-based clusters as shown in Figure 8.2(c). Alternatively,
MAX takes the proximity between the farthest two points in different clusters
to be the cluster proximity, or using graph terms, the longest edge between
two nodes in different subsetsof nodes. (If our proximities are distances,then
the names, MIN and MAX, are short and suggestive.For similarities, however,
where higher values indicate closer points, the names seem reversed. For that
reason, we usually prefer to use the alternative names, single link and com-
plete link, respectively.) Another graph-basedapproach, the group average
technique, definescluster proximity to be the averagepairwise proximities (av-
erage length of edges)of all pairs of points from different clusters. Figure 8.14
illustrates these three approaches.

(a) MIN (single link.) (b) MAX (complete link.) (c) Group average.

Figure8.14.Graph-based
definitions proximity
ofcluster

If, instead, we take a prototype-based view, in which each cluster is repre-

sentedby a centroid, different definitions of cluster proximity are more natural.
When using centroids, the cluster proximity is commonly defined as the prox-
imity between cluster centroids. An alternative technique, Wardts method,
also assumesthat a cluster is representedby its centroid, but it measuresthe
proximity between two clusters in terms of the increase in the SSE that re-
5L8 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

sults from merging the two clusters. Like K-means, Ward's method attempts
to minimize the sum of the squared distances of points from their cluster
centroids.

Time and Space Complexity

The basic agglomerative hierarchical clustering algorithm just presented uses

a proximity matrix. This requires the storage of lm2 proximities (assuming
the proximity matrix is symmetric) where rn is the number of data points.
The space needed to keep track of the clusters is proportional to the number
of clusters, which is m- 1, excluding singleton clusters. Hence, the total space
complexity is O(m2).
The analysis of the basic agglomerative hierarchical clustering algorithm
is also straightforward with respect to computational complexity. O(m2) time
is required to compute the proximity matrix. After that step, there are m - 1
iterations involving steps 3 and 4 because there are rn clusters at the start and
two clusters are merged during each iteration. If performed as a Iinear search of
the proximity matrix, then for the ith iteration, step 3 requires O((*-i+D2)
time, which is proportional to the current number of clusters squared. Step
4 only requires O(* - i + 1) time to update the proximity matrix after the
merger of two clusters. (A cluster merger affects only O(m- i + 1) proximities
for the techniques that we consider.) Without modification, this would yield
a time complexity of O(ms). If the distances from each cluster to all other
clusters are stored as a sorted list (or heap), it is possible to reduce the cost
of finding the two closest clusters to O(m - i + 1). However, becauseof the
additional complexity of keeping data in a sorted list or heap, the overall time
required for a hierarchical clustering based on Algorithm 8.3 is O(m2logm).
The spaceand time complexity of hierarchical clustering severelylimits the
size of data sets that can be processed.We discuss scalability approachesfor
clustering algorithms, including hierarchical clustering techniques, in Section
9.5.

8.3.2 SpecificTechniques

Sample Data

To illustrate the behavior of the various hierarchical clustering algorithms,

we shall use sample data that consists of 6 two-dimensional points, which are
shown in Figure 8.15. The r and g coordinates of the points and the Euclidean
distances between them are shown in Tables 8.3 and 8.4. respectivelv.
 8.3 Agglomerative Hierarchical Clustering 519

Point r Coordinate gr Coordinate

p1 0.40 0.53
p2 0.22 0.38
p3 0.35 0.32
p4 0.26 0.19
p5 0.08 0.41
p6 0.45 0.30

Figure points.
8.15.Setof6 two-dimensional of 6 points.
Table8.3, rg coordinates

p1 p2 p3 p4 p5 p6
p1 0.00 0.24 0.22 0.37 0.34 0.23
p2 0.24 0.00 0.15 0.20 0.14 0.25
p3 0.22 0.15 0.00 u.li) 0.28 0.11
p4 0.37 0.20 0.15 0.00 0.29 0.22
p5 0.34 0.14 0.28 0.29 0.00 0.39
p6 0.23 0.25 0.11 0.22 0.39 0.00
Table8.4. Euclidean matrix
distance for6 ooints.

Single Link or MIN

For the single link or MIN version of hierarchical clustering, the proximity
of two clusters is defined as the minimum of the distance (maximum of the
similarity) between any two points in the two different clusters. Using graph
terminology, if you start with all points as singleton clusters and add links
between points one at a time, shortest links first, then these single links com-
bine the points into clusters. The single link technique is good at handling
non-elliptical shapes,but is sensitive to noise and outliers.
Example 8.4 (Single Link). Figure 8.16 shows the result of applying the
single link technique to our example data set of six points. Figure 8.16(a)
shows the nested clusters as a sequenceof nested ellipses,where the numbers
associatedwith the ellipsesindicate the order of the clustering. Figure S.16(b)
shows the same information, but as a dendrogram. The height at which two
clusters are merged in the dendrogram reflects the distance of the two clusters.
For instance, from Table 8.4, we see that the distance between points 3 and 6
52O Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

(a) Single link cluslering. (b) Single link dendrogram.

Figure
8.16,Single ofthesixpoints
linkclustering shown
inFigure
8.15.

is 0.11, and that is the height at which they are joined into one cluster in the
dendrogram. As another example, the distance between clusters {3,6} and
{2,5} is given by

, ,5}) :
d i , s t ( { 3 , 6 {} 2 min(dest(3,2),di,st(6,2),di,st(3,5),di,st(6,5))
min(O.
15,0.25,0.28,0.39)
0.15.

Complete Link or MAX or CLIQUE

For the complete link or MAX version of hierarchical clustering, the proximity
of two clusters is defined as the maximum of the distance (minimum of the
similarity) between any two points in the two different clusters. Using graph
terminology, if you start with all points as singleton clusters and add links
between points one at a time, shortest links first, then a group of points is
not a cluster until all the points in it are completely linked, i.e., form a clique.
Complete link is less susceptible to noise and outliers, but it can break large
clusters and it favors globular shapes.
Example 8.5 (Complete Link). Figure 8.17 shows the results of applying
MAX to the sample data set of six points. As with single link, points 3 and 6
 8.3 Agglomerative Hierarchical Clustering 521

36412

(a) Complete link clustering. (b) Complete link dendrogram.

Figure
8.17.Complete ofthesixpoints
linkclustering inFigure
shown 8.15.

are merged first. However,{3,6} is merged with {4}, instead of {2,5i or {1}
because

dist({3,6}, {4}) : max(di,st(3,

4), di,st(6,4))
: max(0.15,0.22)
= 0.22.
dist({3,6},{2, 5}) : max(dzst(3 di,st(3,5),
,2),di,st(6,2), di,st(6,5))
: max(0.15,0.25,0.28,0.39)
= 0.39.
dist({3,6},{1}) : max(di,st(3,L), 7))
di,st(6,
: max(0.22,0.23)
= 0.23.

Group Average

For the group average version of hierarchical clustering, the proximity of two
clusters is defined as the average pairwise proximity among all pairs of points
in the different clusters. This is an intermediate approach between the single
and complete link approaches. Thus, for group average, the cluster proxim-
522 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

0.1

364251

(a) Group average clustering. (b) Group average dendrogram.

Figure8.18.Groupaverage ofthesixpoints
clustering shownin Figure
8.15.

ity prori,mi,ty(Ci,Ci) of clusters C,i and Cr', which are of size mi and mi,
respectively, is expressed by the following equation:

\*.co Prori'mitY(x,Y)
r€ci
prori,mi.ty(Cr,C) : . (8.6)
rni * mj
Example 8.6 (Group Average). Figure 8.18showsthe resultsof applying
the group averageapproachto the sampledata set of six points. To illustrate
how group averageworks, we calculatethe distancebetweensomeclusters.

d i s t ( { 3 , 6 , 4 }{,i } ) : ( 0 . 2 2+ 0 . 3 7+ 0 . 2 3 ) / ( 3x t )
: 0.28
di,st({2,5},{1}) : (0.2357 + 03a21)l(2 * I)
: 0.2889
d i , s t ( { 3 , 6 , 4 } , { 2 , 5 } ): ( 0 . 1 5+ 0 . 2 8+ 0 . 2 5+ 0 . 3 9+ 0 . 2 0+ 0 . 2 9 ) l ( 6* 2 )
: 0.26

Because is smallerthandist({3,6,4},{1}) anddi,st({2,5},

di,st({3,6,4},{2.,5}) {1}),
clusters{3,6,4} and {2,5} are mergedat the fourth stage. r
 8.3 AgglomerativeHierarchicalClustering 523

(a) Ward's clustering. (b) Ward's dendrogram.

Figure8.19.Ward's ofthesixpoints
clustering shownin Figure
8.15.

Wardts Method and Centroid Methods

For Ward's method, the proximity between two clusters is defined as the in-
creasein the squared error that results when two clusters are merged. Thus,
this method uses the same objective function as K-means clustering. While
it may seem that this feature makes Ward's method somewhat distinct from
other hierarchical techniques, it can be shown mathematically that Ward's
method is very similar to the group average method when the proximity be-
tween two points is taken to be the square of the distance between them.
Example 8.7 (Ward's Method). Figure 8.19 shows the results of applying
Ward's method to the sample data set of six points. The clustering that is
produced is different from those produced by single link, complete link, and
group average. r
Centroid methods calculate the proximity between two clusters by calcu-
lating the distance between the centroids of clusters. These techniques may
seem similar to K-means, but as we have remarked, Ward's method is the
correct hierarchical analog.
Centroid methods also have a characteristic----often considered bad-that
is not possessedby the other hierarchical clustering techniques that we have
discussed: the possibility of inversions. Specifically, two clusters that are
merged may be more similar (less distant) than the pair of clusters that were
merged in a previous step. For the other methods, the distance between
524 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

Table8.5.Table
of Lance-Williams
coefficients hierarchical
forcommon clustering
approaches.
Clustering Method Q.A 0.B, /'l 'l

Sinele Link 1/r r/2 0 -1 /'

Complete Link 1/' 1/9 0 1/'

mA
Group Average IIL B
0 0
MA MB
Centroid nalmn ma+mB (n o+mo\2 0
tt.HT'roc) _frLQ
Ward's mA+mB+mO mA+nB+nO ma+nR+mo
0

merged clusters monotonically increases (or is, at worst, non-increasing)

we proceed from singleton clusters to one all-inclusive cluster.

8.3.3 The Lance-Williams Formula for Cluster Proximitv

Any of the cluster proximities that we have discussedin this section can be
viewed as a choice of different parameters (in the Lance-Williams formula
shown below in Equation 8.7) for the proximity between clusters Q and R,
where,R is formed by merging clusters A and B. In this equation, p(.,.) is
a proximity function, while rn4, mB, and me are the number of points in
clusters A, B, and Q, respectively. In other words, after we merge clusters A
and B to form cluster .8, the proximity of the new cluster, ft, to an existing
cluster, Q, is a linear function of the proximities of Q with respect to the
original clusters A and B. Table 8.5 shows the values of these coefficientsfor
the techniques that we have discussed.

p ( R , Q ): a A p (A ,Q)+ *e p (B ,Q)+ 0 p( A,B) + t lp@,Q)- p( B,A) l ( 8.7)

Any hierarchical clustering technique that can be expressed using the

Lance-Wiiliams formula does not need to keep the original data points. In-
stead, the proximity matrix is updated as clustering occurs. While a general
formula is appealing, especially for implementation, it is easier to understand
the different hierarchical methods by looking directly at the definition of clus-
ter proximity that each method uses.

8.3.4 Key Issues in Hierarchical Clustering

Lack of a Global Objective Function

We previously mentioned that agglomerative hierarchical clustering cannot be

viewed as globally optimizing an objective function. Instead, agglomerative
hierarchical clustering techniques use various criteria to decide locally, at each
 8.3 Agglomerative Hierarchical Clustering 525

step, which clusters should be merged (or split for divisive approaches). This
approach yields clustering algorithms that avoid the difficulty of attempting
to solve a hard combinatorial optimization problem. (It can be shown that
the general clustering problem for an objective function such as "minimize
SSE" is computationally infeasible.) Furthermore, such approaches do not
have problems with local minima or difficulties in choosing initial points. Of
course,the time complexity of O(m2log m) and the spacecomplexity of O(m2)
are prohibitive in many cases.

Ability to Handle Different Cluster Sizes

One aspect of agglomerative hierarchical clustering that we have not yet dis-
cussedis how to treat the relative sizesof the pairs of clusters that are merged.
(This discussionapplies only to cluster proximity schemesthat involve sums,
such as centroid, Ward's, and group average.) There are two approaches:
weighted, which treats all clusters equally, and unweighted, which takes
the number of points in each cluster into account. Note that the terminology
of weighted or unweighted refers to the data points, not the clusters. In other
words, treating clusters of unequal size equally gives different weights to the
points in different clusters, while taking the cluster size into account gives
points in different clusters the same weight.
We will illustrate this using the group averagetechnique discussedin Sec-
tion 8.3.2, which is the unweighted version of the group average technique.
In the clustering literature, the full name of this approach is the Unweighted
Pair Group Method using Arithmetic averages(UPGMA). In Table 8.5, which
gives the formula for updating cluster similarity, the coefficientsfor UPGMA
involve the size of each of the clusters that were merged:.aA: *ffi,(tB :
,"ffi,0:0,?:0. For the weighted versionof group average-known as
WPGMA-the coefficientsare constants:aa: Lf 2,aB : L12,P: 0,7 : 0.
In general, unweighted approachesare preferred unless there is reason to be-
lieve that individual points should have different weights; e.g., perhaps classes
of objects have been unevenly sampled.

Merging Decisions Are Final

Agglomerative hierarchical clustering algorithms tend to make good local de-

cisions about combining two clusters since they can use information about the
pairwise similarity of all points. However, once a decision is made to merge
two clusters, it cannot be undone at a later time. This approach prevents
a local optimization criterion from becoming a global optimization criterion.
526 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

For example, although the "minimize squared error" criterion from K-means
is used in deciding which clusters to merge in Ward's method, the clusters at
each level do not represent local minima with respect to the total SSE. Indeed,
the clusters are not even stable, in the sensethat a point in one cluster may
be closer to the centroid of some other cluster than it is to the centroid of its
current cluster. Nonetheless,Ward's method is often used as a robust method
of initializing a K-means clustering, indicating that a local "minimize squared
error" objective function does have a connection to a global "minimize squared
error" objective function.
There are some techniques that attempt to overcome the limitation that
merges are final. One approach attempts to fix up the hierarchical clustering
by moving branches of the tree around so as to improve a global objective
function. Another approach uses a partitional clustering technique such as K-
means to create many small clusters, and then performs hierarchical clustering
using these small clusters as the starting point.

8.3.5 Strengths and'Weaknesses

The strengths and weaknessof specific agglomerative hierarchical clustering

algorithms were discussedabove. More generally, such algorithms are typi-
cally used because the underlying application, e.g., creation of a taxonomy,
requires a hierarchy. Also, there have been some studies that suggest that
these algorithms can produce better-quality clusters. However, agglomerative
hierarchical clustering algorithms are expensive in terms of their computa-
tional and storage requirements. The fact that all merges are final can also
cause trouble for noisy, high-dimensional data, such as document data. In
turn, these two problems can be addressedto some degree by first partially
clustering the data using another technique, such as K-means.

8.4 DBSCAN
Density-based clustering locates regions of high density that are separated
from one another by regions of low density. DBSCAN is a simple and effec-
tive density-basedclustering algorithm that illustrates a number of important
conceptsthat are important for any density-basedclustering approach. In this
section, we focus solely on DBSCAN after first considering the key notion of
density. Other algorithms for finding density-based clusters are described in
the next chapter.
 8.4 DBSCAN 527

8.4.1 Tladitional Density: Center-Based Approach

Although there are not as many a,pproachesfor defining density as there are for
defining similarity, there are several distinct methods. In this section we dis-
cuss the center-basedapproach on which DBSCAN is based. Other definitions
of density will be presented in Chapter 9.
In the center-basedapproach, density is estimated for a particular point in
the data set by counting the nurnber of points within a specified radius, .Eps,
of that point. This includes the point itself. This technique is graphically
illustrated by Figure 8.20. The number of points within a radius of Eps of
point A is 7, including A itself.
This method is simple to irnplement, but the density of any point will
depend on the specified radius. For instance, if the radius is large enough,
then all points will have a density of rn, the number of points in the data set.
Likewise, if the radius is too small, then all points will have a density of 1.
An approach for deciding on ther appropriate radius for low-dimensional data
is given in the next section in the context of our discussionof DBSCAN.

Classification of Points According to Center-Based Density

The center-basedapproach to density allows us to classify a point as being (1)

in the interior ofa denseregion (a core point), (2) on the edge ofa denseregion
(a border point), or (3) in a sparsely occupied region (a noise or background
point). Figure 8.21 graphically illustrates the concepts of core, border, and
noise points using a collection of two-dimensional points. The following text
provides a more precise description.

Core points: These points are in the interior of a density-based cluster. A

point is a core point if the number of points within a given neighborhood
around the point as determined by the distance function and a user-
specifieddistance parametr:r, -Eps,exceedsa certain threshold, Mi'nPts,
which is also a user-speci{iedparameter. In Figure 8.21, point A is a
core point, for the indicated radius (Ept) 1f MinPts < 7.

Border points: A border point is not a core point, but falls within the neigh-
borhood of a core point. In Figure 8.21, point B is a border point. A
border point can fall within the neighborhoods of several core points.

Noise points: A noise point is any point that is neither a core point nor a
border point. In Figure 8.21, point C is a noise point.
528 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

core point

Figure8.20. Center-based
density. Figure
8.21.Core, points.
andnoise
border,

8.4.2 The DBSCAN Algorithm

Given the previous definitions of core points, border points, and noise points,
the DBSCAN algorithm can be informally described as follows. Any two core
points that are close enough-within a distance Eps of one another-are put
in the same cluster. Likewise, any border point that is close enough to a core
point is put in the same cluster as the core point. (Ties may need to be resolved
if a border point is close to core points from different clusters.) Noise points
are discarded. The formal details are given in Algorithm 8.4. This algorithm
uses the same concepts and finds the same clusters as the original DBSCAN,
but is optimized for simplicity, not efficiency.

Algorithm 8.4 DBSCAN algorithm.

1: Label all points as core, border, or noise points.
2: Eliminate noise points,
3: Put an edge between all core points that are within Eps of each other.
4: Make each group of connected core points into a separate cluster.
5: Assign each border point to one of the clusters of its associatedcore points.

Time and Space Complexity

The basic time complexity of the DBSCAN algorithm is O(m x time to find
points in the -Eps-neighborhood),where rn is the number of points. In the
worst case, this complexity is O(^'). However, in low-dimensional spaces,
there are data structures, such as kd-trees, that allow efficient retrieval of all
 8.4 DBSCAN 529

points within a given distance of a specified point, and the time complexity
can be as low as O(mlogm). The space requirement of DBSCAN, even for
high-dimensional data, is O(m) because it is only necessaryto keep a small
amount of data for each point, i.e., the cluster label and the identification of
each point as a core, border, or noise point.

Selection of DBSCAN Parameters

There is, of course, the issue of how to determine the parameters Eps and
MinPts. The basic approach is to look at the behavior of the distance from
a point to its kth nearest neighbor, which we will call the k-dist. For points
that belong to some cluster, the value of k-dist will be small if k is not larger
than the cluster size. Note that there will be some variation, depending on the
density of the cluster and the r:r,ndomdistribution of points, but on average,
the range of variation will not be huge if the cluster densities are not radically
different. However, for points that are not in a cluster, such as noise points,
the k-dist will be relatively large. Therefore, if we compute the k-dist for
all the data points for some ,k, sort them in increasing order, and then plot
the sorted values, we expect to see a sharp change at the value of k-dist that
correspondsto a suitable value of Eps. lf we select this distance as the Eps
parameter and take the value oi k as the MinPts parameter, then points for
which ,k-dist is less than -Opswill be labeled as core points, while other points
will be labeled as noise or borderrpoints.
Figure 8.22 shows a sample data set, while the k-dist graph for the data is
given in Figure 8.23. The value of Eps that is determined in this way depends
on k, but does not change dramatically as k changes. If the value of k is too
small, then even a small number of closely spaced points that are noise or
outliers will be incorrectly labeled as clusters. If the value of k is too large,
then small clusters (of size less than k) are likely to be labeled as noise. The
original DBSCAN algorithm used a value of k : 4, which appears to be a
reasonablevalue for most two-dimensional data sets.

Clusters of Varying Density

DBSCAN can have trouble with density if the density of clusters varies widely.
Consider Figure 8.24, which shows four clusters embedded in noise. The den-
sity of the clusters and noise regions is indicated by their darkness. The noise
around the pair of denser clusters, A and B, has the same density as clusters
C and D. If the -Eps threshold is low enough that DBSCAN finds C and D as
clusters, then A and B and the points surrounding them will become a single
530 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

z
6

Poinb Sodd by Disan€ro 4th Neared Neighbor

Figure data.
8.22.Sample Figure8,23.K-distplotforsample
data.

Figure8.24.Fourclusters in noise.
embedded

cluster. If the .Opsthreshold is high enough that DBSCAN finds A and B as

separate clusters, and the points surrounding them are marked as noise, then
C and D and the points surrounding them will also be marked as noise.

An Example

To illustrate the use of DBSCAN, we show the clusters that it finds in the
relatively complicated two-dimensional data set shown in Figure 8.22. This
data set consists of 3000 two-dimensional points. The Eps threshold for this
data was found by plotting the sorted distancesofthe fourth nearest neighbor
of each point (Figure 8.23) and identifying the value at which there is a sharp
increase. We selected Eps :10, which correspondsto the knee of the curve.
The clusters found by DBSCAN using these parameters, i.e., Mi,nPts: 4 and
Eps : 10, are shown in Figure 8.25(a). The core points, border points, and
noise points are displayed in Figure 8.25(b).

8.4.3 Strengths and Weaknesses

BecauseDBSCAN uses a density-based definition of a cluster, it is relatively

resistant to noise and can handle clusters of arbitrary shapesand sizes. Thus,
 8.4 DBSCAN 531

I xxX Ix x
XX
X
f XX

'i+
X
X I X
,j
1 i+-I+.tJtlr,
I
f
la X
X

X
""t

x X
X
X

X
*.
"1 x
X

r trX
€
I
X

i.
X

(a) Clusl;ers found by DBSCAN

x - NoisePoint + - BorderPoint e - CorePoint

(b) Core, border, and nolse points.

Figure8.25,DBSCAN
clustering points.
of3000two-dimensional
532 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

DBSCAN can find many clusters that could not be found using K-means,
such as those in Figure 8.22. As indicated previously, however, DBSCAN has
trouble when the clusters have widely varying densities. It also has trouble
with high-dimensional data becausedensity is more difficult to define for such
data. One possible approach to dealing with such issues is given in Section
9.4.8. Finally, DBSCAN can be expensive when the computation of nearest
neighbors requires computing all pairwise proximities, as is usually the case
for high-dimensional data.

8.5 Cluster Evaluation

In supervisedclassification,the evaluation of the resulting classificationmodel
is an integral part of the process of developing a classification model, and
there are well-accepted evaluation measures and procedures, €.g., accuracy
and cross-validation,respectively. However, becauseof its very nature, cluster
evaluation is not a well-developedor commonly used part of cluster analysis.
Nonetheless,cluster evaluation, or cluster validation as it is more tradition-
ally called, is important, and this section will review some of the most common
and easily applied approaches.
There might be some confusion as to why cluster evaluation is necessary.
Many times, cluster analysis is conducted as a part of an exploratory data
analysis. Hence, evaluation seemslike an unnecessarilycomplicated addition
to what is supposed to be an informal process. Furthermore, since there
are a number of different types of clusters-in some sense, each clustering
algorithm defines its own type of cluster it may seem that each situation
might require a different evaluation measure. For instance, K-means clusters
might be evaluated in terms of the SSE, but for density-basedclusters, which
need not be globular, SSE would not work well at all.
Nonetheless, cluster evaluation should be a part of any cluster analysis.
A key motivation is that almost every clustering algorithm will find clusters
in a data set, even if that data set has no natural cluster structure. For
instance, consider Figure 8.26, which shows the result of clustering 100 points
that are randomly (uniformly) distributed on the unit square. The original
points are shown in Figure 8.26(a), while the clusters found by DBSCAN, K-
means, and complete link are shown in Figures 8.20(b), 8.26(c), and 8.26(d),
respectively. Since DBSCAN found three clusters (after we set Eps by looking
at the distancesof the fourth nearest neighbors), we set K-means and complete
link to find three clusters as well. (In Figure 8.26(b) the noise is shown by
the small markers.) However, the clusters do not look compelling for any of
 8.5 Cluster Evaluation 533

the three methods. In higher dimensions, such problems cannot be so easily

detected.

8.5.1 Overview

Being able to distinguish whether there is non-random structure in the data

is just one important aspect of cluster validation. The following is a list of
several important issues for cluster validation.

1. Determining the clustering tendency of a set of data, i.e., distinguish-

ing whether non-random structure actually exists in the data.

2. Determining the correct number of clusters.

3. Evaluating how well the results of a cluster analysis fit the data wi,thout
referenceto external information.

4. Comparing the results of a cluster analysis to externally known results,

such as externally provided class labels.

5. Comparing two sets of clusters to determine which is better.

Notice that items L, 2, and 3 do not make use of any external information-
they are unsupervised techniques while item 4 requires external information.
Item 5 can be performed in either a supervised or an unsupervised manner. A
further distinction can be made with respect to items 3, 4, and 5: Do we want
to evaluate the entire clustering or just individual clusters?
While it is possible to deverlop various numerical measures to assessthe
different aspects of cluster validity mentioned above, there are a number of
challenges. First, a measure of cluster validity may be quite limited in the
scope of its applicability. For example, most work on measures of clustering
tendency has been done for two- or three-dimensional spatial data. Second,
we need a framework to interprr:t any measure. If we obtain a value of 10 for a
measurethat evaluateshow well cluster labels match externally provided class
labels, does this value represent a good, fair, or poor match? The goodness
of a match often can be measured by looking at the statistical distribution of
this value, i.e., how likely it is that such a value occurs by chance. Finally, if
a measure is too complicated to apply or to understand, then few will use it.
The evaluation measures, or indices, that are applied to judge various
aspects of cluster validity are traditionally classified into the following three
types.
534 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

o V
oo o a VVYV
a ao a VV
Oa VVV
o oo V VV
ao
a
o v vV
v v
1O ao
yY
V Yv
oo to O
VV
Y' Y
tt. a OO
V VY
tt y VV
3 o 'ti. O-
r o YVV v
oO vvv ttg
oo ' a o t vwvv
vv t
1O o V VV

a o *
o .a
oot a x xtur
10 a o ** x
ooor ****
a I

!. t ;o a
a o ***
y* * *
I
I

oo o o oo ** * ta
, I
o

(a) Original points. (b) Three clusters found by DBSCAN.

OO
* *
*x * ** * x
ooo ** * ,(* * *)( *
* 'K * t(
l lo
* )( *.*x * *
)K. *
* *
I
1O * 4*
;{<.
* *
a
o )( * x* *
** * ** x**
x' * 4*
3tt. a * )'( **. ... *
a o x*
**.
x * i *
#
oo o g* )K *oo *
**
o VY * oo
o * v O
o v! *Y v$
yV yV
Y o
V Vvv .Y
Y
VV Y
Y
V'
v
V
Y V
Y' YV
VY
r.':. a
o O o

vv VV o oo
V
v oo a
(c) Three clusters found by K-means. (d) Three clusters found by complete

Figure8.26.Clustering
of 100uniformly points.
distributed
 8.5 Cluster Evaluation 535

IJnsupervised. Measuresthe goodnessof a clustering structure without re-

spect to external informal;ion. An example of this is the SSE. Unsu-
pervised measuresof cluster validity are often further divided into two
classes:measuresof cluster cohesion (compactness,tightness), which
determine how closely relabedthe objects in a cluster are, and measures
of cluster separation (isolation), which determine how distinct or well-
separated a cluster is from other clusters. Unsupervised measures are
often called internal indices becausethev use only information present
in the data set.

Supervised. Measuresthe extent to which the clustering structure discovered

by a clustering algorithm rnatchessome external structure. An example
of a supervised index is entropy, which measureshow well cluster labels
match externally supplied class labels. Supervised measures are often
called external indices Lrecausethey use information not present in
the data set.

Relative. Compares different clusterings or clusters. A relative cluster eval-

uation measure is a supervised or unsupervised evaluation measure that
is used for the purpose of comparison. Thus, relative measuresare not
actually a separate type of cluster evaluation measure, but are instead a
specific use of such measures. As an example, two K-means clusterings
can be compared using either the SSE or entropy.

In the remainder of this section, we provide specificdetails concerning clus-

ter validity. We first describetopics related to unsupervisedcluster evaluation,
beginning with (1) measuresba,sedon cohesion and separation, and (2) two
techniques based on the proximity matrix. Since these approachesare useful
only for partitional sets of clusters, we also describe the popular cophenetic
correlation coefficient, which can be used for the unsupervised evaluation of
a hierarchical clustering. We end our discussion of unsupervised evaluation
with brief discussionsabout fincling the correct number of clusters and evalu-
ating clustering tendency. We then consider supervised approachesto cluster
validity, such as entropy, purity, and the Jaccard measure. We conclude this
section with a short discussionof how to interpret the values of (unsupervised
or supervised) validity measures.
536 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

8.5.2 lJnsupervised Cluster Evaluation Using Cohesion and

Separation

Many internal measuresof cluster validity for partitional clustering schemes

are based on the notions of cohesion or separation. In this section, we use
cluster validity measuresfor prototype- and graph-basedclustering techniques
to explore these notions in some detail. In the process,we will also see some
interesting relationships between prototype- and graph-based clustering.
In general, we can consider expressing overall cluster validity for a set of
1{ clusters as a weighted sum of the validity of individual clusters,

K
oueralluatidi,ty: t wi uali,d,i,ty(C.i). (8.8)
t:l

The uali,di,tyfunction can be cohesion,separation, or somecombination of these

quantities. The weights will vary depending on the cluster validity measure.
In some cases,the weights are simply 1 or the size of the cluster, while in other
casesthey reflect a more complicated property, such as the square root of the
cohesion. SeeTable 8.6. If the validity function is cohesion,then higher values
are better. If it is separation, then lower values are better.

Graph-Based View of Cohesion and Separation

For graph-basedclusters, the cohesionof a cluster can be defined as the sum of

the weights of the links in the proximity graph that connect points within the
cluster. SeeFigure 8.27(a). (Recall that the proximity graph has data objects
as nodes, a link between each pair of data objects, and a weight assignedto
each link that is the proximity between the two data objects connected by the
link.) Likewise, the separation between two clusters can be measured by the
sum of the weights of the links from points in one cluster to points in the other
cluster. This is illustrated in Figure 8.27(b).
Mathematically, cohesion and separation for a graph-based cluster can be
expressedusing Equations 8.9 and 8.10, respectively. The prori,mi,ty function
can be a similarity, a dissimilarity, or a simple function of these quantities.

cohesi,on(C6) : I pro*i*i.ty(x,y) ( 8.e)

i,2Z:
separat'ion(Ct,C) : pro"i^i.tg(x,y) (8.10)
I
xeL i
v€ci
 8.5 Cluster Evaluation 537

(a) Cohesion. (b) Separation.

Figure viewofcluster
8.27.Graph-based cohesion
andseparation.

Prototype-Based View of Cohesion and Separation

For prototype-based clusters, the cohesion of a cluster can be defined as the

sum of the proximities with respect to the prototype (centroid or medoid) of
the cluster. Similarly, the separation between two clusters can be measured
by the proximity of the two cluster prototypes. This is illustrated in Figure
8.28, where the centroid of a cluster is indicated by a "+".
Cohesion for a prototype-based cluster is given in Equation 8.11, while
two measures for separation are given in Equations 8.12 and 8.13, respec-
tively, where ca is the prototype (centroid) of cluster C; and c is the overall
prototype (centroid). There are two measuresfor separation because,as we
will seeshortly, the separation of cluster prototypes from an overall prototype
is sometimes directly related to the separation of cluster prototypes from one
another. Note that Equation 8.11 is the cluster SSE if we let proximity be the
squared Euclidean distance.

cohesion(Ci) :
\ pro.rimilg(x,ci) (8.11)
xeLt

separation(Ct,Ci) prori,mity\ct, cj) (8.12)

separati.on(Ci) prori,mi.ty(c6,c) (8.13)

Overall Measures of Cohesion and Separation

The previous definitions of cluster cohesionand separation gave us some sim-

ple and well-defined measures of cluster validity that can be combined into
an overall measure of cluster validity by using a weighted sum, as indicated
538 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

(a) Cohesion. (b) Separation

Figure8.28,Prototype-based
viewof cluster andseparation.
cohesion

in Equation 8.8. However, we need to decide what weights to use. Not sur-
prisingly, the weights used can vary widely, although typically they are some
measure of cluster size.
Table 8.6 provides examples of validity measuresbased on cohesion and
separation. Iy is a measure of cohesion in terms of the pairwise proximity of
objects in the cluster divided by the cluster size. 12 is a measure of cohesion
based on the sum of the proximities of objects in the cluster to the cluster
centroid. t1 is a measure of separation defined as the proximity of a cluster
centroid to the overall centroid multiplied by the number of objects in the
cluster. 9t, which is a measure based on both cohesion and separation, is
the sum of the pairwise proximity of all objects in the cluster with all objects
outside the cluster-the total weight of the edgesof the proximity graph that
must be cut to separate the cluster from all other clusters-divided bv the
sum of the pairwise proximity of objects in the cluster.

Table ofgraph-based
8.6.Table cluster measures.
evaluation
Name Cluster Measure Cluster Weisht Type
graph-based
a I
L7
mi
cohesion
prototype-based
,7.
L2 D*ect Pr ori'mi'tY(x, c 1) I cohesion
prototype-based
c
u'l prorzmity(ci,c) mi separation
graph-based
sk
9t Ii .l D l:: , ProtimitY(x.Y D;:g; Prori,mits(x,Y) separation and
Jc"J
cohesion
 8.5 Cluster Evaluation 539

Note that any unsupervised measure of cluster validity potentially can be

used as an objective function lbr a clustering algorithm and vice versa. The
ClUstering TOolkit (CLUTO) (see the bibliographic notes) uses the cluster
evaluation measuresdescribed in Table 8.6, as well as some other evaluation
measuresnot mentioned here, to drive the clustering process. It does this by
using an algorithm that is similar to the incremental K-means algorithm dis-
cussedin Section 8.2.2. Specifically,each point is assignedto the cluster that
produces the best value for the cluster evaluation function. The cluster eval-
uation measure Z2 corresponds to traditional K-means and produces clusters
that have good SSE values. The other measuresproduce clusters that are not
as good with respect to SSE, but that are more optimal with respect to the
specified cluster validity measure.

Relationship between Prototype-Based Cohesion and Graph-Based

Cohesion

While the graph-based and prototype-based approachesto measuring the co-

hesion and separation of a cluster seem distinct, for some proximity measures
they are equivalent. For instance, for the SSE and points in Euclidean space,
it can be shown (Equation 8.14) that the average pairwise distance between
the points in a cluster is equivzr,lentto the SSE of the cluster. SeeExercise 27
on page 566.

clusterssE: I :
dtrt1"6,*)2 d,tst(x,y)2 (8.14)
-""1 D
r^L I
xeCi xQCay€Ci

Two Approaches to Prototype-Based Separation

When proximity is measuredby Euclidean distance, the traditional measureof

separation between clusters is the between group sum of squares (SSB), which
is the sum of the squared distance of a cluster centroid, c6, to the overall meanT
c, of all the data points. By summing the SSB over all clusters, we obtain the
total SSB, which is given by Equation 8.15, where ca is the mean of the i,th
cluster and c is the overall mean. The higher the total SSB of a clustering,
the more separated the clusters are from one another.

Total SSB:D*o d,i,st(ci,c)2 (8.15)

x:l

It is straightforward to show that the total SSB is directly related to the

pairwise distances between the centroids. In particular, if the cluster sizesare
54O Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

equal, i.e., mi : m I K , then this relationship takes the simple form given by
Equation 8.16. (SeeExercise 28 on page 566.) It is this type of equivalencethat
motivates the definition of prototype separation in terms of both Equations
8 . 1 2a n d 8 . 1 3 .

1KK
T o taSl S B: + t, dist( c;.ci) 2 (8.16)
2 K ?r -?t ' x r y
l= I

Relationship between Cohesion and Separation

In some cases,there is also a strong relationship between cohesionand separa-

tion. Specifically,it is possible to show that the sum of the total SSE and the
total SSB is a constant; i.e., that it is equal to the total sum of squares (TSS),
which is the sum of squaresof the distance of each point to the overall mean
of the data. The importance of this result is that minimizing SSE (cohesion)
is equivalent to maximizing SSB (separation).
We provide the proof of this fact below, since the approach illustrates
techniques that are also applicable to proving the relationships stated in the
last two sections. To simplify the notation, we assume that the data is one-
dimensional,i.e., d'ist(r,A) : (r-A;2. Also, we usethe fact that the cross-term
- c,1)(c- q) is 0. (SeeExercise29 on page 566.)
Di:tD"ecn@

K
TSS \-
ZJ
\-
Zr
(* - ,)'
reu

t \-
/J

TE L;
((" -
"n)
- (r-
"n))'
K KK
(* - ro)'- 2DI (" - -",) l k - "u)'
"o)(" + D
\-
Z-/ /\-
reC ?:I JEU): i:7 r€Ct
K K

?:1 reu, i:l r€Ct

K K

t
i:L
- -
D (" "n)'+Dlcil(c c)2
r€C;

SSE+ SSB
 8.5 Cluster Evaluation 54L

Evaluating Individual Clusters and Objects

So far, we have focused on using cohesion and separation in the overall eval-
uation of a group of clusters. Many of these measuresof cluster validity also
can be used to evaluate individual clusters and objects. For example, we can
rank individual clusters according to their specific value of cluster validity, i.e.,
cluster cohesion or separation. A cluster that has a high value of cohesion may
be considered better than a clutter that has a lower value. This information
often can be used to improve the quality of a clustering. If, for example, a
cluster is not very cohesive,then we may want to split it into several subclus-
ters. On the other hand, if two clusters are relatively cohesive,but not well
separated, we may want to merge them into a single cluster.
We can also evaluate the objects within a cluster in terms of their con-
tribution to the overall cohesion or separation of the cluster. Objects that
contribute more to the cohesion and separation are near the "interior" of the
cluster. Those objects for which the opposite is true are probably near the
"edge" of the cluster. In the following section, we consider a cluster evalua-
tion measure that uses an approach based on these ideas to evaluate points,
clusters, and the entire set of clusters.

The Silhouette Coefficient

The popular method of silhouette coefficientscombinesboth cohesionand sep-

aration. The following steps explain how to compute the silhouette coefficient
for an individual point, a process that consists of the following three steps.
We use distances, but an analog;ousapproach can be used for similarities.

1. For the ith object, calculate its average distance to all other objects in
its cluster. Call this value a;.

2. For the i,th object and any cluster not containing the object, calculate
the object's averagedistance to all the objects in the given cluster. Find
the minimum such value with respect to all clusters; call this value b.;.

3. For the i,th object, the silhouette coefficient is s; : (b'i- a.i)f rnax(ai,b1).

The value of the silhouette coefficient can vary between -1 and 1. A

negative value is undesirable becausethis corresponds to a case in which a;,
the averagedistance to points in the cluster, is greater than ba,the minimum
averagedistance to points in another cluster. We want the silhouette coefficient
to be positive (a; .--b), and for o; to be as close to 0 as possible, since the
coefficient assumesits maximum value of 1 when at.:0.
542 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

e :t,,ti'tt r:,;:!:1:i
zu
"i-'11,.,,
;I'ir,'irr', -
et$&s di r" t, :i;,f{,e
* 4#F*
o

* It
,,ti l:i* s ..d{-.*4 "'" :"j:r,
s ".tiii:
gr:'ri:i:*b + i-:Y,$ f.i,:i,.r:,bs.
, r;- "l'+*-I
'u ti'u,in*

o 01 02 03 07 08 oe 1
:,lr**,"0:*n.,:":

Figure8.29.Silhouette forpointsintenclusters.
coefficients

We can compute the average silhouette coemcient of a cluster by simply

taking the average of the silhouette coefficients of points belonging to the
cluster. An overall measureof the goodnessof a clustering can be obtained by
computing the averagesilhouette coefficient of all points.
Example 8.8 (Silhouette Coefficient). Figure 8.29 shows a plot of the
silhouette coeffi.cientsfor points in 10 clusters. Darker shades indicate lower
silhouette coefficients. I

8.5.3 lJnsupervised Cluster Evaluation Using the Proximity

Matrix
In this section, we examine a couple of unsupervised approachesfor assessing
cluster validity that are based on the proximity matrix. The first comparesan
actual and idealized proximity matrix, while the second uses visualization.

Measuring Cluster Validity via Correlation

If we are given the similarity matrix for a data set and the cluster labels from
a cluster analysis of the data set, then we can evaluate the "goodness" of
the clustering by looking at the correlation between the similarity matrix and
an ideal version of the similarity matrix based on the cluster labels. (With
minor changes,the following applies to proximity matrices, but for simplicity,
we discuss only similarity matrices.) More specifically, an ideal cluster is one
whose points have a similarity of 1 to all points in the cluster, and a similarity
of 0 to all points in other clusters. Thus, if we sort the rows and columns
of the similarity matrix so that all objects belonging to the same class are
together, then an ideal similarity matrix has a block diagonal structure. In
other words, the similarity is non-zero, i.e., 1, inside the blocks of the similarity
 8.5 Cluster Evaluation 543

matrix whose entries represent intra-cluster similarity, and 0 elsewhere. The

ideal similarity matrix is constructed by creating a matrix that has one row
and one column for each data point just like an actual similarity matrix-
and assigning a 1 to an entry if the associatedpair of points belongs to the
same cluster. All other entries are 0.
High correlation between the ideal and actual similarity matrices indicates
that the points that belong to the same cluster are close to each other, while
low correlation indicates the opposite. (Since the actual and ideal similarity
matrices are symmetric, the correlation is calculated only among the n(n-L) l2
entries below or above the diagonal of the matrices.) Consequently,this is not
a good measure for many density- or contiguity-based clusters, becausethey
are not globular and may be closely intertwined with other clusters.
Example 8.9 (Correlation of Actual and Ideal Similarity Matrices).
To illustrate this measure,we calculated the correlation between the ideal and
actual similarity matrices for the K-means clusters shown in Figure 8.26(c)
(random data) and Figure S.30(a) (data with three well-separated clusters).
The correlations were 0.5810 and 0.9235, respectively, which reflects the ex-
pected result that the clusters found by K-means in the random data are worse
than the clusters found by K-means in data with well-separatedclusters. r

Judging a Clustering Visually by Its Similarity Matrix

The previous technique suggestsia more general, qualitative approach to judg-

ing a set of clusters: Order the riimilarity matrix with respect to cluster labels
and then plot it. In theory, if we have well-separatedclusters, then the simi-
larity matrix should be roughly block-diagonal. If not, then the patterns dis-
played in the similarity matrix can reveal the relationships between clusters.
Again, all of this can be applied to dissimilarity matrices, but for simplicity,
we will only discusssimilarity matrices.
Example 8.10 (Visualizing a Similarity Matrix). Considerthe points in
Figure 8.30(a), which form three well-separatedclusters. If we use K-means to
group these points into three clusters, then we should have no trouble finding
these clusters since they are well-separated. The separation of these clusters
is illustrated by the reordered similarity matrix shown in Figure 8.30(b). (For
uniformity, we have transformed the distances into similarities using the for-
mula s - 1- (d-rni,n-d)l(mar^d-min-d).) Figure 8.31 showsthe reordered
similarity matrices for clusters found in the random data set of Figure 8.26 by
DBSCAN, K-means, and complete link.
544 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

Similarity

(a) Well-separated clusters. (b) Similarity matrix sorted by K-means

cluster labels.

Figure8.30.Similarity
matrix
forwell-separated
clusters.

The well-separated clusters in Figure 8.30 show a very strong, block-

diagonal pattern in the reordered similarity matrix. However, there are also
weak block diagonal patterns-see Figure 8.31 in the reordered similarity
matrices of the clusterings found by K-means, DBSCAN, and complete link
in the random data. Just as people can find patterns in clouds, data mining
algorithms can find clusters in random data. While it is entertaining to find
patterns in clouds, it is pointless and perhaps embarrassingto find clusters in
noise. r
This approach may seem hopelessly expensive for large data sets, since
the computation of the proximity matrix takes O(m2) time, where rn is the
number of objects, but with sampling, this method can still be used. We can
take a sample of data points from each cluster, compute the similarity between
these points, and plot the result. It may be necessaryto oversample small
clusters and undersample large ones to obtain an adequate representation of
all clusters.

8.5.4 lJnsupervised Evaluation of Hierarchical Clustering

The previous approaches to cluster evaluation are intended for partitional
clusterings. Here we discuss the cophenetic correlation, a popular evaluation
measure for hierarchical clusterings. The cophenetic distance between two
objects is the proximity at which an agglomerativehierarchical clustering tech-
 8.5 Cluster Evaluation 545

(a) Similarity matrix (b) Similarity matrix (c) Similarity matrix

sorted by DBSCAN sorted by K-means sorted by complete link
cluster labels. cluster labels. cluster labels.

matrices
Figure8.31.Similarity fromrandom
forclusters data.

nique puts the objects in the same cluster for the first time. For example, if at
some point in the agglomerative hierarchical clustering process, the smallest
distance between the two clusters that are merged is 0.1, then all points in
one cluster have a cophenetic clistanceof 0.1 with respect to the points in the
other cluster. In a cophenetic distance matrix, the entries are the cophenetic
distances between each pair of objects. The cophenetic distance is different
for each hierarchical clustering of a set of points.
Example 8.L1- (Cophenetic Distance Matrix). Table 8.7 showsthe cophen-
tic distance matrix for the single link clustering shown in Figure 8.16. (The
data for this figure consists of the 6 two-dimensional points given in Table
8.3.)

distance
Table8.7. Cophenetic forsinglelinkanddataintable8.3
matrix
Point P1 P2 P3 P4 P5 P6
P1 0 0.222 0.222 0.222 0.222 0.222
P2 0.222 0 0.148 0 . 1 5 r 0.139 0.148
P3 0.222 0.148 0 0 . 1 5 1 0.148 0.110
P4 0.222 0.151 0.151 0 0.151 0.151
P5 0.222 0.139 0.148 0.151 0 0.148
P6 0.222 0.148 0 . 11 0 0 . 1 5 1 0.148 0

The CoPhenetic Correlation Coefficient (CPCC) is the correlation

between the entries of this matrix and the original dissimilarity matrix and is
546 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

a standard measureof how well a hierarchical clustering (of a particular type)

fits the data. One of the most common uses of this measure is to evaluate
which type of hierarchical clustering is best for a particular type of data.
Example 8.12 (Cophenetic Correlation Coefficient). We calculatedthe
CPCC for the hierarchical clusterings shown in Figures 8.16-8.1g.Thesevalues
are shown in Table 8.8. The hierarchical clustering produced by the single
link technique seemsto fit the data less well than the clusterings produced by
complete link, group average,and Ward's method.

Table8.8. Cophenetic
correlation
coefficient
fordataofTable8.3andfouragglomerative
hierarchical
clustering
techniques.
Technique CPCC
Single Link 0.44
Complete Link 0.63
Group Average 0.66
Ward's 0.64

8.5.5 Determining the Correct Number of Clusters

Various unsupervised cluster evaluation measures can be used to approxi-

mately determine the correct or natural number of clusters.
Example 8.13 (Number of Clusters). The data set of Figure 8.29 has 10
natural clusters. Figure 8.32 shows a plot of the SSE versus the number of
clusters for a (bisecting) K-means clustering of the data set, while Figure 8.33
shows the average silhouette coefficient versus the number of clusters for the
same data. There is a distinct knee in the SSE and a distinct peak in the
silhouette coefficient when the number of clusters is equal to 10. r
Thus, we can try to find the natural number of clusters in a data set by
looking for the number of clusters at which there is a knee, peak, or dip in
the plot of the evaluation measure when it is plotted against the number of
clusters. Of course,such an approach does not always work well. Clusters may
be considerably more intertwined or overlapping than those shown in Figure
8.29. Also, the data may consist of nested clusters. Actually, the clusters in
Figure 8.29 are somewhat nested; i.e., there are 5 pairs of clusters since the
clusters are closer top to bottom than they are left to right. There is a knee
that indicates this in the SSE curve, but the silhouette coefficient curve is not
 gr-br.v*,'F/r"P 8.5 Cluster Evaluation 547

055
o
05

a 045
04

035

03
051015202530 051015202530
Numberof Clusters Numberof ClusteF

Figure
8.32,SSEversusnumber
ofclusters
for 8.33.Average
Figure silhouette ver-
coefficient
thedataofFigure
8.29. susnumber forthedataof Figure
of clusters
8.29,

as clear. In summary, while caution is needed, the technique we have just

described can provide insight into the number of clusters in the data.

8.5.6 Clustering Tendency

One obvious way to determine if a data set has clusters is to try to cluster
it. However, almost all clusterirrg algorithms will dutifully find clusters when
given data. To addressthis issue,we could evaluate the resulting clusters and
only claim that a data set has clusters if at least someof the clusters are of good
quality. However, this approactr does not address the fact the clusters in the
data can be of a different type than those sought by our clustering algorithm.
To handle this additional problem, we could use multiple algorithms and again
evaluate the quality of the resulting clusters. If the clusters are uniformly poor,
then this may indeed indicate that there are no clusters in the data.
Alternatively, and this is the focus of measuresof clustering tendency, we
can try to evaluate whether a data set has clusters without clustering. The
most common approach, especially for data in Euclidean space, has been to
use statistical tests for spatial randomness. Unfortunately, choosing the cor-
rect model, estimating the parameters, and evaluating the statistical signifi-
cance of the hypothesis that the data is non-random can be quite challenging.
Nonetheless,many approaches have been developed, most of them for points
in low-dimensional Euclidean space.
Exarnple 8.14 (Hopkins Statistic). For this approach,we generateppoints
that are randomly distributed acrossthe data space and also sample p actual
548 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

data points. For both sets of points we find the distance to the nearest neigh-
bor in the original data set. Let theui be the nearest neighbor distancesofthe
artificially generated points, while the ui are the nearest neighbor distances
of the sample of points from the original data set. The Hopkins statistic fI is
then defined by Equation 8.17.

H- DT:t.t (8.17)
+ Dl:twt
D,Pl:rut,
If the randomly generated points and the sample of data points have
roughly the same nearest neighbor distances, then 11 will be near 0.5. Values
of fI near 0 and 1 indicate, respectively, data that is highly clustered and
data that is regularly distributed in the data space. To give an example, the
Hopkins statistic for the data of Figure 8.26 was computed for p : 20 and 100
different trials. The average value of 11 was 0.56 with a standard deviation
of 0.03. The same experiment was performed for the well-separatedpoints of
Figure 8.30. The average value of fI was 0.95 with a standard deviation of
0.006. r

8.5.7 Supervised Measures of Cluster Validity

When we have external information about data, it is typically in the form of
externally derived class labels for the data objects. In such casesTthe usual
procedure is to measurethe degreeof correspondencebetweenthe cluster labels
and the class labels. But why is this of interest? After all, if we have the class
labels, then what is the point in performing a cluster analysis? Motivations for
such an analysis are the comparison of clustering techniques with the "ground
truth" or the evaluation of the extent to which a manual classification process
can be automatically produced by cluster analysis.
We consider two different kinds of approaches. The first set of techniques
use measuresfrom classification, such as entropy, purity, and the F-measure.
These measuresevaluate the extent to which a cluster contains objects of a
single class. The secondgroup of methods is related to the similarity measures
for binary data, such as the Jaccard measurethat we saw in Chapter 2. These
approachesmeasurethe extent to which two objects that are in the same class
are in the same cluster and vice versa. For convenience,we will refer to these
two types of measuresas classification-oriented and similarity-oriented,
respectively.
 8.5 Cluster Evaluation 549

Classification-Oriented Measures of Cluster Validity

There are a number of measures entropy, purity, precision, recall, and the
F-measure that are commonly used to evaluate the performance of a classi-
fication model. In the case of classification?we measure the degree to which
predicted class labels correspond to actual class labels, but for the measures
just mentioned, nothing fundamental is changed by using cluster labels in-
stead of predicted classlabels. Next, we quickly review the definitions of these
measures,which were discussedin Chapter 4.

Entropy: The degreeto which each cluster consistsof objects of a single class.
For each cluster, the class distribution ofthe data is calculated first, i.e.,
for cluster jl we compute pU, the probability that a member of cluster ri
belongs to classj as pU : mt j /mt, where mi is the number of objects in
cluster i, and mii is the number of objects of class j in cluster i. Using
this class distribution, the entropy of each cluster ri is calculated using
the standard formula, €i == -Di-tptilogzptj, where tr is the number of
classes. The total entropy for a set of clusters is calculated as the sum
of the e_ntropiesof each cluster weighted by the size of each cluster, i.e.,
e : DLtffei, where 1{ is the number of clusters and rn is the total
number of data points.

Purity: Another measureof ttLe extent to which a cluster contains objects of

a single class. Using the previous terminology, the purity of cluster i is
pi : rta;xpri, the overall purity of a clustering is puri,ta : Di:tTpt,.

Precision: The fraction of a cluster that consistsof objects of a specifiedclass.

j) : pij.
The precision of cluster i with respect to classj is preci,si,on(i.,

Recall: The extent to which a cluster contains all objects of a specifiedclass.

The recall of cluster z with respect to classj is recall(i.,j) : miif mi,
where m3 is the number of objects in classj.

F-measure A combination of both precision and recall that measures the

extent to which a cluster contains only objects of a particular class and all
objects of that class. The F-measure of cluster i with respect to class3 is
F(i,,j): (2xpreci,si,on(i, j)xrecall(i, j))l(precision(i,,j)*recall(i, j)).

Example 8.15 (Supervised Evaluation Measures). We present an exam-

ple to illustrate these measures. Specifically,we use K-means with the cosine
similarity measure to cluster 3204 newspaper articles from the Los Angeles
 550 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

Tabfe8.9. K-means
clustering
results
fortheLATimes
document
dataset.
Cluster Enter- Financial Foreign Metro National Sports Entropy Purity
tainment
1 3 K
40 506 96 27 r.2270 0.7474
2 4 7 280 29 39 2 1.1472 0.7756
.-) 1 1 I 7 4 677 0 . 1 8 1 3 0.9796
4 10 r62 ,f119 IJ 2 r.7487 0.4390
r r
331 22 70 IJ 23 1.3976 0.7134
o ( 358 12 2r2 48 IJ r.5523 0.5525
Total D<A
cDo 34r 943 273 738 1.1450 0.7203

Ti,mes. These articles come from six different classes:Entertainment, Finan-

cial, Foreign, Metro, National, and Sports. Table 8.9 shows the results of a
K-means clustering to find six clusters. The first column indicates the clus-
ter, while the next six columns together form the confusion matrix; i.e., these
columns indicate how the documents of each category are distributed among
the clusters. The last two columns are the entropy and purity of each cluster,
respectively.
Ideally, each cluster will contain documents from only one class. In reality,
each cluster contains documents from many classes.Nevertheless,many clus-
ters contain documents primarily from just one class. In particular, cluster
3, which contains mostly documents from the Sports section, is exceptionally
good, both in terms of purity and entropy. The purity and entropy of the
other clusters is not as good, but can typically be greatly improved if the data
is partitioned into a larger number of clusters.
Precision. recall. and the F-measure can be calculated for each cluster. To
give a concrete example, we consider cluster 1 and the Metro class of Table
8.9. The precisionis 5061677: 0.75, recall is 5061943:0.26, and hence,the
F value is 0.39. In contrast, the F value for cluster 3 and Sports is 0.94. r

Similarity-Oriented Measures of Cluster Validity

The measures that we discuss in this section are all based on the premise
that any two objects that are in the same cluster should be in the same class
and vice versa. We can view this approach to cluster validity as involving
the comparison of two matrices: (1) the ideal cluster similarity rnatrix
discussedpreviously, which has a 1 in the i,jth entry if two objects, i. and j,
are in the same cluster and 0, otherwise, and (2) an ideal class similarity
matrix defined with respect to class labels, which has a 1 in the i,jth entry if
 8.5 Cluster Evaluation 551

two objects, i and j, belong to the same class,and a 0 otherwise. As before, we

can take the correlation of these two matrices as the measure of cluster validity.
This measure is known as the f statistic in clustering validation literature.
Example 8.16 (Correlation between Cluster and Class Matrices). To
demonstrate this idea more concretely, we give an example involving five data
points,pttpztpstp4,p5,two clusters,Ct: {pt,p2,p3} and Cz: {pq,p5}, and
two classes,Lr : {pt,,pz} and L2 : {pz,p+,p5}. The ideal cluster and class
similarity matrices are given in Tables 8.10 and 8.11. The correlation between
the entries of these two matrices is 0.359.

Table
8.10.ldeal
cluster matrix.
similaritv Table
8.11.ldeal
class matrix.
similariW
Point pl p2 p3 p4 p5 Point p1 p2 p3 p4 p5
p1 11100 p1 11000
p2 11100 p2 11000
p3 11100 p3 00111
p4 00011 p4 00111
p5 00011 p5 00111

More generally, we can use any of the measures for binary similarity that
we sav/ in Section 2.4.5. (For erxample,we can convert these two matrices into
binary vectors by appending the rows.) We repeat the definitions of the four
quantities used to define those similarity measures,but modify our descriptive
text to fit the current context. Specifically,we need to compute the following
four quantities for all pairs of distinct objects. (There are m(m - I) 12 stch
pairs, if m is the number of objects.)

,foo: number of pairs of objects having a different class and a different cluster
.for : number of pairs of objects having a different class and the same cluster
,fio : number of pairs of objects having the same class and a different cluster
.fir : number of pairs of objects having the same class and the same cluster

In particular, the simple matching coefficient, which is known as the Rand

statistic in this context, and the Jaccard coefficient are two of the most fre-
quently used cluster validity measures.

foo-l ft
Rand statistic : (8.18)
,foo* "for-l frc -l fn
552 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

.l1r
Jaccard coefficient : ( 8.1e)
fot -t fn'l ft
Example 8.17 (Rand and Jaccard Measures). Based on these formulas,
we can readily compute the Rand statistic and Jaccard coefficient for the
examplebasedon Tables8.10and 8.11. Noting that foo: 4, fot:2, frc:2,
and /rr : 2, the Rand statistic : (2 + 4) 110: 0.6 and the Jaccard coefficient
:2lQ+2+2):0.33. r
We also note that the four quantities, foo, fot, fis, and fi1, define a con-
t'ingency table as shown in Table 8.12.

Table8.12.Two-way tablefordetermining
contingency pairsof objects
whether areinthesameclass
andsamecluster.
Same Cluster Different Cluster
Same Ulass .t17 Iro
Diflerent Class .l07 ./ 00

Previously, in the context of association analysis-see Section 6.7.1-we

presented an extensive discussionof measuresof association that can be used
for this type of contingency table. (Compare Table 8.12 with Table 6.7.) Those
measurescan also be applied to cluster validity.

Cluster Validity for Hierarchical Clusterings

So far in this section, we have discussedsupervised measures of cluster va-

lidity only for partitional clusterings. Supervised evaluation of a hierarchical
clustering is more difficult for a variety of reasons, including the fact that a
preexisting hierarchical structure often does not exist. Here, we will give an
example of an approach for evaluating a hierarchical clustering in terms of a
(flat) set of classlabels, which are more likely to be available than a preexisting
hierarchical structure.
The key idea of this approach is to evaluate whether a hierarchical clus-
tering contains, for each class, at least one cluster that is relatively pure and
includes most of the objects of that class. To evaluate a hierarchical cluster-
ing with respect to this goal, we compute, for each class, the F-measure for
each cluster in the cluster hierarchy. For each class, we take the maximum F-
measureattained for any cluster. Finally, we calculate an overall F-measurefor
the hierarchical clustering by computing the weighted averageof all per-class
F-measures, where the weights are based on the class sizes. More formally,
 8.5 Cluster Evaluation 553

this hierarchical F-measure is defined as follows:

S- 7Zl
F-
z

where the maximum is taken over all clusters z at all levels, m1 is the number
of objects in classj, and rn is the total number of objects.

8.5.8 Assessing the Significance of Cluster Validity Measures

Cluster validity measuresare intended to help us measurethe goodnessof the

clusters that we have obtained. Indeed, they typically give us a single number
as a measure of that goodness. However, we are then faced with the problem
of interpreting the significance of this number, a task that may be even more
difficult.
The minimum and maximum values of cluster evaluation measures may
provide some guidance in many cases. For instance, by definition, a purity of
0 is bad, while a purity of 1 is good, at least if we trust our class labels and
want our cluster structure to reflect the class structure. Likewise, an entropy
of 0 is good, as is an SSE of 0.
Sometimes, however, there may not be a minimum or maximum value,
or the scale of the data may affect the interpretation. Also, even if there
are minimum and maximum values with obvious interpretations, intermediate
values still need to be interpreted. In some cases, we can use an absolute
standard. If, for example, we Erreclustering for utility, we may be willing to
tolerate only a certain level of error in the approximation of our points by a
cluster centroid.
But if this is not the case, then we must do something else. A common
approach is to interpret the value of our validity measure in statistical terms.
Specifically,we attempt to judge how likely it is that our observedvalue may
be achieved by random chance. The value is good if it is unusual; i.e., if it is
unlikely to be the result of random chance. The motivation for this approach
is that we are only interested in clusters that reflect non-random structure in
the data, and such structures should generate unusually high (low) values of
our cluster validity measure, at least if the validity measuresare designed to
reflect the presenceof strong cluster structure.
Example 8.18 (Significance of SSE). To show how this works, we present
an example basedon K-means and the SSE. Supposethat we want a measureof
how good the well-separatedclusters of Figure 8.30 are with respect to random
data. We generate many random sets of 100 points having the same range as
554 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

50

Figure8.34.Histogram
of SSEfor500random
datasets.

the points in the three clusters, find three clusters in each data set using K-
means, and accumulate the distribution of SSE values for these clusterings. By
using this distribution of the SSE values, we can then estimate the probability
of the SSE value for the original clusters. Figure 8.34 shows the histogram of
the SSE from 500 random runs. The lowest SSE shown in Figure 8.34 is 0.0173.
For the three clusters of Figure 8.30, the SSE is 0.0050. We could therefore
conservativelyclaim that there is less than a I%ochancethat a clustering such
as that of Figure 8.30 could occur by chance. r
To conclude, we stressthat there is more to cluster evaluation-supervised
or unsupervised than obtaining a numerical measure of cluster validity. Un-
lessthis value has a natural interpretation based on the definition of the mea-
sure, we need to interpret this value in some way. If our cluster evaluation
measure is defined such that lower values indicate stronger clusters, then we
can use statistics to evaluate whether the value we have obtained is unusually
low, provided we have a distribution for the evaluation measure. We have pre-
sented an example of how to find such a distribution, but there is considerably
more to this topic, and we refer the reader to the bibliographic notes for more
pointers.
Finally, even when an evaluation measure is used as a relative measure,
i.e., to compare two clusterings, we still need to assessthe significancein the
difference between the evaluation measuresof the two clusterings. Although
one value will almost always be better than another, it can be difficult to
determine if the difference is significant. Note that there are two aspects to
this significance: whether the differenceis statistically significant (repeatable)
 8.6 BibliographicNotes 555

and whether the magnitude of the differenceis meaningful with respect to the
application. Many would not regard a differenceof 0.1% as significant, even if
it is consistently reproducible.

8.6 Bibliographic Notes

Discussion in this chapter has been most heavily influenced by the books on
cluster analysis written by Jain and Dubes [396], Anderberg [374], and Kauf-
man and Rousseeuw [400]. Additional clustering books that may also be of
interest include those by Aldenderfer and Blashfield [373], Everitt et al. [388],
Hartigan [394],Mirkin [405],Murtagh [407],Romesburg[409],and Spzith[413].
A more statistically oriented approach to clustering is given by the pattern
recognition book of Duda et al. [385], the machine learning book of Mitchell
[406], and the book on statistical learning by Hastie et al. 1395]. A general
survey of clustering is given by Jain et al. 1394, while a survey of spatial data
mining techniques is provided by Han et al. 1393]. Behrkin [379] provides a
survey of clustering techniques for data mining. A good source of references
to clustering outside of the data mining field is the article by Arabie and Hu-
bert [376]. A paper by Kleinberg [401] provides a discussion of some of the
trade-offs that clustering algorithms make and proves that it is impossible to
for a clustering algorithm to simultaneously possessthree simple properties'
The K-means algorithm has a long history, but is still the subject of current
research. The original K-means algorithm was proposed by MacQueen [403].
The ISODATA algorithm by Ball and Hall [377] was an early, but sophisticated
version of K-means that employed various pre- and postprocessingtechniques
to improve on the basic algorithm. The K-means algorithm and many of its
variations are described in detail in the books by Anderberg PTal and Jain
and Dubes [396]. The bisecting K-means algorithm discussedin this chapter
was described in a paper by Steinbach et al. [4L4], and an implementation
of this and other clustering approaches is freely available for academic use in
the CLUTO (ClUstering TOolkit) package created by Karypis [382]. Bolev
[380] has created a divisive partitioning clustering algorithm (PDDP) based
on finding the first principal direction (component) of the data, and Savaresi
and Boley [411] have explored its relationship to bisecting K-means. Recent
variations of K-means are a new incremental version of K-means (Dhilion et al.
[383]), X-means (Pelleg and Moore 1408]),and K-harmonic means (Zhang et aI
pro-
[416]). Hamerly and Elkan [392] discuss some clustering algorithms that
duce better results than K-means. While some of the previously mentioned
approaches address the initialization problem of K-means in some manller)
556 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

other approachesto improving K-means initialization can also be found in the

work of Bradley and Fayyad [381]. Dhillon and Modha [384] present a gen-
eralization of K-means, called spherical K-means, that works with commonly
used similarity functions. A general framework for K-means clustering that
usesdissimilarity functions based on Bregman divergenceswas constructed by
Banerjeeet al. 1378].
Hierarchical clustering techniques also have a long history. Much of the
initial activity was in the area of taxonomy and is coveredin books by Jardine
and Sibson [398] and Sneath and Sokal 1412].General-purposediscussionsof
hierarchical clustering are also available in most of the clustering books men-
tioned above. Agglomerative hierarchical clustering is the focus of most work
in the area ofhierarchical clustering, but divisive approacheshave also received
some attention. For example, Zahn l4l5l describesa divisive hierarchical tech-
nique that uses the minimum spanning tree of a graph. While both divisive
and agglomerative approachestypically take the view that merging (splitting)
decisions are final, there has been some work by Fisher [389] and Karypis et
al. [399] to overcomethese limitations.
Ester et al. proposed DBSCAN 1387],which was later generalizedto the
GDBSCAN algorithm by Sander et al. 1410]in order to handle more general
types of data and distance measures)such as polygons whose closenessis mea-
sured by the degree of intersection. An incremental version of DBSCAN was
developed by Kriegel et al. [386]. One interesting outgrowth of DBSCAN is
OPTICS (Ordering Points To Identify the Clustering Structure) (Ankerst et
al. [375]), which allows the visualization of cluster structure and can also be
used for hierarchical clustering.
An authoritative discussion of cluster validity, which strongly influenced
the discussion in this chapter, is provided in Chapter 4 of Jain and Dubes'
clustering book [396]. More recent reviews of cluster validity are those of
Halkidi et al. [390,391] and Milligan [ 0a]. Silhouette coefficientsare described
in Kaufman and Rousseeuw'sclustering book [400]. The sourceof the cohesion
and separation measuresin Table 8.6 is a paper by Zhao and Karypis [417],
which also contains a discussion of entropy, purity, and the hierarchical F-
measure. The original source of the hierarchical F-measure is an article by
Larsen and Aone [402].

Bibliography
f3731 M. S. Aldenderfer and R. K. Blashfield. Cluster Analysr,s. Sage Publications, Los
Angeles, 1985
[374] M. R. Anderberg. Cluster Analysis for Appli.cati,ons. Academic Press, New York,
December 1973.
 Bibliography 557

13751M. Ankerst, M. M. Breunig, H.-P. Kriegel, and J. Sander. OPTICS: Ordering Points
To Identify the Clustering Structure. In Proc. of 1999 ACM-SIGMOD Intl. Conf. on
Managementof Data, pages49 60, Philadelphia,Pennsylvania,June 1999.ACM Press.
[376] P. Arabie, L. Hubert, and G. D. Soete. An overviewof combinatorialdata analysis.
In P. Arabie, L. Hubert, and G. D. Soete,editors, Clusteri,ngand Classifi,cation,
pages
188 2L7.World Scientific,Singapore,January 1996.
13771
G. Ball and D. Hall. A Clustering Technique for Summarizing Multivariate Data.
Behauior Sc'ience,12:153-155,March 1967.
[378]A. Banerjee, S. Merugu, I. S. Dhillon, and J. Ghosh. Clustering with Bregman Diver-
gences. In Proc. of the 2001 SIaIM Intl. Conf. on Data M'ini,ng, pages 234 245, Lake
Buena Vista, FL, April 2004.
[37e]P. Berkhin. Survey Of Clusterirrg Data Mining Techniques. Technical report, Accrue
Software. San Jose. CA. 2002.
[380]D. Boley. Principal Direction Divisive Partitioning. Data Mi,ni.ng and Knowledge
Discouery, 2(4):325-344, 1998.
P. S. Bradley and U. M. Fayyad. Refining Initial Points for K-Means Clustering. In
Proc. of the 15th Intl. Conf. on Machine Learni,ng, pages 91 99, Madison, WI, July
1998. Morgan Kaufmann Publishers Inc.
[382] C L U T O 2 . I . 7 : Software for Ciustering High-Dimensional Datasets.
/www.cs.umn.edu/-61ypis, November 2003.
[383] L S. Dhillon, Y. Guan, and J. Kogan. Iterative Clustering of High Dimensional Text
Data Augmented by Local Search In Proc. of the 2002 IEEE Intl. Conf. on Data
Mining, pages 131 138. ItrEE Computer Society, 2002
I. S. Dhillon and D. S. Modha. Concept Decompositions for Large Sparse Text Data
Using Clustering. M ach'ine Learn,,ing,a2Q I 2) :1a3 I75, 2001.
13851 R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classi.ficati,on John Wiley & Sons,
Inc., New York, second edition, 2001.
13861 M. Ester, H.-P. Kriegel, J. Sander, M. Wimmer, and X. Xu. Incremental Clustering
for Mining in a Data Warehousing Environment In Proc. of the 2/1th VLDB Conf.,
pages 323 333, New York City, August 1998. Morgan Kaufmann.
13871 M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A Density-Based Algorithm for Dis-
covering Clusters in Large Spatial Databases with Noise. In Proc. of the Znd Intl. Conf.
on Knowledge Discouery and Data M'in'ing, pages 226 231, Portland, Oregon, August
1996. AAAI Press.
[388] B. S. trveritt, S. Landau, and M. Leese. Cluster Analysis. Arnold Publishers, London,
fourth edition, May 2001.
l38elD. Fisher. Iterative Optimization and Simplification of Hierarchical Clusterings. -/ozr-
naL of Artif"cial Intelli,gence Research, 4:147-179, 1996.
M. Halkidi, Y. Batistakis, and M Vazirgiannis. Cluster validity methods: part L
SIGMOD Record (ACM Speci,alInterest Group on Management of Data),31(2):40 45,
June 2002.
M. Halkidi, Y. Batistakis, and N{. Vazirgiannis. Clustering validity checking methods:
part II. SIGMOD Record (ACM Special Interest Group on Managernent of Data), 3L
(3):19-27, Sept. 2002.
1392] G. Hamerly and C. trlkan. Alternatives to the k-means algorithm that find better
clusterings. In Proc. of the 1lth IntI. Conf . on Informat'ion and Knowledge Management,
pages 600 607, Mclean, Virginia, 2002. ACM Press.
558 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

[393] J. Han, M. Kamber, and A. Tirng. Spatial Clustering Methods in Data Mining: A
review. In H. J. Miller and J. Han, editors, Geographi'cData Mining and Knowledge
Discouery, pages 188-217. Taylor and Francis, London, December 2001.
[394] J. Hartigan. Clusteri,ng Algorithms. Wiley, New York, 1975.
[395] T. Hastie, R. Tibshirani, and J. H. Friedman. The Elements of Statistical Learning:
Data Mini,ng, Inference, Predi,ct'ion.Springer, New York, 2001.
f396] A. K. Jain and R. C. Dubes. Algorithms for Clustering Data. Prentice Hall
Advanced Reference Series. Prentice Hall, March 1988. Book available online at
http: / /www.cse. msu.edu/ -j ain/ Clustering-Jain-Dubes. pdf .
[397] A. K. Jain, M. N. Murty, and P. J. Flynn. Data clustering: A review. ACM Computing
Surueys,3L(3):26a 323, September 1999.
[398] N. Jardine and R. Sibson. Mathemat'ical Tanonomg. Wiley, New York, 1971.
13991G. Karypis, E.-H. Han, and V. Kumar. Multilevel Refinement for Hierarchical CIus-
tering. Technical Report TR 99-020, University of Minnesota, Minneapolis, MN, 1999.
[400] L. Kaufman and P. J. Rousseeuw. Find,ing Groups i,n Data: An Introducti,on to Cluster
Analgsi,s. Wiley Series in Probability and Statistics. John Wiley and Sons, New York,
November 1990.
[401] J. M. Kleinberg. An Impossibility Theorem for Clustering. ln Proc. of the 16th Annual
Conf. on Neural Informat'ion Processi.ng Systems, December, I 142002.
[402] B. Larsen and C. Aone. Fast and Effective Text Mining Using Linear-Time Document
Clustering. In Proc. of the 5th IntI. Conf. on Knowledge Discouery and Data M'ining,
pages 16-22, San Diego, California, 1999. ACM Press.
[403] J. MacQueen. Some methods for classification and analysis of multivariate observa-
tions. In Proc. of the Sth Berkeleg Sgmp. on Mathernatical Stati,sti,cs and, Probabi,Iitg,
pages 281-297. University of California Press, 1967.
1404] G. W. Milligan. Clustering Validation: Results and Implications for Applied Analyses.
In P. Arabie, L. Hubert, and G. D. Soete, editors, Clusteri,ng and Classif,cati,on,pages
345 375. World Scientific, Singapore, January 1996.
[405] B. Mirkin. Mathematical Classi,fication and Clustering, volume Il of Nonconuer Opti'-
mi:zati,onand Its Appli,cations. Kluwer Academic Publishers, August 1996.
[406] T. Mitchell. Machine Learning. McGraw-Hill, Boston, MA, 1997.
[407] F. Murtagh. Multidimensi.onal Clustering Algori.thms. Physica-Verlag, Heidelberg and
Vienna, 1985.
[408] D. Pelleg and A. W. Moore. X-means: Extending l(-means with Efficient Estimation
of the Number of Clusters. It Proc. of the 17th Intl. Conf. on Machine Learning, pages
727 734. Morgan Kaufmann, San Francisco, CA, 2000.
[409] C. Romesburg. Cluster Analysis for Researchers. Life Time Learning, Belmont, CA,
1984.
[410] J. Sander, M. Ester, H.-P. Kriegel, and X. Xu. Density-Based Clustering in Spatial
Databases: The Algorithm GDBSCAN and its Applications. Data M'ini,ng and"Knowl-
edgeDiscouery,2(2):169-194, 1998.
[411] S. M. Savaresi and D. Boley. A comparative analysis on the bisecting K-means and
the PDDP clustering algorithms. Intelligent Data Analysi.s, 8(4):345 362,2004.
14121P. H. A. Sneath and R. R. Sokal. Nurnerical Taronomg. Freeman, San Francisco, 1971.
f4131 H. Spd,th. Clusler Analgsi,s Algorithms for Data Red,uction and Classificati,on of Ob-
jecls, volume 4 of Computers and Thei,r Appli,cation. ElIis Horwood Publishers, Chich-
ester, 1980. ISBN 0-85312-14l-9.
 8.7 Exercises 559

[414] M. Steinbach, G. Karypis, and V. Kumar. A Comparison of Document Clustering

Techniques. In Proc. of KDD Workshop on Tert Minr,ng, Proc. of the 6th Intl. Conf.
on Knowled,geDiscouery and, Data Mintng, Boston, MA, August 2000.
f415] C. T. Zahn. Graph-Theoretical Methods for Detecting and Describing Gestalt Clusters.
IEEE Tfansactions on Computers,C-20(1):68 86, Jan.1971.
f416] B. Zhang, M. Hsu, and U. Dayal. K-Harmonic Means-A Data Clustering Algorithm.
Technical Report HPL-1999-124, Hewlett Packard Laboratories, Oct. 29 1999.
l4l7l Y. Zhao and G. Karypis. Empirical and theoretical comparisons of selected criterion
functions for document clusterirLg. Machine Lear-ning,55(3):311-331,2004.

8.7 Exercises
1. Consider a data set consisting of 22o data vectors, where each vector has 32
components and each component is a 4-byte value. Suppose that vector quan-
tization is used for compression and that 216 prototype vectors are used. How
many bytes of storage does t;hat data set take before and after compression and
what is the compressionratio?
2. Find all well-separated clusl;ersin the set of points shown in Figure 8.35.

a
a a
oo
oo a'
a
a
a oo a
oo
oo ao

iigure8.35.Points
forExercise
2.

J. Many partitional clustering algorithms that automatically determine the num-

ber of clusters claim that this is an advantase. List two situations in which this
is not the case.
4. Given K equally sized clusters, the probability that a randomly chosen initial
centroid will come from any given cluster is lf K, but the probability that each
cluster will have exactly one initial centroid is much lower. (It should be clear
that having one initial centroid in each cluster is a good starting situation for
K-means.) In general, if there are K clusters and each cluster has n points, then
the probability, p, of selecting in a sample of size K one initial centroid from each
cluster is given by Equation 8.20. (This assumessampling with replacement.)
From this formula we can ca,lculate, for example, that the chance of having one
initial centroid from each of four clusters is 4!l4a: 0.0938.
560 Chapter Cluster Analysis: Basic Concepts and Algorithms

number of wavs to select one centroid from each cluster KInK KI

v - (8.20)
number of wavs to select .I( centroids (Kn)n KK

(a) PIot the probability of obtaining one point from each cluster in a sample
of size K for values of K between 2 and 100.
(b) For K clusters, K : 10,100, and 1000, find the probability that a sample
of size 21( contains at least one point from each cluster. You can use
either mathematical methods or statistical simulation to determine the
answer.

5. Identify the clusters in Figure 8.36 using the center-, contiguity-, and density-
based definitions. AIso indicate the number of clusters for each case and give
a brief indication of your reasoning. Note that darkness or the number of dots
indicates density. If it helps, assume center-based means K-means, contiguity-
based means sinsle link. and densitv-based means DBSCAN.

(a) (b) (d)

Figure8,36.Clusters 5.
forExercise

6. For the following sets of two-dimensional points, (1) provide a sketch of how
they would be split into clusters by K-means for the given number of clusters
and (2) indicate approximately where the resulting centroids would be. Assume
that we are using the squared error objective function. If you think that there
is more than one possible solution, then please indicate whether each solution
is a global or local minimum. Note that the label of each diagram in Figure
8.37 matches the corresponding part of this question, e.g., Figure 8.37(a) goes
with part (a).

(") K : 2. Assuming that the points are uniformly distributed in the circle,
how many possible ways are there (in theory) to partition the points
into two clusters? What can you say about the positions of the two
centroids? (Again, you don't need to provide exact centroid locations,
just a qualitative description.)
 8.7 Exercises 561-

/tr-l-l']\ @-..*O
q#@(s@@ffift#
(a.) (b) (c) (d) (eJ

Figure
8.37.Diagrams
forExercise
6.

(b) 1( : 3. The distance between the edges of the circles is slightly greater
than the radii of the circles.
(") K : 3. The distance between the edges of the circles is much less than
the radii of the circles.
a\ K :2.
(e) 1( : 3. Hint: Use the symmetry of the situation and remember that we
are looking for a rough sketch of what the result would be.

7. Suppose that for a data set

o there are m points and K clusters,
o half the points and clusters are in "more dense" regions,
o half the points and clusters are in "less dense" regions, and
o the two regions are well-separated from each other.

For the given data set, which of the following should occur in order to minimize
the squared error when finding If clusters:

(a) Centroids should be equally distributed between more dense and less dense
regrons.
(b) More centroids should be allocated to the less dense region.
(c) More centroids should be allocated to the denser region.

Note: Do not get distracted by special cases or bring in factors other than
density. However, if you feel the true answer is different from any given above,
justify your response.

8 . Consider the mean of a cluster of objects from a binary transaction data set.
What are the minimum and maximum values of the components of the mean?
What is the interpretation of components of the cluster mean? Which compo-
nents most accurately charzr,cterizethe objects in the cluster?
9 . Give an example of a data set consisting of three natural clusters, for which
(almost always) K-means would likely find the correct clusters, but bisecting
K-means would not.
562 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

10. Would the cosine measure be the appropriate similarity measureto use with K-
means clustering for time series data? Why or why not? If not, what similarity
measure would be more appropriate?
1 1 . Total SSE is the sum of the SSE for each separateattribute. What does it mean
if the SSE for one variable is low for all clusters? Low for just one cluster? High
for all clusters? High for just one cluster? How could you use the per variable
SSE information to improve your clustering?
12. The leader algorithm (Hartigan 1394])represents each cluster using a point,
known as a leader, and assigns each point to the cluster corresponding to the
closest leader, unless this distance is above a user-specifiedthreshold. In that
case,the point becomesthe leader of a new cluster.

(a) What are the advantages and disadvantages of the leader algorithm as
compared to K-means?
(b) Suggest ways in which the leader algorithm might be improved.

1 3 The Voronoi diagram for a set of 1( points in the plane is a partition of all
the points of the plane into K regions, such that every point (of the plane)
is assigned to the closest point among the 1( specified points. (See Figure
8.38.) What is the relationship between Voronoi diagrams and K-means clus-
ters? What do Voronoi diagrams tell us about the possible shapes of K-means
clusters?

Figure8.38.Voronoi 13.
forExercise
diagram

L 4 . You are given a data set with 100 records and are asked to cluster the data.
You use K-means to cluster the data, but for all values of K, | < 1{ < 100,
the K-means algorithm returns only one non-empty cluster. You then apply
an incremental version of K-means, but obtain exactly the same result. How is
this possible? How would single link or DBSCAN handle such data?
r o . Traditional agglomerative hierarchical clustering routines merge two clusters at
each step. Does it seem likely that such an approach accurately captures the
 8.7 Exercises 563

(nested) cluster structure of a set of data points? If not, explain how you might
postprocessthe data to obtain a more accurate view of the cluster structure.

1 6 . Use the similarity matrix in Table 8.13 to perform single and complete link
hierarchical clustering. Show your results by drawing a dendrogram. The den-
drogram should clearly show the order in which the points are merged.

Table8.13.Similarity
matrix
forExercise
16.
p1 ^t pr p4 p5
p1 1.00 0.10 0.41 0.55 0.35
p2 0.10 1.00 0.64 0.47 0.98
p3 0.41 0.64 1.00 0.44 0.85
p4 0.55 0.47 0.44 1.00 0.76
p5 0.35 0.98 0.85 0.76 1.00

17. Hietarchical clustering is sometimes used to generate K clusters, K > I by

taking the clusters atl,he Ktt'Ievel of the dendrogram. (Root is at level 1.) By
looking at the clusters produced in this way, we can evaluate the behavior of
hierarchical clustering on different types of data and clusters, and also compare
hierarchical approachesto K-means.
The following is a set of one-dimensionalpoints: {6,12,18,24,30,42,48}.

(a) For each of the following sets of initial centroids, create two clusters by
assigning each point to the nearest centroid, and then calculate the total
squared error for each set of two clusters. Show both the clusters and the
total squared error for each set of centroids.
i. {18,45}
ii. {15,40}
(b) Do both sets of centroids represent stable solutions; i.e., if the K-means
algorithm was run on this set of points using the given centroids as the
starting centroids, would there be any change in the clusters generated?
(c) What are the two clusters produced by single link?
(d) Which technique, K-means or single link, seems to produce the "most
natural" clustering in this situation? (For K-means, take the clustering
with the lowest squared error.)
(e) What definition(s) of clustering does this natural clustering correspond
to? (Well-separated, center-based,contiguous, or density.)
(f) What well-known characteristic of the K-means algorithm explains the
previous behavior?
564 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

1 8 . Supposewe find K clusters using Ward's method, bisecting K-means, and ordi-
nary K-means. Which of these solutions represents a local or global minimum?
Explain.
10 Hierarchical clustering algorithms require O(m2 Iog(m)) time, and consequently,
are impractical to use directly on larger data sets. One possible technique for
reducing the time required is to sample the data set. For example, if l( clusters
are desired and 1/m points are sampled from the m points, then a hierarchi-
cal clustering algorithm will produce a hierarchical clustering in roughly O(rn)
time. K clusters can be extracted from this hierarchical clustering by taking
the clusters on the Kth level of the dendrogram. The remaining points can
then be assigned to a cluster in linear time, by using various strategies. To give
a specific example, the centroids of the 1( clusters can be computed, and then
each of the m - J* remaining points can be assigned to the cluster associated
with the closest centroid.

For each of the following types of data or clusters, discussbriefly if (1) sampling
will causeproblems for this approach and (2) what those problems are. Assume
that the sampling technique randomly choosespoints from the total set of m
points and that any unmentioned characteristics of the data or clusters are as
optimal as possible. In other words, focus only on problems caused by the
particular characteristic mentioned. Finally, assume that K is very much less
than m.

(a) Data with very different sized clusters.

(b) High-dimensional data.
(c) Data with outliers, i.e., atypical points.
(d) Data with highly irregular regions.
(e) Data with globular clusters.
(f) Data with widely different densities.
(g) Data with a small percentage of noise points.
(h) Non-Euclidean data.
(i) Euclidean data.
(j) Data with many and mixed attribute types.

20. Consider the following four faces shown in Figure 8.39. Again, darkness or
number of dots represents density. Lines are used only to distinguish regions
and do not represent points.

(a) For each figure, could you use single link to find the patterns represented
by the nose, eyes, and mouth? Explain.
(b) For each figure, could you use K-means to find the patterns represented
by the nose, eyes)and mouth? Explain.
 8.7 Exercises 565

@@
@
\#

(b) (d)

Figure
8.39.Figure
forExercise
20.

(c) What limitation does clustering have in detecting all the patterns formed
by the points in Figure 8.39(c)?

21. Compute the entropy and purity for the confusion matrix in Table 8.14.

Table8.14.Confusion
matrix
forExercise
21.
Cluster Entertainment Financial Foreign Metro National Sports Total
#1 I 1 0 11 / 676 693
4' 27 89 333 827 253 .).) 7562
-JJ.'
326 40,1 8 105 16 29 949
Total 354 555 347 943 273 738 3204

22. You are given two sets of 100 points that fall within the unit square. One set
of points is arranged so that the points are uniformly spaced. The other set of
points is generated from a uniform distribution over the unit square.

(a) Is there a difference between the two sets of points?

(b) If so, which set of points will typically have a smaller SSE for K:10
clusters?
(c) What will be the behavior of DBSCAN on the uniform data set? The
random data set?

23. Using the data in Exercise 24, compute the silhouette coefficient for each point,
each of the two clusters, and the overall clustering.
24. Given the set of cluster labels and similarity matrix shown in Tables 8.15 and
8.16, respectively, compute the correlation between the similarity matrix and
the ideal similarity matrix, i.e., the matrix whose ijth enfty is 1 if two objects
belons to the same cluster. and 0 otherwise.
566 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

Table8,15.Table labels
ofcluster forExercise forExercise
matrix
24. Table8.16.Similarity 24.
Point Cluster Label Point P1 P2 P3 P4
P1 1 P1 1 0 . 8 0.65 t.r.5l)
P2 1 P2 0.8 1 0.7 0.6
'I
P3 2 P3 tr.oc 0.7 0.9
P4 2 P4 u . 5 5 0 . 6 0.9 1

25. Compute the hierarchical F-measure for the eight objects {p1, p2, p3, p4, p5,
p6, p7, p8) and hierarchical clustering shown in Figure 8.40. Class A contains
points pI,p2, and p3, while p4, p5, p6, p7, and p8 belong to classB.

(p1,p2,p3,p4, p5,p6, p7,p8)

{ p 1 ,p 2 ,p 4 ,p 5 , }

8.40.Hierarchical
Figure 25.
forExercise
clustering

26. Compute the cophenetic correlation coefficient for the hierarchical clusterings
in Exercise 16. (You will need to convert the similarities into dissimilarities.)
27. Prove Equation 8.14.
28. Prove Equation 8.16.
29. Prove that Df-rD,.. (r - m)(m - *o) :0. This fact was used in the proof
that TSS : SSE + SSB in Section8.5.2.
30. Clusters of documents can be summarized by finding the top terms (words) for
the documents in the cluster, e.g., by taking the most frequent k terms, where
k is a constant, say 10, or by taking all terms that occur more frequently than
a specified threshold. Suppose that K-means is used to find clusters of both
documents and words for a document data set.

(a) How might a set of term clusters defined by the top terms in a document
cluster differ from the word clusters found by clustering the terms with
K-means?
(b) How could term clustering be used to define clusters of documents?

3 1 . We can represent a data set as a collection of object nodes and a collection of

attribute nodes, where there is a link between each object and each attribute,
 8.7 Exercises 567

and where the weight of that link is the value of the object for that attribute. For
sparse data, if the value is 0, the Iink is omitted. Bipartite clustering attempts
to partition this graph into disjoint clusters, where each cluster consists of a
set of object nodes and a set of attribute nodes. The objective is to maximize
the weight of links between the object and attribute nodes of a cluster, while
minimizing the weight of links between object and attribute links in different
clusters. This type of clustering is also known as co-clustering since the
objects and attributes are clustered at the same time.

(a) How is bipartite clust;ering (co-clustering) different from clustering the

sets of objects and attributes separately?
(b) Are there any cases in which these approaches yield the same clusters?
(c) What are the strengths and weaknessesof co-clustering as compared to
ordinary clustering?

32. In Figure 8.41, match the sirnilarity matrices, which are sorted according to
cluster labels, with the sets of points. Differences in shading and marker shape
distinguish between clusters, and each set of points contains 100 points and
three clusters. In the set of points labeled 2, there are three very tight, equal-
sized clusters.
568 Chapter 8 Cluster Analysis: Basic Concepts and Algorithms

'1

09
08
07
06
>0 5
o4
03

,i#' ds}{
..oh"
4*'
04 06 08 '1
02
x
B

40 60 80 100 2A 40 60

matrices
andsimilarity
8,41.Points
Figure 32.
forExercise
ClusterAnalysis:
Additionallssuesand
Algorithms
A large number of clustering algorithms have been developed in a variety
of domains for different types of applications. None of these algorithms is
suitable for all types of data, clusters, and applications. In fact, it seemsthat
there is arlwaysroom for a new clustering algorithm that is more effi.cientor
better suited to a particular type of data, cluster, or application. Instead,
we can only claim that we have techniques that work well in some situations.
The reason is that, in many cases,what constitutes a good set of clusters is
open to subjective interpretation. Furthermore) when an objective measure is
employed to give a precise definition of a cluster, the problem of finding the
optimal clustering is often computationally infeasible.
This r:hapter focuseson important issuesin cluster analysis and explores
the concerptsand approachesthat have been developed to address them. We
begin with a discussionof the key issuesof cluster analysis, namely, the char-
acteristics of data, clusters, and algorithms that strongly impact clustering.
These issuesare important for understanding, describing, and comparing clus-
tering techniques,and provide the basis for deciding which technique to use in
a specific situation. For example) many clustering algorithms have a time or
spaceconnplexityof O(m2) (m being the number of objects) and thus, are not
suitable fbr large data sets. We then discuss additional clustering techniques.
For each rbechnique, we describethe algorithm, including the issuesit addresses
and the rnethods that it uses to address them. We conclude this chapter by
providing; some general guidelines for selecting a clustering algorithm for a
given application.
57O Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

9.1 Characteristics of Data, Clusters, and Cluster-

ing Algorithms
This section explores issuesrelated to the characteristics of data, clusters, and
algorithms that are important for a broad understanding of cluster analysis.
Some of these issuesrepresent challenges,such as handling noise and outliers.
Other issues involve a desired feature of an algorithm, such as an ability to
produce the same result regardlessof the order in which the data objects are
processed.The discussionin this section, along with the discussionof different
types of clusterings in Section 8.1.2 and different types of clusters in Section
8.1.3, identifies a number of "dimensions" that can be used to describe and
compare various clustering algorithms and the clustering results that they pro-
duce. To illustrate this, we begin this section with an example that compares
two clustering algorithms that were described in the previous chapter, DB-
SCAN and K-means. This is followed by a more detailed description of the
characteristics of data, clusters, and algorithms that impact cluster analysis.

9.1.1 Example: Comparing K-means and DBSCAN

To simplify the comparison, we assumethat that there are no ties in distances

for either K-means or DBSCAN and that DBSCAN always assigns a border
point that is associatedwith several core points to the closest core point.

o Both DBSCAN and K-means are partitional clustering algorithms that

assign each object to a single cluster, but K-means typically clusters all
the objects, while DBSCAN discards objects that it classifiesas noise.

o K-means uses a prototype-based notion of a cluster; DBSCAN uses a

density-basedconcept.

o DBSCAN can handle clusters of different sizes and shapes and is not
strongly affected by noise or outliers. K-means has difficulty with non-
globular clusters and clusters of different sizes. Both algorithms can
perform poorly when clusters have widely differing densities.

o K-means can only be used for data that has a well-defined centroid,
such as a mean or median. DBSCAN requires that its definition of
density, which is based on the traditional Euclidean notion of density,
be meaningful for the data.

o K-means can be applied to sparse, high-dimensional data, such as doc-

ument data. DBSCAN typically performs poorly for such data because
 9.1_ Characteristics of Data, Clusters, and Clustering Algorithms 57L

the traditional Euclidean definition of density does not work well for
high-dimensional data.

The original versions of K-means and DBSCAN were designed for Eu-
clidean data, but both have been extended to handle other types ofdata.

DIISCAN makes no assumption about the distribution of the data. The

basic K-means algorithm is equivalent to a statistical clustering approach
(nLixture models) that assumesall clusters come from spherical Gaussian
dir;tributions with different means but the same covariance matrix. See
Section 9.2.2.

o DI3SCAN and K-means both look for clusters using all attributes, that
is, they do not look for clusters that may involve only a subset of the
attbributes.

K-means can find clusters that are not well separated, even if they over-
lap (see Figure 8.2(b)), but DBSCAN merges clusters that overlap.

Tlre K-means algorithm has a time complexity of O(m), while DBSCAN

tdkes O(m2) time, except for special casessuch as low-dimensional Eu-
clidean data.

o DIISCAN produces the same set of clusters from one run to another,
wtrile K-means, which is typically used with random initialization of
centroids. does not.

DIISCAN automatically determinesthe number of clusters; for K-means,

th,e number of clusters needs to be specified as a parameter. However,
DIISCAN has two other parameters that must be specified, Eps and
Mi,nPts.

K-means clustering can be viewed as an optimization problem; i.e., min-

imize the sum of the squared error of each point to its closest centroid,
and as a specific case of a statistical clustering approach (mixture mod-
els). DBSCAN is not based on any formal model.

9.I.2 Data Characteristics

The foll:wing are some characteristics of data that can strongly affect cluster
analvsis,,
572 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

High Dimensionality In high-dimensional data sets, the traditional Eu-

clidean notion of density, which is the number of points per unit volume,
becomesmeaningless. To see this, consider that as the number of dimensions
increases,the volume increasesrapidly, and unlessthe number of points grov/s
exponentially with the number of dimensions,the density tends to 0. (Volume
is exponential in the number of dimensions. For instance, a hypersphere with
radius, r, and dimension, d, has volume proportional to rd.) Also, proximity
tends to become more uniform in high-dimensional spaces. Another way to
view this fact is that there are more dimensions (attributes) that contribute to
the proximity between two points and this tends to make the proximity more
uniform. Since most clustering techniques are based on proximity or density,
they can often have difficulty with high-dimensional data. One approach to
addressing such problems is to employ dimensionality reduction techniques.
Another approach, as discussedin Sections 9.4.5 andg.4.7, is to redefine the
notions of proximity and density.

Size Many clustering algorithms that work well for small or medium-size
data sets are unable to handle larger data sets. This is addressed further
in the discussion of the characteristics of clustering algorithms scalability is
one such characteristic-and in Section 9.5, which discussesscalableclustering
algorithms.

Sparseness Sparsedata often consists of asymmetric attributes, where zeto

values are not as important as non-zero values. Therefore, similarity measures
appropriate for asymmetric attributes are commonly used. However, other,
related issues also arise. For example, are the magnitudes of non-zero en-
tries important, or do they distort the clustering? In other words, does the
clustering work best when there are only two values, 0 and 1?

Noise and Outliers An atypical point (outlier) can often severely degrade
the performance of clustering algorithms, especially algorithms such as K-
means that are prototype-based. On the other hand, noise can cause tech-
niques, such as single link, to join clusters that should not be joined. In some
cases, algorithms for removing noise and outliers are applied before a clus-
tering algorithm is used. Alternatively, some algorithms can detect points
that represent noise and outliers during the clustering processand then delete
them or otherwise eliminate their negative effects. In the previous chapter,
for instanceTwe saw that DBSCAN automatically classifieslow-density points
as noise and removes them from the clustering process. Chameleon (Section
 9.1 Characteristics of Data, Clusters, and Clustering Algorithms 573

9.4.4),SNN density-basedclustering(Section9.4.8),and CURE (Section9.5.3)

are three of the algorithms in this chapter that explicitly deal with noise and
outliers during the clustering process.

Type of Attributes and Data Set As discussedin Chapter 2, data sets

can be ofvarious types, such as structured, graph, or ordered, while attributes
can be categorical (nominal or ordinal) or quantitative (interval or ratio), and
are binary, discrete, or continuous. Different proximity and density measures
are appropriate for different types of data. In some situations, data may need
to be discretized or binarized so that a desired proximity measureor clustering
algorithm can be used. Another complication occurs when attributes are of
widely differing types, e.g., continuous and nominal. In such cases,proximity
and density are more difficult to define and often more ad hoc. Finally, special
data structures and algorithms may be neededto handle certain types of data
efficiently.

Scale Different attributes, e.g., height and weight, m&y be measured on dif-
ferent scales. These differencescan strongly affect the distance or similarity
between two objects and, consequently,the results of a cluster analysis. Con-
sider clustering a group of people based on their heights, which are measured
in meters, and their weights, which are measured in kilograms. If we use Eu-
clidean distance as our proximity measure)then height will have little impact
and people will be clustered mostly based on the weight attribute. If, however,
we standardize each attribute by subtracting off its mean and dividing by its
standard deviation, then we will have eliminated effects due to the difference
in scale. More generally, normalization techniques, such as those discussedin
Section 2.3.7, are typically used to handle these issues.

Mathematical Properties of the Data Space Some clustering tech-

niques calculate the mean of a collection of points or use other mathematical
operations that only make sensein Euclidean space or in other specific data
spaces. Other algorithms require that the definition of density be meaningful
for the data.

9.1.3 Cluster Characteristics

The different types of clusters, such as prototype-, graph-, and density-based,
were described earlier in Section 8.1.3. Here, we describe other important
characteristics of clusters.
574 Chapter I Cluster Analysis: Additional Issuesand Algorithms

Data Distribution Some clustering techniques assume a particular type

of distribution for the data. More specifically, they often assume that data
can be modeled as arising from a mixture of distributions, where each cluster
correspondsto a distribution. Clustering based on mixture models is discussed
in Section9.2.2.

Shape Some clusters are regularly shaped, e.g., rectangular or globular, but
in general, clusters can be of arbitrary shape. Techniques such as DBSCAN
and single link can handle clusters of arbitrary shape, but prototype-based
schemesand some hierarchical techniques, such as complete link and group
average) cannot. Chameleon (Section 9.4.4) and CURE (Section 9.5.3) are
examplesof techniquesthat were specifically designedto addressthis problem.

Differing Sizes Many clustering methods, such as K-means, don't work well
when clusters have different sizes. (See Section 8.2.4.) This topic is discussed
further in Section9.6.

Differing Densities Clusters that have widely varying density can cause
problems for methods such as DBSCAN and K-means. The SNN density-
based clustering technique presented in Section 9.4.8 addressesthis issue.

Poorly Separated Clusters When clusters touch or overlap, some cluster-

ing techniquescombine clusters that should be kept separate. Even techniques
that find distinct clusters arbitrarily assign points to one cluster or another.
Ftzzy clustering, which is described in Section 9.2.1, is one technique for deal-
ing with data that does not form well-separatedclusters.

Relationships among Clusters In most clustering techniques,there is no

explicit consideration of the relationships between clusters, such as their rel-
ative position. Self-organizing maps (SOM), which are described in Section
9.2.3, are a clustering technique that directly considers the relationships be-
tween clusters during the clustering process. Specifically,the assignment of a
point to one cluster affects the definitions of nearby clusters.

Subspace Clusters Clusters may only exist in a subset of dimensions (at-

tributes), and the clusters determined using one set of dimensions may be
quite different from the clusters determined by using another set. While this
issue can arise with as few as two dimensions, it becomes more acute as di-
mensionality increases,since the number of possible subsets of dimensions is
 9.1 Characteristics of Data, Clusters, and Clustering Algorithms 575

exponential in the total number of dimensions. For that reason, it is not feasi-
ble to simply look for clusters in all possible subsets of dimensions unless the
number of dimensions is relatively low.
One approach is to apply feature selection, which was discussedin Sec-
tion 2.3.4. However, this approach assumesthat there is only one subset of
dimensions in which the clusters exist. In reality, clusters can exist in many
distinct subspaces(sets of dimensions), some of which overlap. Section 9.3.2
considerstechniques that addressthe general problem of subspaceclustering,
i.e., of finding both clusters and the dimensions they span.

9.L.4 General Characteristics of Clustering Algorithms

Clustering algorithms are quite varied. We provide a general discussion of
important characteristicsof clustering algorithms here, and make more specific
comments during our discussionof particular techniques.

Order Dependence For some algorithms, the quality and number of clus-
ters produced can vary, perhaps dramatically, depending on the order in which
the data is processed.While it would seemdesirable to avoid such algorithms,
sometimesthe order dependenceis relatively minor or the algorithm may have
other desirable characteristics. SOM (Section 9.2.3) is an example of an algo-
rithm that is order dependent.

Nondeterminism Clustering algorithms, such as K-means, are not order-

dependent, but they produce different results for each run since they rely on
an initialization step that requires a random choice. Because the quality of
the clusters can vary from one run to another, multiple runs can be necessary.

Scalability It is not unusual for a data set to contain millions of objects, and
the clustering algorithms used for such data sets should have linear or near-
linear time and space complexity. Even algorithms that have a complexity of
O(*') are not practical for large data sets. Furthermore, clustering techniques
for data sets cannot always assumethat all the data will fit in main memory or
that data elements can be randomly accessed.Such algorithms are infeasible
for large data sets. Section 9.5 is devoted to the issue of scalabilitv.

Parameter Selection Most clustering algorithms have one or more pa-

rameters that need to be set by the user. It can be difficult to choose the
576 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

proper valuesl thus, the attitude is usually, "the fewer parameters, the bet-
ter." Choosing parameter values becomes even more challenging if a small
change in the parameters drastically changes the clustering results. Finally,
unless a procedure (which may involve user input) is provided for determining
parameter values, a user of the algorithm is reduced to using trial and error
to find suitable parameter values.
Perhaps the most well-known parameter selectionproblem is that of "choos-
ing the right number of clusters" for partitional clustering algorithms, such as
K-means. One possible approach to that issue is given in Section 8.5.5, while
referencesto others are provided in the bibliographic notes.

Tlansforming the Clustering Problem to Another Domain One ap-

proach taken by some clustering techniques is to map the clustering problem
to a problem in a different domain. Graph-basedclustering, for instance, maps
the task of finding clusters to the task of partitioning a proximity graph into
connected components.

TYeating Clustering as an Optirnization Problem Clustering is often

viewed as an optimization problem: divide the points into clusters in a way
that maximizes the goodnessof the resulting set of clusters as measured by a
user-specifi.edobjective function. For example, the K-means clustering algo-
rithm (Section 8.2) tries to find the set of clusters that minimizes the sum of
the squared distance of each point from its closest cluster centroid. In theory,
such problems can be solved by enumerating all possible sets of clusters and
selectingthe one with the best value of the objective function, but this exhaus-
tive approach is computationally infeasible. For this reason, many clustering
techniques are based on heuristic approachesthat produce good, but not op-
timal clusterings. Another approach is to use objective functions on a greedy
or local basis. In particular, the hierarchical clustering techniques discussedin
Section 8.3 proceed by making locally optimal (greedy) decisionsat each step
of the clustering process.

Road Map

We arrange our discussionof clustering algorithms in a manner similar to that

of the previous chapter, grouping techniques primarily according to whether
they are prototype-based, density-based, or graph-based. There is, however,
a separate discussion for scalable clustering techniques. We conclude this
chapter with a discussionof how to choosea clustering algorithm.
 9.2 Prototype-BasedClustering 577

9.2 Prototype-Based Clustering

In prototype-based clustering, a cluster is a set of objects in which any object
is closer to the prototype that defines the cluster than to the prototype of
any other cluster. Section 8.2 described K-means, a simple prototype-based
clustering algorithm that uses the centroid of the objects in a cluster as the
prototype of the cluster. This section discussesclustering approaches that
expand on the concept of prototype-based clustering in one or rrore ways, as
discussednext:

o Objects are allowed to belong to more than one cluster. More specifically,
an object belongs to every cluster with some weight. Such an approach
addressesthe fact that some objects are equally close to several cluster
prototypes.

o A cluster is modeled as a statistical distribution, i.e., objects are gen-

erated by a random process from a statistical distribution that is char-
acterized by a number of statistical parameters, such as the mean and
variance. This viewpoint generalizesthe notion of a prototype and en-
ables the use of well-establishedstatistical techniques.

o Clusters are constrained to have fixed relationships. Most commonly,

these relationships are constraints that specify neighborhood relation-
ships; i.e., the degree to which two clusters are neighbors of each other.
Constraining the relationships among clusters can simplify the interpre-
tation and visualization of the data.

We consider three specific clustering algorithms to illustrate these exten-

sions of prototype-based clustering. Ftzzy c-means uses concepts from the
field of fitzzy logic and fiizzy set theory to propose a clustering scheme,which
is much like K-means, but which does not require a hard assignmentof a point
to only one cluster. Mixture model clustering takes the approach that a set of
clusters can be modeled as a mixture of distributions, one for each cluster. The
clustering schemebased on Self-Organizing Maps (SOM) performs clustering
within a framework that requires clusters to have a prespecified relationship
to one another, e.8., & two-dimensional grid structure.

9.2.L hnzy Clustering

If data objects are distributed in well-separated groups, then a crisp clas-
sification of the objects into disjoint clusters seems like an ideal approach.
However, in most cases,the objects in a data set cannot be partitioned into
578 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

well-separatedclusters, and there will be a certain arbitrariness in assigningan

object to a particular cluster. Consider an object that lies near the boundary
of two clusters, but is slightly closer to one of them. In many such cases,it
might be more appropriate to assign a weight to each object and each cluster
that indicates the degree to which the object belongs to the cluster. Mathe-
matically, wii is the weight with which object 4 belongs to cluster Cr.
As shown in the next section, probabilistic approaches can also provide
such weights. While probabilistic approaches are useful in many situations,
there are times when it is difficult to determine an appropriate statistical
model. In such cases, non-probabilistic clustering techniques are needed to
provide similar capabilities. Fuzzy clustering techniques are based on fitzzy
set theory and provide a natural technique for producing a clustering in which
membership weights (the w61)have a natural (but not probabilistic) interpre-
tation. This section describes the general approach of finzy clustering and
provides a specific example in terms of finzy c-means (fuzzy K-means).

F\rzzy Sets

Lotfi Zadeh introduced frzzy set theor:y and finzy logic in 1965 as a way
of dealing with imprecision and uncertainty. Briefly, fuzzy set theory allows an
object to belong to a set with a degreeof membership between 0 and 1, while
htzzy logic allows a statement to be true with a degreeof certainty between 0
and 1. Traditional set theory and logic are special casesof their finzy counter-
parts that restrict the degree of set membership or the degree of certainty to
be either 0 or 1. Ftzzy concepts have been applied to many different areas,in-
cluding control systems, pattern recognition, and data analysis (classification
and clustering).
Consider the following example of hnzy logic. The degree of truth of the
statement "It is cloudy" can be defined to be the percentageof cloud cover in
the sky, e.g., if the sky is 50% covered by clouds, then we would assign "It is
cloudy" a degreeof truth of 0.5. If we have two sets, "cloudy days" and "non-
cloudy days," then we can similarly assign each day a degree of membership
in the two sets. Thus, if a day were 2570cloudy, it would have a 25Vodegreeof
membership in "cloudy days" and a 75Todegreeof membership in "non-cloudy
days.tt

F\tzzy Clusters

Assume that we have a set of data points X : {*t, . . . ,x-}, where each point,
x4, is an n-dimensionalpoint, i.e., x4 : (rn,. ..,rin). A collectionof finzy
 9.2 Prototype-Based Clustering 579

clusters, Ct, Cz, ..., Cn is a subset of all possiblefizzy subsetsof .t. (This
simply means that the membership weights (degrees),wii,have been assigned
values between 0 and 1 for each point, xi, o,ndeach cluster, Ci.) However, we
also want to impose the following reasonableconditions on the clusters in order
to ensure that the clusters form what is called a fitzzy psuedo-partition.

1. All the weights for a given point, xi, add up to 1.

k
D.ej : I
j:t

2. Each cluster, Ci, contains, with non-zero weight, at least one point, but
does not contain, with a weight of one, all of the points.
rn
0< tuUlm
i:I

F\tzzy c-means

While there are many types of fuzzy chstering-indeed, many data analysis
algorithms can be "ftrzzified" we only consider the fuzzy vercion of K-means,
which is called fiizzy c-rneans. In the clustering literature, the version of K-
means that does not use incremental updates of cluster centroids is sometimes
referred to as c-means, and this was the term adapted by the fuzzy community
for the finzy versionof K-means. The fizzy c-meansalgorithm, also sometimes
known as FCM, is given by Algorithm 9.1.

Algorithm 9.1 Basic fizzy c-meansalgorithm.

1: Select an initial fuzzy pseudo-partition, i.e., assign values to all the tu23.
2: repeat
3: Compute the centroid of each cluster using the fuzzy pseudo-partition.
4: Recompute the finzy pseudo-partition, i.e., the wii.
5: until The centroids don't change.
(Alternative stopping conditions are "if the ch,angein the error is below a specified
threshold" or "if the absolute change in any uri.; is below a given threshold.")

After initialization, FCM repeatedly computes the centroids of each cluster

and the fuzzy pseudo-partition until the partition does not change. FCM
is similar in structure to the K-means algorithm, which after initialization,
alternates between a step that updates the centroids and a step that assigns
each object to the closest centroid. Specifically, computing a fitzzy pseudo-
partition is equivalent to the assignment step. As with K-means, FCM can
580 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

be interpreted as attempting to minimize the sum of the squared error (SSE),

although FCM is based on a fwzy version of SSE. Indeed, K-means can be
regarded as a special case of FCM and the behavior of the two algorithms is
quite similar. The details of FCM are described below.

Computing SSE The definition of the sum of the squared error (SSE) is
modified as follows:

ktn
. . ., Cu): t
SSE(C1,C2, \uf,dtst(xt, cj)2 (e.1)
.J-L .-L

where c7 is the centroid of the j'h cluster and p, which is the exponent that
determines the influence of the weights, has a value between 1 and oo. Note
that this SSE is just a weighted version of the traditional K-means SSE given
in Equation 8.1.

Initialization Random initialization is often used. In particular, weights

are chosen randomly, subject to the constraint that the weights associated
with any object must sum to 1. As with K-means, random initialization is
simple, but often results in a clustering that represents a local minimum in
terms of the SSE. Section 8.2.1, which contains a discussionon choosinginitial
centroids for K-means, has considerablerelevancefor FCM as well.

Computing Centroids The definition of the centroid given in Equation g.2

can be derived by finding the centroid that minimizes the fitzzy SSE as given
by Equation 9.1. (Seethe approach in Section 8.2.6.) For a cluster, C7, the
corresponding centroid, c7, is defined by the following equation:

cj: S:Lwlrx;l\ul,
n i\-
(s.2)
i-r i:\

The finzy centroid definition is similar to the traditional definition except

that all points are considered (any point can belong to any cluster, at least
somewhat) and the contribution of each point to the centroid is weighted by
its membership degree. In the case of traditional crisp sets, where all uii are
either 0 or 1, this definition reducesto the traditional definition of a centroid.
There are a few considerations when choosing the value of p. Choosing
p : 2 simplifies the weight update formula-see Equation 9.4. However, 1f p
 9.2 Prototype-BasedCiustering 581

is chosen to be near L, then fuzzy c-meansbehaveslike traditional K-means.

Going in the other direction, as p gets larger, all the cluster centroids approach
the global centroid of all the data points. In other words, the partition becomes
fuzzier as p increases.

Updating t};re Fluzzy Pseudo-partition Since the fuzzy pseudo-partition

is defined by the weight, this step involves updating the weights ?/r;7associ-
ated with the i,th point and jtb cluster. The weight update formula given in
Equation 9.3 can be derived by minimizing the SSE of Equation 9.1 subject
to the constraint that the weishts sum to 1.

ysit : (Lldist(x6,"j)'),-
/ fplonu(*.,")')h (e.3)
, n:t
This formula may appear a bit mysterious. However, note that if P :2,
then we obtain Equation 9.4, which is somewhat simpler. We provide an
intuitive explanation of Equation 9.4, which, with a slight modification, also
applies to Equation 9.3.

k
lxii : 1fd,ist(xi,")' Itla*t@6 (e.4)
f s:I "r)'

Intuitively, the weight u,;i, which indicates the degree of membership of

point x; in cluster C7, should be relatively high if x6 is close to centroid c7 (if
di,st(xi,c7) is low) and relatively low if xa is far from centroid ci (if dist(x6,ci)
is high). If uii : lf dist(xi,c7)2, which is the numerator of Equation 9.4, then
this will indeed be the case. However, the membership weights for a point
will not sum to one unless they are normalized; i.e., divided by the sum of all
the weights as in Equation 9.4. To summarize, the membership weight of a
point in a cluster is just the reciprocal of the square of the distance between
the point and the cluster centroid divided by the sum of all the membership
weights of the point.
Now consider the impact of the exponent tl(p - 1) in Equation 9.3. If
p > 2, then this exponent decreasesthe weight assigned to clusters that are
close to the point. Indeed, as p goes to infinity, the exponent tends to 0 and
weights tend to the value 1/k. On the other hand, as P approaches 1, the
exponent increasesthe membership weights of points to which the cluster is
close. As p goes to 1, the membership weight goes to 1 for the closest cluster
and to 0 for all the other clusters. This correspondsto K-means.
582 Chapter I Cluster Analysis: Additional Issuesand Algorithms

A^'
A ^'
^' A
AA . ^.
^'
^.
A Ar a
^'
AA
A A
A A
@
tr@r
o o@ o
T

OO ao rEo @ l1 rl
a o tr ll ll
@ t
ooto o @ I I I
I
oa o @ T I
I
I
o @
I I
ta' E I
o O@ 1l
T
o I
I

#ffiT,:fl],
Figure9.1. Fuzzyc-means
clustering pointset.
ofa two-dimensional

Example 9.L (Ftzzy c-means on Three Circular Clusters). Figure 9.1

shows the result of applying fuzzy c-rneans to find three clusters for a two-
dimensional data set of 100 points. Each point was assignedto the cluster in
which it had the largest membership weight. The points belonging to each
cluster are shown by different marker shapes,while the degreeof membership
in the cluster is shown by the shading. The darker the points, the stronger their
membership in the cluster to which they have been assigned. The membership
in a cluster is strongest toward the center of the cluster and weakest for those
points that are between clusters.

Strengths and Limitations

A positive feature of FCM is that it produces a clustering that provides an

indication of the degreeto which any point belongs to any cluster. Otherwise,
it has much the same strengths and weaknessesas K-means, although it is
somewhat more computationally intensive.
 9.2 Prototype-BasedClustering 583

9.2.2 Clustering Using Mixture Models

This section considers clustering based on statistical models. It is often con-

venient and effective to assume that data has been generated as a result of
a statistical process and to describe the data by finding the statistical model
that best fits the data, where the statistical model is described in terms of a
distribution and a set of parameters for that distribution. At a high level, this
process involves deciding on a statistical model for the data and estimating
the parameters of that model from the data. This section describes a par-
ticular kind of statistical model, mixture models, which model the data by
using a number of statistical distributions. Each distribution corresponds to
a cluster and the parameters of each distribution provide a description of the
corresponding cluster, typically in terms of its center and spread.
The discussion in this section proceeds as follows. After providing a de-
scription of mixture models, we consider how parameters can be estimated for
statistical data models. We first describe how a procedure known as maxi-
mum likelihood estimation (MLE) can be used to estimate parameters
for simple statistical models and then discusshow we can extend this approach
for estimating the parameters of mixture models. Specifically,we describe the
well-known Expectation-Maximization (EM) algorithm, which makes an
initial guessfor the parameters, and then iteratively improves these estimates.
We present examples of how the EM algorithm can be used to cluster data by
estimating the parameters of a mixture model and discuss its strengths and
limitations.
A firm understanding of statistics and probability, as covered in Appendix
C, is essential for understanding this section. Also, for conveniencein the
following discussion,we use the term probability to refer to both probability
and probability density.

Mixture Models

Mixture models view the data as a set of observationsfrom a mixture of differ-

ent probability distributions. The probability distributions can be anything,
but are often taken to be multivariate normal, since this type of distribution
is well understood, mathematically easy to work with, and has been shown
to produce good results in many instances. These types of distributions can
model ellipsoidal clusters.
Conceptually, mixture models correspond to the following processof gen-
erating data. Given several distributions, usually of the same type, but with
different parameters, randomly select one of these distributions and generate
584 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

an object from it. Repeat the process m times, where rn is the number of
objects.
More formally, assumethat there are K distributions and m objects, ,{ :
{*t, . . . ,x*}. Let the 7rh distribution have parameters 0i, and let O be the set
of all parameters,i.e., @ : {0r,...,0x}. Then, prob(x,101)is the probability
of the ith object if it comes from the 7'h distribution. The probability that
the jth distribution is chosento generate an object is given by the weight u7,
I < j < 1(, where these weights (probabilities) are subject to the constraint
that they sum to one, i.e., Ditul: 1. Then, the probability of an object x
is given by Equation 9.5.

K
prob(xl}): t wpi@lli) (e.5)
j:t

If the objects are generatedin an independent manner) then the probability

of the entire set of objects is just the product of the probabilities of each
individual x;.

l@): fin,ou(4l@)- iIi

prob(x wipi @,10
1) ( e.6)
i:1 i:l i:7

For mixture models, each distribution describes a different group, i.e., a

different cluster. By using statistical methods, we can estimate the parame-
ters of these distributions from the data and thus describe these distributions
(clusters). We can also identify which objects belong to which clusters. How-
ever, mixture modeling does not produce a crisp assignment of objects to
clusters, but rather gives the probability with which a specific object belongs
to a particular cluster.
Example 9.2 (Univariate Gaussian Mixture). We provide a concrete
illustration of a mixture model in terms of Gaussian distributions. The prob-
ability density function for a one-dimensionalGaussian distribution at a point
ris
prob(r,ll):+ ,-\# (e.7)
v zlTo
The parameters of the Gaussian distribution are given by 0 : (p,o), where
p is the mean of the distribution and o is the standard deviation. Assume
that there are two Gaussian distributions, with a common standard deviation
of 2 and means of -4 and 4, respectively. Also assumethat each of the two
 9.2 Prototvpe-BasedClusterine 585

o
n

(a) Probability density function for (b) 20,000 points generated from the
the mixture model. mixture model.

Figure9.2. Mixture oftwonormal

modelconsisting distributions of -4and4, respectively.
withmeans
Bothdistributions
havea standard of 2.
deviation

distribrrtions is selected with equal probability, i.e., wy : w2 : 0.5. Then

Equation 9.5 becomesthe following:

1 - trt4t2 1 _
' :) - c
Drob(rltO 8 + _e . (e.8)
2\/2iT 2t/2r
Figure 9.2(a) shows a plot of the probability density function of this mix-
ture m,rdel, while Figure 9.2(b) shows the histogram for 20,000 points gener-
ated from this mixture model.

Estimiating Model Parameters Using Maximum Likelihood

Given ix statistical model for the data, it is necessaryto estimate the param-
eters of that model. A standard approach used for this task is maximum
likelihc'od estimation, which we now explain.
To begin, consider a set of rn points that are generated from a one-
dimensional Gaussian distribution. Assuming that the points are generated
independently, the probability of these points is just the product of their in-
dividual probabilities. (Again, we are dealing with probability densities, but
to keep our terminology simple, we will refer to probabilities.) Using Equa-
tion 9.7, we can write this probability as shown in Equation 9.9. Since this
probability would be a very small number, we typically will work with the log
probability, as shown in Equation 9.10.
586 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

!- | t,i-ut2
p ro b (X l @) II
: ,; e- - ";z- (e.e)
. a v a^u
a:l

!\ (r - ut2 -
Ios prob(XlO) : - Lt;+ - mlogo
0.5m1og2tr ( e.10)
i=I

We would like to find a procedure to estimate z and o if they are unknown.

One approach is to choosethe values of the parameters for which the data is
most probable (most likely). In other words, choosethe p and o that maximize
Equation 9.9. This approach is known in statistics as the maximum like-
lihood principle, and the processof applying this principle to estimate the
parameters of a statistical distribution from the data is known as maximum
likelihood estimation (MtE).
The principle is called the maximum likelihood principle because,given a
set of data, the probability of the data, regarded as a function of the parame-
ters, is called a likelihood function. To illustrate, we rewrite Equation 9.9
as Equation 9.11 to emphasizethat we view the statistical parameters p, and
o as our variables and that the data is regarded as a constant. For practical
reasons,the log likelihood is more commonly used. The log likelihood func-
tion derived from the log probability of Equation 9.10 is shown in Equation
9.72. Note that the parameter values that maximize the log likelihood also
maximize the likelihood since log is a monotonically increasing function.
m1.

: L(@lx): TI +
ti,ketihood,(@lx) (e.11)
LL ^ /r*^
. 1 v a t t v " "Ett

: l(OlX): - t
losti,kelihood(Olx) - 0.bmlog2tr
- mlogo (9.12)
z--I
ry

Example 9.3 (Maximum Likelihood Parameter Estimation). We pro-

vide a concrete illustration of the use of MLE for finding parameter values.
Supposethat we have the set of 200 points whose histogram is shown in Figure
9.3(a). Figure 9.3(b) showsthe maximum log likelihood plot for the 200 points
under consideration. The values of the parameters for which the log probabil-
ity is a maximum are LL- -4.1 and o:2.1, which are closeto the parameter
values of the underlying Gaussian distribution, LL: -4.0 and o : 2.0. r
 9.2 Prototype-BasedClustering 587

40

40

470

E 40
z
J90

-s0

-510

15 IOg proD

(a) Ilistogram of 200 points from a (b) Log likelihood plot of the 200 points for
Gaussian distribution. different values of the mean and standard
deviation.

9.3. 200points
Figure froma Gaussian andtheirlogprobability
distribution parameter
fordifferent
values.

Graphing the likelihood of the data for different values of the parameters is
not practical, at least if there are more than two parameters. Thus, standard
statisrbicalprocedure is to derive the maximum likelihood estimates of a statis-
tical parameter by taking the derivative of likelihood function with respect to
that parameter, setting the result equal to 0, and solving. In particular, for a
Gauss;iandistribution, it can be shown that the mean and standard deviation
of the sample points are the maximum likelihood estimates of the correspond-
ing peurameters of the underlying distribution. (SeeExercise 9 on 648.) Indeed,
for the 200 points consideredin our example, the parameter values that max-
imized the log likelihood were precisely the mean and standard deviation of
t h e 2 0 0 p o i n t s ,i . e . ,u : - 4 . 1 a n d o : 2 . 1 .

Estinnating Mixture Model Parameters Using Maximum Likelihood:

The .EM Algorithm

We cia,nalso use the maximum likelihood approach to estimate the model

parameters for a mixture model. In the simplest case, we know which data
objecl;s come from which distributions, and the situation reduces to one of
estimratingthe parameters of a single distribution given data from that distri-
bution. For most common distributions, the maximum likelihood estimates of
the parameters are calculated from simple formulas involving the data.
588 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

In a more general (and more realistic) situation, we do not know which

points were generated by which distribution. Thus, we cannot directly cal-
culate the probability of each data point, and hence, it would seem that we
cannot use the maximum likelihood principle to estimate parameters. The
solution to this problem is the EM algorithm, which is shown in Algorithm
9.2. Briefly, given a guess for the parameter values, the EM algorithm cal-
culates the probability that each point belongs to each distribution and then
usesthese probabilities to compute a new estimate for the parameters. (These
parameters are the ones that maximize the likelihood.) This iteration con-
tinues until the estimates of the parameters either do not change or change
very little. Thus, we stil employ maximum likelihood estimation, but via an
iterative search.

Algorithm 9.2 EM algorithm.

Select an initial set of model parameters.
(As with K-means, this can be done randomly or in a variety of ways.)
z : repeat
J: Expectation Step For each object, calculate the probability
that each object belongs to each distribution, i.e., calculate
pr ob(distribution, lxi, O).
4: Maximization Step Given the probabilities from the expectation step,
find the new estimates of the parameters that maximize the expected
likelihood.
5: until The parameters do not change.
(Alternatively, stop if the change in the parameters is below a specified
threshold.)

The trM algorithm is similar to the K-means algorithm given in Section

8.2.1. Indeed, the K-means algorithm for Euclidean data is a special caseof the
EM algorithm for spherical Gaussian distributions with equal covariance ma-
trices, but different means. The expectation step correspondsto the K-means
step of assigning each object to a cluster. Instead, each object is assigned
to every cluster (distribution) with some probability. The maximization step
correspondsto computing the cluster centroids. Instead, all the parameters of
the distributions, as well as the weight parameters, are selectedto maximize
the likelihood. This process is often straightforward, as the parameters are
typically computed using formulas derived from maximum likelihood estima-
tion. For instance, for a single Gaussian distribution, the MLE estimate of the
 9.2 Prototype-BasedClustering 589

mean is the mean of the objects in the distribution. In the context of mixture
models and the trM algorithm, the computation of the mean is modified to
account for the fact that every object belongs to a distribution with a certain
probability. This is illustrated further in the following example.

Example 9.a (Simple Example of EM Algorithm). This example illus-

trates how EM operates when applied to the data in Figure 9.2. To keep the
example as simple as possible) we assume that we know that the standard
deviation of both distributions is 2.0 and that points were generated with
equal probability from both distributions. We will refer to the left and right
distributions as distributions 1 and 2, respectively.
We begin the EM algorithm by making initial guessesfor p,1 and p,2)say)
Ft : -2 and p,2 : 3. Thus, the initial parameters,0 : (Lt,o), for the
two distributions are?respectively,01 : (-2,2) and 02: (3,2). The set of
parametersfor the entire mixture model is O: {0t,02}.For the expectation
step of EM, we want to compute the probability that a point came from
a particular distribution; i.e., we want to compute prob(di,stri,bution 1lr;, O)
and prob(di,stri,bution 2lri,O). These values can be expressedby the following
equation, which is a straightforward application of Bayes rule (see Appendix
c),
0.5prob(r.il?i)
pr ob(distribution j l, i, 0) : ( e.13)
0.5prob(ril?t)+ 0.5prob(ril?2)'
where 0.5 is the probability (weight) of each distribution and j is 7 or 2.
For instance, assume one of the points is 0. Using the Gaussian den-
sity function given in Equation 9.7, we compute that prob(010t) : 0.12 and
prob(0102): 0.06. (Again, we are really computing probability densities.) Us-
ing these values and Equation 9.13, we find that prob(di,stribution 110,O) :
0.12I Q.12+ 0.06) : 0.66 and pr ob(distri,buti,on210,O) : 0.06/(0.12+ 0.06) :
0.33. This means that the point 0 is twice as likely to belong to distribution 1
as distribution 2 based on the current assumptions for the parameter values.
After computing the cluster membership probabilities for all 20,000points,
we compute new estimates for pr,1and pt2 (using Equations 9.14 and 9.15) in
the maximization step of the EM algorithm. Notice that the new estimate for
the mean of a distribution is just a weighted averageof the points, where the
weights are the probabilities that the points belong to the distribution, i.e.,
the pr ob(di,stri,buti,on7 |16) values.
20,000
\- pr obldi,stri
bution 1|r;, O)
14: ) . r' (e.14)
i:7 D}foo p, ob(d,istri bution Llr ;, Q)
590 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

Table9.1. Firstfewiterations
oftheEMalgorithm
forthesimple
example.
Iteration l-rt Hz
0 -2.00 3.00
1 , aA
4.r0
-3.94 4.07
.l -3.97 4.04
4 -3.98 4.03
-3.98 4.03

20,000
U2: \-
L
( e.15)

We repeat these two steps until the estimates of p1 and p,2 either don't
change or change very little. Table 9.1 gives the first few iterations of the EM
algorithm when it is applied to the set of 20,000 points. For this data, we
know which distribution generated which point, so we can also compute the
mean of the points from each distribution. The means are p1 : -3.98 and
ltz: 4.03. I
Example 9.5 (The EM Algorithm on Sample Data Sets). We give
three examples that illustrate the use of the EM algorithm to find clusters
using mixture models. The first example is based on the data set used to
illustrate the finzy c-means algorithm-see Figure 9.1. We modeled this data
as a mixture of three two-dimensional Gaussian distributions with different
means and identical covariance matrices. We then clustered the data using
the EM algorithm. The results are shown in Figure 9.4. Each point was
assignedto the cluster in which it had the largest membership weight. The
points belonging to each cluster are shown by different marker shapes, while
the degreeof membership in the cluster is shown by the shading. Membership
in a cluster is relatively weak for those points that are on the border of the two
clusters, but strong elsewhere. It is interesting to compare the membership
weights and probabilities of Figures 9.4 and 9.1. (See Exercise 11 on page
648.)
For our second example, we apply mixture model clustering to data that
contains clusters with different densities. The data consists of two natural
clusters, each with roughly 500 points. This data was created by combining
two sets of Gaussiandata, one with a center at (-4,1) and a standard deviation
of 2, and one with a center at (0,0) and a standard deviation of 0.5. Figure 9.5
shows the clustering produced by the EM algorithm. Despite the differences
 9.2 Prototvpe-Based Clusterins 591

in the density, the EM algorithm is quite successfulat identifying the original

clusters.
For our third example, we use mixture model clustering on a data set that
K-means cannot properly handle. Figure 9.6(a) showsthe clustering produced
by a mixture model algorithm, while Figure 9.6(b) shows the K-means cluster-
ing of the same set of 1000 points. For mixture model clustering, each point
has been assignedto the cluster for which it has the highest probability. In
both figures, different markers are used to distinguish different clusters. Do
not confuse the 'f' and 'x' markers in Figure 9.6(a).

Advantages and Limitations of Mixture Model Clustering Using the

EM Algorithm

Finding clusters by modeling the data using mixture models and applying the
EM algorithm to estimate the parameters of those models has a variety of
advantages and disadvantages. On the negative side, the trM algorithm can
be slow, it is not practical for models with large numbers of components, and
it does not work well when clusters contain only a few data points or if the
data points are nearly co-linear. There is also a problem in estimating the
number of clusters or7 more generally, in choosingthe exact form of the model
to use. This problem typically has been dealt with by applying a Bayesian
approach, which, roughly speaking, givesthe odds of one model versusanother,
based on an estimate derived from the data. Mixture models may also have
difficulty with noise and outliers, although work has been done to deal with
this problem.
On the positive side, mixture models are more general than K-means or
finzy c-meansbecausethey can use distributions of various types. As a result,
mixture models (basedon Gaussiandistributions) can find clusters of different
sizesand elliptical shapes. AIso, a model-basedapproach provides a disciplined
way of eliminating some of the complexity associatedwith data. To see the
patterns in data, it is often necessaryto simplify the data, and fitting the data
to a model is a good way to do that if the model is a good match for the data.
Furthermore, it is easy to characterize the clusters produced, since they can
be described by a small number of parameters. Finally, many sets of data are
indeed the result of random processes)and thus should satisfy the statistical
assumptions of these models.
592 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

o
ao
a O o
a
a oo o a I
oo
a
a 1O a 85
OO a
oa
oo
8
a o
a

A 7
^. AA
r .4. ^. A .65
tl A
atta
.6
ll A
I
lr ^' A
0.55
lr ^. A
lrr 1A

I
^. 0.5
Maximum
Probablility

Figure9.4. EMclustering pointsetwiththreeclusters.

of a two-dimensional

!-o ..

Figure9.5. EMclustering pointsetwithtwoclusters

ofa two-dimensional of differing
density.
 9,2 Prototype-Based Clustering

X<
{x
x (,\
x
*
XX *
rx +*
x* :l<*
x*
++ x?(x *
.+ :X a<*
*
fl
T *
+' + ' t *,r
+]-
+ T +
+TJ
+ -+=
j;tr+if;

''+

(a) Clusters produced by mixture model clustering.

;r

*
,( h*
)e

*x *
,x ,t*
* :x
)(
* **

(b) Clusters produced by K-means clustering.

modelandK-means
Figure9.6. Mixture clustering points.
of a setoftwo-dimensional
594 Chapter I Cluster Analysis: Additional Issuesand Algorithms

9.2.3 Self-Organizing Maps (SOM)

The Kohonen Self-Organizing Feature Map (SOFM or SOM) is a clustering

and data visualization technique basedon a neural network viewpoint. Despite
the neural network origins of SOM, it is more easily presented-at least in the
context of this chapter-as a variation of prototype-based clustering. As with
other types of centroid-based clustering, the goal of SOM is to find a set of
centroids (reference vectors in SOM terminology) and to assign each object
in the data set to the centroid that provides the best approximation of that
object. In neural network terminology, there is one neuron associated with
each centroid.
As with incremental K-means, data objects are processedone at a time and
the closest centroid is updated. Unlike K-means, SOM imposes a topographic
ordering on the centroids and nearby centroids are also updated. F\rrthermore,
SOM does not keep track of the current cluster membership of an object, and,
unlike K-means, if an object switches clusters, there is no explicit update of the
old cluster centroid. Of course,the old cluster may well be in the neighborhood
of the new cluster and thus may be updated for that reason. The processing
of points continues until some predetermined limit is reached or the centroids
are not changing very much. The final output of the SOM technique is a set
of centroids that implicitly define clusters. Each cluster consists of the points
closest to a particular centroid. The following section explores the details of
this process.

The SOM Algorithm

A distinguishing feature of SOM is that it imposes a topographic (spatial)

organization on the centroids (neurons). Figure 9.7 shows an example of a
two-dimensional SOM in which the centroids are represented by nodes that
are organized in a rectangular lattice. Each centroid is assigneda pair of coor-
dinates (2,j). Sometimes,such a network is drawn with links between adjacent
nodes, but that can be misleading becausethe influence of one centroid on an-
other is via a neighborhood that is defined in terms of coordinates, not links.
There are many types of SOM neural networks, but we restrict our discussion
to two-dimensional SOMs with a rectangular or hexagonal organization of the
centroids.
Even though SOM is similar to K-means or other prototype-based ap-
proaches, there is a fundamental difference. Centroids used in SOM have a
predetermined topographic ordering relationship. During the training process,
SOM useseach data point to update the closestcentroid and centroids that are
 9.2 Prototype-BasedClustering 595

@@@
@@@
r.;)
\r\rvr.;) G)
SOMneural
rectangular
3-by-3
Figure9,7.Two-dimensional network.

nearby in the topographic ordering. In this way, SOM produces an ordered set
of centroids for any given data set. In other words, the centroids that are close
to each other in the SOM grid are more closely related to each other than to
the centroids that are farther away. Becauseof this constraint, the centroids of
a two-dimensional SOM can be viewed as lying on a two-dimensional surface
that tries to fit the n-dimensional data as well as possible. The SOM centroids
can also be thought of as the result of a nonlinear regressionwith respect to
the data points.
At a high level, clustering using the SOM technique consists of the steps
described in Algorithm 9.3.

Algorithm 9.3 Basic SOM Algorithm.

1: Initialize the centroids.
2: repeat
3: Select the next object.
4: Determine the closest centroid to the object.
5: Update this centroid and the centroids that are close, i.e., in a specified neigh-
borhood.
6: until The centroids don't change much or a threshold is exceeded.
7: Assign each object to its closest centroid and return the centroids and clusters'

Initialization This step (line 1) can be performed in a number of ways.

One approach is to choose each component of a centroid randomly from the
range of values observed in the data for that component. While this approach
works, it is not necessarilythe best apploach, especially for producing rapid
convergence. Another approach is to randomly choose the initial centroids
596 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

from the available data points. This is very much like randomly selecting
centroids for K-means.

Selection of an object The first step in the loop (line 3) is the selectionof
the next object. This is fairly straightforward, but there are some difficulties.
Since convergencemay require many steps, each data object may be used
multiple times, especially if the number of objects is small. However, if the
number of objects is large, then not every object needs to be used. It is also
possible to enhance the influence of certain groups of objects by increasing
their frequency in the training set.

Assignment The determination of the closestcentroid (line 4) is also straight-

forward, although it requires the specification of a distance metric. The Eu-
clidean distance metric is often used, as is the dot product metric. When using
the dot product distance, the data vectors are typically normalized beforehand
and the referencevectors are normalized at each step. In such cases,using the
dot product metric is equivalent to using the cosine measure.

Update The update step (line 5) is the most complicated. Let m1, ...,
rn7, be the centroids. (For a rectangular grid, note that k is the product of
the number of rows and the number of columns.) For time step f, let p(t)
be the current object (point) and assume that the closest centroid to p(l) is
m;. Then, for time t+I, the jth centroid is updated by using the following
equation. (We will see shortly that the update is really restricted to centroids
whose neurons are in a small neighborhood of m;.)

rni(t t 1) : + hj(t)(p(t) - ( e.16)

"'i(r) "'i(t))

Thus, at time f, a centroid mi(t) is updated by adding a term, hj(t) (p(t) -

mj(t)), which is proportional to the difference, p(t) - -i(t), between the
centroid, mi(t), and the current object, p(t). hj(t) determines the effect that
the difference, p(l) -^i(t), will have and is chosen so that (t) it diminishes
with time and (2) it enforcesa neighborhood effect, i.e., the effect of an object
is strongest on the centroids closest to the centroid m7. Here we are referring
to the distance in the grid, not the distance in the data space. Typically, h7(l)
is chosento be one of the followins two functions:
 9.2 Prototype-BasedClustering 597

hi(t) : a(t)erp(-di,st(ri,rp72 1 1Zo2(t)) (Gaussianfunction)

hi?) : a(t) if di,st(ri,ri < threshold,0 otherwise (step function)

These functions require more explanation. a(t) is a learning rate param-

eter, 0 < a(t) < 1, which decreasesmonotonically with time and controls
the rate of convergence.r/, : (rk,gp) is the two-dimensional point that gives
the grid coordinatesof the k'h centroid. dist(ri,16) is the Euclidean distance
betweenthegridlocationofthetwocentroids,i.e.,'](r1-r4f+(gi-ar),.
Consequently,for centroids whose grid locations are far from the grid location
of centroid m;, the influence of object p(t) wilt be either greatly diminished or
non-existent. Finally. note that o is the typical Gaussian variance parameter
and controls the width of the neighborhood, i.e., a small o will yield a small
neighborhood, while a large o will yield a wide neighborhood. The threshold
used for the step function also controls the neighborhood size.
Remember, it is the neighborhood updating technique that enforcesa Ie-
Iationship (ordering) between centroids associatedwith neighboring neurons.

Termination Deciding when we are closeenough to a stable set of centroids

is an important issue. Ideally, iteration should continue until convergence
occurs, that is, until the referencevectors either do not change or change very
little. The rate of convergence will depend on a number of factors, such as
the data and a(f). We will not discussthese issuesfurther, except to mention
that, in general, convergencecan be slow and is not guaranteed.
Example 9.6 (Document Data). We present two examples. In the first
case, we apply SOM with a 4-by-4 hexagonal grid to document data. We
clustered 3204 newspaper articles from the Los Angeles T'imes, which come
from 6 different sections: Entertainment, Financial, Foreign, Metro, National,
and sports. Figure 9.8 shows the SoM grid. we have used a hexagonal
grid, which allows each centroid to have six immediate neighbors instead of
four. Each SOM grid cell (cluster) has been labeled with the majority class
label of the associatedpoints. The clusters of each particular category form
contiguous groups, and their position relative to other categories of clusters
gives us additional information, e.g., that the Metro section contains stories
related to all other sections. I

Example 9.7 (Two-Dimensional Points). In the second case) we use a

rectangular SOM and a set of two-dimensional data points. Figure 9.9(a)
598 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

Figure9.8. Visualization
of therelationships
between lor LosAngeles
SOMcluster Times
document
dataset.

shows the points and the positions of the 36 referencevectors (shown as x's)
produced by SOM. The points are arranged in a checkerboard pattern and
are split into five classes:circles, triangles, squares,diamonds, and hexagons
(stars). A 6-by-6 two-dimensional rectangular grid of centroids was used with
random initialization. As Figure 9.9(a) shows, the centroids tend to distribute
themselves to the dense areas. Figure 9.9(b) indicates the majority class of
the points associatedwith that centroid. The clusters associatedwith triangle
points are in one contiguous area, as are the centroids associated with the
four other types of points. This is a result of the neighborhood constraints
enforced by SOM. While there are the same number of points in each of the
five groups, notice also that the centroids are not evenly distributed. This is
partly due to the overall distribution of points and partly an artifact of putting
each centroid in a sinsle cluster.

Applications

Once the SOM vectors are found, they can be used for many purposes other
than clustering. For example, with a two-dimensional SOM, it is possible to
associatevarious quantities with the grid points associatedwith each centroid
(cluster) and to visualize the results via various types of plots. For example,
plotting the number of points associatedwith each cluster yields a plot that
revealsthe distribution of points among clusters. A two-dimensional SOM is a
non-linear projection of the original probability distribution function into two
 9.2 Prototype-BasedClustering599

o e o
osn { ^ Xd o
dX'o oo o E^o q X- diamond diamond diamond h9Xagon nexagon nexagon
oo o gl i " J"
;;; "x*""?
oxe.i"
* S""\
" .*r*
'oX ** diamond dlamond diamond ctrde hexagon hengon
- o o "
"^ ox
"o o "8>E
?X o-xo o
^s diamond dimond ctrcle nexagon
^&
" :,^ ^ oxo
o
9{"- .o
o o^' o > q t^-g( o
oo
" .'x
- -o@ square cfcb triangle tiangle
"l -4
rn%a^aA
8k"."
'",x8 k '^Aa a da ^ ")\ a
^X X quare tiangle hangle
."XF"
"d r^ o A A A
X .Fo
" EooX D ^Xa a
oo "A
E Bo
quare quare tiangl€ tdan9le hangb

(a) Distribution of SOM reference vectors (b) Classes of the SOM cen-
(X's) for a two-dimensional point set. broids.

datapoints.
to two-dimensional
Figure9.9. SOMapplied

dimensions. This projection attempts to preserve topological features; thus,

using SOM to capture the structure of the data has been compared to the
processof ttpressinga flower.tt

Strengths and Limitations

SOM is a clustering technique that enforcesneighborhood relationships on the

resulting cluster centroids. Because of this, clusters that are neighbors are
more related to one another than clusters that are not. Such relationships
facilitate the interpretation and visualization of the clustering results. Indeed,
this aspect of SOM has been exploited in many areas, such as visualizing Web
documents or gene array data.
SOM also has a number of limitations, which are listed next. Some of
the listed limitations are only valid if we consider SOM to be a standard
clustering technique that aims to find the true clusters in the data, rather
than a technique that uses clustering to help discover the structure of the
data. Also, some of these limitations have been addressedeither by extensions
of SOM or by clustering algorithms inspired by SOM. (See the bibliographic
notes.)
o The user must choosethe settings of parameters, the neighborhood func-
tion, the grid type, and the number of centroids.
600 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

A SOM cluster often does not correspond to a single natural cluster.

In some cases,a SOM cluster may encompassseveral natural clusters,
while in other cases a single natural cluster is split into several SOM
clusters. This problem is partly due to the use of a grid of centroids and
partly due to the fact that SOM, like other prototype-based clustering
techniques, tends to split or combine natural clusters when they are of
varying sizes,shapes,and densities.

SOM lacks a specific objective function. SOM attempts to find a set

of centroids that best approximate the data, subject to the topographic
constraints among the centroids, but the successof SOM in doing this
cannot be expressedby a function. This can make it difficult to compare
different SOM clustering results.

o SOM is not guaranteed to converge, although, in practice, it typically

does.

9.3 Density-Based Clustering

In Section 8.4, we considered DBSCAN, a simple, but effective algorithm for
finding density-basedclusters, i.e., denseregions ofobjects that are surrounded
by low-density regions. This section examines additional density-based clus-
tering techniquesthat addressissuesof efficiency,finding clusters in subspaces,
and more accurately modeling density. First, we consider grid-based cluster-
ing, which breaks the data space into grid cells and then forms clusters from
cells that are sufficiently dense. Such an approach can be efficient and effec-
tive, at least for low-dimensional data. Next, we consider subspaceclustering,
which looks for clusters (denseregions) in subsetsof all dimensions. For a data
spacewith n dimensions,potentially 2" - 1 subspacesneed to be searched,and
thus an efficient technique is needed to do this. CLIQUE is a grid-based clus-
tering algorithm that provides an efficient approach to subspace clustering
based on the observation that dense areas in a high-dimensional space imply
the existence of dense areas in lower-dimensional space. Finally, we describe
DENCLUE, a clustering technique that useskernel density functions to model
density as the sum of the influences of individual data objects. While DEN-
CLUE is not fundamentally a grid-basedtechnique, it does employ a grid-based
approach to improve efficiency.
 9.3 Density-Based Clustering 601

9.3.1 Grid-Based Clustering

A grid is an efficient way to organize a set of data, at least in low dimensions.
The idea is to split the possible values of each attribute into a number of
contiguous intervals, creating a set of grid cells. (We are assuming, for this
discussion and the remainder of the section, that our attributes are ordinal,
interval, or continuous.) Each object falls into a grid cell whose corresponding
attribute intervals contain the values ofthe object. Objects can be assignedto
grid cells in one pass through the data, and information about each cell, such
as the number of points in the cell, can also be gathered at the same time.
There are a number of ways to perform clustering using a grid, but most
approachesare based on density, at least in part, and thus, in this section, we
will use grid-based clustering to mean density-based clustering using a grid.
Algorithm 9.4 describes a basic approach to grid-based clustering. Various
aspects of this approach are explored next.

Algorithm 9.4 Basic grid-based clustering algorithm.

1: Defi.nea set of grid cells.
2: Assign objects to the appropriate cells and compute the density of each cell.
3: Eliminate cells having a density below a specified threshold, r.
4: Form clusters from contiguous (adjacent) groups of dense cells.

Defining Grid Cells

This is a key step in the process, but also the least well defined, as there
are many ways to split the possible values of each attribute into a number
of contiguous intervals. For continuous attributes, one common approach is
to split the values into equal width intervals. If this approach is applied to
each attribute, then the resulting grid cells all have the same volume, and the
density of a cell is conveniently defined as the number of points in the cell.
However, more sophisticated approaches can also be used. In particular,
for continuous attributes any of the techniques that are commonly used to
discretize attributes can be applied. (See Section 2.3.6.) In addition to the
equal width approach already mentioned, this includes (1) breaking the values
of an attribute into intervals so that each interval contains an equal number
of points, i.e., equal frequency discretization, or (2) using clustering. Another
approach, which is used by the subspaceclustering algorithm MAFIA, initially
602 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

breaks the set of values of an attribute into a large number of equal width
intervals and then combines intervals of similar density.
Regardlessof the approach taken, the definition of the grid has a strong
impact on the clustering results. We will consider specific aspectsof this later.

The Density of Grid Cells

A natural way to define the density of a grid cell (or a more generally shaped
region) is as the number of points divided by the volume of the region. In other
words, density is the number of points per amount of space,regardlessof the
dimensionality of that space. Specific, low-dimensional examples of density
are the number of road signs per mile (one dimension), the number of eagles
per square kilometer of habitat (two dimensions), and the number of molecules
of a gas per cubic centimeter (three dimensions). Ar mentioned, however, a
common approach is to use grid cells that have the same volume so that the
number of points per cell is a direct measure of the cell's density.
Example 9.8 (Grid-Based Density). Figure 9.10 shows two sets of two-
dimensional points divided into 49 cells using a 7-by-7 grid. The first set
contains 200 points generatedfrom a uniform distribution over a circle centered
at (2, 3) of radius 2, while the second set has 100 points generated from a
uniform distribution over a circle centered at (6, 3) of radius 1. The counts
for the grid cells are shown in Table 9.2. Since the cells have equal volume
(area), we can consider these values to be the densities of the cells. r

0000000
0000000
417186000
L4L413130L827
1118102L02431
320144000
0000000

Figute9.10.Grid-based
density. Table9.2. Pointcountsfororidcells.
 9.3 Density-Based Clustering 603

Forming Clusters from Dense Grid Cells

Forming clusters from adjacent groups of dense cells is relatively straightfor-

ward. (In Figure 9.10, for example, it is clear that there would be two clusters.)
There are, however, some issues. We need to define what we mean by adjacent
cells. For example, does a two-dimensional grid cell have 4 adjacent cells or
8? Also, we need an efficient technique to find the adjacent cells, particularly
when only occupied cells are stored.
The clustering approach defined by Algorithrn 9.4 has some limitations
that could be addressedby making the algorithm slightly more sophisticated.
For example, there are likely to be partially empty cells on the boundary of
a cluster. Often, these cells are not dense. If so, they will be discarded and
parts of a cluster will be lost. Figure 9.10 and Table 9.2 show that four parts
of the larger cluster would be lost if the density threshold is 9. The clustering
processcould be modified to avoid discarding such cells, although this would
require additional processing.
It is also possibleto enhancebasic grid-basedclustering by using more than
just density information. In many cases,the data has both spatial and non-
spatial attributes. In other words, some of the attributes describe the location
of objects in time or space, while other attributes describe other aspects of
the objects. A common example is houses, which have both a location and
a number of other characteristics, such as price or floor space in square feet.
Because of spatial (or temporal) autocorrelation, objects in a particular cell
often have similar values for their other attributes. In such cases,it is possible
to filter the cells based on the statistical properties of one or more non-spatial
attributes, e.9., average house price, and then form clusters based on the
density of the remaining points.

Strengths and Limitations

On the positive side, grid-based clustering can be very efficient and effective.
Given a partitioning of each attribute, a single pass through the data can
determine the grid cell of every object and the count of every grid. Also,
even though the number of potential grid cells can be high, grid cells need to
be created only for non-empty cells. Thus, the time and space complexity of
defining the grid, assigning each object to a cell, and computing the density
of each cell is only O(m), where m is the number of points. If adjacent,
occupied cells can be efficiently accessed,for example, by using a search tree,
then the entire clustering process will be highly efficient, e.g., with a time
complexity of O(m log rz). For this reason,the grid-based approach to density
604 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

clustering forms the basis of a number of clustering algorithms, such as STIlttrG,

GRIDCLUS, WaveCluster, Bang-Clustering, CLIQUE, and MAFIA.
On the negative side, grid-based clustering, like most density-based clus-
tering schemes,is very dependent on the choice of the density threshold z'
If r is too high, then clusters will be lost. If r is too low, two clusters that
should be separate may be joined. Furthermore, if there are clusters and noise
of differing densities, then it may not be possible to fi.nd a single value of r
that works for all parts of the data space.
There are also a number of issuesrelated to the grid-based approach. In
Figure 9.10, for example, the rectangular grid cells do not accurately capture
the density of the circular boundary areas. We could attempt to alleviate
this problem by making the grid finer, but the number of points in the grid
cells associatedwith a cluster would likely show more fluctuation since points
in the cluster are not evenly distributed. Indeed, some grid cells, including
those in the interior of the cluster, might even be empty. Another issue is
that, depending on the placement or size of the cells, a group of points may
appear in just one cell or be split between several different cells. The same
group of points might be part of a cluster in the first case, but be discarded
in the second. Finally, as dimensionality increases,the number of potential
grid cells increasesrapidly-exponentially in the number of dimensions. Even
though it is not necessaryto explicitly consider empty grid cells, it can easily
happen that most grid cells contain a single object. In other words, grid-based
clustering tends to work poorly for high-dimensional data.

9.3.2 Subspace Clustering

The clustering techniques considered until now found clusters by using all
of the attributes. However, if only subsets of the features are considered,i.e.,
subspacesofthe data, then the clusters that we find can be quite different from
one subspaceto another. There are two reasons that subspaceclusters may
be interesting. First, the data may be clustered with respect to a small set of
attributes, but randomly distributed with respect to the remaining attributes.
Second, there are cases in which different clusters exist in different sets of
dimensions. Consider a data set that records the sales of various items at
various times. (The times are the dimensions and the items are the objects.)
Some items might show similar behavior (cluster together) for particular sets
of months, e.g.) summer, but different clusters would likely be characterized
by different months (dimensions).
 9.3 Density-Based Clustering 605

a
ttt
. ! ,a'
r.tl :
a
a: ,a
t
al

06 08 1 12 14 16 18
x

(a) Four clusters in three dimensions. (b) View in the rg plane

(c) View in the rE plane. (d) View in the rg plane.

Figure9.11. Example
figures
forsubspace
clustering.

Example 9.9 (Subspace Clusters). Figure 9.11(a) showsa set of points in

three-dimensional space. There are three clusters of points in the full space,
which are representedby squares)diamonds, and triangles. In addition, there
is one set of points, represented by circles, that is not a cluster in three-
dimensional space. Each dimension (attribute) of the example data set is split
into a fixed number (rl) of equal width intervals. There are n - 20 intervals,
each of size 0.1. This partitions the data space into rectangular cells of equal
volume, and thus, the density of each unit is the fraction of points it contains.
Clusters are contiguous groups of dense cells. To illustrate, if the threshold
606 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

0.14

o.12

U'
c
o
(L

o
(U
LL
0.04

0.02

00.5 1*t.u22.5

Figure9.12,Histogram
showing of points
thedistribution forther attribute.

for a dense cell is ( : 0.06, or 6%oof the points, then three one-dimensional
clusters can be identified in Figure 9.12, which shows a histogram of the data
points of Figure 9.11(a) for the r attribute.
Figure 9.11(b) shows the points plotted in the ry plane. (The z attribute
is ignored.) This figure also contains histograms along the r and g axes that
show the distribution of the points with respect to their r and g coordinates,
respectively. (A higher bar indicates that the corresponding interval contains
relatively more points, and vice versa.) When we consider the gr axis, we see
three clusters. One is from the circle points that do not form a cluster in the
full space,one consists of the square points, and one consists of the diamond
and triangle points. There are also three clusters in the r dimension; they
correspond to the three clusters diamonds, triangles, and squares in the
full space. These points also form distinct clusters in the ry plane. Figure
9.11(c) shows the points plotted in the rz plane. There are two clusters,
if we consider only the z attribute. One cluster corresponds to the points
representedby circles, while the other consists of the diamond, triangle, and
square points. These points also form distinct clusters in the rz plane. In
Figure 9.11(d), there are three clusters when we consider both the y and z
coordinates. One of these clusters consists of the circlesl another consists
 9.3 Density-Based Clustering 60T

of the points marked by squares. The diamonds and triangles form a single
cluster in the yz plane. I
These figures illustrate a couple of important facts. First, a set of points-
the circles may not form a cluster in the entire data space, but may form a
cluster in a subspace.Second,clusters that exist in the full data space (or even
a subspace)show up as clusters in lower-dimensionalspaces.The first fact tells
us that we may need to look in subsetsof dimensionsto find clusters, while the
second fact tells us that many of the clusters we find in subspacesmay only
be "shadows" (projections) of higher-dimensional clusters. The goal is to find
the clusters and the dimensions in which they exist, but we are typically not
as interested in clusters that are projections of higher-dimensional clusters.

CLIQUE

CLIQUtr (Clustering In QUEst) is a grid-based clustering algorithm that me-

thodically finds subspaceclusters. It is impractical to check each subspacefor
clusters since the number of such subspacesis exponential in the number of
dimensions. Instead, CLIQUE relies on the following property;

Monotonicity property of density-based clusters If a set of points forms

a density-basedcluster in k dimensions (attributes), then the sameset of points
is also part of a density-basedcluster in all possiblesubsetsof those dimensions.

Consider a set of adjacent, k-dimensional cells that form a cluster; i.e., there is
a collection of adjacent cells that have a density above the specified threshold
(. A corresponding set of cells in k - 1 dimensions can be found by omitting
one of the k dimensions (attributes). The lower-dimensional cells are still ad-
jacent, and each low-dimensional cell contains all points of the corresponding
high-dimensional cell. It may contain additional points as well. Thus, a low-
dimensional cell has a density greater than or equal to that of its corresponding
high-dimensional cell. Consequently,the low-dimensional cells form a cluster;
i.e., the points form a cluster with the reduced set of attributes.
Algorithm 9.5 gives a simplified version of the steps involved in CLIQUE.
Conceptually, the CLIQUtr algorithm is similar to the Apriori, algorithm for
finding frequent itemsets. See Chapter 6.

Strengths and Limitations of CLIQUE

The most useful feature of CLIQUE is that it provides an efficient technique
for searching subspacesfor clusters. Since this approach is based on the well-
608 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

Algorithm 9.5 CLIQUE.

1: Find all the dense areas in the one-dimensional spaces corresponding to each
attribute. This is the set of dense one-dimensionalcells.
2: k-2
3: repeat
4: Generate all candidate densek-dimensional cells from dense (k - 1)-dimensional
cells.
5: Eliminate cells that have fewer than { points.
6: k-k*l
7: until There are no candidate dense k-dimensional cells.
8: Find clusters by taking the union of all adjacent, high-density cells.
9: Summarize eachcluster using a small set of inequalities that describethe attribute
ranses of the cells in the cluster.

known Apri,ori, principle from association analysis, its properties are well un-
derstood. Another useful feature is CLIQUE's ability to summarize the list of
cells that comprises a cluster with a small set of inequalities.
Many limitations of CLIQUE are identical to the previously discussedlim-
itations of other grid-based density schemes. Other limitations are similar to
those of the Apri,ori. algorithm. Specifically,just as frequent itemsets can share
items, the clusters found by CLIQUE can share objects. Allowing clusters to
overlap can greatly increase the number of clusters and make interpretation
difficult. Another issue is that Apri,orz (and CLIQUE) potentially have expo-
nential time complexity. In particular, CLIQUE will have difficulty if too many
dense cells are generated at lower values of k. Raising the density threshold
( can alleviate this problem. Still another potential limitation of CLIQUE is
explored in Exercise 20 on page 650.

9.3.3 DENCLUE: A Kernel-Based Scheme for Density-Based

Clustering

DENCLUE (DENsity ClUstEring) is a density-basedclustering approach that

models the overall density of a set of points as the sum of influence functions
associated with each point. The resulting overall density function will have
local peaks, i.e., local density maxima, and these local peaks can be used
to define clusters in a natural way. Specifically, for each data point, a hill-
climbing procedure finds the nearest peak associated with that point, and
the set of all data points associated with a particular peak (called a local
density attractor) becomes a cluster. However, if the density at a local
peak is too low, then the points in the associatedcluster are classifiedas noise
 9.3 Density-Based Clustering 609

.=
a
c
o
o

Figure9.13.lllustration
of DENCLUE
density
concepts
inonedimension.

and discarded. Also, if a local peak can be connected to a second local peak
by a path of data points, and the density at each point on the path is above
the minimum density threshold, then the clusters associatedwith these local
peaks are merged. Therefore, clusters of any shape can be discovered.

Example 9.L0 (DENCLUE Density). We illustrate these conceptswith

Figure 9.13, which shows a possible density function for a one-dimensional
data set. Points A-E are the peaks of this density function and represent
local density attractors. The dotted vertical lines delineate local regions of
influence for the local density attractors. Points in these regions will become
center-definedclusters. The dashed horizontal line shows a density threshold,
€. All points associatedwith a local density attractor that has a density less
than (, such as those associatedwith C, will be discarded. All other clusters
are kept. Note that this can include points whose density is lessthan (, as long
as they are associatedwith local density attractors whose density is greater
than {. Finally, clusters that are connected by a path of points with a density
above { are combined. Clusters A and B would remain separate,while clusters
D and E would be combined. r
The high-level details of the DENCLUE algorithm are summarized in Al-
gorithm 9.6. Next, we explore various aspects of DENCLUE in more detail.
First, we provide a brief overview of kernel density estimation and then present
the grid-based approach that DENCLUE uses for approximating the density.

Kernel Density Estimation

DENCLUE is based on a well-developedarea of statistics and pattern recogni-

tion that is known as kernel density estirnation. The goal of this collection
610 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

Algorithm 9.6 DENCLUE algorithm.

1: Derive a density function for the space occupied by the data points.
2: Identify the points that are local maxima.
(These are the density attractors.)
3: Associate each point with a density attractor by moving in the direction of max-
imum increase in density.
4: Define clusters consisting of points associatedwith a particular density attractor.
5: Discard clusters whose density attractor has a density less than a user-specified
threshold of (.
6: Combine clusters that are connected by a path of points that all have a density
of { or higher.

of techniques (and many other statistical techniques as well) is to describe

the distribution of the data by a function. For kernel density estimation, the
contribution of each point to the overall density function is expressedby an
influence or kernel function. The overall density function is simply the sum
of the influence functions associatedwith each point.
Typically, the influence or kernel function is symmetric (the same in all
directions) and its value (contribution) decreasesas the distance from the
point increases.For example, for a particular point, x, the Gaussian function,
K(a): is often used as a kernel function. (o is a parame-
"-distance(x,v)tl2o2,
ter, which is analogousto the standard deviation) that governshow quickly the
influence of a point drops off. Figure 9.14(a) shows what a Gaussian density
function would look like for a single point in two dimensions, while Figures
9.14(c) and 9.14(d) show the overall density function produced by applying
the Gaussian influence function to the set of points shown in Figure 9.14(b).

Irnplementation Issues

Computation of kernel density can be quite expensive,and DENCLUE uses a

number of approximations to implement its basic approach efficiently. First, it
explicitly computes density only at data points. However,this still would result
in an O(m2) time complexity since the density at each point is a function of the
density contributed by every point. To reduce the time complexity, DENCLUE
usesa grid-based implementation to efficiently define neighborhoods and thus
limit the number of points that need to be consideredto define the density at
a point. First, a preprocessingstep creates a set of grid cells. Only occupied
cells are created, and these cells and their related information can be efficiently
accessedvia a search tree. Then, when computing the density of a point and
finding its nearest density attractor, DENCLUtr considers only the points in
 9.3 Density-Based Clustering 61-1
,*-tr.ea&/t e

(b) Set of 12 points.

(c) Overall density gray scale plot. (cl) Overall density surlace plot.

Figure9,14.Example
oftheGaussian (kernel)
influence function
andanoverall
density
function.

the neighborhood; i.e., points in the samecell and in cells that are connectedto
the point's cell. While this approach can sacrifice some accuracy with respect
to density estimation, computational complexity is greatly reduced.

Strengths and Limitations of DENCLUE

DENCLUE has a solid theoretical foundation becauseis based on kernel den-
sity functions and the notion of kernel density estimation, which is a well-
developed area of statistics. For this reason, DENCLUE provides a more
flexible and potentially more accurate way to compute density than other
grid-based clustering techniques and DBSCAN. (DBSCAN is a special case
of DENCLUE.) An approach based on kernel density functions is inherently
6L2 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

computationally expensive,but DENCLUE employs grid-based techniques to

addresssuch issues. Nonetheless,DENCLUE can be more computationally ex-
pensive than other density-basedclustering techniques. AIso, the use of a grid
can adverselyaffect the accuracy of the density estimation, and it makes DEN-
CLUE susceptible to problems common to grid-based approaches; e.g., the
difficulty of choosing the proper grid size. More generally, DtrNCLUE shares
many of the strengths and limitations of other density-basedapproaches. For
instance, DENCLUE is good at handling noise and outliers and it can find
clusters of different shapes and size, but it has trouble with high-dimensional
data and data that contains clusters of widelv different densities.

9.4 Graph-Based Clustering

Section 8.3 discussed a number of clustering techniques that took a graph-
based view of data, in which data objects are represented by nodes and the
proximity between two data objects is represented by the weight of the edge
between the corresponding nodes. This section considers some additional
graph-based clustering algorithms that use a number of key properties and
characteristics of graphs. The following are some key approaches, different
subsets of which are employed by these algorithms.

1. Sparsify the proximity graph to keep only the connections of an object

with its nearest neighbors. This sparsification is useful for handling noise
and outliers. It also allows the use of highly efficient graph partitioning
algorithms that have been developed for sparse graphs.

2. Define a similarity measurebetween two objects based on the number of

nearest neighbors that they share. This approach, which is based on the
observationthat an object and its nearest neighborsusually belong to the
same class, is useful for overcoming problems with high dimensionality
and clusters of varying density.

3 . Define core objects and build clusters around them. To do this for graph-
based clustering, it is necessaryto introduce a notion ofdensity-based on
a proximity graph or a sparsified proximity graph. As with DBSCAN,
building clusters around core objects leads to a clustering technique that
can find clusters of differing shapes and sizes.

A Use the information in the proximity graph to provide a more sophisti-

cated evaluation of whether two clusters should be merged. Specifically,
 9.4 Graph-Based Clustering 613

two clusters are merged only if the resulting cluster will have character-
istics similar to the original two clusters.
We begin by discussing the sparsification of proximity graphs, providing
two examplesof techniqueswhose approach to clustering is based solely on this
technique: MST, which is equivalent to the single link clustering algorithm,
and Opossum. We then discuss Chameleon, a hierarchical clustering algorithm
that uses a notion of self-similarity to determine if clusters should be merged.
We next define Shared Nearest Neighbor (SNN) similarity, a new similarity
measure)and introduce the Jarvis-Patrick clustering algorithm, which usesthis
similarity. Finally, we discuss how to define density and core objects based
on SNN similarity and introduce an SNN density-basedclustering algorithm,
which can be viewed as DBSCAN with a new similarity measure.

9.4.t Sparsification

The m by m proximity matrix for rn data points can be represented as a dense

graph in which each node is connected to all others and the weight of the edge
between any pair of nodes reflects their pairwise proximity. Although every
object has some level of similarity to every other object, for most data sets,
objects are highly similar to a small number of objects and weakly similar
to most other objects. This property can be used to sparsify the proximity
graph (matrix), by setting many of these low-similarity (high-dissimilarity)
values to 0 before beginning the actual clustering process. The sparsification
may be performed, for example, by breaking all links that have a similarity
(dissimilarity) below (above) a specified threshold or by keeping only links to
the k nearest neighbors of point. This latter approach creates what is called
a k-nearest neighbor graph.
Sparsification has several beneficial effects:
Data size is reduced. The amount of data that needs to be processed
to cluster the data is drastically reduced. Sparsification can often elim-
inate more than 99% of the entries in a proximity matrix. As a result,
the size of problems that can be handled is increased.

Clustering may work better. Sparsification techniques keep the con-

nections to their nearest neighbors of an object while breaking the con-
nections to more distant objects. This is in keeping with the nearest
neighbor principle that the nearest neighbors of an object tend to
belong to the same class (cluster) as the object itself. This reduces
the impact of noise and outliers and sharpens the distinction between
clusters.
61.4 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

Construct a Modified

Figure9.15.ldealprocess usingsparsification.
ofclustering

o Graph partitioning algorithms can be used. There has been a

considerableamount of work on heuristic algorithms for finding min-cut
partitionings of sparsegraphs, especiallyin the areas of parallel comput-
ing and the design of integrated circuits. Sparsification of the proximity
graph makes it possible to use graph partitioning algorithms for the
clustering process. For example, Opossum and Chameleon use graph
partitioning.

Sparsification of the proximity graph should be regarded as an initial step

before the use of actual clustering algorithms. In theory, a perfect sparsifi-
cation could leave the proximity matrix split into connected components cor-
responding to the desired clusters, but in practice, this rarely happens. It is
easy for a single edge to link two clusters or for a single cluster to be split
into several disconnectedsubclusters. Indeed, as we shall seewhen we discuss
Jarvis-Patrick and SNN density-basedclustering, the sparse proximity graph
is often modified to yield a new proximity graph. This new proximity graph
can again be sparsified. Clustering algorithms work with the proximity graph
that is the result of all these preprocessingsteps. This processis summarized
in Figure 9.15.

9.4.2 Minimum Spanning T[ee (MST) Clustering

In Section 8.3, where we described agglomerative hierarchical clustering tech-

niques, we mentioned that divisive hierarchical clustering algorithms also exist.
We saw an example of one such technique, bisecting K-means, in Section 8.2.3.
Another divisive hierarchical technique, MST, starts with the minimum span-
ning tree of the proximity graph and can be viewed as an application of sparsi-
fication for finding clusters. We briefly describe this algorithm. Interestingly,
this algorithm also produces the same clustering as single link agglomerative
clustering. See Exercise 13 on page 648.
A minimum spanning tree of a graph is a subgraph that (1) has no
cycles,i.e., is a tree, (2) contains all the nodes of the graph, and (3) has the
minimum total edge weight of all possible spanning trees. The terminology,
 9.4 Graph-Based Clustering 615

Figure9.16.Minimum
spanning points.
treefora setofsixtwo-dimensional

minimum spanning tree, assumesthat we are working only with dissimilarities

or distances, and we will follow this convention. This is not a limitation,
however, since we could convert similarities to dissimilarities or modify the
notion of a minimum spanning tree to work with similarities. An example of
a minimum spanning tree for some two-dimensional points is shown in Figure
9.16.
The MST divisive hierarchical algorithm is shown in Algorithm 9.7. The
first step is to find the MST of the original dissimilarity graph. Note that a
minimum spanning tree can be viewed as a special type of sparsified graph.
Step 3 can also be viewed as graph sparsification. Hence, MST can be viewed
as a clustering algorithm based on the sparsification of the dissimilarity graph.

Algorithm 9.7 MST divisive hierarchical clustering algorithm,

1: Compute a minimum spanning tree for the dissimilarity graph.
2: repeat
3: Create a new cluster by breaking the link corresponding to the largest dissimi-
Iarity.
4: until Only singleton clusters remain.
616 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

9.4.3 OPOSSUM: Optimal Partitioning of Sparse Similarities

Using METIS

OPOSSUM is a clustering technique that was specifically designedfor cluster-

ing sparse, high dimensional data, such as document or market basket data.
Like MST, it performs clustering based on the sparsification of a proximity
graph. However, OPOSSUM uses the METIS algorithm, which was specifi-
cally created for partitioning sparcegraphs. The steps of OPOSSUM are given
in Aleorithm 9.8.

Algorithm 9.8 OPOSSUM clustering algorithm.

1: Compute a sparsified similarity graph.
2: Partition the similarity graph into k distinct components (clusters) using METIS.

The similarity measuresused are those appropriate for sparse,high dimen-

sional data, such as the extended Jaccard measureor the cosine measure. The
METIS graph partitioning program partitions a sparse graph into k distinct
components, where k is a user-specifiedparameter, in order to (1) minimize
the weight of the edges (the similarity) between components and (2) fulfiIl
a balance constraint. OPOSSUM uses one of the following two balance con-
straints: (1) the number of objects in each cluster must be roughly the same,
or (2) the sum of the attribute values must be roughly the same. The second
constraint is useful when, for example, the attribute values represent the cost
of an item.

Strengths and Weaknesses

OPOSSUM is simple and fast. It partitions the data into roughly equal-sized
clusters, which, depending on the goal of the clustering, can be viewed as an
advantage or a disadvantage. Becausethey are constrained to be of roughly
equal size, clusters can be broken or combined. However, if OPOSSUM is
used to generate a large number of clusters, then these clusters are typically
relatively pure pieces of larger clusters. Indeed, OPOSSUM is similar to the
initial step of the Chameleon clustering routine, which is discussednext.

9.4.4 Chameleon: Hierarchical Clustering with Dynamic

Modeling

Agglomerative hierarchical clustering techniques operate by merging the two

most similar clusters, where the definition of cluster similarity depends on
 9.4 Graph-Based Clustering 6L7

a '

aats

(a) (b) (c) (d)

Figure9.17.Situation
inwhichcloseness
is nottheappropriate
merging
criterion.
@1999,IEEE

the particular algorithm. Some agglomerative algorithms, such as group aver-

age, base their notion of similarity on the strength of the connectionsbetween
the two clusters (e.g., the pairwise similarity of points in the two clusters),
while other techniques, such as the single link method, use the closenessof
the clusters (e.g., the minimum distance between points in different clusters)
to measure cluster similarity. Although there are two basic approaches,using
only one of these two approaches may lead to mistakes in merging clusters.
Consider Figure 9.17, which shows four clusters. If we use the closenessof
clusters (as measured by the closest two points in different clusters) as our
merging criterion, then we would merge the two circular clusters, (c) and
(d),which almost touch, instead of the rectangular clusters, (a) and (b), which
are separated by a small gap. However, intuitively, we should have merged
rectangular clusters, (a) and (b). Exercise 15 on page 649 asks for an exam-
ple of a situation in which the strength of connections likewise leads to an
unintuitive result.
Another problem is that most clustering techniques have a global (static)
model of clusters. For instance, K-means assumesthat the clusters will be
globular, while DBSCAN defines clusters based on a single density threshold.
Clustering schemesthat use such a global model cannot handle casesin which
cluster characteristics, such as size, shape, and density, vary widely between
clusters. As an example of the importance of the local (dynamic) modeling of
clusters, consider Figure 9.18. If we use the closenessof clusters to determine
which pair of clusters should be merged, as would be the case if we used, for
example, the single link clustering algorithm, then we would merge clusters (a)
and (b). However, we have not taken into account the characteristics of each
individual cluster. Specifically, we have ignored the density of the individual
clusters. For clusters (a) and (b), which are relatively dense, the distance
between the two clusters is significantly larger than the distance between a
point and its nearest neighbors within the same cluster. This is not the case
for clusters (c) and (d), which are relatively sparse. Indeed, when clusters (c)
618 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

and (d) are merged, they yield a cluster that seemsmore similar to the original
clusters than the cluster that results from merging clusters (a) and (b).
Chameleon is an agglomerative clustering algorithm that addressesthe is-
suesof the previous two paragraphs. It combines an initial partitioning of the
data, using an efficient graph partitioning algorithm, with a novel hierarchical
clustering schemethat usesthe notions of closenessand interconnectivity, to-
gether with the local modeling of clusters. The key idea is that two clusters
should be merged only if the resulting cluster is similar to the two original
clusters. Self-similarity is described first, and then the remaining details of
the Chameleon algorithm are presented.

Deciding Which Clusters to Merge

The agglomerative hierarchical clustering techniques consideredin Section 8.3

repeatedly combine the two closest clusters and are principally distinguished
from one another by the way they define cluster proximity. In contrast, Cha-
meleon aims to merge the pair of clusters that results in a cluster that is most
similar to the original pair of clusters, as measured by closenessand intercon-
nectivity. Becausethis approach depends only on the pair of clusters and not
on a global model, Chameleon can handle data that contains clusters with
widely different characteristics.
Following are more detailed explanations of the properties of closenessand
interconnectivity. To understand these properties, it is necessaryto take a
proximity graph viewpoint and to consider the number of the links and the
strength of those links among points within a cluster and across clusters.

o Relative Closeness (RC) is the absolute closenessof two clusters nor-

malized bv the internal closenessof the clusters. Two clusters are com-
bined only if the points in the resulting cluster are almost as close to
each other as in each of the original clusters. Mathematically,

Sp6'(C;,Ci)
,Dr u
/ 1--
, ( e.17)

where rn6 and mi are the sizes of clusters Ct and Cj, respectively,
Snc(Cn,Cr) is the averageweight of the edges(of the k-nearest neighbor
graph) that connect clusters Cr a;ndCi; Sec(Ct) is the averageweight of
edgesif we bisect cluster C,;; and Ssc(C.i) is the averageweight of edges
if we bisect cluster Cj. (EC stands for edge cut.) Figure 9.18 illustrates
the notion of relative closeness.As discussedpreviously, while clusters
 9.4 Graph-Based Clustering 619

ll&sss

##ffi
s
t&9
a
I t
e*
tq
I 4
a
te
tl 'g**

rrg*
t
(c) (d)

Figure9.18.lllustration
ofthenotion
of relative
closeness. IEEE
G)1999,

Figure
9.19.lllustration
ottl'rlnotion
ofretative
in,.Jl'onn.r,.oness.
@1999,tEEE

(a) and (b) are closer in absolute terms than clusters (c) and (d), this is
not true if the nature of the clusters is taken into account.

Relative Interconnectivity (RI) is the absolute interconnectivity of

two clusters normalized by the internal connectivity of the clusters. Two
clusters are combined if the points in the resulting cluster are almost as
strongly connected as points in each of the original clusters. Mathemat-
icallv.
EC(Ci,C1)
RI: ( e.18)
[tecQ,)+ EC(ciD'
where EC(Ci,C3) is the sum of the edges (of the k-nearest neighbor
graph) that connect clusters Ct and C1; EC(C1) is the minimum sum
of the cut edges if we bisect cluster Ci; and EC(Cj) is the minimum
sum of the cut edges if we bisect cluster C7. Figure 9.19 illustrates the
notion of relative interconnectivity. The two circular clusters, (c) and
620 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

(d), have more connections than the rectangular clusters, (a) and (b).
However, merging (c) and (d) produces a cluster that has connectivity
quite different from that of (c) and (d). In contrast, merging (a) and (b)
produces a cluster with connectivity very similar to that of (a) and (b).

RI and RC can be combined in many different ways to yield an overall

measure of self-similarity. One approach used in Chameleon is to merge the
pair of clusters that maximizes RI(C6,Ci) * RC(Ci,Cj)o, where a is a user-
specified parameter that is typically greater than 1.

Chameleon Algorithm

Chameleon consists of three key steps: sparsification, graph partitioning, and

hierarchical clustering. Algorithm 9.9 and Figure 9.20 describe these steps.

Algorithm 9.9 Chameleon algorithm.

1: Build a k-nearest neighbor graph.
2: Partition the graph using a multilevel graph partitioning algorithm.
3: repeat
4: Merge the clusters that best preservethe cluster self-similarity with respect to
relative interconnectivity and relative closeness.
5: until No more clusters can be merged.

Figure process
9.20.Overall bywhich performs
Chameleon clustering. IEEE
O1999,

Sparsification The first step in Chameleon is to generatea k-nearest neigh-

bor graph. Conceptually, such a graph is derived from the proximity graph,
and it contains links only between a point and its k nearest neighbors, i.e., the
points to which it is closest. As mentioned, working with a sparsified prox-
imity graph instead of the full proximity graph can significantly reduce the
effects of noise and outliers and improve computational efficiency.
 9.4 Graph-Based Clustering 62I

Graph Partitioning

Once a sparsified graph has been obtained, an efficient multilevel graph par-
titioning algorithm, such as METIS (see bibliographic notes) can be used to
partition the data set. Chameleon starts with an all-inclusive graph (cluster)
and then bisects the largest current subgraph (cluster) until no cluster has
more than MIN-SIZE points, where MIN_SIZE is a user-specifiedparameter.
This processresults in a large number of roughly equally sized groups of well-
connectedvertices (highly similar data points). The goal is to ensurethat each
partition contains objects mostly from one true cluster.

Agglomerative Hierarchical Clustering As discussedpreviously, Cha-

meleon merges clusters based on the notion of self-similarity. Chameleon can
be parameterized to merge more than one pair of clusters in a single step and
to stop before all objects have been merged into a single cluster.

Complexity Assume that rn is the number of data points and p is the

number of partitions. Performing an agglomerative hierarchical clustering of
the p partitions obtained from the graph partitioning requires time O(p2 tosp).
(See Section 8.3.1.) The amount of time required for partitioning the graph
is O(mp * mlogm). The time complexity of graph sparsification depends
on how much time it takes to build the k-nearest neighbor graph. For low-
dimensional data, this takes O(mlogrn) time if a k-d tree or a similar type
of data structure is used. Unfortunately, such data structures only work well
for low-dimensional data sets, and thus, for high-dimensional data sets, the
time complexity of the sparsification becomesO(*'). Since only the k-nearest
neighbor list needsto be stored, the spacecomplexity is O(km) plus the space
required to store the data.
Example 9.11. Chameleonwas applied to two data sets that clustering algo-
rithms such as K-means and DBSCAN have difficulty clustering. The results
of this clustering are shown in Figure 9.21. The clusters are identified by the
shading of the points. In Figure 9.21,(a),the two clusters are irregularly shaped
and quite closeto each other. AIso, noise is present. In Figure 9.21(b), the two
clusters are connected by a bridge, and again, noise is present. Nonetheless,
Chameleon identifies what most people would identify as the natural clusters.
Chameleon has specifically been shown to be very effective for clustering spa-
tial data. Finally, notice that Chameleon does not discard noise points, as do
other clustering schemes,but instead assignsthem to the clusters. I
622 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

(a) (b)

Figure9.21.Chameleon tocluster
applied setsofpoints.
a pairoftwo-dimensional IEEE
@1999,

Strengths and Limitations

Chameleon can effectively cluster spatial data, even though noise and outliers
are present and the clusters are of different shapes, sizes, and density. Cha-
meleon assumesthat the groups of objects produced by the sparsification and
graph partitioning process are subclusters; i.e., that most of the points in a
partition belong to the same true cluster. If not, then agglomerative hier-
archical clustering will only compound the errors since it can never separate
objects that have been wrongly put together. (See the discussion in Section
8.3.4.) Thus, Chameleon has problems when the partitioning processdoes not
produce subclusters, as is often the case for high-dimensional data.

9.4.5 Shared Nearest Neighbor Similarity

In some cases,clustering techniques that rely on standard approachesto sim-

ilarity and density do not produce the desired clustering results. This section
examinesthe reasonsfor this and introduces an indirect approach to similarity
that is based on the following principle:

If two points are similar to many of the same points, then they are
similar to one another, even if a direct measurement of similarity
does not indicate this.

We motivate the discussion by first explaining two problems that an SNN

version of similarity addresses:low similarity and differencesin density.

Problerns with Tladitional Sirnilarity in High-Dirnensional Data

In high-dimensional spaces)it is not unusual for similarity to be low. Con-

sider, for example, a set of documents such as a collection of newspaperarticles
 9.4 Graph-Based Clustering 623

Table9.3. Similarity
among
documents
indifferent of a newspaper.
sections
Section Average Cosine Similarity
Entertainment 0.032
Financial 0.030
Foreign 0.030
Metro 0.021
National 0.027
Sports 0.036
AII Sections 0.014

that come from a variety of sectionsof the newspaper: Entertainment, Finan-

cial, Foreign, Metro, National, and Sports. As explained in Chapter 2, these
documents can be viewed as vectors in a high-dimensional space)where each
component of the vector (attribute) records the number of times that each
word in a vocabulary occurs in a document. The cosine similarity measure
is often used to assessthe similarity between documents. For this example,
which comes from a collection of articles from the Los Angeles T'imes,Table
9.3 gives the averagecosine similarity in each section and among the entire set
of documents.
The similarity of each document to its most similar document (the first
nearest neighbor) is better, 0.39 on average. However, a consequenceof low
similarity among objects of the same class is that their nearest neighbor is
often not of the same class. In the collection of documents from which Table
9.3 was generated, about 20% of the documents have a nearest neighbor of
a different class. In general, if direct similarity is low, then it becomes an
unreliable guide for clustering objects, especiallyfor agglomerativehierarchical
clustering, where the closest points are put together and cannot be separated
afterward. Nonetheless,it is still usually the casethat a large majority of the
nearest neighbors of an object belong to the same class; this fact can be used
to define a proximity measure that is more suitable for clustering.

Problems with Differences in Density

Another problem relates to differencesin densities between clusters. Figure

9.22 shows a pair of two-dimensional clusters of points with differing density.
The lower density of the rightmost cluster is reflected in a lower average dis-
tance among the points. Even though the points in the less densecluster form
an equally valid cluster, typical clustering techniques will have more difficulty
finding such clusters. Also, normal measuresof cohesion,such as SSE, will in-
dicate that these clusters are less cohesive. To illustrate with a real example.
624 Chapter 9 Cluster Analysis:

Figure9.22.Twocircular
clusters
of 200uniformly points.
distributed

the stars in a galaxy are no less real clusters of stellar objects than the planets
in a solar system?even though the planets in a solar system are considerably
closer to one another on average,than the stars in a galaxy.

SNN Similarity Computation

In both situations, the key idea is to take the context of points into account in
defining the similarity measure. This idea can be made quantitative by using
a shared nearest neighbor definition of similarity in the manner indicated
by Algorithm 9.10. Essentially, the SNN similarity is the number of shared
neighbors as long as the two objects are on each other's nearest neighbor lists.
Note that the underlying proximity measure can be any meaningful similarity
or dissimilarity measure.

Algorithm 9.10 Computing shared nearest neighbor similarity

1: Find the k-nearest neighbors of all points.
2: if two points, x and y are not among the k-nearest neighbors of each other then
3: si,mi,lari,ty(x,y) - 0
4: else
5: si,rni,lari,ty(x,y) - number of shared neighbors
6: end if

The computation of SNN similarity is described by Algorithm 9.10 and

graphically illustrated by Figure 9.23. Each of the two black points has eight
nearest neighbors, including each other. Four of those nearest neighbors
the points in gray are shared. Thus, the shared nearest neighbor similarity
between the two points is 4.
 9.4 Graph-Based Clustering 625

Figure9.23.Computation
of SNNsimilarity twopoints.
between

The similarity graph of the SNN similarities among objects is called the
SNN similarity graph. Since many pairs of objects will have an SNN simi-
larity of 0, this is a very sparse graph.

SNN Similarity versus Direct Similarity

SNN similarity is useful becauseit addressessome of the problems that occur
with direct similarity. First, since it takes into account the context of an object
by using the number of shared nearest neighbors, SNN similarity handles the
situation in which an object happens to be relatively close to another object,
but belongs to a different class. In such cases,the objects typically do not
share many near neighbors and their SNN similarity is low.
SNN similarity also addressesproblems with clusters of varying density.
In a low-density region, the objects are farther apart than objects in denser
regions. However, the SNN similarity of a pair of points only depends on the
number of nearest neighbors two objects share, not how far these neighbors
are from each object. Thus, SNN similarity performs an automatic scaling
with respect to the density of the points.

9.4.6 The Jarvis-Patrick Clustering Algorithm

Algorithm 9.11 expressesthe Jarvis-Patrick clustering algorithm using the con-
cepts of the last section. The JP clustering algorithm replaces the proximity
between two points with the SNN similarity, which is calculated as described
in Algorithm 9.10. A threshold is then used to sparsify this matrix of SNN
similarities. In graph terms, an SNN similarity graph is created and sparsified.
Clusters are simply the connected components of the SNN graph.

Algorithm 9.11 Jarvis-Patrick clustering algorithm.

1: Compute the SNN similarity graph.
2: Sparsify the SNN similarity graph by applying a similarity threshold.
3: Find the connected components (clusters) of the sparsified SNN similarity graph.
626 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

vv
vvw
";qw
'q? j

(a) Original data (b) Clusters found by Jarvis-Patrick

Figure9.24.Jarvis-Patrick
clustering pointset.
of a two-dimensional

The storage requirements of the JP clustering algorithm are only O(km),

since it is not necessaryto store the entire similarity matrix, even initially.
The basic time complexity of JP clustering is Olm2), since the creation of
the k-nearest neighbor list can require the computation of O(*') proximities.
However, for certain types of data, such as low-dimensional Euclidean data,
special techniques, €.8., D k-d tree, can be used to more efficiently find the
k-nearest neighbors without computing the entire similarity matrix. This can
reducethe time complexity frorn Olm2) to O(mlogrn).
Example 9.12 (JP Clustering of a Two-Dimensional Data Set). We
applied JP clustering to the "fish" data set shown in Figure 9.2a@) to find
the clusters shown in Figure 9.24(b). The size of the nearest neighbor list was
20, and two points were placed in the same cluster if they shared at least 10
points. The different clusters are shown by the different markers and different
shading. The points whose marker is an "x" were classifiedas noise by Jarvis-
Patrick. They are mostly in the transition regions between clusters of different
density.

Strengths and Limitations

Because JP clustering is based on the notion of SNN similarity, it is good

at dealing with noise and outliers and can handle clusters of different sizes,
 9.4 Graph-Based Clustering 627

shapes, and densities. The algorithm works well for high-dimensional data
and is particularly good at finding tight clusters of strongly related objects.
However, JP clustering defines a cluster as a connected component in the
SNN similarity graph. Thus, whether a set of objects is split into two clusters
or left as one may depend on a single link. Hence, JP clustering is somewhat
brittle; i.e., it may split true clusters or join clusters that should be kept
separate.
Another potential limitation is that not all objects are clustered. However,
these objects can be added to existing clusters, and in some cases,there is no
requirement for a complete clustering. JP clustering has a basic time complex-
ity of O(*2), which is the time required to compute the nearest neighbor list
for a set of objects in the general case. In certain cases,e.g., low-dimensional
data, special techniques can be used to reduce the time complexity for finding
nearest neighbors to O(mlogm). Finally, as with other clustering algorithms,
choosing the best values for the parameters can be challenging.

9.4.7 SNN Density

As discussedin the introduction to this chapter, traditional Euclidean density

becomesmeaninglessin high dimensions. This is true whether we take a grid-
based view, such as that used by CLIQUtr, a center-basedview, such as that
used by DBSCAN, or a kernel-density estimation approach, such as that used
by DENCLUE. It is possibleto use the center-baseddefinition of density with a
similarity measurethat works well for high dimensions,e.g., cosineor Jaccard,
but as described in Section 9.4.5, such measuresstill have problems. However,
since the SNN similarity measure reflects the local configuration of the points
in the data space, it is relatively insensitive to variations in density and the
dimensionality of the space, and is a promising candidate for a new measure
of density.
This section explains how to define a concept of SNN density by using
SNN similarity and following the DBSCAN approach described in Section
8.4. For clarity, the definitions of that section are repeated, with appropriate
modification to account for the fact that we are using SNN similarity.

Core points. A point is a core point if the number of points within a given
neighborhood around the point, as determined by SNN similarity and a
supplied parameter -Eps exceedsa certain threshold MinPts, which is
also a supplied parameter.
628 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

n$$fiiis
(a) All points. (b) High SNN (c) Medium SNN (d) Low SNN
density. density. density.

Figure
9.25.SNNdensity points.
oftwo-dimensional

Border points. A border point is a point that is not a core point, i.e., there
are not enough points in its neighborhood for it to be a core point, but
it falls within the neighborhood of a core point.

Noise points. A noise point is any point that is neither a core point nor a
border point.

SNN density measuresthe degreeto which a point is surrounded by similar

points (with respect to nearest neighbors). Thus, points in regions of high and
low density will typically have relatively high SNN density, while points in
regions where there is a transition from low to high density-points that are
between clusters will tend to have low SNN density. Such an approach may
be better suited for data sets in which there are wide variations in densitv" but
clusters of low density are still interesting.
Example 9.13 (Core, Border, and Noise Points). To make the preceding
discussionof SNN density more concrete, we provide an example of how SNN
density can be used to find core points and remove noise and outliers. There
are 10,000 points in the 2D point data set shown in Figure 9.25(a). Figures
9.25(b-d) distinguish between these points based on their SNN density. Figure
9.25(b) shows the points with the highest SNN density, while Figure 9.25(c)
shows points of intermediate SNN density, and Figure 9.25(d) shows figures
of the lowest SNN density. Frorn these figures, we see that the points that
have high density (i.e., high connectivity in the SNN graph) are candidates for
being representativeor core points since they tend to be located well inside the
cluster, while the points that have low connectivity are candidates for being
noise points and outliers, as they are mostly in the regions surrounding the
clusters.
 9.4 Graph-Based Clustering 629

9.4.8 SNN Density-Based Clustering

The SNN density defined above can be combined with the DBSCAN algorithm
to creaJe a new clustering algorithm. This algorithm is similar to the JP
clustering algorithm in that it starts with the SNN similarity graph. However,
instead of using a threshold to sparsify the SNN similarity graph and then
taking connected components as clusters, the SNN density-based clustering
algorithm simply applies DBSCAN.

The SNN Density-based Clustering Algorithm

The steps of the SNN density-based clustering algorithm are shown in AIgo-
rithm 9.12.

Algorithm 9.12 SNN density-basedclustering algorithm.

1: Compute the SNN similarity graph.
2: Apply DBSCAN with user-specified parameters for Eps and Mi'nPts

The algorithm automatically determines the number of clusters in the data.

Note that not all the points are clustered. The points that are discarded
include noise and outliers, as well as points that are not strongly connected
to a group of points. SNN density-basedclustering finds clusters in which the
points are strongly related to one another. Depending on the application, we
might want to discard many of the points. For example, SNN density-based
clustering is good for finding topics in groups of documents.
Example 9.14 (SNN Density-based Clustering of Time Series). The
SNN density-basedclustering algorithm presented in this section is more flex-
ible than Jarvis-Patrick clustering or DBSCAN. Unlike DBSCAN, it can be
used for high-dimensional data and situations in which the clusters have dif-
ferent densities. Unlike Jarvis-Patrick, which performs a simple thresholding
and then takes the connectedcomponents as clusters, SNN density-basedclus-
tering uses a less brittle approach that relies on the concepts of SNN density
and core points.
To demonstrate the capabilities of SNN density-based clustering on high-
dimensional data, we applied it to monthly time series data of atmospheric
pressureat various points on the Earth. More specifically,the data consists of
the averagemonthly sea-levelpressure (SLP) for a period of 41 years at each
point on a 2.5" longitude-latitude grid. The SNN density-based clustering
algorithm found the clusters (gray regions) indicated in Figure 9.26. Note
630 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

that these are clusters of time series of length 492 months) even though they
are visualized as two-dimensional regions. The white areasare regions in which
the pressure was not as uniform. The clusters near the poles are elongated
becauseof the distortion of mapping a spherical surface to a rectangle.
Using SLP, Earth scientists have defined time series, called climate in-
dices, that are useful for capturing the behavior of phenomena involving the
Earth's climate. For example, anomalies in climate indices are related to
abnormally low or high precipitation or temperature in various parts of the
world. Some of the clusters found by SNN density-based clustering have a
strong connection to some of the climate indices known to Earth scientists.
Figure 9.27 shows the SNN density structure of the data from which the
clusters were extracted. The density has been normalized to be on a scale
between 0 and 1. The density of a time series may seem like an unusual
concept, but it measuresthe degree to which the time series and its nearest
neighbors have the same nearest neighbors. Since each time seriesis associated
with a location, it is possible to plot these densities on a two-dimensional plot.
Becauseof temporal autocorrelation, these densitiesform meaningful patterns,
e.g., it is possible to visually identify the clusters of Figure 9.27.

Strengths and Limitations

The strengths and limitations of SNN density-based clustering are similar to

those of JP clustering. However, the use of core points and SNN density adds
considerablepower and flexibility to this approach.

9.5 Scalable Clustering Algorithms

Even the best clustering algorithm is of little value if it takes an unacceptably
long time to execute or requires too much memory. This section examines
clustering techniques that place significant emphasis on scalability to the very
large data sets that are becoming increasingly common. We start by discussing
some general strategies for scalability, including approachesfor reducing the
number of proximity calculations, sampling the data, partitioning the data,
and clustering a summarized representation of the data. We then discusstwo
specific examples of scalable clustering algorithms: CURE and BIRCH.

9.5.1 Scalability: General Issues and Approaches

The amount of storage required for many clustering algorithms is more than
linear; e.g., with hierarchical clustering) memory requirements are usually
 9.5 Scalable Clustering Algorithms 631

90

60
F-{^ 6m
sffi P
AE

t4
1*r
q ,

F
t*"

\
J
7r! !
h

k JT
I
AN
fc J- &-
ffi t
JE
t nQ # x( ll
FA'
30
\ f\J 4L k \\) I , rrF
o
-Xffi V"{t rU TJ h .
.Eo x
_:t

"LT ht
"+_E\
j
(
{t F-
-30
til '\
\',

-60
t,
1a
\!
t-

*
p
x

*K
#
q
-90
-180-150-120-90-60 -30 0 30 60 90 120 150 180
Longitude

of pressure
Figure9.26.Clusters foundusingSNNdensity-based
timeseries clustering.

-180-150-120-90 -60 -30 0 30 60 90 120 150 Density

Longitude

ofpressure
9.27.SNNdensity
Figure timeseries.
632 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

O(*'), where m is the number of objects. For 10,000,000objects, for ex-

ample, the amount of memory required is proportional to 1014,a number still
well beyond the capacities of current systems. Note that because of the re-
quirement for random data access,many clustering algorithms cannot easily
be modified to efficiently use secondary storage (disk), for which random data
accessis slow. Likewise, the amount of computation required for some clus-
tering algorithms is more than linear. In the remainder of this section, we
discuss a variety of techniques for reducing the amount of computation and
storage required by a clustering algorithm. CURtr and BIRCH use some of
these techniques.

Multidimensional or Spatial Access Methods Many techniques-K-

means, Jarvis Patrick clustering, and DBSCAN-need to find the closest cen-
troid, the nearest neighbors of a point, or all points within a specifieddistance.
It is possibleto use special techniquescalled multidimensional or spatial access
methods to more efficiently perform these tasks, at least for low-dimensional
data. These techniques, such as the k-d tree or R*-tree, typically produce a
hierarchical partition of the data spacethat can be used to reduce the time re-
quired to find the nearest neighbors of a point. Note that grid-based clustering
schemesalso partition the data space.

Bounds on Proximities Another approach to avoiding proximity compu-

tations is to use bounds on proximities. For instance, when using Euclidean
distance, it is possible to use the triangle inequality to avoid many distance
calculations. To illustrate, at each stage of traditional K-means, it is necessary
to evaluate whether a point should stay in its current cluster or be moved to
a new cluster. If we know the distance between the centroids and the distance
of a point to the (newly updated) centroid of the cluster to which it currently
belongs, then we may be able to use the triangle inequality to avoid computing
the distance ofthe point to any ofthe other centroids. SeeExercise 21 on page
0br.r.

Sampling Another approach to reducing the time complexity is to sample.

In this approach, a sample of points is taken, these points are clustered, and
then the remaining points are assigned to the existing clusters typically to
the closest cluster. If the number of points sampled is Jrn, then the time
complexity of an O(m2) algorithm is reduced to O(m). A key problem with
sampling, though, is that small clusters can be lost. When we discuss CURE,
 9.5 Scalable Clustering Algorithms 633

we will provide a technique for investigating how frequently such problems

occur.

Partitioning the Data Objects Another common approach to reducing

time complexity is to use some efficient technique to partition the data into
disjoint sets and then cluster these sets separately. The fi.nal set of clusters
either is the union of these separatesets of clusters or is obtained by combining
and/or refining the separate sets of clusters. We only discuss bisecting K-
means (Section 8.2.3) in this section, although many other approachesbased
on partitioning are possible. One such approach will be described, when we
describe CURE later on in this section.
If K-means is used to find l( clusters, then the distance of each point to
each cluster centroid is calculated at each iteration. When K is large, this can
be very expensive. Bisecting K-means starts with the entire set of points and
uses K-means to repeatedly bisect an existing cluster until we have obtained
K clusters. At each step, the distance of points to two cluster centroids is
computed. Except for the first step, in which the cluster being bisectedconsists
of all the points, we only compute the distance of a subset of points to the two
centroids being considered. Because of this fact, bisecting K-means can run
significantly faster than regular K-means.

Summarization Another approach to clustering is to summarize the data,

typically in a single pass, and then cluster the summarized data. In particular,
the leader algorithm (see Exercise 12 on page 562) either puts a data object
in the closest cluster (if that cluster is sufficiently close) or starts a new clus-
ter that contains the current object. This algorithm is linear in the number
of objects and can be used to summarize the data so that other clustering
techniques can be used. The BIRCH algorithm uses a similar concept.

Parallel and Distributed Computation If it is not possible to take ad-

vantage of the techniques described earlier, or if these approachesdo not yield
the desired acculacy or reduction in computation time, then other approaches
are needed. A highly effectiveapproach is to distribute the computation among
multiple processors.

9.5.2 BIRCH
BIRCH (Balanced Iterative Reducing and Clustering using Hierarchies) is a
highly efificientclustering technique for data in Euclidean vector spaces,i'e.,
634 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

data for which averagesmake sense. BIRCH can efficiently cluster such data
with one pass and can improve that clustering with additional passes.BIRCH
can also deal effectively with outliers.
BIRCH is based on the notion of a Clustering Feature (CF) and a CF tree.
The idea is that a cluster of data points (vectors) can be represented by a
triple of numbers (N, LS, SS), where N is the number of points in the cluster,
,L,9is the linear sum of the points, and ,9,9is the sum of squaresof the points.
These are common statistical quantities that can be updated incrementally
and that can be used to compute a number of important quantities, such as
the centroid of a cluster and its variance (standard deviation). The variance
is used as a measure of the diameter of a cluster.
These quantities can also be used to compute the distance between clusters.
The simplest approach is to calculate an L1 (city block) or L2 (Euclidean)
distance between centroids. We can also use the diameter (variance) of the
merged cluster as a distance. A number of different distance measures for
clusters are defined by BIRCH, but all can be computed using the summary
statistics.
A CF tree is a height-balanced tree. Each interior node has entries of the
form [CF;, childi], where child; is a pointer to the i,th child node. The space
that each entry takes and the page size determine the number of entries in an
interior node. The space of each entry is, in turn, determined by the number
of attributes of each point.
Leaf nodes consist of a sequenceof clustering features, CFi, where each
clustering feature represents a number of points that have been previously
scanned. Leaf nodes are subject to the restriction that each leaf node must
have a diameter that is less than a parameterized threshold, ?. The space
that each entry takes, together with the page size, determines the number of
entries in a leaf.
By adjusting the threshold parameter T, the height of the tree can be
controlled. 7 controls the finenessof the clustering, i.e., the extent to which
the data in the original set of data is reduced. The goal is to keep the CF tree
in main memory by adjusting the ? parameter as necessary.
A CF tree is built as the data is scanned. As eachdata point is encountered,
the CF tree is traversed, starting from the root and choosing the closest node
at each level. When the closest leaf cluster for the current data point is finally
identified, a test is performed to see if adding the data item to the candidate
cluster will result in a new cluster with a diameter greater than the given
threshold, ?. If not, then the data point is added to the candidate cluster bv
 9.5 Scalable Clustering Algorithms 635

updating the CF information. The cluster information for all nodes from the
leaf to the root is also updated.
If the new cluster has a diameter greater than T, then a new entry is
created if the leaf node is not full. Otherwise the leaf node must be split.
The two entries (clusters) that are farthest apart are selectedas seedsand the
remaining entries are distributed to one of the two new leaf nodes, based on
which leaf node contains the closest seedcluster. Once the leaf node has been
split, the parent node is updated and split if necessary;i.e., if the parent node
is full. This processmay continue all the way to the root node.
BIRCH follows each split with a merge step. At the interior node where the
split stops, the two closestentries are found. Ifthese entries do not correspond
to the two entries that just resulted from the split, then an attempt is made to
merge these entries and their corresponding child nodes. This step is intended
to increasespaceutilization and avoid problems with skeweddata input order.
BIRCH also has a procedure for removing outliers. When the tree needs
to be rebuilt because it has run out of memory, then outliels can optionally
be written to disk. (An outlier is defined to be a node that has far fewer data
points than average.) At certain points in the process, outliers are scanned
to see if they can be absorbed back into the tree without causing the tree to
grow in size. If so, they are reabsorbed. If not, they are deleted.
BIRCH consists of a number of phases beyond the initial creation of the
CF tree. All the phasesof BIRCH are described briefly in Algorithm 9.13.

9.5.3 CURE
CURE (Clustering Using REpresentatives) is a clustering algorithm that uses
a variety of different techniques to create an approach that can handle large
data sets, outliers, and clusters with non-spherical shapes and non-uniform
sizes. CURE representsa cluster by using multiple representative points from
the cluster. These points will, in theory, capture the geometry and shape of the
cluster. The first representative point is chosen to be the point farthest from
the center of the cluster, while the remaining points are chosenso that they are
farthest from all the previously chosen points. In this way, the representative
points are naturally relatively well distributed. The number of points chosen
is a parameter, but it was found that a value of 10 or more worked well.
Once the representative points are chosen, they are shrunk toward the
center by a factor, a. This helps moderate the effect of outliers, which are
usually farther away from the center and thus, are shrunk more. For example,
a representative point that was a distance of 10 units from the center would
636 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

Algorithm 9.13 BIRCH.

1: Load the data into memory by creating a CF tree that summarizes the
data.
2: Build a smaller CF tree if it is necessary for phase 3. 7 is increased,and
then the leaf node entries (clusters) are reinserted. Since ? has increased, some
clusters will be merged.
3: Perform global clustering. Different forms of global clustering (clustering that
uses the pairwise distances between all the clusters) can be used. However, an
agglomerative, hierarchical technique was selected. Because the clustering features
store summary information that is important to certain kinds of clustering, the
global clustering algorithm can be applied as if it were being applied to all the
points in a cluster representedby the CF.
4: Redistribute the data points using the centroids of clusters discovered
in step 3, and thus, discover a new set of clusters. This overcomescertain
problems that can occur in the first phase of BIRCH. Because of page size con-
straints and the 7 parameter, points that should be in one cluster are sometimes
split, and points that should be in different clusters are sometimes combined.
Also, if the data set contains duplicate points, these points can sometimes be
clustered differently, depending on the order in which they are encountered. By
repeating this phase multiple times, the process convergesto a locally optimum
solution.

move by 3 units (for a :0.7), while a representative point at a distance of 1

unit would only move 0.3 units.
CURtr uses an agglomerative hierarchical scheme to perform the actual
clustering. The distance between two clusters is the minimum distance be-
tween any two representative points (after they are shrunk toward their re-
spective centers). While this schemeis not exactly like any other hierarchical
schemethat we have seen,it is equivalent to centroid-basedhierarchical clus-
tering if a :0, and roughly the same as single link hierarchical clustering if
a : r. Notice that while a hierarchical clustering schemeis used, the goal of
CURE is to find a given number of clusters as specified by the user.
CURE takes advantage of certain characteristics of the hierarchical clus-
tering process to eliminate outliers at two different points in the clustering
process. First, if a cluster is growing slowly, then this may mean that it con-
sists mostly of outliers, since by definition, outliers are far from others and
will not be merged with other points very often. In CURE, this first phase
of outlier elimination typically occurs when the number of clusters is 1/3 the
original number of points. The secondphase of outlier elimination occurs when
the number of clusters is on the order of K, the number of desired clusters.
At this point, small clusters are again eliminated.
 9.5 Scalable Clustering Algorithms 637

Since the worst-casecomplexity of CURE is O(m2logm), it cannot be ap-

plied directly to large data sets. For this reason, CURE uses two techniques
to speed up the clustering process. The first technique takes a random sample
and performs hierarchical clustering on the sampled data points. This is fol-
Iowed by a final pass that assigns each remaining point in the data set to one
of the clusters by choosing the cluster with the closest representative point.
We discuss CURE's sampling approach in more detail later.
In some cases,the sample required for clustering is stil too large and a
secondadditional technique is required. In this situation, CURtr partitions the
sample data and then clusters the points in each partition. This preclustering
step is then followed by a clustering of the intermediate clusters and a final
pass that assigns each point in the data set to one of the clusters. CURE's
partitioning schemeis also discussedin more detail later.
Algorithm 9.14 summarizes CURE. Note that 1( is the desired number of
clusters, m is the number of points, p is the number of partitions, and q is
the desired reduction of points in a partition, i.e., the number of clusters in a
partition is Therefore, the total number of clusters is For example, if
ffi. ffi.
m : 10,000,P : I0, and q : 100, then each partition contains 10,000/10:
1000 points, and there would be 1000/100 : 10 clusters in each partition and
10.000/100: 100 clustersoverall.

Algorithm 9.14 CURE.

1: Draw a random sample from the data set. The CURE paper is notable for
explicitly deriving a formula for what the size of this sample should be in order to
guarantee, with high probability, that all clusters are represented by a minimum
number of points.
2: Partition the sample irft,o p equal-sized partitions.
3: Cluster the points in each partition into ffi clusters using CURE's hi-
erarchical clustering algorithm to obtain a total of f clusters. Note that
some outlier elimination occurs during this process.
Use CURE's hierarchical clustering algorithm to cluster the I clusters
found in the previous step until only K clusters remain.
Eliminate outliers. This is the second phase of outlier elimination.
Assign all remaining data points to the nearest cluster to obtain a
complete clustering.
638 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

Sampling in CURE

A key issue in using sampling is whether the sample is representative, that

is, whether it captures the characteristics of interest. For clustering, the issue
is whether we can find the same clusters in the sample as in the entire set of
objects. Ideally, we would like the sample to contain some objects for each
cluster and for there to be a separate cluster in the sample for those objects
that belong to separate clusters in the entire data set.
A more concrete and attainable goal is to guarantee (with a high probabil-
ity) that we have at least some points from each cluster. The number of points
required for such a sample varies from one data set to another and depends
on the number of objects and the sizesof the clusters. The creators of CURtr
derived a bound for the sample size that would be neededto ensure (with high
probability) that we obtain at least a certain number of points from a cluster.
Using the notation of this book, this bound is given by the following theorem.

Theorem 9.L. Let f be a fraction, 0 < < 1. For cluster Ct of s'izemi,

"f
we wi,Il obtai,n at least f * mt objects from cluster C6 wi,th a probabi,li,tyof
1- d,0 < d < 7, 'if our sample s'izes 'isgi,uenby the following:

(e.1e)

where rn i,s the number of objects.

While this expressionmay look intimidating, it is reasonably easy to use.

Suppose that there are 100,000 objects and that the goal is to have an 80%
chance of obtainingT0% of the objects in cluster Ci, which has a size of 1000.
In this case,/ : 0.1, 6 :0.2, rn :100,000, mi:1000, and thus s :17,962. If
the goal is a 5Tosample of Ci, which is 50 objects, then a sample size of 6440
will suffice.
Again, CURE uses sampling in the following way. First a sample is drawn,
and then CURE is used to cluster this sample. After clusters have been found,
each unclustered point is assignedto the closest cluster.

Partitioning

When sampling is not enough, CURE also uses a partitioning approach. The
idea is to divide the points into p groups of size m f p and to use CURE to cluster
each partition in order to reduce the number of objects by a factor of q > I,
where q can be roughly thought of as the averagesize of a cluster in a partition.
 9.6 Which Clustering Algorithm? 639

Overall, clusters are produced. (Note that since CURE represents each
ffi
cluster by a number of representative points, the reduction in the number of
objects is not pq.) This preclustering step is then followed by a final clustering
of the mf pq intermediate clusters to produce the desired number of clusters
(K). Both clustering passesuse CURE's hierarchical clustering algorithm and
are followed by a final pass that assigns each point in the data set to one of
the clusters.
The key issue is how p and q should be chosen. Algorithms such as CURE
have a time complexity of O(^') or higher, and furthermore, require that all
the data be in main memory. We therefore want to choose p small enough so
that an entire partition can be processedin main memory and in a'reasonable'
amount of time. At the current time, a typical desktop computer can perform
a hierarchical clustering of a few thousand objects in a few seconds.
Another factor for choosing p, and also q, concerns the quality of the
clustering. Specifically, the objective is to choosethe values of p and q such
that objects from the same underlying cluster end up in the same clusters
eventually. To illustrate, suppose there are 1000 objects and a cluster of size
100. If we randomly generate 100 partitions, then each partition will, on
average, have only one point from our cluster. These points will likely be put
in clusters with points from other clusters or will be discarded as outliers. If
we generate only 10 partitions of 100 objects, but q is 50, then the 10 points
from each cluster (on average) will likely still be combined with points from
other clusters, since there are only (on average) 10 points per cluster and we
need to produce, for each partition, two clusters. To avoid this last problem'
which concernsthe proper choice of q, a suggestedstrategy is not to combine
clusters if they are too dissimilar.

9.6 Which Clustering Algorithm?

A variety of factors need to be consideredwhen deciding which type of cluster-
ing technique to use. Many, if not all, of these factors have been discussedto
some extent in the current and previous chapters. Our goal in this section is
to succinctly summarize these factors in a way that sheds some light on which
clustering algorithm might be appropriate for a particular clustering task.

Type of Clustering One important factor in making sure that the type of
clustering matches the intended use is the type of clustering produced by the
algorithm. For some applications, such as creating a biological taxonomy, a
640 Chapter I Cluster Analysis: Additional Issues and Algorithms

hierarchy is preferred. In the caseof clustering for summarization, a partitional

clustering is typical. In yet other applications, both may prove useful.
Most clustering applications require a clustering of all (or almost all) of the
objects. For instance, if clustering is used to organize a set of documents for
browsing, then we would like most documents to belong to a group. However,
if we wanted to find the strongest themes in a set of documents, then we might
prefer to have a clustering schemethat produces only very cohesiveclusters,
even if many documents were left unclustered.
Finally, most applications of clustering assumethat each object is assigned
to one cluster (or one cluster on a level for hierarchical schemes).As we have
seen, however, probabilistic and fitzzy schemesprovide weights that indicate
the degreeor probability of membership in various clusters. Other techniques,
such as DBSCAN and SNN density-based clustering, have the notion of core
points, which strongly belong to one cluster. Such concepts may be useful in
certain applications.

Type of Cluster Another key aspect is whether the type of cluster matches
the intended application. There are three commonly encountered types of
clusters: prototype-, graph-, and density-based. Prototype-based clustering
schemes,as well as some graph-basedclustering schemes-complete link, cen-
troid, and Ward's-tend to produce globular clusters in which each object is
sufficiently close to the cluster's prototype or to the other objects in the clus-
ter. If, for example, we want to summarize the data to reduce its size and we
want to do so with the minimum amount of error, then one of these types of
techniques would be most appropriate. In contrast, density-based clustering
techniques, as well as some graph-based clustering techniques, such as single
link, tend to produce clusters that are not globular and thus contain many ob-
jects that are not very similar to one another. If clustering is used to segment
a geographical area into contiguous regions based on the type of land cover,
then one of these techniques is more suitable than a prototype-based scheme
such as K-means.

Characteristics of Clusters Besidesthe generaltype of cluster, other clus-

ter characteristics are important. If we want to find clusters in subspacesof
the original data space, then we must choose an algorithm such as CLIQUE,
which explicitly looks for such clusters. Similarly, if we are interested in enforc-
ing spatial relationships between clusters, then SOM or some related approach
would be appropriate. Also, clustering algorithms differ widely in their ability
to handle clusters of varying shapes,sizes,and densities.
 9.6 Which Clustering Algorithm? 64L

Characteristics of the Data Sets and Attributes As discussedin the

introduction, the type of data set and attributes can dictate the type of algo-
rithm to use. For instance, the K-means algorithm can only be used on data
for which an appropliate proximity measure is available that allows meaning-
ful computation of a cluster centroid. For other clustering techniques, such
as many agglomerative hierarchical approaches,the underlying nature of the
data sets and attributes is less important as long as a proximity matrix can
be created.

Noise and Outliers Noise and outliers are particularly important aspects
of the data. We have tried to indicate the effect of noise and outliers on the
various clustering algorithms that we have discussed. In practice, however, it
may be difficult to evaluate the amount of noise in the data set or the number
of outliers. More than that, what is noise oI an outlier to one person may
be interesting to another pelson. For example, if we are using clustering to
segment an area into regions of different population density, we do not want
to use a density-based clustering technique, such as DBSCAN, that assumes
that regions or points with density lower than a global threshold are noise or
outliers. As another example, hierarchical clustering schemes,such as CURE,
often discard clusters of points that are growing slowly since such groups tend
to representoutliers. However, in some applications we may be most interested
in relatively small clustersl e.g., in market segmentation, such groups might
represent the most profitable customers.

Number of Data Objects We have considered how clustering is affected

by the number of data objects in considerabledetail in previous sections. We
reiterate, however,that this factor often plays an important role in determining
the type of clustering algorithm to be used. Suppose that we want to create
a hierarchical clustering of a set of data, we are not interested in a complete
hierarchy that extends all the way to individual objects, but only to the point
at which we have split the data into a few hundred clusters. If the data is
very large, we cannot directly apply an agglomerative hierarchical clustering
technique. We could, however, use a divisive clustering technique, such as
the minimum spanning tree (MST) algorithm, which is the divisive analog to
single link, but this would only work if the data set is not too large. Bisecting
K-means would also work for many data sets, but if the data set is large enough
that it cannot be contained completely in memory, then this schemealso runs
into problems. In this situation, a technique such as BIRCH, which does not
require that all data be in main memory, becomesmore useful'
642 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

Number of Attributes We have also discussedthe impact of dimension-

ality at some length. Again, the key point is to realize that an algorithm
that works well in low or moderate dimensions may not work well in high
dimensions. As in many other casesin which a clustering algorithm is inap-
propriately applied, the clustering algorithm may run and produce clusters,
but the clusters may not represent the true structure of the data.

Cluster Description One aspect of clustering techniquesthat is often over-

looked is how the resulting clusters are described. Prototype clusters are suc-
cinctly described by a small set of cluster prototypes. In the case of mixture
models, the clusters are described in terms of small sets of parameters, such as
the mean vector and the covariance matrix. This is also a very compact and
understandable representation. For SOM, it is typically possible to visualize
the relationships between clusters in a two-dimensional plot, such as that of
Figure 9.8. For graph- and density-basedclustering approaches,however, clus-
ters are typically described as sets of cluster members. Nonetheless,in CURE,
clusters can be described by a (relatively) small set of representative points.
Also, for grid-based clustering schemes,such as CLIQUE, more compact de-
scriptions can be generated in terms of conditions on the attribute values that
describe the grid cells in the cluster.

Algorithmic Considerations There are also important aspects of algo-

rithms that need to be considered. Is the algorithm non-deterministic or
order-dependent? Does the algorithm automatically determine the number
of clusters? Is there a technique for determining the values of various pa-
rameters? Many clustering algorithms try to solve the clustering problem by
trying to optimize an objective function. Is the objective a good match for
the application objective? If not, then even if the algorithm does a good job
of finding a clustering that is optimal or close to optimal with respect to the
objective function, the result is not meaningful. Also, most objective functions
give preferenceto larger clusters at the expenseof smaller clusters.

Sumrnary The task of choosing the proper clustering algorithm involves

considering all of these issues,and domain-specific issuesas well. There is no
formula for determining the proper technique. Nonetheless,a general knowl-
edge of the types of clustering techniques that are available and consideration
of the issues mentioned above, together with a focus on the intended appli-
cation, should allow a data analyst to make an informed decision on which
clustering approach (or approaches)to try.
 9.7 BibliographicNotes 643

9.7 Bibliographic Notes

An extensive discussion of fiizzy clustering, including a description of finzy
c-meansand formal derivations of the formulas presentedin Section 9.2.7, can
be found in the book on fuzzy cluster analysis by Hcippner et al. f4al]. While
not discussed in this chapter, AutoClass by Cheeseman et al. 1424]is one
of the earliest and most prominent mixture-model clustering programs. An
introduction to mixture models can be found in the tutorial of Bilmes [420],
the book by Mitchell [450] (which also describeshow the K-means algorithm
can be derived from a mixture model approach), and the article by Fraley and
Raftery [429].
Besidesdata exploration, SOM and its supervised learning variant, Learn-
ing Vector Quantization (LVQ), have been used for many tasks: image segmen-
tation, organization of document files, and speechprocessing. Our discussion
of SOM was cast in the terminology of prototype-based clustering. The book
on SOM by Kohonen et al. 14471contains an extensive introduction to SOM
that emphasizesits neural network origins, as well as a discussionof some of
its variations and applications. One important SOM-related clustering devel-
opment is the Generative Topographic Map (GTM) algorithm by Bishop et
al. [421], which uses the EM algorithm to find Gaussian models satisfying
two-dimensional topographic constraints.
The description of Chameleon can be found in the paper by Karypis et
al. [ 5]. Capabilities similar, although not identical to those of Chameleon
have been implemented in the CLUTO clustering package by Karypis [425].
The METIS graph partitioning packageby Karypis and Kumar [446] is used
to perform graph partitioning in both programs, as well as in the OPOSSUM
clustering algorithm by Strehl and Ghosh 1459].The notion of SNN similarity
was introduced by Jarvis and Patrick 1442). A hierarchical clustering scheme
basedon a similar concept of mutual nearestneighborswas proposed by Gowda
and Krishna la3\. Guha et al. [437] created ROCK, a hierarchical graph-
based clustering algorithm for clustering transaction data, which among other
interesting features, also uses a notion of similarity based on shared neighbors
that closely resemblesthe SNN similarity developed by Jarvis and Patrick. A
description of the SNN density-basedclustering technique can be found in the
publications of Ertoz et al. 1426,427]. SNN density-basedclustering was used
by Steinbach et al. l457l to find climate indices.
Examples of grid-based clustering algorithms are OptiGrid (Hinneburg and
Keim [4a0]), the BANG clustering system (Schikuta and Erhart [a55]), and
WaveCluster (Sheikholeslami et al. [456]). The CLIQUtr algorithm is de-
scribed in the paper by Guha et al. [418]. MAFIA (Nageshet al. [452]) is
644 Chapter 9 Cluster Analysis: Additional Issues and Alsorithms

a modification of CLIQUE whose goal is improved efficiency. Kailing et al.

14441have developed SUBCLU (density-connected SUBspace ClUstering), a
subspaceclustering algorithm based on DBSCAN. The DENCLUtr algorithm
was proposed by Hinneburg and Keim [439].
Our discussionof scalability was strongly influenced bv the article of Ghosh
1432].A wide-ranging discussion of specific techniques for clustering massive
data sets can be found in the paper by Murtagh [451]. CURE is work by Guha
et al. [436], while details of BIRCH are in the paper by Zhang et al. [460].
CLARANS (Ng and Han [453]) is an algorithm for scaling K-medoid clustering
to larger databases. A discussion of scaling EM and K-means clustering to
large data sets is provided by Bradley et al. 1422,423].
There are many aspectsof clustering that we have not covered. Additional
pointers are given in the books and surveys mentioned in the bibliographic
notes of the previous chapter. Here, we mention four areas-omitting, un-
fortunately, many more. Clustering of transaction data (Ganti et al. [430],
Gibson et al. [433],Han et al. l 38], and Peters andZaki [ 5 ]) is an important
area, as transaction data is common and of commercial importance. Streaming
data is also becoming increasingly common and important as communications
and sensor networks become pervasive. Two introductions to clustering for
data streams are given in articles by Barbard [a19] and Guha et al. [4gb].
Conceptual clustering (Fisher and Langley la28], Jonyer et al. 1443],Mishra
et al. [449], Michalski and Stepp [448], Stepp and Michalski l 5S]), which uses
more complicated definitions of clusters that often correspond better to human
notions of a cluster, is an area of clustering whose potential has perhaps not
been fully realized. Finally, there has been a great deal of clustering work for
data compressionin the area of vector quantization. The book bv Gersho and
Gray [a31] is a standard text in this area.

Bibliography
1418] R. Agrawal, J. Gehrke, D. Gunopulos, and P. Raghavan. Automatic subspaceclustering
of high dimensional data for data mining applications. In Proc. of 1998 ACM-SIGMOD
IntL. Conf. on Managernent of Data, pages 94 105, Seattle, Washington, June 1998.
ACM Press.
[419] D. Barbar6. Requirements for clustering data streams. SIGKDD Etplorations Newslet-
ter, 3(2):23-27, 2002.
1420] J. Bilmes. A Gentle Tutorial on the EM Algorithm and its Application to Parameter
Estimation for Gaussian Mixture and Hidden Markov Models. Technical Report ICSI-
TR-97-021, University of California at Berkeley, 1997.
l42Il C. M. Bishop, M. Svensen, and C. K. I. Williams. GTM: A principled alternative to
the self-organizing map. In C. von der Malsburg, W. von Seelen,J. C. Vorbruggen, and
 Bibliography 645

B. Sendhoff, editors, Arti,fi,c'ial Neural Netuorks-ICANN96. Intl. Conf, Proc., pages

165 170. Springer-Verlag, Berlin, Germany, 1996.
14221 P. S. Bradley, U. M. Fayyad, and C. Reina. Scaling Clustering Algorithms to Large
Databases. In Proc. of the lth IntI. Conf. on Knowledge Discouerltr and Data Mining,
pages 9 15, New York City, August 1998. AAAI Press.
1423] P. S. Bradley, U. M. Fayyad, and C. Reina. Scaling EM (Expectation Maximization)
Clustering to Large Databases. Technical Report MSR-TR-98-35, Microsoft Research,
October 1999.
1424] P. Cheeseman, J. Kelly, M. Self, J. Stutz, W. Taylor, and D. Freeman. AutoClass:
a Bayesian classification system In Read'ings 'in knowled,ge acqu'isition and" learning:
automating the construction and improuement of erper"tsgstems,pages 431 441. Morgan
Kaufmann Publishers Inc., 1993.
[425] CLUTO 2.1.1: Software for Clustering High-Dimensional Datasets'
/www.cs.umn.edu/-161to1s, November 2003.
14261L. Ertoz, M. Steinbach, and V. Kumar. A New Shared Nearest Neighbor Clustering
Algorithm and its Applications. In Workshop on Clustering Hi,gh Dimensional Data
and,'its Appl'ications, Proc. of Tent Mine'}l, Fi,rst SIAM IntI. Conf. on Data M'in'ing,
Ch'icago, IL, USA, 2001.
14271L. Ertriz, M. Steinbach, and V. Kumar. Finding Clusters of Different Sizes, Shapes,
and Densities in Noisy, High Dimensional Data. In Proc. of the 2003 SIAM Intl. Conf.
on Data Min'ing, San FYancisco,May 2003. SIAM.
1428] D. Fisher and P. Langley. Conceptual clustering and its relation to numerical taxon-
olny. Artifi,cial Intelli,gence and Stati,sti,cs,pages 77-1L6, 1986.
14291 C. Ftaley and A. E. Raftery. How Many Clusters? Which Clustering Method? Answers
Via Model-Based Cluster Analysis. The Computer Journal,4l(8):578 588, 1998.
1430] V. Ganti, J. Gehrke, and R. Ramakrishnan. CACTUS Clustering Categorical Data
Using Summaries. In Proc. of the 5th Intl. Conf. on Knowled,ge Discouery and Data
Min'ing, pages 73 83. ACM Press, 1999.
[431] A. Gersho and R. M. Gray. Vector Quantizati,on and Si,gnal Cornpression,volume 159
of Kluwer Intemr,ational Series i,n Engr,neering and Computer Sc'ience. Kluwer Academic
Publishers, 1992.
1432] J. Ghosh. Scalable Clustering Methods for Data Mining. In N. Ye, editor, Handbook
of Data Mining, pages 247-277. Lawrence Ealbaum Assoc, 2003.
[433] D. Gibson, J. M. Kleinberg, and P. Raghavan. Clustering Categorical Data: An
Approach Based on Dynamical Systems. VLDB Journal' 8(3-4):222 236,2000.
1434] K. C. Gowda and G. Krishna. Agglomerative Clustering Using the Concept of Mutual
Nearest Neighborhoo d. P att ern Recognit'ion,10(2) :105-1 12, 1978.
[435] S. Guha, A. Meyerson, N. Mishra, R. Motwani, and L. O'Callaghan. Clustering Data
Streams: Theory and Practice. IEEE Transacti,ons on Knowled,ge and Data Eng'ineering,
I 5(3):515-528, May/June 2003.
1436] S. Guha, R. Rastogi, and K. Shim. CURE: An Efficient Clustering Algorithm for Large
Databases. In Proc. of 1998 ACM-SIGMOD Intl. Conf. on Management of Data, pages
73 84. ACM Press, June 1998.
[437] S. Guha, R. Rastogi, and K. Shim. ROCK: A Robust Clustering Algorithm for Cate-
gorical Attributes. In Proc. of the 15th Intl. Conf. on Data Engineering, pages 512 521.
IEEE Computer Society, March 1999.
[438] tr.-H. Han, G. Karypis, V. Kumar, and B. Mobasher. Hypergraph Based Clustering
in High-Dimensional Data Sets: A Summary of Results. IEEE Data Eng. Bulletin,2l
(1):15-22,1998.
646 Chapter 9 Cluster Analysis: Additional Issuesand Algorithms

1439] A. Hinneburg and D. A. Keim. An Efficient Approach to Clustering in Large Multi-

media Databases with Noise. In Proc. of the lth Intl. Conf. on Knowledge D'iscouerg
and Data M'ining, pages 58 65, New York City, August 1998. AAAI Press.
[440] A. Hinneburg and D. A. Keim. Optimal Grid-Clustering: Towards Breaking the Curse
of Dimensionality in High-Dimensional Clustering. In Proc. of the 25th VLDB Conf.,
pages 506 517, trdinburgh, Scotland, UK, September 1999. Morgan Kaufmann.
l44ll F. Hcippner, F. Klawonn, R. Kruse, and T. Runkler. Fuzzy Cluster Analysis: Methods
for Classi,fi,cat'ion,Data Analysi,s and Image Recogni,tion. John Wiley & Sons, New York,
Julv 2 1999.
14421R. A. Jarvis and E. A. Patrick. Clustering Using a Similarity Measure Based on Shared
Nearest Neighbors. IEEE Transactions on Computers, C-22(ll):1025 1034, 1973.
14431 I. Jonyer, D. J. Cook, and L. B. Holder. Graph-based hierarchical conceptual cluster-
ing. Journal of Machine Learning Research,2:19-43, 2002.
1444] K. Kailing, H.-P. Kriegel, and P. Kroger. Density-Connected Subspace Clustering for
High-Dimensional Data. In Proc. of the 2001 SIAM Intl. Conf. on Data Mini,ng, pages
428-439, Lake Buena Vista, Florida, April 2004. SIAM.
1445] G. Karypis, E.-H. Han, and V. Kumar CHAMELEON: A Hierarchical Clustering
Algorithm Using Dynamic Modeling. IEEE Computer,32(8):68-75, August 1999.
1446] G. Karypis and V. Kumar. Multilevel k-way Partitioning Scheme for Irregular Graphs.
Journal of Parallel and Di,stri,butedComputing, 48(1):96-129, 1998.
14471T. Kohonen, T. S. Huang, and M. R. Schroeder. Self-Organ'izing Maps. Springer-
Verlag, December 2000.
[448] R. S. Michalski and R. E. Stepp. Automated Construction of Classifications: Concep-
tual Clustering Versus Numerical Taxonomy. IEEE Transactions on Pattern Analyszs
and"M achine Intelli,gence, 5(4) :396 409, 1983.
[449] N. Mishra, D. Ron, and R. Swaminathan. A New Conceptual Clustering Framework.
M achine Learni,ng J ournal, 56( 1-3) : 1 15-1 5 1, July/August/September 2004.
f450] T. Mitchell. Mach'ine Learning. McGraw-Hill, Boston, MA, 1997.
[451] F. Murtagh. Clustering massive data sets. In J. Abello, P. M. Pardalos, and M. G. C.
Reisende, editors, Handbook of Massi,ueData Sets. Kluwer, 2000.
14521H. Nagesh, S. Goil, and A. Choudhary. Parallel Algorithms for Clustering High-
Dimensional Large-Scale Datasets. In R. L. Grossman, C. Kamath, P. Kegelmeyer,
V Kumar, and R. Namburu, editors, Data Mining for Sci,entific and Eng'ineering Appli-
cat'ions,pages 335 356. Kluwer Academic Publishers, Dordrecht, Netherlands, October
2001.
1453] R. T. Ng and J. Han. CLARANS: A Method for Clustering Objects for Spatial Data
Mining. IEEE Transactr,ons on Knowled.ge and Data Engineering, 14(5):1003-1016,
2002.
[454] M. Peters and M. J. Zaki. CLICKS: Clustering Categorical Data using K-partite
Maximal Cliques. ln Proc. of the 21st IntI. Conf. on Data Eng'ineering, Tokyo, Japan,
April 2005.
1455] E. Schikuta and M. Erhart. The BANG-Clustering System: Grid-Based Data Analy-
sis. In Aduances in Intelli,gent Data Analgsis, Reasoning about Data, Second Intl. Sym-
Ttos'ium, IDA-97, London, volume 1280 of Lecture Notes 'in Computer Sc'ience, pages
513 524. Springer, August 1997.
[456] G. Sheikholeslami, S. Chatterjee, and A. Zhang. Wavecluster: A multi-resolution
clustering approach for very large spatial databases. In Proc. of the 2lth VLDB Conf.,
pages 428 439, New York City, August 1998. Morgan Kaufmann.
 9.8 Exercises 647

[457] M. Steinbach, P.-N. Tan, V. Kumar, S. Klooster, and C. Potter. Discovery of climate
indices using clustering. In KDD '03: Proceed,ingsof the ninth ACM SIGKDD 'interna'
tional conference on Knowledge discouery and data mi'n'ing, pages 446-455, New York,
NY. USA. 2003. ACM Press.
[45S] R. E. Stepp and R. S. Michalski. Conceptual clustering of structured objects: A
goal-oriented approach. Arti,fici,al Intelligence, 28(1) :43-69, 1986
1459] A. Strehl and J. Ghosh. A Scalable Approach to Balanced, High-dimensional Cluster-
ing of Market-Baskets. In Proc. of the 7th Intl. Conf. on High Performance Computing
(H|PC 2000), vohtme 1970 of Lecture Notes in Computer Science, pages 525 536, Ban-
galore, India, December 2000. Springer.
[460] T. Zhang, R. Ramakrishnan, and M. Livny. BIRCH: an efficient data clustering method
for very large databases. In Proc. of 1996 ACM-SIGMOD Intl. Conf. on Management
of Data, pages 103-114, Montreal, Quebec, Canada, June 1996. ACM Press.

9.8 Exercises
1 . For sparse data, discuss why considering only the presence of non-zero values
might give a more accurate view of the objects than considering the actual
magnitudes of ralues. When would such an approach not be desirable?
2 . Describe the change in the time complexity of K-means as the number of clusters
to be found increases.
.). Consider a set of documents. Assume that all documents have been normalized
to have unit length of 1. What is the "shape" of a cluster that consists of all
documents whose cosine similarity to a centroid is greater than some specified
constant? In other words, cos(d,c) ) d, where 0 < d < 1.
4 . Discuss the advantages and disadvantages of treating clustering as an optimiza-
tion problem. Among other factors, consider efficiency, non-determinism, and
whether an optimization-based approach captures all types of clusterings that
are of interest.
K
What is the time and space complexity of fuzzy c-means? Of SOM? How do
these complexities compare to those of K-means?

6 . Tladitional K-means has a number of limitations, such as sensitivity to outliers

and difficulty in handling clusters of different sizes and densities, or with non-
globular shapes. Comment on the ability of fivzy c-means to handle these
situations.
7. For tlne fuzzy c-means algorithm described in this book, the sum of the mem-
bership degree of any point over all clusters is 1. Instead, we could only require
that the membership degree of a point in a cluster be between 0 and 1. What
are the advantages and disadvantages of such an approach?

8. Explain the difference between likelihood and probability.

648 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

** ,(
*

Figure9.28.DatasetforExercise
12,EMclustering pointsetwithtwoclusters
of a two-dimensional
ofdiffering
density.

a Equation 9.12 gives the likelihood for a set of points from a Gaussian distribu-
tion as a function of the mean p and the standard deviation o. Show math-
ematically that the maximum likelihood estimate of p and o are the sample
mean and the sample standard deviation, respectively.
10. We take a sample of adults and measure their heights. If we record the gender of
each person, we can calculate the average height and the variance of the height,
separately, for men and women. Suppose, however, that this information was
not recorded. Would it be possible to still obtain this information? Explain.
1 1 . Compare the membership weights and probabilities of Figures 9.1 and 9.4,
which come, respectively, from applying fuzzy and EM clustering to the same
set of data points. What differences do you detect, and how might you explain
these differences?
t 2 . Figure 9.28 shows a clustering of a two-dimensional point data set with two
clusters: The leftmost cluster, whose points are marked by asterisks, is some-
what diffuse, while the rightmost cluster, whose points are marked by circles, is
compact. To the right of the compact cluster, there is a single point (marked by
an arrow) that belongs to the diffuse cluster, whose center is farther away than
that of the compact cluster. Explain why this is possible with EM clustering,
but not K-means clustering.
 9.8 Exercises 649

13. Show that the MST clustering technique of Section 9.4.2 produces the same
clusters as single link. To avoid complications and special cases) assume that
all the pairwise similarities are distinct.
1 4 . One way to sparsify a proximity matrix is the following: For each object (row
in the matrix), set all entries to 0 except for those corresponding to the objects
k-nearest neighbors. However, the sparsified proximity matrix is typically not
symmetric.

(a) If object o is among the k-nearest neighbors of object b, why is b not

guaranteed to be among the k-nearest neighbors of a.?
(b) Suggest at least two approaches that could be used to make the sparsified
proximity matrix symmetric.

1 5 . Give an example of a set of clusters in which merging based on the closeness

of clusters Ieads to a more natural set of clusters than merging based on the
strength of connection (interconnectedness)of clusters.
1 6 . Table 9.4 lists the two nearest neighbors of four points.

Table
9,4.Twonearest offourpoints.
neighbors
Point First Neiehbor Second Neighbor
I A

2 J 4
a A 2
4 .) 1

Calculate the SNN similarity between each pair of points using the definition
of SNN similarity defined in Algorithm 9.10.
7 7 . For the definition of SNN similarity provided by Algorithm 9.10, the calculation
of SNN distance does not take into account the position of shared neighbors
in the two nearest neighbor lists. In other words, it might be desirable to give
higher similarity to two points that share the same nearest neighbors in the
same or roughly the same order.

(a) Describe how you might modify the definition of SNN similarity to give
higher similarity to points whose shared neighbors are in roughly the same
order.
(b) Discuss the advantages and disadvantages of such a modification.

1 8 Name at least one situation in which you would not want to use clustering
based on SNN similarity or density.
19. Grid-clustering techniques are different from other clustering techniques in that
they partition space instead of sets of points.
650 Chapter 9 Cluster Analysis: Additional Issues and Algorithms

(a) How does this affect such techniques in terms of the description of the
resulting clusters and the types of clusters that can be found?
(b) What kind of cluster can be found with grid-based clusters that cannot
be found by other types of clustering approaches? (Hint: See Exercise 20
in Chapter 8, page 564.)

20. In CLIQUE, the threshold used to find cluster density remains constant, even
as the number of dimensions increases.This is a potential problem since density
drops as dimensionality increases;i.e., to find clusters in higher dimensions the
threshold has to be set at a level that may well result in the merging of low-
dimensional clusters. Comment on whether you feel this is truly a problem and,
if so, how you might modify CLIQUE to address this problem.
2 r . Given a set of points in Euclidean space, which are being clustered using the
K-means algorithm with Euclidean distance, the triangle inequality can be used
in the assignment step to avoid calculating all the distances of each point to
each cluster centroid. Provide a general discussion of how this might work.
22. Instead of using the formula derived in CURE see Equation 9.19--we could
run a Monte Carlo simulation to directly estimate the probability that a sample
of size s would contain at least a certain fraction of the points from a cluster.
Using a Monte Carlo simulation compute the probability that a sample of size
s contains 50% of the elements of a cluster of size 100, where the total number
of points is 1000, and where s can take the values 100, 200, or 500.
 10

AnomalyDetection
In anomaly detection, the goal is to find objects that are different from most
other objects. Often, anomalous objects are known as outliers, since, on a
scatter plot of the data, they lie far away from other data points. Anomaly
detection is also known as deviation detection, becauseanomalous objects
have attribute values that deviate significantly from the expected or typical
attribute values, or as exception mining, becauseanomalies are exceptional
in some sense.In this chapter, we will mostly use the terms anomaly or outl'ier.
There are a variety of anomaly detection approaches from several areas,
including statistics, machine learning, and data mining. All try to capture the
idea that an anomalousdata object is unusual or in someway inconsistent with
other objects. Although unusual objects or events are, by definition, relatively
rare, this does not mean'that ttrey do not occur frequently in absolute terms.
For example, an event that is "one in a thousand" can occur millions of times
when billions of events are considered.
In the natural world, human society, or the domain of data sets, most
events and objects are, by definition, commonplace or ordinary. However, we
have a keen awarenessof the possibility of objects that are unusual or extraor-
dinary. This includes exceptionally dry or rainy seasons)famous athletes, or
an attribute value that is much smaller or larger than all others. Our inter-
est in anomalous events and objects stems from the fact that they are often
of unusual importance: A drought threatens crops, an athlete's exceptional
skill may lead to victory, and anomalous values in experimental results may
indicate either a problem with the experiment or a new phenomenon to be
investigated.
The following examples illus;trate applications for which anomalies are of
considerableinterest.
652 Chapter 10 Anomaly Detection

o Fbaud Detection. The purchasing behavior of someonewho steals a

credit card is probably different from that of the original owner. Credit
card companies attempt to detect a theft by looking for buying patterns
that characterize theft or by noticing a change from typical behavior.
Similar approachesare used for other types of fraud.

o Intrusion Detection. Unfortunately, attacks on computer systems

and computer networks are commonplace. While some of these attacks,
such as those designed to disable or overwhelm computers and networks,
are obvious, other attacks, such as those designed to secretly gather
information, are difficult to detect. Many of these intrusions can only be
detected by monitoring systems and networks for unusual behavior.

o Ecosystem Disturbances. In the natural world, there are atypical

events that can have a significant effect on human beings. Examples
include hurricanes, floods, droughts, heat waves, and fires. The goal is
often to predict the likelihood of these events and the causes of them.

o Public Health. In many countries, hospitals and medical clinics re-

port various statistics to national organizations for further analysis. For
example, if all children in a city are vaccinated for a particular disease,
e.g., measles,then the occurrenceofa few casesscattered acrossvarious
hospitals in a city is an anomalous event that may indicate a problem
with the vaccination programs in the city.

o Medicine. For a particular patient, unusual symptoms or test results

may indicate potential health problems. However, whether a particular
test result is anomalous may depend on other characteristics of the pa-
tient, such as age and sex. Furthermore, the categorization of a result
as anomalous or not incurs a cost-unneeded additional tests if a pa-
tient is healthy and potential harm to the patient if a condition is left
undiagnosed and untreated.

Although much of the recent interest in anomaly detection has been driven
by applications in which anomalies are the focus, historically, anomaly detec-
tion (and removal) has been viewed as a technique for improving the analysis
of typical data objects. For instance, a relatively small number of outliers can
distort the mean and standard deviation of a set of values or alter the set
of clusters produced by a clustering algorithm. Therefore, anomaly detection
(and removal) is often a part of data preprocessing.
 10.1 Preliminaries 653

In this chapter, we will focur;on anomaly detection. After a few preliminar-

ies, we provide a detailed discussionof some important approachesto anomaly
detection, illustrating them with examples of specific techniques.

10.1 Preliminaries
Before embarking on a discussionof specific anomaly detection algorithms, we
provide some additional background. Specifically,we (1) explore the causesof
anomalies, (2) consider various anomaly detection approaches, (3) draw dis-
tinctions among approachesbased on whether they use classlabel information,
and (4) describe issuescommorr to anomaly detection techniques.

10.1.1 Causes of Anoma.lies

The following are some common causes of anomalies: data from different
classes,natural variation, and rlata measurement or collection errors.

Data from Different Classes An object may be different from other ob-
jects, i.e., anomalous, because it is of a different type or class. To illustrate,
someone committing credit cald fraud belongs to a different class of credit
card users than those people v'ho use credit cards legitimately. Most of the
examples presented at the beginning of the chapter, namely, fraud, intrusion,
outbreaks of disease,and abnormal test results, are examplesof anomaliesthat
represent a different class of ob.iects. Such anomalies are often of considerable
interest and are the focus of anomaly detection in the field of data mining.
The idea that anomale6 6['jects come from a different source (class) than
most of the data objects is stal;ed in the often-quoted definition of an outlier
by the statistician Douglas Har'ikins.
Definition 10.1 (Hawkins' lDefinition of an Outlier). An outlier is an
observation that differs so much from other observations as to arouse susoicion
that it was generated by a diffe,rent mechanism.

Natural Variation Many daL,ta sets can be modeled by statistical distribu-

tions, such as a normal (Gaussian) distribution, where the probability of a
data object decreasesrapidly as the distance of the object from the center
of the distribution increases. In other words, most of the objects are near a
center (average object) and the likelihood that an object differs significantly
from this averageobject is small. For example, an exceptionally tall person is
not anomalous in the senseof t,eing from a separate class of objects, but only
654 Chapter 10 Anomaly Detection

in the senseof having an extreme value for a characteristic (height) possessed

by all the objects. Anomalies that represent extreme or unlikely variations are
often interesting.

Data Measurement and Collection Errors Errors in the data collection

or measurement process are another source of anomalies. For example, a
measurement may be recorded incorrectly becauseof human error) a problem
with the measuring device, or the presenceof noise. The goal is to eliminate
such anomalies, since they provide no interesting information but only reduce
the quality of the data and the subsequentdata analysis. Indeed, the removal
of this type of anomaly is the focus of data preprocessing, specifically data
cleaning.

Summary An anomaly may be a result of the causes given above or of

other causesthat we did not consider. Indeed, the anomalies in a data set
may have several sources,and the underlying causeof any particular anomaly
is often unknown. In practice, anomaly detection techniques focus on finding
objects that differ substantially from most other objects, and the techniques
themselves are not affected by the source of an anomaly. Thus, the under-
lying cause of the anomaly is only important with respect to the intended
application.

10.1.2 Approaches to Anomaly Detection

Here, we provide a high-level description of some anomaly detection tech-

niques and their associateddefinitions of an anomaly. There is some overlap
between these techniques, and relationships among them are explored further
in Exercise 1 on page 680.

Model-Based Techniques Many anomaly detection techniques first build

a model of the data. Anomalies are objects that do not fit the model very well.
For example, a model of the distribution of the data can be created by using
the data to estimate the parameters of a probability distribution. An object
does not fit the model very well; i.e., it is an anomaly, if it is not very likely
under the distribution. If the model is a set of clusters, then an anomaly is an
object that does not strongly belong to any cluster. When a regressionmodel
is used, an anomaly is an object that is relatively far from its predicted value.
Becauseanomalous and normal objects can be viewed as defining two dis-
tinct classes,classification techniques can be used for building models of these
 10.1 Preliminaries 655

two classes.Of course,classification techniques can only be used if classlabels

are available for some of the objects so that a training set can be constructed.
Also, anomalies are relatively rare, and this needs to be taken into account
when choosing both a classification technique and the measuresto be used for
evaluation. (SeeSection 5.7.)

In some cases, it is difficult to build a model; e.g., because the statistical

distribution of the data is unknown or no training data is available. In these
situations, techniques that do not require a model, such as those described
below, can be used.

Proximity-Based rechniques It is often possible to define a proximity

measure between objects, and a number of anomaly detection approachesare
based on proximities. Anomalous objects are those that are distant from most
of the other objects. Many of the techniquesin this area are basedon distances
and are referred to as distance-based outlier detection techniques. When
the data can be displayed as a two- or three-dimensionalscatter plot, distance-
basedoutliers can be detected visually, by looking for points that are separated
from most other points.

Density-Based Techniques Estimates of the density of objects are rela-

tively straightforward to compute, especially if a proximity measure between
objects is available. Objects that are in regions of low density are relatively
distant from their neighbors, and can be considered anomalous. A more so-
phisticated approach accommodates the fact that data sets can have regions
of widely differing densities, and classifiesa point as an outlier only if it has a
local density significantly less than that of most of its neighbors.

10.1.3 The Use of Class Labels

There are three basic approachesto anomaly detection: unsupervised, super-
vised' and semi-supervised.The major distinction is the degreeto which class
labels (anomaly or normal) are available for at least some of the data.

Supervised anomaly detection Techniquesfor supervised anomaly detec-

tion require the existence of a training set with both anomalous and
normal objects. (Note that there may be more than one normal or
anomalousclass.) As mentioned previously, classificationtechniquesthat
addressthe so-calledrare classproblem are particularly relevant because
656 Chapter 10 Anomaly Detection

anomalies are relatively rare with respect to normal objects. SeeSection

rn
r).1 .

IJnsupervised anornaly detection In many practical situations, class Ia-

bels are not available. In such cases,the objective is to assign a score (or
a label) to each instance that reflects the degreeto which the instance is
anomalous. Note that the presenceof many anomalies that are similar
to each other can causethem all to be labeled normal or have a low out-
Iier score. Thus, for unsupervised anomaly detection to be successful,
anomalies must be distinct from one another, as well as normal objects.

Semi-supervised anomaly detection Sometimestraining data contains la-

beled normal data, but has no information about the anomalousobjects.
In the semi-supervisedsetting, the objective is to find an anomaly label
or score for a set of given objects by using the information from labeled
normal objects. Note that in this case, the presence of many related
outliers in the set of objects to be scored does not impact the outlier
evaluation. However, in many practical situations, it can be difficult to
find a small set of representative normal objects.

AII anomaly detection schemesdescribed in this chapter can be used in

supervisedor unsupervisedmode. Supervisedschemesare essentiallythe same
as classification schemesfor rare classesdiscussedin Section 5.7.

LO.L.4 Issues

There are a variety of important issuesthat need to be addressedwhen dealing

with anomalies.

Number of Attributes Used to Define an Anomaly The question of

whether an object is anomalous based on a single attribute is a question of
whether the object's value for that attribute is anomalous. However, since an
object may have many attributes, it may have anomalous values for some at-
tributes, but ordinary values for other attributes. Furthermore, an object may
be anomalous even if none of its attribute values are individually anomalous.
For example, it is common to have people who are two feet tall (children) or
are 300 pounds in weight, but uncommon to have a two-foot tall person who
weighs 300 pounds. A general definition of an anomaly must specify how the
values of multiple attributes are used to determine whether or not an object
is an anomaly. This is a particularly important issue when the dimensionality
of the data is high.
 10.1 Preliminaries 657

Global versus Local Perspective An object may seem unusual with re-
spect to all objects, but not with respect to objects in its local neighborhood.
For example, a person whose height is 6 feet 5 inches is unusually tall with re-
spect to the generalpopulation, but not with respect to professionalbasketball
players.

Degree to Which a Point Is an Anomaly The assessmentof whether an

object is an anomaly is reported by some techniques in a binary fashion: An
object is either an anomaly or it is not. Frequently, this does not reflect the
underlying reality that some objects are more extreme anomaliesthan others.
Hence, it is desirable to have some assessmentof the degreeto which an object
is anomalous. This assessmentis known as the anomalv or outlier score.

Identifying One Anomaly at a Time versus Many Anomalies at Once

In some techniques,anomaliesare removed one at a time; i.e., the most anoma-
lous instance is identified and removed and then the processrepeats. For other
techniques, a collection of anomalies is identified together. Techniques that
attempt to identify one anomaly at a time are often subject to a problem
known as masking, where the presenceof several anomalies masks the pres-
ence of all. On the other hand, techniquesthat detect multiple outliers at once
can experienceswamping, where normal objects are classifiedas outliers. In
model-basedapproaches,these effects can happen becausethe anomalies dis-
tort the data model.

Evaluation If class labels are available to identify anomalies and normal

data, then the effectivenessof an anomaly detection schemecan be evaluated
by using measuresof classification performance discussedin Section 5.7. But
since the anomalous class is usually much smaller than the normal class, mea-
sures such as precision, recall, and false positive rate are more appropriate
than accuracy. If class labels are not available, then evaluation is difficult.
However, for model-based approaches, the effectivenessof outlier detection
can be judged with respect to the improvement in the model once anomalies
are eliminated.

Efficiency There are significant differences in the computational cost of var-

ious anomaly detection schemes.Classification-basedschemescan require sig-
nificant resourcesto create the classificationmodel, but are usually inexpensive
to apply. Likewise, statistical approaches create a statistical model and can
658 Chapter 10 Anomaly Detection

then categorize an object in constant time. Proximity-based approachesnat-

urally have a time complexity of O(*'), where rn is the number of objects,
because the information they require can usually only be obtained by com-
puting the proximity matrix. This time complexity can be reduced in specific
caseslsuch as low-dimensional data, by the use of special data structure and
algorithms. The time complexity of other approachesis consideredin Exercise
6 on page 681.

Road Map

The next four sections describe several major categoriesof anomaly detection
approaches: statistical, proximity-based, density-based, and cluster-based.
One or more specific techniques are consideredwithin each of these categories.
In these sections, we will follow common practice and use the term outlier
instead of anomaly.

1-O.2 Statistical Approaches

Statistical approaches are model-based approaches; i.e., a model is created
for the data, and objects are evaluated with respect to how well they fit the
model. Most statistical approachesto outlier detection are based on building
a probability distribution model and considering how Iikely objects are under
that model. This idea is expressedby Definition 10.2.
Definition 10.2 (Probabilistic Definition of an Outlier). An outlier is
an object that has a low probability with respect to a probability distribution
model of the data.
A probability distribution model is created from the data by estimating the
parameters of a user-specifieddistribution. If the data is assumed to have a
Gaussiandistribution, then the mean and standard deviation of the underlying
distribution can be estimated by computing the mean and standard deviation
of the data. The probability of each object under the distribution can then be
estimated.
A wide variety of statistical tests basedon Definitiont0.2 have been devised
to detect outliers, or discordant observations, as they are often called in the
statistical literature. Many of these discordancy tests are highly specialized
and assumea level of statistical knowledgebeyond the scopeof this text. Thus,
we illustrate the basic ideas with a few examples and refer the reader to the
bibliographic notes for further pointers.
 LO.2 Statistical Approaches 659

Issues

Among the important issuesfacing this approach to outlier detection are the
following:

Identifying the specific distribution of a data set. While many types

of data can be described by a small number of common distributions, such
as Gaussian, Poisson, or binomial, data sets with non-standard distributions
are relatively common. Of course, if the wrong model is chosen, then an
object can be erroneously identified as an outlier. For example, the data
may be modeled as coming from a Gaussian distribution, but may actually
come from a distribution that has a higher probability (than the Gaussian
distribution) of having values far from the mean. Statistical distributions with
this type of behavior are common in practice and are known as heavy-tailed
distributions.

The number of attributes used. Most statistical outlier detection tech-

niques apply to a single attribute, but some techniques have been defined for
multivariate data.

Mixtures of distributions. The data can be modeled as a mixture of distri-

butions, and outlier detection schemescan be developedbased on such models.
Although potentially more powerful, such models are more complicated, both
to understand and to use. For example, the distributions need to be identi-
fied before objects can be classified as outliers. See the discussion of mixture
models and the EM algorithm in Section 9.2.2.

10.2.1 Detecting Outliers in a Univariate Normal Distribution

The Gaussian (normal) distribution is one of the most frequently used dis-
tributions in statistics, and we will use it to describe a simple approach to
statistical outlier detection. This distribution has two parameters, p and o,
which are the mean and standard deviation, respectively, and is represented
using the notation N(p,o). Figure 10.1 showsthe density function of l[(0,1).
There is little chancethat an object (value) from a N(0, 1) distribution will
occur in the tails of the distribution. For instance, there is only a probability
of 0.0027 that an object lies beyond the central area between t3 standard
deviations. More generally, if c is a constant and r is the attribute value of
an object, then the probability that lzl ) c decreasesrapidly as c increases.
Let a : prob(lrl > c). Table 10.1 shows some sample values for c and the
660 Chapter 10 Anomaly Detection

o4

) 0.25
q
c
o

> 0.2

G
!
g 0.15
(!

-1 012345
x

Figure10,1. Probability function

density of a Gaussian witha meanof 0 anda standard
distribution
deviation
of 1.

corresponding values for a when the distribution is l/(0,1). Note that a value
that is more than 4 standard deviations from the mean is a one-in-ten-thousand
occurrence.

Table10.1, Sample pairs(c,a), a : prob(lrl > c) fora Gaussian withmean0 and

distribution
standard 1.
deviation
c a for N(0,1)
1.00 0.3173
1.50 0.1336
2.00 0.0455
2.50 0.0124
3.00 0.0027
3.50 0.0005
4.00 0.0001

Becausea value's distance c from the center of the N(0,1) distribution is

directly related to the value's probability, it can be used as the basis of a test
for whether an object (value) is an outlier as indicated in Definition 10.3.
 LO.2 Statistical Approaches 661

Definition 10.3 (Outlier for a Single N(0,1) Gaussian Attribute). At

object with attribute value r from a Gaussian distribution with mean of 0 and
standard deviation 1 is an outlier if

lrl > ", (10.1)

where c is a constant chosenso that prob(lrl) ) c: e.

To use this definition it is necessaryto specify a value for a. From the
viewpoint that unusual values (objects) indicate a value from a different dis-
tribution, c indicates the probability that we mistakenly classify a value from
the given distribution as an outlier. From the viewpoint that an outlier is a
rare value of a l/(0,1) distribution, a specifiesthe degree of rareness.
If the distribution of an attribute of interest (for the normal objects) has a
Gaussiandistribution with mean p and a standard deviation o, i.e., a N(p,,o)
distribution, then to use Definition 10.3, we need to transform the attribute
r to a new attribute z, which has a ,A/(0,1) distribution. In particular, the
approach is to set z: (r - p)lo. (z is typically called a z score.) However, pr
and o are typically unknown and are estimated using the sample mean 7 and
sample standard deviation s". In practice, this works well when the number
of observations is large. However, we note that the distribution of z is not
actually N(0,1). A more sophisticatedstatistical procedure (Grubbs' test) is
explored in Exercise 7 on page 681.

LO.2.2 Outliers in a Multivariate Normal Distribution

For multivariate Gaussian observations, we would like to take an approach

similar to that given for a univariate Gaussian distribution. In particular,
we would like to classify points as outliers if they have low probability with
respect to the estimated distribution of the data. Furthermore? we would like
to be able to judge this with a simple test, for example, the distance of a point
from the center of the distribution.
However, because of the correlation between the different variables (at-
tributes), a rnultivariate normal distribution is not symmetrical with respect
to its center. Figure 10.2 shows the probability density of a two-dimensional
multivariate Gaussiandistribution with mean of (0,0) and a covariancematrix
of

\' - / 1'oo
"- \ o.zs 3r3)
662 Chapter 1O Anomaly Detection

If we are to use a simple threshold for whether an object is an outlier, then

we will need a distance measure that takes the shape of the data distribution
into account. The Mahalanobis distance is such a distance. SeeEquation 2.14
on page 81. The Mahalanobis distance between a point x and the mean of the
data x is shown in Equation 10.2.

x) : (x - x)S*1(x - *)t,
mahalanobzs(x, (10.2)

where S is the covariancematrix of the data.

It is easy to show that the Mahalanobis distance of a point to the mean of
the underlying distribution is directly related to the probability of the point.
In particular, the Mahalanobis distance is equal to the log of the probability
density of the point plus a constant. See Exercise 9 on page 682.
Example 10.1 (Outliers in a Multivariate Normal Distribution). Fig-
ure 10.3 shows the Mahalanobis distance (from the mean of the distribution)
for points in a two-dimensional data set. The points A (-4,4) and B (5, 5)
are outliers that were added to the data set, and their Mahalanobis distance is
indicated in the figure. The other 2000 points of the data set were randomly
generated using the distribution used for Figure 10.2.
Both A and B have large Mahalanobis distances. However, even though A
is closer to the center (the large black x at (0,0)) as measuredby Euclidean dis-
tance, it is farther away than B in terms of the Mahalanobis distance because
the Mahalanobis distance takes the shape of the distribution into account.
In particular, point B has a Euclidean distance of 5t/2 and a Mahalanobis
distance of 24, while the point A has a Euclidean distance of 4Jd and a Ma-
halanobis distance of 35. I

10.2.3 A Mixture Model Approach for Anomaly Detection

This section presents an anomaly detection technique that uses a mixture

model approach. In clustering (seeSection 9.2.2), the mixture model approach
assumesthat the data comes from a mixture of probability distributions and
that each cluster can be identified with one of these distributions. Similarly,
for anomaly detection, the data is modeled as a mixture of two distributions,
one for ordinary data and one for outliers.
For both clustering and anomaly detection, the goal is to estimate the
parameters of the distributions in order to maximize the overall likelihood
(probability) of the data. In clustering, the EM algorithm is used to esti-
mate the parameters of each probability distribution. However, the anomaly
detection technique presented here uses a simpler approach. Initially, all the
 IO.2 Statistical Approaches 663

Figure10,2.Probability of pointsfortheGaussian
density usedto generate
distribution thepointsof
Figure
10.3.

B
@ 24'

A
E
IFt-

lcq @
@ o- e@
p@
%^@
Ftf ^ @@
@
su
@

***

4-3-2-1012345 Mahalanobis
x Distance

Figure10,3.Mahalanobis of points
distance fromthecenter setof 2002points.
of a two-dimensional
664 Chapter 10 Anomaly Detection

objects are put in a set of normal objects and the set of anomalous objects
is empty. An iterative procedure then transfers objects from the ordinary set
to the anomalous set as long as the transfer increasesthe overall likelihood of
the data.
Assume that the data set D contains objects from a mixture of two prob-
ability distributions: M, the distribution of the majority of (normal) objects,
and A, the distribution of anomalous objects. The overall probability distri-
bution of the data can be written as

D (x) : (1- ) ) M( x) +.laix;. (10.3)

where x is an object and ,\ is a number between 0 and 1 that gives the expected
fraction of outliers. The distribrtion M is estimated from the data, while the
distribution A is often taken to be uniform. Let Mt and A1 be the set of
normal and anomalous objects, respectively,at time t. Initially, at time f : 0,
M0 : D and As is empty. At an arbitrary time l, the likelihood and log
likelihood of the entire data set D are given by the following two equations,
respectively:

L,(D) II po(xe):
xi eD
-
(,t ,l),-,,*,ll. e,,i*,)){ro.a)
r,,,r*,))(^,^,,_,U,

Lh(D) lMltog(I - ^) + logPv, (*u)+ lA| Iog.\* ! logPa,(x;) (to.s)

!-
x,€Mt xi€47

where Po, Pnr, and P4, are the probability distribution functions for D, M1
and A1, respectively. This equation can be derived from the general definition
of a mixture model given in Equation 9.6 (Section 9.2.2). To do so, it is
necessaryto make the simplifying assumption that the probability is 0 for
both of the following situations: (1) an object in ,4 is a normal object, and (2)
an object in M is an outlier. Algorithm 10.1 gives the details.
Becausethe number of normal objects is large compared to the number of
anomalies, the distribution of the normal objects may not change much when
an object is moved to the set of anomalies. In that case, the contribution of
each normal object to the overall likelihood of the normal objects will remain
relatively constant. Furthermore, if a uniform distribution is assumed for
anomalies,then each object moved to the set of anomalies contributes a fixed
amount to the likelihood of the anomalies. Thus, the overall change in the
total likelihood of the data when an object is moved to the set of anomalies
is roughly equal to the probability of the object under a uniform distribution
 LO.2 Statistical Approaches 665

Algorithm 10.1 Likelihood-based outlier detection.

1: Initialization: At time t : 0, let Mr contain all the objects, while A1 is empty.
Let LL1(D) : LL(MI) + LL(AI) be the log likelihood of all the data.
2: for each point x that belongs to M1 do
3: Move x from M1 to A7 to produce the new data sets A1a1 and M7a1.
4: Compute the new log likelihood of D, LLt+r(D) : LL(Mt+r) + LL(At+r)
5: Compute the difference, A : LL^D) - Lh+r(D)
6: if A > c, where c is some threshold then
7: x is classified as an anomaly, i.e., M11y and ,4111 are left unchanged and
become the current normal and anomaly sets.
8: end if
9: end for

(weighted by ,\) minus the probability of the object under the distribution of
the normal data points (weighted by 1- )). Consequently,the set of anomalies
will tend to consist of those objects that have significantly higher probability
under a uniform distribution rather than under the distribution of the normal
objects.
In the situation just discussed,the approach described by Algorithm 10.1
is roughly equivalent to classifying objects with a low probability under the
distribution of normal objects as outliers. For example, when applied to the
points in Figure 10.3, this technique would classify points A and B (and other
points far from the mean) as outliers. However, if the distribution of the nor-
mal objects changessignificantly as anomalies are removed or the distribution
of the anomalies can be modeled in a more sophisticated manner, then the
results produced by this approach will be different than the results of simply
classifying low-probability objects as outliers. Also, this approach can work
even when the distribution of obiects is multimodal.

IO.2.4 Strengths and Weaknesses

Statistical approaches to outlier detection have a firm foundation and build

on standard statistical techniques, such as estimating the parameters of a
distribution. When there is sufficient knowledge of the data and the type
of test that should be applied these tests can be very effective. There are a
wide variety of statistical outliers tests for single attributes. Fewer options
are available for multivariate data, and these tests can perform poorly for
high-dimensional data.
666 Chapter 10 Anomaly Detection

10.3 Proximity-Based Outlier Detection

Although there are several variations on the idea of proximity-based anomaly
detection, the basic notion is straightforward. An object is an anomaly if it
is distant from most points. This approach is more general and more easily
applied than statistical approaches,since it is easierto determine a meaningful
proximity measure for a data set than to determine its statistical distribution.
One of the simplest ways to measure whether an object is distant from
most points is to use the distance to the k-nearest neighbor. This is captured
by Definition 10.4. The lowest value of the outlier score is 0, while the highest
value is the maximum possible value of the distance function-usually infinity.
Definition 10.4 (Distance to k-Nearest Neighbor). The outlier score of
an object is given by the distance to its k-nearest neighbor.
Figure 10.4 shows a set of two-dimensional points. The shading of each
point indicates its outlier score using a value of k : 5. Note that outlying
point C has been correctly assigneda high outlier score.
The outlier score can be highly sensitive to the value of k. If k is too
small, e.8., 1, then a small number of nearby outliers can cause a low outlier
score. For example, Figure 10.5 showsa set of two-dimensional points in which
another point is close to C. The shading reflects the outlier score using a value
of k : 1. Note that both C and its neighbor have a low outlier score. If k is
too large, then it is possible for all objects in a cluster that has fewer objects
than k to become outliers. For example, Figure 10.6 shows a two-dimensional
data set that has a natural cluster of size 5 in addition to a larger cluster of
size 30. For k : 5, the outlier score of all points in the smaller cluster is very
high. To make the schememore robust to the choice of k, Definition 10.4 can
be modified to use the averageofthe distancesto the first k-nearest neighbors.

10.3.1 Strengths and Weaknesses

The distance-basedoutlier detection scheme described above, and other re-

lated schemes, are simple. However, proximity-based approaches typically
take O(m2) time. For large data sets this can be too expensive, although
specializedalgorithms can be used to improve performance in the caseof low-
dimensional data. Also, the approach is sensitive to the choice of parameters.
Furthermore, it cannot handle data sets with regions of widely differing den-
sities because it uses global thresholds that cannot take into account such
density variations.
 10.3 Proximity-Based Outlier Detection 667

a @
@

6n @.

@ o I @ @ B
@oo OO 6@
o^^ ooo @oo
^
o t-J &
6
@o B
o
@
o a 6
6 o
@o a@

Outlier Outlier
Score Score

Figure10,4. Outlier
scorebasedon the Figure 10.5.Outlierscore
based
onthedis-
distance
tofifthnearest
neiohbor. tancetothefirstnearest
neighbor.
Nearby
out-
liershavelowoutlier
scores.
.c
a c
a
a
ao

@@ @@B
& @6 @ @@

@ oE
6@e oO
o ^w^ ^ o
o08
6
o
6
@@

Outlier Outlier
Score Score

Figure 10.6.Outlier
score
basedondistance Figure10.7.Outlier
scorebasedonthedis-
to thefifthnearest
neighbor.
A smallcluster fifth
tancetothe nearest neiohbor.
Clusters
of
becomes anoutlier. differing
density.

To illustrate this, consider the set of two-dimensional points in Figure

10.7. This figure has one rather loose cluster of points, another dense cluster
of points, and two points, C and D, that are quite far from these two clusters.
Assigning the outlier score to points according to Definition 10.4 for k : 5,
correctly identifies point C to be an outlier, but shows a low outlier score for
668 Chapter 10 Anomaly Detection

point D. In fact, the outlier score for D is much lower than many points that
are oart of the loose cluster.

LO.4 Density-Based Outlier Detection

From a density-basedviewpoint, outliers are objects that are in regions of low
density.
Definition 10.5 (Density-Based Outlier). The outlier scoreof an object
is the inverse of the density around an object.
Density-basedoutlier detection is closely related to proximity-based outlier
detection since density is usually defined in terms of proximity. One common
approach is to define density as the reciprocal of the averagedistance to the k
nearest neighbors. If this distance is small, the density is high, and vice versa.
This is captured by Definition 10.6.
Definition 10.6 (Inverse Distance).

:(ffi)-'
d,ensi,ty(x,-l (10.6)

where l/(x, k) is the set containing the k-nearest neighborsof x, ll/(x, k)l is
the size of that set, and y is a nearest neighbor.
Another definition of density is the one used by the DBSCAN clustering
algorithm. SeeSection 8.4.
Definition 10.7 (Count of Points within a Given Radius). The density
around an object is equal to the number of objects that are within a specified
distance d of the object.
The parameter d needsto be chosencarefully. If d is too small, then many
normal points may have low density and thus a high outlier score. If d is
chosento be large, then many outliers may have densities (and outlier scores)
that are similar to normal points.
Detecting outliers using any of the definitions of density has similar char-
acteristics and limitations to those of the proximity-based outlier schemes
discussedin Section 10.3. In particular, they cannot identify outliers correctly
when the data contains regions of differing densities. (See Figure 10.7.) To
correctly identify outliers in such data sets, we need a notion of density that
is relative to the neighborhood of the object. For example, point D in Figure
 LO.4 Density-Based Outlier Detection 669

10.7 has a higher absolute density, according to Definitions 10.6 and 10.7, than
point A, but its density is lower relative to its nearest neighbors.
There are many ways to define the relative density of an object. One
method that is used by the SNN density-basedclustering algorithm is discussed
in Section 9.4.8. Another method is to compute the relative density as the ratio
of the density of a point x and the averagedensity of its nearest neighbors y
as follows:

density(x, k)
&ueragerelat'iue densi,ty(x,k) : (10.7)
k) llN (x' k)l'
Dyen(*,r)densi'tv(v,

10.4.1 Detection of Outliers Using Relative Density

In this section, we describe a technique that is based on the notion of relative

density. This technique, which is a simplified version of the Local Outlier
Factor (LOF) technique (see bibliographic notes), is described in Algorithm
10.2. The details of the algorithm are examined in more detail below, but in
summary, it works as follows. We calculate the outlier score for each object
for a specified number of neighbors (k) by first computing the density of an
object density(x,k) based on its nearest neighbors. The average density of
the neighbors of a point is then calculated and used to compute the average
relative density of the point as indicated in Equation 10.7. This quantity
provides an indication of whether x is in a denser or sparser region of the
neighborhood than its neighbors and is taken as the outlier score of x.

Algorithm 10.2 Relative density outlier score algorithm.

1: {k is the number of riearest neighbors}
2: for all objects x do
3: Determine l[(x, k), the k-nearest neighbors of x.
4: Determine dens'ity(x,k), the density of x usinlg its nearest neighbors, i.e., the
objects in l[(x, k).
5 : end for
o : for all objects x do
Set the outli.er score(x,k) : ou"roge relat'iae dens'ity(x,k) from Equation
r0.7.
8: end for

Example 10.2 (Relative Density Outlier Detection). We illustrate the

performance of the relative density outlier detection method by using the ex-
ample data set shown in Figure 10.7. Here, k : 10. The outlier scores for
670 Chapter 10 Anomaly Detection

o.uc

oo
On
t

B
o

Figure10.8.Relative (LOF)outlier
density scores points
fortwo-dimensional of Figure
10.7.

these points are shown in Figure 10.8. The shading of each point is determined
by its score; i.e., points with a high score are darker. We have labeled points
A, C, and D, which have the largest outlier scores,with these values. Respec-
tively, these points are the most extreme outlier, the most extreme point with
respect to the compact set of points, and the most extreme point in the loose
set of points.

LO.4.2 Strengths and Weaknesses

Outlier detection based on relative density gives a quantitative measureof the

degree to which an object is an outlier and can work well even if data has
regions of differing density. Like distance-basedapproaches,these approaches
naturally have O(m2) time complexity (where rn is the number of objects),
although this can be reduced to O(rnlog rn) for low-dimensional data by using
special data structures. Parameter selection can also be difficult, although the
standard LOF algorithm addressesthis by looking at a variety of values for k
and then taking the maximum outlier scores. However, the upper and lower
bounds of these values still need to be chosen.
 10.5 Clustering-Based Techniques 67L

10.5 Clustering-Based Techniques

Cluster analysis finds groups of strongly related objects, while anomaly detec-
tion finds objects that are not strongly related to other objects. It should not
be surprising, then, that clustering can be used for outlier detection. In this
section, we will discuss several such techniques.
One approach to using clustering for outlier detection is to discard small
clusters that are far from other clusters. This approach can be used with any
clustering technique, but requires thresholds for the minimum cluster size and
the distance between a small cluster and other clusters. Often, the processis
simplified by discarding all clusters smaller than a minimum size. This scheme
is highly sensitive to the number of clusters chosen. AIso, it is hard to attach
an outlier score to objects using this scheme. Note that considering groups
of objects as outliers extends the notion of outliers from individual objects to
groups of objects, but does not change anything essential.
A more systematic approach is to first cluster all objects and then assess
the degree to which an object belongs to any cluster. For prototype-based
clustering, the distance of an object to its cluster center can be used to mea-
sure the degree to which the object belongs to a cluster. More generally, for
clustering techniques that are based on an objective function, we can use the
objective function to assesshow well an object belongs to any cluster. In par-
ticular, if the elimination of an object results in a substantial improvement in
the objective, then we would classify the object as an outlier. To illustrate,
for K-means, eliminating an object that is far from the center of its associated
cluster can substantially improve the sum of the squared error (SSE) of the
cluster. In summary, clustering creates a model of the data and anomalies
distort that model. This idea is captured in Definition 10.8.
Definition 10.8 (Clustering-Based Outlier). An object is a cluster-based
outlier if the object does not strongly belong to any cluster.
When used with clustering schemesthat have an objective function, this
definition is a special case of the definition of a model-based anomaly. At-
though Definition 10.8 is more natural for prototype-based schemesor schemes
that have an objective function, it can also encompassdensity- and connectivity-
based clustering approaches to outlier detection. In particular, for density-
basedclustering, an object does not strongly belong to any cluster ifits density
is too low, while for connectivity-based clustering, an object does not strongly
belong to any cluster if it is not strongly connected.
672 Chapter 10 Anomaly Detection

Below, we will discuss issuesthat need to be addressedby any technique

for clustering-basedoutlier detection. Our discussionwill focus on prototype-
based clustering techniques, such as K-means.

10.5.1 Assessing the Extent to Which an Object Belongs to a

Cluster

For prototype-based clusters, there are several ways to assessthe extent to

which an object belongs to a cluster. One method is to measure the distance
of the object to the cluster prototype and take this as the outlier score of
the object. However, if the clusters are of differing densities, then we can
construct an outlier score that measures the relative distance of an object
from the cluster prototype with respect to the distances of the other objects
in the cluster. Another possibility, provided that the clusters can be accurately
modeled in terms of Gaussiandistributions, is to use the Mahalanobis distance.
For clustering techniques that have an objective function, we can assign
an outlier score to an object that reflects the improvement in the objective
function when that obiect is eliminated. However, assessingthe degree to
which a point is an outlier based on the objective function can be compu-
tationally intensive. For that reason) the distance-basedapproaches of the
previous paragraph are often preferred.

Example 10.3 (Clustering-Based Example). This example is based on

the set of points shown in Figure 10.7. Prototype-based clustering uses the
K-means algorithm, and the outlier score of a point is computed in two ways:
(t) bV the point's distance from its closest centroid, and (2) by the point's
relative distance from its closest centroid, where the relative distance is the
ratio of the point's distance from the centroid to the median distance of all
points in the cluster from the centroid. The latter approach is used to adjust
for the large difference in density between the compact and loose clusters.
The resulting outlier scoresare shown in Figures 10.9 and 10.10. As before,
the outlier score, measured in this case by the distance or relative distance,
is indicated by the shading. We use two clusters in each case. The approach
based on raw distance has problems with the differing densities of the clusters,
e.8., D is not considered an outlier. For the approach based on relative dis-
tances, the points that have previously been identified as outliers using LOF
(A, C, and D) also show up as outliers here. r
 10.5 Clustering-Based Techniques 673

4.6
a

3.5

@@
2.5
Oa

I
1.5

B
a 1

05

Distance

Figure ofpoints
10.9.Distance fromclosest
centroid.

769 70
o
60

50

40

30

6
f:) 20
x

10

BiJXffi
10.10.Relative
Figure ofpoints
distance fromclosest
centroid.
674 Chapter 10 Anomaly Detection

LO.5.2 Impact of Outliers on the Initial Clustering

If outliers are detected by clustering, there is a question of whether the results

are valid since outliers affect the clustering. To addressthis issue,the fbllowing
approach can be used: objects are clustered, outliers are removed, and then
the objects are clustered again. While there is no guarantee that this approach
will yield optimal results, it is easy to use. A more sophisticated approach is
to have a special group for objects that do not currently fit well in any cluster.
This group represents potential outliers. As the clustering process proceeds,
clusters change. Objects that no longer belong strongly to any cluster are
added to the set of potential outliers, while objects currently in the set are
tested to seeif they now strongly belong to a cluster and can be removed from
the set of potential outliers. The objects remaining in the set at the end of
the clustering are classified as outliers. Again, there is no guarantee of an
optimal solution or even that this approach will work better than the simpler
one described previously. For example, a cluster of noise points may look like a
real cluster with no outliers. This problem is particularly serious if the outlier
score is computed using the relative distance.

10.5.3 The Number of Clusters to IJse

Clustering techniques such as K-means do not automatically determine the

number of clusters. This is a problem when using clustering in outlier detec-
tion, since whether an object is considered an outlier or not may depend on
the number of clusters. For instance, a group of 10 objects may be relatively
close to one another, but may be included as part of a larger cluster if only
a few large clusters are found. In that case, each of the 10 points could be
regarded as an outlier, even though they would have formed a cluster if a large
enough number of clusters had been specified.
As with some of the other issues,there is no simple answer to this problem.
One strategy is to repeat the analysis for different numbers of clusters. Another
approach is to find a large number of small clusters. The idea here is that (1)
smaller clusters tend to be more cohesive and (2) if an object is an outlier
even when there are a large number of small clusters, then it is likely a true
outlier. The downside is that groups of outliers may form small clusters and
thus escapedetection.

LO.5.4 Strengths and Weaknesses

Some clustering techniques, such as K-means, have linear or near-linear time

and space complexity and thus, an outlier detection technique based on such
 10.6 Bibliographic Notes 675

. the definition of a cluster is often com-

plementary to that of an outlier and thus, it is usually possible to find both
clusters and outliers at the same time. On the negative side, the set of outliers
proclucedand their scorescan be heavily dependent upon the number of clus-
ters used as well as the plesence of outliers in the data. For example, clusters
produced by prototype-based algorithms can be distorted by the presenceof
outliers. The quality of outliers produced by a clustering algorithm is heavily
impacted by the quality of clusters produced by the algorithm. As discussed
in Chapters 8 and 9, each clustering algorithm is suitable only for a certain
type of d.ata; hence the clustering algorithm needs to be chosen carefully.

10.6 Bibliographic Notes

Anomaly detection has a long history, particularly in statistics, where it is
known as outlier detection. Relevant books on the topic are those of Barnett
and Lewis [464], Hawkins [483], and Rousseeuwand Leroy [513]. The article
by Beckman and Cook [466] provides a general overview of how statisticians
look at the subject of outlier detection and provides a history of the subject
dating back to comments by Bernoulli rn 1777. Also see the related articles
is the one by Barnett
1467,4841.Another general article on outlier detection
data include those by Davies
[+0f1. erti"les on finding outliers in multivariate
and Gather 1474),Gnanadesikanand Kettenring 1480],Rocke and woodruff
Scott [516]. Rosner [512]
[tr11],Rousr""n* and van Zomerenand 1515],and
provides a discussionof finding multiple outliers at the same time.
An extensivesurvey of outlier detection methods is provided by Hodge and
Austin [486]. Markou and singh [506, 507] give a two-part review of techniques
for novelty detection that covers statistical and neural network techniques,
respectively. Grubbs' procedure for detecting outliers was originally described
in la81]. Thg mixture model outlier approach discussedin section 10-2'3 is
from Eskin [476]. The notion of a distance-basedoutlier and the fact that this
definition can include many statistical definitions of an outlier was described
by Knorr et al. 1496-498]. The LoF technique (Breunig et al. [468, 469])
grew out of DBSCAN. Ramaswamy et al. 1510]propose a distance-based
outlier detection procedure that gives each object an outlier score based on
the distance of its k-nearest neighbor. Efficiency is achieved by partitioning
the data using the first phase of BIRCH (Section 9.5.2). Chaudhary et al.
detection, while Bay and
[470] use k-d trees to improve the efficiency of outlier
schwabacher [465] use randomization and pruning to improve performance.
Aggarwal and Yu [462] use projection to address outlier detection for high-
676 Chapter 1O Anomaly Detection

dimensional data, while Shyu et al. [518] use an approach based on principal
components. A theoretical discussion of outlier removal in high-dimensional
space can be found in the paper by Dunagan and Vempala [475]. The use
of information measuresin anomaly detection is described by Lee and Xiang
1504],while an approach based on the 12 measure is given by Ye and Chen
[520].
Many different types of classification techniques can be used for anomaly
detection. A discussion of approachesin the area of neural networks can be
found in papers by Hawkins et al. [485], Ghosh and Schwartzbard 1479],and
Sykacek [519]. Recent work on rare class detection includes the work of Joshi
et al. [490-494]. The rare class problem is also sometimes referred to as the
imbalanced data set problem. Of relevanceare an AAAI workshop (Japkowicz
1488]),an ICML workshop (Chawla et al. [a71]),and a specialissueof SIGKDD
Explorations (Chawla et al. la72l).
Clustering and anomaly detection have a long relationship. In Chapters 8
and 9, we considered techniques, such as BIRCH, CURE, DENCLUE, DB-
SCAN, and SNN density-based clustering, which specifically include tech-
niques for handling anomalies. Statistical approaches that discuss this re-
lationship are described in papers by Scott [516] and Hardin and Rocke [482].
In this chapter, we have focused on basic anomaly detection schemes.We
have not considered schemesthat take into account the spatial or temporal
nature of the data. Shekhar et al. [517] provide a detailed discussion of the
problem of spatial outliers and present a unified approach to spatial outlier
detection. The issue of outliers in time series was first considered in a sta-
tistically rigorous way by Fox [478]. Muirhead [508] provides a discussion of
different types of outliers in time series. Abraham and Chuang [ 6t] propose a
Bayesian approach to outliers in time series,while Chen and Liu [473] consider
different types of outliers in time seriesand propose a technique to detect them
and obtain good estimates of time seriesparameters. Work on finding deviant
or surprising patterns in time seriesdatabaseshas been performed by Jagadish
et aI. [487] and Keogh et al. [495]. Outlier detection based on geometric ideas,
such as the depth of convex hulls, has been explored in papers by Johnson et
al. [489],Liu et al. [505],and Rousseeuwet al. [51a].
An important application area for anomaly detection is intrusion detection.
Surveys of the applications of data mining to intrusion detection are given by
Lee and Stolfo [502] and Lazarevic et al. [501]. In a different paper, Lazarevic
et al. [500] provide a comparison of anomaly detection routines specific to
network intrusion. A framework for using data mining techniques for intrusion
detection is provided by Lee et al. [503]. Clustering-based approachesin the
 Bibliography 677

area of intrusion detection include work by Eskin et al. 14771,Lane and Brodley
[499], and Portnoy et al. [509].

Bibliography
f461] B. Abraham and A. Chuang. Outlier Detection and Time Series Modeling. Techno-
metrics, Sl(\:2aI 248, May 1989.
14621 C. C. Aggarwal and P. S. Yu. Outlier detection for high dimensional data. In Proc.
of 2001 ACM-SIGMOD Intl. Conf. on Managernent of Data, pages 37-46. ACM Press,
2001.
f463] V. Barnett. The Study of Outliers: Purpose and Model. Applied Stati.stics, 27(3):
242 250, 1978.
'in Statistical Data. Wiley Series in Probability and
[464] V. Barnett and T. Lewis. Outliers
Statistics. John Wiley & Sons, 3rd edition, April 1994.
[465] S. D. Bay and M. Schwabacher. Mining distance-basedoutliers in near linear time with
randomization and a simple pruning rule. In Proc. of the 9th Intl. Conf. on Knowledge
Discouerg and Data M'ining, pages 29 38. ACM Press, 2003.
f 4 6 6 ] R . J . B e c k m a n a n d R . D . C o o k . ' O u t l i e r . . . . . . . . . . s ' . T e c h n o r n e t r i c s , 2 S ( 2 ) : 1 1194 9 , M a y
1983.
[467] R. J. Beckman and R. D. Cook. f'Outlier..........s']:Response. Technometrics,25(2):
161-163, May 1983.
[468] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. OPTICS-OF: Identifying
Local Outliers. In Proceedings of the Thi,rd European Conference on Princ'iples of Data
Mining and Knowledge D'iscouery, pages 262-270. Springer-Verlag, 1999.
[469] M. M. Breunig, H.-P. Kriegel, R. T. Ng, and J. Sander. LOF: Identifying density-based
local outliers. In Proc. of 2000 ACM-SIGMOD Intl. Conf. on Management of Data,
pages 93-104. ACM Press, 2000.
14701A. Chaudhary A. S. Szalay, and A. W. Moore. Very fast outlier detection in large
multidimensional data sets. In Proc. ACM SIGMOD Workshop on Research Issues 'in
Data Min'ing and Knowledge D'iscouery @MKD),2002.
f471] N. V. Chawla, N. Japkowicz, and A. Kolcz, editors. Workshop on Learni'ng from
Imbalanced Data Sets II, 20th IntI. Conf. on Machine Leamuing,2000. AAAI Press.
[472] N. V. Chawla, N. Japkowicz, and A. Kolcz, editors. SIGKDD Erplorations Newsletter,
Speci,al'issueon learn'ingfrom imbalanced datasets,volume 6(1), June 2004. ACM Press.
1473] C. Chen and L.-M. Liu. Joint Estimation of Model Parameters and Outlier Effects in
Time Series. Jout'nal of the American Stati.stical Assoc'iation, 88(421):284-297, Marc}l
1993.
1474] L. Davies and U. Gather. The Identification of Multiple Outliers. Journal of the
Am eri,can Stati,sti,cal A ssoc'iation, 88 ( 423) :782-792, September I 993.
14751 J. Dunagan and S. Vempala. Optimal outlier removal in high-dimensional spaces.
Journal of Computer and System Sciences, Special Issue on STOC 2001, 68(2):335-
373, March 2004.
[476] E. Eskin. Anomaly Detection over Noisy Data using Learned Probability Distributions.
In Proc. of the 17th Intl. Conf. on Machine Learning, pages 255 262,2000.
[477] E Eskin, A. Arnold, M. Prera,u, L. Portnoy, and S. J. Stolfo. A geometric framework for
unsupervised anomaly detection. In Applications of Data Mini,ng in Computer Securitg,
pages 78-100. Kluwer Academics, 2002.
678 Chapter 10 Anomaly Detection

[478] A. J. Fox. Outliers in Time Series. Journal of the Royal Statisti,cal Society. Series B
( M ethodological), 34(3) :350-363, 1972.
14791 A. Ghosh and A. Schwartzbard. A Study in Using Neural Networks for Anomaly and
Misuse Detection. In 8th USENIX Security Symposiurn, August 1gg9.
1480] R. Gnanadesikan and J. R. Kettenring. Robust Estimates, Residuals, and Outlier
Detection with Multiresponse Data. B'iometrics, 28(I):8I-I2 , March Ig72.
[481] F. Grubbs. Procedures for Testing Outlying Observations. Annal o.f Mathemati.cal
Statistics, 2I(l):27 58, March 1950.
[482] J. Hardin and D. M. Rocke. Outlier Detection in the Multiple Cluster Setting using
the Minimum Covariance Determinant Estimator. Computational Statisti,cs and Data
Analg sis, 44:625-638, 2004.
f483] D. M. Hawkins. Identi,fication of Outliers. Monographs on Applied Probability and
Statistics. Chapman & Hall, May 1980.
'fOutlier..........s]':
f484] D. M. Hawkins. Discussion. Technometrics,2b(2):Ib5_156,May
1983.
1485] S. Hawkins, H. He, G. J. Williams, and R. A. Baxter. Outlier Detection Using Repli-
cator Neural Networks. rn DawaK 2000: Proc. of the lth Intnl. conf. on Data Ware-
hous'ing and Knowledge Discouerg, pages 170 180. Springer-Yerlag,2OO2.
f486] V. J. Hodge and J. Austin. A Survey of Outlier Detection Methodologies. Arti,ficial
Intelligence Reui,ew,22:85 126, 2004.
[487] H. V. Jagadish, N. Koudas, and S. Muthukrishnan. Mining Deviants in a Time Series
Database. In Proc. of the 25th VLDB Conf., pages 102-113, 1999.
f4881 N. Japkowicz, editor. Workshop on Learn'ing from Imbalanced Data Sets I, Seuenteenth
National Conference on Artificial Intelligence, Published, as Techn'ical Report WS-00-05,
2000. AAAI Press.
[489] T. Johnson, I. Kwok, and R. T. Ng. Fast Computation of 2-Dimensional Depth Con-
tours. In KDD98, pages 224-228, 1998.
f490] M. V. Joshi. On Evaluating Performance of Classifiers for Rare Classes. In Proc. of
the 2002 IEEE IntI. Conf. on Data Mining, pages 641 044,2002.
[491] M. V. Joshi, R. C. Agarwal, and V. Kumar. Mining needle in a haystack: Classifying
rare classes via two-phase rule induction. In Proc. of 2001 ACM-SIGM)D Intt. Conf.
on Managernent of Data, pages 91-102. ACM Press, 2001.
1492] M' V. Joshi, R. C. Agarwal, and V. Kumar. Predicting rare classes: can boosting make
any weak learner strong? rn Proc. of 2002 ACM-SIGM)D IntI. Conf. on Management
of Data, pages 297-306. ACM Press, 2002.
[493] M. V. Joshi, R. c. Agarwal, and v. Kumar. Predicting Rare classes: comparing
Two-Phase Rule Induction to Cost-Sensitive Boosting. In Proc. of the 6th European
conf. of Princi'ples and Practice of Knowledge Di,scoueryin Databases, pages 237 24g.
Springer-Verl ag, 2002.
1494] M. V. Joshi, V. Kumar, and R. C. Agarwal. Evaluating Boosting Algorithms to Classify
Rare Classes: Comparison and Improvements. In Proc. of the 2001 IEEE Intl. Conf.
on Data M,ining, pages 257 264,200I.
[495] E. Keogh, S. Lonardi, and B. Chiu. Finding Surprising Patterns in a Time Series
Database in Linear Time and Space. rn Proc. of the 8th IntI. conf. on Knowledge
Discouery and Data Mining, Edmonton, Alberta, Canada, JuJy 2002.
[496] E. M. Knorr and R. T. Ng. A Unified Notion of Outliers: Properties and Computation.
rn Proc. of the Srd IntI. conf . on Knowledge D'iscouerg and Data Mining, pages 2lg-222,
1997.
 Bibliography 679

1497 E. M. Knorr and R. T. Ng. Algorithms for Mining Distance-Based Outliers in Large
Datasets. In Proc. of the Zlth VLDB Conf., pages 392 403,24-27 1998.
[498] E. M. Knorr, R. T. Ng, and V. Tucakov. Distance-based outliers: algorithms and
applications. The V L D B J ournal, 8(3-4):237-253, 2000.
1499] T. Lane and C. E. Brodley. An Application of Machine Learning to Anomaly Detection.
In Proc. 20th NIST-NCSC Nati,onal Informat'ion Systems Security Conf.,pages 366 380,
1997.
[500] A. Lazarev\c,L.Ertoz, V. Kumar, A. Ozgur, and J. Srivastava. A Comparative Study
of Anomaly Detection Schemes in Network Intrusion Detection. In Proc. of the 2003
SIAM Intl. Conf. on Data Mini,ng,2OO3.
[501] A. Lazarevic, V. Kumar, and J. Srivastava. Intrusion Detection: A Survey. In Manag-
ing Cyber Threats: Issues, Approaches and Challenges, pages 19-80. Kluwer Academic
Publisher, 2005.
[502] W. Lee and S. J. Stolfo. Data Mining Approaches for Intrusion Detection. In 7th
USENIX Security Sympos'ium, pages 26-29, January 1998.
[503] W. Lee, S. J. Stolfo, and K. W. Mok. A Data Mining Framework for Building Intrusion
Detection Models. In IEEE Sgmposium on Securitg and Priuacg, pages 120-132, 1999.
[504] W. Lee and D. Xiang. Information-theoretic measures for anomaly detection In Proc.
of the 2001 IEEE Syrnpos'ium on Security and Priuacy, pages 130-143, May 2001.
1505] R. Y. Liu, J. M. Parelius, and K Singh. Multivariate analysis by data depth: descrip-
tive statistics, graphics and inference. Annals of Stat'istics,27(3):783-858, 1999.
1506] M. Markou and S. Singh. Novelty detection: A review part 1: Statistical approaches.
Signal Processi,ng, 83(L2):248I 2497, 2003.
15071M. Markou and S. Singh. Novelty detection: A review part 2: Neural network based
approaches. Signal Processi,ng, 83(12):2499 2521, 2003.
[508] C. R. Muirhead. Distinguishing Outlier Types in Time Series. Journal of the Royal
Stati,stical Soci,ety.Series B (Methodologi,cal),48(l):39 47, 1986.
[509] L. Portnoy, E. Eskin, and S. J. Stolfo. Intrusion detection with unlabeled data using
clustering. In In ACM Workshop on Data Mi.ning Appli,ed to Securi'ty,200I.
[510] S. Ramaswamy, R. Rastogi, and K Shim. Efficient algorithms for mining outliers from
large data sets. In Proc. of 2000 ACM-SIGMOD IntI. Conf. on Management of Data,
pages 427 438. ACM Press, 2000.
1511] D. M. Rocke and D. L. Woodruff. Identification of Outliers in Multivariate Data.
Journal oJ the Ameri,can Stati,stical Associat'ion, 91(435):1047-106t, September 1996.
f512] B. Rosner. On the Detection of Many Outliers. Technometrics, 17(3):227 227, 1975.
[513] P. J. Rousseeuw and A. M. Leroy. Robust Regression and Outlier Detection. Wlley
Series in Probability and Statistics. John Wiley & Sons, September 2003.
[514] P. J. Rousseeuw, I. Ruts, and J. W. T[rkey. The Bagplot: A Bivariate Boxplot. ?he
American Statist'ic'ian, 53(4) :382-387, November 1999.
1515] P. J. Rousseeuw and B. C. van Zomeren. Unmasking Multivariate Outliers and Lever-
age Points. Journal of the American Stati,st'icalAssociation,85(411):633-639, September
1990.
[516] D. W. Scott. Partial Mixture Estimation and Outlier Detection in Data and Regres-
sion. In M. Hubert, G. Pison, A. Struyf, and S. V. Aelst, editors, Theory and' Appli''
cat'ions of Recent Robust Methods, Statistics for Industry and Technology. Birkhauser,
2003.
[517] S. Shekhar, C.-T. Lu, and P. Zhang. A Unified Approach to Detecting Spatial Outliers.
G eoInf orrnatica, 7 (2) :139-166, June 2003.
680 Chapter 10 Anomaly Detection

1518] M.-L. Shyu, S.-C. Chen, K. Sarinnapakorn, and L. Chang. A Novel Anomaly Detection
Scheme Based on Principal Component Classifier. In Proc. of the 2003 IEEE Intl. Conf.
on Data Mi,n'ing, pages 353 365, 2003.
1519] P. Sykacek. Equivalent error bars for neural network classifiers trained by bayesian
inference. ln Proc. of the European Symposium on Artifi,cial Neural Networks, pages
l2t 126, 1997.
f5201 N. Ye and Q. Chen. Chi-square Statistical Profiling for Anomaly Detection. In Proc.
of the 2000 IEEE Workshop on Inforrnation Assurance and Securitg, pages 187 193,
June 2000.

LO.7 Exercises
1 . Compare and contrast the different techniques for anomaly detection that were
presentedin Section I0.7.2. In particular, try to identify circumstancesin which
the definitions of anomalies used in the different techniques might be equivalent
or situations in which one might make sense, but another would not. Be sure
to consider different types of data.

2 . Consider the following definition of an anomaly: An anomaly is an object that

is unusually influential in the creation of a data model.

(a) Compare this definition to that of the standard model-based definition of

an anomaly.
(b) For what sizes of data sets (small, medium, or large) is this definition
appropriate?

3 . In one approach to anomaly detection, objects are represented as points in a

multidimensional space,and the points are grouped into successiveshells,where
each shell represents a layer around a grouping of points, such as a convex hull.
An object is an anomaly if it lies in one of the outer shells.

(a) To which of the definitions of an anomaly in Section 10.1.2is this definition

most closely related?
(b) Name two problems with this definition of an anomaly.

4 . Association analysis can be used to find anomalies as follows. Find strong asso-
ciation patterns, which involve some minimum number of objects. Anomalies
are those objects that do not belong to any such patterns. To make this more
concrete, we note that the hyperclique association pattern discussedin Section
6.8 is particularly suitable for such an approach. Specifically, given a user-
selected h-confidencelevel, maximal hyperclique patterns of objects are found.
All objects that do not appear in a maximal hyperclique pattern of at least size
three are classified as outliers.
 LO.7 Exercises 681

(a) Does this technique fall into any ofthe categoriesdiscussedin this chapter?
If so, which one?
(b) Name one potential strength and one potential weaknessofthis approach.

K Discuss techniques for combining multiple anomaly detection techniques to im-

prove the identification of anomalous objects. Consider both supervised and
unsupervised cases.

b. Describe the potential time complexity of anomaly detection approachesbased

on the following approaches: model-based using clustering, proximity-based,
and density. No knowledge of specific techniques is required. Rather, focus
on the basic computational requirements of each approach, such as the time
required to compute the density of each object.

The Grubbs' test, which is described by Algorithm 10.3, is a more statistically

sophisticated procedure for detecting outliers than that of Definition 10.3. It is
iterative and also takes into account the fact that the z-score does not have a
normal distribution. This algorithm computes the z-score of each value based
on the sample mean and standard deviation of the current set of values. The
value with the largest magnitude z-score is discarded if its z-score is larger
tlnan g., the critical value of the test for an outlier at significance level cr. This
processis repeated until no objects are eliminated. Note that the sample mean,
standard deviation, and g" are updated at each iteration.

Algorithm 10.3 Grubbs' approach for outlier elimination.

1: Input the values and o
{m is the number of values, a is a parameter, and f. is a value chosen so that
61: prob(r ) l.) for a t distribution with m - 2 degreesof freedom.)
2 : repeat
3 : Compute the sample mean (r) and standard deviation (s").
4: Compute a value g. so that prob(lzl 2 9) : a.
(In termsof t. andm. 9.: lF^-** I
#
Compute the z-scoreof each value, i.e., z : (r - *)lt'.
o: Let g: maxlzl, i.e., find the z-scoreof largest magnitude and call it g.
ifg>9.then
8: Eliminate the value corresponding to g.
o. m+m-I
1 0 : end if
1 1 : until No objects are eliminated.

(a) What is the limit of the value usedfor Grubbs' test as m

#{H"
approaches infinity? Usea significancelevel of 0.05.
682 Chapter 10 Anomaly Detection

(b) Describe, in words, the meaning of the previous result.

8 . Many statistical tests for outliers were developed in an environment in which

a few hundred observations was a large data set. We explore the limitations of
such approaches.

(a) For a set of 1,000,000values, how likely are we to have outliers according
to the test that says a value is an outlier if it is more than three standard
deviations from the average? (Assume a normal distribution.)
(b) Does the approach that states an outlier is an object of unusually low
probability need to be adjusted when dealing with large data sets? If so,
how?

9 . The probability density of a point x with respect to a multivariate normal

distribution having a mean p and covariance matrix I is given by the equation

7/, I -(x-ptx-1(x-!)
Problxl:- s- 2 (10.8)
(t/2r)-lEl'/'

Using the sample mean x and covariance matrix S as estimates of the mean p
and covariance matrix !, respectively, show that the logprob(x) is equal to the
Mahalanobis distance between a data point x and the sample mean x plus a
constant that does not depend on x.

1 0 . compare the following two measures of the extent to which an object belongs
to a cluster: (1) distance of an object from the centroid of its closest cluster
and (2) the silhouette coefficient described in Section 8.5.2.

1 1 . Consider the (relative distance) K-means schemefor outlier detection described

in Section 10.5 and the accompanying figure, Figure 10.10.

(a) The points at the bottom of the compact cluster shown in Figure 10.10
have a somewhat higher outlier score than those points at the top of the
compact cluster. Why?
(b) Suppose that we choose the number of clusters to be much larger, e.g.,
10. Would the proposed technique still be effective in finding the most
extreme outlier at the top of the figure? Why or why not?
(c) The use of relative distance adjusts for differences in density. Give an
example of where such an approach might lead to the wrong conclusion.

12, If the probability that a normal object is classified as an anomaly is 0.01 and
the probability that an anomalous object is classifiedas anomalous is 0.99, then
 LO.7 Exercises 683

what is the false alarm rate and detection rate if 99Yaof the objects are normal?
(Use the definitions given below.)

number of anomalies detected

detection rate : (10.e)
total number of anomalies

number of false anomalies

false alarm rate (10.10)
number of objects classified as anomalies

13. When a comprehensive training set is available, a supervised anomaly detection

technique can typically outperform an unsupervised anomaly technique when
performance is evaluated using measures such as the detection and false alarm
rate. However, in some cases, such as fraud detection, new types of anomalies
are always developing. Performance can be evaluated according to the detection
and false alarm rates, because it is usually possible to determine upon investiga-
tion whether an object (transaction) is anomalous. Discuss the relative merits
of supervised and unsupervised anomaly detection under such conditions.

14, Consider a group of documents that has been selected from a much larger set
of diverse documents so that the selected documents are as dissimilar from one
another as possible. If we consider documents that are not highly related (con-
nected, similar) to one another as being anomalous, then all of the documents
that we have selected might be classified as anomalies. Is it possible for a data
set to consist only of anomalous objects or is this an abuse of the terminology?

1 5 . Consider a set of points, where most points are in regions of low density, but a
few points are in regions of high density. If we define an anomaly as a point in
a region of low density, then most points will be classified as anomalies. Is this
an appropriate use of the density-based definition of an anomaly or should the
definition be modified in some way?

1 6 . Consider a set of points that are uniformly distributed on the interval 10,1]. Is
the statistical notion of an outlier as an infrequently observed value meaningful
for this data?

1 7 . An analyst applies an anomaly detection algorithm to a data set and finds a

set of anomalies. Being curious, the analyst then applies the anomaly detection
algorithm to the set of anomalies.

(a) Discuss the behavior of each of the anomaly detection techniques described
in this chapter. (If possible, try this for real data sets and algorithms.)
(b) What do you think the behavior of an anomaly detection algorithm should
be when applied to a set of anomalous objects?
LinearAlgebra
This appendix provides a brief introduction to linear alp;ebrawith a focus on
topics that are relevant to the material in this text. Wr begin by describing
vectors, which can be used to represent both data objects and attributes. We
then discuss matrices, which can be used both to reprer;entdata sets and to
describe transformations on them.

A.1 Vectors
A.1.1 Definition

In Euclidean space, such as the ordinary two- and three-dimensional space

with which we are familiar, a vector is a quantity that has magnitude and
direction. It is traditionally representedas an arrow that has a length equal
to its magnitude and an orientation given by its direction, Figure A.1(a) shows
two vectors: vector u, which has a length of 1 and is parallel to the g axis,
and vector v, which has a length of 2 and a direction c,f 45" with respect to
the r axis. (We shall use lowercasebold letters, such as u and v, to represent
vectors. They are often also represented by italic lower(;aseletters, such as z
and u.) Since a point can be regarded as a displacemenbfrom the origin in a
particular direction, it can be represented by a vector from the origin to the
point.

4.L.2 Vector Addition and Multiplication b;y a Scalar

Various operations can be performed on vectors. (In what follows, we assume
that the vectors are all from the same space space, i.e., have the same di-
mensionality.) For instance, vectors can be added and subtracted. This is
686 Appendix A Linear Algebra

(a) Twovectors. (b) Their difference. (c) Their sum.

Figure
A.1.Twovectors
andtheirsumanddifference.

best illustrated graphically, and vector subtraction and addition are shown in
Figures A.1(b) and A.1(c), respectively.Like the addition of numbers,vector
addition has some familiar properties. If u, v, and w are three vectors, then
these properties can be described as follows:

o Commutativity of vector addition. The order of addition does not

m a t t e r .u * v : v + u .

o Associativity of vector addition. The grouping of vectors during

a d d i t i o nd o e sn o t m a t t e r . ( u + v ) * w : uf (v+*)

o Existence of an identity element for vector addition. There exists

a zero vector, simply denoted as 0, which is the identitv element. For
anyvectoru,u+O:u.

o Existence of additive inverses for vector addition. For every vec-

tor u, there is an inverse vector -u such that u * (-u) : g.

Another important operation is the multiplication of a vector by a number,

which, in the context of linear algebra, is typically called a scalar. Scalar
multiplication changesthe magnitude of the vector; the direction is unchanged
if the scalar is positive and is reversedif the scalar is negative. If u and v are
vectors and a and B are scalars (numbers), then the properties of the scalar
multiplication of vectors can be described as follows:
 ,\.1 Vectors 687

Associativity of scalar rnultiplication. The ortler of multiplication

by two scalarsdoes not matter. a(/u) : (aB)u.

Distributivity of scalar addition over multiplication of a scalar

by a vector. Adding two scalars and then multiplying the resulting
sum by a vector is the same as multiplying each scalar by the vector and
then adding the two resultant vectors. (a + B)u : ztt -l 0,,r .

Distributivity of scalar multiplication over vet:tor addition. Add-

ing two vectors and then multiplying the sum by a scalar is the same
as multiplying each vector by the scalar and then adding. a(u + v) :
au + 0v.

Existence of scalar identity. If a : 1, then for arry vector u, atl : u.

A.1.3 Vector Spaces

A vector space is a set of vectors, along with an associated set of scalars

(e.g., the real numbers) that satisfies the properties given above and that is
closed under vector addition and multiplication by a scillar. (BV closed, we
mean that every result of vector addition and/or scalar rnultiplication results
in a vector in the original set.) Vector spaces have the property that any
vector can be representedas a linear combination of a small set of vectors,
which are known as a basis. More specifically,if u1 ,...,un are the basis
vectors, then we can find a set of n scalars{*r,...,an} for any vector v, so
that v : lf,-ra,;q. We say that the basis vectors span the vector space.
The dimension of a vector space is the minimum numbr:r of vectors that are
necessaryto form a basis. Typically, the basis vectors at'e taken to have unit
Iength.
The basis vectors are usually orthogonal. The orthogonality of vectors is
an extension of the two-dimensional notion of perpendicrrlar lines and will be
defined more precisely later on. Conceptually orthogonal vectors are unrelated
or independent. If basis vectors are mutually orthogonill, then expressing a
vector as a linear combination of basisvectors effectively decomposesthe vector
into a number of independent components.
Thus, a vector in an n-dimensional space can be considered to be an n-
tuple of scalars (numbers). To provide a concrete illustration, consider two-
dimensional Euclidean space,where each point is associatedwith a vector that
represents the displacement of the point from the origin. The displacement
vector to any point can be written as the sum of a displacement in the r
688 Appendix A Linear Algebra

direction and a displacement in the grdirection, which are, respectively,the r

and g coordinates of the point.
We will refer to the components of a vector v by using the notation v :
(ut,rz,...,un-r,un). (With reference t o t h e e q u a t i o n ,v : DLr airti, n,i:
ai.) Note that ui is a component of v, while vi is one of a set of vectors.
With a component view of vectors, the addition of vectors becomes simple
to understand; to add two vectors, we simply add corresponding components.
For example, (2,3) + (4,2) : (6,5). To multiply a vector by a scalar, we
multiply each component by the scalar, e.g., 3 * (2,3) : (6,9).

A.1.4 The Dot Product, Orthogonality, and Orthogonal

Projections

We now define what it means for two vectors to be orthogonal. For simplicity,
we restrict ourselvesto Euclidean vector spaces,although the definitions and
results are easily generalized. We begin by defining the dot product of two
vectors.
Definition A.1 (Dot Product). The dot product u .v of two vectors, u
and v, is given by the following equation:

rL
\-
ll'V :
)_rUlui.
(A.1)
t--1

In words, the dot product of two vectors is computed by multiplying cor-

responding components of a vector and then adding the resulting products.
F o r i n s t a n c e (, 2 , 3 ) - ( 4 , 7 ) : 2 * 4 * 3 * 1 : 1 1 .
In Euclidean space it can be shown that the dot product of two (non-zero)
vectors is 0 if and only if they are perpendicular. Geometrically, two vectors
define a plane, and their dot product is 0 if and only if the angle (in the plane)
between the two vectors is 90o. We say that such vectors are orthogonal.
The dot product can also be used to compute the length of a vector in
Euclidean space, namely, length(u) : Vffi. The length of a vector is also
known as its 12 norm and is written as llull. Given a vector u, we can find
a vector that is pointing in the same direction as u, but is of unit length, by
dividing eachcomponentof u by its length; i.e., by computing r/llrll. We say
that we have normalized the vector to have an L2 norm of 1.
Given the notation for the norm of a vector, the dot product of a vector
can be written as
u . v : l l r . llll ' l l c o s ( O ) , (A.2)
 A.1 Vectors 689

vr4

proiection
FigureA.2, Orthogonal of vectoru.
ofvectorv inthedirection

where 0 is the angle between the two vectors. By grouping terms and reorder-
ing, this can be rewritten as

u . v : ( l l v l cl o s ( dl)l)" l l: v " l l u l l , (A.3)

where v, : llvll cos(d) represents the Iength of v in the direction of u as

illustrated in Figure A.2. If u is a unit vector, then the dot product is the
component of v in the direction of u. We refer to this as the orthogonal
projection of v onto u. Of course, it is also true that if v is a unit vector,
then the dot product is the projection of u in the direction of v.
An important consequenceof this is that, given a set of orthogonal vectors
of norm 1 that form a basis of a vector space,we can find the components of
any vector with respect to that basis by taking the dot product of the vector
with each basis vector.
A concept that is closely related to that of orthogonality is the notion of
linear independence.
Definition A.2 (Linear Independence). A set of vectors is linearly inde-
pendent if no vector in the set can be written as a linear combination of the
other vectors in another set.
If a set of vectors is not linearly independent, then they are linearly de-
pendent. Note that we want our basis to consist of a set of vectors such that
no vector is linearly dependent with respect to the remaining basis vectors,
becauseif this were so, then we could eliminate that vector and still have a
690 Appendix A Linear Algebra

set of vectors that span the entire vector space. If we chooseour basis vectors
to be mutually orthogonal (independent), then we automatically obtain a lin-
early independent set since any two vectors that are orthogonal are linearly
independent.

A.1.5 Vectors and Data Analysis

Although vectors were originally introduced to deal with quantities such as

force, velocity, and acceleration, they have proven useful for representing and
understanding many other kinds of data. In particular, we can often regard
a data object or an attribute as a vector. For example, Chapter 2 described
a data set that consisted of 150 Iris flowers that were characterized by four
attributes: sepal length, sepal width, petal length, and petal width. Each
flower can be regarded as a four dimensional vector, and each attribute can be
regarded as a 150 dimensional vector. As another example, a document can be
representedas a vector, where each component correspondsto a term (word)
and the value of each component is the number of times the term appears in
the document. This yields a very sparse, high-dimensional vector, where by
sparse,we mean that most of the entries of the vector are 0.
Once we have representedour data objects as vectors, we can perform var-
ious operations on the data that derive from a vector viewpoint. For example,
using various vector operations, we can compute the similarity or distance of
two vectors. In particular, the cosine similarity of two vectors is defined as

/\uv
COS(tl.\//:' -
'
, (A.4)
ll,'ll llull'
This similarity measure does not take into account the magnitude (length) of
the vectors, but is only concerned with the degreeto which two vectors point
in the same direction. In terms of documents, this means that two documents
are the same if they contain the same terms in the same proportion. Terms
that do not appear in both documents play no role in computing similarity.
We can also simply define the distance between two vectors (points). If
u and v are vectors, then the Euclidean distance between the two vectors
(points) is simply
di,st(r,v) : (t -.r) .(t -.r). (A.5)
This type of measureis more appropriate for the Iris data, since the magnitude
of the various components of the vectors does make a difference in whether
they are consideredto be similar.
 4.2 Matrices 691

Also, for vector data, it is meaningful to compute the mean of the set of
vectors, which is accomplished by computing the mean of each component.
Indeed, some clustering approaches, such as K-means (Chapter 8) work by
dividing the data objects into groups (clusters) and characterizing each cluster
by the mean of the data objects (data vectors). The idea is that a good cluster
is one in which the data objects in the cluster are close to the mean, where
closenessis measured by Euclidean distance for data like the Iris data and by
cosine similarity for data like document data.
Other common operations that are performed on data can also be thought
of as operations on vectors. Consider dimensionality reduction. In the sim-
plest approach, someof the componentsof the data vector are eliminated, while
leaving the others unchanged. Other dimensionality reduction techniques pro-
duce a new set of components (attributes) for the data vector that are linear
combinations of the previous components. Still other methods changethe vec-
tors in more complicated ways. Dimensionality reduction is discussedfurther
in Appendix B.
For certain areasofdata analysis, such as statistics, the analysis techniques
are expressedmathematically in terms of operations on data vectors and the
data matrices that contain these data vectors. Thus, a vector representation
brings with it powerful mathematical tools that can be used to represent,
transform, and analyze the data.
In the remainder of this appendix, we will complete the story, by discussing
matrices.

4.2 Matrices
4.2.L Matrices: Definitions

A matrix is a tabular representation of a set of numbers as a collection of

rows and columns. We will use uppercasebold letters, such as A, to represent
matrices. (Uppercaseitalic letters, such as A, are also used.) The term "mby
n matrix" is commonly used to refer to a matrix with rn rows and n columns.
For example, the matrix A, shown below, is a 2 by 3 matrix. If m : ??,we say
that the matrix is a square rnatrix. The transpose of A is written as A"
and is produced by interchanging the rows and columns of A.

12 71
":l?gN A7:
Ll,)
692 Appendix A Linear Algebra

The matrix entries are representedby subscripted, lowercaseletters. For

matrix A, for example, a,ii rs the entry in the ith row and jth column. Rows
are numbered from top to bottom and columns from left to right. As a specific
illustration, &2r :7 is the entry in the secondrow and first column of A.
Each row or column of a matrix defines a vector. For a matrix A, the zrh
row vector can be represented using the notation aia and the jth column
vector using the notation a*3. Using the previous example, a.2*: 17 5 2],
while a*3 : II 2]7. Notice that row and column vectors are matrices and must
be distinguished; i.e., a row vector and column vector that have the same
number of entries and the same values reoresent different matrices.

4.2.2 Matrices: Addition and Multiplication by a Scalar

Like vectors, matrices can be added by adding their corresponding entries
(components). (Here we are assuming that the matrices have the same number
of rows and columsn.) More specifically, if A and B are two matrices having
dimensions m by n, then the sum of A and B is defined as follows:
Definition A.3 (Matrix Addition). The sum of two m by n matrices,
A and B, is an m by n matrix C, whose entries are given by the following
equation:
cij : aij -f bij- (A.6)
For example,
[s 1] ' fs 41 fs bl
ttft t:t I

L1 2) L2 el L3 11.1
Matrix addition has the following properties:

o Commutativity of matrix addition. The order of addition does not

m a t t e r .A + B : B + A .

o Associativity of matrix addition. The grouping of matrices during

addition doesnot matter. (A+ B) * C : A+ (B+ C) .

o Existence of an identity element for matrix addition. There

exists a zero matrix, having all 0 entries and simply denoted as O,
which is the identity element. For any matrix A, A + 0 : A.

o Existence of additive inverses for matrix addition. For every

matrix A there is a matix -A such that A + (-A) : 0. The entries of
-A are -aii.

As with vectors, we can also multiply a matrix by a scalar.

 A.2 Matrices 693

Definition A.4 (Scalar Multiplication of a Matrix). The product of a

scalar a and a matrix A is the matrix B : aA, whose entries are given by the
following equation.
b6i : a aii (A.7)
Scalar multiplication has properties that are very similar to those of mul-
tiplying a vector by a scalar.

o Associativity of scalar multiplication. The order of multiplication

by two scalarsdoes not matter. a(pe) : @0)A.
o Distributivity of scalar addition over multiplication of a scalar
by a rnatrix. Adding two scalars and then multiplying the sum by a
matrix is the same as multiplying each scalar times the matrix and then
adding the two resultant matrices. (a+ 0)A: aA + BA .

o Distributivity of scalar multiplication over matrix addition.

Adding two matrices and then multiplying the sum by a scalar is the
same as multiplying each matrix by the scalar and then adding. a(A +
B):aAiaB.

o Existence of scalar identity. If a:1, then for any matrix A, aA :

A.

None of the previous properties should be surprising since we can think

of a matrix as being composed of row or column vectors, and hence, matrix
addition or the multiplication by a scalar amounts to adding corresponding
row or column vectors or multiplying them by a scalar.

A.2.3 Matrices: Multiplication

We can define a multiplication operation for matrices. We begin by defining

multiplication between a matrix and a vector.
Definition A.5 (Multiplication of a Matrix by a Column Vector). The
product of an m by n matrix A and an n by 1 column matrix u is the rn by
1 column matrix v: Au, whose entries are given by the following equation.

ui: ai.*'l)T (A.8)

In other words, we take the dot product of the transpose of u with each
row vector of A. In the following example, notice that the number of rows in
694 Appendix A Linear Algebra

u must be the same as the number of columns of A.

[e r] lrl
lttt:t
frzl I

Ll 2)L2l Lel
We can similarly define the multiplication of a matrix by a row vector on
the left side.
Definition ,4'.6 (Multiplication of a Matrix by a Row Vector). The
product of a 1 by m row matrix u and an m by n matrix A is the 1 by n row
matrix v: uA, whose entries are given by the following equation.

?)i: t ' (..r)t (A.e)

In other words, we take the dot product of the row vector with the trans-
pose of each column vector of A. An example is given below.

rr 4li 3l: [o 22]

We define the product of two matrices as an extension to the above idea.

Definition A.7. The product of an m by n matrix A and an n by p matrix
B is the rn by p matrix C : AB, whose entries are given by the equation

cij : a1*. (b-;)t (A.10)

In words, the i,jth entry of C is the dot product of the i,th row vector of A
and the transpose of the jth column vector of B.

fr rl fs 41 ltz 2r1
tttt:tl

Ll 2)12 el Le 22)
Matrix multiplication has the following properties:

o Associativity of matrix rnultiplication. The order of multiplication

of matrices does not matter. (AB)C : A(BC).

o Distributivity of matrix multiplication. Matrix multiplication is

distributive with respect to matrix addition. A(B + C) : AB + AC
and (B + C)A : BA + CA.
 4.2 Matrices 695

o Existence of an identity element for matrix multiplication. If

I, is the p by p matrix with 1's only on the diagonal and 0 elsewhere,
then for aty m by n matrix A, AI,,: A and l*A: A. (Note that the
identity matrix is an example of a diagonal matrix, which is a matrix
whose off diagonal entries are all 0, i.e., &.i,j:0, if i + i.)

In general, matrix multiplication is not commutative, i.e., AB I BA.

A.2.4 Linear TYansformations and Inverse Matrices

If we have an n by 1 column vector u, then we can view the multiplication

of an m by n matrix A by this vector on the right as a transformation of u
into an rn-dimensional column vector v : Au. Similarly, if we multiply A
by a (row) vector u : [rr, .. .,u*] on the left, then we can view this as a
transformation of u into an n-dimensional row vector v : uA. Thus, we can
view any m by n matrix A as a function that maps one vector space onto
another.
In many cases,the transformation (matrix) can be described in easily un-
derstood terms.

o A scaling matrix leaves the direction of the vector unchanged, but

changesits length. This is equivalent to multiplying by a matrix that is
the identity matrix multiplied by a scalar.

o A rotation matrix changes the direction of a vector, but leaves the

magnitude of the vector unchanged. This amounts to a change of coor-
dinate system.

o A reflection matrix reflects a vector across one or more coordinate

axes. This would be equivalent to multiplying some of the entries of the
vector by -1, while leaving the other entries unchanged.

o A projection matrix takes vectors into a lower dimensional subspace.

The simplest example is the modified identity matrix where one or more
of the 1's on the diagonal have been changed into 0's. Such a matrix
eliminates the vector components corresponding to those zero entries,
while preserving all others.

Of course, a single matrix can do two kinds of transformations at once, e.g.,

scaling and rotation.
Following are a few properties of matrices when viewed as functions that
map vectors from one vector space to another.
696 Appendix A Linear Algebra

Matrices are linear transformations, i.e., A(au + 0v): aAu -f /Av

and (ou + pv)A : auA + |vA.

The set of all transformed row vectors of a matrix A is called the row
space of A because the row vectors of the matrix, or some subset of
them, form a basis for the subspaceof transformed row vectors. This is
evident from the following equation, which expressesthe product of a 1
by m row vector r: [rr, ...,urnl and an mby n matrix A as a linear
combination of the rows of the matrix.

TIL
^ \-a
V: 11A: ,tL;.d;.* (A.11)
)
ZJ

The dimension of the row spacetells us the number of linearly indepen-

dent rows of A.

o The set of all transformed column vectors is called the column space
of A. The column vectors of the matrix, or some subset of them, form
a basis for the subspace of transformed column vectors. This is clear
from the following equation, which expressesthe product of an n by I
c o l u m nv e c t o r r : lrr,...,un]T and an mby n matrix A as a linear
combination of the columns of the matrix.

TL
^S-
v: Au: ) .uja*j (A.12)
j:r

The dimension of the column space tells us the number of linearly inde-
pendent columns of A.

o The left nullspace is the set of row vectors that the matrix maps to 0.

o The right nullspace (or more commonly, just nullspace) is the set of
column vectors that the matrix maps to 0.

Note that the rank of a matrix is the minimum of the dimensionality of

the row space and column space and is often used to characterize a matrix.
For instance, if we take a single 7 by n row vector and duplicate it m times to
create an m by rz matrix, we would only have a matrix of rank 1.
A question of practical and theoretical importance is whether matrices,
like real numbers, have multiplicative inverses. First, we note that because of
the nature of matrix multiplication (i.e., dimensions have to match), a matrix
 4.2 Matrices 697

must be square if it is to have an inverse matrix. Thus, for an mby m matrix

A, we are asking if we can find a matrix A-1 such that AA-1 : A-1A - l*.
The answer is that some square matrices have inversesand some do not.
More abstractly, an m by rn matrix has an inverse only if both of its null
spacescontain only the 0 vector, or if, equivalently, the row and column spaces
are both of dimension rn. (This is equivalent to the rank of the matrix being
nr,.) Conceptually, an m by m matrix has an inverse if and only if it uniquely
maps every non-zero m-dimensional row (column) vector onto a unique, non-
zero m-d\mensional row (column) vector.
The existenceof an inversematrix is important when solving various matrix
equations.

A.2.5 Eigenvalue and Singular Value Decomposition

We now discuss a very important area of linear algebra: eigenvaluesand eigen-

vectors. Eigenvaluesand eigenvectors,along with the related concept ofsingu-
lar values and singular vectors, capture the structure of matrices by allowing
us to factor or decomposematrices and expressthem in a standard format. For
that reason, these concepts are useful in the solution of mathematical equa-
tions and for dimensionality and noise reduction. We begin with the definition
of eigenvalues and eigenvectors.
Definition A.8 (Eigenvectors and Eigenvalues). The eigenvaluesand
eigenvectors of an m by n matrix A are, respectively, the scalar values ,\ and
the vectors u that are solutions to the following equation.

Au: )u (A.13)

In other words, eigenvectors are the vectors that are unchanged, except
for magnitude, when multiplied by A. The eigenvalues are the scaling fac-
tors. This equation can also be written as (A - )I)u : 0.
For square matrices, it is possible to decomposethe matrix using eigenval-
ues and eigenvectors.

Theorem A.l. Assume that A i,s an n by n matrir wi.th n i,ndependent(or-

thogonal)e'igenuectors,1f,r,...,un andn correspondi,ngei,genualues, ),1,...,\n.
'i.e., :
LetIJ be the matrir whosecolurnnsare these e'igenuectors, U lur, . . . , u",]
and let lt be a d'iagonalmatrir, whose diagonal entries are the \;, I I i I n.
Then A can be erpressedas

A: UAU-I (A.14)
698 Appendix A Linear Algebra

Thus, A can be decomposedinto a product of three matrices. u is known

as the eigenvector matrix and A as the eigenvalue matrix.
More generally, an arbitrary matrix can be decomposedin a similar way.
Specifically) any rn by n matrix A can be factored into the product of three
matrices as described by the following theorem.

Theorem 4.2. Assume that A i,san m by n matrir. Then A can be erpressed

as follows
A: UEV". (A.15)
WhereU i,sm by m, D is m by n, andY i,sn by n. U andY are orthonormal
matrices, 2.e., thei,r columns are of uni,t length and are mutually orthogonal.
Thus, IJIJT :lm and,VV" : I",. E i,s a d,iagonalmatrir whose di,agonal
entries are non-negat'iueand are sorted so that the larger entries appearfi,rst,
' i . e . ,o i , , i ) o * t , r + t

The column vectors of V, v1 , . . . ,yn are the right singular vectors, while
the columns of U are the left singular vectors. The diagonal elements of
E, the singular value matrix, are typically written as or)...,on and are
called the singular values of A. (This use of o should not be confused with
the use of o to represent the standard deviation of a variable.) There are at
most rank(A) < min(m,n) non-zero singular values.
It can be shown that the eigenvectorsof A"A are the right singular vectors
(i.e., the columns of V), while the eigenvectorsof AAT are the left singular
vectors (i.e., the columns of U). The non-zero eigenvaluesof A"A and AA"
are the o|, i.e., the squares of the singular values. Indeed, the eigenvalue
decomposition of a square matrix can be regarded as a special caseof singular
value decomposition.
The singular value decomposition (SVD) of a matrix can also be expressed
with the following equation. Note that while u;vfl might look like a dot
product, it is not, and the result is a rank I mby n matrix.

rank(A)
^ \- 'l-
A: ) o;u;v; (A.16)
;-1

The importance of the above representation is that every matrix can be

expressedas a sum of rank 1 matrices that are weighted by singular values.
Since singular values, which are sorted in non-increasing order, often decline
rapidly in magnitude, it is possibleto obtain a good approximation of a matrix
by using only a few singular values and singular vectors. This is useful for
dimensionality reduction and will be discussed further in Appendix B.
 4.2 Matrices 699

4.2.6 Matrices and Data Analysis

We can represent a data set as a data matrix, where each row is a data ob-
ject and each column is an attribute. (We can, with equal validity, have
attributes as rows and objects as columns.) Matrix representation provides a
compact and well-structured representation for our data and permits the easy
manipulation of the objects or attributes of the data through various matrix
operations.
Systemsof linear equations are one very common example of the usefulness
of the matrix representation of data. A system of linear equations can be
written as the matrix equation Ax : b and solved using matrix operations.

atrt*atzrzl...ta1nr, u1

&ztrt * azzrz 1 .. . I a2nrn : b z

:
a r n 1 " rl a
1 ^. 2 r 2 + . . . + a m n f i n : b*

In particular, if A has an inverse, the system of equations has a solution

x : A-1b. If not, then the system of equations has either no solution or an
infinite number of solutions. Note that in this case, our rows (data objects)
were equations and our columns were variables (attributes).
For many statistical and data analysis problems, we want to solve linear
systems of equations, but these equations cannot be solved in the manner
just described. For example, we may have a data matrix where the rows
represent patients and the columns represent characteristics of the patients-
height, weight, and age-and their responseto a particular medication, e.g., a
changein blood pressure. We want to expressblood pressure(the independent
variable) as a linear function of the other (dependent) variables, and we can
write a matrix equation in much the same way as above. However, if we have
more patients than variables-the usual case the inverse of the matrix does
not exist.
In this case) we still want to find the best solution for the set of equa-
tions. This means that we want to find the best linear combination of the
independent variables for predicting the dependent variable. Using linear al-
gebra terminology, we want to find the vector Ax that is as close to B as
possible; in other words, we want to minimize llb - Axll, which is the length
of the vector b - Ax. This is known as the least squares problem. Many
statistical techniques (e.g., linear regression, which is discussedin Appendix
700 Appendix A Linear Algebra

D) require the solution of a least squares problem. It can be shown that the
least squaressolution of the equation Ax: b is x : (A"A)-1A"b.
Singular value and eigenvalue decomposition are also very useful in an-
alyzing data, particularly in the area of dimensionality reduction, which is
discussedin Appendix B. Note that noise reduction can also occur as a side
effect of dimensionality reduction.
While we have given a few examples of the application of linear algebra,
we have omitted many more. Examples of other areas where linear algebra is
important in the formulation and solution of problems include solving systems
of differential equations, optimization problems (such as linear programming),
and graph partitioning.

A.3 Bibliographic Notes

There are many books that provide good coverageof linear algebra, including
those by Demmel [521], Golub and Van Loan 1522],and Stramg [523].

Bibliography
f521] J. W. Demmel. Applied Numerical Linear Algebra. SIAM Press, September 1997.
1522] G. H. Golub and C. F. Van Loan. Matrir Cornputat'ions. Johns Hopkins University
Press. 3rd edition. November 1996.
f523] G. Strang. L,inear Algebra anil Its Applications. Harcourt Brace & Company, Orlando,
FL, 3rd edition. 1986.
Dimensionality
Reduction
This appendix considersvarious techniques for dimensionality reduction. The
goal is to expose the reader to the issuesinvolved and to describe some of the
more common approaches. We begin with a discussion of Principal Compo-
nents Analysis (PCA) and Singular Value Decomposition (SVD). These meth-
ods are described in some detail since they are among the most commonly
used approaches and we can build on the discussion of linear algebra in Ap-
pendix A. However, there are many other approachesthat are also employed
for dimensionality reduction, and thus, we provide a quick overview of several
other techniques. We conclude with a short review of important issues.

8.1 PCA and SVD

PCA and SVD are two closely related techniques. For PCA, the mean of the
data is removed, while for SVD, it is not. These techniques have been widely
used for decades in a number of fields. In the following discussion, we will
assumethat the reader is familiar with linear alqebra at the level presented in
Appendix A.

B.1.1 Principal Components Analysis (PCA)

The goal of PCA is to find a new set of dimensions (attributes) that better
captures the variability of the data. More specifically, the first dimension is
chosento capture as much of the variability as possible. The seconddimension
is orthogonal to the first, and, subject to that constraint, captures as much of
the remaining variability as possible, and so on.
7O2 Appendix B Dimensionality Reduction

PCA has several appealing characteristics. First, it tends to identify the

strongest patterns in the data. Hence, PCA can be used as a pattern-finding
technique. Second, often most of the variability of the data can be captured
by a small fraction of the total set of dimensions. As a result, dimensionality
reduction using PCA can result in relatively low-dimensional data and it may
be possible to apply techniques that don't work well with high-dimensional
data. Third, sincethe noise in the data is (hopefully) weaker than the patterns,
dimensionality reduction can eliminate much of the noise. This is beneficial
both for data mining and other data analysis algorithms.
We briefly describe the mathematical basis of PCA and then present an
example.

Mathematical Details

Statisticians summarize the variability of a collection of multivariate data; i.e.,

data that has multiple continuous attributes, by computing the covariance
matrix S of the data.
Definition B.1. Given an m by n data matrix D, whose ?n,rows are data
objects and whose n columns are attributes, the covariancematrix of D is the
matrix S, which has entries si1 defined as

sij : couariance(d*i, d*). (8.1)

In words, s;7 is the covariance of the i,th andjth attributes (columns) of the
data.
The covarianceof two attributes is defined in Appendix C, and is a measure
ofhowstronglytheattributesvarytogether.Ifi.:j,i.e.,theattributesarethe
same, then the covariance is the variance of the attribute. If the data matrix
D is preprocessedso that the mean of each attribute is 0, then S : D"D.
A goal of PCA is to find a transformation of the data that satisfies the
following properties:

1. Each pair of new attributes has 0 covariance (for distinct attributes).

2. The attributes are ordered with resoect to how much of the variance of
the data each attribute captures.

3. The first attribute captures as much of the variance of the data as pos-
sible.
 B.1 PCA and SVD 703

4. Subject to the orthogonality requirement, each successiveattribute cap-

tures as much of the remaining variance as possible.

A transformation of the data that has these properties can be obtained by using
eigenvalueanalysis of the covariancematrix. Let ,\1, . . . , \n be the eigenvalues
of S. The eigenvaluesare all non-negative and can be ordered such that
)1 ) A2 ) . . . \m-r > x^. (Covariance matrices are examples of what are
called positive semidefinite matrices, which, among other properties, have
non-negative eigenvalues.)Let U : [rr,. . . , ur] be the matrix of eigenvectors
of S. These eigenvectorsare ordered so that the ith eigenvector corresponds
to the ith largest eigenvalue. Finally, assume that data matrix D has been
preprocessedso that the mean of each attribute (column) is 0. We can make
the following statements.

o The data matrix D' : DU is the set of transformed data that satisfies
the conditions posed above.

o Each new attribute is a linear combination of the original attributes.

Specifically, the weights of the linear combination for the irh attribute
are the components of the irh eigenvector. This follows from the fact that
the jth column of D/ is given by Du, and the definition of matrix-vector
multiplication given in trquation A.12.

o The variance of the ith new attribute is ,\;.

o The sum of the variance of the original attributes is equal to the sum of
the variance of the new attributes.

o The new attributes are called principal components; i.e., the first new
attribute is the first principal component, the second new attribute is
the second principal component, and so on.

The eigenvector associatedwith the largest eigenvalueindicates the direc-

tion in which the data has the most variance. In other words, if all of the data
vectors are projected onto the line defined by this vector, the resulting val-
ues would have the maximum variance with respect to all possible directions.
The eigenvector associatedwith the second largest eigenvalueis the direction
(orthogonal to that of the first eigenvector) in which the data has the largest
remaining variance.
The eigenvectorsof S define a new set of axes. Indeed, PCA can be viewed
as a rotation of the original coordinate axes to a new set of axes that are
aligned with the variability in the data. The total variability of the data is
preserved,but the new attributes are now uncorrelated.
7O4 Appendix B Dimensionality Reduction

-1
-2
-3

-5
-2024 10 -6

(a) origi;al points 101eointsatteitransformation

thedata.
Figure8.1. UsingPCAto transform

Example B.1 (Two-Dimensional Data). We illustrate the use of PCA

for aligning the axes in the directions of the maximum variability of the data.
Figure B.1 shows a set of 1000 two-dimensional data points, before and after
a PCA transformation. The total variance for the original set of points is the
sum of the variance of the r and g attributes, which is equal to 2.84 -t 2.95 :
5.79. After transformation, the varianceis 4.81 + 0.98 : 5.79. I

Example B.2 (Iris Data). This example uses the Iris data set to demon-
strate the use of PCA for dimensionality reduction. This data set contains
150 data objects (flowers); there are 50 flowers from each of three different
Iris species: Setosa, Versicolour, and Virginica. Each flower is described by
four attributes: sepal length, sepal width, petal length, and petal width. See
Chapter 3 for more details.
Figure 8.2(a) shows a plot of the fraction of the overall variance accounted
for by each eigenvalue (principal component) of the covariance matrix. This
type of plot is known as a scree plot and is useful for determining how many
principal components need to be kept to capture most of the variability of the
data. For the Iris data, the first principal component accounts for most of
the variation (92.5%), the second for only 5.3T0,and the last two components
for just 2.2%0.Thuq keeping only the fi.rst two principal components preserves
most of the variability in the data set. Figure 8.2(b) showsa scatter plot of the
Iris data based on the first two principal components. Note that the Setosa
flowers are well separated from the Versicolour and Virginica flowers. The
latter two sets of flowers, while much closer to each other, are still relatively
well separated.
 B.1 PCA and SVD 7O5

0.9

0.8

n '7
o
o
o
0.6
G

o AE

o
o 0.4
E
Lr
0.3

0.2

0.1

0
123
Eigenvalue
(a) Fractionof varianceaccounted
for by eachprincipalcomponent.

* * E o-
* -tr
**

**E
E
*

t-2-1012
FirstPrincipalComponet

(b) Plotof firsttwo principalcomponents

of lris data.

Figure
8.2,PCAapplied
tothelrisdataset.
706 Appendix B Dimensionalitv Reduction

8.T.2 SVD
PCA is equivalent to an SVD analysis of the data matrix, once the mean
of each variable has been removed. Nonetheless,it is informative to look at
dimensionality reduction from the SVD point of view, since it is not always
desirable to remove the mean from data, especially if the data is relatively
sparse.

Mathematical Details

From Appendix A, we know that an m by n matrix A can be written as

rank(A)

A- t oppf,:UEV". (B.2)
i:l

where oi is the ith singular value of A (the ith diagonal entry of )), u6 is the
ith left singular vector of A (the ith column of U), and the v; is the i.th right
singular vector of A (the frh column of V). (See Section A.2.5.) An SVD
decomposition of a data matrix has the following properties.

o Patterns among the attributes are captured by the right singular vectots,
i.e., the columns of V.

o Patterns among the objects are captured by the left singular vectors,
i.e., the columns of IJ.

o A matrix A can be successively approximated in an optimal manner by

taking, in order, the terms of Equation B.2. We do not explain what
we mean by optimal, but refer the reader to the bibliographic notes.
Informally, the larger a singular value, the larger the fraction of a matrix
that is accounted for by the singular value and its associated singular
vectors.

o To obtain a new data matrix with k attributes, we compute the matrix

D/: Dx [v1,v2,.'.,vk]. It might seemfrom the previousdiscussion
that we would take the matrix that results from the first k terms of
Equation A.12. However, while the resulting matrix is of rank k, it still
has n columns (attributes).

Example B.3 (Document Data). SVD decomposition can be used to an-

alyze document data. The data for this example consists of 3204 newspaper
 B.1 PCA and SVD 7O7

articles from the Los Angeles T'imes. These articles come from 6 different sec-
tions: Entertainment, Financial, Foreign, Metro, National, and Sports. The
data matrix is a document-term matrix, where each row represents a docu-
ment and each column is a term (word). The value of the i,jth entry is the
number of times the jth term occurs in the iih document. The data was pro-
cessedusing standard techniques to remove common words, to adjust for the
different frequencieswith which terms appear?and to adjust for the different
lengths of documents. (See Section 2.3.7 for more details.)
An SVD analysis of the data was performed to find the first 100 singular
values and vectors. (For many data sets, it is too expensiveto find a full SVD
or PCA decomposition and often pointless since relatively few of the singular
values or eigenvaluesare required to capture the structure of the matrix.)
The largest singular value is associatedwith common terms that are frequent,
but not eliminated by the preprocessing. (It can happen that the strongest
patterns represent noise or uninteresting patterns.)
However, the patterns associated with other singular values were more
interesting. For example, the following are the top 10 terms (words) associated
with the strongest components in the second right singular vector:

game, score, lead, team, plaj, rebound, season, coach, league,

goal

These are all terms associatedwith sports. Not surprisingly, the documents
associatedwith the strongest components of the secondleft singular vector are
predominantly from the Sports section
The top 10 terms associated with the strongest components in the third
right singular vector are the following:

earn, million, quarter, bank, rose, billion, stock, company,

corporation, revenue

These are all financial terms, and, not surprisingly, the documents associated
with the strongest components in the third left singular vector are predomi-
nantly from the Financial section.
We reduced the dimensionality of the data using the second and third
singular vectors, i.e., D/: D x [vz,ve]. In other words, all documents were
expressedin terms of two attributes, one relating to Sports and one relating to
Finance. A scatter plot of documents is given by Figure B.3. For clarity, non-
Sports, non-Financial documents have been eliminated. The Sports documents
are shown in a lighter shade of gray, while the Financial documents are a
darker gray. The two different categories of documents are well separated for
708 Appendix B Dimensionality Reduction

0.6

Documents

E ot
c

E
o ne

E
f
a
.= v.1 a
a
a
c a'
,

"i.

-o.2 -0.1 0 0.1 0.2 0.3 0.4

SecondSingularComponent

FigureB.3. Plotof SportsandFinancial

documents andthird
fromtheLA Timesusingthesecond
singular
values.

the most part. Indeed, the Sports documents do not vary much with respect
to the Financial variable (component 3) and the Financial documents do not
vary much with respect to the Sports variable (component 2).

8.2 Other Dimensionality Reduction Techniques

In this section, we review a few other dimensionality reduction techniques.
These techniques will be discussedmore briefly, with a focus on their general
motivation and approach.

8.2.I Factor Analysis

For PCA and SVD, the new attributes that are produced are linear combina-
tions of the original variables. With factor analysis, the goal is to expressthe
original variables as linear combinations of a small number of hidden or la-
tent attributes. The motivation is based on the following observation. Often
there are characteristics of data objects that are hard to measuredirectly, but
that seem to be related to measurable characteristics. One common example
is intelligence and performance on various types of IQ tests. Another common
 8.2 Other Dimensionality Reduction Techniques 7Og

example is the connection between performance in various athletic events and

an athlete's speed and strength. If a small number of attributes can be found
that group and summarize the original attributes, then we will have achieved
both a reduction in dimensionality and an increase in our understanding of
the data.
The motivation for factor analysis is sometimes also explained in terms
of the covariance or correlation matrix of the data. Suppose that a group of
attributes are not very highly correlated to other attributes, but are strongly
correlated to one another, perhaps becausethey measurethe same underlying
quantity. In this case,it would seemdesirableto develop techniquesthat could
find a single underlying attribute that summarizes each such group.
For example, consider a data set that records the performance of a group
of athletes in the ten separate events that comprise the decathlon. We might
find that athletes tend to show the same performance in all events that em-
phasize speed; i.e., slow athletes are consistently slow and fast athletes are
consistently fast. Likewise, we might find that an athlete's behavior in an
event that requires strength indicates how he or she will perform in another
event that emphasizesstrength. Hence, we might hypothesizethat an athlete's
performance in any given event is really determined by the nature of the event
and two underlying factors: speed and strength. Factor analysis attempts to
discover such relationships.
More formally, let ft,fz,.. . , fo be the latent factors, i.e., the underlying
or hidden attributes. Note that these are the new attributes and have a value
for each object. If the original data matrix is D, an mby n matrix, then the
new data matrix is F : lh,fz, . . . ,fp], which is an rn by p matrix. (Note that
f*j : fi.) The i'jth entry of F is f.;i, the jth component of fi.
Assume that the mean of each attribute is 0. If d,;* is the 'ith row of
the original data matrix D, then f;* is the corresponding row of the new
data matrix, F. The standard factor analysis model assumesthe following
relationship between the old and new data objects:

afi: trffi+ e (B.3)

or equivalently by

dii : Airft I \izfn,. . ., \jpftu I er. (B.4)

A, which has entries ,\61, is an n by p matrix of factor loadings that

indicate, for each of the original attributes, how the original value depends
on the latent factors, i.e., the new attributes. To illustrate, in the decathlon
7LO Appendix B Dimensionality Reduction

example, there would be two latent factors: speed and strength. These corre-
spond to columns of F. Each athlete would be representedby a row of F with
entries recording the athlete's speed and strength. Each column of D would
correspond to one of the ten events of the decathlon, while each row again
corresponds to an athlete. The ijth entry of D is the performance of the i,th
athlete in the jth event. A would be a 10 by 2 matrix. If the first column of D
records the performance of the athletes on the 100-meter dash, then the per-
formanceof athlete i in the 100-meterdash is written as di1- \nft* \tzfn,
where fi1 is a value indicating the speed of athlete i, and f,;2 is a value indi-
cating the strength of athlete i. \tr and .\12 indicate how an athlete's speed
and strength, respectively, should be weighted to predict an athlete's perfor-
mance in the 100 meter dash. We would expect that .\11 would be relatively
large compared to )rz. Note that these weights are the same acrossall objects
(athletes).
Since all latent factors are involved in the determination of the value of any
original attribute, they are known as common factors. e is an error term
that accounts for the portion of the attributes that is not accounted for by the
common factors, and hence, the components of e are known as the specific
factors.
Example B.4 (Factor Analysis of Iris Data). This example is based on
the Iris data set. For this data, only a single factor could be found. The
flowers in the Iris data set are organized so that the first 50 flowers are of
speciesSetosa, the second 50 are Versicolour, and the last 50 are Virginica.
This single factor (attribute) is plotted against flower as shown in Figure B.4.
This factor seemsto capture the distinction among the three species.

8.2.2 Locally Linear Embedding (LLE)

LLE is a technique for dimensionality reduction based on the idea of analyzing

overlapping local neighborhoods in order to determine the local structure. The
LLE algorithm is given below.

Algorithm E|.1 LLE algorithm.

1: Find the nearest neighbors of each data point.
2: Express each point X1 d,S& linear combination of the other points, i.e., x4 :
Diw4x7, wherefr.oj:I andwii:0 i f x j i s n o t a n e a r n e i g h b o ro f x ; .
3: Find the coordinates of each point in lower-dimensional space of specified dimen-
sion p by using the weights found in step 2.
 8.2 Other Dimensionality Reduction Techniques 7Ll

()
:0
o
o
J
Setosa
I
I
I
Virginica

-1 i

50 100 150
Flower

Figure8.4. Plotoftheflower
ofthelrisdatasetversus
thesinglelatent
factor.

In step 2, the weight matrix W, whose entries are wij) is found by minimiz-
ing the squared approximation error as measured by the following equation.
W can be found by solving a least squares problem. (Such problems were
discussedin Appendix A.)

error(W): (B.5)
T
Step 3 performs the actual dimensionality reduction. Given the weight
matrix and a number of dimensiotrs,?, specified by the user, the algorithm
constructs a "neighborhood preserving embedding" of the data into the lower-
dimensional space. If y6 is the vector in the lower-dimensional space that
correspondsto x6 and Y is the new data matrix whose 'ith row is y;, then this
can be accomplished by finding a Y that minimizes the following equation.

e rro r(Y ): t (B.6)

7L2 Appendix B Dimensionality Reduction

o oo o @ o@.r6Doo@o& @@ooo o@

*
irx

EI

tr E
D D

-5-4-3-2-101234
First New Attribute

Figure8.5. Plotoftheflowers
ofthelrisdatasetbasedontwonewattributes
fromLLE.

Example B.5. the use of LLtr for dimensionality reduction is illustrated using
the Iris data set. Specifically, the data was projected to two dimensions. A
neighborhood of 30 points was used. A scatter plot of the projected data is
shown in Figure B.5. The data can also be projected to one dimension. In
that case, it looks much like Figure B.4.

8.2.3 Multidimensional Scaling, FastMap, and ISOMAP

Multidimensional scaling is a technique that is often used for dimensionality

reduction. A number of variations of this technique have been proposed, but
the general strategy of these techniques is the same: Find a projection of the
data to a lower-dimensionalspace that preservespairwise distances as well as
possible, as measuredby an objective function. Becauseof this strategy, MDS
starts from a dissimilarity matrix, and thus, can be used even for data that
does not originally have a vector space representation, e.g., strings.

Standard MDS Techniques

We begin by describing the classical MDS approach for projecting data to a

p-dimensional space. Assume that we are given a distance matrix D, where
the entry d6, is the distance between Lhe i,th and jth objects. Let dtoobe the
 8.2 Other Dimensionality Reduction Techniques 713

distance between the objects after they have been transformed. ClassicalMDS
tries to assign each object to a p-dimensional point such that a quantity called
stress is minimized, where stress is defined as

t ,,2
Du \di, - ari)
stTess: (B.7)

The classical version of MDS is an example of metric MDS techniques,

which assumethat the dissimilarities are continuous variables (interval or ra-
tion). Non-metric MDS techniques assumethat the data is categorical (at
best ordinal). We will not discussthe details of these algorithms, except to say
that the typical approach is to initially assign objects to p-dimensional points
in some manner and then try to modify the points to reduce the stress.
When classical MDS or some of the other standard variants of MDS are
applied to the Iris data set, they yield almost the same results as shown in
Figure B.2. Indeed, classicalMDS for Euclidean distance is equivalent to PCA.

FastMap

A recent development in the area of MDS is the algorithm FastMap. It has

the same goal as other MDS techniques, but has two important differences.

It is faster-linear complexity.

It can operate incrementally.

The FastMap algorithm identifies a pair of objects and then computes the
distance of each remaining object in this direction. This can be accomplished
using only pairwise distances by employing certain facts of geometry, namely,
the law of cosines. This distance is taken as the value of the first attribute. The
objects are then projected onto an (n - 1)-dimensional subspace. Again, this
can be performed using only pairwise distances. The processis then repeated.
The FastMap algorithm is initially applied to an entire data set. However,
if we keep track of the pairs of objects that are chosen at each step, then
we can incrementally apply FastMap to a new object. The only information
needed is the distance of the new object to the selectedpairs.
714 Appendix B Dimensionality Reduction

Figure8.6. Plotof Swissrolldataset.

ISOMAP

MDS and PCA are not good at dimensionality reduction when the points have
a complicated, non-linear relationship to one another. (An exceptions is kernel
PCA-see bibliographic notes.) ISOMAP, which is an extension of traditional
MDS, was developedto handle such data sets. An example of the type of data
set that it can handle is given in Figure 8.6, which shows a plot of the "Swiss
roll" surface. A data set with this structure constitutes a two-dimensional
set of data in a three-dimensional space, but one that cannot be successfully
handled by PCA or MDS. However, ISOMAP can successfully analyze this
data set.
Algorithm B.2 outlines the basic ISOMAP algorithm. Nearest neighbors

Algorithm B.2 ISOMAP Algorithm.

1: Find the nearest neighbors of each data point and create a weighted graph by
connecting a point to its nearest neighbors. The nodes are the data points and
the weights of the links are the distances between points.
2: Redefine the distances between points to be the length of the shortest path be-
tween the two points in the neighborhood graph.
3: Apply classical MDS to the new distance matrix.

can be defined, either by taking the k-nearest points, where k is a parameter,

or by taking all points within a specified radius of the point. The purpose
of step 2 is to compute the geodesic distance; i.e., the distance between two
points that stays on the surface, rather than the Euclidean distance. As an
 8.2 Other Dimensionality Reduction Techniques 7L5

**
o.4 * x*
lEn * * )t(
UU
n_ *
o u-z UN * * *
3
tr
D n *__. n *
tsn E
f
E
"n
tr
h#..#"
D* iF
2 -o.z tr tr-"E
r
.
1 F ^x x
ti.*x
x+*
c tr*
8 -0.+ ** *
o
a * It(

-0.6 tro
*

tr

-1.2
-4-3-2-101234
First New Attribute

Figure8.7. Plotof theflower

ofthelrisdatasetbasedontwonewattributes
fromISOMAP.

example, the Euclidean distance between two cities on opposite sides of the
Earth is the length of a line segment that passesthrough the Earth, while the
geodesicdistance between two cities is the length of the shortest arc on the
surface of the Earth.
Example El.6. ISODATA was used to project the Iris data into two dimen-
sions. See Figure B.7. The result is similar to previous techniques.

8.2.4 Common Issues

As with other data analysis techniques, we can distinguish between different

dimensionality techniques in a number of areas. One key issue is the quality
of the result: Can a technique produce a reasonably faithful representation of
the data in a lower-dimensional space? Does this representation capture the
characteristicsof the data that are important to the intended application (e.g.,
clusters), while eliminating aspects that are irrelevant or even detrimental
(e.g.,noise)?
To a large extent, the answer depends on the kind of data and data dis-
tributions that can be analyzed by the dimensionality reduction approach.
Techniquessuch as PCA, SVD, and factor analysis assumethat there is a lin-
ear relationship between the old and new sets of attributes. Although this may
716 Appendix B Dimensionality Reduction

be approximately true in many cases,there are many caseswhere a non-linear

approach is necessary. In particular, algorithms such as ISOMAP and LLE
have been developed to deal with nonlinear relationships.
The time and space complexity of dimensionality reduction algorithms is
a key issue. Most of the algorithms that we have discussedhave time and/or
space complexity of O(m2) or higher, where rn is the number of objects. This
limits their applicability to larger data sets, although sampling can sometimes
be used quite effectively. FastMap is the only algorithm presented here that
has linear time and space complexity.
Another important aspect of dimensionality reduction algorithms is whether
they produce the same answer every time they are run. PCA, SVD, and LLE
do. Factor analysis and the MDS techniques can produce different answers
on different runs. Many of the techniques that we did not discuss also have
this characteristic because they try to optimize some objective, and this re-
quires a search that may become trapped in a local minimum. Search-based
approachescan also have poor time complexity.
Finally, a key issue is determining the number of dimensions for the di-
mensionality reduction. The techniques that we have consideredcan typically
perform a dimensionality reduction to almost any number of dimensions. The
goodnessof the reduction is typically measured by some quantity that can be
plotted, as in a screeplot. In some cases,this curve provides a clear indication
of the intrinsic dimensionality. In many other situations, a choice needs to
be made between a smaller number of dimensions and a larger approximation
error) and a smaller approximation error and more dimensions.

B.3 Bibliographic Notes

Dimensionality reduction is a broad topic, and the relevant references are
scattered across many fields. A comprehensive discussion of PCA can be
found in the book by Jolliffe [531], while an introduction to SVD is given by
Demmel [527] and other linear algebra texts. Kernel PCA is described by
Scholkopf et al. [534]. Many books on multivariate statistical analysis, such
as that by Anderson 15241,also include discussionson PCA, as well as factor
analysis. More details on MDS can be found in the book by Kruskal and
Wish [532]. The FastMap algorithm was proposed by Faloutsos and Lin [529].
The papers for LLE (Roweis and Saul [535]) and ISOMAP (Tenenbaum et al.
[533]) appeared in the same issue of Sc'ience.MATLAB code for the ISOMAP
and LLtr algorithms is available on the Web. Other articles that may be of
 BIBLIOGRAPHY 7L7

interest include those by M. Belkin and P. Niyogi [525], Donoho and Grimes
[528],and Ye et al. [536, 537]
There are many other techniques that are often used for dimensionality
reduction or are strongly related to it. These areas include principal curves
and surfaces, non-linear PCA (including neural network approaches),vector
quantization, random projections, Independent Components Analysis (ICA),
Self-OrganizingMaps (SOM), projection pursuit, regression-basedapproaches,
genetic algorithms, and optimization-based approaches such as simulated or
deterministic annealing. Descriptions of these areas and additional references
can be found in two surveys on dimensionality reduction by Fodor [530] and
Carreira-Perpinan[526]. SOM is discussedin Section9.2.3.

Bibliography
1524] T. W. Anderson. An Introduct'ion to Multi,uariate Statistical Analysis. Wiley, 2nd
edition, July 2003.
15251 M. Belkin and P. Niyogi. Laplacian eigenmaps for dimensionality reduction and data
representation. Technical Report TR 2002-01, Department of Computer Science and
Statistics, University of Chicago, Jarluary 2002.
1526] M. A. Carreira-Perpinan. A Review of Dimension Reduction Techniques. Technical
Report CS-96 09, Dept. of Computer Science, University of Sheffield, January 1997.
15271J. W. Demmel. Appli,ed Numer,ical L,inear Algebra. SIAM Press, September 1997.
15281D. L. Donoho and C. Grimes. Hessian eigenmaps: Locally linear embedding techniques
for high-dimensional data. PNAS, 100(10):5591-5596, 2003.
[529] C. Faloutsos and K.-I. Lin. FastMap: A Fast Algorithm for Indexing, Data-Mining
and Visualization of Traditional and Multimedia Datasets. In Proc. of the 1995 ACM
SIGMOD IntI. Conf. on Management of Data, pages 163-174, San Jose, California,
June 1995.
[530] I. K. Fodor. A survey of dimension reduction techniques. Technical Report UCRL-ID-
t48494, LLNL, June 2002
f5311 L T. Jolliffe. Principal Component Anallls'is. Springer-Verlag, 2nd edition, October
2002.
[532] J. B. Kruskal and M. Wish. Multi.dzmensional Scali,ng. SAGE Publications, January
7978.
1533] S. T. Roweis and L. K. Saul. Nonlinear Dimensionality Reduction by Locally Linear
Embedding. Science, 290( 5500):2323-2326, 2000.
[534] B. Schcilkopf, A. J. Smola, and K.-R. Miiller. Nonlinear Component Analysis as a
Kernel Eigenvalue Problem. N eural Computat'ion, 10(5) : 1299-1319, 1998.
[535] J. B. Tenenbaum, V. d. Silva, and J. C. Langford. A Global Geometric Framework for
Nonlinear Dimensionality Reduction Science, 290(5500):2319-2323, 2OOO.
[536] J. Ye, R. Janardan, and Q. Li. GPCA: an efficient dimension reduction scheme for
image compression and retrieval. In Proc. of the 1Oth Intl. Conf. on Knowledge Discouery
and"Data Min'ing, pages 354 363, Seattle, Washington, August 2004. ACM.
[537] J. Ye, Q. Li, H Xiong, H. Park, R. Janardan, and V. Kumar. IDR/QR: an incremental
dimension reduction algorithm via QR decomposition. In Proc. of the 1Oth Intl. Conf.
on Knowledge Dzscouery and Data Mining, pages 364-373, Seattle, Washington, 2004.
ACM.
Probabilityand
Statistics
This appendix presentssome of the basic conceptsin probability and statistics
used throughout this book.

C.1 Probability
A random experirnent is the act of measuring a process whose outcome
is uncertain. Examples include rolling a die, drawing from a deck of cards,
and monitoring the types of traffic across a network router. The set of all
possible outcomes of a random experiment is known as the sample space,
f). For example,O: {1,2,3,4,5,6} is the samplespacefor rolling a die. An
event .D correspondsto a subset of these outcomes, i.e., E e f). For example
E : {2,4,6} is the event of observing an even number when rolling a die.
A probability P is a real-valued function defined on the sample space Q
that satisfiesthe following properties:

1. For any event E C 0, 0 < P(E) < 1.

2. P(Q) :7.

3. For any set of disjoint events, Er, E2, . . . , Ep e {'1,

kk
.p, Y
t l " ,I 'R
- . ' r: \ - p r\ n
u .t tr .
,?t

The probability of an event E, which is written as P(E), is the fraction of

times event-E is observedin a potentially unlimited numberof experiments.
72O Appendix C Probability and Statistics

In a random experiment, there is often a quantity of interest we want to

measure; e.g., counting the number of times a tail turns up when tossing a
coin fifty times or measuring the height of a person taking a roller coaster ride
at a theme park. Since the value of the quantity depends on the outcome of a
random experiment, the quantity of interest is known as a random variable.
The value of a random variable can be discrete or continuous. A Bernoulli
random variable, for example, is a discrete random variable whose only possible
values are 0 and 1.
For a discrete random variable X, the probability X takes on a particular
value z is given by the total probability of all outcomes e in which X(e) : v:

P ( X : u ) : P ( A : { e l ee Q ,X ( e ) : , } ) . (c.1)

The probability distribution of a discrete random variable X is also known as

its probability mass function.
Example C.l-. Consider a random experiment where a fair coin is tossed
four times. There are 16 possibleoutcomes of this experiment: HHHH, HHHT,
HHTH, HTHH, THHH, HHTT, HTHT, THHT, HTTH, THTH, TTHH, HTTT,
THTT, TTHT, TTTH, and TTTT, where H (T) indicates that a head (tail)
is observed. Let X be a random variable that measuresthe number of times
a tail is observedin the experiment. The five possible values fot X are 0, 7) 2)
3, and 4. The probability mass function for X is given by the following table:

X 0 1 2 3 4
P(X) rl76 4 l 1 6 6 1 7 6 4l16 1 1 1 6

For example, P(X - 2) :6/16 becausethere are six outcomesin which the
tail is observed twice during the four tosses. r

On the other hand, if X is a continuous random variable, then the proba-

bility that X has a value between a and b is

P(o<r<b): f (r)dr (c.2)

The function f (r) is known as the probability density function (pdf). Be-
cause / is a continuous distribution, the probability that X takes a particular
value r is always zero.
 C.1 Probabilitv 72L

of probability
TableC.1. Examples (f (n + 1) : nf (n) andf (1) : 1)
functions.

Probability F\rnction Parameters

, t\f, P)2
Gaussian plrl : --+- 'z o
VzTo
exp 11,o

Binomial p(r):(7)p"(t-p)n-" nrP

Poisson p(r) : j.0" exp o 0

Exponential P(r) : 0 exP-o' 0

Gamma p(r) : exp-)z \,d

{}r'-t
b l u-_ t
€xP -' -
I r/)
Chi-square P\:t ) : 2ro1-n7O"c'"1

Table C.1 shows some of the well-known discrete and continuous proba-
bility functions. The notion of a probability (mass or density) function can
be extended to more than one random variable. For example, 1f X and Y are
random variables, then p(X,Y) denotes their joint probability function. The
random variables are independent of each other if P(X,Y) : P(X) x P(Y).
If two random variables are independent, it means that the value for one vari-
able has no impact on the value for the other.
Conditional probability is another useful concept for understanding the
dependenciesamong random variables. The conditional probability for vari-
able Y given X, denoted as P()'lX), is defined as

P-(4J)
P f f t x 1, : (c.3)
P(X)

If X and Y are independent, then P(YIX) : P(Y). The conditional proba-

bilities P(YIX) and P(XIY) can be expressedin terms of one another using
a formula known as the Baves theorem:

P( xlY) P( Y)
P(Ylx) : (c.4)
P(X)

If {X1, Xz,...,Xn} is the set of mutually exclusiveand exhaustiveoutcomes

of a random variable X, then the denominator of the above equation can be
722 Appendix C Probabilitv and Statistics

expressedas follows:

kk
P(x): I P(x,Y): t P(xlV)P(Y). (c.5)
i:7 i:r

Equation C.5 is called the law of total probability.

C.1.1 Expected Values

The expected value of a function g of a random variable X, denoted as

Elg(X)], is the weighted-averagevalue of g(X), where the weights are given
by the probability function for X. If X is a discrete random variable, then the
exoected value can be computed as follows:

nlg(xl: s @ 6 ) P :( ,x0 ) . (c.6)

T
On the other hand, if X is a continuous random variable,

nlg(x)l: (x)d.x, (c.7)

I:s@) f
where /(X) is the probability density function for X. The remainder of this
section considersonly the expected values for discrete random variables. The
corresponding expected values for continuous random variables are obtained
by replacing the summation with an integral.
There are severalparticularly useful expected values in probability theory.
First, if g(X): X, then

px: ElXl: ri P(X : ro). (c.8)

t
This expected value corresponds to the mean value of the random variable
X. Another useful expected value is when g(X) : (X - p,y). The expected
value of this function is

o 2 *: o 1 1 x- t t x ) ' l : \ ( r i - I r x ) 2P ( x : r i ) . (c.e)

This expected value corresponds to the variance of the random variable X.

The square root of the variance corresponds to the standard deviation of
the random variable X.
 C.2 Statistics 723

Example C.2. Consider the random experiment described in Example C.1.

The average number of tails expected to show up when a fair coin is tossed
four times is

t r x : 0 x 1 l 1 6 * 1 x 4 l L 6 + 2x 6 l 1 6 + 3x 4 1 1 6 -4t x r f 1 6 : 2 . (C.10)

The variance for the number of tails expected to show up is

: (0 _ 2)' xLl76+ (1 - 2)2x 4116+(2 -2)2 x 6lL6

"2y
+ ( 3 - 2 ) 2x a l 7 6+ ( 4 - 2 ) 2x t f t o : t .

For pairs of random variables, a useful expected value to compute is the

covariance function. Cou. which is defined as follows:

Cou(X,Y): El(X - px)(Y - PY)l (c.11)

Note that the variance of a random variable X is equivalent Cou(X, X). The
expected value of a function also has the following properties:

t. E[a]: a, if a is a constant.

2 . El a x l : a E l X l .
3. ElaX + bYl : aElXl + bElYl.
Based on these properties, Equations C.9 and C.11 can be rewritten as follows:

: n11x- px)21:E[x2]- Elxl' (c.12)

"\
Cou(X,Y): EIXY)- ElXlElYl (c.13)

C.2 Statistics
To draw conclusionsabout a population, it is generally not feasible to gather
data from the entire population. Instead, we must make reasonableconclusions
about the population based on evidence gathered from sampled data. The
process of drawing reliable conclusions about the population based on sampled
data is known as statistical inference.
724 Appendix C Probability and Statistics

C.z.I Point Estimation

The term statistic refers to a numeric quantity derived from sampled data.
Two examples of useful statistics include the sample mean (z) and the sample
variance (s|):

'\-r.
1N
(c.14)
N 'L"o
;-1

1 '\tr
r
.Sv : \-/ l-. - -t2 (c.15)
A/ - 1 /-t\1,x
i:I

The processof estimating the parameters of a population using sample statis-

tics is known as point estimation.
Example C.3. Let Xr, Xz, . . . , XN be a random sample of l/ independent
and identically distributed observations drawn from a population with mean
p,y and variance o2*. t'et 7 be the sample mean. Then

Ewl:"[# # t : t,x,
Elxo]:f ' Nt,x (c.16)
\",):
where E[Xd - px since all the observationscome from the same distribution
with mean p76. This result suggeststhat the sample mean 7 approachesthe
population mearr p,x) especiaily when ly' is sufficiently large. In statistical
terms, the sample mean is called an unbiased estimator of the population
mean. It is possible to show that the variance of the sample mean is

alfz- ur"rf]:o?tN (c.17)

Becausethe population variance is usually unknown, the variance of the sample

mean is often approximated by replacing o| with the sample variance s2r. The
quantity sy lfi is known as the standard error of the mean. r

C.2.2 Central Limit Theorem

The normal distribution is perhaps one of the most widely used probability
distributions becausethere are many random phenomenathat can be modeled
using this distribution. This is a consequenceof a statistical principle known
as the central limit theorem.
 C.2 Statistics 725

Theorem C.1- (Central Limit Theorem). Consider a random sample of

si,zeN drawn from a probabi,Ii,tydi,stri,buti,onwith mean p,x and uariance o2*.
If E is the sample n'Lean,then the di,stri,buti,onof 7 approachesa normal dis-
tribution with mean p,y and uariance when the sample size i,s large.
"2xlN
The central limit theorem holds regardlessof the distribution from which
the random variable is drawn. For example, suppose we randomly draw l/ in-
dependent examples from a data set with an unknown distribution. Let X.i be
a random variable that denotes whether the ith example is predicted correctly
by a given classifier; i.e., Xi: 1 if the example is classified correctly, and 0
otherwise. The sample mean, X, denotes the expected accuracy of the classi-
fier. The central limit theorem suggeststhat the expected accuracy (i.e., the
sample mean) tends to be normally distributed even though the distribution
from which the examples are drawn may not be normally distributed.

C.2.3 fnterval Estimation

When estimating the parameters of a population, it is useful to indicate the re-
liability of the estimate. For example, supposewe are interested in estimating
the population mean p,y from a set of randomly drawn observations. Using a
point estimate such as the sample mean, r, may not be sufficient, especially
when the sample size is small. Instead, it may be useful to provide an interval
that contains the population mean with high probability. The task of esti-
mating an interval in which the population parameter can be found is termed
interval estimation. Let 0 be the population parameter to be estimated. If

P(h<g<9il:7-d., (c.18)

then (91,02) is the confidence interval for 0 at a confidence level of 1- a.

Figure C.1 shows the 95% confidence interval for a parameter derived from a
normal distribution with mean 0 and variance 1. The shaded region under the
normal distribution has an area equal to 0.95. In other words, if we generate
a sample from this distribution, there is a g57o chance that the estimated
parameter falls between -2 and *2.
Considera sequenceof randomly drawn observations,Xl, X2,..., X1,'. We
would like to estimate the population mean, ,u;, based upon the sample mean,
T, at 68Toconfidence interval. According to the central limit theorem, 7 ap-
proaches a normal distribution with mean px and variance when.ly'
"klN
is sufficiently large. Such a distribution can be transformed into a standard
normal distribution (i.e., a normal distribution with mean 0 and variance 1)
726 Appendix C Probability and Statistics

o4

0.3

0.25

o2

015

o.'r

0.05

-4 -3 -2-101234

Figure
C.1,Confidence ofa parameter.
interval

in the following way.

r-1"
Z:r-ltx x =N(0.1), (c.1e)
ox/t/N sx/{N

where the population standard deviation is approximated by the standard

error of the sample mean. Fhom the probability table of a standard normal
distribution, P(-L < Z < 1) :0.69. The probability can be rewritten in the
following way:

P(-sy IJN < r - p,x< sx/'/N) : 0.68,

or equivalently,

P(T - sxlt/N 1 px <E + sxlJN) :0.08.

Therefore, the 68% confidenceinterval for p,ssis r* sxlr/N.

C.3 Hypothesis Testing

Hypothesis testing is a statistical inference procedure to determine whether a
conjecture or hypothesis should be accepted or rejected based on the evidence
gathered from data. Examples of hypothesis tests include verifying the quality
of patterns extracted by data mining algorithms and validating the significance
of the performance difference between two classification models.
 C.3 Hypothesis Testing 727

In hypothesis testing, we are usually presented with two contrasting hy-

potheses,which are known, respectively, as the null hypothesis and the al-
ternative hypothesis. The general procedure for hypothesis testing consists
of the following four steps:
1. Formulate the null and alternative hypothesesto be tested.

2. Define a test statistic 0 that determines whether the null hypothesis

should be accepted or rejected. The probability distribution associated
with the test statistic should be known.

3. Compute the value of 0 from the observeddata. Use the knowledgeof the
probability distribution to determine a quantity known as the p-value.

4. Define a significance level, a, which controls the range of 0 values in

which the null hypothesis should be rejected. The range of values for I
is known as the rejection region.
Consider an association pattern X derived using the algorithms presented
in Chapter 6. Supposewe are interested in evaluating the quality of the pattern
from a statistical perspective. The criterion for judging whether the pattern
is interesting depends on a quantity known as the support of the pattern (see
Equation 6.2), s(X). Support measuresthe fraction of records in which the
pattern is actually observed. X is considered interesting if s(X) > m'insup,
where mi.nsup is a user-specified minimum threshold.
The problem can be formulated into the hypothesis testing framework in
the following way. To validate the pattern X, we need to decide whether to
accept the null hypothesis, Hs: s(X) - m'insup, or the alternative hypothesis
H1 : s(X) > minsup. If the null hypothesis is rejected, then X is considered
an interesting pattern. To perform the test, the probability distribution for
s(X) must also be known. We can apply the binomial distribution to model
this problem because determining the number of times pattern X appears in
.ly' records is analogous to determining the number of heads that shows up
when tossing I{ coins. The former can be described by a binomial distribution
with mean s(X) and variances(X) x s(X)lN. The binomial distribution can
be further approximated using a normal distribution if l/ is sufficiently large,
which is typically the case in most market basket analysis problems.
Under the null hypothesis, s(X) is assumed to be normally distributed
with mean minsup and variance m,insup x minsupfli. To test whether the
null hypothesis should be accepted or rejected, the following Z-stalistic can
be used:
s(X) - m'insup
Z- (c.20)
1f-minsupx (1- minsup)lN
728 Appendix C Probability and Statistics

Z has a standard normal distribution with mean 0 and variance 1. The statistic
essentiallymeasuresthe differencebetween the observedsupport s(X) and the
m'insup threshold in units of standard deviations. Let ltr : 10000, s(X) :
17%o,and mi,nsup : 7070. The Z-statistic under the null hypothesis is Z :
(0.11 - 0.1)//0.09n0-000 : 3.33. From the probability table of a standard
normal distribution, a one-sided test witln Z : 3.33 correspondsto a p-value
of 4.34 x 10-4.
Suppose a : 0.001 is the desired significance level. a controls the prob-
ability of falsely rejecting the null hypothesis even though the hypothesis is
true (in the statistics literature, this is known as the Type 1 error). For ex-
ample, an a value of 0.01 suggeststhat there is one in a hundred chance the
discovered pattern is spurious. At each significance level a, there is a corre-
sponding threshold Zo, srch that when the Z value of a pattern exceeds the
threshold, the pattern is consideredstatistically significant. The threshold Zo
can be looked upin a probability table for the standard normal distribution.
For example, the choice of a :0.001 sets up a rejection region with Zo: 3.09.
Since p < o, or equivalently, Z ) Zo, lhe null hypothesis is rejected and the
pattern is consideredstatistically interesting.
Regression
Regression is a predictive modeling technique where the target variable to
be estimated is continuous. Examples of applications of regression include
predicting a stock market index using other economic indicators, forecasting
the amount of precipitation in a region based on characteristics of the jet
stream, projecting the total sales of a company based on the amount spent
for advertising, and estimating the age of a fossil according to the amount of
carbon-14 Ieft in the organic material.

D.1 Preliminaries
Let D denote a data set that contains ly' observations,

D : { ( x t , a t ) li : 1 , 2 , . . . ,l'r}.

Each xa correspondsto the set of attributes of the ith observation (also known
as the explanatory variables) and !, correspondsto the target (or response)
variable. The explanatory attributes of a regressiontask can be either discrete
or continuous.
Definition D.1 (Regression). Regressionis the task of learning a target
function / that maps each attribute set x into a continuous-valuedoutput 3t.

The goal of regression is to find a target function that can fit the input
data with minimum error. The error function for a regressiontask can be
73O Appendix D Regression

expressedin terms of the sum of absolute or squared error:

AbsoluteError : - /(*r)l (D.1)

\,lru
1,

SquaredError : - f!r))2 (D.2)

D,@o
1,

D.2 Simple Linear Regression

Consider the physiological data shown in Figure D.1. The data correspondsto
measurementsof heat flux and skin temperature of a person during sleep. Sup-
pose we are interested in predicting the skin temperature of a person based on
the heat flux measurementsgenerated by a heat sensor. The two-dimensional
scatter plot shows that there is a strong linear relationship between the two
variables.

HeatFlux SklnTemDerature HeatFlux SkinTemperature Heat Flux SkinTemperature

tu.oco 5f ,uvz 6.3221 31.581 JIZZl
1 06 1 7 31.021 6.0325 31.618 4-2951 32.259
10.183 31.058 5.7429 31-674 4.2469 32.296
9.7003 31.095 5.5016 31 712 4 0056 32.334
Y OCZ 31.133 5 2603 31.768 3 716 32.391
10 086 31 '188 5 1638 31 825 3 523 32.448
9.459 31 226 5 0673 31.862 3 4265 32.505
I 3972 31 263 4.9708 31.919 33782 32543
7 6251 3 13 1 9 48743 31_975 3 4265 32.6
71907 31 356 4.7777 32.013 3.3752 32.657
7.046 31.412 4.7295 32-O7 3.3299 32 696
6.9494 31.468 4.633 32.126 3 3299 32.753
6.7081 31.524 A AAA' 32.164 3 4265 32-791

32.5

I
f

932
o
o
E
o
-
c 31.5
a
+
+T + .

3U.5
3 4 5 6 7 I 9 10 11
Heat Flux

FiguteD.1,Measurements ofa person.

ofheatfluxandskintemperature
 D.2 Simple Linear Regression 73L

D.2.1 Least Square Method

Suppose we wish to fit the following linear model to the observed data:

f (r) : uLr + ao) (D.3)

where aJgand url are parameters of the model and are called the regression
coefficients. A standard approach for doing this is to apply the method of
least squares, which attempts to find the parameters (r,.16,&r1)
that minimize
the sum of the squared error
NN
:D[vo - a1r- u012,
ssE :Llou - f ("0)12 (D.4)
;-1 i:I

which is also known as the residual sum of squares.

This optimization problem can be solved by taking the partial derivative
of -E with respect to c.rgand c,,r1,setting them to zero, and solving the corre-
sponding system of linear equations.

AI
OE - Z^ \\-.
lUr-arri-uo] :0
0uo ;-1

-lV
AE ^ S-.
-Z\[Ut - arri - ws]ri: Q (D.5)
0at
i:l

These equations can be summarized by the following matrix equation'

which is also known as the normal equation:

(::) : 1r,D::;,)
(d",?,u) (D.6)

Since!o rt : 229.9,Dt"? - 1569.2,DtAt : L242.9,and !, niyi : 7279.7,

the normal equationscan be solvedto obtain the followingestimatesfor the
parameters.
-'
:
39 229.9\ 1 |Z+Z.O
\
(;: ( 22s.sr56s.2
) ) \ zzzo.r)
:(
0.1881 -0.0276\ / 1242.e\
-0.0276 o.oo47) \ zzzs.z)
33.1699
-0.2208 \I
:(
732 Appendix D Regression

Thus, the linear model that best fits the data in terms of minimizing the
SSEis
f (r) :33.17 - 0.22n-
Figure D.2 showsthe line corresponding
to this model.

SkinTemp= 33.t2- 0.22 Heat Flux

q)

832
o
c
o)
F
.E 31.5
x
a

\.. .

30.5
678 I 10 11
Heat Flux

FigureD.2.A linearmodel
thatfitsthedatagiveninFigure
D.1.

We can show that the general solution to the normal equations given in
D.6 can be expressed as follow:

wu u-afx
onA
wl (D.7)
ort

w h e r e7 : D t q l N , y:Dryif N, and

\-r -\ /
ors : ) .\rt-r)\h-A) (D.8)
x

orr : \t*o-z)' (D.e)

1,
[-. .2
ova : )-\ut-T) (D.10)
L
 D.2 Simple Linear Regression 733

Thus, linear model that results in the minimum squared error is given by

f (*) :g" + lzY

orr'
l" -zl. (D.11)

In summary, the least squares method is a systematic approach to fit a lin-

ear model to the responsevariable g by minimizing the squared error between
the true and estimated value of g. Although the model is relatively simple, it
seems to provide a reasonably accurate approximation because a linear model
is the first-order Taylor series approximation for any function with continuous
derivatives.

D.2.2 Analyzing Regression Errors

Some data sets may contain errors in their measurements of x and y. In
addition, there may exist confounding factors that affect the response variable
g, but are not included in the model specification. Becauseof this, the response
variable grin regressiontasks can be non-deterministic, i.e., it may produce a
different value even though the same attribute set x is provided.
We can model this type of situation using a probabilistic approach, where
g is treated as a random variable:

/(*)+ [s- /(")]

/(x)+ e. ( D.12)

Both measurement errors and errors in model specification have been absorbed
into a random noise term, e. The random noise present in data is typically
assumedto be independent and follow a certain probability distribution.
For example, if the random noise comes from a normal distribution with
zero mean and variance o2, then

1 - ts-J(**o)]2
P ( el x ,O ) ----: eXD 2oz (D.13)
Jhro2
1-
log[P(elx, o)] -:(u - f(x. Q))' * constant (D.14)
2',"

This analysis showsthat minimizing the SSE, [g - f (*,O]2, impticitly assumes

that the random noise follows a normal distribution. Furthermore, it can be
shown that the constant model, /(*, A) : c, that best minimizes this type of
error is the mean, i.e., c : y.
734 Appendix D Regression

Another typical probabilitymodelfor noiseusesthe Laplaciandistribution:

P ( e l x ,Q ) (x'ft) |
c exp-clu-"f (D.15)
Iog[P(e lx,0)] -clA - /(*, CI)l* constant (D.16)

This suggeststhat minimizing the absolute error lA - f 6,O)l implicitly as-

sumes that the random noise follows a Laplacian distribution. The best con-
stant model for this case corresponds to /(x, Q) : A, the median value of
a.
Besidesthe SSE given in Equation D.4, we can also define two other types
of errors:

ssr : \,(uo-d, (D.17)

ssM : \,(f("0)-il, (D.18)

where ,9^9? is known as the total sum of squares and SSM is known as the
regression sum of squares. S^97 represents the prediction error when the
average value g is used as an estimate for the response variable. SSM, on
the other hand, representsthe amount of error in the regressionmodel. The
relationship among SST, SSE, and SSM is derived as follows:

ssE : -T +T - f @)12
Dlro
: -s)2+DVI-S -sl2+z\(u, -r)@- f (re))
\W,
: Llro -vl2+Dlr(,,) -sl' - z\(u - fiay(r;-r)
ii1.

: Dlro-sl' +DIti,,o)-sl' - zlal@r -z)2

iii

: -\lf @ r-)d 2
Dlr'-Tl2
it
: SST _ SSM ( D.1e)

wherewe have applied the following relationships:

-o.'l\ri/ - \ - I
T-f@r) )

Dlr -ellq -nl - -Ir.

\-r t2
ora LDlorn : u7 ) Lri
 D.2 SimpleLinear Regression 735

Thus,we can write ^9S7: SSE + SSM.

D.2.3 Analyzing Goodness of Fit

One way to measure the goodness of the fit is by computing the following
measure:
p2- SSM -el'
Dnlf@o) (D.20)
SSr Dtlat-Tl"
The R2 (or coffici,ent of determinati,on) for a regression model may range
between 0 and 1. Its value is close to 1 if most of the variability observed in
the responsevariable can be explained by the regressionmodel.
-R2 is also related to the correlation coefficient. r. which measures the
strength of the linear relationship between the explanatory and response vari-
ables
u r|.t
T:--2. (D.21)
t/o'tona
From Equations D.9, D.10, and D.11, we can write

-s)'
Dolf@u)
R2: -g)'
Dulao
Dolffi@-E))"
oaa

(D.22)

The above analysis shows that the correlation coefficient is equivalent to the
square root of the coefficient of determination (except for its sign, which de-
pends on the direction of the relationship, whether positive or negative).
It is worth noting that R2 increases as we add more explanatory variables
into the model. One way to correct for the number of explanatory variables
added to the model is by using the following adjusted fi2 measure:

R2: r- (ffi)
Adjusted $ - R2), (D.23)
736 Appendix D Regression

where l/ is the number of data points and d * 1 is the number of parameters

of the regressionmodel.

D.3 Multivariate Linear Regression

The normal equations can be written in a more compact form using the
following matrix notation. Let X : (1 x), where 1 : (1,1,1,...)" and
x: (rtrr2t. . . ,rN)T . Then, we can show that

X"X:( 1;l l;x) : (;,, B:#), Q24)

which is equivalent to the left-hand side matrix of the normal equation. Sim-
ilarly, if y : (At,Az,...,Ux)T, we can show that

( 1 * )'y: ( 1;l) : ( j,:;,), (D.25)

which is equivalent to the right-hand side matrix of the normal equation. Sub-
stituting Equations D.24 and D.25 into Equation D.6 we obtain the following
equation:
x"xe - x'y, (D.26)
where Q: (r0,c,,,1)". We can solve for the parameters in f) can as follows:

E2: (XrX)-tXty, (D.27)

The above notation is useful because it allows us to extend the linear

regressionmethod to the multivariate case. More specifically, if the attribute
set consistsof d explanatoryattributes (r1, r2t...,ra),X becomesan ly' x d
design matrix:

I rn rr2 r1.d
I rZt r22 r2d
(D.28)
1 rl,rt rN2 rNd.

while Q : (oo,(trt.. .,aa-t)r is a d-dimensionalvector. The parameterscan

be computed by solving the matrix equation given in Equation D.26.
 D.4 Alternative Least-Square Methods 737
Regression

D.4 Alternative Least-Square Regression Methods

The Ieast squares method can also be used to find other types of regression
models that minimize the SSE. More specifically, if the regressionmodel is

A : / ( x , C l )* e (D.2e)
: ,o+Iuagi(x)le, (D.30)

and the random noise is normally distributed, then we can apply the same
methodology as before to determine the parameter vector O. The g6'scan be
any type of basis functions, including polynomial, kernel, and other nonlinear
functions.
For example, suppose x is a two-dimensional feature vector and the regres-
sion model is a polynomial function of degree 2

f (rt,rz,0) : ,o I u1r1* azrz I wsr1r2 -l uarl * u5n?r. (D.31)

If we create the following design matrix:

I rn rr2 rrrr1z r?t *72

I rzr !x22 r2rr22 *3r rZz
X- (D.32)
I rNt rN2 rNtfrNz ,2Nt ,2Nz

where r;7 is the jth attribute of the zth observation, then the regressionprob-
lem becomesequivalent to solving Equation D.26. The least-squaresolution to
the parameter vector Q is given by EquationD.27. By choosing the appropri-
ate design matrix, we can extend this method to any type of basis functions.
Optimization
Optimization is a methodology for finding the maximum or minimum value of
a function. It is an important topic in data mining because there are many
data mining tasks that can be cast as optimization problems. For example,
the K-means clustering algorithm described in Section 8.2.1 seeks to find a
set of clusters that minimizes the sum of the squared error (SSE). Similarly,
the method of least squares presented in Section D.2.1 is designed to learn
the regressioncoefficients that minimize the SSE of the model. This section
presents a brief overview of the various techniques used to solve optimization
problems.

E.1 lJnconstrained Optimization

Suppose /(r) is a univariate function with continuous first-order and second-
order derivatives. In an unconstrained optimization problem, the task is to
locate the solution r* that maximizes or minimizes f (r) without imposing any
constraints on tr*. The solution r*, which is known as a stationary point,
can be found by taking the first derivative of / and setting it to zero:

dfl
o'
hl,__,.:
f (r*) can take a maximum or minimum value depending on the second-order
derivative of the function:

o r* is a maximum stationary point if #J a 0 at r : r*.

o u* is a minimum stationary point if # , O at tr : tr*.

74O Appendix E Optimization

o r* is a point of inflection when ffi: 0 at tr : tr*.

Figure E.1 illustratesan exampleof a function that containsall three station-
ary points (maximum,minimum, and point of inflection).

Pointof
Inflection

Minimum
Figure points
E.l. Stationary ofafunction.

This definition can be extendedto a multivariate function, f (rt, 12,..., rd),

where the condition for finding a stationary point **: [r{, ri,...,rfi]T is

(E.1)

However, unlike univariate functions, it is more difficult to determine whether

x* corresponds to a maximum or minimum stationary point. The difficulty
arisesbecausewe need to consider the partial derivatives for all possible
#J,
pairs of i, and j. The complete set of second-orderpartial derlvatives is given
by the Hessian matrix

a2f
"' }rt0rs
o'J
"' 0r20ra (8.2)
a2f
"' Ox40r4

A Hessian matrix H is positive definite if and only if xTHx > 0 for any
non-zero vector x. If H(x.) is positive definite, then x* is a minimum
stationary point.
 E.1 Unconstrained Optimization 74I

o A Hessian is negative definite if and only if x?Hx ( 0 for any non-zero

vector x. If H(x-) is negative definite, then x* is a maximum stationary
point.

o A Hessian is indefinite if x"Hx is positive for some value of x and neg-

ative for others. A stationary point with indefinite Hessian is a saddle
point, which can have a minimum value in one direction, and a maxi-
mum value in another.

Example E.1. Supposef (r,A) :3r2 *2A3 -2ry. Figtxe E.2 showsa plot of
this function. The conditions for finding the stationary points of this function
are

af 6r-2y:Q
0r
af 6 a '- 2 r : 0 (E.3)
0s
whosesolutionsaretr* : A* :0 or r* : L127,A* : Il9.

-2
v-
FigureE.2. f (",a):3r2 *2y3 -2ry.
Plotforthefunction
742 Appendix E Optimization

The Hessianof / is
H(r,a):
[j, ,;]
At r: A:0,
:
H(o,o)
lj, ?]
Since [r g] H(0,0) l" A]r :612 - 4rA :2r(3r _ 2A), which can be either
positive or negative, the Hessian is indefinite and (0,0) is a saddle point.
At r: I l 2 T ,y : I l g ,

: I u,
H(7l27,rlr)
L-a

Since[r y] H(I I 27,t I 9) [r a)r : 4r2- 2rs t +a2I J : 4(r - aI q2 + Ba2I 4 > 0
for non-zero z and y, the Hessianis positive definite. Therefore, (Il2T,1/g) is
a minimum stationary point. The minimum value of / is -0.0014. I

E.1.1 Numerical Methods

The preceding approach works if Equation E.l can be solved analytically for
x*. In many cases, finding analytical solutions is a very difficult problem,
thus necessitatingthe use of numerical methods to find approximate solutions.
Some of the numerical methods for finding the minimum value of a function
include golden search, Newton's method, and gradient descent search. While
the techniquespresentedhere are used to minimize the objective function /(x),
they are also applicable to maximization problems because a maximization
problem can be easily turned into a minimization problem by converting the
function /(x) to -/(").

Golden Search Consider the unimodal distribution illustrated in Figure

E.3, where the minimum value is bracketed between a and b. The golden
search method iteratively finds successively smaller brackets that contain the
minimum value until the interval width is small enough to approximate the
stationary point. To determine the smaller brackets, two additional points, c
and d, are chosenso that the intervals (a,c,d) and (c,d,b) have equal width.
Let c- a:b - d: a(b _ a) and d_ c: 0 x (b_ a). Therefore,

(b - d) + (d - c) + (c - o)
1- :e-1 0+a,
b-a
 E.l Unconstrained Optimization 743

ofa unimodalfunction.
FigureE.3.Example

or equivalently,
0:L-2a. (E.4)

The widths are also chosen to obey the following condition so that a re-
cursive procedure can be applied:

d-c c-a
r)
b-c o-a

or equivalently,
- - aa. (E.5)
l-a
Together, Equations E.4 and E.5 can be solved to yield c : 0.382 and B :
0.236. By comparing /(c) with /(d), it is possible to detect whether the mini-
mum value occurs in the interval (a,c,d) or (c,d,b). The interval that contains
the minimum value is then recursively partitioned until the interval width is
small enough to approximate the minimum value, as shown in Algorithm E.1.
The golden search method makes no assumption about the function, other
than it must be continuous and unimodal within the initial bracket [a,b]. lt
converges linearly to the solution for the minimum value.

Newtonts Method Newton's method is based on using a quadratic approx-

imation to the function f ("). BV using a Taylor series expansion of / around
744 Appendix E Optimization

Algorithm E.1 Golden search algorithm,

l: c: o+0.382(b-a).
2: whileb-a>e do
3: d,:b-0.382(b-a).
4: if f (d) > /(c) then
5: b: d.
6: else
7: a:qc:d.
8: end if
9: end while
10: return c.

zs, the following expressionis obtained:

f (*)= f (ro)+ @- ro)f' (*o)+ @+t f" (ro). (E.6)

Taking the derivative of the function with respect to r and setting it to zero
leads to the following equation:

'(") : '(ro) + (, - rg)

f f ftt(rs) : g
'(ro)
r : 1 0 _ 7f - l' 1il. (E.7)

Equation E.7 can be used to update r until it converges to the location of

the minimum value. It can be shown that Newton's method has quadratic
convergence, although it may fail to converge in some cases, especially when
the initial point rs is located far away from the minimum point. A summary
of this method is given in Algorithm E.2

Algorithm E.2 Newton's method.

1: Let as be the initial point.
2: while lf'(ro)l ) e do
rt (-^\
3: tr:ro-fffi.
4: tro: n.
5: end while
6: return r.

Newton's method can be extended to multivariate data by replacing the

first order derivative //(r) with the gradient operator V/(x) and the second
 E.1- Unconstrained Optimization 745

order derivative ftt(r) with the Hessian matrix fI:

x: x - r r - lv/( x) .

However, instead of computing the inverse of the Hessian matrix, it is easier

to solve the following equation:

Hz: -V/(*)

to obtain the vector z. The iterative formula for finding the stationary point
ismodifiedtox:x*2.

Gradient Descent Method Newton's method is one of severalincremental

methods to progressively locate the stationary point of a function using the
following update formula:
x:x+)9(x)), (E.8)
The function g(x) determines the direction in which the searchshould proceed
and .\ determines the step size.
The gradient descent method assumes that the function /(x) is differen-
tiable and computes the stationary point as follows:

x: x - )V/(x), (E.e)

In this method, the location of x is updated in the direction of the steepest

descent, which means that x is moved towards the decreasing value of /.
Section 5.4.2 described how the gradient descent method can be used to learn
the weight parameters of an artificial neural network. A summary of this
method is given in Algorithm E.3. Notice that the algorithm looks very similar
to AlgorithmE.2, except for the update formula.

Algorithm E.3 Gradient descent method.

1: Let xs be the initial point.
2: while llv/(*o)ll ) e do
3: tr:no-)V/(x).
4: fio: tr.
5: end while
6: return r.
746 Appendix E Optimization

8.2 Constrained Optimization

This section examineshow to solve an optimization problem when the variables
are subjected to various types of constraints.

8.2.L Equality Constraints

Consider the problem of finding the minimum value of f (ryn2t...,ra) sub-
jected to equaiity constraints of the form

'i: I,2,. ..
7 t ( x ): 0 , ,P.

A method known as Lagrange multipliers can be used to solve the constrained

optimization problem. This method involves the following steps:

1. Define the Lagrangian, I(x,,\) : /(x) +Dl:r\gilx), where )i is a

dummy variable called the Lagrange multiplier.

2. Set the first-order derivatives of the Lagrangian with respect to x and

the Lagrange multipliers to zero,

Vi:1,2,...,d
#:0,
and
AL
V z:i I ' 2 " " ' P '
N:0'
3. Solve the (d * p) equations in step 2 to obtain the stationary point x*
and the corresponding values for )6's.

The following example illustrates how the Lagrange multiplier method

works.
Example 8.2. Let f (r,g) : r l2A. Suppose we want to minimize the
function f (",A) subject to the constraint ,'+ A2 - 4 : 0. The Lagrange
multiplier method can be used to solve this constrained optimization problem
in the following way.
First, we introduce the Lagrangian

L ( r , A , , ) ): , I 2 y - t ) , ( r 2+ A 2- 4 ) ,
 8.2 Constrained Optimization 747

where .\ is the Lagrange multiplier. To determine its minimum value, we need

to differentiate the Lagrangian with respect to its parameters:

AL :
L l2),r :0 (E.10)
0r
AL :
2+2^a:0 (E.11)
0y
AL :
, , + a ,- 4 : o
0^
SolvingtheseequationsyieldsS: +t/51a.7 :121t/5, andg : a4lJB.
WhenA: l|l4, f (-2lr/b,-4lr/6) : -r0l\/5.Similarly,when) : -t/614,
f (21'/5,41t/5): L\lt/5.Thus, the functionf(r,g) hasits minimumvalue
at r: -2lt/5 andg: -41'/5.

8.2.2 Inequality Constraints

Consider the problem of finding the minimum value of.f (q,r2t..., z6) sub-
jected to inequality constraints of the form

ht@)< 0, i: t,2,...,e.

The method for solving this problem is quite similar to the Lagrange method
described above. However, the inequality constraints impose additional con-
ditions to the optimization problem. Specifically, the optimization problem
stated above leads to the following Lagrangian

ri : ?,, 9,,,
f 8) + l\ihi$). (E.12)
i.:r

and constraints known as the Karush-Kuhn-Tucker (KKT) conditions:

AL : ll Yi : I,2,...,d (E.13)
0u
hil*) V'i:1,2r...rQ (E.14)
),,i Y'i:1,2r...rQ (E.i5)
,\ah6(x) : 0, Y ' i : 7 , 2 , . . . r e . (E.16)

Notice that the Lagrange multipliers are no longer unbounded in the presence
of inequality constraints.
748 Appendix E Optimization

Example E.3. Suppose we want to minimize the function f (r,A) : -

@
7)' + (A - 3)2 subject to the following constraints:

r*y12, and y2r.

The Lagrangian for this problem is .L : (r - I)' + @- g)' + ),t(r i a -

2) + \2(r - g) subjected to the following KKT constraints:

Ar.
(E.17)
#:2(r-i)+)r*)z:o
AT,
F :2(a-3)*)r-lz:o (E.18)
oa
)1(z*A-2):0 (E.1e)
^ z ( r - A ): 0 (E.20)
)r)0,,\z)0, r+A<2,U2r (E.21)

To solve the above equations, we need to examine all the possible casesof
Equations E.19 and E.20.

Case 1: )r : 0, )z : 0. In this case,we obtain the following equations:

2(t- 1):0 and 2(Y-3):0,

whose solution is given by r : 1 and U : 3. Since z * A :4, this is not

a feasible solution becauseit violates the constraint r * y < 2.

Case 2: )r : 0, \z 10. In this case,we obtain the following equations:

r - a : 0 , 2 (r- 1 )+ ) z : 0, 2( a- 3) - \z : 0,

whose solution is given by r : 2, A : 2, and \z : -2, which is not a

feasible solution becauseit violates the conditions )2 ) 0 and r * y 1 2.

Case 3: \ 10, )z : 0. In this case,we obtain the following equations:

r+A-2:0,2(*- 1 )- | ) r : 0 , -2(r +1) +)r:0,

whose solution is given by r : 0, U :2, and \t : 2, which is a feasible

solution.
 8.2 Constrained Optimization 749

Case 4: \r # 0, \z 10. In this case,we obtain the following equations:

r+A-2:0, tr-A:0,2(r -1)+)r*lz :0,2(A -3)+)1-.\2: Q,

whose solution is r : L, a : 1, ,\1 : 2, and \2 : -2, which is not a

feasible solutiorr.

Therefore, the solution for this problem is r : 0 and A :2. I

Solving the KKT conditions can be quite a laborious task especially if the
number of constraining inequalities is large. In such cases,finding a closed-
form solution is no longer possible and it is necessary to use numerical tech-
niques such as linear and quadratic programming.
AuthorIndex
Abdulghani,.A., 399 Bay,S. D., 397,677
A b e ,N . , 3 1 5 Bayardo,R., 397
A b e l l o ,J . , 6 4 6 Beckman,R. J.,677
Abiteboul, S., 402 Belanche,L., 88
Abraham8 , .,677 B e l k i n ,M . , 7 1 7
Adams,N. M., 397 Ben-David,S., 16
A e l s t ,S . V . , 6 7 9 Benjamini,Y., 397
Agarwal, R. C., 314, 397, 678 Bennett,K.,312
Aggarwal,C. C., 397,677 Berkhin, P., 557
Agrawal,R., 14, 88, 197, 198, 397, 401, Berry, M. J. A., 14
402,47L,472, 644 Bertino, E., 16
A h a , D . W . , 1 9 6 ,3 1 2 B i l m e s J, . , 6 4 4
Aldenderfer,M. S., 556 Bins, M., 1-97
Alexandridis,M. G., 197 Bishop,C. M., 196,312,644
Ali, K., 397 Blashfield,R. K., 556
Allwein,E. L.,312 Blum,A.,86
Alsabti,K., 196 Bock, H. H., 87
Altman, R. 8., 15 Boley,D., 557, 558
Anderberg,M. R., 86, 556 Bolton, R. J., 397
Anderson,T. W., 7LT Borg, I., 87
Andrews,R., 312 Boswell,R., 312
Ankerst, M., 557 Bosworth,4., I4l
Antonie,M.-L.,47I Boulicaut,J.-F.,47L
Aone,C., 558 Bowyer,K. W., 312
Arabie,P., 557,558 Bradley,A. P., 312
A r n o l d ,4 . , 6 7 7 Bradley,P. S., 14, 403,557,645
A u m a n n ,Y . , 4 7 I Bratko, I., 197
Austin, J., 678 Breiman,L., L96,3L2
Ayres,J., 471 Breslow,L. A., 196
Breunig,M. M., 557,677
Bakiri, G., 313 Brin, S., 397,40L
Ball, G., 557 Brockwell,A., 16
Banerjee,4., 557 Brodley C. E., 198,679
Barbar6, D., 397, 644 Buntine, W., 196
Barnett,V.,677 B u r g e sC , . J. C.,3I2
Bastide,Y., 401 Burn, D. A., 140
Batistakis,Y., 557 Bykowski,L.,47I
Baxter, R. A., 678
 Author Index 75L

Cai, C. H., 397 Demmel,J. W., 87, 700,717

Campbell,C.,3L2 Dhillon,L S., 557
Cantri-Paz,tr., 196 Diday, E., 87
Card, S. K., l4l Diederich,J ., 3L2
Carreira-Perpinan,M. A., 7L7 Dietterich,T. G., 313,314
Carralho, C., 401 Ding, C., 403
Ceri, S., 400 Dokas,P., 398
Chakrabarti,S., 14 Domingos,P., 15, 196,313
Chan,P. K., 16, 313 Dong,G., 398
Chang,L., 680 D o n o h oD , . L.,7I7
Chatterjee,S., 646 Doster,W., 198
Chaudhary,A.,677 Dougherty,J., 87
Chaudhuri,S., 141 Drummond,C., 313
Chawla,N. V., 312,677 Dubes,R. C., 87, 558
Cheeseman, P.,645 D u d a ,R . O . , 1 5 , 1 9 6 ,3 1 3 ,5 5 7
Chen,B., 402 D u i n , R . P . W . , 1 9 6 ,3 1 5
Chen,C., 677 DuMouchel,W., 398
Chen,M.-S.,14, 40L,472 D u n a g a nJ, . , 6 7 7
C h e n ,Q . , 3 9 8 , 4 7 2 , 6 8 0 Dunham,M. H., 15, 313
Chen,S., 400 Dunkel,8., 398
Chen,S.-C.,680
Cheng,C. H., 397 Efron, B., 196
C h e n gH , .,471 Elkan,C., 313,557
Cherkassky,V., 15, 796,312 Elomaa,T., 87
Cheung,D. C., 398 Erhart, M., 646
Chiu, B., 678 Ertoz, L.,398, 645,679
Choudhary,A.,646 E s k i n ,E . , 6 7 7 , 6 7 9
C h u a n g4, , 6 7 7 Esposito,F., 196
Chung,S. M., 399 Ester,M., 557,558
Clark, P., 312 Everitt, B. S., 557
Clifton, C., 15, 402
Coatney,M.,40I,472 Fiirnkranz, J., 313
Cochran,W. G., 87 Fabris,C. C., 398
C o d d ,E . F . , 1 4 1 Faloutsos, C., 16, 88,717
C o d d ,S . B . , 1 4 1 Fan,W., 313
Cohen,P. R., 196 Fawcett,T.,3L4
Cohen,W. W., 312 Fayyad,U. M., 15, 87,l4l, 403,557,645
C o o k ,D . J . , 6 4 6 Feng,L., 398,402
C o o k ,R . D . , 6 7 7 Ferri,C., 313
Cooley,R., 398 Fisher, D., 557,645
C o s t ,S . , 3 1 2 Fisher,N. I., 398
C o u t o ,J . , 3 9 7 Fisher,R. A., 196
Cover,T. M., 312 Flach,P., 313
Cristianini, N., 313 Flannick, J., 47L
Flynn, P. J., 558
Dash,M., 87 F o d o r ,I . K . , 7 1 7
Datta, S., 16 Fovino,I. N., 16
D a v i e sL, . , 6 7 7 F o x ,A . J . , 6 7 8
Dayal,U., 398,472,559 Fraley,C.,645
752 Author Index

F Y a n kE, . , 1 6 , 3 1 5 Hall, L. O., 312

FbawleyW., 401 Hamerly,G., 557
FYeeman, D., 645 Hamilton, H. J., 399
Freitas,A. A., 398 Han, E.-H., 197,313,398, 399,47I,558,
Fleund, Y., 313 645,646
Fliedman,J. H., 15, 196,313,398,558 llan, J., 14, 15, 313, 398-403,477,472,
Fliendly, M., 141 558,646
Fu, A., 397,399 Hand, D. J., 15, 87, 313,397
F\r,Y., 399,47t Hardin, J., 678
Fukuda,T., 398, 400,47I H a r t , P . E . , 1 5 ,1 9 6 , 3 I 2 , 3 1 3 5
, 57
Fukunaga,K., 196,313 Hartigan, J., 558
F\rruichi,E., 401 Hastie,T., 15, 196,313,558
Hatonen,K.,402
Gada,D., 87 Hawkins,D. M., 678
Ganti, V., 196,645 Hawkins,S., 678
GaohuaGu, F. H., 87 He, H., 678
Garofalakis,M. N., 471 He, X., 403
G a t h e r ,U . , 6 7 7 He,Y.,4O2
Geatz,M., 16 Hearst,M., 313
Gehrke,J., L4, L6, L4l, L96, 471,644,645 Heath, D., 196
Gersema,8., L97 Heckerman,D., 314
Gersho,A., 645 Hernandez-Orallo, J., 313
Ghosh,A., 678 Hidber,C., 399
G h o s h J, . , 5 5 7 , 6 4 5 , 6 4 7 Hilderman,R. J., 399
Giannella,C., 15 Hinneburg,A., 646
Gibson,D., 645 Hipp, J., 399
Glymour,C., 15 H o , C . - T . ,1 6
Gnanadesikan, R., 678 Hoaglin, D., 141
Goil, S., 646 Hochberg,Y., 397
Golub,G. H., 700 Hodge,V. J., 678
Goodman,R. M., 315 Hofmann,H., 399
G o w d a ,K . C . , 6 4 5 Holbrook, S. R., 403
Grama,A., 16 Holder,L. 8., 646
Gray, J., 141 Holt, J. D., 399
Gray R. M., 645 Holte,R. C., 313,314
G r i m e sC , .,7t7 Hong,J., 314
Grinstein,G. G., 15, 141 Hornick,M. F., 15
Groenen,P., 87 Houtsma,M., 399
Grossman,R. L., 15, 646 H s i e h ,M . J . , 4 7 2
Grubbs, F., 678 Hsu, M., 398,472,559
Guan,Y., 557 Hsu, W., 400
Guha,S., 15,645 Huang,T. S., 646
Gunopulos,D., 398, 644 Huang, Y., 399
Guntzer,U., 399 Hubert, L., 557,558
Hubert, M., 679
Haight,R.,402 Hulten, G., 15
Halic,M., 197 Hussain,F., 87
Halkidi, M., 557 Hoppner,F., 646
Hall, D., 557
 Author Index 753

Iba, W., 314 Klooster,S., 401, 402,647

Imielinski, T., 397,399 Knorr, E. M., 678,679
Inokuchi,4.,399, 47I Kogan, J., 557
Irani, K. B., 87 Kohavi, R., 87, 197
Kohonen,T.,646
Jagadish,H. V., 678 K o I c z ,4 . , 3 L 2 , 6 7 7
Jain, A. K., 16, 87, 196,558 Kong, E. B:, 314
Jajodia,S., 397 Koperski,K., 399
Janardan,R.,7I7 Kosters,W. A., 399
Japkowicz,N., 312,314,677,678 Koudas,N., 678
Jardine,N., 558 Krciger,P.,646
Jaroszewicz, S.,399 Krantz, D., 87, 88
Jarvis, R. A., 646 Kriegel,H.-P.,557,558,646,677
Jensen,D., 88, 196 Krishna, G., 645
Jeudy,B., 471 Kruse, R., 646
John, G. H., 87 Kruskal, J. 8., 87, 717
Johnson,T., 678 Kubat, M., 314
J o l l i f f eI,. T . , 8 7 , 7 1 7 Kuhara,S., 401
Jonyer,L, 646 Kulkarni, S. R., 197
Joshi,A., 15 Kumar, V., 15, L96, 197,313,314, 398-
J o s h i ,M . V . , 1 9 6 ,I 9 7 , 3 I 4 , 4 7 L , 6 7 8 403, 47r, 472, 558, 559, 645-
647,678,679,7L7
Kailing,K.,646 Kuok, C. M., 399
K u r a m o c h iM , .,400,471
Kwok; I., 678
Kwong, W. W., 397

L a k h a l ,L . , 4 0 L
Lakshmanan,L. V. S., 400
Lambert,D., 16
, 471,472, Landau, S., 557
558,559,645,646 Landeweerd,G., 197
Kasif, S., 796, I97 Landgrebe,D., 197,198
Kass,G. V., 196 Lane,T., 679
Kaufman, L., 87, 558 Langford,J. C., 315,717
Kegelmeyer,P., 15, 646 Langley.P., 86, 314, 645
Kegelmeyer,W. P., 312 Larsen,8., 558
Keim, D. A., 646 Lavrac,N., 314
Kelly, J., 645 Law, M. H. C., 16
Keogh, E., 678 Layman, 4., L4L
K e o g h ,E . J . , 8 7 Lazarevic,A., 398, 679
Kettenring, J. R., 678 Lee,S. D., 398
Khardon, R., 398 Lee, W., 400, 679
Kifer, D., 16 L e e ,Y . W . , 8 8
Kim, B., 197 Leese,M., 557
Kitsuregawa,M., 401 Lent, B., 472
Klawonn,F.,646 Leroy,A. M., 679
Kleinberg,J. M., 558, 645 Lewis,D. D., 314
Klemettinen,M., 399, 402 Lewis, T., 677
754 Author Index

Li, J., 398 Mirkin, 8., 558

Li, q,',717 Mishra,N., 15, 645,646
Li, w., 88, 400,403 MitchellT , . , 1 6 ,L 9 7 , 3 1 4 , 5 5 8 , 6 4 6
Lin, K.-I., 400,7I7 Mobasher,B., 399, 645
L i n d e l l ,Y . , 4 7 1 Modha,D. S., 557
Lindgren.B. W., 87.14- Mok, K. W., 400,679
Linoff, G., 14 Molina, L. C., 88
Liu, B., 400,403,472 Moore,A. W., 558,677
Liu, H., 87, 88 Moret, B. M. E., 197
Liu, L.-M., 677 Morimoto,Y., 398, 400,47I
Liu, R. Y.,679 Morishita, S., 398, 471
Livny, M., 647 Mortazavi-Asl,8., 401,472
Lonardi, S., 678 Mosteller,F., 400
Lu, C.-T., 679 Motoda,H., 87, 88, 399,471
Lu, H. J., 398,40I,402 Motwani,R., 15, 397,40L,402,645
Lu, Y., 403 Mozetic,I., 314
Luce,R. D., 87, 88 Mueller, A., 400
Lugosi,G.,I97 Muggleton,S., 314
Muirhead, C. R., 679
Ma, Y., 400 Mulier, F., 15, 196,312
MacKinlay J. D., 141 Murtagh, F., 558, 646
MacQueen,J., 558 Murthy, S. K., 197
Madigan,D., 15 Murty, M. N., 558
Malerba,D., 196 Muthukrishnan,S., 678
Maletic,J. I., 400 Myneni,R., 401
Manganaris,S., 397 Miiller, K.-P".,717
Mangasarian,O., 314
Mannila, H., 15, 313, 398, 402,472 Nagesh,H., 646
M a o ,J . , 1 9 6 Nakhaeizadeh,G., 399
Maranell,G. M., 88 Namburu, R., 15, 646
Marchiori, E., 399 Navathe,5., 4Ol, 472
Marcus,A., 400 Nebot,A., 88
Margineantu,D. D., 314 Nemani,R., 401
Markou, M., 679 Nestorov,S., 402
Masand,B., 398 Ng, R. T., 400,646,677-679
Matsuzawa,H., 400 N i b l e t t ,T . , 1 9 7 , 3 I 2
Matwin, S., 314 Nishio, S., 401
Megiddo,N., 400 N i y o g i ,P . , 7 1 7
Mehta,M., 197,198
Meo, R., 400 O'Callaghan,L., 15, 645
Merugu,S., 557 Oates,T., 88
Meyer,G., 15 Oerlemans,A., 399
Meyerson,4., 15,645 Ogihara,M., 88, 403
Michalski,R. S., 197,314,646,647 Ohsuga,S., 403
Michie,D., 197 Olken,F., 88
Miller, H. J., 558 Olshen,R., 196
M i l l e r ,R . J . , 4 0 0 , 4 7 2 Omiecinski,E., 400, 401,472
Milligan,G. W., 558 Ono, T., 401
Mingers,J., 197 Orihara, M., 403
 Author Index 755

Osborne,J., 88 Redman,T. C., 88

Ozden,B., 400 Reichart,D., I4I
O z g t r ,A . , 4 0 0 , 6 7 9 Reina,C., 645
Reisende,M. G. C., 646
Padmanabhan,8., 403, 472 Rissanen,J.,I97
Palmer, C. R., 88 Rivest, R. L., 197
Pang, A., 400 Rocke,D. M., 678, 679
Papadimitriou,S., 16 Roiger,R., 16
Pardalos,P. M., 646 Romesburg,C., 558
Parelius,J. M., 679 Ron, D., 646
P a r k ,H . , 7 L 7 Ronkainen,P.,402
P a r k ,J . S . , 4 0 1 Rosenthal,4.,402
Parr Rud, O., 16 Rosner,B., 679
Parthasarathy,S., 88, 40I, 403,472 Rotem, D., 88
Pasquier,N., 401 Rousseeuw, P. J., 87, 558, 679
Patrick,E. A., 646 Rousu,J., 87
Pattipati, K. R., 197 Roweis,S. T., 717
Pazzani,M., 313, 397 Runkler,T.,646
Pazzan| M. J., 87 Ruts, L, 679
Pei, J., 15, 399-40I, 472
Pelleg,D., 558 Safavian,S. R., 198
Pellow,F., 141 Sahami,M., 87
Peters,M., 646 Salzberg,S., 196,197,312
Piatetsky-Shapiro,G., 15, 16, 401 Sander,J., 557, 558,677
Pirahesh,H., 1,4L Sarawagi,S., 401
Pison, G., 679 Sarinnapakorn,K., 680
P o r t n o yL, . , 6 7 7 , 6 7 9 Satou,K., 401
Potter, C., 401, 402,647 SaulL, . K.,717
Prasad,V. V. V., 397 Savaresi,S. M., 558
Pregibon,D., 15, 16,398 Savasere, 4.,40I,472
P r e r a u ,M . , 6 7 7 Saygin,Y., 16
Provenza,L. P., 16 Schrilkopf,B., 315
P r o v o s tF, . J.,88,314 Schaffer,C., 198
Psaila,G., 400 Schapire,R. E., 198,312,313
Puttagunta, V., 87 Scheuermann, P., 402
Pyle, D., 16 Schikuta,8.,646
Schroeder,M. R., 646
Quinlan,J. R., 197,314 Schuermann,J., 198
Schwabacher, M., 677
Raftery,A. E., 645 Schwartzbard,A., 678
Raghavan,P.,644,645 Scholkopf, 8.,7I7
Ramakrishnan,N., 16 Scott,D. W., 679
Ramakrishnan,R., 14, l4l, L96,645,647 Sebastiani,P., 314
Ramaswamy,S., 400, 679 Self,M., 645
Ramkumar,G. D.,401 Semeraro, G., 196
ILamoni,M.,314 Sendhoff,8.,644
Ranka, S., 196,401 Seno,M., 40I,472
Rao, C. R., 140 Shafer,J. C., 198,397
Rastogi,R., 471,645,679 Shasha,D. E., 16
756 Author Index

Shawe-Taylor,J., 313 S u z u k iE
, .,402
Sheikholeslami, G., 646 S v e n s e nM, . , 6 4 4
Shekhar,S., 399, 40L,403,679 Swami,4., 397, 399,472
Shibayama,G., 401 Swaminathan,R., 646
S h i m ,K . , 4 7 I , 6 4 5 , 6 7 9 Sykacek,P., 680
Shinghal,R., 399 Szalay,A. S., 677
Shintani,T., 401
Shneiderman,8., 141 Takagi,T., 401
Shoshani, A., 141 Tan, C. L., 87
Shyu,M.-L., 680 Tan, P.-N., 647
Sibson,R., 558 Tan, P. N., 197,398,400-403,472
Siebes,A. P. J. M., 399 Tang, S., 401
Silberschatz,A., 400, 401 Taouil, R., 401
S i l v a ,V . d . , 7 L T Tax, D. M. J., 315
Silverstein,C., 397, 40I T a y ,S . H . , 4 0 3 , 4 7 2
Simovici,D., 399 Taylor,C. C., 197
Simpson,E.-H.,402 Taylor, W., 645
Singer,Y.,3I2 T e n e n b a u mJ,. 8 . , 7 1 7
Singh,K., 679 T e n g ,W . G . , 4 7 2
Singh,L., 402 Theodoridis,Y., 16
Singh,S., 679 Thomas,S., 401
Singh,V., 196 Thompson,K.,3I4
Sivakumar,K., 15, 16 Tibshirani,R., 15, 196,313,558
Smalley,C. T., 141 Tickle,A., 312
Smola,A. J., 315,717 Timmers, "I ., 197
Smyth,P., 15, 16, 313,315 Toivonen,H., 16, 88, 398, 402,472
Sneath,P. H. A., 88, 558 Tokuyama,T., 398, 400,47L
Soete,G. D., 557,558 Torregrosa,4., 402
Sokal,R. R., 88, 558 Tsur, S., 397,40L,402
Soparkar,N., 398 T\rcakov,V.,679
Spence,R., 141 Tufte, E. R., 141
Spiegelhalter, D. J., 197 T\rkey,J. W., 88, L4L,679
Spiliopoulou,M., 398 T\rng, A., 402,558
Spd,th,H., 558 T\rzhilin, 4., 40I, 403,472
Srikant,R., 14,88, 397,400,402,477,472 Tversky,A., 87, 88
Srivastava, J., 398, 402,472,679
Steinbach, M., 197,401 403,51i9,64Et,647 U l l m a n ,J . , 3 9 7 , 4 0 2
S t e p p ,R . 8 . , 6 4 6 , 6 4 7 Uslaner,E. M., 87
S t e v e n sS, . S . , 8 8 Utgoff, P. E., 198
Stolfo,S. J., 313, 400,677,671) Uthurusamy,R., 15
Stone,C. J., 196
Stork,D. G., 15, 196,313,557' Vaidya, J., 15
Strang,G., 700 van Rijsbergen,C. J., 315
S t r e h l ,4 . , 6 4 7 van Zomeren,B. C., 679
Struyf,L.,679 Van Loan, C. F., 700
S t u t z ,J . , 6 4 5 Vapnik, V., 315
Suen,C. Y., 198 Vazirgiannis,M., 557
S:ln,2.,397 Velleman,P., 141
Suppes,P., 87, 88 Vempala,5.,677
 Author fndex 757

Venkatesh,S. S., 197 Yitt, T., 47I

Venkatrao,M., 141 Yoda,K., 400
Verkamo,A. I., 472 Yu, J. X., 398
Verykios,V. S., 16 Yu, L., 88
Virmani, A., 399 Yu, P. S., 14-16,397,401,677
von Seelen,W.,644
von der Malsburg,C.,644 Zaiane,O. R., 399,471
Vorbruggen,J. C.,644 Zadrozny,B., 315
Vu, Q., 402 Za}irn,C.'f., 559
Zaki, M. J., 16, 88, 403,472,646
Wang,H., 198 Zaniolo,C., 198
Wang,J. T. L., 16 Zhang,A.,646
Wang,K., 402,403,472 Z h a n g ,B . , 4 0 3 , 5 5 9
Wang,Q., 16 Zhang,C.,472
Wang,Q. R., 198 Zhang,H., 403,472
W a n g ,R . Y . , 8 8 Zhang,J.,313
Wang, W., 399 Zhang,N., 16
W a n g ,Y . R . , 8 8 Zhang, P.. 679
Washio,T.,399, 471 Zhang,5., 472
Webb,A. R., 16, 315 Zhang,T.,647
W e b b ,G . I . , 4 0 3 , 4 7 2 Zhang,Y.,403
Weiss,G. M., 315 Zhang,2." 403
Widmer, G., 313 Zhao,Y.,559
W i d o m ,J . , 4 7 2 Zhong,N., 403
Wierse,A., 15, 141 Zhtt, H., 40L
Wilhelm,A. F. X., 399 Ziad, M., 88
Williams,C. K. I.,644
Williams, G. J., 678
W i m m e r ,M . , 5 5 7
Wish, M., 717
Witten, I. H., 16,315
Wong, M. H., 399
Woodruff,D. L.,679
Wu, N., 397
Wu, X., 16,472

Xiang,D., 679
Xiong,H., 399,402,403,717
X u , X . , 5 5 7 ,5 5 8

Yamamura,Y., 401
Y a n ,X . , 1 5 ,4 0 3 ,4 7 L , 4 7 2
Yang, C., 403
Yang,Y., 400,472
Yao, Y. Y., 403
Y e ,J . , 7 I 7
Ye, N., 88, 645,680
Yesha,Y., 15
Yin, Y., 399
SubjectInclex
accuracy, 149,208 rare class, 654
activation function, 248 relative density, 669 670
AdaBoost, 288 algorithm, 669
adjacency matrix, 448 example, 669-670
aggregation, 45-47 semi-supervised, 656
anomaly detection statistical, 658-665
applications, 651-652 Gaussian, 659
approaches, 655-656 Grubbs, 681
semi-supervised, see anomal;r de- likelihood approach, 662
tection, semi-supervised multivariate, 661
supervised, see anornaly detection, strengths and weaknesses, 665
supervised strengths and weaknesses, 670
unsupervised, see anomaly detec- supervised, 655
tion, unsupervised techniques, 654-655
causes of, 653-654 unsupervised, 656, 658-675
clustering-based, 671-675 Apriori
definition, 671 algorithm, 335
example, 672 principle, 333
impact of outliers, 674 association
membership in a cluster, 672 analysis, 327
number of clusters, 674 categorical attributes, 415
strengths and weaknesses, 67 4 continuous attributes, 418
definitions, 654 655 indirect, 467
density-based, 655 pattern, 328
distance-based, 655 rule, see rule
model-based, 654 attribtfte,22-29
density-based, 668-670 definition of, 23
deviation detection, 651 number of values, 28
distance based set, 146
definition, 666 type,23 27
exception mining, 651 asymmetric, 28-29
issues, 656-658 binary, 28
outliers, 651 continuous, 26, 28
proximity-based, 666-668 discrete, 28
distance-based, see anomaly detec- interval, 25, 26
tion, distance-based nominal, 25, 26
strenghts and weaknesses, 666 ordinal,25, 26
 Subject Index 759

qualitative, 26 classifier
quantitative, 26 bagging, 283
ratio, 25 base,277
axon,246 Bayes,227
Bayesian belief, 240
back-propagation, 254 boosting, 285
bagging, see classifier combination, 276
Bayes decision tree, 150
classifier, s€e classifier ensemble, 276
error, 238 maximal margin, 259
naive, see classifier naive-Bayes, 231
network, see classifier nearest-neighbor, 223
theorem, 228 neural networks,246
bias, 281 perceptron,24T
bias variance decomposition, 281 random forest, 290
binarization, see discretization, binariza- Rote, 223
tion, 416, 419 rule-based, 207
BIRCH, 633-635 support vector machine, 256
boosting, see classifier unstable, 280
bootstrap, 188 climate indices, 630
box plot, see visualization, box plot cluster analysis
Bregman divergence, 79-80 algorithm characteristics, 575-576
mapping to another domain, 576
c4.lniles,2I2 nondeterminism, 575
candidate optimization, 576
elimination, 448 order dependence, 575
generation, 337, 338, 433,448 parameter selection, see parame-
itemset, 332 ter selection
pruning, 338, 435, 453 scalability, s€e scalability
rule, 350 applications, 487-489
sequence,433 as an optimization problem, 576
subgraph, 445 chameleon, see chameleon
capacity, 258 choosing an algorithm, 639-642
case, see object cluster characteristics, 57 3-57 4
chameleon, 616*622 data distribution, 574
algorithm, 620-621 densrty, c/4
graph partitioning, 621 poorly separated, 574
merging strategy, 618 relationships, 574
relative closeness,618 shape, 574
relative intereconnectivity, 619 size, 574
self-similarity, 613, 6l-8, 620, 62I subspace, 574
strengths and limitatiorts, 622 cluster density, 574
characteristic, se€ attribute cluster shape, 510, 574
city block distance, see distance, city block cluster size,574
class data characteristics, 571-573
imbalance, 294 attribute types, 573
Iabel, 146 data types, 573
classification high-dimensionality, 572
model, 146 mathematical properties, 573
760 Subject Index

noise,572 cohesion, 536-540

outliers,572 cophenetic correlation, 545
scale,573 for individual clusters, 541
size,572 for individual objects, 541
sparseness, 572 hierarchical, 544,552
DBSCAN, seeDBSCAN number of clusters, 546
definition of,487,490 relative measures, 535
DENCLUE, seeDENCLUE separation, 536-540
density-basedclustering, 600-012 silhouette coefficient, 54L-542
fuzzy clustering, seefuzzy clustering supervised, 548 553
graph-basedclustering, 612-630 classification measures, 549-550
sparsification,613-OL4 similarity measures, 550-552
grid-basedclustering,seegrid-based supervised measures, 535
clustering unsupervised, 536-548
hierarchical, see hierarchical cluster- unsupervised measures, 535
lng with proximity matrix, 542-544
CURE, see CURE, see CURE codeword, 308
minimum spanning tree, 614-615 compaction factor, 370
Jarvis-Patrick, see Jarvis-Patrick concept hierarchy, 426
K-means, see K-means conditional independence, 241
mixture modes, see mixture models confidence
opossum, see opossum factor, 208
parameter selection, 529, 546,575 interval, 186
prototype-based clustering, 577-600 level,725
seeshared nearest neighbor, density- measure, see measure
based clustering, 629 confusion matrix, 149
self-organizing maps, see self-organizing constraint
maps maxgap, 437
subspace clustering, see subspace clus- maxspan, 437
tering mingap, 437
subspace clusters, 574 timing, 436
types of clusterings, 491-493 window size,439
complete, 493 constructive induction, 172
exclusive, 492 contingency table,372
fin2y,492 convex optim\zation, 262
hierarchical, 491 correlation
overlapping, 492 fcoefficient, 375
partial, 493 limitation, 375
partitional, 491 cost
types of clusters, 493-495 learning, 302
conceptual, 495 cost-sensitive learning, see cost
density-based, 494 coverage, 208
graph-based, 494 cross-validation, 187
prototype-based, 494 cuRE, 635-639
well-separated, 493 algorithm, 636, 637
validation, see cluster validation cluster feature,634
cluster validation, 532-555 clustering feature
assessmentof measures, 553-555 tree, 634
clustering tendency, 533, 547 use of partitioning, 638-639
 Subject Index 761

use of sampling, 638 noise, 37-39

curse of dimensionality, 271 outliers, 40
precision, 39
dag, see graph significant digits, 39-40
data data set, 22
attribute, see attribute characteristics, 29 30
attribute types, 573 dimensionality, 29
cleaning, see daha quality, data clean- resolution,29
rng sparsity, 29
distribution, 574 types of, 29-36
exploration, see data exploration graph-based, 32
fragmentation, 170 matrix, see matrix
high dimensional ordered, 33-35
problems with similarity, 622 record, 30-32
high-dimensional, 572 sequence,34
market basket, 327 sequential, 33
mathematical properties, 573 spatial, 35
noise, 572 temporal, 33
object, see object time series, 35
outliers, 572 transaction, 30
preprocessing. see preprocessing DBSCAN, 526-532
quality, see data quality algorithm, 528
scale, 573 comparison to K-means, 570-571
set, see data set complexity, 528
similarity, see similarity definition of density 527
size, 572 parameter selection, 529
sparse, 572 types of points, 527
transformations, see transformations border, 527
types, 573 core, 527
types of, 19,22-36 noise, 527
data exploration, 97-98 decision
OLAP, see OLAP boundary, 170
statistics, see statistics list,211
visualization, see visualization stump, 284
data quality, 19,36-44 tree, see classifier
application issues, 43-44 decision boundary
data documentation, 44 linear, 259
relevance, 44 deduplication, 43
timliness, 44 DENCLUE,608 612
data cleaning, 36 algorithm, 610
errors, 37-43 implementation issues, 610
accuracy, 39 kernel density estimation, 609-610
artifacts, 39 strengths and limitations, 611
bias, 39 dendrite, 246
collection, 37 density
duplicate data,42-43 center based, 527
inconsistent values, 41-42 dimension, see attribute
measurment, 37 dimensionality
missing values, 40-41 curse, 51-52
762 Subject Index

dimensionality reduction, 50-52, 7Ol-7tG edge, 443

factor analysis, 708-710 g5rowing, 449
FastMap, 713 element, 430
ISOMAP, 714-715 EM algorithm, 587-591
issues, 715-716 ensenrble method, see classifier
Locally Linear Embedding, 710-712 entity, see object
multidimensional scaling, 7 L2-7 13 entropy,60, 158
PCA, 52 rrse in discretization, see discretiza-
svD, 52 tion, supervised
discretization, 57-63, 233 equivalent sample size, 236
association, see association error
binarization, 58-59 :rpparent, 172
clustering, 420 t:rror-correcting output coding, 307
equal frequency,420 lunction, 253
equal width, 420 generalization, L72, 179, 258
of binary attributes, see discreliza- optimistic, 180
tion, binarization pessimistic, 181
of categorical variables, 62-63 r:esubstitution, 172
of continuous attributes, 59-62 1;raining, 172
supervised, 60-62 error rate, 149
unsupervised, 59-60 Eucli,lean distance, seedistance, Euclidean
disjunct, 207 evaluation
dissimilarity, 69-72, 79-80 association, 370
choosing, 83-84 example, 146
definition of, 66 exhaustive, 210
distance, see distance exploratory data analysis, see data explo-
non-metric, 72 ration
transformations, 66-69
distance, 69-71 factor: analysis, see dimensionality reduc-
city block, 70 tion, factor analysis
Euclidean, 70, 690 false negative, 297
Hamming, 308 false positive, 296
L1 norm, 70 FastMap, see dimensionality reduction, FastMap
L2 norm, 70 feature
L*,70 redundant, 169,256
L*orr 70 feature construction, see featrre creation,
Mahalanobis, 81 feature construction
Manhattan, 70 featu.re creation, 55-57
metric, 70-71 feature construction, 57
positivity 70 feature extraction, 55
symmetry, 71 .mapping data to a new space, 56-57
triangle inequality, 71 feature extraction, see feature creation, fea-
Minkowski, 69-70 ture extraction
supremum, 70 feature selection, 52-55
distribution ,architecture for, 53-54
binomial, 183 feature weighting, 55
Gaussian, 183, 233 irrelevant features, 52
.redundant features, 52
eager learner, see learner bypes of, 52-53
 Subject Index 763

embedded, 53 inversions, 523

filter, 53 MAX, see complete link
wrapper, 53 merging decisions, 525
field, see attribute MIN, see single link
Fourier transform, 56 single link, 517, 519-520
FP-growth, 363 Ward's method, 523
FP-tree, see tree high-dimensionality
frequent subgraph, 442 seedata,high-dimensional, 572
Apriori-like algorithm, 447 historgram, see visualization, histogram
fuzzy clustering, 577-582 holdout,186
fitzzy c-means, 579-582 hypothesis
algorithm, 579 alternative, 423,727
centroids, 580 ns,ll, 423,727
example, 582
initialization, 580 independence
ssE,580 conditional, 231
strenths and limitations, 582 information gain
weight update, 581 entropy-based, 160
fuzzy sets, 578 FOIL's, 218
fuzzy psuedo-partition, 579 instance, 146
interest, see measure
gain ratio, 164 ISOMAP, see dimensionality reduction, ISOMAP
general-to-specific, 216 item, see attribute, 328
generalization, 148, see rule competing, 457
gini index, 158 negative, 458
gradient descent, 254 itemset, 329
graph, 443 candidate, see candidate
connected, 444 closed, 355
directed acyclic, 24I, 426 frequent, 331
isomorphism, 454 maximal, 354
undirected, 444
greedy search, 216 Jarvis-Patrick, 625-627
grid-based clustering, 601 604 algorithm, 625
algorithm, 601 example, 626
clusters, 603 strengths and limitations, 626
density, 602
grid cells, 601 K-means, 496-515
algorithm, 497 498
hidden bisecting, 508-509
layer, 251 centroids, 499,lOL
node, 251 choosing initial, 501-505
hierarchical clustering, 515-526 comparison to dBSCAN, 570-57t
agglomerative algorithm, 516 complexity, 505
centroid methods, 523 derivation, 513 515
cluster proximity, 517 empty clusters, 506
Lance-Williams formula, 524 incremental, 508
complete link, 517, 520-521 limitations, 510-513
complexity, 518 objective functions, 499, 50I
group average, 517, 52L-522 outliers, 506
764 Subject Index

reducing SEE, 507-508 document-term, 32

k-nearest neighbor graph, 613, 619,620 eigenvalue, 697
Karush-Kuhn-T\rcker, see KKT eigenvalue decomposition, 697-698
kernel density estimation, 609-610 eigenvector, 697
kernel function, 273 in data analysis, 699-700
kernel trick, 273 inverse, 696-697
KI{T,263 Iinear combination, 703
linear transformations, 695-697
L1 norm, see distance. Lr norm column space, 696
L2 norfir, see distance, L2 norm left nullspace, 696
Lagrange multiplier, 262 nullspace, 696
Lagrangian,262 projection, 695
dual, 263 reflection, 695
Laplace,2lT rotation, 695
Iazy learner, see learner row space, 696
learner scaling, 695
eager,223,226 multiplication, 693-695
Ia2y,223,226 permutation, 455
learning algorithm, 148 positive semidefinite, 703
learning rate,254 rank, 696
least squares, 699 row vector, 692
Ieave-one-out, 187 scalar multiplication, 692-693
lexicographic order, 341 singular value, 698
Iikelihood ratio,2I7 singular value decomposition, 698
linear algebra, 685-700 singular vector, 698
matrix, see matrix sparse, 31
vector, see vector maxgap, see constraint
linear regression, 699 maximum likelihood estimation, 585-587
Iinear separable, 256 maxspan, see constraint
linear systems of equations, 699 MDL, 182, 259
linear transformation, see matrix, Iinear mean,233
transformatiorr measure
Locally Linear Embedding, see dimension- asymmetric, 377
ality reduction, Locally Linear confidence, 329
Embedding consistency, 377
interest, 373
m-estimate, 217, 236 IS, 375
majority voting, see voting objective, 371
Manhattan distance, see distance, Man- properties, 379
hattan subjective, 371
margrn support, 329
maximal, 256 symmetric, 377
soft, 266 measurement, 23-27
market basket data, see data definition of, 23
matrix, 30, 691-697 scale, 23
addition, 692 permissible transformations, 26-27
column vector, 692 types,23-27
confusion, see confusion matrix Mercer's theorem, 274
definition, 691-692 metric
 Subject fndex 765

accuracy,149 object, 22
performance, 149 objective function, 261
min-Apriori, 425 observation, see object
mingap, see constraint Occam's razor, 181
minimum description length, see MDL OLAP, 46, 131-139
missing values, see data quality, errors, computing aggregates, 135-137
missing values cross tabulation, 137
mixture models, 583-59- data cube, 135-139
advantages and limitations, 591 dicing, 135, L38
definition of, 583-585 dimensionality reduction, 135, 138
EM algorithm, 587-591 drill-down, 135, 138-139
maximum likelihood estimation, 585- fact table, 134
D6/ MOLAP, 139
model, 146 multidimensinal data, 131-135
comparison, 188 multidimensional data analysis, 135-
descriptive, l46 139
evaluation, 136 pivoting, 138
overfitting, 165, I72 ROLAP, 139
predictive, 147 roll-up, 135, 138-139
selection, 186 slicing, 135, 138
monotone, 258 online analytical processing, see OLAP
monotonicity, 334 opposum, 616
multiclass, 306 algorithm, 616
multidimensional data analysis, see OLAP, strengths and weaknesses,616
multidimensional data analysis outliers, see data quality
multidimensional scaling, see dimension- overfitting, see modeI, I74
ality reduction, multidimensional oversampling, 305
scaling
multiple comparison, 178 pattern
mutual exclusive, 210 cross-support, 387
hyperclique, 390
nearest-neighbor classifier, see classifier infrequent, 457
network negative, 458
Bayesian, see classifier negatively correlated, 458, 460
feed-forward, 251 sequential, see sequential
multilayer, see classifier subgraph, see subgraph
neural, see classifier PCA, TOL_704
recurrent, 251 examples, 704
topology, 243 mathematics, 702-703
neuron, 246 perceptron, see classifier
node learning, 248
hidden, see hidden pie cha,rt, see visualization, pie chart
internal, 150 point, see object
Ieaf, 150 precision, 297
non-terminal, 150 precondition, 208
root, 150 preprocessing, 19, 44 -65
terminal, 150 aggregation, see aggregation
noise, 175, 227,281 binarization, see discretization, bina-
rization
766 Subject Index

dimensionality reduction, 50 association, 329

discretization, see discretization candidate, see candidate
feature creation, see feature creation classification, 207
feature selection, see feature selec- consequent, 208
tion evaluation, 216
sampling, see sampling generalization, 422
transformations, see transformations generation, 222, 33L, 349, 422
prevalence, 213 growing,2L5,22O
probability ordered, 211-
class-conditional, 230 orderftrg, 222
density, 234 class-based,212
posterior, 229 rule-based, 211
prior,229 pruning, 218
table,24I quantitative, 418
proximity, 65-84 discretization-based, 418
choosing, 83-84 non-discretiz ation, 424
cluster, 517 statistics-based, 422
definition of, 65 redundant, 422
dissimilarity see dissimilarity specialization, 422
distance, see distance unordered, 211
for simple attributes, 67-69 validation, 423
issues, 80-83 rule set, 207
attribute weights, 82 83
coiabining proximities, 81-82 sample, see object
correlation, 81 sampling, 47-50, 305
standardization, 81 approaches, 48
similarity, see similarity progressive, 50
transformations, 66-67 random, 48
pruning stratified, 48
post-pruning, 185 with replacement, 48
prepruning, l84 without replacement, 48
sample size,48-49
quadratic programming, 264 scalability
clustering algorithms, 630-639
random forest BIRCH, 633-635
seeclassifier,290 cuRE, 635-639
random subsampling, 187 general issues, 630-633
recalI,297 scatter plot, see visualization, scatter plot
Receiver Operating Characteristic curve, segmentation, 491
see ROC self-organizing maps, 594-600
record, see object, 146 algorithm, 594-597
reduced error pruning, 201, 316 applications, 598
Reproducing kernel Hilbert space, see RKHS strengths and limitations, 599
RIPPER, 2r2,220 sensitivity, 296
RKHS, 274 sequence
ROC, 298 data sequence,431
Rote classifier, see classifier definition, 430
rule sequential
antecedent, 208 pattern,429
 Subject Index 767

pattern discovery, 431 summary, 98-105

timing constraints, see constraint variance, 102-103
sequential covering, 213 stem and leafplot, see visualization, stem
shared nearest neighbor, 613 and leaf plot
density, 627-628 stopping criterion, 216
density-based clustering, 629 630 structural risk minimization, 258
algorithm, 629 subclass, 147
example, 629 subgraph
strengths and limitations, 630 core, 448
principle, 613 definition, 443
similarity, 622-625 pattern,442
computation, 624-625 support, see support
differences in density, 623-624 subsequence,430
versus direct similarity, 625 contiguous, 438
significance subspace clustering, 604-608
IeveI,727 CLIQUE,607
similarity, 20,72 79 algorithm, 608
choosing, 83-84 monotonicity property, 607
correlation, 76-79 strengths and limitations, 607
cosine, 74-76, 690 example, 604
definition of, 66 subtree
extended Jaccard, 76 raising,186
Jaccard, T4 replacement, 186
shared nearest neighbor, see shared superclass, 147
neafest neighbor, similarity support
simple matching coefficient, 73-74 count, 329
Tanimoto, 76 counting, 342, 435, 439, 457
transformations, 66 69 Iimitation, 372
Simpson's paradox, 384 measure, see mea,sure
soft splitting, 194 pruning, 334
SOM, 574, see self-organrzlng maps sequence,431
specidlization, see rule subgraph, 443
specific-to-general, 216 support vector, 256
specificity, 296 support vector machire, see classifier
split information, 164 svD, 706-708
statistical databases, 139 example, 706-708
statistics mathematics, 706
absolute average deviation, 103 SVM, see classifier
covarinace matrix,702 linear, 259
frequencies, 99-100 linear separabIe,262
interquartile range, 103 nonlinear, 270
mean, 101-102 svm
trimmed, 102 nonseparable, 266
median, 101-102 synapse, 246
median average deviation, 103
mode, 99-100 target
percentiles, 100-101 attribute, 146
range,102-103 function, 146
skewness,105 taxonomy, see concept hierarchy
768 Subject Index

test set, 149 vector quantization, 489

topological equivalence, 450 vertex, 443
training set, 148 growing, 448
transaction, 328 visualization, 105-131
extended, 428 animation, 123
width, 346 basic concepts
transformations, 63-65 arrangement, 108-109
between similarity and dissimilarity, representation, 107-108
66-67 selection, 109-110
normalization, 64-65 box plot, 114
simple functions, 63-64 Chernoff faces, 126 130
standardization, 64-65 contour plot, 121.
tree correlation matrix, 125-126
conditional FP-tree, 369 cumulative distribution functions, 115-
decision, see classifier 116
FP-tree, 363 data matrix, 124-125
hash, 344 do's and don'ts, 130-131
oblique, 172 histogram
pruning, 165 basic, 111-112
triangle inequality, 71 two-dimensional, 113
true negative, 296 motivations for, 105-106
true positive, 296 parallel coordinates, 126
percentile plot.116
underfitting,174 pie chart, 114-115
undersampling, 305 scatter plot, 116-119
universal approximator, 255 extended,119
matrix, 116
validation set, 184, 218 three-dimensional, 119
variable, see attribute two-dimensional, 116-1 19
variance, 233,28L slices, L22-I23
vector, 685-691 star coordinates, 126-130
addition, 685-686 stem and leaf plot, 111
column, see matrix, column vector surface pIot, l2I 122
definition, 685 techniques, 110-131
dot product, 688-690 for few attributes, 111-119
in data analysis, 690-691 for many attributes, 724-730
Iinear independence, 689-690 for spatio-temporal data, IL9-I23
mean, 691 vector field plot,122
mulitplication by a scalar, 686-687 vote
norm, 688 majority, 304
orthogonal,68T 689 voting
orthogonal projection, 689 distance-weighted, 226
row, see matrix, row vector majority,226
space,687 688
basis, 687 wavelet transform, 57
dimension, 687 Web crawler, 166
independent components, 687 window size, see constraint
Iinear combination, 687
span, 687
 769

Copyright Permissions

Some figures and part of the text of Chapter 9 originally appeared in the
article "Finding Clusters of Different Sizes, Shapes, and Densities in Noisy,
High Dimensional Data," Levent Ertoz, Michael Steinbach, and Vipin Kumar,
Proceedingsof the Thi,rd SIAM International Conference on Data Mi,ning, San
FYancisco,CA, May 1-3, 2003, SIAM. @2003, SIAM. The material is reprinted
here by permission of the publisher, SIAM.

Some figures and part of the text of Chapter 7 appeared in the article "select-
ing the Right Objective Measure for Association Analysis," Pang-Ning Tan,
Vipin Kumar, and Jaideep Srivastava, Informati,on Systems,29(4), 293-313,
2004, Elsevier. @2004, Elsevier. The material is reprinted here by permission
of the publisher, Elsevier.

Someof the figures and text of Chapters 9 appearedin the article "Discovery of
Climate Indices Using Clustering," Michael Steinbach, Pang-Ning Tan, Vipin
Kumar, Steven Klooster, and Christopher Potter, KDD '03: Proceedi,ngs of the
Ni,nth ACM SIGKDD International Conferenceon Knowledge Di,scoueryand
Data Mi,ni,ng,446-455, Washington, DC, August 2003, ACM. @)2003,ACM,
INC. The material is reprinted here by permission of the publisher, ACM. DOI
: http : / / aoi. acm.orgl10 . IL45/956750.956801

Some of the figures (1-7,13) and text of Chapter 8 originally appeared in the
chapter "The Challenge of Clustering High-Dimensional Data," Levent Ertoz,
Michael Steinbach, and Vipin Kumar in New Di,rectzonsi,n Stati,sti.calPhysi,cs,
E conophysi,cs,Bi,oi,nformat'ics,and P attern Recogni,ti,on,
273-312, Editor, Luc
Wille, Springer, ISBN 3-540-43182-9.@)2004,Springer-Verlag. The material
is reprinted here by permission of the publisher, Springer-Verlag.

Some of the figures and text of Chapter 9 originally appeared in the article
"Chameleon: Hierarchical Clustering Using Dynamic Modeling," by Geogre
Karypis, Eui-Hong (Sam) Han, and Vipin Kumar, IEEE Computer, Volume
32(8), 68-75, August, 1999, IEEE. @1999, IEEE. The material is reprinted
here by permission of the publisher, IEEE.

gfrr*f trf;ali. e,/.ioia; yi,

l#||$f
l;T