Persiapan Data

1. Data Cleaning
2. Data Reduction
3. Data Transformation and Data Discretization
Data Integration
CRISP-DM

2
Why Preprocess the Data?
Measures for data quality: A multidimensional view

• Accuracy: correct or wrong, accurate or not

• Completeness: not recorded, unavailable, …
• Consistency: some modified but some not, …
• Timeliness: timely update?
• Believability: how trustable the data are correct?
• Interpretability: how easily the data can be
understood?

3
Major Tasks in Data Preprocessing
1. Data cleaning
• Fill in missing values
• Smooth noisy data
• Identify or remove outliers
• Resolve inconsistencies
2. Data reduction
• Dimensionality reduction
• Numerosity reduction
• Data compression
3. Data transformation and data discretization
• Normalization
• Concept hierarchy generation
4. Data integration
• Integration of multiple databases or files

4
Data Preparation Law (Data Mining Law 3)
Data preparation is more than half of every data
mining process

• Maxim of data mining: most of the effort in a data

mining project is spent in data acquisition and
preparation, and informal estimates vary from 50 to
80 percent
• The purpose of data preparation is:
1. To put the data into a form in which the data mining
question can be asked
2. To make it easier for the analytical techniques (such as
data mining algorithms) to answer it

5
1. Data Cleaning

6
Data Cleaning
Data in the Real World Is Dirty: Lots of potentially
incorrect data, e.g., instrument faulty, human or computer
error, transmission error
• Incomplete: lacking attribute values, lacking certain
attributes of interest, or containing only aggregate data
• e.g., Occupation=“ ” (missing data)
• Noisy: containing noise, errors, or outliers
• e.g., Salary=“−10” (an error)
• Inconsistent: containing discrepancies in codes or names
• e.g., Age=“42”, Birthday=“03/07/2010”
• Was rating “1, 2, 3”, now rating “A, B, C”
• Discrepancy between duplicate records
• Intentional (e.g., disguised missing data)
• Jan. 1 as everyone’s birthday?
7
Incomplete (Missing) Data

• Data is not always available

• E.g., many tuples have no recorded value for several attributes, such as
customer income in sales data
• Missing data may be due to
• equipment malfunction
• inconsistent with other recorded data and thus deleted
• data not entered due to misunderstanding
• certain data may not be considered important at the time of entry
• not register history or changes of the data
• Missing data may need to be inferred

8
Contoh Missing Data

• Dataset: MissingDataSet.csv

9
MissingDataSet.csv

• Jerry is the marketing manager for a small Internet design and advertising firm
• Jerry’s boss asks him to develop a data set containing information about Internet users
• The company will use this data to determine what kinds of people are using the Internet
and how the firm may be able to market their services to this group of users
• To accomplish his assignment, Jerry creates an online survey and places links to the
survey on several popular Web sites
• Within two weeks, Jerry has collected enough data to begin analysis, but he finds that his
data needs to be denormalized
• He also notes that some observations in the set are missing values or they appear to
contain invalid values
• Jerry realizes that some additional work on the data needs to take place before analysis
begins.

10
Relational Data

11
View of Data (Denormalized Data)

12
Contoh Missing Data

• Dataset: MissingDataSet.csv

13
How to Handle Missing Data?
• Ignore the tuple:
• Usually done when class label is missing (when doing
classification)—not effective when the % of missing values
per attribute varies considerably
• Fill in the missing value manually:
• Tedious + infeasible?
• Fill in it automatically with
• A global constant: e.g., “unknown”, a new class?!
• The attribute mean
• The attribute mean for all samples belonging to the same
class: smarter
• The most probable value: inference-based such as
Bayesian formula or decision tree

14
Latihan
• Lakukan eksperimen mengikuti buku
Matthew North, Data Mining for the Masses
2nd Edition, 2016, Chapter 3 Data
Preparation
1. Handling Missing Data, pp. 34-48 (replace)
2. Data Reduction, pp. 48-51 (delete/filter)

• Dataset: MissingDataSet.csv

• Analisis metode preprocessing apa saja yang

digunakan dan mengapa perlu dilakukan
pada dataset tersebut?
15
Missing Value Detection

16
Missing Value Replace

17
Missing Value Filtering

18
Noisy Data

• Noise: random error or variance in a measured variable

• Incorrect attribute values may be due to
• Faulty data collection instruments
• Data entry problems
• Data transmission problems
• Technology limitation
• Inconsistency in naming convention
• Other data problems which require data cleaning
• Duplicate records
• Incomplete data
• Inconsistent data

19
How to Handle Noisy Data?

• Binning
• First sort data and partition into (equal-frequency) bins
• Then one can smooth by bin means, smooth by bin median, smooth by bin
boundaries, etc.
• Regression
• Smooth by fitting the data into regression functions
• Clustering
• Detect and remove outliers
• Combined computer and human inspection
• Detect suspicious values and check by human (e.g., deal with possible
outliers)

20
Data Cleaning as a Process
• Data discrepancy detection
• Use metadata (e.g., domain, range, dependency, distribution)
• Check field overloading
• Check uniqueness rule, consecutive rule and null rule
• Use commercial tools
• Data scrubbing: use simple domain knowledge (e.g., postal code,
spell-check) to detect errors and make corrections
• Data auditing: by analyzing data to discover rules and relationship
to detect violators (e.g., correlation and clustering to find outliers)
• Data migration and integration
• Data migration tools: allow transformations to be specified
• ETL (Extraction/Transformation/Loading) tools: allow users to
specify transformations through a graphical user interface
• Integration of the two processes
• Iterative and interactive (e.g., Potter’s Wheels)

21
Latihan

• Lakukan eksperimen mengikuti buku Matthew North, Data

Mining for the Masses 2nd Edition, 2016, Chapter 3 Data
Preparation, pp. 52-54 (Handling Inconsistence Data)

• Dataset: MissingDataSet.csv

• Analisis metode preprocessing apa saja yang digunakan dan

mengapa perlu dilakukan pada dataset tersebut!

22
23
 Setting Regex

Ujicoba Regex
24
Latihan
• Impor data MissingDataValue-Noisy.csv
• Gunakan Regular Expression (operator Replace)
untuk mengganti semua noisy data pada atribut
nominal menjadi “N”

25