Instance Based Learning

UNIT-4
Chapter 8
Instance Based Learning
15CS73 - Machine Learning Harivinod N

Chapter 8: Instance Based Learning
1. Introduction
2. K-nearest neighbor Learning
3. Locally Weighted regression
4. Radial basis functions
5. Case based reasoning
6. Summary
4
• Approaches in previous chapters
Training examples
…
New instances Target function Classified
• Instance-based learning
Training examples
– Lazy : processing is postponed

• Until queries are encountered
• Stores all training examples
– When a new query is encountered, examples related to the query
instance are retrieved and processed
• Instance-based learning
– In machine learning, instance-based
learning (sometimes called memory-based
learning) is a family of learning algorithms
that, instead of performing explicit
generalization, compares new
problem instances with instances seen in
training, which have been stored in memory.
Introduction
5
Introduction
6
Outline

1. Introduction
6. Summary
7
K nearest neighbors
K nearest neighbors is a simple algorithm that
stores all available cases and classifies new
cases based on a similarity measure (e.g.,
distance functions).
K nearest neighbors
1. Pick a value for K (i.e. 5).

K nearest neighbors
2. Take the K nearest neighbors of the new data point according to their
Euclidean distance.
K nearest neighbors
3. Among these neighbors, count the number of data points in each

category and assign the new data point to the category where you counted
the most neighbors.
K-nearest neighbor learning
(For classification and regression )
8
9
KNN Algorithm for classification
10
K-NN Hypothesis Space
• Note the k-NEAREST NEIGHBOR algorithm never forms an explicit

general hypothesis f regarding the target function f .
• It simply computes the classification of each new query instance as
needed.
11
K-NN Hypothesis Space
Voronoi diagram
11
12
Distance Weighted Nearest Neighbor
13
Remarks on K-NN
14
1. Introduction
6. Summary
15
Locally Weighted Regression
16
The phrase "locally weighted regression" is called

• local because the function is approximated based only on
data near the query point,
• weighted because the contribution of each training example
is weighted by its distance from the query point, and
• Regression because this is the term used widely in the
statistical learning community for the problem of
approximating real-valued functions.
PROCEDURE:
• Given a new query x q , construct an approximation f̂ that

fits the training examples in the neighbourhood surrounding x q
• This approximation is used to calculate f̂  x q  , which is as
the estimated target value assigned to the query instance.
How can f̂  x q  be calculated?
• Linear function
• Quadratic function
• Multilayer neural network
• The description of f̂ may change, because a different

local approximation will be calculated for each instance
Locally weighted linear regression
18
19
20
Outline

1. Introduction
6. Summary
22
Radial basis function
One of the approach to functionapproximation that is closely

related to distance-weighted regression and artificial neural
networks is learning with radial basis functions.
23
( for regression)
24
25
Given a set of training examples of the target function, RBF
networks are typically trained in a two-stage process.
• First, the number k of hidden units is determined and each
hidden unit u is defined by choosing the values of xu and a:
that define its kernel function Ku(d(xu , x)).
• Second, the weights wu are trained to maximize the fit of

the network to the training data, using the global error
criterion given by Equation
Because the kernel functions are held fixed during this second
stage, the linear weight values wu can be trained very 25
efficiently.
Outline

1. Introduction
6. Summary
29
Case Based Reasoning
• Case-based reasoning (CBR), is the process of

solving new problems based on the solutions of
similar past problems.
• An auto mechanic who fixes an engine by recalling

another car that exhibited similar symptoms is using
case-based reasoning.
• A lawyer who advocates a particular outcome in a trial
based on legal precedents or a judge who creates case
law is using case-based reasoning.
30
• CBR starts with a set of cases or training examples;

• it forms generalizations of these examples, by
identifying commonalities between a retrieved case
and the target problem.
• If for instance a procedure for plain pancakes is

mapped to blueberry pancakes, a decision is made to
use the same basic batter and frying method, thus
implicitly generalizing the set of situations under
which the batter and frying method can be used.
30
Instance-based methods such as K-NN, locally
weighted regression share three key properties.
1. They are lazy learning methods
They defer the decision of how to generalize
beyond the training data until a new query
instance is observed.
2. They classify new query instances by analyzing
similar instances while ignoring instances that
are very different from the query.
3. Third, they represent instances as real-valued
points in an n-dimensional Euclidean space.
Case-based reasoning (CBR) is a learning
paradigm based on the first two of these principles,
but not the third. 30
In CBR, instances are typically represented using more
rich symbolic descriptions, and the methods used to
retrieve similar instances are correspondingly more
elaborate.
31
•CBRhas been applied to problems such as conceptual
design of mechanical devices based on a stored library
of previous designs (Sycara et al. 1992),
•reasoning about new legal cases based on previous
rulings (Ashley 1990),
•solving planning and scheduling problems by reusing
and combining portions of previous solutions to
similar problems (Veloso 1992).
31
Case Study
The CADET(Case-based Design Tool) system

(Sycara et al. 1992)
•employs case based reasoning to assist in the conceptual
design of simple mechanical devices such as water
faucets.
•It uses a library containing approximately 75
previous designs and
•design fragments to suggest conceptual designs to
meet the specifications of new design problems.
32
CADET (Case-based Design Tool)
• Each instance stored in memory (e.g., a water pipe)

is represented by describing both its structure and its
qualitative function.
• New design problems are then presented by

specifying the desired function and requesting the
corresponding structure.
32
CADET (Case-based Design Tool)
• Each instance stored in memory (e.g., a water pipe)

is represented by describing both its structure and its
qualitative function.
• New design problems are then presented by

specifying the desired function and requesting the
corresponding structure.
32
32
32
32
32
Summary
33

Instance Based Learning

Uploaded by

Instance Based Learning

Uploaded by

UNIT-4

15CS73 - Machine Learning Harivinod N

– Lazy : processing is postponed

Chapter 8: Instance Based Learning

1. Pick a value for K (i.e. 5).

3. Among these neighbors, count the number of data points in each

• Note the k-NEAREST NEIGHBOR algorithm never forms an explicit

The phrase "locally weighted regression" is called

• Given a new query x q , construct an approximation f̂ that

• The description of f̂ may change, because a different

Chapter 8: Instance Based Learning

One of the approach to functionapproximation that is closely

• Second, the weights wu are trained to maximize the fit of

Chapter 8: Instance Based Learning

• Case-based reasoning (CBR), is the process of

• An auto mechanic who fixes an engine by recalling

• CBR starts with a set of cases or training examples;

• If for instance a procedure for plain pancakes is

The CADET(Case-based Design Tool) system

• Each instance stored in memory (e.g., a water pipe)

• New design problems are then presented by

• Each instance stored in memory (e.g., a water pipe)

• New design problems are then presented by

You might also like