Module 1

Introduction :Artificial Neuron-Activation function-Single

layer and Multi layer networks-Training Artificial Neural
Networks- Perceptron-Representation-Linear SeparabilityLearning-Training Algorithms

What is Artificial Neural Networks?

Artificial neural systems can be considered as
simplified

mathematical models of brain like

systems and they function as parallel distributed

computing networks.
Artificial neural systems, or neural networks, are
physical cellular systems which can acquire, store,
and utilize experiential knowledge

Definition of ANN (by Hetct-Nelsen)

An artificial Neural Network is a parallel ,distributed
information processing structure consisting of
Processing Units interconnected via unidirectional
signal channels called connections.
Each processing unit has a single output connection that
branches into as many collateral connections as desired.each carries the same signal-the out put of processing
units

Contd...

Processing
Units:Can Possess a Local Memory
Can carry out localized information processing
Processing unit output can be of any desired
mathematical type desired.
The information processes in the unit is
completely local, ie, input arriving at the
units,values stored in memory.

Traditional Algorithamic Approach

Vs
Artificial Neural Network
Traditional algorithmic approach is useful for problems
where it is possible to find

a precise sequence of mathematical operations

a precise sequence of rules.

Main weakness of Traditional approach

Sequential Computation
Usually instructions have to be executed in sequentially
even if the two instruction are not related
Local Representation
Any corruption in the sequence can ruin the entire process.
As complexity of program increases the reliability decreases.
OOPS tries to eliminate this problem

Main weakness of Traditional approach (Contd)

Learning Difficulties
If we define learning as the construction or modification
of some computational representation or model, it is
difficult to simulate learning by traditional methods.
Self decision Problems
Digital computer can solve problems that are difficult for
human,but it is very difficult to use them to automate tasks
that human can solve with little effort.

ANN: Is it similar to Neuron?

Artificial Neural Networks are biologically inspired
ANN composed of elements similar to biological neuron
The anatomy may similar to that of anatomy of brain

ANN also has surprising number of brain Characteristics

such as..
1.Learn from previous examples
2.Generalize from previous examples to new ones.
3.Abstract essentials data from input with irrelevant data
Despite of this similarities nobody can say ANN will soon
replace the functions of human brain
The actual intelligence exhibited by the most sophisticated
ANN works is below the level of a tape worm.

REPRESENTATION OF A NEURON

Artificial Neuron
The Artificial Neuron was designed to mimic the first order
Characteristics of biological neuron.

Set of input applied is output from another neuron

Input is multiplied by a corresponding weight ~ synaptic

strength.

All the weighted inputs are summed to get the activation

level of the neuron.

Artificial Neuron
X1

w1

X2

w2

X3

w3

X4

NET= XW

w4

NET=X1*w1+ X2*w2+X3*w3+X4*w4

ACTIVATION FUNCTION
(Transfer Functions)
A function used in between the actual output and the
NET.
Activation function processes the NET .
Activation function can be
Simple Linear function
The Threshold Function
The Sigmoid Function.

NET= X*W
OUT=F(NET)
X,W are vectors

Simple Linear Function

Threshold Function

OUT = K(NET)
K is a constant

OUT=1 if NET>T
OUT=0 otherwise.

Sigmoid Function
OUT=

1
(1 EXP NET )

Hyperbolic Tangent Functions

OUT=tanh(x)

Simple Linear
Function
0.1

OUT = K(NET)
K is a constant

0.08
0.06
0.04
f(NET)

0.02

K=0.01

0
-0.02
-0.04
-0.06
-0.08
-0.1
-10

-5

0
NET

10

Threshold Function
OUT=1 if NET>T
OUT=0 otherwise.

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
-5

-2.5

2.5

Sigmoid Function provides a nonlinear gain for Artificial

Neuron.
1
0.9
0.8
0.7

f(NET)

0.6
0.5
0.4
0.3
0.2
0.1
0
-10

-5

NET

Sigmoid Function

10

Matlab Activation (Transfer) Functions

Single Input Neuron.

Scalar input given (p) is multiplied by the scalar weight.
A bias b is passed to the neuron.
The summer output is given to the activation function
The output a=f (wp+b)

Single Input Neuron.

b
Input

General Neuron
a= f( w*p+b)

ARTIFICIAL NEURON WITH ACTIVATION FUNCTION

X1
X2

W1,1

F
W1,2

NET= XW

X3
X4

W1,3
W1,4

OUT

Artificial Neuron

Multi input Neuron.

Neuron with R inputs (Abbreviated notation)

p
Rx1

a
W

+
b
R

n
1x1

1x1

Weight Indices
The first index represent the particular neuron destination for
that weight.
The second index represents the source of the signal fed to
the neuron.
The indices in W1,2 say that weight represents the
connection to the first neuron from the second source.

Single Layer Neural Networks

One Neuron , even with many inputs, may not be
sufficient.
A Layer of neurons connected in parallel is called
single layer neuron.
Each elements of the input vector is connected to all of
the neuron of that layer.
Layer includes: weight matrix , the summers (soma)
,bias b, activation function.

Single Layer Neural Networks with 3 Neuron

W1,1

p1

n1

b1

W1,2

a2

p2

p3
p4

W3,3

a1

n2
b2

n3
b3

a3

The Vector Notation

Input vector

P=

p1
p
2
p3

(Rx1)

Weight Matrix

W=

w1,1 w1, 2 w1, R

w
w
w
2
,
1
2
,
2
2, R

ws ,1 ws , 2 ws , R

Multiple Layer Neural Networks

p1

n1

a1

b1
p2

PR

n2

n3
b3

a1

b1

b2
p3

n1

a2

a3

n2
b2

n3
b3

a2

a3

Layer of s neurons- Abbreviated notation.

p
Rx1

a
W
SxR
b

Sx1

n
S x1

S x1

Abbreviated representation-Two Layer Network

p
Rx1 W1
S xR
1

b1
R

S1 x 1

S1x1 W2
S2x S1
b
S1 x 1

n
S x1
2

S2 x1

TRAINING OF NEURAL NETWORKS

A network

is trained so that a set of inputs

produces the desired set of outputs.

Training

is accomplished by sequentially

applying input vectors ,while adjusting networks

weights according to a predetermined procedure.

During training the network weights gradually

converge to a value such that each input vector
produces ad desired output vector.

Types Of Training

Supervised Training.
Unsupervised Training.

Supervised Training.

Supervised

training requires the pairing of input

vector with a target vector representing the desired

output. (Training Pair)

The Network is usually trained with a number of

such training pairs.

Supervised Training.(Contd)

An input vector is applied output vector calculated

difference (error) fed back-Network weights are changed
accordingly-to minimize the error.

The Training pairs are applied sequentially, errors are

calculated and the weights adjusted for each vector, until the
error for entire training set is in the acceptably low level.

Unsupervised training

Supervised training methods are biologically

implausible.

Unsupervised training Methods are far more plausible

It requires no target vectors for output.
No comparison to predetermined ideal response.
Training set consists solely of input vectors.

The Training Algorithms modifies the weights to

produce output vectors that are consistent.

Consistent similar input vectors will produce same

output.

Unsupervised method utilizes the statistical property

of input vectors.

Applying a vector from a given class to the input will

produce a specific output vector, but there is no way to
determine prior to training which output pattern will be
produced by a given input vector class.

TYPES OF NETWORKS

FEED FORWARD NETWORKS

COMPETATIVE NETWORKS
RECURRENT NETWORKS

Perceptron
Perceptron is a feed forward network. In this the
summing unit multiplies the input vector by a weight
and sums the weighted output.
If this sum is greater than a predetermined threshold
value, the output is one; otherwise Zero (in case of
Hardlim and -1 in case of Hardlims)

X1

w1

X2

w2

X3

w3

X4

w4

Threshold
F
NET= XW

Artificial Neuron

OUT

Perceptron Representation
Representation & Learning
Representation refers to the ability of the network to
simulate a specified function.
Learning requires the existence of a systematic procedure for
adjusting the weights to produce that function.

Example: Representation
Can we represent a Odd/Even Number discriminating
machine by a perceptron?

A Basic Pattern Recognition Problem using

Perceptron.
ANN

SENSOR

SORTER

APPLE ORANGE

P=

Shape
Texture

Weight

1
P1=

1
1
1
P2=

1
1

Prototype of orange

Prototype of apple

Two Input case: Single Neuron Preceptron

p1

p2
b
a = hardlims (wp+b)

Single input neuron can classify the input vectors into

two categories.
Example 1:
Let for the above two input perceptron w11=1 and w12=1
Then
a=hardlims([ 1 1 ]p+b)
if b=1,
n=[1 1 ]p+1=0 represent a boundary line.

Perceptron Decision Boundary

P2

-1

n>0
P1

n<0

-1

Example 2:
Let for the above two input perceptron w11=-1 and
w12=1
Then
a=hardlims([ -1 1 ]p+b)
if b= -1,
n=[-1 1 ]p-1=0 represent a boundary line.

n>0

n<0
-1

The Key Property of Single-neuron perceptron is that it

can separate the input vectors into two category.
This category is determined by the equation
Wp + b =0.
Single layer perceptron can be used only to recognize
patterns which are LINEARLY SEPARABLE

Pattern recognition Example (Contd.)

There are only two category, Hence we can use singleneuron perceptron.
The input vector is of order

3x1.

Perceptron equation
a=Hardlims([w11 w12 w13]

p1
p
2
p3

+b )

Here ,to implement this pattern recognition problem we

have to select a linear boundary which separates the
proto type vectors ( Here it is Apple and Orange ).

1
1
Orange =

Apple =

1
1

1

P3

Orange (1 -1 -1 )

P1

P2

Apple (1 1 -1)

Hence the linear boundary between the output are a

plane P1P3 . That is, P2 .=0.
Wp+b=0 is here P2 .=0.

([w11 w12 w13]

p1
p
2
p3

+b )=0.

[0 1

p1
0] p
2
p3

+ 0 =0.

Hence weight matrix = [0 1 0 ].

Bias ,b =0.
Here the weight matrix is orthogonal to the
Boundary Layer.

Example 2
Is X-OR Problem is representational?
Take two input XOR gate

X Value Y Value Desired Point

Output
0

A0

B0

B1

A1

Y
A1

B0

xw1+ yw2

A0

B1
X

Exapmle:3
Check whether AND, OR functions are linearly seperable ?

Linear Separability
For some class of function the input vectors can be
separated geometrically .For two input case ,the
separator is a straight line. For three inputs it can
be done with a flat plane., cutting the resultant
three dimensional space. For four or more inputs
visualization is difficult . we can generalize to a
space of n dimensions divided by a
HYPERPLANE, which divides the space into four
or more regions.

Overcoming Linear separability Limitation

Linear separability limitation of single layer networks
can be overcome by adding more layers.
Multilayer networks can perform more general tasks.

Perceptron
training
Algorithmas follows
Training methods
used can
be summarized
1. Apply an input pattern and calculate the output.
2. .
a) If the output is correct, go to step 1.
b) If the output is incorrect, and is zero, add each input to
its corresponding weight;or
c) If the output is incorrect, and is one, subtract each
input to its corresponding weight.
3.Go to step 1.

THE DELTA RULE

Delta rule is an important generalization of perceptron
training algorithm .
Perceptron training algorithm is generalized by
introducing a term
= ( T-A )
T = Target Output.
A = Actual Output
If

= 0 step 2a
>0 step2b
< 0 step2c

In any of these case ,the perceptron training algorithm

is satisfied if is multiplied the value of each input xi
and this product is added to the corresponding weight.a
Learning Rate coeifficent is multiplied with xi
product to allow the average size of weight changes.

 = ( T-A )
i = xi
wi(n+1) =wi(n)+ i
Where
i
= the correction associated with i th input xi
wi(n+1) = the value of weight i after adjustment
wi(n)
= the value of weight i before adjustment

Problems with Perceptron Training Algorithm

It is difficult to determine whether the input sets are

lineaerly seperable or not.

In real world situation the inputs are often time

varying and may be sepearable at one time and not at
another.

The number of steps required is not properly

defined.

There is no proof that the perceptron algorithms are

faster than simply changing the values.

Module2
Back propagation: Training Algorithm - Application Network Configurations - Network Paralysis - Local
Minima - Temporal instability.

BACK PROPAGATION
Back propagation is a systematic method for training
multilayer artificial neural networks