PRINCIPAL

COMPONENT
ANALYSIS(PCA)
ENGR. SAJID SALEEM
CSAL4243
 PRINCIPAL COMPONENT ANALYSIS
• Principal component analysis (PCA) is a statistical procedure that uses an orthogonal
transformation to convert a set of observations of possibly correlated variables into a set of
values of linearly uncorrelated variables called principal components.

OR

• PCA also called Principal component analysis is used for reducing the dimension of the dataset
to a lower dimension
• where it can be analyzed more easily
• and also keep all the relevant information of the dataset without any loss.

High Dimension Data  Low Dimension Data

 PRINCIPAL COMPONENT ANALYSIS
• When to use PCA?

• To reduce the number of dimensions in your factor analysis.

• To categorize the dependent and independent variables in your data
• To eliminate the noise components in your dimension analysis
 PRINCIPAL COMPONENT ANALYSIS
• Quick review (Vector)
• In the context of linear algebra and matrices:
• A vector is a list of numbers,
• arranged either in a row (referred to as a row vector) or
• a column (known as a column vector)

• In order to multiply a matrix by a vector, the matrix must have as many columns as the vector
has rows
• We might also find ourselves multiplying matrices by vectors when solving linear equations.
• Multiplying a vector by a matrix is much simpler than multiplying two different matrices.

• When we multiply a vectors by matrix, the vector might rotate, stretch, shrink or change
direction.
• Example: x = =
 PRINCIPAL COMPONENT ANALYSIS
• Quick review (Eigenvector and Eigenvalue)
• An eigenvector is a non-zero vector that doesn't change direction when a linear transformation
is applied to it ( when you multiply it by a matrix)
• Eigenvectors are those vectors that are only stretched, with neither rotation nor trim, and keep
pointing in the same direction.
• The corresponding eigenvalue is the factor by which an eigenvector is stretched or
compressed.
• If the eigenvalue is negative, the eigenvector's direction is reversed.

• Consider an n × n matrix A and a nonzero vector v of length n. If multiplying A with v denoted by

Av simply scales v by a factor of λ, where λ is a scalar, then v is called an eigenvector of A, and
λ is the corresponding eigenvalue.

• How much eigen vector is stretched or compressed:

• If λ = 2 => eigenvector is twice as long
• If λ = 0.5 => eigenvector is half as long
 PRINCIPAL COMPONENT ANALYSIS
• Quick review (Eigenvector and Eigenvalue)

• We will be using the formula: det(S- λI)=0

• Where
• det: determinent (scalar valued function of the entries of a square matrix)
• S or A: Matrix
• λ: eigenvalue of S
• I : N x N identity matrix
 PRINCIPAL COMPONENT ANALYSIS
• Quick review (Covariance Matrix)

• Covariance: measure of joint variability of 2 random variables

• shows the tendency in the linear relationship between the variables

• A covariance matrix (also known as auto-covariance matrix, dispersion matrix, variance matrix,
or variance–covariance matrix)
• is a square matrix giving the covariance between each pair of elements of a given random
vector.
 PRINCIPAL COMPONENT ANALYSIS
• Let us understand PCA by an example for conversion of date from 2D to 1D
• Reduce the 2D data given below to 1D data:
Feature Example 1 Example 2 Example 3 Example 4
X1 4 8 13 7
X2 11 4 5 14

• Step 1: Calculate Mean (One can also use Z-Normalization)

• = =8
• = = 8.5
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• Step 2: Calculate Covariance Matrix Feature
1 2 3 4
• S= X1 4 8 13 7 = 8
X2 11 4 5 14 = 8.5

• = =
= 14
• =
=
= -11
• = = -11
• =
= So, Covariance Matrix:
= 23
S=
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• Step 3: Eigenvalues of Covariance Matrix Feature
1 2 3 4
X1 4 8 13 7 = 8
X2 11 4 5 14 = 8.5
• The characteristic equation of the covariance matrix is,
• 0 = det(S- λI) S=

• = I=

• = ()() – (-11) (-11) λI =

• = 2 - 37 + 201
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• Step 3: Eigenvalues of Covariance Matrix Feature
1 2 3 4
X1 4 8 13 7 = 8
X2 11 4 5 14 = 8.5
• = 2 - 37 + 201
• Finding the roots of the quadratic equation:
• = = =
• 1 = = 30.39
• 2 = = 6.61
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• Step 4: Computation of Eigenvectors Feature
1 2 3 4
X1 4 8 13 7 = 8
X2 11 4 5 14 = 8.5
• Eigenvector => =
• = (S- λI) => =>

• Now:
• =0 -①
• =0 -②

• We need to find and for the value of Eigenvectors.

• We can choose any 1 of the above equations
• Let us consider eq ①
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• =0 Feature
1 2 3 4
X1 4 8 13 7 = 8
• = X2 11 4 5 14 = 8.5

• = =t  where t is any real number

• So
• = 11t
• = (14- )t Taking t=1, we get eigenvector corresponding to Covariance Matrix as:

=
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• Next we calculate PCA, for that we consider Feature 1 2 3 4
the largest Eigenvalue X1 4 8 13 7 = 8
• = 30.39 X2 11 4 5 14 = 8.5
1

• 2 = 6.61

• If we need another PCA, we can choose the next value.

• Next, to find a unit eigenvector, we compute the length of X1 which is given by:
• = => =>
• = 19.7348
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• Finally, a unit eigenvector corresponding Feature
1 2 3 4
to λ1 is : X1 4 8 13 7 = 8
X2 11 4 5 14 = 8.5

• e1 = =>
1 = 30.39

= 19.7348
• e1 =

• By carrying out similar computations (if required),

• the unit eigenvector e2 corresponding to the eigenvalue λ= λ2 can be calculated as:

2 = 6.61
• e2 =
= ??
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• Step 5: Compute first principal components Feature
1 2 3 4
X1 4 8 13 7 = 8
• Let X2 11 4 5 14 = 8.5

• be the kth sample in the above Table (dataset).

• The first principal component of this example is given by:

• e1T =>

• = (0.5574) () + (-0.8303) ()
 PRINCIPAL COMPONENT ANALYSIS
Example Example Example Example
• First Principal Component for Example1: Feature
1 2 3 4
X1 4 8 13 7 = 8
• X2 11 4 5 14 = 8.5

• =

• = (0.5574) () + (-0.8303) () => -4.30535

 PRINCIPAL COMPONENT ANALYSIS
• D.I.Y. you may solve it for all examples :

Feature Example 1 Example 2 Example 3 Example 4

X1 4 8 13 7 = 8
X2 11 4 5 14 = 8.5
First Principal Components -4.30535 3.7361 5.6928 -5.1238
 PRINCIPAL COMPONENT ANALYSIS
• Step 6: Geometrical meaning of first principal components
Feature Example 1 Example 2 Example 3 Example 4
X1 4 8 13 7 = 8
X2 11 4 5 14 = 8.5
First Principal Components -4.30535 3.7361 5.6928 -5.1238

• First, we shift the origin to the “center” => (, )

• and then change the directions of coordinate axes

to the directions of the eigenvectors e1 and e2

The coordinate system for principal

components
 PRINCIPAL COMPONENT ANALYSIS
Feature Example 1 Example 2 Example 3 Example 4
X1 4 8 13 7 = 8
X2 11 4 5 14 = 8.5
First Principal Components -4.30535 3.7361 5.6928 -5.1238

• Next, we drop perpendiculars from the given data points to the e1-axis

Projections of data points on the axis of the first

 PRINCIPAL COMPONENT ANALYSIS
• The first principal components are the
e1-coordinates of the feet of
perpendiculars, that is, the projections
on the e1-axis.
• The projections of the data points on the
e1-axis may be taken as approximations
of the given data points hence we may
replace the given data set with these
points.
• Now, each of these approximations can
be unambiguously specified by a single
number, namely, the e1-coordinate of
approximation.
• Thus the two-dimensional data set can Geometrical representation of one-
be represented approximately by the dimensional approximation to the data set
following one-dimensional data set.
THANK YOU