Outline
Review of Linear Algebra
vectors and matrices products and norms vector spaces and linear transformations eigenvalues and eigenvectors
Introduction to Matlab
Why Linear Algebra?
For each data point, we will represent a set of features as feature vector
[length, weight, color,]
Collected data will be represented as collection of (feature) vectors
[l1, w 1 , c 1 , ] [l2, w2 , c2 ,] [l 3 , w3 , c3 ,]
Linear models are simple and computationally feasible
Vectors
X2
[x 1 , x 2 ]
X1
z y-z
n-dimensional row vector x = [x1 x2
x1 x Transpose of row vector is column vector xT = 2
Vector product (or inner or dot product)
xn ]
xn
i =1 k
x, y = x y = xT y = x1y1 + x2y2 + + xnyn =
xi yi
More on Vectors
Euclidian norm or length x =
x, x =
x
i =1 n
2 i
If |x|=1 we say x is normalized or unit length xT y Angle between vectors x and y cos = x y y x x y x
cos = 0 xT y = 0 xy
cos = 1 xT y =| x || y |> 0
y
cos = 1 xT y = | x || y |< 0
x orthogonal to y
Thus inner product captures direction relationship between x and y
More on Vectors
Vectors x and y are orthonormal if they are orthogonal and |x|=|y|=1 Euclidian distance between vectors x and y
xy =
x
i =1 n
( xi y i )
x-y y
Linear Dependence and Independence
Vectors x1, x2 , , xn are linearly dependent if there exist constants 1, 2 , , n s.t. 1. 1 x 1 + 2 x 2 + + n x n = 0 2. at least one i 0 Vectors x1, x2 , , xn are linearly independent if 1 x1 + 2 x2 + + n xn = 0 1 = = n = 0
Vector Spaces and Basis
The set of all n-dimensional vectors is called a vector space V A set of vectors u 1,u 2 , ,u n are called a basis for vector space if any v in V can be written as v = 1u 1 + 2u 2 + + nu n
u 1,u 2 , ,u n are independent implies they form a basis, and vice versa u 1,u 2 , ,u n give an orthonormal basis if 1. ui = 1 i 2. ui u j i j
Matrices
n by m matrix A and its m by n transpose A
T
x11 x12 A = x21 x22 xn1 xn2
x1m x2m xnm
x11 x12 x x T 12 22 A = x1m x2 m
xn1 x n2 xnm
Matrix Product
AB = a11 a12 a13 an1 an2 an3 a1d and b11 b21 b31 bd1 b1m b2m b3m = bdm cij =C
cij = a i , b j
a i is row i of A b j is column j of B
# of columns of A = # of rows of B even if defined, in general AB BA
Matrices
Rank of a matrix is the number of linearly independent rows (or equivalently columns) A square matrix is non-singular if its rank equal to the number of rows. If its rank is less than number of rows it is singular. 10 0 0 1 0 Identity matrix I = 00 AI=IA=A 00 1 1 2 9 5 T 2748 Matrix A is symmetric if A=A 9436 5864
Matrices
Matrix A is positive definite if xT Ax = Ai , j xi x j > 0
i,j
Matrix A is positive semi-definite if T x Ax = Ai , j xi x j 0
i,j
Trace of a square matrix A is sum on the elements on the diagonal
tr[A] =
n i =1
aii
Matrices
Inverse of a square matrix A is matrix A -1 s.t. AA = I
-1
If A is singular or not square, inverse does not exist. Pseudo-inverse A is defined T whenever A A is not singular (it is square) T -1 T A = (A A ) A -1 T T AA =(A A) AA=I
Matrices
Determinant of n by n matrix A is
det( A) =
n k =1
( 1)
k +i
aik det(Aik )
Where Aik obtained from A by removing the ith row and kth column Absolute value of determinant gives the volume of parallelepiped spanned by the matrix rows
1 2 n } a + a + ... + a 1 2 n
i [0 ,1] i
Linear Transformations
A linear transformation from vector space V to vector space U is a mapping which can be represented by a matrix M:
u = Mv
V U
M
Mv
If U and V have the same dimension, M is a square matrix In pattern recognition, often U has smaller dimensionality than V, i.e. transformation M is used to reduce the number of features. M v= u
Eigenvectors and Eigenvalues
Note: A0=0 for any , not interesting
Given n by n matrix A, and nonzero vector x. Suppose there is which satisfies Ax= x
x is called an eigenvector of A is called an eigenvalue of A Linear transformation A maps an eigenvector v in a simple way. Magnitude changes by , direction If > 0 If < 0
Av v v Av
Eigenvectors and Eigenvalues
If A is real and symmetric, then all eigenvalues are real (not complex) If A is non singular, all eigenvalues are non zero If A is positive definite, all eigenvalues are positive
MATLAB
Starting matlab
xterm -fn 12X24 matlab
Basic Navigation
quit more help general
Scalars, variables, basic arithmetic
Clear + - */ ^ help arith
Relational operators
==,&,|,~,xor help relop
Lists, vectors, matrices
A=[2 3;4 5] A
Matrix and vector operations
find(A>3), colon operator * / ^ .* ./ .^ eye(n),norm(A),det(A),eig(A) max,min,std help matfun
Elementary functions help elfun Data types double Char Programming in Matlab .m files scripts function y=square(x) help lang Flow control if i== 1else end, if else if end for i=1:0.5:2 end while i == 1 end Return help lang Graphics help graphics help graph3d File I/O load,save fopen, fclose, fprintf, fscanf
