Page 1 of 20
Matrices & Linear
Algebra
Vector Spaces
Scalars are the elements of a number field (for
example, R and C), which
o Is a set of elements on which the operations of
addition and multiplication are defined, and
satisfy
the
usual
laws
of
arithmetic
(commutative, associative and distributive).
o Is closed under addition and multiplication
o Includes identity elements for addition and
multiplication (0 and 1).
o Includes inverses (negatives and reciprocals) for
addition and multiplication, except 0.
Vectors are elements of a linear vector space defined
over a number field F. A vector space V
o Is a set of elements on which the operations of
vector addition and scalar multiplication are
defined and satisfy certain axioms.
o Is closed under these operations.
o Includes an identity element (0) for vector
addition.
If the number field F over which the linear vector
space is defined is real, then the vector space is real.
Notes:
o Vector multiplication is not, in general, defined
for a vector space.
o The basic example of a vector space is a list of
n scalars, Rn. Vector addition and scalar
multiplication are defined component-wise.
Maths Revision Notes
Daniel Guetta, 2007
Page 2 of 20
o R2 is not exactly the same as C, because C has
a rule for multiplication.
o Similarly, R3 is not quite the same as physical
space, because physical space has a rule
(Pythagoras) for the distance between two
points.
The Inner Product
The inner product is used to give a meaning to
lengths and angles in a vector space.
It is a scalar function, x, y of two vectors x and y.
An inner product must
o Be linear in the second argument
x, ay = a x, y
x , y + z = x, y + x , z
o Have Hermitian symmetry
y, x = x , y
o Be positive definite
x, x 0
with equality if and only if x = 0.
Notes:
o The inner product has an existence without
reference to any basis.
o The hermitian symmetry is required so that
x, x is real. It is not required in a real vector
space.
o It follows from the above that the inner
product is antilinear in the first argument:
ax, y = a* x, y
x + y, z = x, z + y, z
o In Cn, the standard (Euclidean) inner product,
the dot product, is:
x, y = x i*yi
Maths Revision Notes
Daniel Guetta, 2007
Page 3 of 20
The complex conjugation is needed to maintain
Hermitian symmetry, and to ensure that the
product is linear in the second argument.
The Cauchy-Schwarz Inequality states that
x, y
x , x y, y
Or, equivalently:
x, y x y
With equality if and only if x = ay .
To prove, assume that x, y 0 (in which case the
inequality is trivial). We first say:
x - ay | x - ay = x - ay | x - a x - ay | y
= x | x - a* y | x - a x | y + aa * y | y
*
= x | x - a* x | y - a x | y + aa * y | y
= x 2 + a 2 y - 2 Re (a x | y
2
Now, this quantity must be positive, because of the
positive definite property of the inner product. If we
choose the phase of a (which is arbitrary) such that
a x |y
is
real
and
non-negative,
so
that
a x | y = a x | y , we then have that:
2
x 2 + a 2 y -2 a x |y 0
2
(x - a y ) +2 a x y -2 a x |y 0
And now, if we choose a = x / y , we get:
x y x |y
As required.
We can use the Cauchy-Schwarz inequality to prove
the triangle inequality ( x + y x + y ), by writing
x +y
in terms of the inner product, expanding,
using the inequality, and factorising.
The Cauchy-Schwarz Inequality allows us to define, in
real vector space, the angle q between two vectors,
though
x, y = x y cos q
Maths Revision Notes
Daniel Guetta, 2007
Page 4 of 20
[This is possible because, by the Cauchy Schwartz,
x ,y
x y
1 . If x, y = 0 , in any vector space, x and y
are said to be orthogonal.
A knowledge of the inner product of the basis vectors
is sufficient to determine the inner product of any two
vectors x and y. Let:
ei | e j = Gij
Then
x, y = Gij x i*y j
Where the Gij are the metric coefficients of the basis.
The Hermetian Symmetry of the inner product implies
that
Gij = G ji*
The matrix G is hermitian.
For an orthonormal basis, in which ei | e j = dij , we
have that
x | y = x i*yi
Bases
Let S = {e1, e2, , em } be a subset of the vectors in V.
A linear combination of S is any vector of the form
x 1e1 + x 2e2 + + x mem , where the x are scalars.
The span of S is the set of all vectors that are linear
combinations of S. If the span of S is the entire vector
space V, then S is said to span V.
The vectors of S are said to be linearly independent if
x 1e1 + x 2e2 + + x mem = 0
x 1, x 2, , x m = 0
If, on the other hand, such an equation holds for nontrivial values of the coefficients, one of the vectors is
redundant, and can be written as a linear combination
of the other vectors.
If an additional vector is added to a spanning set, it
remains a spanning set. If a vector is removed from a
Maths Revision Notes
Daniel Guetta, 2007
Page 5 of 20
linearly
independent
set,
it
remains
linearly
independent set.
A basis for a vector space V is the subset of vectors S
that spans V and is also linearly independent. The
properties of basis sets are:
o All bases of V have the same number of
elements, n, which is called the dimension of V.
o Any n linearly independent vectors in V form a
basis of V.
o Any vector x V can be written [prove by
considering V {x } as a linearly dependent set]
in a unique way [prove by contradiction] as a
linear combination of the vectors in a basis.
The relevant scalars are called the components
of x with respect to that particular basis.
The same vector (a geometrical identity) has different
components with respect to different bases. To see
how we can change from one to the other, consider
two bases of V S = {ei } and S = {ei} .
o Because both are bases, the elements of one
basis can be written in terms of the other:
e j = eiRij
Where Rij is the transformation matrix between
the two bases
o Now,
consider
vector
x V .
The
representation of the vector in each basis is:
x = e j x j = eix i
However, using our result from above, we can
write this as:
x = eix i = e j x j = eiRij x j
o From this, we can deduce the transformation
law for vector components:
x i = Rij x j
Maths Revision Notes
Daniel Guetta, 2007
Page 6 of 20
Note that:
The law is the reverse of that for
basis vector transformation. This is
to ensure that, overall, the vector x
stays unchanged by transformation.
The first suffix of R corresponds to
the same basis in both relations.
We defined Rij, above, by
e j = eiRij
The condition of the basis {ej} to be orthonormal is
eie j = dij
(ekRki ) elRlj
= dij
Rki* Rljekel = dij
If the second basis is also orthonormal, this becomes:
Rki* Rkj = dij
RR = I
In other words, transformations between orthonormal
bases is described by unitary matrices. In real vector
space, an orthogonal matrix does this in R2 and R3,
this corresponds to a rotation and/or reflection.
Given any m vectors u1 um
that span an n-
dimensional vector space (m > n), it is possible to
construct an orthogonal basis e1 en using the GramSchmidt procedure:
e1 = u1
es ur
es
s =1 es es
r -1
er = u r -
What we are effectively doing is taking each vector u
and removing any bits of vectors weve already
added to the basis from it, to leave us with a final
vector that is orthogonal to all others already added
We can prove, by induction, that this works:
Maths Revision Notes
Daniel Guetta, 2007
Page 7 of 20
Inductive step
Assume that vectors e1 et have already been added
to the orthogonal basis (such that ei e j = 0 " i j ),
and now consider the vector et+1 that were about to
add:
es ut +1
es
s =1 es es
r -1
et +1 = ut +1 -
And now, consider dotting it with ev (v < t), any of
the vectors already in the basis:
es ut +1
es ev ]
[
s =1 es es
=0 if v s
e u
= ut +1 ev - v t +1 [es es ]
es es
= ut +1 ev - ev ut +1
r -1
et +1 ev = ut +1 ev -
=0
So the new vector is indeed orthogonal to all the
vectors already in the set.
"Starting off" step
Consider e1 e2 :
e u
e1 e2 = u1 u2 - 1 2 e1
e1 e1
u u
= u1 u2 - 1 2 [u1 e1 ]
e1 e1
u u
= u1 u2 - 1 2 [e1 e1 ]
e1 e1
=0
So the first two vectors are, indeed, orthogonal.
Matrices
ARRAY VIEWPOINT
o Matrices can be regarded, simply, as an array
of numbers, Rij.
o The rule for multiplying a matrix by a vector is
then
(Ax )i = Aij x j
Maths Revision Notes
Daniel Guetta, 2007
Page 8 of 20
o The
rules
for
matrix
addition
and
multiplication are
(A + B )ij = Aij + Bij
(AB )ij = Aik Bkj
LINEAR OPERATOR VIEWPOINT
o A linear operator A acts on a vector space V to
produce other elements of V.
o The property of linearity means that:
L(ax ) = aL(x )
A(x + y ) = A(x ) + A(y )
o A linear operator can exist without reference to
any basis. It can be thought of as a linear
transformation or mapping of the space V.
[Some linear operators can even transform
between different bases].
o The components of A with respect to a basis
{ei } is defined by the action of A on the basis
vectors:
Ae j = Aijei
The components form a square matrix. [In
other words, the jth column of A contains the
compoenents of the result of A acting on ej].
o We now know enough to determine the action
of A only any x:
Ax = A(x je j ) = x j Ae j = x j Aijei = Aij x jei
So:
(Ax )i = Aij x j
This corresponds to the rule for multiplying a
matrix by a vector.
o Furthermore, the sum of two linear operators is
defined by
(A + B)x = Ax + Bx = ei (Aij + Bij ) x j
o And the product of two linear operators is
defined by
Maths Revision Notes
Daniel Guetta, 2007
Page 9 of 20
(AB)x = A(Bx ) = A(ei Bij x j ) = (Aek )Bkj x j = ei Aik Bkj x j
o Both these operations satisfy the rules of
matrix addition and multiplication and action
on a vector. As such, a linear operator can be
represented as a matrix.
BACK TO CHANGE OF BASIS
o Above, we wrote one set of basis vectors in
terms of the other:
e j = eiRij
But we could also have written
e j = eiSij
Substituting one into the other, we have
e j = ek Ski Rij
e j = ekRkiSij
But this can only be true if
SkiRij = RkiSij = dkj
Which implies that
RS = SR = 1
R = S -1
o We noted, above, that the transformation laws
for vector components could be written
x i = Rij x j
We can write this in matrix form, as
x = Rx
With the inverse relation
x = R-1x
LINEAR OPERATORS CHANGE OF BASIS
o To find how the components of a linear
operator A transform under a change of basis,
we note that we require
Ax = ei Aij x j = eiAij x j
Using e j = Rijei , we have that:
Maths Revision Notes
Daniel Guetta, 2007
Page 10 of 20
ekRki Aij x j = ekAkj x j
Rki Aij x j = Akj x j
RAx = A x
RA (R-1x ) = A x
Which means that
A = RAR-1
MATRIX MULTIPLICATION
o Matrix multiplication does not commute. But it
does distribute, so, with a bit of care, normal
rules of algebra can be applied. For example:
(1 -W )(1 +W )
= 1(1 +W ) -W (1 +W )
= 1 + 1W -W1 -W 2
= 1 -W 2
= (1 +W )(1 -W )
Hermitian Conjugate
We define the Hermitian conjugate of a matrix as
follows:
A = (AT )*
(A )ij
= Aji*
Importantly:
(AB ) = B A
(Note the reversal of the order).
We can also write the inner product as:
x, y = x Gy
Where G is the matrix of metric coefficients. This
preserve the hermitian symmetry of the inner product
as long as the matrix is hermitian G = G .
The adjoint of a linear operator A with respect to a
given inner product is a linear operator A satisfying
Ax | y = x | Ay
Maths Revision Notes
Daniel Guetta, 2007
Page 11 of 20
With the standard inner product, we find that the
matrix defining A is, indeed, the hermitian conjugate
of A.
Special Matrices
SYMMETRY
o A symmetric matrix is equal to its transpose
A = AT
o An hermitian matrix is equal to its hermitian
conjugate.
A = A
o An antisymmetric (or skew-symmetric) matrix
satisfies
AT = -A
o An anti-hermitian (or skew-hermitian) matrix
satisfies
A = -A
ORTHOGONALITY
o An orthogonal matrix is one whose transpose is
equal to its inverse
AT = A-1
o A unitary matrix is one whose hermitian
conjugate is equal to its inverse
A = A-1
We note that if U is a unitary matrix, then
AA = 1 . This implies that the columns of A
are orthonormal vectors.
A normal matrix is one that commutes with its
Hermitian conjugate:
AA = AA
It is easy to verify that hermitian, anti-hermitian and
unitary matrices are all normal.
RELATIONSHIPS
Maths Revision Notes
Daniel Guetta, 2007
Page 12 of 20
o If A is Hermitian, then Ai is anti-hermitian,
and vice-versa.
o If A is Hermitian, then
(Ai )n
exp(Ai ) =
n!
n =0
Is unitary.
o [This can be remembered by bearing in mind
that if z is a real number, iz is imaginary, and
if z is a real number, then eiz has unit modulus
(see below when talking about eigenvalues of
normal matrices)]
To prove that a matrix is a certain type of special
matrix, find an expression for the determining
property. For example, to prove its unitary, find
UU .
Eigenvalues and Eigenvectors
An eigenvector of a linear operator A is a non-zero
vector x satisfying
Ax = lx
(A - l1)x = 0
For some scalar l , called the eigenvalue.
The equation (in its second form) effectively says that
a linear combination of the columns of the matrix
(A - lI ) is equal to 0. This is equivalent to the
statement
det(A - lI ) = 0
Which is called the characteristic equation of the
matrix.
There are two possibilities in terms of roots:
o If there are n distinct solutions to the
characteristic
equation,
then
there
are
linearly independent eigenvectors. We prove
this as follows. Assume that
Maths Revision Notes
Daniel Guetta, 2007
Page 13 of 20
a e
a a
=0
We can multiply both sides by whatever we
want, so:
(A - l1I ) a aea = 0
y = (l2 - l1 )a2e2 + (l3 - l1 )a 3e3 + + (ln - l1 )anen = 0
We can do the same again with y:
(A - l2I )y = 0
(l3 - l1 )(l3 - l2 )a 3e3 + + (ln - l1 )(ln - l2 )anen = 0
We can then repeat this until we obtain:
(
ln - l1 )(ln - l2 )(ln - ln -2 )(ln - ln -1 )anen = 0
Now, if all the l are distinct, the expression
enclosed by a brace is non-zero. Therefore, an
must be 0. Removing the last vector and
repeatedly applying this method shows us that
all the an must be 0. Therefore,
a e
a a
=0
is only true if all the a a are 0. Therefore, the
vectors are linearly independent.
o If the roots are not all distinct, then the
repeated values are said to be degenerate. If a
value l occurs m times, there may be any
number
between
independent
and
eigenvectors.
of
linearly
Any
linear
combination of these is also an eigenvector.
A defective matrix is one who vector space is not
spanned by its eigenvectors. Such a matrix cannot be
diagonalised by a change of basis.
It can be shown that a normal matrix is never
defective. In fact, an orthonormal basis can always be
constructed from the eigenvectors of a matrix, if and
only if the matrix is normal.
Some interesting properties can be derived regarding
the properties of the eigenvectors and eigenvalues of
normal matrices:
Maths Revision Notes
Daniel Guetta, 2007
Page 14 of 20
o The eigenvectors corresponding to distinct
eigenvalues are orthogonal.
o The eigenvalues are
Real for hermitian matrices.
Imaginary for anti-Hermitian matrices.
Of unit modulus for unitary matrices.
A good way to remember these properties is to
consider that a 1 1 matrix is just a number l ,
and to be Hermitian, imaginary or unitary, it
must satisfy
l = l*
l = -l *
l * = l-1
Which are precisely the conditions for l being
real, imaginary or of unit modulus.
The method to prove these results is, in general, as
follows:
o Choose two arbitrary eigenvectors and write
the eigenvector equations:
Ay = my
Ax = lx
o Take one of these equations, and find the
hermitian conjugate.
o Then
For a hermitian matrix, construct two
expressions for y Ax .
For a unitary matrix, multiply both
sides by the other eigenvector equation
that hadnt be used.
o Re-arrange in the form something = 0.
o Assume that x = y, and using the fact that
x, y 0 ,
deduce
something
about
the
eigenvalues.
o Now, assume that x y
and deduce that
y x = 0 as long as l m , proving that the
vectors are orthogonal.
Matrices are given particular names:
Maths Revision Notes
Daniel Guetta, 2007
Page 15 of 20
o If all eigenvalues are < 0 (> 0), the matrix is
negative (positive) definite.
o If all eigenvalues are < 0 (> 0), the matrix is
negative (positive) semi-definite.
o A matrix is definite if it is either positive
definite or negative definite.
Diagonalization
Two square matrices A and B are said to be similar if
they are related by
B = S -1AS
In other words, if they are representations of the same
linear transformation in different bases. S is called the
similarity matrix.
A matrix is said to be diagonalisable if it is similar to
a diagonal matrix in other words, if
S -1AS = L
Where L is a diagonal matrix.
Consider
matrix
whose
columns
are
the
eigenvectors of the matrix A:
AS = A e1
= l1e1
= e1
e2
en
l2e2 l3en
0
l
2
e2 en
l
n
= SL
We can therefore say that
S -1AS = L
Maths Revision Notes
Daniel Guetta, 2007
Page 16 of 20
Provided that S is invertible ie:, provided that the
columns of S are linearly independent ie: provided
that the eigenvectors of A are linearly independent.
Notes:
o We notice that S is the transformation matrix
to
the
eigenvector
basis.
Therefore,
diagonalisation is the process of expressing a
matrix in its simplest form by transforming to
its eigenvector basis.
o An n n matrix is diagonalisable if and only if
it has n linearly independent eigenvectors. That
is to say, only if it is normal. Furthermore, if
the eigenvectors are chosen to be orthonormal,
then the columns of S are orthonormal and S is
therefore unitary (= a matrix whose columns
are orthonormal vectors).
Diagonalisation is rather useful in carrying out certain
operations on matrices:
Am = (S LS -1 )(S LS -1 )(S LS -1 ) = S LmS -1
det(A) = det(S LS -1 ) = det(S )det(L)det(S -1 ) = det(L)
tr(A) = tr(S LS -1 ) = tr(LSS -1 ) = tr(L)
tr(Am ) = tr(Lm )
Where we have used the following properties of
determinants and traces:
det(AB) = det(A)det(B)
det(S )det(S -1 ) = 1
tr(AB) = (AB)ii = Aij B ji = B ji Aij = (BA)jj = tr(BA)
Note that in general, for any matrix A
n
det(A) = li
i =1
n
tr(A) = li
i =1
Quadratic & Hermitian Forms
Maths Revision Notes
Daniel Guetta, 2007
Page 17 of 20
The quadratic form associated with a real symmetric
matrix A is
Q(x ) = x T Ax = Aij x ix j
is
homogeneous
quadratic
function
ie:
Q(ax ) = a2Q(x ) .
In fact, any homogenous quadratic equation is the
quadratic form of a symmetric matrix:
a b x
= x T Ax
Q = ax 2 + 2bxy + cy 2 = x y
b c y
In fact, A can be diagonalised by a real orthogonal
transformation:
(ST = S -1 )
ST AS = L
And the vector x transforms according to x = Sx , so
Q = x T Ax = (x T ST )(S LST )(Sx ) = x T Lx
The quadric form can therefore be reduced to:
n
Q = li x i
i =1
Where the x i are given by:
x = S -1x = S T x
We have effectively rotated the coordinates to reduce
the quadric form to its simplest form.
The quadric surfaces (or quadrics) are the family of
surfaces
Q(x ) = k = constant
In the eigenvector basis, this simplifies to
l1x 12 + l2x 22 + l3x 32 = k
The conic and quadric surfaces that can result are
depicted on the next page. The relevant semi-axes are
given by 1/ l . If l 0 , the shape comes apart.
A few special cases:
o If l1 = l2 = l3 , we have a sphere.
o If (for example), l1 = l2 , we have a surface of
revolution about the third axis, whatever it
might be.
Maths Revision Notes
Daniel Guetta, 2007
Page 18 of 20
o If
(for
example),
l3 = 0 ,
we
have
the
translation of a conic section along the relevant
axis (an elliptic or hyperbolic cylinder).
Maths Revision Notes
Daniel Guetta, 2007
Page 19 of 20
In a complex vector space, the Hermitian form
associated with an Hermitian matrix A is:
Maths Revision Notes
Daniel Guetta, 2007
Page 20 of 20
H (x ) = x Ax = x i*Aij x j
H is a real scalar, because
*
H * (x ) = (x i*Aij x j ) = x j*Aij*x i = x j*Aji x i = H (x )
We also know that A can be diagonalised by a unitary
transformation
U = U -1
U AU = L
And therefore:
H (x ) = x (U LU )x = (U x ) L (U x ) = x Lx = ln x i2
i =1
Therefore, a hermitian form can be reduced to a real
quadratic form by transforming to the eigenvector
basis.
Maths Revision Notes
Daniel Guetta, 2007
