Scribd
Upload
2 views4 pages

clustering_notes

The document outlines the steps involved in spectral clustering, which includes representing data as a graph, computing the adjacency and degree matrices, and deriving the Laplacian matrix to find eigenvectors for clustering. It explains the significance of the Laplacian matrix in relation to the clustering problem and provides mathematical formulations to support the process. An example graph and its corresponding Laplacian are presented to illustrate the concepts discussed.

Uploaded by

andy
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
Download as pdf or txt
2 views4 pages

clustering_notes

The document outlines the steps involved in spectral clustering, which includes representing data as a graph, computing the adjacency and degree matrices, and deriving the Laplacian matrix to find eigenvectors for clustering. It explains the significance of the Laplacian matrix in relation to the clustering problem and provides mathematical formulations to support the process. An example graph and its corresponding Laplacian are presented to illustrate the concepts discussed.

Uploaded by

andy
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
Download as pdf or txt
Download as pdf or txt
You are on page 1/ 4

SPECTRAL CLUSTERING

Notes

Matteo Lugli

1
1 Intuition
Spectral clustering algorithms all rely on some simple steps:

1. Represent the non-convex space using a graph;

2. Compute the adjacency matrix and the degree matrix of that graph;

3. Compute the Laplacian by doing L = A − D;

4. Compute the eigenvector and the eigenvalues of the matrix L;

5. Take the first k eigenvectors (from the smallest corresponding eigenvalue) except for
the first one, where k is the dimension of the new space;

6. The eigenvectors are going to be the features of the elements of the new space, so they
become the columns of a new matrix P ;

7. Use k-means on the matrix P to find the clusters;

But why do we use the Laplacian matrix? Why do we compute the eigenvectors
of such matrix?

2 Example

Figure 1: example graph

nodes 1 2 3 4 5 6
1 3 -1 -1 0 -1 0
2 -1 2 -1 0 0 0
3 -1 -1 3 -1 0 0
4 0 0 -1 3 -1 -1
5 0 0 0 -1 3 -1
6 0 0 0 -1 -1 2

2
Above you can see an example graph and the corresponding Laplacian. If you notice,
the sum of all of the rows is always zero. You can easily verify that one possible eigenvector
is [1, 1, . . . , 1] with the corresponding eigenvalue being λ = 0. Also remember that all of the
eigenvectors are always real and orthogonal.
The min-max theorem (read Wikipedia page) says that finding the smallest eigenvalue
is the same as solving the following equation:

xt M x
λ2 = min (1)
xt x
Let’s see if plugging the Laplacian in that equation produces something usefull that can
relate to our clustering problem:
n
X n
X
xT Lx = Li,j xi xj = (Di,j − Ai,j )xi xj (2)
i,j=1 i,j=1
X X
= Dii x2i − 2xi xj (3)
i i,j∈E

Going from 2 to 3 is not trivial: you need to consider that D is a diagonal matrix, so when
you successfully select an element that is ̸= from zero in matrix D it means that i and j
are the same. For the second part you need to consider that the in matrix A you find a 1
if node i and node j are connected. So we can replace the original loop with a loop that
iterates on the edges. The matrix is symmetrical so we encounter the product xi xj twice
through the loop.
X X
= Dii x2i − 2xi xj
i i,j∈E
X
= x2i + x2j − 2xi xj (4)
i,j∈E
X
= (xi − xj )2 (5)
i,j∈E

Let’s recall that x is a vector that ”selects” some of the elements and place them in cluster
A, the other ones will be placed in cluster B. Let’s re-write the full problem and make some
considerations:
2
P
i,j∈E (xi − xj )
λ2 = min P 2 (6)
i xi

• because x is an eigenvector and L is a Hermitian matrix, his length is 1, so i x2i = 1;


P

• because eigenvector x in orthogonal to the first eigenvector that we found in the


P P
previous example, then i xi · 1 = i xi = 0. This means that our function will place
some nodes on one cluster and the some nodes in the other cluster (if you think of
labels as 1 and -1).

3
3 Summary
We learned that finding the eigenvector corresponding to the smallest eigenvalue is the same
as solving this: X
argminy∈[−1,1]n f (y) = (yi − yj )2 = y t Ly (7)
i,j∈E

which looks really similar to the clustering problem that we want to solve. This particular
formulation is obtained because we used the Laplacian matrix!

You might also like