TwoStep Cluster Analysis

The TwoStep Cluster Analysis procedure is an exploratory tool designed to reveal

natural groupings (or clusters) within a dataset that would otherwise not be apparent.
The algorithm employed by this procedure has several desirable features that
differentiate it from traditional clustering techniques:

Handling of categorical and continuous variables. The ability to create clusters

based on both categorical and continuous variables. By assuming variables to
be independent, a joint multinomial-normal distribution can be placed on
categorical and continuous variables.

Automatic selection of number of clusters. By comparing the values of a

model-choice criterion across different clustering solutions, the procedure can
automatically determine the optimal number of clusters.

Scalability. The ability to analyze large data files efficiently. By constructing a

cluster features (CF) tree that summarizes the records, the TwoStep algorithm
allows you to analyze large data files.

Clustering Principles
In order to handle categorical and continuous variables, the TwoStep Cluster Analysis
procedure uses a likelihood distance measure which assumes that variables in the
cluster model are independent. Further, each continuous variable is assumed to have a
normal (Gaussian) distribution and each categorical variable is assumed to have a
multinomial distribution.
Empirical internal testing indicates that the procedure is fairly robust to violations of
both the assumption of independence and the distributional assumptions, but you
should try to be aware of how well these assumptions are met.
The two steps of the TwoStep Cluster Analysis procedure's algorithm can be
summarized as follows:
Step 1. The procedure begins with the construction of a Cluster Features (CF) Tree.
The tree begins by placing the first case at the root of the tree in a leaf node that
contains variable information about that case. Each successive case is then added to an
existing node or forms a new node, based upon its similarity to existing nodes and
using the distance measure as the similarity criterion. A node that contains multiple
cases contains a summary of variable information about those cases. Thus, the CF tree
provides a capsule summary of the data file.
Step 2. The leaf nodes of the CF tree are then grouped using an agglomerative
clustering algorithm. The agglomerative clustering can be used to produce a range of

solutions. To determine which number of clusters is "best", each of these cluster

solutions is compared using Schwarz's Bayesian Criterion (BIC) or the Akaike
Information Criterion (AIC) as the clustering criterion.

Distance Measure. This selection determines how the similarity between two clusters
is computed.

Log-likelihood. The likelihood measure places a probability distribution on the

variables. Continuous variables are assumed to be normally distributed, while
categorical variables are assumed to be multinomial. All variables are assumed
to be independent.
Euclidean. The Euclidean measure is the "straight line" distance between two
clusters. It can be used only when all of the variables are continuous.

Number of Clusters. This selection allows you to specify how the number of clusters is
to be determined.

Determine automatically. The procedure will automatically determine the

"best" number of clusters, using the criterion specified in the Clustering
Criterion group. Optionally, enter a positive integer specifying the maximum
number of clusters that the procedure should consider.
Specify fixed. Allows you to fix the number of clusters in the solution. Enter a
positive integer.

Count of Continuous Variables. This group provides a summary of the continuous

variable standardization specifications made in the Options dialog box. See the topic
TwoStep Cluster Analysis Options for more information.
Clustering Criterion. This selection determines how the automatic clustering algorithm
determines the number of clusters. Either the Bayesian Information Criterion (BIC) or
the Akaike Information Criterion (AIC) can be specified.
TwoStep Cluster Analysis Data Considerations
Data. This procedure works with both continuous and categorical variables. Cases
represent objects to be clustered, and the variables represent attributes upon which
the clustering is based.
Case Order. Note that the cluster features tree and the final solution may depend on
the order of cases. To minimize order effects, randomly order the cases. You may want
to obtain several different solutions with cases sorted in different random orders to
verify the stability of a given solution. In situations where this is difficult due to
extremely large file sizes, multiple runs with a sample of cases sorted in different
random orders might be substituted.

Assumptions. The likelihood distance measure assumes that variables in the cluster
model are independent. Further, each continuous variable is assumed to have a
normal (Gaussian) distribution, and each categorical variable is assumed to have a
multinomial distribution. Empirical internal testing indicates that the procedure is fairly
robust to violations of both the assumption of independence and the distributional
assumptions, but you should try to be aware of how well these assumptions are met.
Use the Bivariate Correlations procedure to test the independence of two continuous
variables. Use the Crosstabs procedure to test the independence of two categorical
variables. Use the Means procedure to test the independence between a continuous
variable and categorical variable. Use the Explore procedure to test the normality of a
continuous variable. Use the Chi-Square Test procedure to test whether a categorical
variable has a specified multinomial distribution.
To Obtain a TwoStep Cluster Analysis
This feature requires the Statistics Base option.
1. From the menus choose:
Analyze > Classify > TwoStep Cluster...
2. Select one or more categorical or continuous variables.
Optionally, you can:

Adjust the criteria by which clusters are constructed.

Select settings for noise handling, memory allocation, variable standardization,
and cluster model input.
Request model viewer output.
Save model results to the working file or to an external XML file.

TwoStep Cluster Analysis Options

Outlier Treatment. This group allows you to treat outliers specially during clustering if
the cluster features (CF) tree fills. The CF tree is full if it cannot accept any more cases
in a leaf node and no leaf node can be split.

If you select noise handling and the CF tree fills, it will be regrown after placing
cases in sparse leaves into a "noise" leaf. A leaf is considered sparse if it
contains fewer than the specified percentage of cases of the maximum leaf
size. After the tree is regrown, the outliers will be placed in the CF tree if
possible. If not, the outliers are discarded.
If you do not select noise handling and the CF tree fills, it will be regrown using
a larger distance change threshold. After final clustering, values that cannot be
assigned to a cluster are labeled outliers. The outlier cluster is given an

identification number of 1 and is not included in the count of the number of

clusters.
Memory Allocation. This group allows you to specify the maximum amount of memory
in megabytes (MB) that the cluster algorithm should use. If the procedure exceeds this
maximum, it will use the disk to store information that will not fit in memory. Specify a
number greater than or equal to 4.

Consult your system administrator for the largest value that you can specify on
your system.
The algorithm may fail to find the correct or specified number of clusters if this
value is too low.

Variable standardization. The clustering algorithm works with standardized

continuous variables. Any continuous variables that are not standardized should be left
as variables in the To be Standardized list. To save some time and computational
effort, you can select any continuous variables that you have already standardized as
variables in the Assumed Standardized list.
Advanced Options
CF Tree Tuning Criteria. The following clustering algorithm settings apply specifically to
the cluster features (CF) tree and should be changed with care:

Initial Distance Change Threshold. This is the initial threshold used to grow the
CF tree. If inserting a given case into a leaf of the CF tree would yield tightness
less than the threshold, the leaf is not split. If the tightness exceeds the
threshold, the leaf is split.
Maximum Branches (per leaf node). The maximum number of child nodes that
a leaf node can have.
Maximum Tree Depth. The maximum number of levels that the CF tree can
have.
Maximum Number of Nodes Possible. This indicates the maximum number of
CF tree nodes that could potentially be generated by the procedure, based on
the function (b d+1 1) / (b 1), where b is the maximum branches and d is the
maximum tree depth. Be aware that an overly large CF tree can be a drain on
system resources and can adversely affect the performance of the procedure.
At a minimum, each node requires 16 bytes.

Cluster Model Update. This group allows you to import and update a cluster model
generated in a prior analysis. The input file contains the CF tree in XML format. The
model will then be updated with the data in the active file. You must select the
variable names in the main dialog box in the same order in which they were specified
in the prior analysis. The XML file remains unaltered, unless you specifically write the
new model information to the same filename. See the topic TwoStep Cluster Analysis
Output for more information.

If a cluster model update is specified, the options pertaining to generation of the CF

tree that were specified for the original model are used. More specifically, the distance
measure, noise handling, memory allocation, or CF tree tuning criteria settings for the
saved model are used, and any settings for these options in the dialog boxes are
ignored.
Note: When performing a cluster model update, the procedure assumes that none of
the selected cases in the active dataset were used to create the original cluster model.
The procedure also assumes that the cases used in the model update come from the
same population as the cases used to create the original model; that is, the means and
variances of continuous variables and levels of categorical variables are assumed to be
the same across both sets of cases. If your "new" and "old" sets of cases come from
heterogeneous populations, you should run the TwoStep Cluster Analysis procedure
on the combined sets of cases for the best results.
To Set Options for TwoStep Cluster Analysis
This feature requires the Statistics Base option.
1. From the menus choose:
Analyze > Classify > TwoStep Cluster...
2. In the TwoStep Cluster Analysis dialog box, click Options.
3. Change the settings as needed.
TwoStep Cluster Analysis Output
Output. This group provides options for displaying the clustering results.

Pivot tables. Results are displayed in pivot tables.

Charts and tables in Model Viewer. Results are displayed in the Model Viewer.
Evaluation fields. This calculates cluster data for variables that were not used
in cluster creation. Evaluation fields can be displayed along with the input
features in the model viewer by selecting them in the Display subdialog. Fields
with missing values are ignored.

Working Data File. This group allows you to save variables to the active dataset.

Create cluster membership variable. This variable contains a cluster

identification number for each case. The name of this variable is tsc_n, where n
is a positive integer indicating the ordinal of the active dataset save operation
completed by this procedure in a given session.

XML Files. The final cluster model and CF tree are two types of output files that can be
exported in XML format.

Export final model. The final cluster model is exported to the specified file in
XML (PMML) format. You can use this model file to apply the model
information to other data files for scoring purposes. See the topic Scoring
Wizard for more information.
Export CF tree. This option allows you to save the current state of the cluster
tree and update it later using newer data. See TwoStep Cluster Analysis Options
for more information on reading this file.

The Cluster Viewer

Cluster models are typically used to find groups (or clusters) of similar records based
on the variables examined, where the similarity between members of the same group
is high and the similarity between members of different groups is low. The results can
be used to identify associations that would otherwise not be apparent. For example,
through cluster analysis of customer preferences, income level, and buying habits, it
may be possible to identify the types of customers who are more likely to respond to a
particular marketing campaign.
There are two approaches to interpreting the results in a cluster display:

Examine clusters to determine characteristics unique to that cluster. Does one

cluster contain all the high-income borrowers? Does this cluster contain more
records than the others?
Examine fields across clusters to determine how values are distributed among
clusters. Does one's level of education determine membership in a cluster? Does
a high credit score distinguish between membership in one cluster or another?

Using the main views and the various linked views in the Cluster Viewer, you can gain
insight to help you answer these questions.
Who uses clustering?
Clustering techniques are useful in a wide variety of situations, including:

Market Segmentation. Identify distinct groups among a customer base,

allowing precise targeting of sales efforts.
Product Bundling. Identify groups of products that tend to appeal to specific
customer types.
Formal Classification. Classify groups, such as plants or animals into formal
taxonomies.
Medical Diagnosis. Use biological patterns to uncover rules for identifying or
diagnosing medical disorders.

To see information about the cluster model, activate (double-click) the Model Viewer
object in the Viewer.
Cluster Viewer

The Cluster Viewer is made up of two panels, the main view on the left and the linked,
or auxiliary, view on the right. There are two main views:

Model Summary (the default). See the topic Model Summary View for more
information.
Clusters. See the topic Clusters View for more information.

There are four linked/auxiliary views:

Predictor Importance. See the topic Cluster Predictor Importance View for
more information.
Cluster Sizes (the default). See the topic Cluster Sizes View for more
information.
Cell Distribution. See the topic Cell Distribution View for more information.
Cluster Comparison. See the topic Cluster Comparison View for more
information.
Model Summary View
Clusters View
Cluster Predictor Importance View
Cluster Sizes View
Cell Distribution View
Cluster Comparison View

Model Summary View

The Model Summary view shows a snapshot, or summary, of the cluster model,
including a Silhouette measure of cluster cohesion and separation that is shaded to
indicate poor, fair, or good results. This snapshot enables you to quickly check if the
quality is poor, in which case you may decide to return to the modeling node to amend
the cluster model settings to produce a better result.
The results of poor, fair, and good are based on the work of Kaufman and Rousseeuw
(1990) regarding interpretation of cluster structures. In the Model Summary view, a
good result equates to data that reflects Kaufman and Rousseeuw's rating as either
reasonable or strong evidence of cluster structure, fair reflects their rating of weak
evidence, and poor reflects their rating of no significant evidence.
The silhouette measure averages, over all records, (BA) / max(A,B), where A is the
record's distance to its cluster center and B is the record's distance to the nearest
cluster center that it doesn't belong to. A silhouette coefficient of 1 would mean that
all cases are located directly on their cluster centers. A value of 1 would mean all
cases are located on the cluster centers of some other cluster. A value of 0 means, on
average, cases are equidistant between their own cluster center and the nearest other
cluster.
The summary includes a table that contains the following information:

Algorithm. The clustering algorithm used, for example, "TwoStep".

Input Features. The number of fields, also known as inputs or predictors.
Clusters. The number of clusters in the solution.

Clusters View
The Clusters view contains a cluster-by-features grid that includes cluster names, sizes,
and profiles for each cluster.
The columns in the grid contain the following information:

Cluster. The cluster numbers created by the algorithm.

Label. Any labels applied to each cluster (this is blank by default). Double-click
in the cell to enter a label that describes the cluster contents; for example,
"Luxury car buyers".
Description. Any description of the cluster contents (this is blank by default).
Double-click in the cell to enter a description of the cluster; for example, "55+
years of age, professionals, earning over $100,000".
Size. The size of each cluster as a percentage of the overall cluster sample. Each
size cell within the grid displays a vertical bar that shows the size percentage
within the cluster, a size percentage in numeric format, and the cluster case
counts.
Features. The individual inputs or predictors, sorted by overall importance by
default. If any columns have equal sizes they are shown in ascending sort order
of the cluster numbers.
Overall feature importance is indicated by the color of the cell background
shading; the most important feature is darkest; the least important feature is
unshaded. A guide above the table indicates the importance attached to each
feature cell color.

When you hover your mouse over a cell, the full name/label of the feature and the
importance value for the cell is displayed. Further information may be displayed,
depending on the view and feature type. In the Cluster Centers view, this includes the
cell statistic and the cell value; for example: Mean: 4.32. For categorical features the
cell shows the name of the most frequent (modal) category and its percentage.
Within the Clusters view, you can select various ways to display the cluster
information:

Transpose clusters and features. See the topic Transpose Clusters and Features
for more information.
Sort features. See the topic Sort Features for more information.
Sort clusters. See the topic Sort Clusters for more information.
Select cell contents. See the topic Cell Contents for more information.

Transpose Clusters and Features

Sort Features

Sort Clusters
Cell Contents

Cluster Predictor Importance View

The Predictor Importance view shows the relative importance of each field in
estimating the model.
Cluster Sizes View
The Cluster Sizes view shows a pie chart that contains each cluster. The percentage
size of each cluster is shown on each slice; hover the mouse over each slice to display
the count in that slice.
Below the chart, a table lists the following size information:

The size of the smallest cluster (both a count and percentage of the whole).
The size of the largest cluster (both a count and percentage of the whole).
The ratio of size of the largest cluster to the smallest cluster.

Cell Distribution View

The Cell Distribution view shows an expanded, more detailed, plot of the distribution
of the data for any feature cell you select in the table in the Clusters main panel.
Cluster Comparison View
The Cluster Comparison view consists of a grid-style layout, with features in the rows
and selected clusters in the columns. This view helps you to better understand the
factors that make up the clusters; it also enables you to see differences between
clusters not only as compared with the overall data, but with each other.
To select clusters for display, click on the top of the cluster column in the Clusters main
panel. Use either Ctrl-click or Shift-click to select or deselect more than one cluster for
comparison.
Note: You can select up to five clusters for display.
Clusters are shown in the order in which they were selected, while the order of fields is
determined by the Sort Features By option. When you select Within-Cluster
Importance, fields are always sorted by overall importance .
The background plots show the overall distributions of each features:

Categorical features are shown as dot plots, where the size of the dot indicates
the most frequent/modal category for each cluster (by feature).
Continuous features are displayed as boxplots, which show overall medians and
the interquartile ranges.

Overlaid on these background views are boxplots for selected clusters:

For continuous features, square point markers and horizontal lines indicate the
median and interquartile range for each cluster.
Each cluster is represented by a different color, shown at the top of the view.

Navigating the Cluster Viewer

The Cluster Viewer is an interactive display. You can:

Select a field or cluster to view more details.

Compare clusters to select items of interest.
Alter the display.
Transpose axes.

Using the Toolbars

You control the information shown in both the left and right panels by using the
toolbar options. You can change the orientation of the display (top-down, left-to-right,
or right-to-left) using the toolbar controls. In addition, you can also reset the viewer to
the default settings, and open a dialog box to specify the contents of the Clusters view
in the main panel.
The Sort Features By, Sort Clusters By, Cells, and Display options are only available
when you select the Clusters view in the main panel. See the topic Clusters View for
more information.
Table 1. Toolbar icons
Icon

Topic
See Transpose Clusters and Features
See Sort Features By
See Sort Clusters By
See Cells

Control Cluster View Display

To control what is shown in the Clusters view on the main panel, click the Display
button; the Display dialog opens.
Features. Selected by default. To hide all input features, deselect the check box.
Evaluation Fields. Choose the evaluation fields (fields not used to create the cluster
model, but sent to the model viewer to evaluate the clusters) to display; none are

shown by default. Note: This check box is unavailable if no evaluation fields are
available.
Cluster Descriptions. Selected by default. To hide all cluster description cells, deselect
the check box.
Cluster Sizes. Selected by default. To hide all cluster size cells, deselect the check box.
Maximum Number of Categories. Specify the maximum number of categories to
display in charts of categorical features; the default is 20.
Filtering Records
If you want to know more about the cases in a particular cluster or group of clusters,
you can select a subset of records for further analysis based on the selected clusters.
1. Select the clusters in the Cluster view of the Cluster Viewer. To select multiple
clusters, use Ctrl-click.
2. From the menus choose:
Generate > Filter Records...
3. Enter a filter variable name. Records from the selected clusters will receive a
value of 1 for this field. All other records will receive a value of 0 and will be
excluded from subsequent analyses until you change the filter status.
4. Click OK.

Example. Car manufacturers need to be able to appraise the current market to

determine the likely competition for their vehicles. If cars can be grouped according to
available data, this task can be largely automatic using cluster analysis.
Information for various makes and models of motor vehicles is contained in
car_sales.sav. See the topic Sample Files for more information. Use the TwoStep
Cluster Analysis procedure to group automobiles according to their prices and physical
properties.
Retail and consumer product companies regularly apply clustering techniques to data
that describe their customers' buying habits, gender, age, income level, etc. These
companies tailor their marketing and product development strategies to each
consumer group to increase sales and build brand loyalty.

Running the Analysis

1. To run a TwoStep Cluster Analysis, from the menus choose:
2. Analyze > Classify > TwoStep Cluster...
3. If the variable list does not display variable labels in file order, right-click
anywhere in the variable list and from the context menu choose Display
Variable Labels and Sort by File Order.
4. Select Vehicle type as a categorical variable.
5. Select Price in thousands through Fuel efficiency as continuous variables.
6. Click Output.
7. Select Sales in thousands [sales] and 4-year resale value [resale] as evaluation
fields. These fields will not be used to create the cluster model, but can give
you further insight to the clusters created by the procedure.
8. Click Continue and then click OK.

Figure 1. TwoStep Cluster Analysis main dialog box

Figure 2. TwoStep Cluster Analysis main dialog box

Model Summary and Cluster Quality

Figure 1. Model Summary view

The Viewer contains a Model Viewer object. By activating (double-clicking) this object,
you gain an interactive view of the model. The default main view is the Model
Summary view.

The model summary table indicates that three clusters were found based on
the ten input features (fields) you selected.

The cluster quality chart indicates that the overall model quality is "Fair".

Cluster Distribution
Figure 1. Cluster distribution table

The Cluster Sizes view shows the frequency of each cluster. Hovering over a slice in the
pie chart reveals the number of records assigned to the cluster. 40.8% (62) of the
records were assigned to the first cluster, 25.7% (39) to the second, and 33.6% (51) to
the third.
Cluster Profiles
1. In the main view, select Clusters from the dropdown to display the Clusters
view.
Figure 1. Clusters table

By default, clusters are sorted from left to right by cluster size, so they are currently
ordered 1, 3, 2.
Figure 2. Cluster profiles: cells show cluster centers

The cluster means suggest that the clusters are well separated.
o

Motor vehicles in cluster 1 are cheap, small, and fuel efficient

automobiles, except for a single truck (the 1.6% of the cluster not
comprised of automobiles).

Motor vehicles in cluster 2 (column 3) are moderately priced, heavy,

and have a large gas tank, presumably to compensate for their poor fuel
efficiency. Cluster 2 is also entirely comprised of trucks.

Motor vehicles in cluster 3 (column 2) are expensive, large, and are

moderately fuel efficient automobiles.

Figure 3. Cluster profiles: cells show absolute distributions

2. The cluster means (for continuous fields) and modes (for categorical fields) are
useful, but only give information about the cluster centers. In order to get a
visualization of the distribution of values for each field by cluster, click on the
Cells show absolute distributions button in the toolbar.
Now you can see, for example, that there is some overlap between clusters 1 and 3 on
curb weight, engine size, and fuel capacity. There is considerably more overlap
between clusters 2 and 3 on these fields, with the difference that the vehicles with the
very highest curb weight and fuel capacity are in cluster 2 (column 3) and the vehicles
with the very highest engine size appear to be in cluster 3 (column 2).
Figure 4. Cluster profiles: cells show absolute distributions

3. To see this information for the evaluation fields, click on the Display button in
the toolbar.
4. Select Evaluation fields.
5. Click OK.
The evaluation fields should now appear in the cluster table.
Figure 5. Cluster profiles for evaluation fields: cells show absolute distributions

The distribution of sales is similar across clusters, except that clusters 1 and 2 have
longer tails than cluster 3 (column 2). There is a fair amount of overlap in the
distributions of 4-year resale value, but clusters 2 and 3 are centered on a higher value
than cluster 1, and cluster 3 has a longer tail than either cluster 1 or 2.
6. For another way to compare clusters, select (control-click) on the cluster
numbers (column headings) in the clusters table.
7. In the auxiliary view, select Cluster Comparison from the dropdown.
Figure 6. Cluster comparison view : first four fields shown

For each categorical field, this shows a dot plot for the modal category of each cluster,
with dot size corresponding to the percentage of records. For continuous fields, this
shows a boxplot for the distribution of values within each cluster overlaid on a boxplot
for the distribution of values overall. These plots generally confirm what you've seen in
the Clusters view. The Cluster Comparison view can be especially helpful when there
are many clusters, and you want to compare only a few of them.