Clustering and Dimensionality Reduction

Introduction

So far every model had labels to learn from. In unsupervised learning there are no right answers, only data. The goal is to find hidden structure: groups of similar items, or a simpler way to describe the data.

k-means clustering

Clustering groups similar items. k-means is the classic method:

1. Choose the number of clusters, k.
2. Place k centre points at random.
3. Assign every item to its nearest centre.
4. Move each centre to the average of its items.
5. Repeat steps 3 and 4 until the groups stop changing.

We try it on the iris flower measurements, pretending we do not know the species:

from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
import numpy as np

X, _ = load_iris(return_X_y=True)
km = KMeans(n_clusters=3, n_init=10, random_state=42).fit(X)
print(np.bincount(km.labels_))

[62 50 38]

It found three groups of sizes 62, 50 and 38. In the real data each species has 50 flowers, so the clustering matched one species exactly and blurred the other two, because those species overlap.

Choosing k and judging clusters

• You must choose k. Try several values and look at measures such as the inertia (the total distance to centres) or the silhouette score, which compares how close items are to their own cluster versus others.
• Clusters are only useful if they make sense. Look at examples from each group, and ask whether a person would agree.
• k-means assumes roughly round, similar-sized clusters, and is affected by feature scales, so scale features first. Other methods, such as DBSCAN and hierarchical clustering, handle other shapes.

Uses: customer segments, grouping documents, image compression, finding unusual items.

Dimensionality reduction with PCA

Data can have dozens or thousands of features, which is hard to visualise and can hurt models. Principal component analysis (PCA) finds new axes, called components, that capture the most variation in the data. You keep only the first few.

from sklearn.decomposition import PCA

pca = PCA(n_components=2).fit(X)
print(pca.explained_variance_ratio_.round(3), round(pca.explained_variance_ratio_.sum(), 3))

[0.925 0.053] 0.978

The four iris measurements can be squashed to two components that keep 97.8 percent of the variation, so a flat two-dimensional chart shows almost everything.

Using PCA wisely

• Scale features before PCA when they use different units.
• Components are mixtures of the original features, so they are harder to explain.
• Use it for visualisation, removing noise, speeding up models or compressing data.
• It finds straight-line structure only. For complex shapes, other methods such as t-SNE and UMAP are used for visualisation, and should be interpreted carefully because distances in those plots can mislead.

Other unsupervised tasks

• Anomaly detection: finding items unlike the rest, such as fraud or machine faults
• Association rules: items often bought together
• Topic modelling: themes in a set of documents

Try it yourself

Run k-means on the iris data with k = 2, 3 and 4. For each, look at the inertia (km.inertia_). How does it change, and what does that suggest about choosing k?

Summary

• k-means groups unlabelled data around centres, and you choose k and check the groups make sense.
• PCA compresses many features into a few components that keep most of the variation.
• Unsupervised results need human judgement, because there is no answer key.