Skip to content
Machine Learning

2.1 Clustering Overview

Clustering in machine learning is the task of grouping a set of objects in such a way that objects in the same group (cluster) are more similar to each other than to those in other groups. It is an unsupervised learning approach where the algorithm automatically finds the patterns or structure in data without prior labels.

1. Techniques of Clustering

  1. K-Means Clustering

    Overview: K-means clustering aims to partition observations into clusters where each observation belongs to the cluster with the nearest mean.

    • Algorithm:

      1. Initialization: Choose initial centroids randomly.
      2. Assignment: Assign each data point to the nearest centroid.
      3. Update Centroids: Recalculate the centroid as the mean of all points assigned to it.
      4. Repeat: Iterate steps 2 and 3 until convergence (no change in centroids or minimal change).

    • Pros: Simple, efficient for large datasets, easy to interpret clusters.

    • Cons: Requires predefined , sensitive to initial centroids, may converge to local optima.

  2. Hierarchical Clustering

    Overview: Hierarchical clustering builds a hierarchy of clusters either from the bottom up (agglomerative) or from the top down (divisive).

    • Agglomerative Algorithm:

      1. Start: Each data point is a cluster.
      2. Merge: Iteratively merge the closest pairs of clusters until all points belong to one cluster.
      3. Hierarchy: Represented as a dendrogram showing the merging process.

    • Divisive Algorithm: Starts with one cluster containing all data points and recursively splits into smaller clusters.

    • Pros: No need to specify , provides a hierarchy of clusters.

    • Cons: Computationally intensive for large datasets, dendrogram interpretation can be subjective.

  3. Density-Based Spatial Clustering of Applications with Noise (DBSCAN)

    Overview: DBSCAN groups together points that are closely packed together (dense regions) and marks points as outliers (noise) that lie alone in low-density regions.

    • Algorithm:

      1. Core Points: Points with at least points within a radius .
      2. Border Points: Points within of a core point but with fewer than neighbors.
      3. Noise Points: Points that are neither core nor border points.

    • Pros: Can discover clusters of arbitrary shapes, robust to noise and outliers.

    • Cons: Requires tuning of and , sensitive to density variations.

  4. Gaussian Mixture Models (GMM)

    Overview: GMM assumes that the data is generated from a mixture of several Gaussian distributions, each associated with a cluster.

    • Algorithm:

      1. Initialization: Choose initial parameters for Gaussian distributions.
      2. Expectation-Maximization (EM) Algorithm: Iteratively update parameters:
        • E-step: Calculate probability of each point belonging to each cluster (using current parameters).
        • M-step: Update parameters (mean, covariance, and mixture coefficients) based on the expected cluster assignments.

    • Pros: Can model complex cluster shapes, provides probabilistic cluster assignments.

    • Cons: Sensitive to initialization, slower convergence compared to K-means.

2. Progression of Techniques

  • Early Techniques:

    • 1960s-1970s: Basic clustering algorithms like K-means and hierarchical clustering were developed, primarily focusing on partitioning data into non-overlapping clusters.

  • Advancements in 1980s-1990s:

    • Density-Based Methods: DBSCAN introduced the concept of density-based clustering, which could handle noise and outliers effectively.
    • Hierarchical Clustering: Algorithms for hierarchical clustering improved, providing better visualization and hierarchy of clusters.

  • Modern Techniques (2000s onwards):

    • Probabilistic Models: GMM and Bayesian methods brought probabilistic interpretations to clustering, allowing uncertainty modeling.
    • Deep Learning Approaches: Embedding-based clustering using autoencoders and deep neural networks to learn data representations for clustering.

  • Current Trends:

    • Big Data Clustering: Scalable algorithms for large datasets, such as parallel K-means and streaming clustering algorithms.
    • Hybrid Approaches: Combining different clustering techniques or integrating clustering with other machine learning tasks (e.g., semi-supervised learning).

===================

Clustering as a PreProcessing Tool

Using clustering as a preprocessing technique is a smart way to extract structure from your data before applying other machine learning algorithms. Clustering groups similar data points together, and you can use the cluster assignments to enrich or transform your dataset. Here’s how and when to do that, with examples to make it concrete:

✅ Ways to Use Clustering for Preprocessing:

1. Feature Engineering: Add Cluster Labels as Features

  • After clustering, assign each data point a cluster label (e.g., from k-means).
  • This label becomes a new categorical feature that can be used in models like logistic regression, decision trees, etc.

Example:

  • Dataset: Customer data with features like age, income, and spending score.
  • Apply K-Means (k=5) → get cluster labels.
  • Add “cluster_id” as a new column.
  • Train a classifier (e.g., predict who will churn) using the enriched dataset.

2. Dimensionality Reduction

  • Instead of using raw high-dimensional features, use cluster centroids or distances to centroids.
  • Especially useful when dealing with sparse or noisy data.

Example:

  • Text documents represented as TF-IDF vectors.
  • Apply clustering (e.g., k-means or hierarchical clustering).
  • Replace original vectors with:
    • Distances to each cluster center (k features), or
    • Just the cluster label.

3. Outlier Detection / Noise Reduction

  • Clusters with very few members may indicate outliers.
  • You can remove or flag them before training your model.

Example:

  • Sensor data for anomaly detection.
  • After DBSCAN clustering, label points as “noise” and remove them.

4. Data Segmentation for Targeted Models

  • Use clustering to divide data into segments, then train separate models for each segment.

Example:

  • E-commerce user behavior data.
  • Cluster users into behavior groups.
  • Train a recommender model separately per cluster.

5. Initializing Other Algorithms

  • Some algorithms (e.g., EM for Gaussian Mixture Models) benefit from cluster-based initialization.

🛠 Common Clustering Algorithms for Preprocessing

  • K-Means – Fast, simple, works well with numeric data.
  • DBSCAN – Great for density-based noise detection.
  • Agglomerative Clustering – Good for hierarchical structure.
  • Gaussian Mixture Models (GMM) – Probabilistic soft clustering.

🧠 Quick Code Example (Python with scikit-learn)

from sklearn.cluster import KMeans
from sklearn.datasets import load_iris
from sklearn.ensemble import RandomForestClassifier
import pandas as pd


# Load data
data = load_iris()
X = data.data
y = data.target


# Apply clustering
kmeans = KMeans(n_clusters=3, random_state=0)
cluster_labels = kmeans.fit_predict(X)


# Add cluster labels as a feature
X_enriched = pd.DataFrame(X)
X_enriched['cluster'] = cluster_labels


# Train a classifier
clf = RandomForestClassifier()
clf.fit(X_enriched, y)

=========================================================================

Great question! Using distances to cluster centroids is a cool and simple technique for dimensionality reduction. Instead of keeping all original features, you represent each data point by how close it is to each cluster centroid — effectively reducing it to just k features, where k is the number of clusters.

🧠 How It Works

Let’s say you use k-means clustering with k = 5. You’ll end up with 5 centroids.
For each data point, calculate the Euclidean distance to each of the 5 centroids.
Now, instead of a high-dimensional input (maybe 100+ features), you just have 5 features — the distances.

This transforms your original feature space into a cluster-distance space.

✅ Why This Is Useful

  • Compresses high-dimensional data into a low-dimensional space.
  • Keeps meaningful structure based on similarity to groups.
  • Works well before classification or regression tasks.
  • Especially handy for sparse or noisy datasets (e.g., text or images).

🔧 Example in Python (Using scikit-learn)

from sklearn.datasets import load_iris
from sklearn.cluster import KMeans
from sklearn.metrics import pairwise_distances
import pandas as pd


# Load some sample data
X, _ = load_iris(return_X_y=True)


# Step 1: Apply KMeans clustering
k = 5
kmeans = KMeans(n_clusters=k, random_state=0)
kmeans.fit(X)


# Step 2: Compute distances to each centroid
centroids = kmeans.cluster_centers_
distances = pairwise_distances(X, centroids)


# distances.shape = (num_samples, k) -> reduced features
X_reduced = pd.DataFrame(distances, columns=[f'dist_to_cluster_{i}' for i in range(k)])


# Now X_reduced can be used for further ML tasks
print(X_reduced.head())

📊 Before vs After

Original FeaturesReduced Features (after clustering)
Sepal length, width, … (4 features)dist_to_cluster_0, …, dist_to_cluster_4 (5 features)

In real datasets with 100s or 1000s of features, this method gives a huge reduction while still preserving useful information.

🧪 Bonus: Works with Any Clustering

You can do this with k-means, GMM, or even hierarchical clustering, as long as you can define a “centroid” or “representative” for each cluster.

Want to see this applied to text data, images, or your own dataset? Just drop an example and I’ll help set it up!

=======================================================================

Vector Qualitizaion

Great question — vector quantization (VQ) is closely related to clustering, especially k-means clustering. In fact, you can think of vector quantization as a signal processing twist on the idea of clustering.

🧠 What is Vector Quantization?

Vector quantization is a technique used to compress high-dimensional data by mapping it to a finite set of representative points — called a codebook. Each data point is approximated by its nearest representative vector (a codeword).

This is used in:

  • Image compression
  • Speech encoding
  • Audio/video codecs
  • Pattern recognition

📌 Connection to Clustering (Especially K-Means)

VQ and k-means are essentially doing the same thing:

Vector QuantizationK-Means Clustering
CodewordsCluster centroids
CodebookSet of cluster centers
Encoding a vectorAssigning to nearest centroid
Quantization errorWithin-cluster sum of squares

In fact, k-means is often used to create the codebook in vector quantization.

So, if you:

  1. Apply k-means to your data
  2. Store the centroids
  3. Approximate any input vector by its nearest centroid

🎉 You’ve just performed vector quantization.

📉 Why Use Vector Quantization?

  • Compression: Replace each high-dimensional vector with an index (which points to a codeword).
  • Speed: Processing and storing indices is faster than working with original vectors.
  • Noise Reduction: Helps in generalizing data by using prototypes.

🔧 Simple Example (Python-ish Pseudo-Code)

from sklearn.cluster import KMeans
import numpy as np


# Suppose we have 2D data
X = np.array([[1, 2], [2, 1], [8, 9], [9, 8]])


# Step 1: Build the codebook using KMeans
kmeans = KMeans(n_clusters=2).fit(X)
codebook = kmeans.cluster_centers_


# Step 2: Quantize a vector (find nearest codeword)
vector = np.array([1.5, 1.5])
quantized_index = np.argmin(np.linalg.norm(codebook - vector, axis=1))
quantized_vector = codebook[quantized_index]

Now quantized_vector is the compressed version of your original vector.

🧠 Summary

  • Vector quantization = representing vectors by the nearest centroid/codeword.
  • It’s basically k-means used for compression or encoding.
  • Often used in signal processing, image compression, and ML feature compression.

Want to go deeper into how it’s used in images (e.g., quantizing pixel colors) or in neural networks (e.g., quantized embeddings)? Let me know — it’s a fun topic!

==============================================================================

Ah, you’re diving into the core power of clustering now — the ability to discover hidden patterns or structures in data. Different clustering algorithms have different “superpowers” when it comes to this, depending on how they define and detect those patterns.

Let’s break it down.

🧠 What Do We Mean by “Hidden Patterns”?

Clustering algorithms try to group similar data points together based on some notion of distance, density, or distribution. The hidden patterns they discover could be:

  • Groups of similar customers (segmentation)
  • Anomalies or outliers (fraud, noise)
  • Natural categories in data (e.g., document topics, gene expression groups)
  • Structural relationships (e.g., hierarchy, manifolds)

🔍 Clustering Algorithms & Their Pattern Discovery Abilities

Here’s a breakdown of how different clustering algorithms find different kinds of patterns, and what they’re good at:

1. K-Means – “Centroid-Based Grouping”

  • 🔎 Finds: Spherical, equally sized clusters based on mean distance.
  • 🚫 Misses: Irregular shapes, clusters of varying density.
  • 🎯 Hidden patterns: Homogeneous groups (e.g., similar purchase behaviors).
  • ✅ Best when: Data is nicely separated and you want fast, scalable clustering.

2. DBSCAN (Density-Based Spatial Clustering of Applications with Noise)

  • 🔎 Finds: Arbitrary-shaped clusters based on density.
  • 🚫 Misses: Thin or very close dense regions may confuse it.
  • 🎯 Hidden patterns: Natural groupings + outliers or anomalies.
  • ✅ Best when: You’re working with spatial data, or want to detect noise.

3. Hierarchical Clustering (Agglomerative/Divisive)

  • 🔎 Finds: Nested cluster structures (hierarchies).
  • 🚫 Misses: Scales poorly for large datasets.
  • 🎯 Hidden patterns: Multi-level groupings (e.g., countries → regions → cities).
  • ✅ Best when: You suspect sub-clusters or want a dendrogram.

4. Gaussian Mixture Models (GMM) – “Soft Clustering”

  • 🔎 Finds: Elliptical clusters based on probability distributions.
  • 🚫 Misses: Non-Gaussian, irregular shapes.
  • 🎯 Hidden patterns: Overlapping clusters, uncertainty in group membership.
  • ✅ Best when: You need soft assignments or deal with probabilistic data.

5. Spectral Clustering

  • 🔎 Finds: Complex, non-convex shapes by using graph theory.
  • 🚫 Misses: Struggles with very large datasets or high noise.
  • 🎯 Hidden patterns: Graph-based clusters, community structures.
  • ✅ Best when: Data lies on a manifold or you need non-linear pattern detection.

6. HDBSCAN (Hierarchical DBSCAN)

  • 🔎 Finds: Clusters of varying density + hierarchy.
  • 🚫 Misses: Still sensitive to some parameter choices.
  • 🎯 Hidden patterns: Multi-scale clusters + better handling of outliers.
  • ✅ Best when: DBSCAN isn’t flexible enough and you want smarter density-based clustering.

🗺️ Summary Table

AlgorithmPattern Type DiscoveredHandles NoiseIrregular ShapesOverlapping Groups
K-MeansCompact, spherical❌❌❌
DBSCANDense regions, arbitrary shapes✅✅❌
HierarchicalNested groupings⚠️ (manual)✅⚠️ (not soft)
GMMElliptical, overlapping❌⚠️✅
SpectralNon-linear, graph-like⚠️✅✅
HDBSCANVarying-density clusters✅✅⚠️

🧪 Bonus Tip: Combine with Visualization

Use clustering + dimensionality reduction (like PCA or t-SNE or UMAP) to actually see the patterns being discovered — this is incredibly helpful for interpretability.

🧠 Final Thought

Each algorithm brings a different lens to the data. The “hidden pattern” it discovers depends on:

  • How similarity is defined
  • The shape and scale of clusters
  • Whether the data has noise, hierarchy, or overlap

If you tell me more about your data (structured? images? text?), I can help suggest the best clustering tool to discover those patterns.

===========================================

Awesome question — you’re touching on a more advanced and fascinating use of clustering: uncovering structural relationships like manifolds in high-dimensional data.

Let’s walk through what this means and how clustering plays a role.

🧠 First: What’s a Manifold?

A manifold is a smooth, low-dimensional structure embedded in a high-dimensional space.
Think of it like this:

  • A curve in 3D space is a 1D manifold.
  • A sheet (like a plane or sphere surface) is a 2D manifold in 3D space.

In data science, many high-dimensional datasets (e.g., images, speech signals, sensor data) actually lie on low-dimensional manifolds — they’re just twisted or curved in the high-dimensional space.

Clustering can help us understand the shape or structure of the manifold by grouping data points that lie on the same local region of it.

🔍 How Clustering Helps Discover Manifold Structures

Certain clustering methods, especially graph-based or density-based ones, are more capable of capturing local geometry and structure, revealing how the data is organized non-linearly.

📌 Example 1: Spectral Clustering on a Swiss Roll

Imagine a “Swiss Roll” — a 2D sheet rolled up into 3D space.

🔧 Steps:

  1. Generate or get Swiss roll data (scikit-learn has it).
  2. Apply Spectral Clustering.
    • It builds a similarity graph.
    • Applies graph Laplacian + eigen decomposition to find clusters.
  3. The result: Clusters that follow the curved surface — not the raw 3D distance.
from sklearn.datasets import make_swiss_roll
from sklearn.cluster import SpectralClustering
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D


X, _ = make_swiss_roll(n_samples=1000, noise=0.05)
sc = SpectralClustering(n_clusters=6, affinity='nearest_neighbors')
labels = sc.fit_predict(X)


fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(X[:, 0], X[:, 1], X[:, 2], c=labels, cmap='Spectral')

🧠 Insight: The clusters respect the manifold, rather than straight-line (Euclidean) distances.

📌 Example 2: Image Data on a Pose Manifold

Say you have images of a rotating object (e.g., a statue turning). Each frame is a high-dimensional vector (pixels), but the only change is the rotation — a 1D manifold in pixel space.

🧰 Use clustering (e.g., DBSCAN or spectral) on lower-dimensional representations (e.g., after t-SNE or UMAP).

This can reveal:

  • Discrete rotation stages (clustered poses).
  • Smooth progression along a curve → can even help generate new frames.

📌 Example 3: Sensor Data in Human Motion

Multiple IMU sensors on a person walking, running, turning — data lives on a nonlinear manifold of motion.

  • Use UMAP or Isomap for manifold learning.
  • Apply density-based clustering (like HDBSCAN) to group motion patterns.

🎯 Result: Natural groupings of movement types, even transitions between them.

✅ Best Clustering Algorithms for Manifold Structures

AlgorithmWhy it works
Spectral ClusteringUses graph Laplacian, captures manifold curvature
DBSCAN/HDBSCANFinds clusters based on density in local neighborhoods
Affinity PropagationUses similarity matrix, flexible shapes
GMM + PCA/IsomapModels curved clusters after unfolding manifolds

🔗 Combine with Manifold Learning Techniques

Clustering works even better when paired with methods that first try to unfold the manifold:

  • 🌀 Isomap – preserves geodesic distances.
  • 🌌 t-SNE / UMAP – great for visualizing and revealing local manifold structure.
  • 🔄 Autoencoders – learn a compressed (manifold-like) latent space.

Then you can cluster in the manifold space to find hidden structures.

🧠 Summary

Clustering can help uncover structural relationships like manifolds by:

  • Grouping points that lie on the same part of the surface
  • Revealing the nonlinear geometry of data
  • Working alongside manifold learning techniques for deeper insights

Want to try this on image data, time series, or something else? I’d be happy to show a custom example or walk through code with your dataset.