4.2. Apply to Real-World Problems

Customer data rarely behaves nicely.

• Some users buy weekly, others yearly.
• Some interact through multiple channels; others are silent.
• Purchase frequency, spending, and engagement vary widely — creating clusters with different densities.

K-Means might fail here because:

• It forces every user into a cluster (even the weird ones).
• It expects spherical groups — unrealistic in customer behavior.

HDBSCAN, however:

• Automatically detects dense customer segments without predefining how many exist.
• Treats low-activity or outlier customers as noise, not forced assignments.
• Uses stability to ensure that clusters represent real behavioral groups, not random noise.

Example intuition:

• Cluster 1: Frequent, high-spending buyers (dense region).
• Cluster 2: Seasonal shoppers (medium density).
• Cluster 3: Rare or dormant users (low density, often marked as noise).

Log data from servers, sensors, or security systems is noisy, high-volume, and unstructured. You often care less about the “main” clusters and more about the rare patterns — anomalies.

DBSCAN can detect anomalies as “points not assigned to any cluster,” but it depends heavily on the eps value — a single global threshold that rarely fits logs with multiple activity patterns.

HDBSCAN:

• Builds a hierarchy of density-based clusters across scales.
• Marks short-lived, low-persistence clusters and isolated points as anomalies automatically.
• Adapts to shifting density distributions common in real systems (e.g., different log activity peaks).

Practical Example:

• Cluster 1: Normal login events (very dense).
• Cluster 2: Scheduled maintenance logs (moderate density).
• Cluster 3: Rare, high-latency errors (low density → identified as outliers).

When you represent text using embeddings (e.g., BERT, Sentence Transformers), the resulting space is high-dimensional and uneven — topics have fuzzy boundaries and varying densities.

K-Means struggles here because:

• It forces a fixed number of clusters (k), but topics don’t have fixed boundaries.
• It assumes spherical shapes — embeddings are rarely isotropic.

HDBSCAN naturally fits:

• Finds semantic topic clusters of varying density and size.
• Marks unrelated or ambiguous sentences as noise.
• Works beautifully when combined with UMAP for dimensionality reduction before clustering.

Example intuition:

• Cluster 1: Tech news (dense, clear topic).
• Cluster 2: Sports articles (moderate density).
• Cluster 3: Abstract sentences spanning multiple topics (noise or weak clusters).

When clusters vary in density, you can’t pick a single parameter that works globally. Instead, use stability plots (from the condensed tree) to visually inspect which clusters persist across density levels.

Look for:

• Long, thick branches → stable, high-confidence clusters.
• Short, weak branches → likely noise or transient structures.

This visual approach replaces arbitrary numeric tuning with interpretability — you see where the data stabilizes.

If you expect groups of different scales (say, large corporate clients vs. small individual users), tune min_cluster_size proportionally to expected variance:

• Higher values → prefer large, consistent groups.
• Lower values → capture fine-grained or niche clusters.

• “HDBSCAN doesn’t need tuning.” While it automates eps, min_samples and min_cluster_size still require domain-aware selection.
• “Outliers are useless.” In anomaly detection and NLP, outliers often carry most of the insight (errors, novelty, or unique events).
• “You can replace K-Means with HDBSCAN everywhere.” For very large, homogeneous datasets (like text embeddings for millions of tweets), K-Means or MiniBatch K-Means might be better suited.