Gaussian Mixture Models

Clustering with probability, not hard boundaries

K-Means assigns every point to exactly one cluster. But real data is rarely so clear-cut. A customer might be a 70% “budget shopper” and 30% “premium shopper.” An email might be 80% likely spam and 20% likely a legitimate promotion.

Gaussian Mixture Models (GMMs) solve this with soft clustering — instead of assigning a hard label, they give each point a probability of belonging to each cluster.

This tutorial covers:

1. The intuition — what is a mixture of Gaussians?
2. Hard vs soft clustering.
3. The EM algorithm — how GMMs learn.
4. Full Python demonstration.
5. GMM vs K-Means — when to use which.

1. The intuition — overlapping bell curves

A Gaussian is a bell curve — a normal distribution described by a mean (centre) and variance (spread).

A Gaussian Mixture Model assumes that the data was generated by several bell curves mixed together:

Cluster 1: Gaussian centred at (2, 3), spread = 0.5
Cluster 2: Gaussian centred at (7, 1), spread = 1.2
Cluster 3: Gaussian centred at (5, 6), spread = 0.8

Each data point was drawn from one of these Gaussians. The model’s job is to figure out which Gaussians exist and what each point’s probability is for each Gaussian.

Memory trick: Imagine two overlapping spotlights on a stage. Every person on stage is lit partly by spotlight 1 and partly by spotlight 2. GMM figures out where the spotlights are and how bright each one shines on each person.

2. Hard vs soft clustering

import numpy as np

# K-Means: hard assignment
# Point [4.5, 2.0] → Cluster 2 (one definitive answer)

# GMM: soft assignment
# Point [4.5, 2.0] → [0.72, 0.21, 0.07]
#                     cluster1  cluster2  cluster3
# The point is 72% cluster 1, 21% cluster 2, 7% cluster 3

Soft assignments are useful when:

• Points genuinely sit between clusters
• You want a confidence level, not just a label
• Data comes from overlapping populations

3. The EM algorithm — how GMMs learn

GMMs are fitted using the Expectation-Maximisation (EM) algorithm:

E-step (Expectation): Given current Gaussian parameters, compute the probability that each point belongs to each Gaussian.

M-step (Maximisation): Given those probabilities, update the Gaussian parameters (means, covariances, weights) to best explain the data.

Repeat until the parameters stop changing.

Initialise: random Gaussian parameters
    ↓
E-step: for each point, compute P(cluster k | point)
    ↓
M-step: update means, covariances, and weights using those probabilities
    ↓
Repeat until convergence

4. Full Python demonstration

import numpy as np
import matplotlib.pyplot as plt
from sklearn.mixture import GaussianMixture
from sklearn.preprocessing import StandardScaler

# Generate data from three overlapping clusters
np.random.seed(42)
cluster1 = np.random.multivariate_normal([2, 3], [[0.5, 0.2],[0.2, 0.5]], 100)
cluster2 = np.random.multivariate_normal([7, 1], [[1.0, 0.0],[0.0, 1.0]], 100)
cluster3 = np.random.multivariate_normal([5, 6], [[0.8, 0.3],[0.3, 0.8]], 100)
X = np.vstack([cluster1, cluster2, cluster3])

# Fit GMM with 3 components
gmm = GaussianMixture(n_components=3, covariance_type='full', random_state=42)
gmm.fit(X)

# Hard labels (most likely cluster)
labels = gmm.predict(X)

# Soft probabilities
probs = gmm.predict_proba(X)
print("Point 0 probabilities:", probs[0].round(3))
# e.g. [0.961, 0.032, 0.007] → 96% cluster 0, 3% cluster 1, 1% cluster 2

# Model parameters
print("\nLearned cluster means:")
for i, mean in enumerate(gmm.means_):
    print(f"  Cluster {i}: {mean.round(2)}")

print("\nMixture weights (how common each cluster is):")
for i, w in enumerate(gmm.weights_):
    print(f"  Cluster {i}: {w:.3f}")

# Visualise
plt.figure(figsize=(8, 5))
scatter = plt.scatter(X[:, 0], X[:, 1], c=labels, cmap='tab10', alpha=0.6)
plt.scatter(gmm.means_[:, 0], gmm.means_[:, 1],
            marker='X', s=200, c='black', label='Cluster centres')
plt.title('Gaussian Mixture Model — Soft Clustering')
plt.legend()
plt.tight_layout()
plt.show()

Choosing the number of components

Use BIC (Bayesian Information Criterion) — lower is better:

bic_scores = []
n_range = range(1, 8)

for n in n_range:
    gmm = GaussianMixture(n_components=n, covariance_type='full', random_state=42)
    gmm.fit(X)
    bic_scores.append(gmm.bic(X))

best_n = n_range[np.argmin(bic_scores)]
print(f"Best number of components: {best_n}")

plt.plot(n_range, bic_scores, 'o-')
plt.xlabel('Number of components')
plt.ylabel('BIC score (lower = better)')
plt.title('Choosing number of GMM components')
plt.tight_layout()
plt.show()

Anomaly detection with GMM

Points with very low probability under the model are anomalies:

# Score each point (log-probability under the model)
log_probs = gmm.score_samples(X)

# Low log-probability = unusual = potential anomaly
threshold = np.percentile(log_probs, 5)   # bottom 5% = anomalies
anomalies = X[log_probs < threshold]

print(f"Detected {len(anomalies)} anomalies out of {len(X)} points")

5. Covariance types

GMM has four options for how each cluster’s shape is described:

for cov_type in ['spherical', 'diag', 'tied', 'full']:
    gmm = GaussianMixture(n_components=3, covariance_type=cov_type)
    gmm.fit(X)
    print(f"{cov_type:12s}  BIC: {gmm.bic(X):.1f}")

6. GMM vs K-Means

Use GMM when:

• Points genuinely overlap between clusters
• You need probability of cluster membership, not just a label
• Clusters have different shapes or sizes
• You want built-in anomaly detection

Use K-Means when:

• Clusters are well-separated and roughly spherical
• Speed is critical
• You have very large datasets

Summary

• GMM assumes data comes from multiple overlapping Gaussian (bell curve) distributions.
• Instead of hard cluster assignments, GMM gives probabilities for each cluster.
• The EM algorithm alternates between computing membership probabilities and updating the Gaussian parameters.
• Use BIC to choose the number of components.
• The covariance_type parameter controls the shape of each cluster.
• GMM doubles as an anomaly detector — low-probability points are unusual.