1.2. Data Drift

AI System Design Interview Guide (2025)

ML System Design — Monitoring & Observability

1.2. Data Drift

5 min read 935 words

🪄 Step 1: Intuition & Motivation

Core Idea (in 1 short paragraph): Data drift happens when the kind of data your model sees in production starts to look different from the data it was trained on. The model might still work for a while, but its predictions will slowly lose relevance — like using last year’s weather to predict today’s crop yields. Detecting data drift early helps us catch those silent changes before they turn into bad business outcomes.
Simple Analogy: Imagine you trained a chef robot to cook based on your kitchen ingredients — salt, flour, sugar. But one day, the flour supplier starts sending almond flour instead of wheat. The robot’s recipes still run, but the results taste wrong. That’s data drift — the inputs changed subtly, but your system didn’t notice.

🌱 Step 2: Core Concept

What’s Happening Under the Hood?

When your model was trained, it “learned patterns” based on how input features looked — their ranges, correlations, and frequencies.
In production, new data keeps coming. Over time, those feature distributions might shift:

A numeric feature (like age) might see a different mean or spread.
A categorical feature (like browser type) might have new dominant categories.
Some features might appear missing or heavily skewed due to upstream data changes.

Monitoring systems compare training (reference) distributions with live (production) distributions to quantify how different they are.
If the difference crosses a threshold, we flag it as potential drift.

Why It Works This Way

The entire purpose of training a model is to capture stable relationships in data. But real-world data often changes because of seasonality, market behavior, user preferences, or pipeline bugs.
By comparing the shape of input distributions between training and production, we catch early warnings that “the world the model learned from” isn’t the same anymore.

How It Fits in ML Thinking

Data drift is like preventive maintenance for ML systems.
While concept drift affects the relationship between inputs and outputs, data drift focuses only on inputs themselves.
Detecting data drift ensures the foundation — your model’s feature inputs — is still valid before checking downstream metrics like accuracy or recall.

📐 Step 3: Mathematical Foundation

Let’s explore the most common ways to quantify drift — these are the “numbers behind the intuition.”

Kolmogorov–Smirnov (KS) Test

$$ D_{KS} = \sup_x |F_1(x) - F_2(x)| $$

$F_1(x)$: cumulative distribution function (CDF) of training data.
$F_2(x)$: CDF of production data.
$\sup_x$: the maximum vertical distance between the two curves.

It measures how far apart two distributions are, regardless of shape. If $D_{KS}$ exceeds a threshold (say 0.1–0.2), you suspect drift.

Plot both data distributions on one graph.
If their cumulative curves overlap closely → stable data.
If they diverge significantly → something’s changed.

Population Stability Index (PSI)

$$ PSI = \sum_i (p_i - q_i) \ln\left(\frac{p_i}{q_i}\right) $$

$p_i$: proportion of samples in bin $i$ (training/reference data).
$q_i$: proportion of samples in bin $i$ (production/live data).

A heuristic rule:

PSI < 0.1 → stable
0.1 ≤ PSI < 0.25 → moderate drift
PSI ≥ 0.25 → significant drift

It’s like comparing two histograms bin by bin — if counts shift a lot, PSI rises.

Jensen–Shannon Divergence (JSD)

$$ JSD(P || Q) = \frac{1}{2} D_{KL}(P || M) + \frac{1}{2} D_{KL}(Q || M) $$

where $M = \frac{1}{2}(P + Q)$ and $D_{KL}$ is the Kullback-Leibler divergence.

$P$, $Q$: training and live distributions.
$JSD$ is always bounded between 0 and 1.

It measures how much “information” one distribution loses when trying to approximate another — smaller is better.

Earth Mover’s Distance (EMD)

$$ EMD(P, Q) = \inf_{\gamma \in \Gamma(P, Q)} \int |x - y| \, d\gamma(x, y) $$

It asks: How much “effort” (mass × distance) is needed to move one distribution into another?

Imagine one distribution as piles of earth and another as holes.
The more effort to fill the holes, the more drift you have.

🧠 Step 4: Assumptions or Key Ideas

The training dataset captures “normal” variation — production data should roughly follow it.
You have enough samples in production to make statistical comparisons.
Drift metrics are computed feature-wise (each column separately), not on the joint distribution.
Detecting drift ≠ model failure — it’s a flag for further investigation.

⚖️ Step 5: Strengths, Limitations & Trade-offs

Provides early warning signs before model accuracy tanks.
Can detect pipeline or upstream data issues without needing labels.
Simple to automate and visualize (histograms, PSI dashboards).

Only looks at inputs, not outcomes — may miss concept drift.
Sensitive to sampling bias or small production volumes.
Requires baseline data storage and periodic recomputation.

Frequent checks vs. compute cost: More frequent detection gives faster insights but consumes resources.
Feature selection trade-off: Monitoring every feature adds clarity but clutters alerts; choose key features.
Threshold tuning: Conservative thresholds cause alert floods; liberal ones delay response.

🚧 Step 6: Common Misunderstandings

🚨 Common Misunderstandings (Click to Expand)

“If drift is detected, the model is broken.”
Not always — some shifts are benign (e.g., holidays, short-term marketing campaigns).
“Drift detection requires labels.”
No — data drift is label-agnostic. It’s about comparing feature distributions only.
“PSI or KS alone is enough.”
Real monitoring uses a combination of metrics plus domain thresholds and visual checks.

🧩 Step 7: Mini Summary

🧠 What You Learned: Data drift tracks how your model’s inputs evolve after deployment — like checking if your ingredients still match your original recipe.

⚙️ How It Works: Compare training vs. production feature distributions using metrics like KS, PSI, or JSD.

🎯 Why It Matters: It’s the earliest warning signal for pipeline bugs, user behavior changes, or silent system degradation.

1.3. Concept Drift 1.10. End-to-End System Design Synthesis