1.5. Data Quality and Integrity Checks

AI System Design Interview Guide (2025)

ML System Design — Monitoring & Observability

4 min read 753 words

🪄 Step 1: Intuition & Motivation

Core Idea (in 1 short paragraph): Even the best model fails if it’s fed bad data. Data quality checks are your early warning radar — they catch silent, upstream failures before they corrupt your training, predictions, or retraining loops. These checks make sure the data you’re trusting actually makes sense — that columns haven’t swapped, values aren’t missing, and distributions are still sane.
Simple Analogy: Think of data pipelines like a restaurant kitchen. Even a Michelin-star chef (your model) will serve terrible food if the ingredients are spoiled, mislabeled, or missing. Data quality checks are the kitchen’s hygiene inspections — quiet but crucial guardians of everything that follows.

🌱 Step 2: Core Concept

What’s Happening Under the Hood?

When data flows from multiple sources (databases, APIs, event streams) into your pipeline, things can silently go wrong:

Columns renamed or dropped after schema updates.
Numeric features suddenly becoming categorical (e.g., “10” → “ten”).
Missing or duplicate rows due to partial ingestion.
Feature leakage — where future data sneaks into training.

Data validation systems compare incoming data to a reference schema and statistical expectations.
They check for:

Schema integrity: Are columns, types, and constraints consistent?
Value integrity: Are there nulls, duplicates, or range violations?
Statistical sanity: Are mean, variance, and distributions similar to reference data?

If violations exceed thresholds, the system flags or blocks the data before it reaches training or inference.

Why It Works This Way

Models assume training and production data follow the same structure and meaning.
When data pipelines break, models may still “run,” producing outputs that look valid but are logically garbage.
Upfront validation creates guardrails — catching issues at the ingestion stage rather than in user-facing predictions.

How It Fits in ML Thinking

Data validation complements monitoring:

Monitoring checks live behavior (drift, performance).
Validation ensures input integrity before it ever reaches monitoring.

Together, they form a “belt and suspenders” system — protecting the model from both upstream and downstream failure modes.

📐 Step 3: Mathematical Foundation

Let’s make the idea of data quality metrics concrete. These metrics quantify how “healthy” the data is before it reaches your model.

Missing Value Rate

$$ \text{MissingRate}(X_j) = \frac{\text{Count of Missing Values in Feature } j}{\text{Total Rows}} $$

If this spikes above a baseline threshold (e.g., 2% → 20%), something in upstream ingestion likely failed.

High missingness doesn’t just reduce data size — it can break the very signals your model depends on.

Outlier Ratio

$$ \text{OutlierRatio}(X_j) = \frac{\text{Count of Values Beyond } [\mu_j \pm k\sigma_j]}{\text{Total Rows}} $$

$\mu_j$: mean of feature $j$
$\sigma_j$: standard deviation
$k$: typical threshold (e.g., 3 for “3-sigma rule”)

If outliers suddenly spike, it could mean a pipeline unit change (“price” now in cents instead of dollars).

Schema Deviation Rate

$$ \text{SchemaDeviationRate} = \frac{\text{Number of Columns with Mismatched Type or Missing}}{\text{Total Columns}} $$

A tiny schema mismatch (float → string) can cause massive downstream failures if unchecked.

🧠 Step 4: Assumptions or Key Ideas

The reference schema (data contract) is reliable and version-controlled.
Validation thresholds reflect natural variation, not transient noise.
Checks run automatically at data ingestion and pre-training.
Violations lead to actionable responses (alerts, retries, quarantine).

⚖️ Step 5: Strengths, Limitations & Trade-offs

Catches silent upstream issues before they poison the model.
Enforces reproducibility and stability across datasets.
Prevents schema mismatches during retraining or integration.

Overly strict checks may block benign data (false positives).
Requires ongoing maintenance as schemas evolve.
Doesn’t detect subtle semantic drifts (same column name, new meaning).

Strictness vs. Flexibility: Too rigid → brittle pipelines; too lenient → hidden failures.
Frequency vs. Cost: Continuous validation is robust but resource-heavy.
Automation vs. Oversight: Full automation is fast but can miss nuanced domain anomalies.

🚧 Step 6: Common Misunderstandings

🚨 Common Misunderstandings (Click to Expand)

“Validation = Monitoring.”
Validation happens before the model runs; monitoring happens after.
“Great Expectations or TFDV is plug-and-play.”
Frameworks still need domain thresholds and human interpretation.
“Schema checks are enough.”
Even perfect schemas can hide semantic issues — like columns filled with zeroes after an ETL bug.

🧩 Step 7: Mini Summary

🧠 What You Learned: Data validation acts as the immune system of your ML pipeline — catching corrupted, missing, or mismatched data before it breaks your model.

⚙️ How It Works: Compare incoming data to schema + statistical expectations → compute quality metrics → trigger alerts or block ingestion if violated.

🎯 Why It Matters: Prevents silent, cascading failures — ensuring your monitoring pipeline watches good data, not garbage.

1.6. Model Explainability in Monitoring 1.4. Model Performance Monitoring