Principal Component Analysis (PCA)

🤖 Core Machine Learning Foundations

1.1: Build a Geometric and Statistical Intuition

• Visualize PCA as a rotation of coordinate axes — aligning them with directions of maximum variance.
• Understand that each principal component is an orthogonal axis capturing a fraction of total variance.
• Relate PCA to covariance structure: high covariance between features implies redundant information, which PCA eliminates.
• Connect this to real-world data compression and noise filtering scenarios.

Deeper Insight:
Interviewers often ask, “Why does PCA use variance as the measure of information?”
Discuss how variance captures data spread and why maximizing variance minimizes reconstruction error.
Be ready to explain why PCA fails for non-linear manifolds (hint: it’s a linear transformation).

1.2: Derive PCA Mathematically (Step-by-Step)

• Start with data matrix $X \in \mathbb{R}^{n \times d}$, centered by subtracting the mean of each feature.
• Compute the covariance matrix:
  \[ \Sigma = \frac{1}{n-1} X^T X \]
• Perform eigen decomposition:
  \[ \Sigma v_i = \lambda_i v_i \]
  where $v_i$ are eigenvectors (directions) and $\lambda_i$ are eigenvalues (explained variances).
• Sort eigenvectors by descending eigenvalues to select top-$k$ components.
• Project data:
  \[ X_{proj} = X V_k \]

Deeper Insight:
Expect a follow-up question: “Why is PCA equivalent to minimizing reconstruction error?”
Show that PCA finds the subspace minimizing the squared distance between original and projected points — a least-squares projection problem.

1.3: Understand SVD-Based Implementation

• Grasp why most modern implementations use Singular Value Decomposition (SVD) instead of eigen decomposition.
• Compute: \[ X = U S V^T \] Here, columns of $V$ are principal directions, and squared singular values correspond to eigenvalues of the covariance matrix.
• Recognize that SVD is numerically stable and works directly on non-square matrices.

Probing Question:
“If your dataset has 100 features but only 20 samples, which decomposition method would you choose?”
Discuss how SVD handles underdetermined systems more robustly than covariance-based approaches.

1.4: Implement PCA from Scratch

• Using NumPy, implement PCA:
  • Center the data.
  • Compute the covariance matrix.
  • Perform eigen decomposition.
  • Select top-$k$ eigenvectors and project the data.
• Verify that your PCA reproduces results from sklearn.PCA.
• Visualize principal components using matplotlib for intuition.

Probing Question:
“How would you decide the number of components ($k$) to keep?”
Be prepared to discuss the explained variance ratio, elbow rule, and cumulative variance threshold (e.g., 95%).

🧠 Advanced Mathematical and Statistical Concepts

2.1: Connect PCA to Optimization

• Understand PCA as the solution to: \[ \max_{w} \quad w^T \Sigma w \quad \text{s.t.} \quad \|w\| = 1 \]
• Explain that this finds the direction of maximum variance (the leading eigenvector).
• Show that subsequent principal components are found under orthogonality constraints.

Deeper Insight:
Common interview twist: “Why does orthogonality matter?”
Be ready to say: “It ensures each new component captures variance not already explained by previous components.”

2.2: Probabilistic PCA and Connection to Latent Variables

• Explain that Probabilistic PCA (PPCA) models observed data as Gaussian distributed around a low-dimensional linear manifold.
• The generative model: \[ x = Wz + \mu + \epsilon, \quad \epsilon \sim \mathcal{N}(0, \sigma^2 I) \]
• Show how PPCA allows uncertainty modeling and integrates naturally into Bayesian frameworks.

Probing Question:
“What’s the advantage of PPCA over standard PCA?”
Discuss its probabilistic interpretation, noise robustness, and suitability for missing data imputation.

2.3: Kernel PCA (Nonlinear Extension)

• Describe how Kernel PCA replaces the dot product with a kernel function (e.g., RBF kernel) to capture nonlinear manifolds.
• Core equation:
  \[ K = \phi(X)\phi(X)^T \] where $\phi$ maps inputs to high-dimensional space.
• Understand why kernel PCA can model nonlinear structures while still using the same eigen decomposition logic.

Deeper Insight:
Be prepared for the question: “How do you center the kernel matrix in feature space?”
The correct transformation:

\[ K' = K - 1_n K - K 1_n + 1_n K 1_n \]

⚙️ Applied ML and Practical Trade-Offs

3.1: Choosing the Number of Components

• Learn methods:
  • Cumulative explained variance plot (retain 95–99%).
  • Cross-validation for downstream performance.
  • Scree plot “elbow” heuristic.
• Discuss trade-offs between accuracy, interpretability, and runtime.

Deeper Insight:
“How would you apply PCA in a production pipeline?”
Mention:

• Fitting PCA on the training set only (to avoid data leakage).
• Applying the same transformation on test data.
• Using incremental PCA for streaming or large-scale data.

3.2: Numerical Stability and Scaling

• Always standardize data before PCA (mean = 0, variance = 1).
• Discuss how unscaled features can dominate variance due to differing magnitudes.
• Recognize the importance of float precision and SVD truncation for massive datasets.

Probing Question:
“Why is PCA sensitive to scaling and outliers?”
Outliers inflate variance, skewing principal directions — discuss Robust PCA and Z-score normalization as mitigation strategies.

3.3: PCA in Model Pipelines and MLOps

• Integrate PCA as part of a scikit-learn Pipeline for reproducible preprocessing.
• Track and version PCA parameters (e.g., mean vector, components_) for consistent transformations.
• Be prepared to explain how PCA impacts:
  • Feature interpretability
  • Model explainability
  • Latency in online systems

Deeper Insight:
Expect a trade-off question: “Would you apply PCA before or after feature selection?”
The ideal answer: “If the goal is to remove multicollinearity, use PCA before. If we want to retain semantic interpretability, perform feature selection first.”

🧩 Final Interview Tip:
Mastery of PCA isn’t about remembering formulas — it’s about articulating the balance between mathematics, intuition, and system-level trade-offs.
Show you understand why PCA works, when it helps, and how to apply it responsibly in production-scale systems.