Linear Models - Loss Functions

🤖 Core Regression Loss Functions

1️⃣ Mean Squared Error (MSE)

Learning Steps

1. Understand the formulation:
   \( \text{MSE} = \frac{1}{n}\sum_{i=1}^n (y_i - \hat{y}_i)^2 \)
2. Derive the gradient of MSE with respect to model weights.
   Practice differentiating the loss with respect to \( \beta \):
   \( \frac{\partial L}{\partial \beta} = -\frac{2}{n} X^T(y - X\beta) \)
3. Visualize the loss surface — note how convexity ensures a single global minimum.
4. Implement from scratch using NumPy’s matrix operations. Confirm gradient descent convergence.

Deeper Insight:
MSE penalizes large errors more severely. Discuss why this is beneficial for normally distributed residuals but harmful when data includes extreme outliers.
Probing Question: “If your model overfits, how would regularization modify the MSE objective?”

2️⃣ Mean Absolute Error (MAE)

Learning Steps

1. Write the loss expression:
   \( \text{MAE} = \frac{1}{n}\sum_{i=1}^n |y_i - \hat{y}_i| \)
2. Explore why MAE uses median as the optimal estimator under L1 norm.
3. Discuss subgradients and how optimizers handle non-differentiability.
4. Compare convergence behavior vs. MSE under gradient-based updates.

Deeper Insight:
MAE is resistant to outliers but leads to slower convergence due to constant gradients.
Probing Question: “Why do some implementations use Smooth-L1 loss as a compromise?”

3️⃣ Root Mean Squared Error (RMSE)

Learning Steps

1. Understand RMSE:
   \( \text{RMSE} = \sqrt{\frac{1}{n}\sum (y_i - \hat{y}_i)^2} \)
2. Recognize it as the square root of MSE and connect their gradients.
3. Discuss interpretability trade-offs — same metric units vs. differentiability concerns.

Deeper Insight:
RMSE emphasizes large errors even more than MSE due to the non-linear transformation.
Probing Question: “Why is RMSE preferred in competitions like Kaggle, but not in parameter optimization?”

4️⃣ Huber Loss

Learning Steps

1. Study the formulation:
   \( L_\delta(y, \hat{y}) = \begin{cases} \frac{1}{2}(y - \hat{y})^2, & \text{if } |y - \hat{y}| \le \delta \\ \delta(|y - \hat{y}| - \frac{1}{2}\delta), & \text{otherwise} \end{cases} \)
2. Visualize how the curve transitions smoothly from quadratic to linear.
3. Implement and experiment with different δ values to balance robustness and sensitivity.
4. Relate to real-world use cases (e.g., noisy data with occasional large errors).

Deeper Insight:
Huber combines smoothness with robustness — an excellent talking point for optimization trade-offs.
Probing Question: “How does δ act as a hyperparameter controlling bias-variance trade-off?”

🧠 Core Classification Loss Functions

5️⃣ Log Loss (Binary Cross-Entropy)

Learning Steps

1. Recall logistic regression output: \( \hat{y} = \sigma(X\beta) = \frac{1}{1 + e^{-X\beta}} \)
2. Define the loss:
   \( L = -\frac{1}{n}\sum [y_i \log(\hat{y}_i) + (1 - y_i)\log(1 - \hat{y}_i)] \)
3. Derive gradients and show how it corresponds to maximizing log-likelihood.
4. Plot loss vs. probability — observe how confident misclassifications are heavily penalized.

Deeper Insight:
Cross-entropy punishes overconfident wrong predictions, ensuring probabilistic calibration.
Probing Question: “Why can’t we use MSE for logistic regression?”

6️⃣ Categorical Cross-Entropy

Learning Steps

1. Define softmax output:
   \( \hat{y}_i = \frac{e^{z_i}}{\sum_{j} e^{z_j}} \)
2. Write the loss:
   \( L = -\sum y_i \log(\hat{y}_i) \)
3. Explain why this measures divergence between true and predicted distributions.
4. Implement from scratch to demonstrate numerical stability using log-sum-exp trick.

Deeper Insight:
This connects to information theory (minimizing KL divergence).
Probing Question: “Why does the softmax’s normalization ensure numerical stability and gradient flow?”

⚔️ Margin-Based Loss Function

7️⃣ Hinge Loss

Learning Steps

1. Define hinge loss:
   \( L = \max(0, 1 - y_i \cdot (w^Tx_i)) \)
2. Understand the geometric interpretation — encouraging a margin of at least 1.
3. Visualize misclassified vs correctly classified samples and how the gradient behaves.
4. Implement and compare to logistic loss — discuss why hinge loss doesn’t output probabilities.

Deeper Insight:
Hinge loss is not differentiable at the margin but still convex — showing your grasp of subgradients is key.
Probing Question: “How would you modify hinge loss for multiclass SVMs?”

🧩 Integration and Interview Depth

Recommended Capstone Steps

1. Create a unified notebook comparing all regression losses on synthetic noisy data.
2. Create another comparing classification losses on separable vs overlapping datasets.
3. Analyze loss landscapes and gradient magnitudes for each case.
4. Be prepared to discuss practical criteria for loss function selection.