2.2 Tune Hyperparameters and Evaluate
🪄 Step 1: Intuition & Motivation
Core Idea: By now, your Logistic Regression model knows how to learn and how to stay disciplined (thanks to regularization). But… how do you find the perfect balance between freedom and control — between underfitting and overfitting?
That’s where hyperparameter tuning comes in. It’s like adjusting the seasoning in a recipe — too little salt (λ too small), it’s bland and overfits; too much salt (λ too big), it’s inedible and underfits.
The tuning process helps your model find that sweet spot of generalization.
Simple Analogy: Think of your Logistic Regression model as a music equalizer 🎚️. Each hyperparameter (like λ or learning rate) is a slider that changes the “tone” of learning. Grid search is like trying different slider combinations to find the most harmonious sound — the one that plays well not just on your headphones (training set) but also on your car speakers (test set).
🌱 Step 2: Core Concept
Let’s now explore how we tune and evaluate our model’s regularization strength using data-driven, fair, and visual methods.
What’s Happening Under the Hood?
In Logistic Regression with regularization, the key hyperparameter is λ (lambda) — the strength of regularization.
In most ML libraries (like scikit-learn), you don’t directly set λ; instead, you set C, where:
$$ C = \frac{1}{\lambda} $$So:
- Small C → Strong regularization (simpler model, higher bias)
- Large C → Weak regularization (complex model, higher variance)
To find the best C, we perform Grid Search with Cross-Validation (GridSearchCV) — it systematically tests many C values and picks the one that performs best across multiple validation splits.
Why It Works This Way
One model trained on one dataset split can easily be lucky (or unlucky). Cross-validation fixes this by dividing the data into k folds, training on k–1 folds, and testing on the remaining one — then averaging the results for stability.
For imbalanced datasets, we use Stratified Cross-Validation, ensuring each fold keeps the same proportion of classes (like 90:10 stays 90:10).
This avoids the “oops, all spam!” problem — where one validation split accidentally contains only one class.
How It Fits in ML Thinking
Hyperparameter tuning teaches a fundamental ML mindset: no model works perfectly out of the box. You must iterate — adjusting, validating, visualizing, and learning.
Evaluation metrics like AUC, ROC, and confusion matrices provide deeper insight than accuracy alone — they help diagnose what kind of mistakes your model makes.
This is the bridge from building a model → trusting it in production.
📐 Step 3: Mathematical Foundation
Let’s peek under the hood of how we evaluate our tuned model.
ROC Curve and AUC
The Receiver Operating Characteristic (ROC) curve plots:
- x-axis: False Positive Rate (FPR) = $\frac{\text{FP}}{\text{FP + TN}}$
- y-axis: True Positive Rate (TPR) = $\frac{\text{TP}}{\text{TP + FN}}$
Each point corresponds to a different classification threshold.
AUC (Area Under the Curve) measures the model’s ability to separate classes — the higher, the better (1 = perfect, 0.5 = random guessing).
Confusion Matrix
| Predicted: Positive | Predicted: Negative | |
|---|---|---|
| Actual: Positive | True Positive (TP) | False Negative (FN) |
| Actual: Negative | False Positive (FP) | True Negative (TN) |
From this, we can compute:
- Precision = $\frac{TP}{TP + FP}$
- Recall = $\frac{TP}{TP + FN}$
- F1-Score = harmonic mean of precision and recall.
Each metric tells a different story:
- Precision → “How many predicted positives were correct?”
- Recall → “How many actual positives did we find?”
- F1 → “Balanced measure of both.”
🧠 Step 4: Assumptions or Key Ideas
- C (1/λ) controls model flexibility — must be tuned, not guessed.
- Use Stratified Cross-Validation for fair evaluation on imbalanced datasets.
- Always evaluate with multiple metrics — accuracy alone can mislead (especially when one class dominates).
⚖️ Step 5: Strengths, Limitations & Trade-offs
- Finds the optimal balance between bias and variance.
- Prevents overfitting through systematic validation.
- Works well for both balanced and imbalanced datasets (via stratification).
- Grid Search can be computationally expensive for large hyperparameter spaces.
- Cross-validation requires extra computation (training multiple times).
- Evaluation still depends on chosen metrics — no single “best” metric fits all tasks.
🚧 Step 6: Common Misunderstandings
🚨 Common Misunderstandings (Click to Expand)
- ❌ “AUC > 0.9 means perfect model.” → High AUC might hide poor performance on minority classes — always check precision and recall.
- ❌ “λ = 0 is ideal since it removes penalty.” → That often causes overfitting — the model memorizes the training data.
- ❌ “Larger λ always helps.” → Too large → coefficients shrink too much → model underfits.
🧩 Step 7: Mini Summary
🧠 What You Learned: Hyperparameter tuning (via GridSearchCV) and stratified cross-validation help you balance model flexibility and generalization.
⚙️ How It Works: You test multiple C (or λ) values, validate across folds, and pick the one that maximizes metrics like AUC or F1.
🎯 Why It Matters: Proper tuning and evaluation transform a good Logistic Regression model into a reliable, production-ready predictor.