5.2. Resampling & Validation

When we only have one sample from a population, we can’t easily measure variability or error. Resampling helps by generating pseudo-samples through random selection or systematic omission.

These pseudo-samples are used to compute statistics (like the mean, regression coefficients, or accuracy), and their variability tells us:

• How stable is our estimate?
• How much does it depend on specific data points?
• How well would our model perform on unseen data?

Resampling simulates the idea of many possible datasets drawn from the same population — even though we only have one. It mimics real-world uncertainty through repeated, controlled randomness.

That’s why bootstrap or cross-validation is sometimes called “poor man’s repeated sampling.”

• Bootstrap → Measures the variability of model parameters (confidence in estimates).
• Jackknife → Measures sensitivity to individual data points.
• Cross-Validation → Measures generalization performance on unseen data.

Together, they form the validation backbone of modern ML — ensuring that your model isn’t just memorizing, but actually learning.

Bootstrap approximates the sampling distribution of a statistic by resampling with replacement.

Algorithm:

1. Given dataset $D$ of size $n$.

2. Draw $B$ bootstrap samples $D^{}_1, D^{}_2, \dots, D^{*}_B$, each of size $n$, with replacement.

3. Compute the statistic $\hat{\theta}^{*}_b$ (e.g., mean, regression coefficient) for each sample.

4. Estimate variability:

   $$ \text{Var}*{boot}(\hat{\theta}) = \frac{1}{B-1} \sum*{b=1}^{B} (\hat{\theta}^{*}_b - \bar{\theta}^{*})^2 $$

5. Confidence Interval (Percentile Method): Take the 2.5th and 97.5th percentiles of $\hat{\theta}^{*}_b$ as a 95% CI.

Example: If you have 100 observations and resample 1000 times, you might get 1000 different means — their spread approximates the uncertainty in your estimate.

Jackknife systematically leaves out one observation at a time to estimate bias and variance of a statistic.

Algorithm:

1. For each $i = 1, 2, …, n$:

   • Leave out the $i^{th}$ observation → $D_{(-i)}$.
   • Compute statistic $\hat{\theta}_{(-i)}$.

2. Compute the Jackknife estimate of variance:

   $$ \text{Var}*{jack}(\hat{\theta}) = \frac{n - 1}{n} \sum*{i=1}^{n} (\hat{\theta}*{(-i)} - \bar{\theta}*{(-i)})^2 $$

Use Cases:

• Bias correction for small-sample estimates.
• Influence diagnostics (how sensitive is the model to each data point?).

Comparison: Bootstrap = stochastic (many random resamples). Jackknife = deterministic (systematic leave-one-out).

Cross-validation evaluates a model’s generalization ability — how well it performs on new, unseen data.

K-Fold Cross-Validation:

1. Split dataset into $K$ roughly equal parts (folds).

2. For each fold $k$:

   • Train on $K-1$ folds.
   • Test on the remaining fold.

3. Average the performance across all folds:

   $$ \text{CV Error} = \frac{1}{K} \sum_{k=1}^{K} E_k $$

Common Variants:

• Leave-One-Out (LOOCV): $K = n$ (like Jackknife).
• Stratified K-Fold: Maintains class ratios (for classification).
• Nested CV: For hyperparameter tuning inside validation.

Trade-offs:

• Larger $K$ → lower bias, higher variance (more computation).
• Smaller $K$ → higher bias, lower variance (faster).

Prediction error can be decomposed into three parts:

• Bias: Error due to simplifying assumptions (underfitting).
• Variance: Error due to sensitivity to training data (overfitting).
• Irreducible Error: Random noise — can’t be reduced.

Resampling methods help visualize this tradeoff:

• High variance → large spread in bootstrap estimates.
• High bias → consistently wrong estimates (regardless of resample).

Goal: Find the “sweet spot” where total error is minimal.

• “Bootstrap and Jackknife give identical results.” → No — bootstrap is stochastic, jackknife is deterministic.
• “Cross-validation improves the model.” → It doesn’t train better — it evaluates better.
• “More folds always means better estimates.” → Beyond a point, more folds increase computation but not insight.