1.3. Gradient Descent in Neural Networks

Every neural network has a loss function $L(W)$ that measures how far off its predictions are. The goal of training is to minimize this loss by adjusting the weights $W$.

1. Compute the gradient of the loss with respect to weights:

   $$\nabla_W L(W) = \left[\frac{\partial L}{\partial W_1}, \frac{\partial L}{\partial W_2}, ..., \frac{\partial L}{\partial W_n}\right]$$

   This tells us which direction in weight space increases the loss fastest.

2. To minimize the loss, we move in the opposite direction of the gradient:

   $$W := W - \eta \nabla_W L(W)$$

   where $\eta$ (eta) is the learning rate, controlling how big a step we take.

3. Repeat this update many times until the loss stops decreasing significantly — meaning, we’ve reached a minimum or a flat region.

The gradient is like a compass that always points uphill — in the direction of maximum increase. So by moving in the negative gradient direction, we’re always taking the steepest path downhill toward a minimum.

The learning rate $\eta$ acts like your stride length:

• Too small → you move safely but slowly.
• Too large → you might leap over the valley and oscillate endlessly.

This simple mechanism, iterated millions of times, allows even massive neural networks to find low-loss configurations in high-dimensional spaces.

Gradient Descent is the beating heart of all learning algorithms — not just neural networks.

In deep learning:

• Backpropagation computes how each weight affects the loss (the gradient).
• Gradient Descent uses those gradients to update the weights intelligently.

Together, they form the learn-update cycle that powers everything from image recognition to language models.

We define the training objective as:

• $N$: number of training examples.
• $f_W(x_i)$: prediction made by the network with parameters $W$.
• $\ell$: loss for one example (e.g., Mean Squared Error or Cross-Entropy).

We want to find the $W$ that minimizes $L(W)$.

• Computes the gradient using the entire dataset at once.
• Provides a very accurate gradient direction.
• But it’s slow and computationally expensive, especially for large datasets.

Think of it like waiting for everyone in a company to vote before making one decision — precise, but time-consuming.

• Updates the weights after every single example.
• Faster and introduces randomness, which can help escape local minima.
• But gradients are noisy, causing the loss curve to fluctuate.

It’s like taking quick steps after hearing one opinion at a time — faster, but sometimes inconsistent.

• A balanced approach: compute gradients over small batches (e.g., 32 or 128 samples).
• Smooths out some noise while retaining efficiency.
• This is the standard method used in almost all modern deep learning frameworks.

The learning rate ($\eta$) controls how big your steps are during optimization.

• If $\eta$ is too small, convergence is painfully slow.
• If $\eta$ is too large, you might overshoot the minimum repeatedly or diverge entirely.

To stabilize learning:

• Use learning rate schedules (e.g., gradually decreasing $\eta$).

• Use adaptive optimizers like Adam, RMSProp, or AdaGrad, which automatically adjust the learning rate for each parameter.

  Adam is like an experienced hiker who learns how slippery each patch of terrain is and adjusts their stride accordingly — taking bigger steps on flat ground and smaller, careful ones on steep slopes.