1.2. Backpropagation

At its heart, backpropagation is just an application of the chain rule from calculus — applied cleverly across the entire neural network.

Here’s the flow:

1. Forward Pass: Compute the output $\hat{y}$ using the current weights and activations.

2. Compute the Loss: Measure how far the output is from the true label $y$ using a loss function $L(y, \hat{y})$.

3. Backward Pass:

   • The error starts at the output layer.
   • The algorithm computes gradients of the loss with respect to each weight — i.e., how much each weight contributed to the error.
   • These gradients are then propagated backward layer by layer using the chain rule.

4. Weight Update: Each weight is adjusted slightly in the opposite direction of its gradient (via Gradient Descent).

This process repeats for every batch of data until the network converges — meaning, it’s learned the right weights to minimize the loss.

If we only looked at the loss, we’d know how bad the prediction was — but not who to blame.

Backpropagation solves this by assigning credit (or blame) to each neuron according to its contribution. Using the chain rule, we can compute how changes in one neuron affect the final loss — layer by layer.

Think of it like tracing cause and effect backward:

• Output was too high → must have come from overactive neurons in the previous layer.
• Adjust their weights slightly to tone them down next time.

This precise blame assignment makes the learning process both efficient and scalable — even for deep networks.

Backpropagation is what turns a static mathematical structure (the FNN) into a learning machine. Without it, we’d have to manually tune weights — an impossible task with millions of parameters.

It’s also the precursor to modern automatic differentiation (autograd) systems used in frameworks like PyTorch and TensorFlow, which automate gradient computation using computational graphs.

So, in the grand timeline of ML, backpropagation was the key that unlocked the modern deep learning era.

The goal of backpropagation is to compute how much each weight affects the final loss.

1. Loss Function: $L(y, \hat{y})$ — measures prediction error.

2. Error Signal at Output Layer:

   $$\delta^{(L)} = \frac{\partial L}{\partial z^{(L)}}$$

   where $z^{(L)} = W^{(L)}a^{(L-1)} + b^{(L)}$.

3. Propagate Error Backward:

   $$\delta^{(l)} = (W^{(l+1)})^T \delta^{(l+1)} \odot f'(z^{(l)})$$
   • $(W^{(l+1)})^T$: transfers error backward to layer $l$.
   • $\odot$: element-wise product (Hadamard).
   • $f’(z^{(l)})$: derivative of activation function — determines how much that neuron “responds” to the error.

4. Compute Gradients for Parameters:

   $$\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T$$
   $$\frac{\partial L}{\partial b^{(l)}} = \delta^{(l)}$$

• “Backpropagation is a separate algorithm from gradient descent.” Wrong — backprop computes gradients; gradient descent uses them to update weights.
• “We need to derive gradients by hand every time.” Not anymore — automatic differentiation tools handle it.
• “If I initialize all weights to zero, learning is stable.” False — zero initialization breaks symmetry, making all neurons learn the same thing.