Neural Network Fundamentals — Core Concepts & Activation Functions

🧩 Core Concepts

1.1: Feedforward Neural Networks (FNNs)

1. Understand the Architecture.

   • Learn the basic structure — input, hidden, and output layers.
   • Derive the forward propagation rule mathematically:
     $$a^{(l)} = f(W^{(l)}a^{(l-1)} + b^{(l)})$$
     where $f$ is the activation function.
   • Be clear about what happens at each stage: linear transformation → non-linearity → next layer input.

2. Visualize the Data Flow.

   • Create simple examples (e.g., XOR function) to see how multiple layers allow non-linear decision boundaries.
   • Implement a forward pass in NumPy for intuition.

3. Link to Capacity & Overfitting.

   • Understand how increasing depth and width affects model capacity.
   • Discuss trade-offs — expressivity vs. overfitting risk.

Deeper Insight:
Expect questions like:
“Why can’t a single-layer perceptron learn XOR?” or “How do hidden layers transform input space geometrically?”
Highlight your grasp of how non-linear transformations enable hierarchical feature learning.

1.2: Backpropagation

1. Derive the Learning Rule.

   • Begin with the loss function $L(y, \hat{y})$.
   • Derive gradients for weights and biases:
     $$\frac{\partial L}{\partial W^{(l)}} = \delta^{(l)} (a^{(l-1)})^T$$
     $$\delta^{(l)} = (W^{(l+1)})^T \delta^{(l+1)} \odot f'(z^{(l)})$$
   • Understand chain rule intuition — how errors propagate backward layer by layer.

2. Implement Backprop from Scratch.

   • Code a simple 2-layer NN in NumPy.
   • Manually compute and update gradients.

3. Connect to Computational Graphs.

   • Visualize how auto-differentiation (like in PyTorch) is a computational implementation of backprop.
   • Understand gradient flow, accumulation, and memory efficiency.

Deeper Insight:
Common probing questions include:

• “Why do we use the chain rule instead of numerical differentiation?”
• “What happens when gradients vanish or explode?”
• “Can you explain why initializing all weights to zero fails?”
  Your ability to articulate these pitfalls reveals depth beyond textbook understanding.

1.3: Gradient Descent in Neural Networks

1. Grasp the Optimization Objective.

   • Express training as minimizing $L(W) = \frac{1}{N}\sum_i \ell(f_W(x_i), y_i)$.
   • Know the role of gradients as directional guides in weight space.

2. Understand Variants.

   • Differentiate between Batch, Stochastic, and Mini-batch GD.
   • Understand the stochasticity-benefit trade-off — noise helps escape local minima.

3. Explore Learning Rate Dynamics.

   • Learn how too large a rate overshoots minima; too small slows convergence.
   • Introduce adaptive optimizers like Adam and RMSProp conceptually.

Deeper Insight:
Expect practical questions such as:
“How do you detect if your model’s learning rate is too high?” or
“Why does Adam converge faster but sometimes generalize worse than SGD?”
These probe whether you understand not just how optimization works, but why certain strategies behave differently.

⚡ Activation Functions

2.1: ReLU (Rectified Linear Unit)

1. Define & Derive Behavior.

   • $f(x) = \max(0, x)$
   • Simple, fast, avoids saturation in positive regime.
   • Gradient: $f’(x) = 1$ if $x > 0$, else $0$.

2. Understand Its Impact.

   • Enables sparse activations and faster convergence.
   • Learn about “dead ReLUs” (when neurons stop activating).

3. Implementation Detail.

   • Try replacing sigmoid with ReLU in a small MLP and compare convergence.

Deeper Insight:
Be ready for:
“Why does ReLU outperform sigmoid in deep networks?”
“How do you mitigate dead ReLUs?” (Hint: Leaky ReLU or proper initialization).

2.2: Sigmoid

1. Formula & Range.

   • $f(x) = \frac{1}{1 + e^{-x}}$
   • Output in $(0,1)$ — ideal for probabilities.

2. Learn Its Limitations.

   • Saturation at extremes leads to vanishing gradients.
   • Outputs not zero-centered → slower convergence.

3. Use Cases.

   • Best suited for output layers in binary classification.

Deeper Insight:
Expect:
“Why do we avoid sigmoid in hidden layers of deep networks?”
“What happens to gradient flow for large $|x|$?”
Explaining the math behind gradient shrinkage shows real understanding.

2.3: Tanh (Hyperbolic Tangent)

1. Definition.

   • $f(x) = \tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$
   • Outputs in $(-1, 1)$, zero-centered.

2. Compare to Sigmoid.

   • Faster convergence due to zero-centered property.
   • Still suffers from vanishing gradients for large magnitudes.

3. Practical Use.

   • Common in RNNs; better normalized activations.

Deeper Insight:
Interviewers often ask:
“Why does tanh outperform sigmoid?” or
“Why do RNNs historically use tanh before LSTM replaced it with gating mechanisms?”
Highlight its symmetry and normalization effects.

2.4: Softmax

1. Understand Its Function.

   • Converts logits to probabilities:
     $$\sigma(z_i) = \frac{e^{z_i}}{\sum_j e^{z_j}}$$

2. Grasp Gradient Properties.

   • Gradient computation links tightly to cross-entropy loss.
   • Understand numerical stability (subtracting max logit before exponentiation).

3. Practical Integration.

   • Used in output layers for multi-class classification.
   • Be comfortable deriving loss gradient w.r.t. logits.

Deeper Insight:
Common probes:

• “Why do we always pair Softmax with Cross-Entropy Loss?”
• “What numerical issue arises when exponentiating large logits?”
• “How does temperature scaling affect Softmax outputs?”

These questions test both your theoretical grounding and your awareness of real-world numerical subtleties.