RNNs - Roadmap

🧠 Core Deep Learning Foundations

1.1: Understand Sequential Data and the Need for Memory

1. Grasp what makes sequential data different — order matters.
   Explain why traditional feedforward networks fail to capture dependencies across time steps.
2. Learn the temporal correlation assumption: $x_t$ depends on $x_{t-1}$, $x_{t-2}$, etc.
3. Identify real-world examples: text (next-word prediction), time-series (stock trends), audio (speech sequences).

Deeper Insight:
Interviewers may ask, “Why can’t we just feed previous inputs as extra features into a dense net?” — be ready to discuss parameter explosion, loss of temporal abstraction, and inefficiency.

1.2: Architecture of a Vanilla RNN

1. Derive the recurrence relation:
   $$ h_t = f(W_{xh}x_t + W_{hh}h_{t-1} + b_h) $$ $$ y_t = W_{hy}h_t + b_y $$
2. Interpret each term intuitively:
   • $W_{xh}$ → how current input affects hidden state.
   • $W_{hh}$ → how past memory contributes.
   • $h_t$ → the “hidden summary” of all prior context.
3. Visualize the unrolled RNN across time and understand parameter sharing across steps.

Probing Question:
“If we unroll an RNN for 100 steps, how many weight matrices exist?”
(Hint: Only 3, since weights are shared — this is key to generalization and efficiency.)

⚙️ Training Dynamics & Backpropagation Through Time (BPTT)

2.1: Derive BPTT Mathematically

1. Express the loss across all time steps:
   $$ L = \sum_t L_t(y_t, \hat{y_t}) $$
2. Compute gradients recursively:
   $\frac{\partial L}{\partial W_{hh}}$ accumulates through all previous time steps.
3. Observe how repeated multiplication of Jacobians leads to gradient decay/explosion.

Deeper Insight:
The gradient term involves $\prod_{k} W_{hh}^T \cdot f’(a_k)$ — if eigenvalues < 1 ⇒ vanishing gradients; > 1 ⇒ exploding gradients.
Expect questions like, “How would you mitigate vanishing gradients in practice?”

2.2: Handling Vanishing & Exploding Gradients

1. Learn gradient clipping to handle exploding gradients.
2. Study better initialization schemes (Xavier/He initialization).
3. Introduce gated architectures (LSTM, GRU) as structural fixes for vanishing gradients.

Probing Question:
“What happens if we clip too aggressively?”
→ You lose learning signal and training stalls. Mention monitoring gradient norms dynamically.

🧩 Advanced Architectures — LSTM & GRU

3.1: LSTM (Long Short-Term Memory)

1. Learn the core gates:
   • Forget Gate: $f_t = \sigma(W_f[h_{t-1}, x_t] + b_f)$
   • Input Gate: $i_t = \sigma(W_i[h_{t-1}, x_t] + b_i)$
   • Cell Update: $\tilde{C}t = \tanh(W_c[h{t-1}, x_t] + b_c)$
   • Output Gate: $o_t = \sigma(W_o[h_{t-1}, x_t] + b_o)$
2. Understand the cell state $C_t$ as a highway that carries long-term gradients effectively.
3. Visualize how LSTM gates selectively update or forget information.

Deeper Insight:
When asked “Why use $\tanh$ for the cell update instead of ReLU?”, note that bounded activations help prevent uncontrolled growth in cell state values.

3.2: GRU (Gated Recurrent Unit)

1. Compare to LSTM:
   • Fewer parameters, no separate cell state.
   • Combines forget and input gates into one update gate.
2. Formulation:
   $$ z_t = \sigma(W_z[x_t, h_{t-1}]) \\ r_t = \sigma(W_r[x_t, h_{t-1}]) \\ \tilde{h}_t = \tanh(W[x_t, (r_t * h_{t-1})]) \\ h_t = (1 - z_t) * h_{t-1} + z_t * \tilde{h}_t $$

Probing Question:
“Why might GRUs train faster than LSTMs?”
→ Simpler structure, fewer parameters, easier optimization — but may underperform on long-context data.

🧮 Implementation & Practice

4.1: Build a Simple RNN from Scratch

1. Implement a minimal RNN using NumPy:
   • Define weight matrices Wxh, Whh, and Why.
   • Write the forward loop for time steps.
   • Backpropagate manually through time for 2–3 steps to visualize gradient flow.
2. Validate with a toy sequence (e.g., predicting next character in “hello”).

Probing Question:
“What’s the computational complexity per sequence?”
→ O(T × H²), where T = timesteps, H = hidden units.

4.2: Implement an LSTM/GRU using PyTorch

1. Build both manually (nn.LSTMCell, nn.GRUCell) and via high-level APIs (nn.LSTM, nn.GRU).
2. Monitor training curves for stability and speed differences.
3. Explore sequence padding, masking, and batch processing for variable-length sequences.

Deeper Insight:
Expect to discuss teacher forcing in sequence-to-sequence tasks — why it speeds convergence but can cause exposure bias during inference.

🚀 Scaling & Modern Extensions

5.1: Limitations of RNNs

1. Sequential nature prevents full parallelization — limits scalability.
2. Long-range dependencies degrade due to cumulative errors and vanishing gradients.
3. Hidden states may not capture all necessary context (information bottleneck).

Probing Question:
“If RNNs are so limited, why do we still use them?”
→ For low-latency, on-device, or streaming scenarios where full-sequence attention is infeasible.

5.2: Transition to Attention and Transformers

1. Understand the motivation:
   • Attention = selective memory.
   • Transformer = fully parallelized, context-aware sequence learner.
2. Draw parallels: RNNs store history in $h_t$; Transformers store it in attention weights across all tokens.
3. Learn how positional encodings replace temporal recurrence.

Deeper Insight:
Interviewers may test your conceptual bridge: “Can we view Transformers as RNNs with infinite memory?”
→ Yes, in a sense — attention replaces recurrence by directly linking all time steps.

5.3: Practical Interview Prep

1. Be ready to derive RNN and LSTM equations on a whiteboard.
2. Discuss when to choose RNNs vs. GRUs vs. Transformers.
3. Articulate real-world trade-offs:
   • RNNs → compact, low-latency.
   • LSTMs → robust for medium sequences.
   • Transformers → best for global context.

Probing Question:
“Your LSTM model is overfitting — what do you do?”
→ Mention dropout on recurrent connections, layer normalization, gradient clipping, and data augmentation (e.g., time warping).

✅ Final Outcome:
By following this roadmap, you’ll be able to:

• Derive RNNs mathematically from first principles.
• Implement and optimize LSTM/GRU architectures.
• Diagnose gradient and scaling issues.
• Connect RNNs to the broader evolution toward Transformers.