Chain Rule & Backpropagation Intuition
Understand how local derivatives compose so error signals can flow backward through a model.
1. Plain English Intuition
Neural networks are composite functions: input passes through layer 1, whose output goes to layer 2, and so on until the loss. The chain rule states that the rate of change of the final loss with respect to an early weight is simply the product of all intermediate local derivatives along that path. Backpropagation is nothing more than the chain rule applied from the end of the network to the beginning, caching intermediate derivatives so they are computed only once.
2. Where It Appears in ML & Deep Learning
- 1Training multi-layer neural networks, CNNs, and Transformers
- 2Autograd engines: building a Directed Acyclic Graph (DAG) during the forward pass and traversing backward
- 3Residual connections (ResNet): adding skip connections so gradients can flow backward without diminishing
- 4Gradient clipping and optimizer step logic
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Running out of GPU memory (OOM) because storing activations for the chain rule requires memory proportional to network depth and batch size.
Broken computation graph: Detaching a tensor (.detach() or numpy conversion) stops the flow of gradients.
Saturating activation functions causing downstream local derivatives to evaluate to zero.
5. Python in Practice
Engineered using: loss.backward(), jax.vjp, autograd graph
import torch
# Multi-stage composite function: x -> h -> loss
x = torch.tensor([2.0])
w1 = torch.tensor([1.5], requires_grad=True)
w2 = torch.tensor([2.0], requires_grad=True)
# Forward pass
h = x * w1 # h = 2.0 * 1.5 = 3.0
y_hat = h * w2 # y_hat = 3.0 * 2.0 = 6.0
target = torch.tensor([10.0])
loss = (y_hat - target) ** 2 # (6 - 10)^2 = 16.0
# Backward pass (Chain rule in action)
loss.backward()
# dLoss/dw1 = dLoss/dy_hat * dy_hat/dh * dh/dw1
# = 2*(6-10) * 2.0 * 2.0 = -8 * 4 = -32.0
print(f"Analytical backprop gradient for w1: {w1.grad.item()}")Fast Study Loop
Complete these 3 steps to mark this topic understood: