Derivatives, Partial Derivatives & Gradients
Derivative = sensitivity/rate of change; gradient = direction of steepest increase across many parameters.
1. Plain English Intuition
A derivative tells you how sensitive an output is to a tiny nudge in one input: "If I increase weight w by 0.001, will loss go up or down, and by how much?" When a model has millions of weights, the partial derivative measures this sensitivity for each weight individually while holding all other weights constant. Packing all these partial derivatives into a single vector gives the Gradient. The gradient points directly in the direction of steepest increase of the loss; to minimize loss, you step in the exact opposite direction.
2. Where It Appears in ML & Deep Learning
- 1Gradient Descent: updating weights in the opposite direction of the gradient
- 2Automatic Differentiation in PyTorch and JAX (torch.autograd)
- 3Feature attribution & saliency maps (gradient of prediction with respect to input pixels)
- 4Detecting exploding or vanishing gradients during training diagnostics
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Vanishing Gradients: Partial derivatives become so close to 0 that weights stop updating (common with deep networks and Sigmoid/Tanh).
Exploding Gradients: Gradients blow up to huge values or NaN, destabilizing optimization.
Attempting to differentiate through non-differentiable operations (e.g. argmax, thresholding).
5. Python in Practice
Engineered using: loss.backward(), torch.autograd.grad, jax.grad
import torch
# Define a learnable parameter with gradient tracking
w = torch.tensor([3.0], requires_grad=True)
# Define a simple quadratic loss function: L(w) = (w - 1)^2
loss = (w - 1.0) ** 2
# Compute gradient: dL/dw = 2*(w - 1)
loss.backward()
print(f"Weight value: {w.item()}")
print(f"Loss value: {loss.item()}")
print(f"Gradient dL/dw: {w.grad.item()} # Expect 2*(3-1) = 4.0")Fast Study Loop
Complete these 3 steps to mark this topic understood: