Jacobians & Hessians — Concept Only
Useful when reading optimization/autodiff literature or debugging advanced models.
1. Plain English Intuition
When a function takes multiple inputs and outputs a single scalar (like a loss function), its first derivative is the Gradient. But if a function outputs multiple values (vector-to-vector, like a neural network layer), its matrix of all first-order partial derivatives is called the Jacobian. The Hessian is the matrix of second-order partial derivatives of a scalar function; it measures curvature, telling you whether the valley is a steep bowl or a flat trough.
2. Where It Appears in ML & Deep Learning
- 1Autograd engines: vector-Jacobian products (VJPs) in PyTorch backward passes
- 2Second-order optimization methods (L-BFGS, Newton-Raphson)
- 3Analyzing loss landscape curvature and flatness around minima
- 4Generative modeling: Jacobian regularization in GANs
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Computing the full Hessian for a 100M parameter model would require a 100M × 100M matrix (impossible to store).
Negative eigenvalues in the Hessian signify saddle points.
5. Python in Practice
Engineered using: torch.autograd.functional.jacobian, torch.autograd.functional.hessian
import torch
# Vector-to-vector function: f([x1, x2]) = [x1^2, x1 * x2]
def func(x):
return torch.stack([x[0]**2, x[0] * x[1]])
x = torch.tensor([2.0, 3.0])
J = torch.autograd.functional.jacobian(func, x)
print("Jacobian matrix:\n", J)
# Expected: [[2*x1, 0], [x2, x1]] = [[4, 0], [3, 2]]Fast Study Loop
Complete these 3 steps to mark this topic understood: