Gradient Descent, SGD & Learning Rate
Understand parameter updates, step size, minibatches, convergence problems, and why training can diverge.
1. Plain English Intuition
Gradient descent is the iterative engine that teaches a model. At each step, it takes the current weights, inspects the gradient, and nudges the weights in the opposite direction by an amount scaled by the Learning Rate: w_new = w_old - lr * gradient. Full-batch gradient descent uses all training examples at once (slow and memory-heavy); Stochastic Gradient Descent (SGD) uses one or a small mini-batch (fast, noisy, and helps escape shallow local minima).
2. Where It Appears in ML & Deep Learning
- 1Every neural network optimizer: SGD, Momentum, Adam, AdamW, RMSProp
- 2Learning rate schedules: warmup, cosine decay, step reduction
- 3Mini-batch training loops in PyTorch, TensorFlow, and JAX
- 4Fine-tuning large language models using low learning rates
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Learning rate too large: Loss oscillates wildly, explodes to infinity, or outputs NaN.
Learning rate too small: Training takes forever and gets stuck in plateau regions.
Saddle points and noisy mini-batch variance stalling convergence without momentum.
5. Python in Practice
Engineered using: torch.optim.SGD, torch.optim.AdamW, lr_scheduler
import torch
# Synthetic problem: find w that minimizes loss = (w - 5)^2
w = torch.tensor([0.0], requires_grad=True)
learning_rate = 0.1
print(f"Initial weight: {w.item():.2f}")
for step in range(10):
loss = (w - 5.0) ** 2
loss.backward()
# In-place parameter update without tracking gradients
with torch.no_grad():
w -= learning_rate * w.grad
w.grad.zero_() # Critical: PyTorch accumulates gradients by default!
print(f"Trained weight after 10 steps: {w.item():.4f} (Target: 5.0)")Fast Study Loop
Complete these 3 steps to mark this topic understood: