Loss Functions: MSE, Binary Log-Loss & Cross-Entropy
Know what the model is optimizing and why regression/classification use different losses.
1. Plain English Intuition
A loss function assigns a penalty score to errors. For continuous regression, Mean Squared Error (MSE) punishes large deviations quadratically: Loss = (y - ŷ)². For classification, MSE is terrible because it creates flat gradients when the model is confident and wrong. Instead, classification uses Cross-Entropy (Log-Loss): Loss = -log(p_correct). If the model assigns 99% probability to the true class, loss is near 0; if it assigns 0.1%, loss shoots up toward infinity.
2. Where It Appears in ML & Deep Learning
- 1MSE: Regression tasks, autoencoder reconstruction loss, diffusion model noise matching
- 2Binary Cross-Entropy (BCE): Binary classification, multi-label tagging, GAN discriminator loss
- 3Categorical Cross-Entropy: Multi-class classification (ImageNet, next-token prediction in LLMs)
- 4Huber Loss: Robust regression combining MSE for small errors and MAE for large errors
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Passing raw logits into nn.BCELoss() instead of nn.BCEWithLogitsLoss() causes numerical instability.
Passing already-softmaxed probabilities into nn.CrossEntropyLoss() (which already includes Softmax internally).
Severe class imbalance causing the loss to be dominated by the easy majority class.
5. Python in Practice
Engineered using: mean_squared_error, log_loss, torch.nn.MSELoss, torch.nn.CrossEntropyLoss, torch.nn.BCEWithLogitsLoss
import torch
import torch.nn as nn
# 1. Regression: Mean Squared Error
mse_criterion = nn.MSELoss()
pred_reg = torch.tensor([2.5, 4.0])
target_reg = torch.tensor([3.0, 4.0])
print(f"MSE Loss: {mse_criterion(pred_reg, target_reg).item():.4f}")
# 2. Multi-class Classification: Cross-Entropy (takes unnormalized logits)
ce_criterion = nn.CrossEntropyLoss()
raw_logits = torch.tensor([[2.0, 0.5, -1.0]]) # Batch=1, Classes=3
true_class = torch.tensor([0]) # True label is class 0
print(f"Cross-Entropy Loss: {ce_criterion(raw_logits, true_class).item():.4f}")Fast Study Loop
Complete these 3 steps to mark this topic understood: