Bias–Variance, Underfitting & Overfitting
Understand the generalization trade-off and why lower training error is not always better.
1. Plain English Intuition
Total prediction error comes from three sources: Bias, Variance, and Irreducible Noise. High Bias (Underfitting) happens when a model is too simple (like a straight line for quadratic data); it performs poorly on both training and test data. High Variance (Overfitting) happens when a model is overly complex; it memorizes the training noise, getting near-zero training error but failing miserably on unseen test data. The sweet spot minimizes total generalization error on unseen data.
2. Where It Appears in ML & Deep Learning
- 1Model capacity tuning: choosing depth, width, tree depth, or number of parameters
- 2Diagnosing training curves: gap between training loss and validation loss
- 3Early stopping criteria during neural network training
- 4Deciding whether you need more data (fixes variance) or a bigger model (fixes bias)
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Relying solely on training accuracy to judge model quality.
Adding model complexity when the real issue is high bias caused by bad features.
Premature stopping before the model has even converged to low bias.
5. Python in Practice
Engineered using: sklearn.model_selection.learning_curve, validation metrics
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
import numpy as np
# Synthetic non-linear data with noise
np.random.seed(42)
X_train = np.linspace(0, 1, 20).reshape(-1, 1)
y_train = np.sin(2 * np.pi * X_train).ravel() + np.random.normal(0, 0.2, 20)
X_test = np.linspace(0, 1, 10).reshape(-1, 1)
y_test = np.sin(2 * np.pi * X_test).ravel()
# Compare Underfitting (Degree 1) vs Overfitting (Degree 15)
for deg in [1, 3, 15]:
model = make_pipeline(PolynomialFeatures(deg), LinearRegression())
model.fit(X_train, y_train)
train_err = mean_squared_error(y_train, model.predict(X_train))
test_err = mean_squared_error(y_test, model.predict(X_test))
print(f"Degree {deg:2d} -> Train MSE: {train_err:.4f}, Test MSE: {test_err:.4f}")Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Machine Learning Fundamentals: Bias and Variance
Watch on YouTube