Regularization: L1 & L2
Understand why parameter penalties can improve generalization and the qualitative difference between L1 and L2.
1. Plain English Intuition
Regularization discourages a model from becoming overly complex by penalizing large weights in the loss function: Total Loss = Loss + Penalty(W). L2 Regularization (Ridge / Weight Decay) penalizes the sum of squared weights (∑ w_i²); it shrinks all weights smoothly toward zero, preventing any single feature from dominating. L1 Regularization (Lasso) penalizes the sum of absolute values (∑ |w_i|); because of its sharp diamond-shaped constraint geometry, it drives irrelevant weights to exactly zero, producing sparse, feature-selecting models.
2. Where It Appears in ML & Deep Learning
- 1Weight Decay in AdamW and SGD optimizers for training Transformers and CNNs
- 2Ridge regression and Lasso regression in scikit-learn for tabular models
- 3Feature selection: using L1 to discover which variables matter
- 4Dropout and data augmentation as implicit forms of regularization
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Applying regularization without feature scaling (features with large raw numbers will be penalized unfairly).
Setting regularization strength λ too high, causing severe underfitting.
Regularizing bias terms (b): standard practice is to regularize weights only, leaving biases free.
5. Python in Practice
Engineered using: Ridge, Lasso, ElasticNet, optimizer weight_decay
from sklearn.linear_model import Ridge, Lasso
import numpy as np
# 5 collinear features predicting target
X = np.random.randn(30, 5)
y = 3.0 * X[:, 0] + 0.0 * X[:, 1] + 2.0 * X[:, 2] + np.random.randn(30) * 0.1
# 1. L2 Regularization (Ridge): Smooth shrinkage, no exact zeros
ridge = Ridge(alpha=1.0).fit(X, y)
print("Ridge Weights (L2):", np.round(ridge.coef_, 3))
# 2. L1 Regularization (Lasso): Sparse selection, drives unneeded features to exact 0
lasso = Lasso(alpha=0.2).fit(X, y)
print("Lasso Weights (L1):", np.round(lasso.coef_, 3))Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Regularization Part 1: Ridge (L2) and Lasso (L1) Regression
Watch on YouTube