Feature Scaling, Standardization & Numerical Stability
Know why scale changes optimization/distance methods; understand overflow/underflow and why stable library implementations matter.
1. Plain English Intuition
If one feature is measured in kilograms (0–100) and another in annual income ($20,000–$200,000), distance-based models (KNN, SVM, k-means) and gradient descent will be completely overwhelmed by the income feature. Standardization scales features to have mean 0 and standard deviation 1: z = (x - μ) / σ. Numerical stability refers to preventing floating-point arithmetic from breaking (underflowing to 0 or overflowing to inf) via tricks like Log-Sum-Exp.
2. Where It Appears in ML & Deep Learning
- 1StandardScaler & MinMaxScaler preprocessing in scikit-learn pipelines
- 2Batch Normalization, Layer Normalization, and RMSNorm in neural networks
- 3Log-Sum-Exp trick in softmax and cross-entropy loss implementations
- 4Gradient clipping (torch.nn.utils.clip_grad_norm_) to prevent gradient explosions
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Slow, zig-zagging gradient descent convergence caused by elongated, unscaled loss contours.
Distance metrics completely dominated by the feature with the largest numerical scale.
NaN values in loss curves due to division by zero or log(0).
5. Python in Practice
Engineered using: StandardScaler, MinMaxScaler, scipy.special.logsumexp, torch.logsumexp
from sklearn.preprocessing import StandardScaler
from scipy.special import logsumexp
import numpy as np
# 1. Feature Standardization
raw_features = np.array([
[25.0, 150000.0], # Age: 25, Income: 150k
[45.0, 45000.0], # Age: 45, Income: 45k
[60.0, 80000.0] # Age: 60, Income: 80k
])
scaler = StandardScaler()
scaled = scaler.fit_transform(raw_features)
print("Standardized features (zero mean, unit variance):\n", scaled)
# 2. Log-Sum-Exp numerical stability demo
large_logits = np.array([1000.0, 1001.0, 999.0])
# np.log(np.sum(np.exp(large_logits))) would overflow to inf!
safe_lse = logsumexp(large_logits)
print(f"Safe Log-Sum-Exp computation: {safe_lse:.4f}")Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Feature Scaling: Standardization vs Normalization
Watch on YouTube