Expected Value, Mean, Variance & Standard Deviation
Understand center, spread and long-run average; recognize why expected loss is central to ML.
1. Plain English Intuition
The Expected Value is the weighted average outcome you expect to see over the long run. In machine learning, training is framed as Empirical Risk Minimization: minimizing the expected value of the loss across your entire dataset. Variance measures how spread out the data points are around that center, and Standard Deviation is simply the square root of variance, bringing the spread back into the original units of measurement.
2. Where It Appears in ML & Deep Learning
- 1Empirical Risk Minimization: E[Loss(y, f(x))]
- 2Batch Normalization & Layer Normalization: subtracting mean and dividing by std dev
- 3Weight initialization (Xavier / He initialization) ensuring activation variance remains constant across layers
- 4Reinforcement learning: maximizing the expected return (sum of future discounted rewards)
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Variance is sensitive to extreme outliers because deviations are squared.
Small batch sizes causing noisy, unstable mean and variance estimates in Batch Normalization.
Numerical division by zero when standard deviation is zero (fixed with eps=1e-5).
5. Python in Practice
Engineered using: np.mean, np.var, np.std, torch.mean, torch.var_mean
import numpy as np
# Feature matrix: 5 samples, 2 features (e.g. house area, age)
X = np.array([
[1200.0, 5.0],
[1500.0, 2.0],
[850.0, 15.0],
[2200.0, 1.0],
[1100.0, 10.0]
])
# Compute center (mean) and spread (standard deviation)
mu = np.mean(X, axis=0)
sigma = np.std(X, axis=0)
# Standardization: zero mean, unit variance
X_normalized = (X - mu) / sigma
print(f"Feature Means: {mu}")
print(f"Feature Std Devs: {sigma}")
print(f"Normalized sample 0: {X_normalized[0]}")Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Mean, Variance and Standard Deviation
Watch on YouTube