Sampling, Central Limit Theorem & Standard Error
Understand why estimates fluctuate between samples and why averages often become approximately normal.
1. Plain English Intuition
The Central Limit Theorem (CLT) is one of the most magical theorems in science: take independent samples from ANY distribution (even wildly skewed or bimodal), calculate their average, and repeat. The distribution of those sample averages will always form a bell-shaped Gaussian curve as the sample size grows. Standard Error tells you how much your sample mean is expected to bounce around the true population mean. It shrinks as the square root of sample size (1/√N).
2. Where It Appears in ML & Deep Learning
- 1Stochastic Gradient Descent: Mini-batch gradient estimates are sample averages that fluctuate around the true full-batch gradient
- 2A/B testing, offline model evaluation, and reporting confidence intervals on test metrics
- 3Bootstrap re-sampling to estimate uncertainty of predictions
- 4Monte Carlo approximations in reinforcement learning and diffusion models
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Diminishing returns: To cut the standard error of your metric in half, you need 4 times as much data (due to √N).
Assuming CLT applies when sample size is tiny (N < 30) or distribution has infinite variance (Cauchy distribution).
Correlated samples: If data samples are clustered (e.g. repeated user actions), standard error will be severely underestimated.
5. Python in Practice
Engineered using: scipy.stats.sem, np.random.choice, bootstrap utilities
import numpy as np
# Draw 100,000 samples from an extremely skewed exponential distribution
population = np.random.exponential(scale=2.0, size=100000)
# Repeatedly take 1,000 mini-batches of size N=50 and compute their means
sample_means = [np.mean(np.random.choice(population, size=50)) for _ in range(1000)]
print(f"True population mean: {np.mean(population):.4f}")
print(f"Mean of sample means: {np.mean(sample_means):.4f}")
print(f"Theoretical Standard Error: {np.std(population) / np.sqrt(50):.4f}")
print(f"Empirical Std of sample means: {np.std(sample_means):.4f}")Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
The Central Limit Theorem, Clearly Explained
Watch on YouTube