Common Distributions: Bernoulli, Categorical & Gaussian
Know what kind of target/noise each models; recognize their shapes and parameters.
1. Plain English Intuition
Three distributions power 90% of practical machine learning: 1) The Bernoulli distribution models binary yes/no outcomes (p and 1-p), matching binary classification. 2) The Categorical distribution models multi-class picks across K discrete categories, matching Softmax outputs. 3) The Gaussian (Normal) distribution models bell-shaped continuous values parameterized by Mean (center) and Variance (spread), matching continuous regression targets and neural weight initializations.
2. Where It Appears in ML & Deep Learning
- 1Binary Logistic Regression assumes Bernoulli-distributed targets
- 2Multi-class classification (ImageNet, LLM next-token generation) assumes Categorical targets
- 3Mean Squared Error regression implicitly assumes Gaussian-distributed noise around predictions
- 4Latent variables in Variational Autoencoders (VAEs) and initializations (He/Xavier)
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Using Mean Squared Error on heavy-tailed data with extreme outliers.
Assuming a single Gaussian when data has multiple distinct clusters (multimodal distributions).
Treating multi-label classification as a single categorical distribution instead of independent Bernoulli targets.
5. Python in Practice
Engineered using: scipy.stats.bernoulli, scipy.stats.multinomial, scipy.stats.norm, torch.distributions
import numpy as np
from scipy import stats
# 1. Bernoulli: P(click = 1) = 0.35
bern = stats.bernoulli(p=0.35)
print("Bernoulli mean:", bern.mean())
# 2. Categorical / Multinomial: P(cat, dog, bird)
probs = [0.6, 0.3, 0.1]
mult = stats.multinomial(n=1, p=probs)
print("Sample drawn from categorical:", mult.rvs(1)[0])
# 3. Gaussian: Mean = 100, Std = 15
gaussian = stats.norm(loc=100, scale=15)
print("Gaussian 95% confidence interval:", gaussian.interval(0.95))Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
The Normal Distribution, Clearly Explained
Watch on YouTube