Confidence Intervals & Hypothesis Testing
Important if your job includes experimentation, A/B testing or scientific inference.
1. Plain English Intuition
When your new model scores 84.2% accuracy versus the old baseline of 83.5%, is that a genuine improvement or pure random noise? A Hypothesis Test sets up a Null Hypothesis ("no real difference") and calculates a p-value: the probability of seeing this result by pure luck. A Confidence Interval provides a range (e.g. 95% confident the true improvement is between +0.2% and +1.2%), giving you both magnitude and certainty.
2. Where It Appears in ML & Deep Learning
- 1Production A/B testing of recommendation algorithms and ranking models
- 2Evaluating whether LLM prompt tweaks or fine-tunes create statistically significant gains
- 3Feature importance significance testing in regression tables
- 4Model safety and compliance reporting
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
p-hacking: Checking results every hour and stopping the experiment early as soon as p < 0.05.
Confusing statistical significance with business practical significance (a 0.001% gain can be p < 0.01 with huge N).
5. Python in Practice
Engineered using: scipy.stats.ttest_ind, statsmodels.stats.weightstats
from scipy import stats
import numpy as np
# A/B test results: Conversion rates for Model A vs Model B
model_a = np.random.normal(loc=0.12, scale=0.03, size=200)
model_b = np.random.normal(loc=0.13, scale=0.03, size=200)
# Two-sample independent t-test
t_stat, p_value = stats.ttest_ind(model_b, model_a)
print(f"Model A Mean: {np.mean(model_a):.3f} vs Model B Mean: {np.mean(model_b):.3f}")
print(f"p-value: {p_value:.4f}")
print("Significant difference (alpha=0.05)?:", p_value < 0.05)Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
p-values: What they are and how to interpret them
Watch on YouTube