Conditional Probability, Independence & Bayes' Theorem
Understand updating beliefs from evidence, priors/posteriors, and independence assumptions.
1. Plain English Intuition
Conditional probability P(A|B) asks: "What is the probability of A given that event B has occurred?" Independence means knowing B gives zero information about A (P(A|B) = P(A)). Bayes’ Theorem is the mathematical rule for updating your initial belief (Prior) in light of new observed evidence to form an updated belief (Posterior): P(A|B) = P(B|A) · P(A) / P(B). This concept underpins everything from spam filters to Bayesian neural networks and prompt conditioning.
2. Where It Appears in ML & Deep Learning
- 1Naive Bayes classification (assuming features are conditionally independent given class)
- 2Language modeling: P(next_word | previous_tokens)
- 3Bayesian Optimization for hyperparameter search
- 4Precision, Recall, and medical diagnostic test interpretation
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Base Rate Fallacy: Ignoring the prior P(H) when interpreting rare disease or rare fraud alerts.
Violating independence: Naive Bayes produces overconfident probability estimates when features are correlated.
Zero-frequency problem: If a word never appeared in training for class C, P(word|C) becomes 0 (solved by Laplace smoothing).
5. Python in Practice
Engineered using: sklearn.naive_bayes.GaussianNB, scipy.stats
from sklearn.naive_bayes import GaussianNB
import numpy as np
# Training data: [feature_1, feature_2], target: 0 or 1
X_train = np.array([[-1, -1], [-2, -1], [-3, -2], [1, 1], [2, 1], [3, 2]])
y_train = np.array([0, 0, 0, 1, 1, 1])
# Fit Naive Bayes classifier (computes class priors & conditional Gaussian likelihoods)
nb = GaussianNB()
nb.fit(X_train, y_train)
# Predict posterior probabilities P(Class | X_test)
X_test = np.array([[0.5, 0.5]])
probs = nb.predict_proba(X_test)[0]
print(f"P(Class=0 | X): {probs[0]:.4f}")
print(f"P(Class=1 | X): {probs[1]:.4f}")Fast Study Loop
Complete these 3 steps to mark this topic understood: