Entropy, Cross-Entropy & KL Divergence
Especially useful for deep learning, generative/probabilistic models and information theory.
1. Plain English Intuition
Entropy measures the inherent uncertainty or surprise in a probability distribution: an unbiased coin flip has high entropy (1 bit), while a two-headed coin has zero entropy. Cross-Entropy measures the total cost of representing true distribution P using model distribution Q. Kullback-Leibler (KL) Divergence measures the "distance" or extra surprise between two distributions: KL(P || Q) = CrossEntropy(P, Q) - Entropy(P). It is always ≥ 0, and equals 0 only if P and Q are identical.
2. Where It Appears in ML & Deep Learning
- 1Variational Autoencoders (VAEs): KL penalty forcing latent codes to match standard normal Gaussian
- 2Reinforcement learning: entropy bonuses to encourage exploration (PPO, Soft Actor-Critic)
- 3Knowledge Distillation: using KL divergence to transfer dark knowledge from teacher to student
- 4Direct Preference Optimization (DPO) and RLHF in Large Language Models
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
If Q(x) = 0 where P(x) > 0, KL divergence blows up to infinity (division by zero).
Confusing KL(P || Q) with a true geometric distance metric (it violates symmetry and the triangle inequality).
5. Python in Practice
Engineered using: scipy.stats.entropy, torch.nn.KLDivLoss
from scipy import stats
import numpy as np
# True distribution P vs Model prediction Q
P = np.array([0.7, 0.2, 0.1])
Q = np.array([0.5, 0.3, 0.2])
# 1. Shannon Entropy of P
h_p = stats.entropy(P, base=2)
# 2. KL Divergence KL(P || Q)
kl_div = stats.entropy(P, Q, base=2)
print(f"Entropy of P: {h_p:.4f} bits")
print(f"KL Divergence KL(P || Q): {kl_div:.4f} bits")Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Entropy, Cross Entropy and KL Divergence
Watch on YouTube