Likelihood, Log-Likelihood & Maximum Likelihood Estimation (MLE)
Know probability-vs-likelihood and why fitting often means maximizing log-likelihood.
1. Plain English Intuition
Probability calculates the chance of future data given fixed parameters. Likelihood reverses this: given fixed observed data, how plausible are different candidate model parameters? Maximum Likelihood Estimation (MLE) is the foundational principle of machine learning: adjust the model’s weights until the likelihood of having generated the training data is maximized. We always take the logarithm (Log-Likelihood) because it turns a product of tiny probabilities into a sum of numbers, which avoids underflow and is mathematically easier to differentiate.
2. Where It Appears in ML & Deep Learning
- 1Cross-Entropy loss is mathematically identical to Negative Log-Likelihood (NLL)
- 2Fitting linear and logistic regression models in scikit-learn
- 3Pretraining Large Language Models: maximizing log P(token_t | token_<t)
- 4Gaussian Mixture Models (GMM) and hidden Markov models via Expectation-Maximization
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Overfitting: MLE with high-capacity models will assign infinite likelihood to single points (memorization). Regularization (MAP) is needed.
Non-convex likelihood surfaces in deep neural networks have many local optima.
Evaluating likelihood on out-of-domain test data without recognizing probability collapse.
5. Python in Practice
Engineered using: Usually inside .fit() / torch.nn.NLLLoss, scipy.optimize.minimize
import numpy as np
from scipy.optimize import minimize
# Suppose we observed sample heights:
data = np.array([170.2, 175.4, 168.1, 182.0, 174.5])
# Negative Log-Likelihood for Gaussian parameters [mu, log_sigma]
def neg_log_likelihood(params):
mu, log_sigma = params
sigma = np.exp(log_sigma)
# Gaussian log PDF: -0.5 * log(2*pi*sigma^2) - (x - mu)^2 / (2*sigma^2)
n = len(data)
nll = 0.5 * n * np.log(2 * np.pi * sigma**2) + np.sum((data - mu)**2) / (2 * sigma**2)
return nll
# MLE optimization: minimize negative log-likelihood
res = minimize(neg_log_likelihood, [170.0, 1.0])
print(f"MLE Estimated Mean: {res.x[0]:.2f} (Sample mean: {np.mean(data):.2f})")
print(f"MLE Estimated Std: {np.exp(res.x[1]):.2f} (Sample std: {np.std(data):.2f})")Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Maximum Likelihood, clearly explained!!!
Watch on YouTube