Functions, Exponentials & Logarithms
Understand why exp/log appear in sigmoid, softmax, likelihood, and cross-entropy. No log-manipulation drills.
1. Plain English Intuition
The exponential function (exp) turns any real number—positive, zero, or negative—into a strictly positive number. The logarithm (log) does the exact inverse: it turns tiny probabilities into manageable negative numbers, and compresses massive multiplications into simple additions. In machine learning, exp lets us turn unbounded model outputs ("logits") into valid probabilities, while log prevents computer hardware from rounding tiny probabilities down to zero.
2. Where It Appears in ML & Deep Learning
- 1Sigmoid and Softmax activations to guarantee non-negative probabilities summing to 1
- 2Negative Log-Likelihood (NLL) and Cross-Entropy loss functions
- 3Log-Sum-Exp trick to prevent numerical overflow and underflow in attention heads
- 4Log-transforming skewed features (e.g. income, latency, word counts)
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Numerical Overflow: exp(89.0) in float32 exceeds ~3.4e38 and results in +inf.
Numerical Underflow: exp(-90.0) in float32 evaluates to 0.0, which causes log(0) = -inf.
Taking log of non-positive numbers (e.g. log(0) or log(-1)) outputs -inf or NaN.
5. Python in Practice
Engineered using: np.exp, np.log, torch.exp, torch.log
import numpy as np
# 1. Exponential guarantees non-negative weights
raw_logits = np.array([-2.5, 0.0, 3.1])
unnormalized = np.exp(raw_logits)
probabilities = unnormalized / np.sum(unnormalized)
# 2. Logarithm turns products of probabilities into sums
p1, p2 = 0.0001, 0.0002
# Direct product risks floating point underflow:
direct_prod = p1 * p2
# Log-space addition preserves precision:
log_sum = np.log(p1) + np.log(p2)
print(f"Probabilities: {probabilities}")
print(f"Log joint likelihood: {log_sum:.4f}")Fast Study Loop
Complete these 3 steps to mark this topic understood: