Covariance & Correlation
Understand how variables move together, what sign/magnitude mean, and why correlation is not causation.
1. Plain English Intuition
Covariance measures whether two variables tend to increase together (positive) or move in opposite directions (negative). However, raw covariance depends on the scale of the features. Pearson Correlation fixes this by normalizing covariance between -1 and +1: +1 means perfect positive linear relationship, -1 means perfect negative linear relationship, and 0 means no linear relationship. Remember: correlation only detects linear patterns and never proves causation.
2. Where It Appears in ML & Deep Learning
- 1Exploratory Data Analysis (EDA) and feature selection (dropping redundant collinear features)
- 2Covariance matrices in PCA (Principal Component Analysis)
- 3Detecting multicollinearity in linear models which inflates weight uncertainty
- 4Data leakage inspection (if a feature has correlation 1.0 with the target label, investigate immediately)
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Spurious correlation: Confounding variables creating apparent relationships.
Multicollinearity: Highly correlated features causing unstable linear regression coefficients.
Assuming zero correlation implies independence (false for non-linear relationships).
5. Python in Practice
Engineered using: np.cov, np.corrcoef, pandas.DataFrame.corr()
import numpy as np
# Two synthetic features: square footage and price
sqft = np.array([1000, 1500, 2000, 2500, 3000], dtype=float)
price = np.array([210, 310, 395, 505, 620], dtype=float)
# Compute 2x2 correlation matrix
corr_matrix = np.corrcoef(sqft, price)
r = corr_matrix[0, 1]
print(f"Correlation coefficient r: {r:.4f}")
# Non-linear relationship example:
x = np.linspace(-3, 3, 100)
y = x ** 2
print(f"Correlation for y=x^2: {np.corrcoef(x, y)[0, 1]:.4f} (Even though y depends entirely on x!)")Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Covariance and Correlation Clearly Explained
Watch on YouTube