Least Squares & Linear Regression
Understand residuals, squared error, and what the fitted coefficients mean.
1. Plain English Intuition
Linear regression finds the hyper-plane that best fits continuous targets by minimizing the sum of squared vertical differences (residuals) between predicted points and actual targets: y = w^T x + b. The fitted coefficient w_i tells you the expected change in target y for a one-unit change in feature x_i, holding all other features fixed. Ordinary Least Squares (OLS) has an exact closed-form solution: w = (X^T X)^(-1) X^T y.
2. Where It Appears in ML & Deep Learning
- 1The foundational baseline for continuous tabular prediction and forecasting
- 2The core building block of deep learning: a linear layer (nn.Linear) is a set of linear regressions before non-linear activations
- 3Interpretable modeling in healthcare, economics, and finance
- 4Residual connection baselines and trend modeling in time series
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Outlier sensitivity: Because residuals are squared, a single massive outlier will pull the line heavily.
Singular matrix error: If features are linearly dependent, (X^T X) is non-invertible.
Misinterpreting coefficients when features are unnormalized or strongly correlated.
5. Python in Practice
Engineered using: sklearn.linear_model.LinearRegression, np.linalg.lstsq
from sklearn.linear_model import LinearRegression
import numpy as np
# Sample data: Hours studied vs Exam score
X = np.array([[1.0], [2.0], [3.0], [4.0], [5.0]])
y = np.array([52.0, 58.0, 68.0, 74.0, 85.0])
# Fit Ordinary Least Squares
model = LinearRegression()
model.fit(X, y)
print(f"Slope (weight): {model.coef_[0]:.2f} pts per hour")
print(f"Intercept (bias): {model.intercept_:.2f} baseline score")
print(f"Predicted score for 6 hours: {model.predict([[6.0]])[0]:.2f}")Fast Study Loop
Complete these 3 steps to mark this topic understood: