Train/Validation/Test Splits & Cross-Validation
Understand unbiased evaluation, model selection, and data leakage.
1. Plain English Intuition
You cannot evaluate an exam with the exact questions the student practiced on. We partition data into three strictly separated buckets: 1) Train Set: used to optimize weights. 2) Validation Set: used to tune hyperparameters and choose between architectures. 3) Test Set: locked in a vault until the very end, used once to measure unbiased generalization performance. K-Fold Cross-Validation splits the training data into K folds, rotating which fold is used for validation to get robust error estimates on smaller datasets.
2. Where It Appears in ML & Deep Learning
- 1Every real-world ML benchmark and model validation pipeline
- 2Stratified splitting for imbalanced classification tasks
- 3Time-based (temporal) splitting to avoid predicting the past using future information
- 4K-Fold Cross-Validation for hyperparameter grid search and model selection
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Data Leakage: Fitting StandardScaler or imputers on the entire dataset before splitting (leaks mean and variance).
Random splitting on time series data: Leads to unrealistically high test metrics because the model memorizes future patterns.
Repeatedly tuning on the test set until it ceases to be an unbiased evaluation.
5. Python in Practice
Engineered using: train_test_split, KFold, StratifiedKFold, cross_val_score
from sklearn.model_selection import train_test_split, KFold, cross_val_score
from sklearn.linear_model import Ridge
import numpy as np
X = np.random.randn(100, 4)
y = 2.0 * X[:, 0] + np.random.randn(100)
# Step 1: Hold out a clean test set (20%)
X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
# Step 2: 5-Fold Cross-Validation on the remaining training data
model = Ridge(alpha=1.0)
cv_scores = cross_val_score(model, X_temp, y_temp, cv=5, scoring='neg_mean_squared_error')
print(f"5-Fold CV Mean MSE: {-np.mean(cv_scores):.4f} (+/- {np.std(cv_scores):.4f})")Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Cross Validation Explained Step-by-Step
Watch on YouTube