Singular Value Decomposition (SVD)
Useful for low-rank approximation, PCA, recommender systems and embeddings.
1. Plain English Intuition
While eigendecomposition only works on square matrices, SVD works on ANY matrix of any rectangular shape (M × N). SVD factors a matrix into three matrices: A = U Σ V^T. U represents row concepts, V represents column concepts, and Σ contains the "singular values" (ordered from largest to smallest) indicating how important each concept is. Truncated SVD keeps only the top-k singular values to create the best possible low-rank approximation of the data.
2. Where It Appears in ML & Deep Learning
- 1Collaborative filtering recommender systems (Netflix Prize matrix factorization)
- 2Latent Semantic Analysis (LSA) in NLP and word co-occurrence factorizations
- 3Model compression: factorizing large weight matrices into two smaller rank-k matrices
- 4The engine powering robust PCA and pseudoinverse calculations
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Full SVD is computationally expensive (O(min(M, N) * M * N)); use randomized or truncated SVD for large matrices.
Missing values: SVD requires complete matrices (imputation needed beforehand).
5. Python in Practice
Engineered using: np.linalg.svd, sklearn.decomposition.TruncatedSVD, torch.svd
import numpy as np
# User-Movie Rating Matrix: 4 users, 5 movies
ratings = np.array([
[5, 4, 0, 1, 0],
[4, 5, 0, 0, 1],
[0, 0, 5, 4, 4],
[1, 0, 4, 5, 5]
], dtype=float)
# Compute SVD: A = U @ diag(S) @ Vt
U, S, Vt = np.linalg.svd(ratings, full_matrices=False)
# Reconstruct low-rank approximation keeping top 2 latent concepts
k = 2
reconstructed = U[:, :k] @ np.diag(S[:k]) @ Vt[:k, :]
print(f"Top 2 Singular Values: {S[:k]}")
print("Reconstructed (imputed) rating predictions:\n", np.round(reconstructed, 2))Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
Singular Value Decomposition (SVD), Clearly Explained!!!
Watch on YouTube