Linear Combinations, Span, Basis & Rank
Useful for dimensionality, redundant features and representation intuition.
1. Plain English Intuition
A linear combination is formed by multiplying vectors by scalar coefficients and adding them together. The Span of a set of vectors is all the points you can possibly reach using those combinations. A Basis is a minimal set of independent vectors that spans the space. The Rank of a matrix is the number of genuinely independent dimensions it contains; if an (N × 100) feature matrix has rank 12, 88 of those features are redundant linear combinations of the others.
2. Where It Appears in ML & Deep Learning
- 1Low-Rank Adaptation (LoRA) for parameter-efficient LLM fine-tuning
- 2Detecting linearly dependent, redundant features in tabular datasets
- 3Understanding embedding subspace geometry in Transformers
- 4Matrix factorization techniques in recommendation systems
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Rank deficiency causing matrix inversion or linear regression solvers to fail.
Assuming high feature count means high information (a 1000-column dataset might have rank 10).
5. Python in Practice
Engineered using: np.linalg.matrix_rank, scipy.linalg.orth
import numpy as np
# Matrix where column 3 is an exact duplicate sum of column 1 + column 2
A = np.array([
[1.0, 2.0, 3.0],
[2.0, 0.0, 2.0],
[3.0, 1.0, 4.0]
])
rank = np.linalg.matrix_rank(A)
print(f"Matrix shape: {A.shape}")
print(f"Matrix Rank: {rank} (Rank deficient! 3 columns but only 2 independent dimensions)")Fast Study Loop
Complete these 3 steps to mark this topic understood: