Vectors, Matrices, Tensors & Shapes
Know what a vector/matrix/tensor represents and how dimensions/shapes map to samples, features, batches, and parameters.
1. Plain English Intuition
In applied ML, almost every piece of data is stored in an n-dimensional rectangular grid called a tensor. A 1D tensor is a vector (a single sample’s features), a 2D tensor is a matrix (a batch of samples × features), a 3D tensor represents sequence data (batch × sequence_length × embedding_dim), and a 4D tensor represents images (batch × channels × height × width). Mastering tensor shapes is 80% of debugging deep learning pipelines.
2. Where It Appears in ML & Deep Learning
- 1PyTorch and TensorFlow model inputs, weights, and layer activations
- 2Batch processing through multi-layer perceptrons (B, D) and CNNs (B, C, H, W)
- 3Token representations in Transformer architectures (B, T, D)
- 4Weight matrices storing learnable linear projections between layers
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Shape Mismatches: Trying to add a vector of shape (5,) to a matrix of shape (3, 4) without broadcast compatibility.
Silent Broadcasting Bugs: Adding (B, 1) to (1, B) produces a (B, B) matrix instead of a (B,) vector without raising an error.
Transposition oversights: Forgetting to swap axes (e.g. channels-first vs channels-last).
5. Python in Practice
Engineered using: np.array, torch.Tensor, .shape, .view(), .reshape()
import torch
# 1. 2D Tensor: Batch of tabular data (Batch=3 samples, Features=4)
batch_data = torch.randn(3, 4)
# 2. 3D Tensor: Transformer sequence (Batch=2, Sequence=5 tokens, Embedding=8 dims)
seq_tokens = torch.randn(2, 5, 8)
# 3. Shape inspection and reshaping
print(f"Batch shape: {batch_data.shape}")
print(f"Tokens shape: {seq_tokens.shape}")
# Flatten tokens into a 2D representation:
flattened = seq_tokens.view(-1, 8)
print(f"Flattened shape: {flattened.shape}")Fast Study Loop
Complete these 3 steps to mark this topic understood: