Matrix Multiplication
Understand it as combining linear transformations / weighted features; be able to reason about compatible shapes.
1. Plain English Intuition
Matrix multiplication is the workhorse of deep learning. When you multiply a batch of data X by a weight matrix W, you are not simply doing arithmetic: you are rotating, scaling, and projecting the input points into a new feature space. The fundamental rule is inner dimension matching: an (M × K) matrix multiplied by a (K × N) matrix yields an (M × N) matrix. The K dimensions collapse by computing dot products along rows and columns.
2. Where It Appears in ML & Deep Learning
- 1Fully Connected (Linear) layers: y = X @ W + b
- 2Query, Key, Value attention projections in Transformers: Q = X @ W_Q
- 3Convolutional operations mapped to matrix multiplication via im2col / GEMM kernels
- 4Dimensionality reduction projections (e.g., PCA, autoencoders)
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Inner dimension mismatch: Trying to multiply (32, 128) by (64, 256) causes RuntimeError: size mismatch.
Confusing element-wise multiplication (A * B) with matrix multiplication (A @ B).
Unintended memory blowup when multiplying large batch matrices without chunking.
5. Python in Practice
Engineered using: A @ B, np.matmul, torch.matmul, torch.bmm
import numpy as np
# Batch of 4 samples, each with 3 input features
X = np.array([
[1.0, 2.0, 3.0],
[4.0, 5.0, 6.0],
[7.0, 8.0, 9.0],
[1.0, 0.0, 1.0]
]) # Shape: (4, 3)
# Layer weights: 3 inputs mapped to 2 hidden neurons
W = np.array([
[0.2, 0.5],
[-0.1, 0.8],
[0.4, -0.3]
]) # Shape: (3, 2)
# Linear transformation: (4, 3) @ (3, 2) -> (4, 2)
hidden = X @ W
print("Output activations shape:", hidden.shape)
print("Output activations:\n", hidden)Fast Study Loop
Complete these 3 steps to mark this topic understood: