Dot Product, Projection, Norms, Distance & Cosine Similarity
Understand similarity, vector length and distance. You do not need repetitive hand calculations.
1. Plain English Intuition
The dot product measures directional alignment between two vectors. If two vectors point in the same direction, their dot product is large and positive; if perpendicular, it is zero; if opposite, negative. The L2 norm measures a vector’s raw physical length. Dividing the dot product by the lengths of the two vectors gives Cosine Similarity (a score from -1 to 1 regardless of magnitude). In modern AI, semantic vector search (RAG) and attention mechanisms rely almost entirely on these concepts.
2. Where It Appears in ML & Deep Learning
- 1Vector Search & RAG: finding relevant documents by cosine similarity of text embeddings
- 2Scaled Dot-Product Attention: softmax(Q @ K.T / sqrt(d_k))
- 3Neuron activation: w · x + b
- 4L2 Weight Decay: penalizing the Euclidean norm of model weights to prevent overfitting
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Comparing raw dot products when embeddings are not normalized (a longer vector gets an artificially high score).
Division by zero when computing cosine similarity of a zero-vector (norm = 0).
The Curse of Dimensionality: In very high dimensions (D > 1000), distances between points tend to concentrate, making raw Euclidean distance less discriminative.
5. Python in Practice
Engineered using: np.dot, np.linalg.norm, torch.cosine_similarity, sklearn.metrics.pairwise.cosine_similarity
import numpy as np
# Two word/document embeddings (e.g. 4-dimensional)
query_vec = np.array([0.9, 0.1, 0.4, 0.0])
doc_vec = np.array([0.85, 0.15, 0.35, 0.05])
# 1. Dot product (magnitude + alignment)
dot_prod = np.dot(query_vec, doc_vec)
# 2. Euclidean L2 norms (lengths)
norm_q = np.linalg.norm(query_vec)
norm_d = np.linalg.norm(doc_vec)
# 3. Cosine similarity (pure directional alignment in [-1, 1])
cosine_sim = dot_prod / (norm_q * norm_d)
print(f"Dot Product: {dot_prod:.4f}")
print(f"Cosine Similarity: {cosine_sim:.4f}")Fast Study Loop
Complete these 3 steps to mark this topic understood: