Extension #8 of 13Good to Do — On-Demand Mathematical Extensions
Convexity & Constrained Optimization
Useful for SVMs, classical optimization and understanding guarantees.
1. Plain English Intuition
A convex function looks like a smooth bowl: any line drawn between two points on the curve lies entirely above the curve. The immense beauty of convex optimization is that ANY local minimum is guaranteed to be the global minimum—there are no traps, bad saddle points, or misleading local dips. Constrained optimization adds boundary rules (e.g. weights must sum to 1, or budget ≤ $1000) solved via Lagrange Multipliers.
2. Where It Appears in ML & Deep Learning
- 1Support Vector Machines (SVMs) solved via quadratic programming duality
- 2Linear and Logistic regression loss landscapes (strictly convex)
- 3Understanding why deep neural networks are NON-convex and lack global convergence guarantees
- 4Portfolio optimization and resource allocation models
3. Inputs, Outputs, Shapes & Assumptions
Inputs
Objective function f(x) subject to inequality constraints g(x) ≤ 0 and equalities h(x) = 0.
Outputs
Optimal point x* satisfying Karush-Kuhn-Tucker (KKT) conditions.
Shapes & Dimensions
Mathematical programming formulation.
Assumptions & Constraints
Convexity requires positive semi-definite Hessian everywhere.
4. What Can Go Wrong (Gotchas & Failures)
Applying linear/quadratic solvers to non-convex neural landscapes.
Ignoring constraint violations during iterative updates.
5. Python in Practice
Engineered using: scipy.optimize.minimize, cvxpy
from scipy.optimize import minimize
# Minimize non-linear objective f(x, y) = (x - 2)^2 + (y - 3)^2
def objective(x):
return (x[0] - 2)**2 + (x[1] - 3)**2
# Subject to constraint: x + y = 4
constraints = {'type': 'eq', 'fun': lambda x: x[0] + x[1] - 4}
res = minimize(objective, [0.0, 0.0], constraints=constraints)
print(f"Constrained minimum at x={res.x[0]:.2f}, y={res.x[1]:.2f}")Fast Study Loop
Complete these 3 steps to mark this topic understood: