Markov Chains / MDP Intuition
Useful if you move into reinforcement learning.
1. Plain English Intuition
The Markov Property states that the future depends ONLY on the present state, not on the entire history of how you got there ("memorylessness"): P(S_{t+1} | S_t, ..., S_0) = P(S_{t+1} | S_t). A Markov Decision Process (MDP) adds an agent that takes actions to transition between states while receiving rewards. The goal is finding an optimal Policy that maximizes total expected long-term rewards.
2. Where It Appears in ML & Deep Learning
- 1The mathematical foundation of Reinforcement Learning (Q-learning, PPO, AlphaGo)
- 2Markov Chain Monte Carlo (MCMC) sampling in Bayesian statistics
- 3Hidden Markov Models (HMMs) for biological sequence alignment and speech recognition
- 4Simulating user navigation trajectories in web apps
3. Inputs, Outputs, Shapes & Assumptions
4. What Can Go Wrong (Gotchas & Failures)
Partially Observable MDPs (POMDP): If the current state hides crucial past context, the Markov assumption breaks (fixed by using RNNs/memory).
The Curse of Dimensionality: Discrete state spaces grow exponentially with feature count.
5. Python in Practice
Engineered using: scipy.stats, gymnasium, stable-baselines3
import numpy as np
# 2-State Markov Chain: [Sunny=0, Rainy=1]
# Transition matrix P: Row=current state, Col=next state
P = np.array([
[0.8, 0.2], # Sunny -> 80% Sunny, 20% Rainy
[0.4, 0.6] # Rainy -> 40% Sunny, 60% Rainy
])
# Compute steady-state stationary distribution by powering transition matrix
P_future = np.linalg.matrix_power(P, 50)
print("Stationary long-run probabilities [P(Sunny), P(Rainy)]:\n", P_future[0])Fast Study Loop
Complete these 3 steps to mark this topic understood:
Curated Video Lesson
A friendly introduction to Markov Chains and MDPs
Watch on YouTube