Understanding Reinforcement Learning Basics is essential for engineers building autonomous decision-making systems, robotics controllers, algorithmic trading strategies, and alignment pipelines (RLHF / GRPO) for modern Large Language Models. Unlike supervised learning—which relies on static labeled datasets—or unsupervised learning—which uncovers hidden structure—reinforcement learning trains an autonomous agent to make sequential decisions by interacting directly with an uncertain environment to maximize cumulative long-term reward.

In this comprehensive chapter of our AI Fundamentals series, we formalize sequential decision-making through Markov Decision Processes (MDPs), derive the Bellman Optimality Equations, analyze the Exploration vs. Exploitation trade-off, and implement tabular Q-Learning from scratch in Python.

Core Concepts in Reinforcement Learning Basics: The Agent-Environment Loop

At the foundation of Reinforcement Learning Basics is a closed-loop feedback cycle operating across discrete time steps t = 0, 1, 2, ...:

1. State (S_t): The agent observes the current state representation S_t from the environment’s state space S.
2. Action (A_t): Guided by its decision-making policy pi(a | s), the agent selects and executes an action A_t from the valid action space A.
3. Reward (R_{t+1}) and Next State (S_{t+1}): The environment transitions to a new state S_{t+1} according to its transition dynamics and emits a scalar feedback signal called the reward R_{t+1}.
4. Discounted Return (G_t): The agent’s goal is not to maximize immediate reward R_{t+1} greedily, but to maximize the expected discounted return G_t = R_{t+1} + gamma * R_{t+2} + gamma^2 * R_{t+3} + ..., where the discount factor gamma in [0, 1) balances immediate versus future rewards.

Markov Decision Processes (MDPs) and the Markov Property

Mathematically, sequential decision problems in Reinforcement Learning Basics are modeled as a Markov Decision Process (MDP) defined by the 5-tuple (S, A, P, R, gamma). An environment satisfies the Markov Property if the future state and reward depend exclusively on the current state and action, independent of all prior history:

P(S_{t+1} = s', R_{t+1} = r | S_t = s, A_t = a)

State-Value Function V(s) vs. Action-Value Function Q(s, a)

• State-Value Function V^pi(s): Quantifies the expected discounted return when starting in state s and following policy pi thereafter.
• Action-Value Function Q^pi(s, a) (Q-Function): Quantifies the expected discounted return when starting in state s, taking a specific action a first, and following policy pi thereafter. Learning the optimal Q-function Q*(s, a) allows an agent to act optimally without needing a physics model of the environment simply by picking argmax_a Q*(s, a).

The Bellman Equation and Temporal-Difference (TD) Q-Learning

Richard Bellman proved that the optimal value of any state-action pair can be decomposed recursively into the immediate reward plus the discounted value of the best possible action in the next state. Off-policy Q-Learning turns the Bellman Optimality Equation into an iterative Temporal-Difference (TD) update rule:

Q(S_t, A_t) <- Q(S_t, A_t) + alpha * [ R_{t+1} + gamma * max_a Q(S_{t+1}, a) - Q(S_t, A_t) ]

• TD Target: R_{t+1} + gamma * max_a Q(S_{t+1}, a) represents the bootstrapped estimate of total return after observing one step.
• TD Error (delta_t): The difference between the TD Target and the current estimate Q(S_t, A_t).
• Learning Rate (alpha): Controls how aggressively new experiences overwrite old Q-table estimates.
• Epsilon-Greedy Exploration: With probability epsilon, the agent explores a random action; with probability 1 - epsilon, it exploits the highest-value action argmax_a Q(s, a). Over training episodes, epsilon decays toward a small floor (such as 0.01).

Comparison Table: Dynamic Programming vs. Monte Carlo vs. Q-Learning vs. SARSA

Algorithm Family	Model-Free or Model-Based?	Policy Type	Update Timing	Key Strength & Limitation
Value / Policy Iteration (DP)	Model-Based (Requires known P(s'|s,a))	Optimal planning	Full sweep over all states	Exact convergence, but requires complete environment transition matrix
Monte Carlo (MC) Control	Model-Free	On-Policy / Off-Policy	End of complete episode only	Unbiased returns, but high variance and requires episodic termination
SARSA (TD On-Policy)	Model-Free	On-Policy (learns safe behavior policy)	Every single time step (TD(0))	Safer in risky environments (like Cliff Walking) by accounting for exploration
Q-Learning (TD Off-Policy)	Model-Free	Off-Policy (learns optimal greedy policy)	Every single time step (TD(0))	Directly learns optimal Q*(s,a) while exploring with any behavior policy

Hands-On Python Implementation: Tabular Q-Learning Agent

The self-contained Python script below implements a complete Q-Learning agent navigating a 1D/Gridworld state space with decaying epsilon-greedy exploration and Bellman TD updates:

import numpy as np

# 1. Define a 6-state corridor environment (State 5 is the terminal goal with +10 reward)
n_states = 6
n_actions = 2  # Action 0 = Move Left, Action 1 = Move Right
Q_table = np.zeros((n_states, n_actions))

# Hyperparameters
alpha = 0.1          # Learning rate
gamma = 0.95         # Discount factor
epsilon = 1.0        # Initial exploration probability
epsilon_decay = 0.98
epsilon_min = 0.05

def step_environment(state, action):
    next_state = max(0, state - 1) if action == 0 else min(n_states - 1, state + 1)
    if next_state == n_states - 1:
        return next_state, 10.0, True   # Goal reached (+10 reward, terminal)
    return next_state, -0.1, False      # Small step penalty (-0.1) to encourage shortest path

# 2. Train Q-Learning Agent across 300 episodes
rng = np.random.default_rng(42)
for episode in range(300):
    state = 0
    done = False
    while not done:
        # Epsilon-greedy action selection
        if rng.random() < epsilon:
            action = int(rng.integers(0, n_actions))
        else:
            action = int(np.argmax(Q_table[state]))
            
        next_state, reward, done = step_environment(state, action)
        
        # Bellman Temporal-Difference Q-Learning Update
        best_next_q = 0.0 if done else np.max(Q_table[next_state])
        td_target = reward + gamma * best_next_q
        Q_table[state, action] += alpha * (td_target - Q_table[state, action])
        state = next_state
        
    epsilon = max(epsilon_min, epsilon * epsilon_decay)

print("Learned Optimal Q-Table (Columns: [Move Left, Move Right]):")
print(np.round(Q_table, 2))

Frequently Asked Questions (FAQ) on Reinforcement Learning Basics

What is the difference between On-Policy (SARSA) and Off-Policy (Q-Learning)?

Q-Learning is off-policy because its Bellman update uses max_a Q(S_{t+1}, a)—evaluating the greedy optimal policy regardless of which exploratory action the agent actually takes next. SARSA is on-policy because it updates using the actual next action Q(S_{t+1}, A_{t+1}) chosen by the current epsilon-greedy policy.

When do we replace Tabular Q-Learning with Deep Q-Networks (DQN)?

When the state space is continuous or astronomically large (such as video frames or robotics joint angles), storing a discrete lookup table Q[s, a] in memory becomes impossible. Deep Q-Networks (DQN) approximate Q(s, a; theta) using a deep neural network stabilized by Experience Replay and Target Networks.