Reinforcement learning from the ground up

Rewards, values and policies, computed by hand

Advanced
10 chapters
161 min read

How a program learns from consequences: returns and discounts, bandits and regret, Markov decision processes, value iteration, Q-learning, the exploration trade-off and policy gradients. Every update is written out in plain Python, and the capstone solves one grid world twice — once by planning, once by experience.

What You'll Learn

Write a return with a discount and explain what the discount changes
Decide what belongs in the state, and spot an aliased observation
Estimate bandit values and measure a policy by its regret
Write a transition table and evaluate a policy with the Bellman equation
Plan an optimal policy with value iteration
Learn action values from experience with Q-learning
Choose an exploration rule and say what it costs
Derive and implement a REINFORCE update with a baseline
Solve and evaluate a grid world agent with a measured success rate

Prerequisites

Comfortable with Python functions, dictionaries and loops
Basic probability: expected value and independence
No ML library is needed — every example is plain Python

Vocabulary

The terms the chapters assume, in plain words.

action: One choice the agent can make at a given moment, from a fixed set available to it.
agent: The learner: it sees a state, chooses an action and receives a reward.
discount factor: A number between zero and one that scales down later rewards, so a reward now is worth more than the same reward later.
environment: Everything outside the agent that responds to its actions and hands back a reward and a new state.
episode: One complete run from a starting state until the task ends, after which the agent starts again.
exploitation: Choosing the action that looks best right now, using only what has been learned so far.
exploration: Trying an action that may not be best, in order to learn whether something better exists.
Markov decision process: A task described by states, actions, rewards and the chances of moving between them, where the current state is enough to decide from.
policy: A rule that picks an action for each state; learning changes this rule, not the world.
Q-value: How good an action is in a given state: the average return from taking it and then following a fixed rule.
return: The total reward collected from this point onward, with later rewards scaled down by the discount factor.
reward: A number the environment sends after an action, saying how good that step was.
state: What the agent uses to decide; it must be complete enough that two situations needing different actions look different.
value: How good a state is: the average return from being there and then following a fixed rule.

Course Content

01
Rewards and states
An agent learns from a single number and a description of where it is. Getting those two right matters more than the algorithm that follows.
Advanced · 15 min read
Not started
02
Multi-armed bandits
One state, several actions and no map. The bandit strips reinforcement learning to its smallest form, where exploration can be studied on its own.
Advanced · 14 min read
Not started
03
Markov decision processes
Add states and transitions to a bandit and you get the standard model of sequential decisions: a table of consequences, a policy, and one equation that ties them together.
Advanced · 16 min read
Not started
04
Value iteration
If you know the transition table, you can compute the best policy by sweeping one equation until the numbers stop moving. Planning, with no learning involved.
Advanced · 16 min read
Not started
05
Q-learning
The transition table is a luxury. Q-learning learns action values from experience alone, bootstrapping its way to a policy without ever modelling the world.
Advanced · 17 min read
Not started
06
Exploration versus exploitation
Every action is either information or income. The rules that balance the two decide how fast an agent learns, and how much it pays while learning.
Advanced · 15 min read
Not started
07
Policy gradients
Instead of learning values and deriving a policy, learn the policy itself by pushing probability toward the actions that paid. The gradient is an average, and the variance is the price.
Advanced · 17 min read
Not started
08
Actor-critic methods
Keep the policy gradient and add a critic that learns the value of a state. The critic supplies the baseline, the actor uses it, and the whole method becomes one loop with two losses.
Advanced · 17 min read
Not started
09
Reward hacking
An optimiser does what the reward says, not what you meant. The gap between the two is where specification gaming lives, and closing it is an evaluation problem before it is a modelling problem.
Advanced · 16 min read
Not started
10
Capstone: a grid world agent
One small world, solved twice: once by planning with a known model, once by learning from experience. Then the only result that matters, a measured success rate.
Advanced · 18 min read
Not started