Capstone: a grid world agent

A world small enough to check

The grid is four by four. The agent starts in the top-left corner, the goal is the bottom-right, a wall sits at row one column one, and a pit waits at row two column one. Every step costs a little, entering the goal pays one, and entering the pit loses one. The reward is small enough that the shortest safe path wins, and the pit is bad enough that a shortcut through it never does. The agent has four actions and a deterministic world, so every mistake is reproducible.

Small is a feature. You can print every value, read every arrow and compare the learned policy with the optimal one by eye. A world that fits on a screen is the right place to find a bug in the loop, before the same bug hides inside a network. The wall and the pit are there to make the policy non-trivial: without obstacles, every greedy action points at the goal. The goal is reachable from every open cell, which is what makes the success rate meaningful.

Plan it first

Because the transitions are known, value iteration gives a reference answer. Sweep the Bellman optimality backup until the largest change is tiny, then take the greedy action at every cell. That policy is optimal for this grid, up to ties, and it is the yardstick for everything that follows. The sweep count is not magic; it is chosen so the remaining change is far below the step cost. A tolerance smaller than the reward scale is usually enough.

The reference is worth the ten lines it costs. Without it, a learned policy that walks into the wall looks like a training problem; with it, the difference is a fact you can point at. It also gives you the optimal return from the start, which is the number the learned agent should approach.

Evaluating a trained policy

from stable_baselines3.common.evaluation import evaluate_policy

mean_reward, std_reward = evaluate_policy(
    model,
    env,
    n_eval_episodes=20,
    deterministic=True,
)
print(round(mean_reward, 2), round(std_reward, 2))

Learn it with experience

The second agent is given no model. It starts with every Q value at zero, moves with an epsilon-greedy behaviour policy, and updates one entry per step. The reward signal is the same, but the agent has to discover the wall and the pit by meeting them. Its exploration is what turns a random walk into a map, and its learning rate decides how quickly the map is rewritten. The table is small enough to print, which makes a wrong update visible immediately.

The comparison is the lesson. Planning needs fewer interactions but a model; learning needs no model but many more interactions, and its answer depends on how much it explored. Neither is better in general, and the right choice is a statement about what you have. In this world, planning is exact and cheap; in a real one, the model may be the expensive part.

Report what you measured

A result is a success rate, a number of steps and a description of the seeds. “It works” is not a result. State how many evaluation runs, from which starts, with what step limit, and what counts as a failure: the pit, the step limit, or a loop. A success rate without the number of attempts is a story, not a measurement. Name the seeds too; a different seed is a different experiment.

Then report the failures. An agent that reaches the goal in twelve steps but falls into the pit from one start is not a success with a caveat; it is a policy with a hole. The capstone is finished when the report says where the hole is, not when the average looks good. The same discipline applies to a model that is ninety percent right: the interesting part is the ten. Write the failure case down before you tune it away.