Rewards and states

The loop with a score

Reinforcement learning is the study of a program that acts in an environment and learns from the consequences. At each step it sees a state, chooses an action, receives a reward and lands in a new state. Nothing tells it which action was correct; the only feedback is the number that arrived, and the next situation it produced. The loop is the whole of it: no labels, no teacher, only the consequence of the last choice.

That number is deliberately thin. A chess engine sees +1 for a win and 0 for a draw, and must work out which of its forty moves deserved the credit. Supervised learning is handed the answer at every step. Reinforcement learning is handed an outcome at the end and has to reconstruct the causes, often from a signal that arrives many actions later.

Return, not reward

A single reward is a poor guide because it says nothing about the future. The quantity the agent maximises is the return: the rewards from this step onward, each discounted by how far away it is. With a discount of 0.9, a reward ten steps away counts about a third as much as one received now. Two policies can collect the same total and differ only in when they collect it, which makes the discount a statement about time, not just arithmetic.

The discount is not a technicality. It expresses impatience, keeps an infinite sum finite, and decides how far ahead the agent bothers to plan. A discount near one makes a distant goal worth pursuing; a discount near zero makes the agent grab whatever pays immediately, even when waiting would pay more. The number is a claim about how much the future is worth today, and it is usually the first thing to tune.

The interface every environment shares

import gymnasium as gym

env = gym.make("FrozenLake-v1", is_slippery=False)
state, info = env.reset(seed=0)

for step in range(10):
    action = env.action_space.sample()      # 0 left, 1 down, 2 right, 3 up
    state, reward, terminated, truncated, info = env.step(action)
    print(step, state, reward, terminated)
    if terminated or truncated:
        break

The state is a decision, not a fact

The environment has some true condition, but the agent rarely sees all of it. What it sees is the observation, and the observation becomes the state when it is enough to decide from. If two situations look identical but call for different actions, the observation is not a state yet. The test is simple: can two different histories produce the same observation and still need different behaviour? If the answer is yes, no algorithm can recover what the observation threw away.

The fix is memory: stack the last few observations, or carry a summary of them. A state that includes the cue seen three steps ago can tell apart two corridors that look the same today. Choosing what goes into the state is the first design decision in every problem, and it is usually made before any learning starts. A richer state makes the problem Markov; a poorer one makes it impossible, and no amount of training will close the gap. The observation is a fact; the state is a choice.

Reward is a specification

The reward function is the only thing the agent optimises, so it is the only thing it will honour. Ask for paperclips and you get paperclips, plus a factory that converts everything else into them. The failure is not malice; it is a faithful solution to a badly written objective. The agent cannot read the comment that says what you actually meant, which is why reward design is a specification exercise rather than a tuning exercise.

Good rewards are dense enough to learn from, bounded enough to compare, and shaped so that the intended behaviour is the cheapest way to collect them. Every shortcut you can imagine is a test case, and the agent will find the ones you did not imagine. Write the reward, then write down three ways to cheat it, then close those holes before training starts.