Actor-critic methods

Two jobs in one loop

A policy gradient learns what to do and pays for it with variance. Actor-critic keeps the policy but adds a second model that learns how good the current state is. The actor is the policy: it turns a state into a distribution over actions. The critic is a value function: it turns a state into a number. The actor chooses, the critic judges, and the judgement comes back as a learning signal.

The two models are trained from the same stream of experience, which is what makes the method practical. There is no wait for an episode to end and no separate regression dataset. Every step produces a reward, a next state and therefore a target for the critic and a weight for the actor. One loop, two losses, and a shared notion of what just happened. The critic can share a trunk with the actor, but it keeps its own output and its own loss.

The TD error is the advantage

The critic’s error at one step is the temporal-difference error: the reward plus the discounted value of the next state, minus the value of the state just left. If the sum is positive, the step went better than the critic expected. That number is also an estimate of the advantage of the action taken, which is exactly the quantity a policy gradient wants. The update is the score of the action multiplied by the TD error.

This is one-step actor-critic. It replaces the sampled return of REINFORCE with a bootstrapped estimate, so the weight is available immediately and its variance is much smaller. The price is bias: the critic’s value at the next state is an estimate, and any error in it flows into the actor’s gradient. The method trades a noisy truth for a steady approximation, and most of its tuning is about keeping that approximation honest. A critic that is merely slow is usually better than one that is clever and unstable.

The two losses, in PyTorch

import torch
from torch import nn

actor = nn.Sequential(nn.Linear(4, 64), nn.Tanh(), nn.Linear(64, 2))
critic = nn.Sequential(nn.Linear(4, 64), nn.Tanh(), nn.Linear(64, 1))
actor_opt = torch.optim.Adam(actor.parameters(), lr=3e-4)
critic_opt = torch.optim.Adam(critic.parameters(), lr=1e-3)

state, action, reward, next_state, done = batch
value = critic(state).squeeze(-1)
next_value = critic(next_state).squeeze(-1) * (1.0 - done)
td_error = reward + 0.99 * next_value - value

log_prob = torch.log_softmax(actor(state), dim=-1)
chosen = log_prob.gather(1, action[:, None]).squeeze(-1)

actor_loss = -(chosen * td_error.detach()).mean()
critic_loss = td_error.pow(2).mean()

actor_opt.zero_grad()
actor_loss.backward()
actor_opt.step()

critic_opt.zero_grad()
critic_loss.backward()
critic_opt.step()

Bias, variance and the bootstrap

The choice between one step and a full return is a dial, not a binary. An n-step target bootstraps after n rewards, and generalized advantage estimation averages every n-step advantage with a decay. A lambda near zero gives the low variance and high bias of one-step updates; a lambda near one gives the unbiased, noisy return of Monte Carlo. The dial is usually set by the environment: long episodes and sparse rewards push it up, short episodes and dense rewards let it down.

The critic needs its own care. A critic trained on the same learning rate as the actor can move the baseline faster than the policy can respond, and the advantage becomes a measurement against a target that keeps running away. Separate learning rates, a target network or a slow-moving average all help. Watch the explained variance of the critic: when it falls toward zero, the advantage is mostly noise and the actor is learning from nothing.

What the pair buys

Actor-critic is the shape of most modern deep RL. A2C runs many environments in parallel and averages the updates; PPO adds a clipped ratio so a single batch cannot move the policy too far; SAC adds an entropy term and learns a soft value function. The names differ, but each one has an actor that chooses and a critic that scores. The pattern scales to continuous actions, image observations and long horizons because both parts are function approximators. That is why the same code often serves a simulated robot and a game player with only the observation shape changed.

The cost is real: two models to train, two sets of hyperparameters and a critic that can be wrong in ways the actor cannot see. On small tabular problems a Q-table is simpler and often better. Actor-critic earns its place when the action space is large or continuous, when the policy must be stochastic, or when the value function is easier to learn than the policy. Reach for it for those reasons, not because it sounds more advanced.