Policy gradients

Learn the policy directly

Value-based methods learn how good each action is, then choose the best. Policy gradients skip the middle step. The policy is a parameterised distribution over actions, and learning means changing its parameters so that the expected return goes up. There is no table to consult at decision time, only the distribution the parameters define. The output is a probability for each action, and the parameters are shared across states.

The representation is often a neural network that outputs logits, one per action, turned into probabilities by a softmax. For continuous actions it outputs the mean and spread of a Gaussian. Either way, the object being optimised is the policy, not a table of values. That change of object is the whole idea; everything else is calculus. The logits are a coordinate system for behaviour, and the softmax is the map back to probabilities.

The log-derivative trick

The objective is an expectation over actions sampled from the policy, and the parameters appear inside that distribution. Differentiating through a sample is awkward, so the gradient is rewritten using the identity that the derivative of a log is the derivative divided by the value. The identity is not a trick specific to RL; it is the score function estimator, used wherever a gradient must pass through a sample. It converts a derivative of an expectation into an expectation of a derivative.

The result says something intuitive: the gradient is the return multiplied by the gradient of the log probability of the action taken. Actions that led to high return get their probability increased; actions that led to low return get it decreased. The return is the weight, not the target. Nothing tells the policy what the right action was; the update only reweights what it happened to try. The size of the step is the learning rate, and the direction is the score.

One update in PyTorch

import torch
from torch.distributions import Categorical

logits = torch.tensor([0.2, 0.1, -0.3], requires_grad=True)
dist = Categorical(logits=logits)
action = dist.sample()
advantage = torch.tensor(1.7)

loss = -dist.log_prob(action) * advantage
loss.backward()
print(logits.grad)

Variance is the enemy

The update is unbiased, which is a promise about the average, not about any single run. A return of ten might come from a good action or from a lucky environment, and the gradient cannot tell the difference. The result is an estimator with high variance and a learning curve that looks like a seizure. Unbiased and useless is a real combination, and it is the default here. A single episode can move the policy a long way in the wrong direction.

The standard cure is a baseline: subtract a value that does not depend on the action, usually an estimate of the state’s value. Subtracting a constant leaves the expected gradient unchanged, because the expected score is zero, but it removes the part of the return that was going to happen anyway. What remains is the advantage. A good baseline can shrink the variance by an order of magnitude without changing the answer in expectation.

Where policy gradients win

Continuous control is the clearest case. A Q-table cannot represent a torque, and taking the maximum over a continuous action set is an optimisation inside every update. A Gaussian policy handles it directly, and the gradient flows through the sampled action. The policy can also keep a standard deviation, so the amount of exploration is learned rather than scheduled. The action space can be a vector, and the policy handles each dimension together.

They also suit problems where the best behaviour is genuinely stochastic, such as a game where unpredictability is an advantage. The cost is sample efficiency: policy gradients usually need more interaction than a well-tuned value method, which is why actor-critic methods learn a value function as well and use it as the baseline. The critic does not choose actions; it only makes the actor’s gradient less noisy.