Failure recovery

Failure is the normal path

A tool times out, a provider rate-limits, a model emits JSON with a trailing comma, a search returns nothing. None of those is exceptional in a system that makes ten calls per request; they are the ordinary weather. An agent that treats every error as fatal will fail on inputs that a person would have handled without noticing. The difference between a resilient agent and a brittle one is mostly how many of these it expects.

The first step is classification. A transient error is worth retrying: a timeout, a 429, a dropped connection. A permanent error is not: a missing argument, an unknown tool, a permission denial. An ambiguous error is the dangerous one, because the action may have succeeded before the connection broke, and a retry would do it twice. That third category is where most duplicate side effects come from. Classification should be a function in the code, not a judgement the model makes after the fact.

Retry only the transient

Retry with a bounded number of attempts and a growing delay. Exponential backoff with jitter stops a hundred clients from retrying in lockstep and turning a blip into an outage. The cap matters as much as the growth: without it, attempt five waits longer than the user is willing to, and the request has already failed in the only way that matters. The user should see a wait, not a spinner that never resolves.

Never retry a validation error. The model sent a malformed request; sending the same request again produces the same rejection and burns the budget. Feed the error back to the model instead, so the next attempt is different. A retry that repeats the identical call is a loop wearing a disguise, and the step budget is the only thing that ends it. The feedback is the point: the second attempt has information the first one did not.

A retry policy in LangGraph

from langgraph.graph import StateGraph
from langgraph.pregel import RetryPolicy

graph = StateGraph(State)
graph.add_node(
    "search",
    search,
    retry_policy=RetryPolicy(
        max_attempts=3,
        initial_interval=0.5,
        backoff_factor=2.0,
        jitter=True,
        retry_on=TimeoutError,
    ),
)

Recovery is a decision

When the retries are exhausted, the agent needs a decision rather than another loop. Four options are usually enough: replan with what is known, degrade to a cached or partial answer, escalate to a human, or stop with a clear error. Each of them is a branch in the loop, and each should be visible in the trace. Write the branch before the failure, so the decision is not made under pressure at 3am. The branch should be chosen by code that knows the budget, not by the model.

The worst outcome is a silent fallback that returns something plausible. A degraded answer must say it is degraded, and the caller must be able to tell that the run did not complete. Honest failure is cheaper than a wrong success, because the wrong success is discovered by a user rather than by a dashboard. A status field in the response is the smallest version of that honesty.

Resume from the last good step

A multi-step run should checkpoint after each completed step: the step name, its output and the side effects it caused. When the process restarts, it resumes from the last checkpoint instead of repeating the search, the read and the charge. That is the difference between a recovery and a second attempt, and only one of them is safe. A checkpoint that records only a step name is not enough to resume safely. The checkpoint is also the record that makes a replay auditable.

A timeout is ambiguous by nature: the request may have reached the server and succeeded. The safe move is to ask the system of record — did the charge land? — rather than to retry blindly. Recording an intent before the call and reconciling after it is what makes that question answerable, and the record is also what the trace shows a reviewer. The intent record is cheap; the duplicate charge is not.